Voice conversion on the emotion trajectories — what was done, and what it measures

Every chain on this site is several separate recordings of what the mining believed was one speaker, played one after another. They were only ever filtered for speaker similarity, never made into one voice, so as a chain plays you can hear the voice itself shift between segments.

Voice conversion takes a clip and re-speaks it in someone else's voice. The words, the timing, the pauses and most of the delivery come from the original clip; the vocal identity — the thing that makes a voice recognisably one person — comes from a short reference recording of the target. Here the target is always segment 1 of that same chain. So the chain ends up sounding like segment 1's speaker throughout, saying what each of the later clips actually said.

What it fixes. Only one thing: who the voice sounds like. If two clips of a chain sounded like two different people, they should now sound like one.

What it does not fix. It does not make the clips one continuous performance — they are still separate takes, recorded at different moments, with different room and microphone, and the joins are still joins. It does not correct the mining: if the wrong clips were chained together, they are still the wrong clips.

What it can break. Delivery and identity are not cleanly separable. A converted clip is re-synthesised, so its pacing, emphasis and emotional colour can move — which matters enormously here, because the only thing these chains exist to demonstrate is an emotional change. If conversion flattened the emotion, these pages would be worse than the ones they copy and there would be no reason to ship them. That is the first thing section 5 measures, and it is the reason the emotion re-scoring runs on every single segment rather than on a sample.

For each chain, segment 1 was loudness-prepared (EBU R128 integrated −23 LUFS, peak ceiling 0.95) and used as the target voice. Segments 2…k were tokenised and re-synthesised onto it with ChatterboxVC (the s3gen voice-conversion path, one candidate, seed derived from the chain id so the run is reproducible), then restored with SIDON, which also takes the output to 48 kHz. TF32 is enabled for s3gen (measured free) and explicitly disabled around SIDON (measured ~15 dB cost).

Then, exactly as the uncorrected pages do, every segment is scaled to −20 dBFS RMS before the chain is concatenated, a 0.15 s silence marks each boundary, and the peak guard is applied once to the finished chain so the level relationship between segments survives.

1,304 chains and 3,438 segments were converted onto a new voice (4,742 segments passed through the pipeline in total, counting the anchors used for the third render below), in 1.5 GPU-hours. 0 chains failed.

The first version of these pages converted only segments 2…k and left segment 1 exactly as recorded. Listening found the problem the measurement had also found: the chain then glues one raw 24 kHz recording to k−1 neural 48 kHz re-syntheses, and the acoustic — most audibly the reverb — changes at the first join. The cosine across that first join measured 0.836 before and 0.728 after, the single worst thing conversion did anywhere in the build.

So segment 1 is now converted too, with itself as the target, and that render is the default on every card. The earlier one is kept behind an expander on each card, because it retains more of the emotional move (96 % against 90 %) — a real trade-off rather than a strict improvement, and the choice stays visible.

Crossfades replace the silence. Segments are now joined with an equal-power crossfade — cos/sin, so that the two sides sum to constant energy. A linear crossfade sums to less than unit energy at its midpoint and dips audibly on uncorrelated material, which two different recordings always are.

The nominal length is 150 ms, shortened where it would do damage: never more than a quarter of either neighbouring segment, and cut to 100 ms wherever the incoming segment starts loud or the outgoing one ends loud, since a crossfade attenuates the incoming signal exactly where a word-initial consonant would be.

Across 6,995 joins: 5,088 took the full 150 ms, 1,382 were shortened for a hot onset and 525 for a hot tail. The shortest fade anywhere in the build was 100 ms — the 25 % duration cap never bound, because the shortest segment here is 3 s. Each card prints its own fade lengths.

The chain is normalised as one signal. Previously every segment was scaled to −20 dBFS RMS independently, which forces them all to the same loudness and erases the dynamics between them. Now a single gain is applied to the finished chain and the peak guard is applied once after it.

That would have been unsafe on the original recordings — those carry up to 8 dB of level steps from the MOSS encoder's own per-clip normalisation, an artifact of the pipeline rather than the performance. It is safe here because SIDON returns every converted segment at a consistent level, and the result is measured rather than assumed: the largest step between adjacent segments is a median 1.17 dB, 3.39 dB at the 95th percentile. For comparison the raw, un-normalised chains reach 17.96 dB. What is left is the conversion's own dynamics, not a per-clip reset.

Measured with Orange/Speaker-wavLM-id, the 250-dimensional speaker verification embedding — the same space the min_cos_* numbers on the original pages live in, and the space in which the mining's 0.80 threshold is meaningful. (Not -tbr, which is a 128-d timbre embedding on an unrelated scale.) For every chain: the worst cosine between any segment and segment 1, and the worst between any two neighbours. Higher is more like one person. All figures are over the same 1,304 chains.

The overall before/after average is close to useless here, because it averages two opposite effects. The conversion pulls every chain towards a common level of identity agreement: it rescues chains whose segments were plainly different people, and it damages chains that were already one voice. Splitting the chains by their original worst-to-anchor cosine shows it directly. Median values; “improved” is the share of chains in that band whose number went up at all.

The number above mixes two things a listener does not hear the same way: how well the conversion unified the voice, and how far neural re-synthesis moves a clip in this embedding space at all — which it does even when the imitated voice is right. In the shipped render the second effect is concentrated in one place: the first join, where a raw recording meets a re-synthesis.

So the full segment×segment cosine matrix was recomputed for every one of the 1,304 chains and split up. seam 1 is that first join. after the first join is the worst agreement among segments 2…k measured against each other — note that this is identical for the corrected and uniform renders by construction, since both contain the same converted segments 2 onward. spread is how far the identity wanders across the whole chain, max minus min over every pair.

Every segment, original and converted, was re-scored through the same emotion stack the corpus was annotated with (laion/BUD-E-Whisper + Empathic-Insight-Voice-Plus) and put on the same tie-aware corpus-percentile scale the mining used (ecdfm over the exact 127 M-utterance global histograms). Then d_b — the end-to-end move on the axis the chain was mined for — was recomputed exactly as mine5.py defines it.

Both sides are scored with the same scorer, so any offset between this scorer and the stored corpus annotation cancels. As a check that it does: re-scoring the originals reproduces the mined d_a/d_b to three decimals.

Retention is d_b(after) / d_b(before), kept signed: a conversion that reverses a trajectory has destroyed it, and must not be able to score 1.0 by flipping. Chains whose original delta is under 0.02 are excluded, because a ratio against zero is noise.

The check, run first. Re-scoring the original audio and comparing the recomputed d_b against the value stored by the miner: median absolute error 0.0003, 95th percentile 0.0215, over 1,304 chains — on a scale where a tier threshold is 0.20. The re-scoring reproduces the mining, so the before/after numbers below are a measurement of the conversion and not of a scorer mismatch.

The Empathic-Insight overall quality head, per segment, on the same audio the players serve. A cheap proxy, not a listening test, and it is scored on 48 kHz SIDON output against a 24 kHz original — so part of any gain here is bandwidth, not conversion. Read it as “the conversion did not wreck the audio”, not as “the conversion improved it”.

The corpora differ enormously in how well “speaker” was ever defined. mls ships real speaker identities; everywhere else it meant “same source recording”. podcast is the loose one — only about 54.6 % of consecutive clips inside one nominal podcast speaker track are actually the same person — so it is where a conversion pass has the most to prove.

1,140 segments from 300 chains, stratified over the six corpora, transcribed with openai/whisper-large-v3-turbo. Character error rate, not word error rate: these chains include Korean, Japanese and Chinese, where word error rate is not defined in any useful way.

The headline number uses the ASR of the original clip as the reference, not the corpus transcript. Whisper's own error on this material is large and multilingual and would swamp the effect being measured; comparing converted-ASR against original-ASR cancels it and answers the actual question — did the conversion change what is said? The language detected on the original is forced on the converted side, so a language flip cannot masquerade as a transcription difference.

The alternative to converting a mismatched chain is to not ship it: apply a hard identity cut and keep only chains that already sound like one person. Whether that is better is answerable from what is already measured here, because the conversion's effect is known separately for the chains each cut would keep and the chains it would drop.

The two embedding spaces, on the same chains. The corpus-wide counts below come from two different populations, so their percentages cannot be compared with each other. The grid is the one place both numbers exist for the same chains: on 298 of them (all emolia, the only corpus here carrying a stored timbre anchor), -id ≥ 0.80 is best matched by -tbr ≥ 0.788 (85.9 % agreement), and the standing calibration of 0.795 agrees on 84.9 %. That confirms the calibration rather than revising it, and it means a timbre cut in the 0.75–0.80 range is slightly more permissive than the identity test at 0.80, not less. Correlation between the two spaces is r = 0.923.

This has to be done store-against-store: the identity cosine recomputed for this build is measured on the rendered concatenation and sits a median -0.0184 from the corpus store (r = 0.967). Pairing across the two sources instead moves the apparent equivalent threshold by nearly 0.1 — a measurement offset masquerading as a disagreement between the models.

Corrected audio is under vcaudio/ (the corrected render) and vcaudio_u/ (the uniform render); the untouched originals stay under audio/. Every number on these pages is in vc_grid.json, per chain, machine-readable. Bulk artefacts — the converted segments as float16, the per-shard measurement files — are on the cluster under traj2/vcgrid/out.