# S2 Pro vs S2.1 Pro: 40-voice blind listening study

Completed 2026-10-26T17:03:03.498750+00:00. Gemini 3.1 Pro evaluated all 120 matched pairs twice, plus 8 identical-audio controls twice: 256 complete schema-validated reviews. All 240 generated WAVs passed file-hash and full-decode checks.

**Primary naturalness difference, S2.1 Pro minus S2 Pro: -0.079 points on the 1–5 scale; descriptive 95% character-bootstrap interval [-0.183, +0.025].**

No clear S2.1 Pro naturalness advantage in this English audition: the primary interval crosses zero, and 64/120 pairs changed preference across presentation order. Identical-audio controls tied 16/16 times, but do not erase order sensitivity on differing recordings.

The statistical unit is a character/voice reference (40 total: 20 male, 20 female). Each model score is averaged over three scripts and both listening orders before character-level analysis. Reversed judgments are not independent voice samples. This is one AI judge’s assessment, not a human preference or business-outcome experiment.

| Stable pair preference | Count |
|---|---:|
| S2 Pro | 27 |
| S2.1 Pro | 20 |
| Tie in both orders | 9 |
| Order-sensitive / inconsistent | 64 |

A stable winner requires the same underlying sample to win in both presentation orders. Tie versus winner and opposite winners remain inconsistent; none is forced into a win.

## Character-weighted scores

| Metric | S2 Pro | S2.1 Pro | Difference | Descriptive 95% interval | Characters |
|---|---:|---:|---:|---|---:|
| Naturalness | 3.254 | 3.175 | -0.079 | [-0.183, +0.025] | 40 |
| intelligibility | 3.992 | 3.987 | -0.004 | [-0.021, +0.013] | 40 |
| expressiveness | 3.242 | 3.221 | -0.021 | [-0.138, +0.092] | 40 |
| audioQuality | 3.750 | 3.692 | -0.058 | [-0.125, +0.000] | 40 |
| textFidelity | 4.050 | 4.021 | -0.029 | [-0.058, -0.004] | 40 |

### Descriptive gender slices

| Voices | Characters | S2 Pro naturalness | S2.1 Pro naturalness | Candidate minus control |
|---|---:|---:|---:|---:|
| Male | 20 | 3.225 | 3.150 | -0.075 |
| Female | 20 | 3.283 | 3.200 | -0.083 |

Gender slices are descriptive; they do not establish separate winners.

All 40 selected characters remain visible. Unassessable sample-reviews: 0; exact case/order/sample list: []. Complete-case metric means require all six scores for each model in a character, with no null-skipping or imputation.

Only naturalness was predeclared primary. Secondary intervals are descriptive and unadjusted for multiple comparisons. No p-value, significance declaration or rollout decision is derived from this table.

## Judge reliability

Identical audio: **16/16 tie responses**; 0 non-tie responses; 0 unassessable. 8/8 control pairs tied in both orders. Non-tie judgments would be retained rather than filtered. Passing this identical-audio check does not establish order invariance for differing recordings.

Mean absolute score difference when A and B contained identical bytes: naturalness=0.0, intelligibility=0.0, expressiveness=0.0, audioQuality=0.0, textFidelity=0.0.

The 64 inconsistent real pairs comprise 21 opposite-winner decisions and 43 tie-versus-winner disagreements.

Presented-position preferences across the 240 real judgments: A=65, B=114, tie=61, unassessable=0.
Pairs preferring the first-presented sample in both orders: 4; second-presented in both: 17. These are order-sensitive decisions, not stable model wins.

Confidence values are the mean of two uncalibrated model self-reports, not probabilities of a real-world win.

## Direct WebSocket timing

Direct cold WebSocket requests from this test host; no warm connection pool, not end-to-end production call latency.

| Requested model | Samples | Median first audio incl. handshake | p95 incl. handshake | Median after start event | Failed attempts |
|---|---:|---:|---:|---:|---:|
| S2 Pro | 120 | 477.6 ms | 580.8 ms | 242.8 ms | 0 |
| S2.1 Pro | 120 | 506.3 ms | 1076.2 ms | 265.8 ms | 0 |

Timings describe successful attempts. Failed attempts are counted separately; requests were not concurrent within a matched pair.

## Selection and safeguards

Top 20 per gender by distinct real registered callers in positive-duration private call transcripts over last 14 days; public non-retired characters with active Fish voice. Tie-break by call count then character id.
Popularity window: 2026-10-12T16:42:47.757Z to 2026-10-26T16:42:47.757Z. Forty character voices were selected before synthesis; all remain included.

Both models use the same reference voice and same synthetic script per pair. Three universal English scripts cover everyday conversation, tagged emotional delivery, and names/numbers/pacing. They contain no private user utterances. The exact scripts and predeclared design are in `protocol.json`.

Judge receives opaque pair code, task directions, exact text and two audio files; no model/character names, portrait, gender, popularity, timings or file paths. Same audio reviewed in reverse order in separate requests. Orchestrator knows mapping; not independent human double-blind.

Selection, scripts, masking, controls and aggregation rules were written before synthesis. Exact transport settings and retry details were appended during generation after the pilot; scoring criteria and samples were not changed.

Masked-result freeze SHA-256: `bea31cc2aa8142d315a1df3845f41c732f029f773dfc9c4d8edb9227bce30f21`. `private/masked-results.json` contains the labels and unaltered reviews before the aggregation code looked up model assignments. This protects the analysis order; it does not make the orchestrator ignorant of the assignment.

Model versions: {"gemini-3.1-pro-preview": 256}. Successful response token usage: {"promptTokenCount": 414054, "candidatesTokenCount": 103435, "thoughtsTokenCount": 107210, "totalTokenCount": 624699}. Timeouts may have incurred unreported usage. There were 0 accepted evaluations completed after a retry.

## Limits

- One generation per model/voice/script; stochastic synthesis is not replicated.
- Three fixed English scripts do not establish multilingual quality.
- 40 popular characters represent 40 voice references, not the full catalog.
- One AI judge; reversed order does not create independent judges or subjects.
- No pronunciation/voice identity ground-truth reference; do not claim voice-clone fidelity.
- Real provider model execution is not independently attested by the model request header; the experiment preserves returned headers where available.
- Cold direct-API measurements do not include LLM thinking, production transport, client playback or a warmed TTS pool.
- Scores cannot establish conversion, retention, user preference or safe universal rollout.
- Public gallery includes only selected public character names/portraits, aggregate popularity counts, synthetic audio and judge results. Voice IDs, credentials and raw production account records are excluded.

Artifacts: `gallery/` listening gallery; `results.csv` per-pair decoded scores; `summary.json` aggregate method and outcomes. Private raw model responses, synthesis receipts and masked evidence remain under `private/`. No production model configuration was changed.