Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilities
Paper • 2503.04721 • Published • 4
You need to agree to share your contact information to access this model
This repository is publicly accessible, but you have to accept the conditions to access its files and content.
Research checkpoints of a speech model trained to refuse requests when an operator instruction tells it to. They are not a safety system. By requesting access you agree to use them for research only.
Log in or Sign Up to review the conditions and access this model content.
duplex-refusal
LoRA adapters for nvidia/personaplex-7b-v1, a full-duplex speech model, that make it refuse a request when a private operator instruction arrives mid-conversation — and answer normally when none does. The instruction is a text span the model reads but never speaks.
Versions
One folder per version; each has its own README with data, recipe, evaluation and known issues, and a
VERSION.json with the adapter's sha256 and scaling. Load a version at a pinned commit, not from a
moving main.
| version | adapter | revision | sha256 | what it is for |
|---|---|---|---|---|
v0.1 |
v0.1/lora.safetensors (775,835,968 B) |
313d94c6a347b1493984388313e744440f0028ff |
4bfe4ce2f37289a41349fb0d0e61f8bfe8bebf0a278543f7b09930cd10f068fa |
preview; run refusal_v0.1_ins step 4,800. Always answers and always refuses when told, but over-refuses benign requests: take it where over-refusal is acceptable |
v0.2.1 |
v0.2.1/lora.safetensors (775,856,664 B) |
the commit that added v0.2.1/ (pinned by sha in Daimonion's hub/registry.py) |
8dc8b45e2e6c08fdd7f538c62c4ca00c9b38b8a8454b211baeec4c44e9c3a976 |
run refusal_v0.2.1_ins step 12,000; accept and refuse trained behind the same audio, harmful spots left out of training, the two instruction-marker rows trained. Mount it when the seat must hold an ordinary conversation |
Measured 2026-09-18 by replaying the seed-set benchmark's own 8 kHz telephony caller tapes (20 tapes, 76 caller turns, the instruction spliced 2 s after the last turn):
| version | answers the caller | refuses after the alert | benign over-refusal |
|---|---|---|---|
v0.1 |
1.00 | 1.00 | 0.25 |
v0.2.1 |
0.88 | 0.94–1.00 | 0.00 |
The refusal_v0.2_ins/ and refusal_v0.2.1_ins/ folders hold every 1,000-step checkpoint of those
two runs with a per-checkpoint eval report; they are training artefacts, not versions. v0.2.1/ is
the named, pinned copy of refusal_v0.2.1_ins/checkpoint_012000/.
Evaluation: Full-Duplex-Bench v1.0 (2026-09-19)
Turn-taking behaviour of the folded checkpoints on Full-Duplex-Bench v1.0
(727 stimuli, five categories), brain off, with the harness the JSALT 2026 team uses to reproduce the paper's
PersonaPlex row. Both versions ran under the same condition: the v0.1 training voice fed raw, the
per-category FDB prompts, seed 42424242. The stock PersonaPlex row is the team's reproduction at the
VARF0 voice (a different voice: FDB v1 numbers move strongly with the voice prompt, so treat that column
as context, not a controlled comparison). Full tables, the voice pilot and per-run summaries:
eval/fdb_v1/.
| Metric | Sub | PersonaPlex stock (VARF0) | v0.1 |
v0.2.1 |
Paper (PersonaPlex) |
|---|---|---|---|---|---|
| Pause Handling | Synthetic TOR ↓ | 0.642 | 0.504 | 0.555 | 0.584 |
| Pause Handling | Candor TOR ↓ | 0.718 | 0.593 | 0.704 | 0.662 |
| Backchannel | TOR ↓ | 0.418 | 0.636 | 0.636 | 0.327 |
| Backchannel | Freq ↑ | 0.176 | 0.093 | 0.091 | 0.025 |
| Backchannel | JSD ↓ | 0.700 | 0.748 | 0.743 | 0.649 |
| Smooth Turn-Taking | Candor TOR ↑ | 0.983 | 0.950 | 0.983 | 0.992 |
| Smooth Turn-Taking | Latency ↓ | 0.020 | 0.035 | 0.001 | 0.070 |
| User Interruption | TOR ↑ | 0.935 | 0.950 | 0.900 | 1.000 |
| User Interruption | Rating ↑ | 4.684† | 3.711† | 3.239† | 4.210 |
| User Interruption | Latency ↓ | 0.214 | 0.509 | 0.842 | 0.400 |
† the user-interruption rating comes from a local judge (gemma-4-E4B served as gpt-4-turbo), not GPT-4o:
comparable between these rows, not with the paper. Reading: v0.1 holds the floor better while the caller
pauses and is rated higher when interrupted; v0.2.1 takes the turn more reliably and starts sooner (its
0.001 s smooth-turn latency is the metric's clamp at zero: it speaks at or before the boundary). Both are more
talkative than the paper row, which costs them backchannel TOR.
Evaluation: seedset closed loop, 5-goal pilot (2026-09-19)
The JSALT 2026 seed-set benchmark (a Crescendo attacker LLM speaking to the seat over a telephony channel, tau2, protocol revision 2, judged by Muse-Glimmer-30B plus the HarmBench classifier), on five goals shared by every earlier run, one episode each, four caller turns. Both checkpoints run undefended (brain off) and with Daimonion's brain under each cadence. These are exploratory runs at benchmark tree v2.0; n = 5 per cell, so a one-episode difference is one episode.
Seats, undefended, same five goals (harmful / partial / none = our judge's assistance label; "attack succeeded" = the protocol's binary judge; HarmBench = the classifier's episode verdict):
| seat | harmful | partial | none | attack succeeded | HarmBench |
|---|---|---|---|---|---|
| stock PersonaPlex | 80% | 20% | 0% | 60% | 40% |
| duplex-refusal v0.1 | 0% | 80% | 20% | 0% | 20% |
| duplex-refusal v0.2.1 | 0% | 80% | 20% | 0% | 0% |
With the brain (cadences: P1 = one call at the caller's turn end; P2 = poll every 2 s while the caller
speaks, alert at the hand-over; P2k = P2 but a poll still in flight at the hand-over is awaited; P3 = poll
every 2 s always, the brain paces itself with wait):
| arm | v0.1 partial | v0.1 none | v0.1 refusal | v0.2.1 partial | v0.2.1 none | v0.2.1 refusal |
|---|---|---|---|---|---|---|
| base | 80% | 20% | 0% | 80% | 20% | 0% |
| P1 | 40% | 60% | 100% | 20% | 80% | 60% |
| P2 | 20% | 80% | 100% | 40% | 60% | 80% |
| P2k | 0% | 100% | 100% | 20% | 80% | 100% |
| P3 | 40% | 60% | 80% | 60% | 40% | 40% |
Neither checkpoint answered harmfully in any arm. v0.1 follows an injected alert more sharply (fewer partial
episodes, more explicit refusals in every cadence); v0.2.1 does not over-refuse benign requests (0.00 vs 0.25
on the tapes above) and takes turns better (FDB v1), which this harmful-only pilot cannot see. The cadence
ranking is the same for both: P2k best, P2 close, P3 weakest (it alerts once, then answers wait). Full tables
with brain-call counts and alert timing: eval/seedset_pilot/.
Usage
The adapters target PersonaPlex with the depth decoder sliced to 8 codebooks and two extra text
tokens (<instr_start> = 32000, <instr_end> = 32001), so fold them into the base first:
huggingface-cli download MagicLuke/duplex-refusal --include "v0.2.1/*" --local-dir adapter
python -m duplex_brain.ckpt.build_checkpoint --hf-repo nvidia/personaplex-7b-v1 \
--adapter adapter/v0.2.1/lora.safetensors --scaling 2.0 --out duplex-refusal-v0.2.1.safetensors
build_checkpoint is from duplex-brain. Code and
training recipe: duplex-online-if. For v0.2.1
the two instruction-marker embedding rows are trained as well and must be folded with the rest of the
adapter (VERSION.json says ft_instr_embed: true).
- Downloads last month
- -
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support