README.md

---
tags:
  - video-understanding
  - multimodal
  - temporal-reasoning
  - qwen
---

# AVTime: data construction, pure BiTSC training, and AVTime-Bench

This repository is the code-only release of the AVTime project. It covers the
complete engineering path used by the project:

1. build the AVTime-50K train/test split;
2. cut complete videos and produce segment-level captions;
3. train the actor with SFT and pure BiTSC GRPO;
4. run LongVALE-compatible dense video captioning inference;
5. run the canonical AVTime-Bench inference, Qwen3.6-27B judging, and scoring
   pipeline.

The caption pipeline explicitly includes **both clipping and segmented
captioning**. Clipping is performed by AVCut, and every rendered segment is
then captioned before its local timestamps are mapped back to the complete
video timeline.

> **Release status.** [Hugging Face](https://huggingface.co/Apryle/AVTime) is
> public with **manual gated access**. The matching
> [GitHub mirror](https://github.com/aprylewu/AVTime) is private while the paper
> is under review and is intended to become public at release time.

> **Code-only boundary.** Videos, full annotations, model checkpoints,
> generated captions, benchmark predictions, judge responses, caches, and
> credentials are not included. The bundled files contain source code,
> configuration, tests, and the frozen 300-video split ID list only.

For a component-by-component explanation of why the system is structured this
way, read the [Chinese design document](docs/DESIGN_ZH.md). Shorter
component-specific notes remain in
[data/README.md](data/README.md),
[data/caption_pipeline/README.md](data/caption_pipeline/README.md),
[training/README.md](training/README.md), and
[benchmark/README.md](benchmark/README.md).

## Contents

- [System overview](#system-overview)
- [Repository layout](#repository-layout)
- [Installation](#installation)
- [Required private assets](#required-private-assets)
- [Part I: data construction](#part-i-data-construction)
  - [AVTime-50K split](#1-avtime-50k-split)
  - [AVCut: clipping complete videos](#2-avcut-clipping-complete-videos)
  - [Segmented captioning](#3-segmented-captioning)
- [Part II: model training](#part-ii-model-training)
  - [SFT](#4-supervised-fine-tuning-sft)
  - [Pure BiTSC GRPO data](#5-build-the-pure-bitsc-grpo-dataset)
  - [BiTSC ORM and GRPO](#6-run-pure-bitsc-grpo)
- [Part III: inference and evaluation](#part-iii-inference-and-evaluation)
  - [LongVALE-compatible inference](#7-run-longvale-compatible-inference)
  - [AVTime-Bench](#8-run-avtime-bench)
- [Outputs and reproducibility](#outputs-and-reproducibility)
- [Validation](#validation)
- [Troubleshooting](#troubleshooting)
- [Release and redistribution boundary](#release-and-redistribution-boundary)

## System overview

```text
complete videos
    |
    +-- AVTime-50K split --------------------------------------+
    |       train.jsonl / test.jsonl / manifest                |
    |                                                          |
    +-- AVCut                                                  |
    |       ASR pauses + scene changes + audio events           |
    |       -> clips/*.mp4 + clips/*.mp3 + segments.json        |
    |                                                          |
    +-- segmented captioning                                   |
            Qwen3-Omni or Gemini                               |
            -> captions_merged.json on the full-video timeline |
                                                               |
annotations + media references                                 |
    |                                                          |
    +-- one-round SFT data -> SFT actor                         |
    |                                                          |
    +-- pure BiTSC data -> GRPO + avtime_btsc ORM               |
                                                               |
trained actor -------------------------------------------------+
    |
    +-- LongVALE-compatible DVC inference + official merge
    |
    +-- AVTime-Bench HQ300
            target inference
            -> official LongVALE adjacent-caption merge
            -> Qwen3.6-27B caption-only judge
            -> deterministic T2E/E2T/E2E scoring
```

The three stages exchange explicit artifacts rather than hidden process state:

| Stage | Main input | Main output | Contract |
| --- | --- | --- | --- |
| Split | complete-video JSONL | `train.jsonl`, `test.jsonl` | video-disjoint rows and recorded hashes |
| Clipping | one complete video | `segments.json` and rendered clips | full coverage and non-overlapping segments |
| Captioning | segmented video directory | `captions_merged.json` | seconds on the original video timeline |
| SFT build | AVTime/LongVALE rows | `sft_train.jsonl` | one-round `From XX to YY` answers |
| GRPO build | SFT rows | `bitsc_grpo.jsonl` | schema `avtime_btsc_grpo_v1`, task `bitsc` only |
| Inference | reference-free video rows | raw model answers | no assistant/reference leakage |
| Normalization | raw answers | canonical event JSONL | frozen percentage-time conversion and merge |
| Evaluation | predictions + private references | six scores and traces | judge handles semantics; code handles time |

## Repository layout

```text
.
├── data/
│   ├── split_avtime_50k.py
│   ├── splits/
│   │   └── avtime_bench_hq300_zeroshot_ids.jsonl
│   └── caption_pipeline/
│       ├── avcut_config.yaml
│       ├── requirements*.txt
│       └── src/pipeline/
│           ├── avcut.py
│           ├── avcut_batch.py
│           ├── cut.sh
│           ├── cut_multinode.sh
│           ├── qwen3_caption.py
│           ├── qwen3_caption_multinode.sh
│           ├── qwen3_caption_progress.py
│           └── gemini_caption.py
├── training/
│   ├── sft/
│   │   ├── build_sft_dataset.py
│   │   └── train_sft.sh
│   └── bitsc/
│       ├── build_grpo_dataset.py
│       ├── bitsc_orm.py
│       ├── serve_rollout.sh
│       ├── serve_reward_model.sh
│       └── train_grpo.sh
├── benchmark/
│   ├── build_target_inputs.py
│   ├── infer_qwen3_omni.py
│   ├── normalize_predictions.py
│   ├── judge_qwen36.py
│   ├── score_avtime_bench.py
│   ├── run_avtime_bench.sh
│   ├── run_avtime_bench_structured.sh
│   └── prompts/
├── tests/
├── docs/DESIGN_ZH.md
├── PROVENANCE.md
└── RELEASE_CHECKLIST.md
```

Run every command below from the repository root unless a command says
otherwise.

## Installation

### 1. System requirements

The CPU-only builders and tests require Python 3.10 or newer. Video processing
also requires `ffmpeg` and `ffprobe` on `PATH`:

```bash
ffmpeg -version
ffprobe -version
python3 --version
```

The clipping, Qwen3-Omni inference, SFT, and GRPO stages are GPU workloads.
Install a CUDA-compatible PyTorch build for the target machine before
installing the remaining GPU packages.

### 2. Lightweight package

For JSONL builders, normalization, scoring, and unit tests:

```bash
python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install --upgrade pip
python3 -m pip install -e .
python3 -m pip install -r requirements-dev.txt
```

The lightweight project dependencies are intentionally limited to PyYAML,
HTTPX, and the OpenAI Python client. They do **not** install a universal GPU
stack.

### 3. Caption pipeline

Install the caption client:

```bash
python3 -m pip install -r data/caption_pipeline/requirements.txt
```

Install AVCut after installing the correct PyTorch and TorchAudio wheels:

```bash
python3 -m pip install -r data/caption_pipeline/requirements-avcut.txt
```

The default AVCut configuration uses NVIDIA Parakeet through NeMo. WhisperX is
an optional ASR backend and is not installed by the provided requirements
file. PySceneDetect provides scene cuts, while PANNs provides music and sound
effect transitions.

PANNs does not download its checkpoint in this release. Provide either:

```bash
export PANNS_CHECKPOINT_PATH=/absolute/path/to/Cnn14_mAP=0.431.pth
```

or place that checkpoint under:

```text
${AVCUT_PANNS_HOME}/Cnn14_mAP=0.431.pth
```

If `AVCUT_PANNS_HOME` is unset, use the location expected by your local
configuration.

### 4. Training and benchmark GPU stacks

The launchers expect compatible installations of:

- PyTorch and the CUDA runtime for the machine;
- Megatron-SWIFT for SFT and GRPO;
- vLLM for rollout, reward-model service, target inference, and judge
  inference;
- Transformers with Qwen3-Omni support;
- `qwen_omni_utils` for multimodal preprocessing.

These packages evolve quickly and have checkpoint- and hardware-specific
compatibility constraints. The repository therefore does not claim that a
single lockfile is valid across all supported clusters. Record the exact
environment used for every reported run.

## Required private assets

Create or point the commands at equivalent locations. The following layout
shows the default paths:

```text
data/
├── AVTime-50K/
│   └── all.jsonl
├── LongVALE/
│   └── train_agent.jsonl
└── AVTime-Bench/
    ├── test.jsonl
    ├── inventory.jsonl
    ├── benchmark_reference_manifest.jsonl
    └── target_inputs.jsonl              # generated when absent

/absolute/path/to/source_videos/
/absolute/path/to/Qwen3-Omni-checkpoint/
/absolute/path/to/SFT-actor/
/absolute/path/to/reward-model/
```

The source media paths may live anywhere. Benchmark inventory rows must point
to existing absolute paths. Do not commit private media, annotations,
credentials, or generated judge artifacts.

# Part I: data construction

## 1. AVTime-50K split

### What the splitter does

`data/split_avtime_50k.py` reads one complete-video record per JSONL line. It
selects AVTime rows conservatively:

1. prefer `metadata.source == "avtime"`;
2. when source metadata has been stripped, require the public `avtime_` ID
   namespace;
3. exclude unknown unlabeled rows instead of guessing their source.

Each retained row must provide:

- a non-empty unique ID;
- one resolvable input-video reference (from `metadata.input_video`, otherwise
  the first entry in `videos`);
- positive duration metadata;
- a positive event count, either in `metadata.event_count` or recoverable by
  scanning assistant messages from the end for the latest parseable, non-empty
  event JSON.

The default run asserts a 49,977-row AVTime corpus. It uses the bundled frozen
300-ID zero-shot manifest and produces 49,677 training rows plus 300 benchmark
rows.

### Reproduce the frozen public split

Place the complete annotation file at `data/AVTime-50K/all.jsonl`, then run:

```bash
python3 -m data.split_avtime_50k
```

Equivalent explicit command:

```bash
python3 -m data.split_avtime_50k \
  --source data/AVTime-50K/all.jsonl \
  --output-root outputs/avtime-50k-split \
  --test-ids data/splits/avtime_bench_hq300_zeroshot_ids.jsonl \
  --expected-rows 49977
```

The output directory contains:

```text
outputs/avtime-50k-split/
├── train.jsonl
├── test.jsonl
├── test_ids.jsonl
└── split_manifest.json
```

- `train.jsonl` and `test.jsonl` preserve the complete source rows.
- `test_ids.jsonl` contains the ID, video reference, duration, event count, and
  the assigned duration/event-count quantile bins.
- `split_manifest.json` records schema version, source and output SHA-256
  values, split sizes, distributions, and disjointness checks.

### Derive a new deterministic holdout

This is useful for an ablation or a private rerun, but it is **not** the frozen
AVTime-Bench split:

```bash
python3 -m data.split_avtime_50k \
  --source data/AVTime-50K/all.jsonl \
  --output-root outputs/avtime-50k-derived-split \
  --derive-test-ids \
  --test-size 300 \
  --seed AVTime-50K-public-split-v1
```

Derived selection uses a 5 x 5 duration/event-count stratification and seeded
SHA-256 ranking. Changing the seed changes the selected IDs while preserving
the deterministic procedure.

Use `--expected-rows 0` only when deliberately processing a different corpus
version. Disabling the size check makes accidental corpus drift harder to
detect.

## 2. AVCut: clipping complete videos

AVCut does not cut on scene changes alone. It fuses three signals:

- ASR sentence ends and pauses;
- visual scene transitions;
- audio event onsets/offsets for music and sound effects.

Candidate boundaries are clustered, shifted away from the middle of speech
toward pause anchors, filtered by confidence and minimum-gap constraints, and
supplemented with soft boundaries when a segment becomes too long. The final
timeline is full-coverage and non-overlapping.

The published YAML defaults include:

| Setting | Default |
| --- | ---: |
| ASR backend | `parakeet` |
| scene threshold | `24.0` |
| minimum boundary gap | `5.0 s` |
| minimum segment duration | `6.0 s` |
| long-segment soft threshold | `16.0 s` |
| MP3 sidecar | enabled |

Review [`avcut_config.yaml`](data/caption_pipeline/avcut_config.yaml) before a
large run. Although the MP3 field is configurable in the single-video
renderer, the released batch validator and Qwen segmented-caption path both
expect the audio sidecar. Keep `render.write_mp3_sidecar: true` for the
standard batch pipeline.

### Cut one video

```bash
python3 data/caption_pipeline/src/pipeline/avcut.py \
  /absolute/path/to/input.mp4 \
  --config data/caption_pipeline/avcut_config.yaml \
  --output-root /absolute/path/to/outputs/caption_pipeline/videos
```

The resulting directory is keyed by the source video stem:

```text
/absolute/path/to/outputs/caption_pipeline/videos/<video_stem>/
├── segments.json
└── clips/
    ├── clip_0001.mp4
    ├── clip_0001.mp3
    ├── clip_0002.mp4
    ├── clip_0002.mp3
    └── ...
```

`segments.json` records:

- source video, measured duration, device, ASR backend/model/language, config,
  and output root;
- detector candidate counts and timing statistics;
- segment index, clip path, start/end/duration in seconds;
- boundary provenance;
- audio, visual, and joint event labels.

`avcut.py --dry-run` still runs detection and writes segmentation metadata but
does not render normal captionable media clips. Use it to inspect boundaries,
not as input to live captioning.

### Cut a flat directory

```bash
python3 data/caption_pipeline/src/pipeline/avcut_batch.py \
  /absolute/path/to/flat_source_videos \
  --output-root /absolute/path/to/outputs/caption_pipeline/videos \
  --config data/caption_pipeline/avcut_config.yaml \
  --max-jobs 4
```

Batch discovery is intentionally non-recursive: only files directly inside the
input directory are scanned. Supported extensions are `.mp4`, `.m4v`,
`.mov`, `.mkv`, `.avi`, and `.webm`. Two files with the same stem but
different extensions are rejected because they would map to the same output
directory.

Batch mode provides:

- a SQLite job table;
- source signatures based on file size and nanosecond modification time;
- per-video staging directories;
- output validation before an atomic staging-to-final rename;
- stable BLAKE2 sharding;
- bounded GPU/device worker pools;
- resumable completed and failed states.

The batch dry run creates a dummy output layout to exercise orchestration. It
does not validate real detector or renderer quality.

### Single-node wrapper

The wrapper derives parallelism from visible GPUs. Its default is four jobs per
GPU:

```bash
bash data/caption_pipeline/src/pipeline/cut.sh \
  --input-root /absolute/path/to/flat_source_videos \
  --output-root /absolute/path/to/outputs/caption_pipeline/videos \
  --config data/caption_pipeline/avcut_config.yaml \
  --jobs-per-gpu 4
```

Always pass explicit input and output roots. This avoids ambiguity from
machine-specific working directories and makes the run record portable.

### Multi-node wrapper

The wrapper detects common scheduler rank variables. When a scheduler does not
export them, set the AVTime-specific values:

```bash
AVCUT_NODE_RANK=0 AVCUT_NUM_NODES=2 \
bash data/caption_pipeline/src/pipeline/cut_multinode.sh \
  --input-root /absolute/path/to/flat_source_videos \
  --output-root /shared/path/to/outputs/caption_pipeline/videos \
  --config data/caption_pipeline/avcut_config.yaml
```

Run the command on every node with a different `AVCUT_NODE_RANK`. By default,
only local rank zero on each node performs work. Each rank receives an
independent state database and staging root.

The multi-node wrapper enables terminal-state skipping by default. A job marked
`completed` or `failed` for the same source signature may be skipped without
revalidating the final media. Inspect per-rank databases/logs before treating a
distributed run as complete.

### Detector degradation

AVCut warns and disables a detector channel when optional NeMo/WhisperX,
PySceneDetect, or PANNs dependencies are unavailable. That behavior permits
partial diagnostics, but it can degrade the cut plan to a small number of
boundaries or even one full-video segment.

For production data construction, treat warnings about missing detectors or a
missing PANNs checkpoint as failed environment validation, even when the
process itself continues.

## 3. Segmented captioning

The caption stage consumes a root directory whose direct child directories
each contain `segments.json` and rendered clips:

```text
videos_root/
├── video_a/segments.json
├── video_a/clips/...
├── video_b/segments.json
└── video_b/clips/...
```

The Qwen path sends the clip video and its MP3 sidecar as separate local media
items. It sets `use_audio_in_video=false` so that the audio sidecar, rather
than audio embedded in the MP4, is the explicit audio source.

The caption prompt asks for one to three visual-first events per clip. Audio
provides supporting evidence. For each successful source video, the script:

1. requires a non-empty event array with numeric timestamps and non-empty
   captions;
2. clamps every timestamp into the clip duration;
3. folds a non-positive interval into the previous caption when possible (or
   expands the first such item to the clip), and pins only the first start and
   last end to the clip boundaries;
4. shifts clip-local seconds by the clip start time;
5. atomically writes one complete-video `captions_merged.json`.

The normalizer does not generally sort events or close every internal
overlap/gap, so malformed chronology remains a generation-quality issue that
should be audited.

The merged schema is:

```json
[
  {
    "start_sec": 0.0,
    "end_sec": 5.84,
    "caption": "A person opens a door while footsteps are audible."
  }
]
```

Published Qwen caption-client defaults:

| Setting | Default |
| --- | ---: |
| request model name | endpoint fallback: `Qwen/Qwen3-Omni-30B-A3B-Thinking`; managed mode: served name or checkpoint basename |
| temperature / top-p / top-k | 0.6 / 0.95 / 20 |
| repetition penalty | 1.0 |
| maximum response tokens | 32,768 |
| video sampling | 4 fps |
| concurrent video jobs | 8 |
| total in-flight clips | 64 |
| retries / request timeout | 5 / 300 seconds |

### Option A: use an existing OpenAI-compatible endpoint

The endpoint must be able to read the exact same filesystem paths because the
client sends `file://` URIs:

```bash
python3 data/caption_pipeline/src/pipeline/qwen3_caption.py \
  /absolute/path/to/outputs/caption_pipeline/videos \
  --base-url http://127.0.0.1:8000/v1 \
  --model Qwen/Qwen3-Omni-30B-A3B-Thinking \
  --max-jobs 8 \
  --max-inflight-clips 64
```

Use repeated `--source-video /absolute/path/to/original.mp4` arguments to
select jobs by the exact `segments.json` `input_video` value. Use
`--video-fps`, `--video-min-frames`, and
`--video-max-frames` to control multimodal preprocessing.

### Option B: let the client launch local vLLM

```bash
python3 data/caption_pipeline/src/pipeline/qwen3_caption.py \
  /absolute/path/to/outputs/caption_pipeline/videos \
  --model-path /absolute/path/to/Qwen3-Omni-30B-A3B-Thinking \
  --tensor-parallel-size 8 \
  --max-jobs 8 \
  --max-inflight-clips 64
```

The managed-server mode starts vLLM, waits for readiness, runs caption jobs,
and records server/raw-response logs. HTTP 400 and invalid-media responses are
treated as deterministic errors and are not retried. Transient errors use the
configured retry budget.

Qwen resume state is **video-directory-level**: `captions_merged.json` is
written only after every clip in a video succeeds. The state signature hashes
`segments.json`, not the clip bytes. If clip media changes without changing
`segments.json`, delete or invalidate the corresponding state/output manually.

### Multi-node segmented captioning

```bash
AVCUT_NODE_RANK=0 AVCUT_NUM_NODES=2 \
bash data/caption_pipeline/src/pipeline/qwen3_caption_multinode.sh \
  /shared/path/to/outputs/caption_pipeline/videos \
  --model-path /shared/path/to/Qwen3-Omni-30B-A3B-Thinking \
  --tensor-parallel-size 8
```

Run once per node. Stable sharding ensures that the same video maps to the same
rank for a fixed world size. The wrapper defaults to `--keep-going`, so a run
can exit successfully even when individual videos failed. Completion must be
checked from the state databases or the progress command:

Unlike the single-node client defaults, the multi-node wrapper supplies
`--max-jobs 128 --max-inflight-clips 128` unless explicitly overridden.

```bash
python3 data/caption_pipeline/src/pipeline/qwen3_caption_progress.py \
  /shared/path/to/outputs/caption_pipeline/videos
```

The monitor opens SQLite databases read-only and reports total, completed,
running, failed, pending, and untracked jobs.

### Gemini fallback

Gemini is a per-video-directory fallback. It uploads MP4 clips one at a time,
writes recoverable per-clip JSON, and then merges the video:

```bash
export GEMINI_API_KEY=...
python3 data/caption_pipeline/src/pipeline/gemini_caption.py \
  /absolute/path/to/outputs/caption_pipeline/videos/video_a/segments.json \
  --model gemini-3.1-pro-preview
```

`GOOGLE_API_KEY` is also accepted. Keys are never read from repository files.
Gemini writes:

```text
video_a/
├── captions/
│   ├── clip_0001.json
│   └── ...
└── captions_merged.json
```

Its resume granularity is one clip, which is finer than the Qwen path. The
Gemini normalizer is also intentionally looser; for comparable dataset
construction, keep the backend and normalization path fixed across the run.

# Part II: model training

## 4. Supervised fine-tuning (SFT)

### Build the one-round dataset

The SFT builder accepts:

- an optional LongVALE training JSONL;
- AVTime training JSONL;
- a deterministic AVTime sample count.

To build the intended mixed recipe from the currently supplied LongVALE file
and a deterministic 1,500-row AVTime sample:

```bash
python3 -m training.sft.build_sft_dataset \
  --longvale-jsonl data/LongVALE/train_agent.jsonl \
  --avtime-jsonl outputs/avtime-50k-split/train.jsonl \
  --avtime-sample-count 1500 \
  --seed 20260702 \
  --output-jsonl outputs/training/sft_train.jsonl \
  --report-json outputs/training/sft_train.report.json
```

The builder consumes every valid row in the supplied LongVALE file; it does
not assert a hard-coded LongVALE row count. Record the input hash and report
instead of assuming a count from an earlier private snapshot.

AVTime-only SFT:

```bash
python3 -m training.sft.build_sft_dataset \
  --avtime-jsonl outputs/avtime-50k-split/train.jsonl \
  --avtime-sample-count 1500
```

Use all valid AVTime rows:

```bash
python3 -m training.sft.build_sft_dataset \
  --avtime-jsonl outputs/avtime-50k-split/train.jsonl \
  --avtime-sample-count 0
```

The builder converts percentage- or millisecond-based source events into one
strict answer:

```text
From 00 to 12, a person enters the room
From 13 to 27, the person sits at a table
```

Every output row has one system message, one user message, and one assistant
message. Events are validated, de-duplicated, chronologically sorted, and
written with one event per line. Rows without usable media or events are
skipped; missing IDs or duplicate IDs are treated as errors. The JSON report
records counts, source statistics, prompt text, and sample rows.

### Launch SFT

```bash
MODEL=/absolute/path/to/Qwen3-Omni-base \
DATASET=outputs/training/sft_train.jsonl \
OUTPUT_DIR=outputs/training/sft-checkpoints \
NNODES=4 \
NPROC_PER_NODE=8 \
NODE_RANK=0 \
MASTER_ADDR=trainer-0 \
bash training/sft/train_sft.sh
```

Run the same command on all four nodes, changing `NODE_RANK` to 0, 1, 2, or
3. The public defaults are:

| Category | Default |
| --- | --- |
| topology | 4 nodes x 8 processes |
| TP / PP / CP / EP / ETP | 2 / 1 / 2 / 8 / 1 |
| global / micro batch | 64 / 1 |
| epochs | 1 |
| max sequence length | 65,536 |
| LoRA rank / alpha / dropout | 128 / 256 / 0.05 |
| trainable modules | LLM LoRA; vision and aligner frozen |
| learning rate / minimum | `5e-5` / `5e-6` |
| held-out split ratio | 0.002 |
| save / evaluation interval | 20 steps |

After building the dataset, print the fully expanded command without launching
training (the launcher validates that `DATASET` exists before its dry-run
branch):

```bash
DRY_RUN=1 \
MODEL=/absolute/path/to/Qwen3-Omni-base \
DATASET=outputs/training/sft_train.jsonl \
bash training/sft/train_sft.sh
```

If Megatron-SWIFT is installed somewhere nonstandard, set
`MEGATRON_SFT_ENTRYPOINT`. Use `PYTHON_BIN` to select the environment that
contains both Swift and PyTorch.

## 5. Build the pure BiTSC GRPO dataset

Convert the one-round SFT rows:

```bash
python3 -m training.bitsc.build_grpo_dataset \
  --input-jsonl outputs/training/sft_train.jsonl \
  --output-jsonl outputs/training/bitsc_grpo.jsonl \
  --report-json outputs/training/bitsc_grpo.report.json \
  --seed 20260729
```

Add `--check-media --media-root /absolute/media/root` when media references
should be verified during conversion.

The converter is deliberately strict:

- exactly one assistant message;
- exactly one video reference;
- assistant output made only of valid `From XX to YY, caption` lines;
- chronological events;
- schema `avtime_btsc_grpo_v1`;
- task exactly `bitsc`.

The `solution` field is a compact JSON string containing the complete
reference event list. The default builder shuffles deterministically. Use
`--no-shuffle` only for a controlled diagnostic.

Neither the dataset builder nor the reward plugin generates SC
(segment-captioning) or TVG (temporal video grounding) tasks.

## 6. Run pure BiTSC GRPO

### What “pure BiTSC” means here

The only registered reward name is `avtime_btsc`. The ORM rejects another
schema or task with reward `-1`. It contains no reward branch for:

- SC;
- TVG;
- data source identity;
- caption style;
- preferred event count;
- task-specific formatting beyond the shared timestamp-caption contract.

The two BiTSC directions are:

1. **Time-to-event (T2E).** At deterministic sampled times, find all active
   reference and predicted events. The caption-only judge scores semantic
   pairs from 0 to 4. Code normalizes these scores to [0, 1], performs
   maximum-weight bipartite matching, and computes
   `2 * matched_weight / (reference_count + prediction_count)`. A probe with
   only one active side scores zero; an all-empty probe is skipped.
2. **Event-to-time (E2T).** Deterministically sample reference-event anchors.
   The judge selects all predicted captions describing the same event. Code
   takes the union of their time intervals and computes temporal union-IoU
   against the reference interval.

The final reward is the normalized weighted combination:

```text
reward = w_t2e * mean(T2E probes) + w_e2t * mean(E2T anchors)
```

The defaults are `w_t2e = 0.5` and `w_e2t = 0.5`.

Only caption text is sent to the semantic reward model. Timestamps remain in
Python, so the language model cannot invent the temporal score. Probe and
anchor sampling are deterministic for a fixed solution. Completions sharing
the same reference solution are grouped into one judge request.

If RLAIF is disabled or a judge call fails, the ORM uses a deterministic
token-F1 semantic fallback. This keeps training alive but is not equivalent to
a successful model-based semantic judgment; monitor the `BTSC-RLAIF` logs.

### Start the rollout service

On the rollout host:

```bash
ACTOR_MODEL=/absolute/path/to/sft-actor \
PORT=8100 \
TP=8 \
bash training/bitsc/serve_rollout.sh
```

Script defaults when not overridden:

- port `8100`;
- TP `1`;
- GPU memory utilization `0.88`;
- maximum model length `34,816`;
- maximum concurrent sequences `64`;
- prefix caching enabled.

### Start the caption-semantic reward service

On a separate reward host:

```bash
REWARD_MODEL=/absolute/path/to/reward-model \
SERVED_MODEL_NAME=reward-model \
PORT=8000 \
TP=8 \
bash training/bitsc/serve_reward_model.sh
```

Defaults are host `0.0.0.0`, port `8000`, TP `1`, GPU memory utilization
`0.90`, maximum model length `32,768`, maximum 32 sequences, and 32,768
batched tokens.

Use network controls appropriate for the cluster. The vLLM service should not
be exposed to an untrusted network.

### Launch GRPO

```bash
ACTOR_MODEL=/absolute/path/to/sft-actor \
DATASET=outputs/training/bitsc_grpo.jsonl \
OUTPUT_DIR=outputs/training/bitsc-grpo-checkpoints \
ROLLOUT_HOST=rollout-host \
ROLLOUT_PORT=8100 \
REWARD_HOST=reward-host \
REWARD_PORT=8000 \
NNODES=2 \
NPROC_PER_NODE=8 \
NODE_RANK=0 \
MASTER_ADDR=trainer-0 \
bash training/bitsc/train_grpo.sh
```

Run on both training nodes with `NODE_RANK=0` and `NODE_RANK=1`. The launcher
checks that:

- world size is divisible by `TP * PP * CP`;
- `EP` divides the resulting data-parallel size;
- global batch size is divisible by the number of generations.

Public GRPO defaults:

| Category | Default |
| --- | --- |
| topology | 2 nodes x 8 processes |
| TP / PP / CP / EP / ETP | 2 / 2 / 2 / 2 / 1 |
| global / micro batch | 128 / 1 |
| generations per prompt | 8 |
| train iterations | 100 |
| max input / completion length | 32,768 / 2,048 |
| temperature / top-p / top-k | 0.9 / 0.95 / 50 |
| LoRA rank / alpha / dropout | 64 / 128 / 0.05 |
| learning rate / minimum | `2e-6` / `2e-7` |
| reward function | `avtime_btsc` only |

Useful ORM environment variables:

| Variable | Default | Meaning |
| --- | ---: | --- |
| `BTSC_PROBE_COUNT` | 8 | deterministic T2E probe count |
| `BTSC_ANCHOR_COUNT` | 4 | reference anchors for E2T |
| `BTSC_TIME_TO_EVENT_WEIGHT` | 0.5 | T2E weight |
| `BTSC_EVENT_TO_TIME_WEIGHT` | 0.5 | E2T weight |
| `BTSC_RLAIF_ENABLED` | 1 | enable semantic model |
| `BTSC_RLAIF_FRACTION` | 1.0 | deterministic fraction sent to judge |
| `BTSC_RLAIF_MAX_CONCURRENCY` | 8 | concurrent judge groups |
| `BTSC_RLAIF_TIMEOUT_SEC` | 120 | request timeout |
| `BTSC_RLAIF_MAX_TOKENS` | 4096 | judge response ceiling |
| `BTSC_RLAIF_JSON_OBJECT` | 1 | request strict JSON schema |
| `BTSC_REWARD_BASE_URL` | `http://127.0.0.1:8000/v1` | OpenAI-compatible endpoint |
| `BTSC_REWARD_MODEL` | `reward-model` | served model name |

After building the GRPO dataset, inspect the expanded command (the launcher
validates that `DATASET` exists before its dry-run branch):

```bash
DRY_RUN=1 \
ACTOR_MODEL=/absolute/path/to/sft-actor \
DATASET=outputs/training/bitsc_grpo.jsonl \
bash training/bitsc/train_grpo.sh
```

# Part III: inference and evaluation

## 7. Run LongVALE-compatible inference

This repository includes the LongVALE-compatible **dense video captioning
inference protocol and the pinned official adjacent-caption merge port**. It
does not redistribute the full third-party LongVALE evaluator.

“LongVALE-compatible” in this release means:

- the frozen training-style prompt;
- one output line per event: `From XX to YY, caption`;
- percentage timestamps in [0, 99];
- official Qwen multimodal preprocessing when explicitly selected;
- the pinned adjacent-caption merge used by the canonical AVTime path.

### Prepare reference-free inference JSONL

Each row must contain one video and a stable video ID. The AVTime target-input
builder produces this contract automatically. For another LongVALE-compatible
set, create rows equivalent to:

```json
{
  "id": "record_0001",
  "tools": [],
  "videos": ["/absolute/path/to/video_0001.mp4"],
  "messages": [
    {"role": "system", "content": "placeholder"},
    {"role": "user", "content": "<video>"}
  ],
  "metadata": {
    "video_id": "video_0001",
    "duration_ms": 120000
  }
}
```

With `--prompt-style train`, the frozen training prompt replaces the row text;
the direct inference runner does not consume the sample messages in that mode.
They are shown to mirror the stricter AVTime target-input structure. With
`--prompt-style jsonl`, the direct runner requires at least one user message
whose content can be extracted; a system message is optional. In all cases,
keep references and assistant messages out of inference inputs.

### Run one LongVALE-style inference job

```bash
python3 -m benchmark.infer_qwen3_omni \
  --jsonl /absolute/path/to/longvale_target_inputs.jsonl \
  --model-path /absolute/path/to/target-model \
  --model-name avtime-model \
  --output-dir outputs/longvale/inference \
  --predictions-jsonl outputs/longvale/raw_predictions.jsonl \
  --prompt-style train \
  --qwen-omni-media-mode official \
  --use-audio-in-video \
  --temperature 0 \
  --top-p 1 \
  --top-k 20 \
  --repetition-penalty 1 \
  --max-completion-tokens 8192 \
  --batch-size 8 \
  --vllm-tensor-parallel-size 4 \
  --vllm-max-model-len 65536
```

The direct inference CLI defaults to resumable and
`continue-on-error=true`. That option continues with later rows after a
row-level failure; it does **not** turn an incomplete run into success. The
process still exits with status 2 when any row failed. For immediate
fail-closed behavior, add:

```text
--no-resume --no-continue-on-error
```

### Apply the pinned official adjacent-caption merge

Normalization requires target identities, raw answers, a model identity, and
an explicit decoding record:

```bash
python3 -m benchmark.normalize_predictions \
  --target-inputs /absolute/path/to/longvale_target_inputs.jsonl \
  --raw outputs/longvale/raw_predictions.jsonl \
  --output outputs/longvale/predictions.jsonl \
  --failures outputs/longvale/prediction_failures.jsonl \
  --report outputs/longvale/prediction_report.json \
  --model-id avtime-model \
  --model-path /absolute/path/to/target-model \
  --protocol longvale_train_dvc_v1 \
  --decoding-json '{"temperature":0,"top_p":1,"top_k":20,"max_tokens":8192}'
```

The parser ignores non-protocol prose with recorded warnings, rejects a result
with no valid event lines, sorts valid events by time, and merges adjacent
captions only when:

```text
SequenceMatcher(caption_a, caption_b).ratio() > 0.98
and next.start_pct - current.end_pct is in [-1, 1]
```

The report records the pinned upstream LongVALE commit, upstream file hash,
merge thresholds, and frozen validation-vector hash. Use the official
LongVALE project separately when full LongVALE benchmark metrics are required.

## 8. Run AVTime-Bench

### Canonical HQ300 protocol

`benchmark/run_avtime_bench.sh` is the canonical entry point. It executes four
stages:

1. build reference-free target inputs if absent;
2. run two target-model shards and enforce exact 300-video coverage;
3. normalize with the LongVALE line protocol and official adjacent-caption
   merge;
4. prepare caption-only requests, run Qwen3.6-27B, and deterministically score
   six AVTime metrics.

Expected private files under `data/AVTime-Bench/`:

| File | Purpose |
| --- | --- |
| `test.jsonl` | source test rows |
| `inventory.jsonl` | audited IDs, durations, hashes, and absolute media paths |
| `benchmark_reference_manifest.jsonl` | private scoring references |
| `target_inputs.jsonl` | generated reference-free model inputs |

If `target_inputs.jsonl` is absent, the wrapper writes it under `BENCH_ROOT`.
Therefore that directory must be writable, or `TARGET_INPUTS` must point to a
writable alternative.

Build inputs explicitly:

```bash
python3 -m benchmark.build_target_inputs \
  --test-jsonl data/AVTime-Bench/test.jsonl \
  --inventory data/AVTime-Bench/inventory.jsonl \
  --output data/AVTime-Bench/target_inputs.jsonl \
  --report data/AVTime-Bench/target_inputs_report.json
```

The builder requires:

- unique inventory `stable_id` values;
- unique benchmark `video_id` values;
- an existing absolute `resolved_media_path` for every row;
- exactly one system and one user message;
- no extra fields inside those message objects;
- full test/inventory coverage.

Assistant/reference content is removed. Output metadata retains audited
duration and source hashes.

### Run the complete benchmark

The default topology expects eight GPUs. Target inference runs as two parallel
TP=4 shards, then the judge reuses eight GPUs with TP=8:

```bash
TARGET_MODEL=/absolute/path/to/target-model \
TARGET_MODEL_ID=avtime-model \
JUDGE_MODEL=Qwen/Qwen3.6-27B \
JUDGE_MODEL_ID=Qwen3.6-27B-avtime-judge-v1 \
RUN_ID=avtime-model-hq300-qwen36-27b \
bash benchmark/run_avtime_bench.sh
```

Important canonical defaults:

| Target setting | Default |
| --- | ---: |
| expected videos | 300 |
| GPU groups | `0,1,2,3` and `4,5,6,7` |
| tensor parallelism per shard | 4 |
| rows per shard | 150 and 150 |
| batch size per replica | 512 |
| max model length | 65,536 |
| max completion tokens | 8,192 |
| temperature / top-p / top-k | 0 / 1 / 20 |
| repetition penalty | 1 |
| seed | 1234 |
| prompt | frozen LongVALE training prompt |
| media preprocessing | official Qwen path |
| audio in video | enabled |
| target resume | disabled |
| target continue-on-error | disabled |

Useful hardware overrides:

```bash
TARGET_MODEL=/absolute/path/to/target-model \
TARGET_SHARD0_GPUS=0,1 \
TARGET_SHARD1_GPUS=2,3 \
TARGET_TP=2 \
TARGET_BATCH_SIZE=32 \
TARGET_MAX_NUM_SEQS=8 \
TARGET_GPU_MEMORY_UTILIZATION=0.80 \
JUDGE_MODEL=/absolute/path/to/Qwen3.6-27B \
JUDGE_TP=4 \
JUDGE_MAX_MODEL_LEN=32768 \
JUDGE_BATCH_SIZE=16 \
RUN_ID=hardware-adjusted-run \
bash benchmark/run_avtime_bench.sh
```

Changing topology or batch size does not change the intended semantic protocol,
but every override must be recorded with the resulting run artifacts.

The wrapper refuses to reuse an existing `OUTPUT_ROOT`. Choose a new `RUN_ID`
or an empty `OUTPUT_ROOT` for every run. It also fails when either target shard
returns a nonzero status or produces anything other than exact expected line
coverage.

### Qwen3.6-27B judge behavior

The judge receives caption text plus a modality rubric; it does not receive
timestamps. It serves three request kinds:

- `t2e_scalar`: semantic score for captions active at a fixed point;
- `semantic_selector`: prediction indices matching one reference event;
- `semantic_pair_selector`: two selector sets for temporal relation scoring.

The first canonical judge pass uses 512 output tokens and batch size 32. If
that process exits nonzero, the wrapper retries with 4,096 tokens and batch
size 16. Resume reuse is allowed only when the query ID, axis, branch, video
ID, request type, model ID, prompt version and hash, schema version and hash,
and request hash all match.

### Six AVTime scores

The summary contains exactly:

| Direction | Audio | Visual |
| --- | --- | --- |
| Time-to-event | `T2E-A` | `T2E-V` |
| Event-to-time | `E2T-A` | `E2T-V` |
| Event-to-event | `E2E-A` | `E2E-V` |

Ownership is intentionally separated:

- Qwen3.6-27B judges semantic equivalence or selects caption indices.
- Deterministic Python converts percentages to intervals, finds active events,
  unions intervals, computes tIoU, and evaluates before/after/overlap
  relations.

For T2E, a query with no active prediction deterministically scores zero and
does not call the judge. E2T uses union-tIoU over all semantically selected
prediction intervals. E2E derives the relation between two selected interval
unions with the benchmark tolerance. Missing or invalid semantic responses are
fatal; there is no lexical fallback in benchmark scoring.

### Canonical output tree

```text
outputs/avtime-bench/<RUN_ID>/
├── target_inputs_report.json            # when inputs were built
├── inference/
│   ├── shard0/
│   │   ├── raw_predictions.jsonl
│   │   └── summary.json
│   └── shard1/
│       ├── raw_predictions.jsonl
│       └── summary.json
├── predictions.jsonl
├── prediction_failures.jsonl
├── prediction_report.json
├── semantic_requests.jsonl
├── semantic_responses.jsonl
├── semantic_failures.jsonl
├── semantic_report.json
└── eval/<RUN_ID>/
    ├── run_config.json
    ├── t2e_trace.jsonl
    ├── e2t_trace.jsonl
    ├── e2e_trace.jsonl
    └── summary.json
```

### Structured-JSON experimental path

The earlier structured-output experiment remains available:

```bash
TARGET_MODEL=/absolute/path/to/target-model \
TARGET_MODEL_ID=avtime-model-structured \
JUDGE_MODEL=Qwen/Qwen3.6-27B \
RUN_ID=structured-experiment \
bash benchmark/run_avtime_bench_structured.sh
```

It is not the canonical main-table protocol.

| Property | Canonical wrapper | Structured wrapper |
| --- | --- | --- |
| target prompt | frozen train prompt | input JSONL messages by default |
| target output | LongVALE lines | strict JSON schema |
| media mode | official | Swift-compatible by default |
| execution | two fail-closed shards | one resumable pass |
| coverage gate | exact 300 by default | no canonical 300-row gate |
| target row errors | abort immediately | process later rows, then exit nonzero if any failed |
| judge | retry pass included | one pass |
| output-root protection | refuses reuse | scorer directory remains immutable |

The structured target subprocess defaults to TP=1, batch size 8, 4,096
completion tokens, top-k -1, resumable inference, and row-level
continue-on-error. Because the wrapper uses `set -e`, any final nonzero target
status still stops the wrapper before normalize/judge/score. Its behavior and
artifacts must not be described as a canonical HQ300 run.

# Outputs and reproducibility

## Time representation

Model-facing answers use inclusive integer percentage buckets from 00 to 99.
When converted to milliseconds, bucket `p` covers its percentage cell:

```text
start_ms = floor((start_pct / 100) * duration_ms)
end_ms   = ceil(((end_pct + 1) / 100) * duration_ms)
```

The resulting internal interval is half-open and clamped to the measured video
duration. This makes `From 00 to 00` a non-empty first bucket and
`From 99 to 99` reach the video end.

Caption-pipeline outputs use seconds directly. Training and benchmark
normalizers perform explicit conversions; do not mix the two representations
without the corresponding duration.

## Run identity

For every publishable run, preserve:

- repository commit;
- input and output hashes;
- model checkpoint identity and revision;
- exact Python/CUDA/PyTorch/vLLM/Swift versions;
- GPU model and count;
- all non-default environment variables;
- split manifest;
- SFT and GRPO builder reports;
- prediction and judge reports;
- benchmark `run_config.json` and trace files.

Do not overwrite an old result with a new checkpoint or judge response.

## Fail-closed versus keep-going stages

The repository uses different failure policies for different operational
needs:

| Stage | Default behavior |
| --- | --- |
| AVTime split | fail on malformed identity/coverage |
| AVCut batch | record per-video state; resumable |
| Qwen caption client | resumable per video |
| Qwen multi-node caption wrapper | keep going; inspect DB |
| direct benchmark inference CLI | resume + continue on error |
| canonical AVTime-Bench target inference | no resume, fail closed |
| canonical benchmark scoring | fail on missing judge response |

Read process exit status together with reports and state databases. A zero exit
from a keep-going wrapper does not prove complete coverage.

# Validation

Install the development requirements and run:

```bash
python3 -m pytest -q
python3 scripts/check_release.py
```

The tests cover:

- split and training data builders;
- strict pure-BiTSC input validation;
- ORM scoring, semantic fallback, and task isolation;
- SFT/GRPO launcher expansion and topology checks;
- target-input reference removal;
- LongVALE parsing and pinned official merge behavior;
- canonical wrapper flags and fail-closed coverage;
- judge request contracts;
- all six AVTime scoring branches.

Shell syntax can be checked without starting GPU work:

```bash
find data training benchmark -type f -name '*.sh' -print0 \
  | xargs -0 -n1 bash -n
```

Use launcher `DRY_RUN=1` modes where provided to inspect resolved arguments.
Unit tests and dry runs do not replace a small end-to-end media smoke test on
the actual GPU environment.

# Troubleshooting

## AVCut produces one very long clip

Inspect warnings for unavailable ASR, scene, or PANNs detectors. Confirm the
PANNs checkpoint path, NeMo/WhisperX installation, PySceneDetect, FFmpeg, and
the selected CUDA device. A process that degraded gracefully is not
necessarily a valid production cut.

## Qwen captioning cannot open `file://` media

The endpoint and client must see the same absolute paths. A containerized or
remote endpoint needs the media directory mounted at the identical path. The
released segmented captioner also expects MP3 sidecars.

## A distributed caption job exits zero with failed videos

The multi-node wrapper enables keep-going. Run
`qwen3_caption_progress.py` and inspect the per-rank SQLite databases and raw
response logs.

## SFT or GRPO entrypoint is not found

Activate the environment containing Megatron-SWIFT or set `PYTHON_BIN` and,
when necessary, `MEGATRON_SFT_ENTRYPOINT`,
`MEGATRON_RLHF_ENTRYPOINT`, or `SWIFT_ROLLOUT_ENTRYPOINT` explicitly.

## GRPO topology is rejected

Check `NNODES * NPROC_PER_NODE` against `TP * PP * CP`, then check that `EP`
divides the derived data-parallel size. Also make `GLOBAL_BATCH_SIZE` divisible
by `NUM_GENERATIONS`.

## BiTSC reward suddenly uses lexical fallback

Look for `BTSC-RLAIF` warnings. Verify the reward endpoint, served model name,
timeout, response schema support, and network reachability. The token-F1 path
keeps execution moving but should be reported as a degraded reward run.

## AVTime-Bench refuses an output directory

Canonical run artifacts are immutable. Choose a new `RUN_ID` or
`OUTPUT_ROOT`. Do not delete an old run merely to reuse its name.

## The benchmark target stage fails coverage

Confirm that `target_inputs.jsonl` has exactly `EXPECTED_VIDEOS` lines and
that each shard produced its exact assigned count. A raw error row still
indicates failed inference and the canonical wrapper aborts on the originating
process.

## The judge retry still fails

Inspect `semantic_failures.jsonl` and `semantic_report.json`. The second pass
uses a larger token ceiling and smaller batch, but it still requires every
response to pass identity and schema validation. Benchmark scoring intentionally
has no silent fallback.

# Release and redistribution boundary

- The frozen split ID list is included; the full dataset is not.
- The code never requires credentials to be stored in tracked files.
- This copy contains no absolute internal cluster paths or bundled secrets.
- The official adjacent-caption merge is a small pinned compatibility port;
  the complete third-party LongVALE evaluator is not redistributed.
- Dataset, checkpoint, and third-party software licenses must be reviewed
  independently before redistribution.
- Complete [`RELEASE_CHECKLIST.md`](RELEASE_CHECKLIST.md) before changing
  repository visibility or publishing scores.
- Repository-level extraction and provenance notes are recorded in
  [`PROVENANCE.md`](PROVENANCE.md).

The public benchmark default is the requested **Qwen3.6-27B** judge. An older
internal launcher used Qwen3.6-35B-A3B. Historical 35B-A3B results must not be
labeled as 27B results; any released 27B number should come from a complete
27B judge rerun with its own immutable artifacts.