# MiniMax-H3 oracle-reference / low-NFE experiment

Status: **complete and validated (28/28 outputs)**

## Executive result

The experiment compares two MiniMax-H3 generation cases at literal 50, 25, 20, 15, and 10 transformer evaluations, plus partition-matched Alibaba PDD 8-NFE and LightX2V Turbo 4-NFE LoRAs. The same explicit target video/audio noise tensors are reused for every setting within a scene.

Option 2 supplies the Option-1 Base-50 output as a full audiovisual oracle reference. In both test cases this reference strongly locks the Base and PDD8 outputs to the oracle trajectory, including at 10 and 8 NFE. LightX2V 4-NFE remains much less visually faithful: it improves on the direct car result when given the oracle, but still changes the frame substantially; in the image-reference desert case it mainly recovers oracle-like audio while visual composition and motion remain different.

This is a controlled two-case diagnostic, not a population benchmark.

## Checkpoint research

- Official base repository: [`MiniMaxAI/MiniMax-H3`](https://huggingface.co/MiniMaxAI/MiniMax-H3), pinned at revision `42ed227ee7df40d41602854ae760620d6eb651fe`.
- The [official Diffusers MiniMax-H3 documentation](https://huggingface.co/docs/diffusers/main/en/api/pipelines/minimax_h3) identifies `transformer/` as the T2VA/FL2VA component and the separate `transformer_ref/` component as Ref2VA. The model files are visible in the official [`transformer_ref` tree](https://huggingface.co/MiniMaxAI/MiniMax-H3/tree/main/transformer_ref).
- Direct T2VA uses `transformer`; both image and video reference generation use `transformer_ref`. The two local partitions have the same 638-key topology but different checkpoint bytes.
- Alibaba PAI provides matching FL2VA and Ref2VA 8-NFE adapters in [`MiniMax-H3-Acc-LoRAs`](https://huggingface.co/alibaba-pai/MiniMax-H3-Acc-LoRAs), pinned at revision `335001fb9e5455d68a0caa18ec2e319072150328`.
  - FL2VA: `MiniMax-H3-FL2VA-Acc-8Step.safetensors`, SHA-256 `0b29be7042d883970eb0c20774a9ba03d95669ed80a721bb4d21be8ea0d0a196`.
  - Ref2VA: `MiniMax-H3-Ref2VA-Acc-8Step.safetensors`, SHA-256 `111c82e669f6e20e628228172edf39395f1a9fc3ad049793895e542c0f55b18c`.
- LightX2V provides matching FL2VA and Ref2VA 4-NFE adapters in [`lightx2v/Minimax-h3-Turbo`](https://huggingface.co/lightx2v/Minimax-h3-Turbo), pinned at revision `0eebcc7e79f9cb200927c80b8e7595265b770e34`.
  - FL2VA: `minimax_h3_fl2v_turbo_4step_v0.1.safetensors`, SHA-256 `5ff4a12c8b4599fec716e1b15a45e504e0d1129111896bdcde5ac4a15e395b29`.
  - Ref2VA: `minimax_h3_ref2v_turbo_4step_v0.1_bf16.safetensors`, SHA-256 `9e642fc8749c74f8da5e2382877ab5c7aa37b9a73b7fd0d6d457bd1b3cb1ae99`.
- LightX2V reference inference was cross-checked against [`ModelTC/Minimax-H3-Turbo`](https://github.com/ModelTC/Minimax-H3-Turbo), pinned at revision `02e26d591f7a04d5d1a074c9566d5dd4f22f6225`.

Static adapter checks passed. The LightX2V Ref2VA adapter maps all 624 tensors; the Alibaba Ref2VA PDD adapter maps all 728 tensors and declares 32 internal steps with block size 4, giving 8 actual model evaluations.

## Experiment matrix

| Scene | Seed | Option 1 | Option 2 |
|---|---:|---|---|
| `car_drift` | 20260904 | Text-only T2VA, high-motion red-car drift | Ref2VA with the Option-1 Base-50 MP4 as Video 1, including its audio |
| `desert_observatory` | 20260905 | Ref2VA with the official telescope Picture 1 | Ref2VA with Picture 1 plus the Option-1 Base-50 MP4 as audiovisual Video 1 |

Each option contains seven settings: Base 50/25/20/15/10, Alibaba PDD8, and LightX2V Turbo4.

## Controlled protocol

- 960x544, 124 frames, 24 fps, 5.1667 seconds, native 32 kHz stereo audio.
- MiniMax-H3 rectified-flow Euler scheduler with `eta=0`, video shift 12, and audio shift 3.
- Step labels are literal NFE. The pipeline receives `NFE + 1` sigma grid points because the final grid point is terminal zero; it does not count as another transformer forward.
- Base checkpoints and adapter partitions are matched to the workflow: `transformer` + FL2VA LoRA for T2VA, `transformer_ref` + Ref2VA LoRA for reference generation.
- LoRA strength 1.0; LightX2V alpha 8; PDD scheduling parameters come from its checkpoint metadata.
- Reference resize mode is `match`. The complete oracle MP4 is supplied to the reference pipeline; the pipeline performs its normal internal multimodal sampling/encoding.
- Each scene has one saved float32 target video-noise tensor and one saved float32 target audio-noise tensor, injected explicitly into all 14 outputs for that scene.

### Exact noise and oracle checksums

| Scene | Video-noise SHA-256 | Audio-noise SHA-256 | Option-1 Base-50 oracle SHA-256 |
|---|---|---|---|
| `car_drift` | `7532dfeeb7a8a006b7b4dae370b80e2b6d6e9311f524854235456bcf3695e8b9` | `bf5467870ff06b605b724bcad7f24a472fec898f3c19fcc7bbf27e813f43f219` | `2d76ee4eee48a5ace742167cc9389b30d241929b2bc1bb57fc7cdca6b932464a` |
| `desert_observatory` | `a594c2fd78001f47b6a7c4633442690303f9f07c9f0bbb213c7f072e369da20c` | `5e4d49256ad04fe79a1ffd5f2a3ebee2aad5ac09c3c682d74dd639b0f0f987ac` | `9079f7898594fe422f271ae22a76e4657a830cf69be788429c3bb74efc292fb7` |

## Similarity results

Metrics use 16 evenly spaced decoded frames. PSNR and SSIM are pixel-relative; motion cosine compares temporal frame differences. Audio waveform and log-spectrum cosine use the native output audio. Higher is better except that these values should not be treated as perceptual-quality scores.

### Option 1 versus its Base-50 oracle

| Scene | Setting | PSNR dB | SSIM | Motion cosine | Audio waveform | Audio spectrum |
|---|---|---:|---:|---:|---:|---:|
| Car | Base 25 | 15.732 | 0.5719 | 0.5360 | 0.8822 | 0.9480 |
| Car | Base 20 | 15.173 | 0.5035 | 0.4906 | 0.8585 | 0.9528 |
| Car | Base 15 | 13.241 | 0.3795 | 0.3445 | 0.7238 | 0.9341 |
| Car | Base 10 | 12.129 | 0.3445 | 0.1439 | 0.6453 | 0.9240 |
| Car | Alibaba PDD 8 | 11.911 | 0.3805 | 0.1725 | 0.8220 | 0.9326 |
| Car | LightX2V 4 | 9.964 | 0.2977 | -0.0056 | 0.4315 | 0.8725 |
| Desert | Base 25 | 26.606 | 0.8263 | 0.7195 | 0.8958 | 0.9568 |
| Desert | Base 20 | 25.640 | 0.8016 | 0.6696 | 0.8134 | 0.9436 |
| Desert | Base 15 | 25.567 | 0.8058 | 0.6220 | 0.8533 | 0.9562 |
| Desert | Base 10 | 24.794 | 0.7819 | 0.5419 | 0.5733 | 0.8967 |
| Desert | Alibaba PDD 8 | 23.734 | 0.7553 | 0.4781 | 0.7934 | 0.9186 |
| Desert | LightX2V 4 | 16.100 | 0.5152 | 0.0082 | 0.1376 | 0.6531 |

The text-only high-motion car trajectory diverges quickly as Base NFE decreases. In the image-reference desert case, Base10 and PDD8 retain substantially more visual correspondence. PDD8 is competitive with Base10 in both cases and has notably stronger car audio correspondence. LightX2V 4-NFE changes the sampled visual trajectory the most.

### Option 2 versus the Option-1 Base-50 oracle

| Scene | Setting | PSNR dB | SSIM | Motion cosine | Audio waveform | Audio spectrum |
|---|---|---:|---:|---:|---:|---:|
| Car | Base 50 | 27.505 | 0.8283 | 0.9793 | 0.8533 | 0.9112 |
| Car | Base 25 | 27.618 | 0.8245 | 0.9787 | 0.8502 | 0.9157 |
| Car | Base 20 | 27.651 | 0.8223 | 0.9782 | 0.8472 | 0.9167 |
| Car | Base 15 | 27.656 | 0.8213 | 0.9773 | 0.8410 | 0.9178 |
| Car | Base 10 | 27.578 | 0.8177 | 0.9759 | 0.8281 | 0.9178 |
| Car | Alibaba PDD 8 | 27.393 | 0.8158 | 0.9777 | 0.8174 | 0.9222 |
| Car | LightX2V 4 | 17.205 | 0.5569 | 0.5294 | 0.7923 | 0.9196 |
| Desert | Base 50 | 33.239 | 0.9238 | 0.9637 | 0.7328 | 0.8945 |
| Desert | Base 25 | 33.349 | 0.9242 | 0.9638 | 0.7393 | 0.8982 |
| Desert | Base 20 | 33.362 | 0.9234 | 0.9631 | 0.7437 | 0.9022 |
| Desert | Base 15 | 33.370 | 0.9230 | 0.9629 | 0.7369 | 0.9050 |
| Desert | Base 10 | 33.308 | 0.9221 | 0.9605 | 0.7012 | 0.9026 |
| Desert | Alibaba PDD 8 | 33.290 | 0.9227 | 0.9625 | 0.6911 | 0.9088 |
| Desert | LightX2V 4 | 16.246 | 0.5033 | 0.0043 | 0.6818 | 0.9217 |

The reference does not produce a pixel copy: even Option2 Base50 is a regenerated output. Nevertheless, the Base10 and PDD8 rows remain nearly flat relative to Base50 in both oracle-reference cases. Supplying the oracle raises PDD8 PSNR by +15.482 dB for car and +9.556 dB for desert relative to Option 1; motion cosine rises by +0.8053 and +0.4844 respectively.

LightX2V 4-NFE is different: for car, the oracle adds +7.241 dB PSNR and +0.5349 motion cosine but still trails the Base/PDD group. For desert, visual gain is negligible (+0.145 dB PSNR, -0.0119 SSIM), while audio waveform and spectrum cosine improve by +0.5441 and +0.2686.

### Low-NFE consistency with each option's own Base50

| Scene | Option | Setting | PSNR dB | SSIM | Motion cosine |
|---|---|---|---:|---:|---:|
| Car | Option 2 | Base 10 | 30.486 | 0.9180 | 0.9874 |
| Car | Option 2 | Alibaba PDD 8 | 32.175 | 0.9348 | 0.9916 |
| Car | Option 2 | LightX2V 4 | 16.677 | 0.5797 | 0.5267 |
| Desert | Option 2 | Base 10 | 38.151 | 0.9629 | 0.9825 |
| Desert | Option 2 | Alibaba PDD 8 | 38.645 | 0.9646 | 0.9843 |
| Desert | Option 2 | LightX2V 4 | 16.002 | 0.4822 | 0.0040 |

Within Option 2, PDD8 is slightly closer to that option's Base50 visual trajectory than Base10 in both scenes. Full per-row oracle and within-option values are in `metrics.json` and the CSV files.

## Runtime and loading

Generation seconds include conditioning, denoising, VAE decode, and media export; checkpoint load is recorded separately. Reference-generation runtime has a large fixed cost from image/video/audio encoding, so NFE reduction gives less than a linear speedup.

| Scene / option | Base50 | Base10 | PDD8 | LightX4 | LightX4 speedup vs Base50 |
|---|---:|---:|---:|---:|---:|
| Car Option 1, text T2VA | 130.4 s | 29.3 s | 28.5 s | 16.4 s | 7.97x |
| Desert Option 1, image Ref2VA | 152.7 s | 43.3 s | 43.8 s | 28.3 s | 5.39x |
| Car Option 2, video-oracle Ref2VA | 844.8 s | 301.1 s | 267.4 s | 215.7 s | 3.92x |
| Desert Option 2, image+video-oracle Ref2VA | 866.0 s | 313.1 s | 289.0 s | 220.1 s | 3.93x |

The optimized loader stages about 197 GiB of immutable model assets once per node to `/raid/scratch/shuchenx/minimax-h3`, initializes large modules on meta, wraps them with FSDP2, and lets every rank read only its local safetensor rows from NVMe. Recorded load time is 8.1-11.7 seconds per model/adapter setting, rather than repeatedly materializing the complete checkpoint from Lustre on every rank.

The denoising transformer uses FlashAttention 3. The native-FP32 audio VAE is explicitly pinned to its native attention backend; this avoids a process-global Diffusers backend dispatch that otherwise sends FP32 audio Q/K/V to FA3. All project tests passed after the fix (`4 passed`). Pre-fix diagnostic failures remain under `controllers/*/failed/` for auditability; none are part of the validated 28-output matrix.

## Validation

`validation.json` confirms:

- 28/28 output and metadata pairs exist.
- Every MP4 has one 960x544, 124-frame, 24-fps video stream and one 32-kHz stereo audio stream.
- Literal NFE/grid size and Euler scheduler metadata match all seven variants.
- Exact target video/audio noise hashes match within each scene.
- T2VA/Ref2VA transformer and LoRA partitions match.
- Every Option-2 video-reference hash equals its Option-1 Base-50 oracle hash.
- All four 2x4 composites decode end-to-end and contain 3840x1088 H.264 video plus 32-kHz stereo audio.
- Slurm jobs 6557794 and 6557795 exited and both interactive allocations were released.

## Composite layout and audio

- Option 1: top row Base50 / Base25 / Base20 / Base15; bottom row Base10 / Alibaba PDD8 / LightX2V4 / input reference tile.
- Option 2: top-left is the Option-1 Base50 oracle, followed by Option2 Base50 / Base25 / Base20; bottom row Option2 Base15 / Base10 / Alibaba PDD8 / LightX2V4.
- The Option-1 composite audio is copied from its Base50 panel. The Option-2 composite audio is copied from the oracle panel, which is the same Option-1 Base50 audiovisual file. Individual MP4s retain their own generated audio and are what the audio metrics use.

## Files

- `configs/oracle_reference_experiment.json`: pinned provenance, prompts, protocol, and matrix.
- `validation.json`: full per-output validation records and hashes.
- `metrics.json`: complete metrics payload.
- `metrics_vs_oracle.csv`, `metrics_vs_own_base50.csv`, `metrics_reference_gain.csv`: tabular metrics.
- `metrics_summary.md`: generated metric tables.
- `comparison_videos/`: four labeled 2x4 MP4s.
- `previews/`: inspected middle-frame previews.
- `outputs/`: all 28 individual audiovisual outputs and metadata.
- `noise/`: the two exact saved target-noise tensor pairs.
- `controllers/`: reproducible two-node task scripts and logs.
- `src/run_oracle_reference_fsdp.py`, `src/meta_fsdp_loader.py`: inference and rank-local loader implementation.

## Limitations

- Two hand-selected prompts reveal useful behavior but cannot establish general quality rankings.
- Pixel metrics penalize plausible changes in composition and are not substitutes for perceptual or preference evaluation.
- Option 1 and Option 2 are different workflows and use different official transformer partitions; same initial target noise controls one source of variation but cannot make their latent trajectories identical.
- The full-video reference makes Option 2 intentionally reconstruction-oriented and creates substantial fixed conditioning cost.