RSIBench-Infra
Can agents optimize their own
training and inference infrastructure?
From kernels to the full software stack. Evaluating AI agents on
real, production-scale infrastructure optimization.
¹ ByteDance Seed ² Tsinghua AIR ³ SIA-Lab
* Equal contribution · † Corresponding author
RSIBench-Infra-Lite leaderboard.
Mean normalized score; Claude uses its three tested workloads, other models use all five.
† Claude Opus 5.5 · 3/5 tasks tested. Faded bars with a dagger mark partial coverage; its mean includes only tasks with formal scores.
Model scores by workload
Five workloads, seven models, one shared scale. Hover or use arrow keys to compare exact scores.
Scroll horizontally to explore all five workloads.
- Qwen3.8-27B, Claude Opus 5.5: 0.000
- Qwen3.8-27B, Qwen3.8 Max: 0.700
- Qwen3.8-27B, GLM-5.3: 0.718
- Qwen3.8-27B, GPT-6 Astra: 0.731
- Qwen3.8-27B, Seed Evolving Auto: 0.716
- Qwen3.8-27B, Kimi K3: 0.606
- Qwen3.8-27B, DeepSeek V4 Flash: 0.717
- Muse-Glimmer-30B, Claude Opus 5.5: 0.000
- Muse-Glimmer-30B, Qwen3.8 Max: 0.618
- Muse-Glimmer-30B, GLM-5.3: 0.563
- Muse-Glimmer-30B, GPT-6 Astra: 0.600
- Muse-Glimmer-30B, Seed Evolving Auto: 0.559
- Muse-Glimmer-30B, Kimi K3: 0.562
- Muse-Glimmer-30B, DeepSeek V4 Flash: 0.560
- Qwen3-VL-32B, Claude Opus 5.5: 0.620
- Qwen3-VL-32B, Qwen3.8 Max: 0.583
- Qwen3-VL-32B, GLM-5.3: 0.679
- Qwen3-VL-32B, GPT-6 Astra: 0.550
- Qwen3-VL-32B, Seed Evolving Auto: 0.511
- Qwen3-VL-32B, Kimi K3: 0.549
- Qwen3-VL-32B, DeepSeek V4 Flash: 0.540
- LLaDA2.1-mini-16B, Claude Opus 5.5: 0.978
- LLaDA2.1-mini-16B, Qwen3.8 Max: 0.715
- LLaDA2.1-mini-16B, GLM-5.3: 0.542
- LLaDA2.1-mini-16B, GPT-6 Astra: 0.826
- LLaDA2.1-mini-16B, Seed Evolving Auto: 0.530
- LLaDA2.1-mini-16B, Kimi K3: 0.528
- LLaDA2.1-mini-16B, DeepSeek V4 Flash: 0.725
- Wan2.2-TI2V-5B, Claude Opus 5.5: 0.796
- Wan2.2-TI2V-5B, Qwen3.8 Max: 0.732
- Wan2.2-TI2V-5B, GLM-5.3: 0.713
- Wan2.2-TI2V-5B, GPT-6 Astra: 0.500
- Wan2.2-TI2V-5B, Seed Evolving Auto: 0.693
- Wan2.2-TI2V-5B, Kimi K3: 0.740
- Wan2.2-TI2V-5B, DeepSeek V4 Flash: 0.000
Normalization follows SOL-ExecBench: . Oracle = 0.5 and SOL = 1.0. Scores use the November 12, 2026 SOL references and the selected experimental results. Speedups at or above SOL receive 1; missing or zero reward receives 0. The Lite mean includes the three formally scored tasks for Claude (including verified zero rewards), and all five tasks for other models. These derived scores are not official rewards. SOL estimates use AISimulate evidence with the task-specific caveats described in the paper.
RSIBench-Infra leaderboard.
How often can an agent make a real workload faster?
† Claude Opus 5.5 · 10/20 tasks tested. Faded bars with a dagger mark partial coverage; all aggregate metrics use only its 10 formally scored tasks.
GPT-6 Astra
#2GLM-5.3
#3Qwen3.8 Max
#4DeepSeek V4 Flash
#5Kimi K3
#6Seed Evolving Auto
#7Claude Opus 5.5: Only the 10 completed tasks are included. The other 10 of 20 tasks could not be tested because the following API error was triggered:
API Error: 400 This request may violate the Claude model's safety policy and has been terminated early.
Untested tasks: infer-deepseekv4flash-284b, infer-glm5p2-744b, infer-glm5p3-744b, infer-glm5p3flash-320b, infer-kimik3-2p8t, infer-museglimmer-30b, infer-nemotron3ultra-550a55b, infer-qwen3p8-27b, train-qwen3-8b-opd, train-qwen3vl-4b-grpo. Their speedup is unavailable and their per-task SOL score is 0; these untested tasks are excluded from Claude’s aggregate metrics. Verified zero rewards remain included.
RSIBench-Infra agent trajectory explorer.
Each cell shows speedup and normalized SOL score. Click to explore the original Agent trajectory.
| WORKLOAD ↓ | Claude Opus 5.5 | GPT-6 Astra | GLM-5.3 | Qwen3.8 Max | DeepSeek V4 Flash | Kimi K3 | Seed Evolving Auto |
|---|---|---|---|---|---|---|---|
| 01 DeepSeek V4 FlashINFERENCE | |||||||
| 02 GLM-5.2INFERENCE | |||||||
| 03 GLM-5.3INFERENCE | |||||||
| 04 GLM-5.3 FlashINFERENCE | |||||||
| 05 InklingINFERENCE | |||||||
| 06 Kimi K3INFERENCE | |||||||
| 07 LingBot World 2.0INFERENCE | |||||||
| 08 LLaDA 2.1 MiniINFERENCE | |||||||
| 09 MiniMax H3INFERENCE | |||||||
| 10 Muse GlimmerINFERENCE | |||||||
| 11 Nemotron 3 UltraINFERENCE | |||||||
| 12 π0.5 LIBEROINFERENCE | |||||||
| 13 Qwen3.8INFERENCE | |||||||
| 14 Qwen3 TTSINFERENCE | |||||||
| 15 Qwen3-VLINFERENCE | |||||||
| 16 Wan 2.2INFERENCE | |||||||
| 17 Qwen3 · GRPOTRAINING | |||||||
| 18 MiMo 7B RL · SFTTRAINING | |||||||
| 19 Qwen3 · OPDTRAINING | |||||||
| 20 Qwen3-VL · GRPOTRAINING |
SOL references · November 12, 2026
Analytical references are task-specific estimates, not measured throughput. GLM-5.2, GLM-5.3 Flash, Kimi K3 and Nemotron 3 Ultra use prefix-average extrapolations. Their workload and timing boundaries differ from the historical baseline; normalized scores are descriptive comparisons.
| Workload | Baseline | SOL | Unit | Reference type |
|---|---|---|---|---|
| DeepSeek V4 Flash(0731 W8A8) | 160.620461 | 1083.400789 | tokens/s | audited analytical reference |
| GLM-5.2 | 12.932500 | 658.039006 | tokens/s | prefix-average extrapolation |
| GLM-5.3 FP8 | 113.460833 | 378.201944 | tokens/s | audited analytical reference |
| GLM-5.3-Flash 320B W8A8 | 4.885000 | 176.883406 | tokens/s | prefix-average extrapolation |
| Inkling 975B(NVFP4 W4A16) | 104.370000 | 586.622688 | tokens/s | existing analytical reference |
| Kimi-K3(2.8T MXFP4) | 26.246667 | 201.485474 | tokens/s | prefix-average extrapolation |
| LingBot-World 2.0 | 7.542009 | 10.784332 | frames/s | existing analytical reference |
| LLaDA2.1-mini(BF16) | 92.709178 | 642.288852 | tokens/s | existing analytical reference |
| MiniMax-H3 FL2VA | 202.796861 | 301.532998 | videos/hour | existing analytical reference |
| Muse-Glimmer-30B(BF16, text-only) | 47.807500 | 71.141944 | tokens/s | existing analytical reference |
| Nemotron 3 Ultra(550B-A55B NVFP4) | 26.690556 | 136.266420 | tokens/s | prefix-average extrapolation |
| π0.5 | 2.707846 | 31.217109 | action_chunks/s | existing analytical reference |
| Qwen3.8-27B(FP8) | 108.927500 | 300.224167 | tokens/s | existing analytical reference |
| Qwen3-TTS-12Hz-1.7B-Base | 9.831496 | 47.555649 | reference-audio-seconds/s | existing analytical reference |
| Qwen3-VL-32B(FP8) | 106.079385 | 301.400723 | tokens/s | existing analytical reference |
| Wan2.2 TI2V-5B (T2V-only) | 6.628043 | 10.209773 | videos/hour | existing analytical reference |
| Qwen3-30B-A3B-Base | 5.888738 | 16.160189 | step/hour | audited analytical reference |
| MiMo-7B-RL | 1472.256060 | 2228.113649 | step/hour | existing analytical reference |
| Qwen3-8B | 53.765306 | 78.768159 | step/hour | existing analytical reference |
| Qwen3-VL-8B-Instruct | 4.509013 | 7.375054 | step/hour | existing analytical reference |
RSIBench-Infra-Lite
& RSIBench-Infra.
RSIBench-Infra evaluates 20 production-scale training and inference workloads. RSIBench-Infra-Lite focuses on five inference workloads, with normalized scores relative to Oracle and analytical SOL references.
Code coming soonFreedom across the stack
Agents can modify the software stack inside a sandbox, rather than being limited to a single layer.
End-to-end evaluation
16 inference and 4 training tasks measure performance on real workloads, with task-specific verification.
Reproducible submissions
An OCI image layer submission protocol captures changes across the environment, beyond a Git patch.