RSIBench / Infra Code coming soon

RSIBench-Infra

Can agents optimize their own
training and inference infrastructure?

From kernels to the full software stack. Evaluating AI agents on
real, production-scale infrastructure optimization.

Hanlin Wu1,2,3,*Yifei Wang1,2,3,* Zhicheng Cai1,2,3Huan-ang Gao1,2,3
Wei-Ying Ma2,3Ya-Qin Zhang2,3 Jingjing Liu2,3Mingxuan Wang1,3 Hao Zhou2,3,†

¹ ByteDance Seed ² Tsinghua AIR ³ SIA-Lab

* Equal contribution · † Corresponding author

Explore the results Code coming soon
20real-world tasks
16 + 4inference + training
7evaluated models
5Lite Set tasks
01 / RSIBench-Infra-Lite

RSIBench-Infra-Lite leaderboard.

Mean normalized score; Claude uses its three tested workloads, other models use all five.

Mean normalized Agent scoreOracle = 0.5SOL = 1

† Claude Opus 5.5 · 3/5 tasks tested. Faded bars with a dagger mark partial coverage; its mean includes only tasks with formal scores.

Model scores by workload

Five workloads, seven models, one shared scale. Hover or use arrow keys to compare exact scores.

Claude Opus 5.5 †Qwen3.8 MaxGLM-5.3GPT-6 AstraSeed Evolving AutoKimi K3DeepSeek V4 Flash
Normalized Agent score · Higher is betterOracle = 0.5SOL = 1

Scroll horizontally to explore all five workloads.

  • Qwen3.8-27B, Claude Opus 5.5: 0.000
  • Qwen3.8-27B, Qwen3.8 Max: 0.700
  • Qwen3.8-27B, GLM-5.3: 0.718
  • Qwen3.8-27B, GPT-6 Astra: 0.731
  • Qwen3.8-27B, Seed Evolving Auto: 0.716
  • Qwen3.8-27B, Kimi K3: 0.606
  • Qwen3.8-27B, DeepSeek V4 Flash: 0.717
  • Muse-Glimmer-30B, Claude Opus 5.5: 0.000
  • Muse-Glimmer-30B, Qwen3.8 Max: 0.618
  • Muse-Glimmer-30B, GLM-5.3: 0.563
  • Muse-Glimmer-30B, GPT-6 Astra: 0.600
  • Muse-Glimmer-30B, Seed Evolving Auto: 0.559
  • Muse-Glimmer-30B, Kimi K3: 0.562
  • Muse-Glimmer-30B, DeepSeek V4 Flash: 0.560
  • Qwen3-VL-32B, Claude Opus 5.5: 0.620
  • Qwen3-VL-32B, Qwen3.8 Max: 0.583
  • Qwen3-VL-32B, GLM-5.3: 0.679
  • Qwen3-VL-32B, GPT-6 Astra: 0.550
  • Qwen3-VL-32B, Seed Evolving Auto: 0.511
  • Qwen3-VL-32B, Kimi K3: 0.549
  • Qwen3-VL-32B, DeepSeek V4 Flash: 0.540
  • LLaDA2.1-mini-16B, Claude Opus 5.5: 0.978
  • LLaDA2.1-mini-16B, Qwen3.8 Max: 0.715
  • LLaDA2.1-mini-16B, GLM-5.3: 0.542
  • LLaDA2.1-mini-16B, GPT-6 Astra: 0.826
  • LLaDA2.1-mini-16B, Seed Evolving Auto: 0.530
  • LLaDA2.1-mini-16B, Kimi K3: 0.528
  • LLaDA2.1-mini-16B, DeepSeek V4 Flash: 0.725
  • Wan2.2-TI2V-5B, Claude Opus 5.5: 0.796
  • Wan2.2-TI2V-5B, Qwen3.8 Max: 0.732
  • Wan2.2-TI2V-5B, GLM-5.3: 0.713
  • Wan2.2-TI2V-5B, GPT-6 Astra: 0.500
  • Wan2.2-TI2V-5B, Seed Evolving Auto: 0.693
  • Wan2.2-TI2V-5B, Kimi K3: 0.740
  • Wan2.2-TI2V-5B, DeepSeek V4 Flash: 0.000

Normalization follows SOL-ExecBench: score={0missing or zero reward1Tk≤TSOL11+Tk−TSOLTb−TSOLotherwise\mathrm{score} = \begin{cases} 0 & \text{missing or zero reward} \\ 1 & T_k \leq T_{\mathrm{SOL}} \\ \displaystyle\frac{1}{1 + \frac{T_k - T_{\mathrm{SOL}}}{T_b - T_{\mathrm{SOL}}}} & \text{otherwise} \end{cases}. Oracle = 0.5 and SOL = 1.0. Scores use the November 12, 2026 SOL references and the selected experimental results. Speedups at or above SOL receive 1; missing or zero reward receives 0. The Lite mean includes the three formally scored tasks for Claude (including verified zero rewards), and all five tasks for other models. These derived scores are not official rewards. SOL estimates use AISimulate evidence with the task-specific caveats described in the paper.

02 / RSIBench-Infra

RSIBench-Infra leaderboard.

How often can an agent make a real workload faster?

Higher is better

† Claude Opus 5.5 · 10/20 tasks tested. Faded bars with a dagger mark partial coverage; all aggregate metrics use only its 10 formally scored tasks.

Faster rate · Share of evaluated tasks with speedup > 1.10× (Claude: 10; others: 20).Median speedup · Across formally scored tasks for Claude; all 20 tasks for other models.Mean SOL score · Mean normalized score, capped at 1 per task; Claude: 10 formally scored tasks, others: all 20.

Claude Opus 5.5 †

#1
80%
1.34×median speedup
10/20 scoredMean SOL: 0.711

GPT-6 Astra

#2
80%
1.33×median speedup
20/20 scoredMean SOL: 0.667

GLM-5.3

#3
70%
1.28×median speedup
20/20 scoredMean SOL: 0.626

Qwen3.8 Max

#4
70%
1.21×median speedup
20/20 scoredMean SOL: 0.577

DeepSeek V4 Flash

#5
60%
1.28×median speedup
20/20 scoredMean SOL: 0.614

Kimi K3

#6
60%
1.13×median speedup
20/20 scoredMean SOL: 0.627

Seed Evolving Auto

#7
50%
1.09×median speedup
20/20 scoredMean SOL: 0.570

Claude Opus 5.5: Only the 10 completed tasks are included. The other 10 of 20 tasks could not be tested because the following API error was triggered:

API Error: 400 This request may violate the Claude model's safety policy and has been terminated early.

Untested tasks: infer-deepseekv4flash-284b, infer-glm5p2-744b, infer-glm5p3-744b, infer-glm5p3flash-320b, infer-kimik3-2p8t, infer-museglimmer-30b, infer-nemotron3ultra-550a55b, infer-qwen3p8-27b, train-qwen3-8b-opd, train-qwen3vl-4b-grpo. Their speedup is unavailable and their per-task SOL score is 0; these untested tasks are excluded from Claude’s aggregate metrics. Verified zero rewards remain included.

03 / RSIBench-Infra

RSIBench-Infra agent trajectory explorer.

Each cell shows speedup and normalized SOL score. Click to explore the original Agent trajectory.

Verified speedup and normalized SOL score by task and model. Click any cell to view the original Agent trajectory. Missing speedup is a dash; missing reward contributes a SOL score of zero.
WORKLOAD ↓Claude Opus 5.5GPT-6 AstraGLM-5.3Qwen3.8 MaxDeepSeek V4 FlashKimi K3Seed Evolving Auto
01
DeepSeek V4 FlashINFERENCE
02
GLM-5.2INFERENCE
03
GLM-5.3INFERENCE
04
GLM-5.3 FlashINFERENCE
05
InklingINFERENCE
06
Kimi K3INFERENCE
07
LingBot World 2.0INFERENCE
08
LLaDA 2.1 MiniINFERENCE
09
MiniMax H3INFERENCE
10
Muse GlimmerINFERENCE
11
Nemotron 3 UltraINFERENCE
12
π0.5 LIBEROINFERENCE
13
Qwen3.8INFERENCE
14
Qwen3 TTSINFERENCE
15
Qwen3-VLINFERENCE
16
Wan 2.2INFERENCE
17
Qwen3 · GRPOTRAINING
18
MiMo 7B RL · SFTTRAINING
19
Qwen3 · OPDTRAINING
20
Qwen3-VL · GRPOTRAINING
Color intensity indicates speedup Best on task (ties included)— No verified score0.00× Verified zero rewardSOL score ∈ [0, 1]; missing reward = 0
SOL references · November 12, 2026

Analytical references are task-specific estimates, not measured throughput. GLM-5.2, GLM-5.3 Flash, Kimi K3 and Nemotron 3 Ultra use prefix-average extrapolations. Their workload and timing boundaries differ from the historical baseline; normalized scores are descriptive comparisons.

WorkloadBaselineSOLUnitReference type
DeepSeek V4 Flash(0731 W8A8)160.6204611083.400789tokens/saudited analytical reference
GLM-5.212.932500658.039006tokens/sprefix-average extrapolation
GLM-5.3 FP8113.460833378.201944tokens/saudited analytical reference
GLM-5.3-Flash 320B W8A84.885000176.883406tokens/sprefix-average extrapolation
Inkling 975B(NVFP4 W4A16)104.370000586.622688tokens/sexisting analytical reference
Kimi-K3(2.8T MXFP4)26.246667201.485474tokens/sprefix-average extrapolation
LingBot-World 2.07.54200910.784332frames/sexisting analytical reference
LLaDA2.1-mini(BF16)92.709178642.288852tokens/sexisting analytical reference
MiniMax-H3 FL2VA202.796861301.532998videos/hourexisting analytical reference
Muse-Glimmer-30B(BF16, text-only)47.80750071.141944tokens/sexisting analytical reference
Nemotron 3 Ultra(550B-A55B NVFP4)26.690556136.266420tokens/sprefix-average extrapolation
π0.52.70784631.217109action_chunks/sexisting analytical reference
Qwen3.8-27B(FP8)108.927500300.224167tokens/sexisting analytical reference
Qwen3-TTS-12Hz-1.7B-Base9.83149647.555649reference-audio-seconds/sexisting analytical reference
Qwen3-VL-32B(FP8)106.079385301.400723tokens/sexisting analytical reference
Wan2.2 TI2V-5B (T2V-only)6.62804310.209773videos/hourexisting analytical reference
Qwen3-30B-A3B-Base5.88873816.160189step/houraudited analytical reference
MiMo-7B-RL1472.2560602228.113649step/hourexisting analytical reference
Qwen3-8B53.76530678.768159step/hourexisting analytical reference
Qwen3-VL-8B-Instruct4.5090137.375054step/hourexisting analytical reference
04 / THE BENCHMARKS

RSIBench-Infra-Lite
& RSIBench-Infra.

RSIBench-Infra evaluates 20 production-scale training and inference workloads. RSIBench-Infra-Lite focuses on five inference workloads, with normalized scores relative to Oracle and analytical SOL references.

Code coming soon
01

Freedom across the stack

Agents can modify the software stack inside a sandbox, rather than being limited to a single layer.

02

End-to-end evaluation

16 inference and 4 training tasks measure performance on real workloads, with task-specific verification.

03

Reproducible submissions

An OCI image layer submission protocol captures changes across the environment, beyond a Git patch.