Rune

Model transformations without brittle graph scripts.

Rune is a powerful graph rewrite system for ONNX, TFLite and torch.export models.

With simple, auditable rules, Rune finds every match site in the model, applies replacements, verifies the result against the original and provides evidence.

$ pip install byteinfer-rune

The problem

Model graphs are often unclean

  • Exported models carry training primitives the inference runtime does not need. That bloat costs latency.
  • Runtimes have fast fused kernels, but their pattern fusers only recognise specific export layouts. Everything else runs op by op.
  • The usual fix is a Python script that edits the graph file. It works for one model and breaks on the next export or opset.
  • Hundreds of lines of graph-editing Python are hard to audit. Silent bugs slip through and nobody catches them.

What Rune does instead

  • Rune rules describe find-and-replace patterns that are short and readable.
  • If the pattern exists, Rune rewrites it. If it doesn't, nothing is changed.
  • One rewrite system, multiple model formats: ONNX, torch.export, TFLite.
V1edit_model.pyedits 12 siteslayernorm.rune12 sites, verifiedV2 · NEW EXPORTedit_model.pyedits the wrong nodesno errorlayernorm.rune12 sites, verifiedV3 · NEW OPSETedit_model.pyedits nothingno errorlayernorm.runerefused: no matchmodel untouchedthe script edits the file;the rule matches the pattern V1modelV2 · NEW EXPORTmodelV3 · NEW OPSETmodeledit_model.pyedits 12 sitesedits the wrong nodesno erroredits nothingno errorlayernorm.rune12 sites, verified12 sites, verifiedrefused: no matchmodel untouchedthe script edits the file; the rule matches the pattern

How it works

What a rule does

Find → Check → Apply → Verify

  1. 1
    Find

    You describe a pattern. Rune finds every place it appears in the graph.

  2. 2
    Check

    Each match is simulated on a copy first. If it would break the graph, the gate refuses it.

  3. 3
    Apply

    Only the changes that passed get applied, all at once.

  4. 4
    Verify

    The rewritten model is run and its outputs compared with the original.

the ruleMul#m -> Add#a -> Relu#r
+ FusedMulAddRelu#f
> m.input[0] -> f.input[0]  ...

swipe to pan →

Case studies

Real model rewrites

BERT

BERT LayerNorm

  • Replace TensorFlow's 13-op LayerNorm expansion with the native ONNX operation.

the rule bert_tf_fusion.rune

37 rewrites · outputs match · +5.07% throughput

removedinsertededge cut

swipe to pan →

LLM serving

LLM prefill + decode

  • Split one decoder export into dedicated prefill and KV-cache decode graphs.

the rules smollm_prefill.rune smollm_decode.rune

512 tokens verified · 2.67× faster decode

ONE EXPORT decoderstatic loop, no cache rule 1 rule 2 TWO GRAPHS prefillprompt in, caches out decodetoken + past cache in KV cache next token in, cache reused weightsone copy, shared

swipe to pan →

SmolLM

SmolLM GQA

  • Replace the exported 126-op attention block with the fused runtime operation.

the rule smollm_gqa_fusion_fp16.rune

30 blocks rewritten · accuracy preserved · +9.8% step performance

removedinsertededge cut

swipe to pan →

LoRA

LoRA

  • Inject adapters directly into an exported ONNX model, train them, then fold them back into the base weights.

the rules smollm_lora_qv.rune smollm_lora_merge.rune

60 injection sites · identical before training · merged within 5e-4

BASE x q_proj Reshape → INJECTED x q_proj + Reshape → A B ×2 rank 8, trainable MERGED x q_proj W + 2AB folded in Reshape →

swipe to pan →

Measured

Real models, real numbers

Each number is a paired A/B measurement on the real model. Every rewrite passed an equivalence gate before it ran.

Show results
ONNX Runtime · CUDA1.47×BERT-SQuADLayerNorm + GELU fusion, TF exportproven within 1e-2found by rune-cli plan
ONNX Runtime · CUDA1.18×SqueezeNet 1.1Discovered sibling-conv merge, all eight Fire modulesexact · within 1e-4found by rune-cli plan
ONNX Runtime · CUDA1.05×MiniLM-L6-v2Eager attention → MultiHeadAttentionexact · max err 1.2e-6bert_hf_mha_fusion.rune
LiteRT1.27×MobileBERTSplit-FC-Pack → BATCH_MATMULwithin 1e-4 · 48 sitesattention_batch_matmul.rune
LiteRT1.54×BERT classifier · MediaPipeSplit-FC-Pack → BATCH_MATMULwithin 1e-3 · 48 sitesattention_batch_matmul.rune
torch · eager1.33×Llama-3.2-1B · decoderepeat_kv → sdpa enable_gqabit-identical · 16 layersgqa_sdpa_enable_gqa.rune
torch · eager1.42×Qwen3-0.6B · decoderepeat_kv → sdpa enable_gqabit-identical · 28 layersgqa_sdpa_enable_gqa.rune

In the wild

Real issues, one rule each

Issues from the runtimes' own trackers. We re-checked each one on current releases: some were closed without a fix, and one fix no longer matches today's exports.

Verified on ORT 1.23 · transformers 5.13 · PyTorch 2.13

Show cases

TFLite

Unrolled batched matmul

An old TFLite converter pass replaced batched matmuls with per-head Split, FullyConnected, Pack stacks. Both reports were closed stale, and models built with that converter still ship: MediaPipe's BERT classifier has 48 of these stacks. One rule replaces each with BATCH_MATMUL.

tensorflow#56984tensorflow#47425attention_batch_matmul.rune

48 sites · within 1e-3 · 1.54× on LiteRT

ONNX Runtime

Attention layouts the fuser skips

ORT's attention fuser is patched one export layout at a time. The CLIP report was fixed in 2024; we re-ran it with transformers 5.13 and ORT 1.23 and the attention comes out unfused again — the export layout changed. MiniLM's eager export was never covered. One rule maps the layout to MultiHeadAttention directly.

onnxruntime#21208onnxruntime#23864bert_hf_mha_fusion.rune

6 of 6 layers fused · exact · 1.05× on CUDA

PyTorch

Materialised KV head copies

GQA models repeat K/V heads before sdpa, materialising copies that grow with the cache. PyTorch's fix is enable_gqa — we verified it bit-identical and 2× faster at a 2048-token decode shape. Exported models still carry the repeat chain, so one rule removes it and sets enable_gqa on the same call.

pytorch#154363gqa_sdpa_enable_gqa.rune

bit-identical · Llama 1.33× · Qwen 1.42× at 2048 tokens

One rule, wherever the pattern appears

ReduceMean#mean -> Identity#sg -> Sub#center -> Mul#sq -> ReduceMean#var -> Add#addeps -> Sqrt#sqrt -> Reciprocal#rstd -> Mul#gmul -> Mul#mul2 -> Sub#bsub -> Add#lnoutmatch the TensorFlow LayerNorm chain
+ LayerNormalization#ln epsilon=1e-12 axis=-1add one native LayerNorm node
> mean.input[0] -> ln.input[X]route the input to the new node
> gmul.input[1] -> ln.input[Scale]connect the scale weight
> bsub.input[0] -> ln.input[B]connect the bias weight
> ln.output[Y] -> lnout.output[0]wire the output to its consumers

Rune finds every matching instance and checks each candidate before committing the rewrite.

Workflow

Multiple expertise levels

Just run it. One command picks rules, applies them, verifies the output.

rune-cli run model.onnx --target ort-cpu --output model_optimized.onnx

Review first. Generate a plan, inspect it, then apply.

rune-cli plan model.onnx --target ort-cpu
rune-cli apply model.onnx model.plan

Full control. optimize, validate, query, analyze, discover are individual commands for when you need them.

Every run leaves a receipt: which rules ran, tagged and versioned. rune-cli versions, reproduce, diff.

REPORT shufflenet-v2-10.onnx → shufflenet-v2-10_bn_fold.onnx graph 8b85e969 → 7bc68317 · rules: shufflenet_bn_fold.rune 261 → 205nodes 56sites matched 0gate refusals passedoutputs match SITE 1 OF 56 · Conv#conv -> BatchNormalization#bn before input Conv BatchNormalization out after input Conv out weights folded OPERATOR CENSUS BatchNormalization   56 → 0 ⋮ 55 more sites, each drawn before and after

swipe to pan →

Automation

Works with coding agents

Rune ships with skills for inspecting models, writing rules, validating them and running rewrites.

The agent writes the transformation. Rune decides whether it is safe to apply.

rune-setupinstall and verify the environment→rune-stageanalyse a model, find rewrite candidates→rune-authorwrite and validate rulesets until gate-clean→rune-executerun a validated ruleset with provenance and a report + rune-playgroundwork alongside a person in the browser playground

Research

Automatic discovery

experimental

  • Rune can inspect a model and propose reusable .rune rules automatically.
  • The output is normal Rune code: readable, editable and reusable.
  • Convolution and attention patterns are supported today.
rune-cli discover model.onnx --out rules/

swipe to pan →

Before you start

Current state

  • Rune is under active development.
  • ONNX and torch.export are the most mature targets. TFLite has the same safety gates but narrower operator coverage.
  • Structural checks happen during rewriting. Numerical equivalence is verified separately by running the rewritten model against the original.
EVERY REWRITEyour rulegatestructure · shadow copyverifynumbers vs originalmodelrefusedmodel untouchedFORMATSONNXstabletorch.exportstableTFLiteexperimentalEXECUTORCH, ONE WAYtorch.exportRunelower.pte EVERY REWRITE your rule gate verify model structure shadow copy numbers vs the original refused · model untouched FORMATS ONNX stable torch.export stable TFLite experimental EXECUTORCH, ONE WAY torch.export Rune lower .pte