Hermes Wiki
Raw/rtk-harness-integration

RTK × Agent Harness — Efficiency Architecture

Central thesis: The harness ladder tells you what infrastructure makes agents reliable. RTK tells you how cheaply you can run it. Together they make the case that a small local model + a strong harness + compressed tool output can match or beat a large cloud model with no harness — at a fraction of the cost.

The Problem RTK Solves Inside a Multi-Agent Pipeline

The multi_agent_coding pipeline has four nodes:

PLANNER → CODER → EVALUATOR → DEBUGGER → EVALUATOR (retry loop)

Every LLM node receives a context window built from:

  • the original task description
  • prior agent outputs (plan, code)
  • tool call returns — and this is where cost explodes

The EVALUATOR runs pytest --tb=short and hands the raw output to the DEBUGGER. On any realistic test suite, that raw pytest output is between 2,000 and 13,000 tokens of tracebacks, assertion diffs, file paths, and boilerplate headers. The DEBUGGER only needs the assertion lines and failed test names — maybe 100–200 tokens of signal. The rest is noise the LLM has to wade through on every debug iteration.

Raw pytest output is the single most compressible content type in any coding agent pipeline, and the DEBUGGER is the node that burns the most tokens because it runs up to three times. That is exactly where RTK pays off.

Where RTK Plugs Into the Harness (3 Integration Points)

Integration Point 1 — run_tests / _run_tests ← highest value

Both projects run pytest as a subprocess and return raw output.

harness_ladder/tools/factory.py, line 89:

# Current
result = subprocess.run(
    ["python", "-m", "pytest", str(test_file), "-v", "--tb=short", "--no-header"],
    ...
)

# RTK-wired (one token change)
result = subprocess.run(
    ["rtk", "python", "-m", "pytest", str(test_file), "-v", "--tb=short", "--no-header"],
    ...
)

multi_agent_coding/system.py, _run_tests():

# Current
result = subprocess.run(
    ["python", "-m", "pytest", tests_path, "-v", "--tb=short", "--no-header"],
    ...
)

# RTK-wired
result = subprocess.run(
    ["rtk", "python", "-m", "pytest", tests_path, "-v", "--tb=short", "--no-header"],
    ...
)

Nothing else changes. The DEBUGGER still reads from state["test_output"]. It just reads compressed output instead of raw tracebacks. The existing _parse_failures() handwritten filter in system.py becomes redundant — RTK's pytest filter does the same job deterministically, without bespoke regex.

Integration Point 2 — read_file tool output (harness_ladder)

When the agent calls read_file("solution.py") to inspect code it previously wrote, the full file content enters the context window. For any non-trivial implementation (50+ lines), this adds 200–800 tokens per read call, and agents call it repeatedly across iterations.

RTK's rtk cat solution.py applies its code-aware compressor, preserving function signatures and reducing body noise. Not as dramatic as the pytest savings, but meaningful across a multi-rung sweep.

Integration Point 3 — state["test_output"] (multi_agent_coding)

In system.py, the EVALUATOR writes raw pytest output into state["test_output"], which the DEBUGGER reads directly. By compressing at the subprocess level (Point 1 above), this state field automatically carries compressed content — no state schema changes required.

Token Math Per Agent Node

The following breakdown uses numbers measured against this repo. "Real suite" refers to a medium-complexity task (email validator with 8 test cases, ~60 lines of implementation).

Node Context without RTK Context with RTK Delta
PLANNER ~500 tokens (task only) same —
CODER ~900 tokens (task + plan) same —
EVALUATOR no LLM — deterministic no LLM — deterministic —
DEBUGGER (1 call, mock suite) 687 tokens 88 tokens −87.2%
DEBUGGER (1 call, real suite) 8,000–13,000 tokens ~231 tokens −97–98%
DEBUGGER (×3 iterations) context accumulates each round context stays flat no bloat

The compounding effect across iterations is the non-obvious win. Each DEBUGGER retry reloads the full context: task + code + all prior tool outputs. Without RTK, the message history grows by 8,000+ tokens per iteration. With RTK, each iteration adds ~231 tokens. By iteration 3, the difference is a 24,000-token gap in the running context.

The Model Routing Implication

The DEBUGGER's large context requirement was the forcing function that made cloud models feel necessary. The reasoning: qwen2.5:7b handles 8,000-token contexts poorly — it loses track of the assertion lines buried in verbose pytest output, produces hallucinated fixes, and fails more often than it should. The instinct is to reach for claude-sonnet-4-6 or a 14B+ local model for the DEBUGGER specifically.

RTK removes that forcing function:

Node         Without RTK                 With RTK
──────────────────────────────────────────────────────────────────
PLANNER      qwen2.5:7b ✓ (short ctx)    qwen2.5:7b ✓ (unchanged)
CODER        qwen2.5:7b ✓ (medium ctx)   qwen2.5:7b ✓ (unchanged)
EVALUATOR    no LLM                       no LLM
DEBUGGER     needs 14B+ or cloud API ✗   qwen2.5:7b ✓ (88 tokens)

With RTK, the entire multi_agent_coding pipeline runs on qwen2.5:7b locally at no cost. Without RTK, the DEBUGGER becomes the bottleneck that either bloats memory (large local model) or introduces API cost (cloud model).

The efficiency ratio at Claude Sonnet 4.6 input pricing ($3/MTok):

Scenario DEBUGGER tokens × 3 iterations API cost Local model cost
No RTK, cloud model ~36,000 tokens ~$0.11 —
RTK, local qwen2.5:7b ~693 tokens $0.00 $0.00

That $0.11 is per task, per pipeline run. At 100 tasks/day it's $11/day. With RTK and local inference it's $0.00/day. The savings compound with volume.

Rung 6 — RTK as a Measurable Harness Layer

The harness ladder (rungs 0–5) measures capability additions:

Rung What's added What it measures
0 bare ReAct loop baseline
1 planning tools structural decomposition
2 scratch memory context offload
3 forced verify node harness-enforced feedback loop
4 sandbox constraints Ashby's Law in action
5 sub-agent delegation isolated context per sub-problem
6 RTK tool-output compression cost/efficiency per rung

Rungs 0–5 answer: does adding this layer improve pass rate? Rung 6 asks a different question: what does each rung cost, and does RTK change the cost/quality tradeoff enough to make a cheaper model viable?

The 3-axis benchmark this enables

Run the eval sweep across three configurations:

A: (rung=N, model=qwen2.5:7b,       rtk=OFF)   ← local, no compression
B: (rung=N, model=qwen2.5:7b,       rtk=ON)    ← local, RTK-compressed
C: (rung=N, model=claude-sonnet-4-6, rtk=OFF)  ← cloud baseline

The hypothesis: B matches or beats C on pass rate at rung 3+, at zero API cost.

The mechanism: at rung 3, the harness forces the verify-and-retry loop regardless of model quality. The model can't skip it. With RTK keeping the DEBUGGER's context small and clean, qwen2.5:7b doesn't fail because of context noise — it fails because of reasoning gaps, which are a different problem. The rung 3 harness compensates for reasoning gaps by forcing retries. RTK compensates for context size limitations by compressing inputs. The combination is what makes the small model viable.

This is "harness engineering > model engineering" with a cost dimension attached.

Implementation sketch for Rung 6

Add an rtk: bool flag to build_agent() and get_tools():

# harness_ladder/tools/factory.py

def _test_tool(task_dir: Path, rtk: bool = False) -> BaseTool:
    cmd_prefix = ["rtk"] if rtk else []

    @make_tool
    def run_tests() -> str:
        """Run the pytest test suite. Returns compressed output if RTK is enabled."""
        result = subprocess.run(
            cmd_prefix + ["python", "-m", "pytest", str(test_file), "-v", "--tb=short", "--no-header"],
            capture_output=True, text=True, cwd=str(task_dir), timeout=30,
        )
        return (result.stdout + result.stderr).strip()

    return run_tests


def get_tools(rung: int, task_dir: Path, model=None, rtk: bool = False) -> list:
    tools = _file_tools(task_dir, sandboxed=(rung >= 4))
    tools.append(_test_tool(task_dir, rtk=rtk))
    ...

And propagate it through build_agent() and run_experiment.py:

# run_experiment.py
for model_name in MODELS:
    for rung in RUNGS:
        for rtk in [False, True]:         # ← sweep both configurations
            agent = build_agent(model, rung, task_dir, rtk=rtk)
            result = agent.invoke(...)
            record(model_name, rung, rtk, task_id, result["passed"])

The eval output then becomes a table with three axes instead of two, and the rtk=True column is the cost story.

Why This Matters Beyond Cost

RTK compression does two things simultaneously:

1. Reduces token count — directly reduces API cost and local inference time.

2. Improves signal-to-noise ratio — the DEBUGGER reads 88 tokens of clean assertion failures rather than 8,000 tokens of mixed signal and boilerplate. A smaller, denser input is often easier for a small model to act on correctly than a large noisy one.

This is the non-obvious efficiency gain: RTK doesn't just make the pipeline cheaper, it makes the DEBUGGER's job structurally easier. It is doing a version of what _parse_failures() in system.py does manually — but transparently, consistently, and across every command type that RTK covers (100+ commands: git, pytest, cargo, docker, kubectl, aws, etc.).

The pattern generalizes to any multi-agent system where tool outputs dominate context:

  • A research agent whose subagents read large docs → Headroom on RAG chunks
  • A DevOps agent running kubectl/docker commands → RTK on every subprocess call
  • A code review agent running linters → RTK on eslint, tsc, clippy, ruff output

The harness enforces correctness. RTK makes it affordable.

Connection to the Deep Research Agent

The deep_research_agent uses DuckDuckGo search as its primary tool. Search results are prose — RTK doesn't touch prose. That's where Headroom's Kompress layer applies (see rtk-vs-headroom.md in rust_token_killer/). But if the research agent is extended to run shell commands (e.g., querying internal CLIs, running scripts to gather data), RTK applies immediately.

The broader principle: RTK owns shell output, Headroom owns everything else. In a full multi-agent system, you want both layers running simultaneously — RTK at the subprocess boundary, Headroom at the RAG/message-history boundary.

Implementation Roadmap

Immediate (one-liner): Wire RTK into _run_tests / run_tests in both projects. No architecture changes. Measure DEBUGGER context size before/after on a real task.

Short term: Add rtk: bool to build_agent() and get_tools() in harness_ladder. Run the 3-axis eval sweep. Produce the pass-rate vs. cost table.

Medium term: Add a model_router node between EVALUATOR and DEBUGGER that reads context size from the EVALUATOR output and selects:

  • RTK-compressed output → qwen2.5:7b (local, free)
  • Uncompressed output above threshold → claude-sonnet-4-6 (cloud, reliable)

This is the practical production pattern: RTK runs always, the model routes based on what the compressed context still demands.

Summary

Lever Effect Mechanism
Harness rung 3+ Enforces verify-retry loop regardless of model LangGraph node the model cannot skip
RTK on run_tests Compresses pytest output 87–98% Rust filter at subprocess boundary
Combined Small model (qwen2.5:7b) viable for full pipeline Harness compensates for reasoning gaps; RTK compensates for context noise
Cost outcome $0.00/task vs $0.11/task (DEBUGGER alone) Local model + compressed context

The harness makes small models work. RTK makes them affordable.

Filed under: Harness Engineering · Context Compression · Model Routing · Cost Optimization

Hermes Wiki