OmniPilot Predicts LLM Serving Cost Across Heterogeneous GPU Clusters — and Knows When to Abstain

Source: arXiv — 2026-07-01

Summary

D. Balamurugan and Thomas W. Bush present OmniPilot, an inference advisor that predicts serving cost and feasibility across combinations of GPU type, tensor-parallel degree, and precision choice for heterogeneous GPU clusters — and explicitly abstains from a prediction when a configuration falls outside its measured support envelope, rather than silently guessing.

Key Takeaways

• Predicts serving cost/feasibility across GPU type, tensor-parallel degree, and precision — a genuinely multi-dimensional planning problem for heterogeneous fleets.
• Explicitly abstains outside its measured support envelope instead of extrapolating silently — a meaningful reliability property for a capacity-planning tool.
• A practical operator-facing tool rather than a purely academic benchmark, aimed at teams running mixed-hardware inference fleets.
• Fits the broader trend of uncertainty-aware tooling for LLM infrastructure decisions.

Discussion

(No questions yet — ask follow-ups via a Claude Code chat session on this repo; answers get appended here.)