Source: LangChain Blog — 2026-08-19
Summary
LangChain launched Tuned Evaluators in LangSmith, starting with a metric called "Perceived Error" — an automated evaluator attached to production agent traces that's specifically tuned to predict when a human reviewer would flag a given agent action as a mistake, rather than scoring against a generic rubric. The feature is now generally available for LangSmith Plus and Cloud Enterprise tiers, aimed at letting teams find and fix agent mistakes at production scale without manually reviewing every trace.
Key Takeaways
- Core idea: "Perceived Error" is tuned to approximate human judgment of what counts as a mistake, rather than a fixed, generic correctness rubric — the evaluator is calibrated against how humans actually perceive agent errors.
- Runs against production traces automatically, meaning teams get error signal at the scale of real traffic rather than only on a hand-curated eval set run occasionally.
- Now GA for LangSmith Plus and Cloud Enterprise — a production-tier feature, not an experimental preview.
- Framed as the first in a series ("starting with Perceived Error"), implying LangChain plans additional tuned evaluators targeting other specific failure modes beyond this one.
- Fits a broader industry pattern (see also OpenAI's harness benchmarking, Elastic's context-engineering push) of treating agent quality as something to be continuously measured in production, not just validated once before ship.
Reel Script
Hook: Most agent evaluation tools grade against a fixed rubric — LangSmith's new evaluator is tuned to predict something harder: whether a human would actually be annoyed by what the agent just did.
Core Concept: Evaluating an AI agent is different from evaluating a single model output, because an agent takes multi-step actions, and "was this correct" often isn't a clean pass/fail — it's closer to "would a human reviewing this trace consider it a mistake." A generic rubric-based evaluator scores against fixed criteria you write in advance, which misses errors you didn't anticipate. LangSmith's Perceived Error evaluator takes a different approach: it's tuned — meaning calibrated using data about how humans actually judge agent traces — to approximate that human perception directly, rather than checking boxes against a rubric a developer wrote before seeing real failures.
Hands-On: In practice, this runs automatically against production traces, not just a curated test set you run before shipping. That distinction matters: a pre-ship eval set only catches the failure modes you thought to test for, while an evaluator running continuously against real traffic surfaces the failure modes actual users are triggering — the ones you didn't anticipate when writing your rubric. It's positioned as the first in a planned series of tuned evaluators, suggesting LangChain intends to build out a library of these human-calibrated signals for different failure categories over time, rather than shipping one evaluator and stopping.
Takeaway: If your team's agent evaluation still consists of a fixed rubric and a periodic manual review pass, that gap between "passes our rubric" and "a human would flag this as wrong" is exactly what's now costing you undetected production failures — a human-calibrated evaluator running continuously against real traces closes that gap in a way spot-checks can't. Worth trying against your noisiest agent surface first.