LLM Eval Pipeline Integration With GitHub Actions CI/CD

Agent CI/CD has to catch behavior regressions that traditional pipelines cannot see, because a prompt edit can pass every lint check, type check, and unit test, but it can still quietly break the task the agent was built to do. A tool-call sequence can silently return the wrong function, or a rephrased system prompt can nudge the model away from calling the right tool at all, and neither one produces a compiler error or a failed assertion.

Why LLM CI/CD cannot borrow traditional pipeline logic

Traditional CI gates, lint, type-check, unit tests, build, exist to catch a narrow and well-defined class of error: code that doesn't compile, code that violates a type contract, code that fails a deterministic assertion. Agent behavior lives in prompts, tool schemas, retrieval configuration, memory policies, and orchestration rules, and none of that is visible to a diff checker. A pull request can look clean in review, yet still ship a regression that appears only once real users start typing real requests into the system.

This is the mechanism behind silent tool failure in production multi-agent pipelines: an orchestrator calls a tool, the tool returns something structurally valid but semantically wrong, a downstream agent treats that result as ground truth, and the final answer comes back wrong with no error anywhere in the stack. A multi-agent pipeline that returns a plausible-sounding but incorrect answer, with no exception thrown and no retry triggered, is almost never a crash but a tool-selection or tool-argument failure that passed every structural check and was never evaluated against expected behavior. If there are no explicit eval gates between a prompt change and a merge, that class of regression reaches users silently, and the first signal of trouble is a production incident, not a blocked pull request.

The three-tier architecture that makes agent evals feasible in CI

The response to that gap is a tiered pipeline, and three tiers specifically reflects an economic calculation. A three-tier structure resolves that tension by assigning each tier a different cost profile and a different job.

Tier 1 sits on every pull request. Tier 3 runs continuously in production: it samples live traces, applies lightweight online scoring, and enforces an error budget before canary traffic is allowed to expand. But if that's missing, a nightly eval can score green against a synthetic golden dataset and still miss the behavioral drift that appears only once real traffic hits it.

Tier 1 tests and cassette replay

Cassette replay keeps Tier 1 cheap, fast, and reliable. A cassette is a recording: during a known-good run, the agent's tool calls and the responses those tools returned are captured and stored. A test that depends on a live API response will occasionally fail for reasons that have nothing to do with the code under review; a test built on a replayed cassette fails only when the agent's tool-selection logic has actually changed.

In practice, a Tier 1 assertion looks something like this: given a recorded user query, assert that the agent calls search_inventory before create_order, and assert that the arguments passed to create_order validate against the tool's JSON schema. Two execution modes make this pattern work across the pipeline: replay mode, which uses recorded tool responses in a fully deterministic environment, for CI and for root-cause attribution when something fails, and live mode, which hits real tools over a real network, reserved for canary validation once a change reaches production traffic.

What Tier 1 catches is narrow and specific: tool schema drift, meaning malformed arguments or the wrong function being selected, along with any structural failure a rule-based check can catch without involving the model. What it does not catch is anything about quality: whether the final output is correct, whether the agent's reasoning holds up, whether the response is a hallucination dressed in confident language. Those questions require a judge with some capacity for semantic understanding, which the next tier is built to provide. Promptfoo's CI integration shows the pattern clearly: its --fail-on-error flag exits non-zero on any test failure, and the project's own GitHub Actions example favors custom shell logic, parsing results with jq and calling exit 1 explicitly, for quality-gate decisions.

Configuring Tier 2 LLM-as-judge scoring with threshold-based pass/fail gates

An LLM judge score only becomes a deployment gate once it is paired with an explicit threshold, version-controlled alongside the agent code it's evaluating, that turns a number into a decision. Without that pairing, a nightly eval run just produces a dashboard someone might glance at, not a gate that blocks a bad merge. The minimum viable threshold stack for an agent eval pipeline covers five distinct dimensions, and each one catches a different kind of regression. Task completion rate serves as the primary CI gate: if it drops, the agent is failing at the thing it exists to do. Hallucination rate acts as a blocking ceiling, since any increase past a defined limit means the agent is inventing information with more confidence than it has any right to. A cost regression gate blocks any increase in token cost beyond a defined percentage, because a prompt change can silently double token usage and still hold output quality steady, so it is a budget problem.

These thresholds need to live in version control next to the agent code they gate, and the baseline they're measured against needs a clear policy for when it's allowed to move. The sounder policy advances the baseline only when a green run on the main branch shows a non-trivial improvement, completion rate meaningfully higher, cost median meaningfully lower, and never just because the run happened to pass.

Promptfoo's quality gate example demonstrates the mechanics directly: parse the pass rate out of results.json with jq, compare it against a hard threshold set in the workflow, and exit with code 1 on failure, which GitHub Actions then treats as a required status check blocking the merge. The LLM-eval-CI-CD-pipeline reference project takes the same idea further by defining its gates entirely in YAML, with explicit fields for max_hallucination_rate, p95_max_ms, min_answer_relevancy, min_faithfulness, and max_mean_cost_per_query_usd, and exiting non-zero on any violation. A single evaluation run can generate all three formats at once, making format choice essentially free.

Building the golden dataset the gates test against

A threshold is only as meaningful as the test cases it's measured against, and a golden dataset built from production traces holds up better than one built from synthetic examples, because it captures the failure modes that actually occur rather than the ones engineers imagined while designing the agent.

The sourcing mechanism that keeps a dataset current runs as a closed loop. Production trace evaluation scores live traffic as it happens, an error feed clusters the failing traces into recognizable patterns, those failing traces auto-promote into the offline eval set, and that dataset expansion then triggers the next scheduled optimization run. This is the compounding payoff of wiring Tier 3 back into Tier 2: every production failure that gets caught and promoted into the golden dataset makes the nightly eval marginally harder to game and marginally more representative of what the agent actually faces.

The test cases populating Tier 2 evals are most reliable when they're sourced from real production traces rather than synthetic golden datasets built ahead of time, because production traces capture the actual input distributions, the actual edge cases, and the actual failure patterns an agent encounters at scale. Each evaluation run should also record its own metadata: git SHA, branch, PR number, model, prompt hash, knowledge-base hash, and dataset version, since that record is what makes it possible to compare runs across a model swap, a prompt rewrite, or a knowledge-base update without guessing at what changed.

How root-cause attribution changes a CI failure's meaning

Knowing that a gate fired tells an engineer almost nothing on its own. The gate becomes useful only once the failure is attributed to a specific layer, prompt, tool schema, workflow logic, memory, or the model itself, because each of those layers calls for a completely different fix. Treating all three as the same kind of failure wastes the engineer's time chasing the wrong layer.

The step where an error becomes visible is frequently not the step that caused it. AgentDebugX (Zhu et al., arXiv:2607.18754, 2026) formalizes this as a closed debugging loop: Detect, Attribute, Recover, Rerun. Its attribution strategies trade cost against resolution deliberately, starting with inexpensive heuristics and a single pass over the whole trace, and escalating to binary search, per-step inspection, and budgeted ensembles only when the cheaper methods fail to localize the fault.

That escalation logic has a direct payoff for CI. Attribution that lands on the harness layer, the prompt, the tool interface, middleware, state handling, or recovery logic, is actionable immediately, without touching the model. Attribution that implicates the model itself is a different and far costlier intervention. The paper Beyond Prompts: Measuring and Optimizing LLM Tool-Agent Harnesses (arXiv:2609.05736, 2026) establishes that agents can be improved without retraining by modifying the runtime harness components around a fixed model, prompts, tool interfaces, middleware, state handling, recovery logic. This means most agent failures in practice originate in the harness. A CI gate that simply reports "task completion fell" without isolating whether the regression sits in the prompt, the tool schema, or the workflow gives the engineer no starting point. A gate that attributes the failure to a layer gives them a fix to start writing.

Wiring Tier 3 production validation into the deployment pipeline

Canary rollout paired with automatic rollback is the mechanism that enforces behavioral quality after code ships, treating a drop in eval score below an error budget the same way an on-call engineer treats a crash-rate spike. Shadow evaluation runs ahead of that: recorded production traffic gets replayed against the new agent version before it ever handles a live request, and any divergence in trajectory from the production baseline reveals a behavioral regression in testing, a failure mode synthetic golden datasets tend to miss.

A graduated canary follows shadow evaluation. A small percentage of live traffic routes to the new version first, a lightweight online judge scores it continuously, and an error budget threshold then gates whether traffic can expand. Only if the online score holds does traffic advance further. If the online score drops below that error budget threshold, the deployment rolls back automatically, which is what a Tier 2 gate firing before a merge is allowed looks like in production.

Five gates, taken together, form a complete pipeline rather than five disconnected checks: golden dataset offline evaluation, regression blocks, a cost gate, shadow evaluation, and canary rollout with auto-rollback. Each one catches a different failure mode at a different point in the deployment lifecycle, and none of them substitutes for the others. Production trace sampling closes the loop back to Tier 2 directly: failing traces captured during a canary get promoted into the golden dataset, so the next nightly eval run is measurably harder to pass on the exact class of problem that just caused a rollback. Cassette replay in Tier 1 catches schema drift and structural breakage early, but it cannot detect output quality or reasoning errors, the subtle behavioral regressions that can slip silently through a PR gate and corrupt production output downstream. If a team replays cassettes in CI but skips production trace analysis entirely, it tends to miss exactly the class of agent error that an orchestrator swallows quietly before it ever reaches a user-visible response.

Implementing the GitHub Actions workflow that wires all three tiers together

A complete implementation splits cleanly into three workflow files, one for the PR tier, one for the nightly tier, and one for production monitoring, each wired to the trigger that matches its job and each failing in the way appropriate to what it's testing.

The Tier 1 workflow triggers on pull_request, scoped with path filters to directories like prompts/** and files like promptfooconfig.yaml, so it only runs when something relevant actually changed. Attaching pipeline context with tags such as --tag git.sha=$CI_COMMIT_SHA and --tag ci.run-id=$CI_PIPELINE_ID means every eval run can be traced back to the exact commit and pull request that triggered it, which matters enormously once a regression needs to be attributed to a specific change weeks later.

The Tier 2 workflow triggers on a cron schedule, cron: '0 6 * * *', running daily at 06:00 UTC, alongside a workflow_dispatch trigger that lets an engineer kick it off manually when needed. Any API key an LLM provider requires has to live as an encrypted GitHub Actions secret, never hardcoded into a workflow file and never committed to the repository under any circumstance. Path filtering applies on this trigger too: full LLM-judge scoring runs only when files under agent/prompts/, agent/tools/, or agent/config/ have changed, and the suite is skipped or reduced for changes that are purely infrastructural.

Tier 3 configuration usually lives outside GitHub Actions proper; instead it runs inside the deployment platform or a monitoring service that handles production trace sampling and canary scoring. Configuring Tier 1 and Tier 2 gates as required status checks under branch protection is what makes the whole pipeline mandatory rather than advisory: merges are blocked outright when either gate fires. Metadata attached to every run, git SHA, branch, PR number, model, prompt hash, dataset version, is what makes a regression attributable to a specific change rather than merely detectable as a vague drop in a score.

Keeping the pipeline from degrading over time

A well-structured eval pipeline rarely fails because a gate was misconfigured. Threshold creep happens when a baseline is allowed to auto-advance on every green run, which lowers the bar gradually and invisibly; the fix is a baseline advancement policy with explicit, non-trivial improvement criteria, not simply "it passed this time." Flaky LLM judges create a third kind of decay: if a judge scores inconsistently, it turns the Tier 2 gate into something closer to a coin flip than a measurement, but prompt-stable rubric definitions, deterministic temperature settings, and running judges only on paths where agent behavior actually changed can all reduce that inconsistency.

The closed-loop production sourcing mechanism described earlier is the structural answer to staleness specifically: failing production traces that auto-promote into the golden dataset keep coverage current continuously, without requiring a dedicated audit sprint every quarter to catch up. Treating it with the same rigor applied to code review and deployment testing is what separates a team shipping agents that actually get better from a team shipping agents merely faster. Moda's trace-driven improvement loop, ingesting production traces, attributing failures to the specific harness layer responsible, generating reviewable fixes, and validating those fixes against historical traces before recommending what to ship, is one concrete instance of that discipline automated: a pipeline that improves itself from production evidence rather than from engineering intuition about what might be wrong.