# How Agents CLI Helps You Catch and Fix AI Agent Regressions

Canonical URL: https://zero2vibecode.com/blog/agents-cli-eval-regressions
Date: 2026-10-29
Tags: agents, tools, evaluation
Source: https://developers.googleblog.com/hill-climbing-agent-quality-with-agents-cli-evaluation/

Learn how Agents CLI's evaluation tools help you systematically improve your AI agents by catching regressions and measuring changes.

When you're building an AI agent, the hardest part isn't getting it to work once—it's keeping it working as you make changes. Agents CLI's evaluation tools give you a systematic way to catch problems before they reach users, measure the impact of changes, and steadily improve your agent's quality.  

## Why evaluation matters  

Building an AI agent that works in a demo is different from building one that stays reliable through model updates, tool changes, and unexpected user inputs. Without systematic testing:  

- Small changes can break behaviors you didn't think to test  
- Model updates might degrade performance without warning  
- You can't measure whether a change actually improved things  

The Agents CLI evaluation tools implement a quality engineering technique called "hill-climbing"—making small changes, measuring their impact, and keeping only what improves your score.  

## Setting up evaluation  

The evaluation workflow starts with a dataset of test cases that represent different ways users might interact with your agent. A basic weather agent might need cases for:  

1. Simple greetings ("hello")  
2. Weather queries for supported cities ("weather in San Francisco")  
3. Queries for unsupported locations ("weather in Rome")  
4. Off-topic questions ("who won the World Series?")  

Start with 4-10 test cases that cover your agent's core behaviors. Too many makes the feedback loop slow; too few won't catch regressions.  

The test cases live in a JSON file with example inputs and expected response formats. Agents CLI provides templates to get you started.  

## Running your first evaluation  

With test cases prepared, you:  

1. Generate responses: `agents-cli eval generate --dataset tests/dataset.json`  
2. Grade those responses: `agents-cli eval grade --traces output/*.json`  

This produces scores like:  

| Metric                | Score |  
|-----------------------|-------|  
| Response accuracy     | 4.2   |  
| Relevance             | 3.8   |  
| Clarity               | 4.5   |  

A healthy baseline shows variation—if every case scores perfectly, your tests might be too easy.  

## Catching regressions  

The real power comes when you make changes. Imagine accidentally changing your weather agent's prompt to act like a chef:  

```python  
## Before  
system_prompt="You are a helpful assistant"  

## After  
system_prompt="You are a professional chef. Suggest recipes at the end of every response."  
```  

Re-running the evaluation catches this:  

| Case          | Before | After | Change |  
|---------------|--------|-------|--------|  
| Greeting      | 5.0    | 4.0   | ↓      |  
| SF Weather    | 4.0    | 2.0   | ↓↓     |  
| Off-topic     | 5.0    | 3.0   | ↓      |  

The scores dropped because:  
- Greetings now include unsolicited recipe advice  
- Weather queries ignore the tool and make up forecasts  
- Off-topic answers force in cooking tips  

Without evaluation, this regression might reach users. Code review won't catch it—the change is just one string!  

## Comparing model performance  

Evaluation also helps when considering model changes. Swapping from `gemini-3.8-flash` to `gemini-2.5-flash` produces:  

| Case          | 3.8-flash | 2.5-flash | Change |  
|---------------|-----------|-----------|--------|  
| Greeting      | 5.0       | 5.0       | ·      |  
| SF Weather    | 4.0       | 3.0       | ↓      |  
| Off-topic     | 5.0       | 1.0       | ↓↓↓    |  

The cheaper model handles simple cases well but struggles with edge cases. This helps you decide if the tradeoff is worth it.  

## Automating the process  

For production agents, set up:  

1. **Scheduled runs** - Daily evaluations against a larger test set catch drift from model updates  
2. **Alerting** - A script compares new scores to baselines and alerts on significant drops  
3. **Optimization** - Advanced tools can automatically test prompt variations to find improvements  

## Getting started  

To implement this for your agent:  

1. Define 5-10 test cases covering key behaviors  
2. Run initial evaluation to establish a baseline  
3. Re-evaluate after every meaningful change  
4. Set up alerts for production agents

## Read next

- [A2UI gets faster with swappable formats and decision models](https://zero2vibecode.com/blog/a2ui-swappable-inference-formats)
- [Agents CLI Adds Framework Flexibility with Extension System](https://zero2vibecode.com/blog/agents-cli-extension-system-framework-flexibility)

Want to try all of this hands-on? Start with the free [Claude Code from Zero](https://zero2vibecode.com/learn/claude-code) course.

**Source.**   
Based on Google Developers' announcement, "Hill-Climbing Agent Quality with Agents CLI Evaluation". Written for people learning to build with these tools.