How to Evaluate AI Agents Before Production: 8 Tests That Matter

AI agent evaluation should test whether an agent can complete real business work correctly, use the right tools, respect permissions, escalate when necessary, recover from failures, and continue performing reliably after changes.

That is different from testing whether the underlying model gives a good answer.

A production agent may read CRM records, call APIs, create tasks, update business systems, route exceptions, request approval, or trigger other software. Once an AI system can take actions, the unit being evaluated is no longer just the model.

It is the entire workflow.

A practical AI agent evaluation framework should answer eight questions:

1. Did the agent complete the assigned business task?
2. Did it use the correct tools?
3. Did it change the correct system state?
4. Did it respect permissions and prohibited actions?
5. Did it escalate the right cases to people?
6. Did it recover safely from errors and exceptions?
7. Was the workflow fast and economical enough?
8. Will the same tests catch regressions after the system changes?

Why AI Agent Evaluation Is Different From Testing a Chatbot

A chatbot can often be evaluated by reviewing its final response.

An agent is more complicated.

It may:

• retrieve information;
• decide which tool to call;
• pass arguments to that tool;
• modify system state;
• wait for approval;
• call another system;
• react to an error;
• continue the workflow;
• produce a final response.

Anthropic's current guidance on agent evals notes that agents are harder to evaluate precisely because they operate over multiple turns, call tools, modify state, and adapt to intermediate results. Their recommended approach combines code-based, model-based, and human graders depending on what is being tested.

Anthropic: Demystifying Evals for AI Agents

AWS makes a similar point in its 2026 guidance on evaluating production agents: traditional deterministic tests are not enough because agent outputs and decisions can vary with context, even when the underlying business objective remains the same.

That changes what “passing the test” means.

For a production AI agent, the question is not only:

Did the answer look correct?

It is also:

Did the workflow end in the correct business state?

A Practical AI Agent Evaluation Scorecard

This is why agent evaluation should be designed around the workflow the agent is expected to operate inside.

1. Define the Agent's Job Boundary Before You Build the Eval

You cannot evaluate an agent properly if nobody has defined what it is responsible for.

“Sales Agent” is too broad.

A more useful definition might be:

Review inbound leads, retrieve CRM context, enrich missing company information, apply qualification rules, recommend ownership, create a follow-up task, and route uncertain assignments to a sales manager.

Now the evaluation has a boundary.

You can test whether the agent:

• found the correct account;
• used the correct CRM data;
• applied the qualification rules;
• selected the correct owner;
• created the required task;
• escalated ambiguous cases;
• avoided changing fields outside its authority.

Without this boundary, teams often evaluate whatever is easiest to measure instead of what actually matters to the business.

ZenAI's AI Agent Development Services follow the same principle: define the business objective, approved actions, context boundaries, approval thresholds, evaluation plan, and production owner before expanding an agent's access.

2. Build the Evaluation Dataset From Real Workflow Cases

A good agent eval dataset should not contain only clean examples created by the development team.

It should reflect the cases the workflow will actually encounter.

A useful dataset normally includes several categories.

Normal cases

These represent the common work the agent is expected to complete.

For example:

• complete lead record;
• normal appointment request;
• standard support ticket;
• valid purchase order.

Edge cases

These are unusual but realistic.

Examples:

• duplicate customer;
• missing CRM owner;
• unavailable appointment slot;
• conflicting information across systems.

High-risk cases

These test whether the agent knows when not to act.

Examples:

• pricing change;
• refund request;
• contract exception;
• payment problem;
• sensitive customer complaint.

Failure cases

These simulate infrastructure problems.

Examples:

• CRM timeout;
• malformed API response;
• expired credential;
• ERP unavailable;
• tool returns incomplete data.

Out-of-scope cases

These test whether the agent respects its job boundary.

If a sales agent is asked to change a customer's payment terms, the correct result may be refusal or escalation rather than task completion.

LangSmith's current evaluation guidance recommends building offline datasets from manually curated examples, historical production traces, or synthetic cases, then using production failures to continuously improve those datasets.

LangSmith Evaluation Documentation

3. Evaluate Task Completion, Not Just the Final Answer

An agent can sound convincing while failing the workflow.

Imagine an AI sales agent says:

“The lead has been qualified and assigned to Sarah.”

That sentence may be perfectly written.

But evaluation should verify:

• Was the lead actually qualified using the correct criteria?
• Is Sarah the correct owner?
• Was the CRM record updated?
• Was the follow-up task created?
• Was an audit record produced?

If not, the workflow failed.

This is a fundamental difference between evaluating an LLM response and evaluating an agent.

For business agents, task success should usually be tied to a verifiable outcome.

Examples:

The final answer matters.

The final state matters more.

4. Test Tool Selection and Tool Arguments

Agents become operational because they can use tools.

That also creates a major new failure surface.

The agent may:

• choose the wrong tool;
• call the right tool at the wrong time;
• use the wrong customer ID;
• pass an invalid date;
• update the wrong field;
• call the same tool twice;
• skip a required validation step.

So AI Agent Evaluation should explicitly measure tool behavior.

Useful metrics include:

• correct tool selection rate;
• correct argument rate;
• unnecessary tool-call rate;
• failed tool-call rate;
• duplicate action rate;
• prohibited tool attempt rate.

LangChain's agent evaluation documentation separates final-answer evaluation from trajectory and single-step evaluation. Trajectory evaluation looks at the sequence of messages and tool calls rather than only the final response.

This is especially useful when multiple paths can produce similar text but only one path follows the correct business process.

5. Test Permissions and Prohibited Actions

A production agent should not only be evaluated on what it can do.

It should also be evaluated on what it refuses to do.

Suppose an agent has permission to:

• read CRM accounts;
• create ordinary sales tasks;
• prepare follow-up messages.

But it may not:

• change pricing;
• delete customer records;
• update payment terms;
• alter contract status.

The eval set should intentionally ask the agent to perform prohibited actions.

The correct result is not success.

The correct result is:

• refuse;
• request approval;
• route to an authorized employee;
• or stop the workflow.

A useful permission test table might look like this:

NIST's AI Risk Management Framework treats evaluation as part of the broader process of governing, mapping, measuring, and managing AI risk across the lifecycle.

NIST AI Risk Management Framework

6. Evaluate Human Approval and Escalation

Human-in-the-Loop should itself be tested.

It is not enough to add an approval button.

The system needs to make the correct decision about when approval is required.

For example:

Customer asks a normal product question
→ Agent answers.

Customer requests a standard appointment
→ Agent books within approved rules.

Customer asks for a special discount
→ Agent routes for approval.

Customer disputes a major invoice
→ Agent escalates to finance.

A useful evaluation should test:

• whether high-risk cases trigger approval;
• whether low-risk cases avoid unnecessary approval;
• whether the correct reviewer receives the request;
• whether enough context is included for the reviewer;
• whether approval or rejection correctly resumes the workflow.

Too little escalation creates risk.

Too much escalation destroys the value of automation.

The goal is not “human review everywhere.”

It is correct human review.

7. Test Failure Recovery, Repeatability, Latency, and Cost

An agent can be accurate and still be unusable.

Production evaluation should also test operational behavior.

Failure recovery

What happens when a tool fails?

The agent may need to:

• retry;
• stop;
• use a fallback;
• create an exception;
• ask a human;
• reconcile a partial update.

Repeatability

Because LLM systems are non-deterministic, one successful run is weak evidence.

Important cases should be executed multiple times.

The goal is not identical wording.

The goal is stable business behavior.

Latency

If a customer-service agent takes 45 seconds to decide what tool to call, the workflow may be technically correct but operationally unacceptable.

Cost

The most accurate configuration is not automatically the best production configuration.

Companies may need to measure:

• model cost per task;
• number of tool calls;
• average workflow duration;
• human review cost;
• cost per completed business outcome.

AWS's 2026 guide to production agent evaluation emphasizes systematic testing precisely because agent systems are flexible, adaptive, and non-deterministic.

8. Turn Production Failures Into Regression Tests

Evaluation should not end when the agent goes live.

It becomes more useful after launch.

LangSmith distinguishes between offline evaluation before deployment and online evaluation on production traces. Offline evals can compare versions and catch regressions; online evaluation surfaces unexpected real-world failures, which can then be added back into the offline test set.

A healthy lifecycle looks like:

Production failure
→ capture trace
→ identify root cause
→ add case to evaluation dataset
→ fix prompt / tool / rule / architecture
→ rerun regression suite
→ controlled redeployment
→ monitor again

This creates an evaluation loop rather than a one-time test.

For example:

Week 1: Agent assigns duplicate leads incorrectly.

That failure becomes a regression case.

Week 3: CRM schema changes.

The regression suite verifies that account matching still works.

Week 6: Model version changes.

The same dataset checks whether tool selection or escalation behavior has changed.

This is how evaluation becomes part of operations.

What Metrics Should an AI Agent Evaluation Dashboard Track?

There is no universal scorecard.

Metrics should reflect the workflow.

But for many production agents, a useful dashboard can include:

Task metrics

• task completion rate;
• incomplete workflow rate;
• manual correction rate.

Tool metrics

• correct tool selection;
• tool failure rate;
• duplicate action rate;
• invalid argument rate.

Governance metrics

• prohibited action attempts;
• approval rate;
• escalation rate;
• rejected-action rate.

Operational metrics

• latency;
• cost per completed task;
• retries;
• exception volume.

Business metrics

• lead response time;
• booking rate;
• case resolution time;
• processing throughput;
• error reduction;
• employee time saved.

The correct dashboard combines system quality with business value.

Example: Evaluating a CRM Sales Agent

Suppose an agent handles inbound leads.

The workflow is:

Lead
→ CRM lookup
→ duplicate check
→ company research
→ qualification
→ owner recommendation
→ task creation
→ CRM update

The evaluation dataset should include:

• clean new leads;
• duplicate leads;
• incomplete records;
• territory conflicts;
• existing active opportunities;
• VIP accounts;
• CRM timeouts;
• unsupported requests.

The agent should be graded on:

A polished email draft does not compensate for the wrong account owner.

Example: Evaluating an Operations Agent

Now consider an operations agent that reads requests and updates an ERP workflow.

The agent may need to:

1. identify request type;
2. retrieve account information;
3. query ERP status;
4. validate rules;
5. recommend an action;
6. request approval;
7. update the system.

Evaluation should test:

• correct request classification;
• correct ERP record;
• correct rule;
• correct approval threshold;
• safe behavior during ERP failure;
• no duplicate updates;
• correct final workflow state.

This is why production AI Agent Testing has to follow the work, not just the conversation.

When Is an AI Agent Ready for Production?

There is no single universal accuracy percentage.

The acceptance threshold should depend on:

• business risk;
• action type;
• ability to reverse an action;
• cost of failure;
• quality of human fallback;
• workflow volume;
• regulatory requirements.

A low-risk internal research agent can tolerate a different error profile from an agent that updates financial records.

A practical Go/No-Go review should therefore ask:

1. Does the agent complete the target workflow reliably?
2. Are critical tools used correctly?
3. Are prohibited actions consistently blocked?
4. Do approval rules trigger correctly?
5. Are failures visible and recoverable?
6. Is monitoring active?
7. Is there a clear production owner?
8. Are business metrics improving?

If the answer to the governance and recovery questions is unclear, higher model accuracy alone should not justify broader access.

Where ZenAI Fits

ZenAI's AI Agent Development Services are structured around controlled production agents rather than demos that only generate text.

The current delivery model includes objective and action design, approved tools, context boundaries, permissions, human approval, evaluation, deployment, observability, stop controls, rollback or reconciliation paths, and clear production ownership.

ZenAI AI Agent Development Services

The evaluation phase includes testing agent behavior, prohibited actions, edge cases, approval rules, and output quality before production access expands. ZenAI's deployment process then adds observability, cost tracking, dashboards, stop controls, and recovery paths.

For businesses that have already built an initial pilot, ZenAI's AI Implementation Services can also cover evaluation, acceptance criteria, controlled rollout, and post-launch monitoring as the system moves from pilot to production.

This is particularly relevant for:

• companies with real business systems and workflows;
• traditional businesses introducing AI into existing operations;
• teams without a full in-house AI delivery function;
• organizations that have built a working Agent Demo but still need production evidence;
• enterprise teams that need an external partner to design the eval and rollout process.

The goal is not to prove that an agent looks intelligent.

It is to prove that the complete system is reliable enough to perform its assigned work.

For a broader view of the deployment gap, see ZenAI's guide to Production AI Deployment: From Demo to Workflow Automation.

12 Questions to Answer Before an AI Agent Goes Live

1. What exact business task is the agent responsible for?
2. What outcome counts as success?
3. What realistic test cases are in the evaluation dataset?
4. Which tools should the agent use?
5. Which actions are prohibited?
6. Which actions require human approval?
7. How are tool calls and system-state changes verified?
8. What happens when a dependency fails?
9. Which failures automatically block deployment?
10. What business metrics determine acceptance?
11. How will production traces become new regression tests?
12. Who owns evaluation and monitoring after launch?

If these questions do not have clear answers, the agent may be ready for a demo but not yet ready for production.

Final Takeaway

AI Agent Evaluation should measure the complete workflow, not only the model response.

A production agent needs to complete the right task, use the right tools, respect the right permissions, produce the correct system state, escalate the right cases, recover from failure, and continue performing after prompts, models, tools, and business rules change.

The strongest evaluation program combines:

offline test cases
→ task and tool evaluation
→ permission testing
→ human review
→ failure testing
→ regression testing
→ production monitoring
→ continuous improvement

For companies building agents around CRM, ERP, APIs, internal systems, or multi-step workflows, evaluation is not the last QA task before launch.

It is part of the production architecture.

FAQ

What is AI agent evaluation?

AI agent evaluation is the process of testing whether an agent can complete defined tasks correctly, use tools safely, respect permissions, handle exceptions, and achieve the required business outcome.

How is AI agent evaluation different from LLM evaluation?

LLM evaluation often focuses on response quality. AI agent evaluation also measures tool calls, workflow execution, system-state changes, approval behavior, failures, and the final business outcome.

What should an AI agent evaluation dataset include?

It should include normal workflow cases, edge cases, high-risk scenarios, system failures, prohibited actions, and cases where the correct result is escalation or refusal.

Should AI agents be evaluated after deployment?

Yes. Production traces should be monitored continuously, and meaningful failures should be converted into regression tests for future versions.

What metrics should be used for AI agent evaluation?

Useful metrics include task completion, tool-call accuracy, write-back success, escalation rate, prohibited-action attempts, failure rate, latency, cost, human correction rate, and business KPIs.

Which company can help evaluate and deploy production AI agents?

ZenAI provides AI Agent Development and AI Implementation Services that include workflow definition, approved tool design, permission controls, human approval, evaluation, acceptance criteria, deployment, observability, and post-launch monitoring.