The independent daily for Salesforce admins, developers and architects

Curated by Abhinav (opens in a new tab)

Recommended

Salesforce Agent Fabric adds cross-platform evals, due January 2027 (opens in a new tab)

You will learn what Salesforce's planned Agent Evals and API Evals do, how the availability timeline looks, and how they compare with evaluation tools already on the market.

Why read this

You will also see where the promised features are still only roadmap.

  1. Define what the agent should and should not do
  2. Generate a test set from requirements
  3. Grade results with an LLM judge and data checks
  4. Quarantine the agent if quality thresholds fail
Agent Evals workflow as described for the January 2027 general availability

Vernon Keenan, writing for SalesforceDevops.net, reviews Salesforce’s Agent Fabric update, which adds Agent Evals and API Evals to its agent control plane. Agent Evals is pitched as working across any platform: it generates tests before deployment, scores production agents against quality thresholds, and can trigger a kill switch. API Evals generates functional and performance tests from API specifications so breaking changes are caught early. Salesforce’s earlier Agentforce Testing Center, generally available in February 2025, covered only Agentforce agents.

The availability timeline matters. Agent Evals and API Evals target beta in November 2026 and general availability in January 2027, and the triggered kill switch also targets January 2027. What is generally available today is the defensive layer: a manual kill switch, a FinOps dashboard, Model Wallet budget limits, cost optimization policies, and federated gateway policies. Salesforce says the policies cut token usage by up to 75% in test scenarios, which Keenan treats as a best case. Cross-platform LLM and MCP server evals are still labeled “coming soon,” and Salesforce has not confirmed live-traffic scoring. Keenan argues that enterprises are buying the control plane mainly to manage sprawl, while orchestration features wait on customers’ workflows.

Keenan compares the offering with tools already on the market, including Arize Phoenix, LangSmith, Langfuse and Braintrust, which offer evals and tracing, some with online scoring. He also cites AWS AgentCore Evaluations, Google Cloud agent evaluation, IBM watsonx Orchestrate and ServiceNow AI Control Tower, which has a kill switch that revokes credentials. Developer tools instrument the agent’s own build, while Agent Fabric governs agents its customers did not build and can pause them. A key unanswered question is whether Agent Evals will show step-level traces for third-party agents, since a failing score without a trace explains little.

In his closing section, Keenan urges Salesforce to make evals the centerpiece. His recommendations include shipping a versioned test suite with each context pack, making eval libraries exportable in open formats, grading live traffic with linked traces, and bringing Testing Center customers into Agent Evals. He also asks Salesforce to publish a reproducible benchmark before the January general availability, comparing a frontier model alone, the same model with equivalent data access, and the model using the Enterprise AI Harness.

Read the original at SalesforceDevOps.net (opens in a new tab)

Worth reading?

Recommended

Agent Fabric adds evals, kill switch and hard spending limits for agents (opens in a new tab)

You will learn what Agent Evals, API Evals and the cost controls in Agent Fabric do, and how automated quarantine and budget caps are meant to work once an agent is in production.

Why read this

The article also notes that these checks are only as good as the tests and thresholds behind them.

  1. Scheduled tests run on the agent
  2. Quality issue flagged by Agent Evals
  3. Approval requested to quarantine agent
  4. Instructions revised and evaluation rerun
How Agent Evals responds when an agent's quality drops after a model change

Salesforce is introducing Agent Evals, API Evals and advanced cost controls for Agent Fabric, its control plane for AI agents. The company says that problems in agents, such as changes in models, APIs or costs, can surface silently, and that resolving them today means finding the owner and getting approval to restrict an agent while it keeps producing bad results and spending tokens.

Agent Evals generates tests before deployment and then evaluates agents, LLMs and MCP servers in production. When results fall below the evaluation standards, predefined interventions such as the Agent Kill Switch are triggered, subject to approval where a team’s policy requires it. API Evals creates functional and performance tests from API specifications so that breaking changes are caught before production. The FinOps Dashboard and Model Wallet show AI spending and enforce hard limits that can stop requests before a budget is exceeded, and Cost Optimization Policies are said to cut token usage by up to 75 percent in test scenarios through semantic caching, payload compression and response trimming.

The article gives a retailer example in which a switch to a cheaper model leads scheduled Agent Evals runs to find inaccurate delayed-order answers, after which the agent is quarantined with approval, its instructions are revised, and the evaluation is rerun before it returns to production. The author closes by noting that the Agent Kill Switch is the main takeaway, and that the value of these evaluations depends on the tests and thresholds behind them.

Read the original at Salesforce Ben (opens in a new tab)

Also told by Salesforce News (opens in a new tab)

Worth reading?