Recommended
How an MCP tool cut triage of 600 test failures to about 90 minutes (opens in a new tab)
You will learn how Salesforce engineers split deterministic code from AI reasoning in an MCP workflow that clusters and reassigns test failures, and which guardrails prevented duplicate work items and misrouting.
Why read this
The reasoning is useful even if you never touch the GUS internals.
- Retrieve failures
- Gather context from stack traces
- Detect patterns or apply categories
- Reassign individually, in bulk or by team
Salesforce’s engineering team faced over 600 test failures triggered by the retirement of the SOAP API login mechanism. Expected failures and genuine regressions could produce similar exceptions and span several scrum teams, so a wrong classification could hide a real defect or send engineers chasing the wrong problem for days. The existing process required manual stack trace reading and reassignment through an interface limited to 50 failures at a time, and it could create duplicate work items or misroute failures.
The team built AI-Assisted Test Failure Buddy as a Python and FastMCP process exposing eight tools to Claude Code or Cursor, connected to the APIs behind GUS, Salesforce’s internal work item system. The design separates an interface layer of typed MCP tools, a core logic layer that controls API calls, identifiers, parsing, validation, batching, clustering and reassignment, and a support layer for authentication and tokens. The workflow runs in four stages: retrieve failures, gather context from stack traces, detect patterns or apply categories, and reassign failures individually, in bulk or by team.
Grouping was the hardest part. Stack traces offer no stable key for root cause, and the same exception type could mean an expected SOAP login failure in one context and an unrelated bug in another. The team combined deterministic processing, such as regular-expression matching and a preset recognizing strings like INSUFFICIENT_ACCESS, with AI reasoning reserved for novel traces, unmatched clusters and ambiguous ownership. To avoid duplicates, the tool assigns related failures to one existing work item and uses explicit failure IDs in place of the broader interface action that had caused misrouting.
The estimated two or three days of manual triage for one 600-failure work item fell to roughly one to one and a half hours for a single engineer. The tool has since been used by three or four teams, though it has no adoption telemetry yet. The article also notes that Claude Code helped build the tool, starting from a markdown design document with human review throughout.
Read the original at Salesforce Engineering (opens in a new tab)