AI Researcher Automates Intellectual Toil, Revolutionizing Agent Performance Analysis

Breaking News: Copilot Applied Science Unveils Self-Optimizing Analysis Tool

In a development that could reshape how software teams evaluate AI agents, a researcher at GitHub's Copilot Applied Science division has built a system that automates the tedious analysis of coding agent performance—freeing developers to focus on creative problem-solving.

“I may have just automated myself into a completely different job,” said the researcher, who requested anonymity to maintain focus on the project. The tool, dubbed eval-agents, uses GitHub Copilot to surface patterns in agent behavior streams called trajectories.

Background: The Trajectory Problem

A large part of the researcher's role involves analyzing coding agent performance against standardized evaluation benchmarks, such as TerminalBench2 and SWEBench-Pro. Each task in these benchmarks produces a trajectory—a JSON file describing the agent's thought process and actions step by step.

With dozens of tasks per benchmark set and multiple runs daily, researchers end up sifting through hundreds of thousands of lines of code. “It's an impossible task to do alone,” the researcher explained. Previously, they used GitHub Copilot to reduce the data from hundreds of thousands of lines to a few hundred—but even that repeatable loop became a form of intellectual toil.

The Repetitive Loop

“The engineer in me saw this repetitive task and said, ‘I want to automate that,’” the researcher noted. Agents provide the means to automate intellectual work, and thus eval-agents was born. The project aims to eliminate the manual pattern-spotting phase entirely.

The Plan: Collaboration as a Guiding Principle

Engineering and science teams work better together. That principle drove the design of eval-agents, which was built with three core goals:

• Make agents easy to share and use across the team.
• Make it easy to author new agents without steep learning curves.
• Make coding agents the primary vehicle for contributions, embedding automation into the workflow.

These goals reflect GitHub's core values and the researcher's experience as an open-source maintainer for the GitHub CLI. “Bullets one and two are in GitHub's lifeblood,” they added.

What This Means: Faster Development Loops

eval-agents enables a cycle where researchers spend less time parsing logs and more time improving agent performance. “Applying these learnings has unlocked an incredibly fast development loop for myself and enabled my teammates to build solutions tailored to their needs,” the researcher said.

The tool represents a shift from manual analysis to agent-driven development. It may accelerate AI research cycles across the industry, as similar teams adopt automated trajectory analysis. The full technical details are expected in an upcoming report.

— Breaking news report, updated [date]