ProgramDistill gives agents working reference web apps and asks them to infer and rebuild the missing behavior, closer to reverse engineering a live internal tool than solving a neatly written ticket. Its automated mine–craft–patch pipeline extracted 1,975 replay-verified behaviors from 26 applications and created 4,063 tasks without human annotation. On cumulative full-application workflows, the paper reports GPT-6 Astra at 49.2% and Claude Opus 5 at 28.8%. In partial reconstruction, success fell sharply as the number of restoration steps rose from one to eight.
The benchmark’s best idea is not the leaderboard. Each feature has replayable behavior tied to a gold patch, so failures can be separated into discovery, implementation and compounding-depth problems. That makes it more useful for deciding whether an agent can maintain unfamiliar production software than another collection of isolated GitHub issues. The caveat is also useful: 26 web apps remain a narrow ecology, and automatically mined behaviors may reward what is easy to replay rather than what is operationally important. ProgramDistill paper