The Tangent study, appearing as ASE 2026 opens today after an August 9 preprint, began with 1,190 Python repositories using 11 agent frameworks. The IBM/Georgia Tech team ultimately hand-labeled 2,572 test methods from 240 modules across 12 projects; two authors reached a Cohen's kappa of 0.92. It also interviewed 10 senior practitioners.
The distribution is more useful than a generic warning. 61.8% of tests exercise a tool or agent in isolation, 34.6% cover interaction behavior, only 3.5% cover multi-tool invocation, and 0.1% test through an exposed API. Just 7.5% address non-functional requirements. 39.9% use simple example-like inputs; 34.4% mock an agent or tool; and only 8.7% assert on reasoning-related behavior—usually whether it appears, not whether it is good.
The paper's strongest engineering implication is about testability. Agent systems need controllability and observability over memory, tool selection, delegation, intermediate state, model outputs, tool responses, environment conditions, and replayable traces. A green suite can otherwise validate deterministic seams while scarcely interrogating the non-deterministic core. Audit test-type distribution before adding more cases, and make trajectories, side effects, and realistic multi-step environments explicit targets. The study diagnoses the gap; it does not prove a magic coverage percentage. Primary paper · IBM Research