Claudia Perlich, Head of Strategic Data Science at Two Sigma Investment Management, uses concrete historical cases in her September 23 essay on human intuition in machine learning.
In a New York City Housing Authority example, many apartment-level rows depend on roughly a hundred boilers. Shared equipment reduces independent evidence. A validation implication is that randomly splitting apartments can place the same boiler on both sides, making the test easier than predicting a genuinely new heating system. Perlich also describes the qualitative tradeoff between observations, feature count and signal strength: weak financial signals call for more evidence or materially fewer features, without a universal numerical ratio.
Her loan example concerns outcomes maturing at different speeds, making recent loans appear artificially default-prone. Her advertising example runs in the opposite direction: predictable ad-fraud bots produced an apparent step-change in predictive performance. The statistical task became easier because the observed population had changed.
A separate deployment case gives trust a specific form. A stronger linear quantile-regression model remained unused for two years; users preferred an objectively weaker nearest-neighbor model because they could inspect the comparable cases. The account does not quantify the sacrificed accuracy or prove that the weaker model led to better decisions. It identifies the inspection method users valued, rather than merely asserting that adoption matters.
This is practitioner commentary, not disclosure of a live Two Sigma investment model. Perlich's essay.