I Hid Four Traps in a Forecasting Task. Here Is What Four AI Assistants Did.

Four AI assistants were tested on a time-series sales forecasting task containing four hidden, realistic traps. These traps included a data leakage feature, a reporting delay in the target variable, a post-promotion sales dip, and a structural break in the data. The experiment evaluated whether the AI models could identify and correctly handle these issues in their generated code. Success was judged not by the AI's commentary but by whether the final, runnable code implemented the correct solutions, with an additional check to ensure reported metrics were reproducible.