The better the numbers, the earlier they should be questioned.

Should investment in AI be evaluated only by technical precision? The issue was an evaluation of a churn-prediction AI model, but there were questions to confirm before that: overfitting risk, separation of seasonality, and data freshness. These three points were challenged first. A simple hit-rate evaluation not linked to the effect of the initiative was rejected, and a validation method was proposed on the spot.

Another issue was how to interpret the favorable figure that “customers who moved to the new pricing plan have low churn-risk scores.” If the population is limited to recent movers, the figure may be statistically unsurprising. Customers who have just moved do not churn.

A premature declaration of success was therefore put on hold. The role of questioning numbers produced by AI cannot be entrusted to AI.