Evidence — Records of Judgment

Should investment in AI be evaluated only by technical precision? The issue was an evaluation of a churn-prediction AI model, but there were questions to confirm before that: overfitting risk, separation of seasonality, and data freshness. These three points were challenged first. A simple hit-rate evaluation not linked to the effect of the initiative was rejected, and a validation method was proposed on the spot.

Another issue was how to interpret the favorable figure that “customers who moved to the new pricing plan have low churn-risk scores.” If the population is limited to recent movers, the figure may be statistically unsurprising. Customers who have just moved do not churn.

A premature declaration of success was therefore put on hold. The role of questioning numbers produced by AI cannot be entrusted to AI.

Adoption of a new generative-AI coding tool was becoming a fait accompli through expense claims. No one had explicitly approved it, yet it was beginning to be used. This is one of the most common governance gaps in the AI era. Before allowing the new tool to touch confidential data, the work was stopped.

Review of live code and live repositories was directed to an existing approved AI bot, while the new tool was limited to writing assistance that contained no confidential information. After designing that division of roles, the practice of treating adoption through an expense claim as “consent” was explicitly rejected.

Data scope, security, and contract terms: until the tool passed all three gates in writing, it could not touch live data. What AI may decide and what people must continue to decide: this is precisely where the boundary is drawn.

Was the fall in productivity in the DX organization a staffing problem or a decision-structure problem? This separation was at issue in a DX development division of a large company. The mood on the ground was that there were not enough people. Task-management tickets had indeed risen by more than 10 percent in a short period, and many were overdue. But the data told a different story. Tickets whose direction had already been decided were accumulating without classification: not a staffing shortage, but a hygiene problem.

Hiring and reminders were rejected. Bulk closure and triage of dormant tickets became the highest priority. A kill condition was made explicit: no hiring decision until the triage was complete.

Solving busyness by adding people is abandoning diagnosis. Name the problem correctly first.

Turnover rose in a data-analytics specialist organization. To keep immediate work moving, business-experienced people and AI use could cover the gap. But devalue engineering capability, and neither it nor analytical capability can escape a structure of outside dependence. Short-term staffing and longer-term capability defense were clearly separated, then the definition of the hiring target was led anew.

The immediate hire was not an “analyst.” It was a “planner” who can articulate purpose and create a path to win.

The competitive field was reset from a salary competition that could not be won to points of differentiation, and priorities were set. Hiring is not replenishing headcount. It is a statement of what the organization will compete on.

It was a decision that, if treated as a field improvement proposal, could miss a risk directly tied to revenue. At a retailer organized around a membership program, the question was whether member-tier update frequency should be shortened from annually to monthly or quarterly. Behind the benefit of stimulating purchase frequency was a risk that could not be ignored: promotional discounts and membership-benefit discounts could stack on the same sales, directly affecting operating profit.

After separating benefits and risks, the decision was not about the content of KPI design itself. Because the issue was likely to have a financial impact, consultation with the corporate-planning function became a mandatory condition. In other words, the line between which decision is subject to whose approval.

KPIs can be corrected when wrong. Get the approval hierarchy wrong, and you lose the very mechanism by which the error is noticed.

In a customer-platform integration project run in parallel by multiple operating companies, requirements definition was approaching closure without approval criteria for moving between phases. Trying to lock the entire design at once would have been the wrong move. The scope was narrowed to the smallest point that had to be decided immediately, and a three-stage approval framework was drafted independently. Reviewers were separated into three layers — the field, project management, and management — rejecting the inefficiency of elevating every item to management.

The release decision was also redesigned from a single go/no-go call into four stages: release, initial monitoring, first monthly-close confirmation, and a declaration of stable operation. At each stage, approvers were deliberately placed asymmetrically according to where the substantive risk sat among participating organizations.

Approval is not a formality. It is placing judgment where the risk is.