When the Model Works but the Idea Doesn’t: Knowing When to Kill an ML Project

Most AI case studies are written backwards. They start at the finished system, describe the architecture that shipped, and quietly imply that it was the plan all along. That version is easier to write and easier to sell, but it teaches you nothing about how these projects actually go, and it hides the decision that mattered most.

This is the other version. It’s the story of a machine-learning initiative I built, deployed, tested against live data — and then killed. What replaced it is now one of the more useful pieces of the system it was built for, and it exists only because the first approach failed in a specific, informative way.

The problem we were trying to solve

The setting was a high-volume support operation. Inbound requests arrived through several channels, in several languages, at a rate no small team can triage by reading each one. The team knew, in aggregate, that some tickets were quick and some were painful. What they couldn’t do was tell which was which before someone opened it.

That’s a real operational problem. If you can predict difficulty at intake, a lot of good things follow: you route hard tickets to senior staff, you staff by expected load rather than raw count, you set realistic response expectations, and you stop burning your best people on work that anyone could close.

So the plan was straightforward. Take a body of historical tickets. Have experienced reviewers rate each one on two dimensions — roughly, how involved the issue was and how much effort it took to resolve. Train a model on those labels. Expose it as an API. Score incoming tickets on arrival.

We built it. It ran.

This is the part that matters for the lesson, so I want to be precise: the technical work succeeded.

The labelling exercise was completed. The model trained. It was deployed behind a live API endpoint and tested against real ticket data, not a held-out slice of the training set. Predictions came back. The plumbing worked end to end.

And it was still the wrong thing to have built.

The three signals

Three problems showed up, and none of them was a modelling problem.

Reviewers didn’t agree with each other. Given the same ticket, different raters produced different scores. Not wildly different — but different enough that the “difficulty” the model was learning wasn’t a property of the ticket. It was a blend of the ticket and whoever happened to be rating it that day. When your labels encode the labeller, your model learns the labeller.

The ratings collapsed toward easy. The distribution didn’t spread across the scale; it bunched at the low end. Partly that’s honest — most support tickets genuinely are routine. But it meant the scale was doing very little work. A model that predicts “easy” for almost everything can post a respectable accuracy number while carrying almost no operational information, and the cases you actually wanted flagged — the rare hard ones — were the cases with the fewest examples to learn from.

Rating fatigue degraded the labels over time. This one is the most interesting, and the least discussed in ML writing. Rating tickets is tedious. Reviewers who are careful at ticket fifty are less careful at ticket four hundred. So label quality wasn’t just noisy — it was noisy in a way that drifted, with later labels systematically worse than earlier ones. That’s not something you fix with more data. More data makes it worse.

Tuning problem or premise problem?

Every one of those symptoms has a plausible tuning-flavoured response. Low inter-rater agreement? Write a better rubric, run a calibration session. Skewed distribution? Rebalance, resample, weight the rare class. Fatigue? Shorter rating sessions, more reviewers.

Any of those could have been the next sprint. That’s the trap. Each fix is individually reasonable, individually affordable, and collectively an indefinite commitment to a premise nobody has re-examined.

The premise here was: ticket difficulty is a stable, knowable quantity that experienced humans can consistently express as a number.

Once you say it out loud, the evidence is already in. Three independent signals were all telling us the same thing — the target variable wasn’t well-defined. You cannot build a reliable predictor of a quantity that doesn’t reliably exist.

The distinction I now apply to any model that underperforms:

• A tuning problem is when the signal is there and you haven’t extracted it. Different architecture, better features, more data, cleaner preprocessing — the ceiling is above you and the work is getting you closer to it.
• A premise problem is when the thing you’re predicting isn’t a coherent thing. No amount of engineering raises the ceiling, because the ceiling is the definition of the target.

The diagnostic question is not “can we improve this?” It is: if this model were perfect, would it be measuring something real? For a target that two qualified people score differently, the answer was no.

We stopped.

The salvage: what the failure was actually telling us

Killing an initiative and learning nothing from it is just a write-off. The value came from one more question after the decision: which tickets could the model classify reliably?

The answer: the highly repetitive ones. The cases the model handled confidently were the cases that recur constantly, follow a predictable shape, and get resolved the same way every time.

Read that finding as an operational fact rather than a model diagnostic, and it inverts. The model wasn’t good at those tickets because they were easy to score. It was good at them because they were nearly identical to hundreds of tickets before them.

And if a request is so repetitive that a machine recognizes it on sight, the interesting question isn’t how to route it faster. It’s why a person is answering it at all.

That’s the pivot. The failed classifier was trying to predict effort. The finding pointed at a way to remove it.

What got built instead

The replacement is a generative-AI content pipeline that turns resolved support conversations into published self-service material. In outline:

1. Intake and classification. Incoming requests are categorized against a defined support taxonomy, with automatic language detection and translation so non-English requests enter the same structured pipeline as everything else.
2. Draft generation. Resolved conversations — the question as it was actually asked, and the answer that actually closed it — are passed to a generative model that drafts a public-facing knowledge article.
3. A human review gate. Nothing publishes automatically. Every draft lands in a queue with an explicit review status and a person who approves, edits, or rejects it.
4. Publication. Approved articles go live as public self-service content, where they intercept the next instance of that question before it becomes a ticket.

That review gate is not a nicety. It’s the same pattern I use for AI-assisted invoice extraction and every other place a model’s output touches something that matters: the model proposes, a human disposes, and anything the system isn’t confident about goes to a queue rather than silently into production. It’s the difference between AI you can put in front of customers and AI you have to apologize for.

Why the second system worked where the first didn’t

Same domain. Same data. Same team. The difference is the label.

The classifier depended on humans generating a subjective score that had no independent existence — an artificial layer of judgment invented purely to feed the model, and one that got worse the more of it you asked for.

The content pipeline depends on something that already exists as a natural byproduct of the work: a ticket that was resolved. The resolution is the ground truth. Nobody rates anything. Nobody gets fatigued. Every closed ticket adds signal for free, and the system gets more useful the more the support team simply does its job.

That’s the general principle, and it’s worth more than either system: prefer targets your business already produces as a side effect of operating over targets you have to manufacture through a labelling exercise. Manufactured labels are expensive, inconsistent, and degrade under volume. Operational ground truth is cheap, consistent, and compounds.

The failed project’s real output wasn’t the model. It was learning which of those two we had.

How to run AI initiatives so this is cheap instead of expensive

If you’re commissioning AI work — from me or anyone — a few things make the difference between a productive dead end and an expensive one.

Write down the kill criteria before you start. “We stop if inter-rater agreement is below X” is a decision made calmly in week one. Made in month five, with budget spent and expectations set, the same decision is a political event. Pre-committing is what makes stopping possible.

Validate the label before you build the model. Have two people independently rate the same hundred records and compare. That exercise costs a day. It would have surfaced everything above before a line of training code was written, and it is now the first thing I do on any supervised-learning proposal.

Make the first version cheap enough to throw away. The initiative described here was killed without much drama precisely because it hadn’t been over-invested in. Deploy small, test against real data early, and keep the sunk cost low enough that evidence can still change your mind.

Debrief the failure before you close the file. The pivot came from asking what the model did get right. If we’d shut it down without that question, the second system wouldn’t exist.

Ask your developer what they’ve killed. Anyone who has shipped AI work of real substance has abandoned something. If the answer is “nothing,” you’re either talking to someone very new or someone who kept tuning past the point where they should have stopped — and you’ll be paying for that habit.

The point

There’s a version of this story where the model gets refined for another two quarters, the rubric gets rewritten twice, a second rating round is commissioned, and the eventual outcome is a difficulty score nobody trusts and nobody uses — with a plausible narrative at every step explaining why the next fix will be the one.

The version that happened cost less and produced more, because someone asked whether the thing being measured was real before asking how to measure it better.

Building AI features is not the hard part anymore. Recognizing which ones are worth building — and being willing to say so when the answer changes mid-project — is.

Thinking about an AI initiative and unsure whether the premise holds up? I build custom business systems on the Claris FileMaker platform, including AI-assisted document processing, support automation, and content pipelines — with the validation and human-review discipline that keeps them trustworthy in production. If you’d like a candid read on whether an idea is worth building, I’m happy to look at it with you. (Contact)