Keeping an AI system honest after launch: monitoring, feedback and drift

Traditional software usually fails loudly. A page will not load, a calculation throws an error, an alarm sounds. AI systems often fail quietly. A model keeps producing answers, the dashboard stays green and the quality slowly erodes, until someone notices that the results no longer match what they see on the ground.

That is why launch is the beginning of the work, not the end. This article describes a practical routine for keeping an AI system honest over months and years, whether it is a land cover model, a document assistant or an agent that handles requests.

Why AI systems drift

• The world changes. Seasons turn, new buildings appear, prices move, regulations are updated, and the patterns that the model learned no longer hold.
• The inputs change. A satellite sensor is replaced, a form is redesigned, a data source changes its format, or users start asking new kinds of questions.
• The documents change. An assistant that was accurate in January quotes outdated rules in June if the index is not updated.
• Upstream models change. If you use an outside model service, an update can shift behaviour without warning.
• People adapt. Users learn how to phrase things, or find ways around the system, and the mix of cases moves.

Three levels of monitoring

Watch the system at three levels. Each catches different problems.

1. 01Health: is it running? Uptime, errors, speed and cost. This is standard monitoring, and it tells you the system is alive, not that it is good.
2. 02Behaviour: what is going in and coming out? The kinds of inputs, the spread of predictions, the share of answers that refuse or hand over to a person. Sudden shifts are a warning.
3. 03Outcomes: is it right? Accuracy against checked examples, user ratings, corrections made by staff and results in the real world.

What to track

• Input statistics: for imagery, brightness and cloud cover. For text, length, topics and language. For tabular data, ranges and missing values.
• Output statistics: the share of each predicted class, the average confidence and the rate of “I do not know”.
• Quality samples: a few dozen results reviewed by a person each week or month.
• A fixed test set: re-run on a schedule, with scores kept in one place so you can see trends.
• User signals: thumbs up and down, corrections, escalations to a person and support requests.
• Cost and speed: because quality fixes can raise both.

Build a feedback loop

Users are your best sensors. Make it easy and safe to report a bad result, ideally in one click, with an optional note. Then read what they send.

• Put a feedback button next to every result.
• Capture context automatically: the input, the output and the version of the system.
• Review reports every week, and sort them into causes: data, model, instructions, missing documents or user error.
• Add the interesting failures to your test set, so they are never forgotten.
• Tell users what was fixed. People report more when they see that it matters.

Decide your triggers in advance

It is far easier to act calmly if you decide beforehand what level of quality is unacceptable and what you will do about it.

• Amber: quality drops by a small amount, or an input statistic drifts. Investigate, and review a larger sample.
• Red: quality falls below the agreed minimum, or a serious error reaches a user. Fix quickly, add a warning to the interface or temporarily restrict the system to cases it handles well.
• Rollback: keep the previous version available, so that you can return to it if an update makes things worse.

When and how to retrain or update

• On a schedule, if the world changes quickly, for example before each growing season for a crop model.
• When monitoring shows a sustained drop in quality, not a single bad day.
• When new regions, sensors or document types are added.
• After collecting enough new, checked examples to make a real difference.

Treat every new model like a new release. Test it on the fixed test set, compare it with the current one, run it alongside the old one for a while if you can and keep the old one ready until you are confident.

Keep records

Keep a log of every version of the model, the data and the settings behind it, with the scores it achieved. When someone asks why this month’s results differ from last month’s, you should be able to answer in minutes. Also keep a short changelog of what was changed and why, so that the team and the users can follow.

Governance and people

• Name an owner for the system’s quality, with time to do the job.
• Agree who may change prompts, models and data sources, and how changes are reviewed.
• Review performance with the business sponsor each quarter.
• Check that access controls and data handling still match the rules, especially as the system grows.
• Plan the end of life: when a system is no longer useful, retire it cleanly.

A simple monthly routine

1. 01Look at the health and behaviour dashboards for unusual changes.
2. 02Review a sample of 30 to 50 results by hand, and mark them against the standard.
3. 03Re-run the fixed test set, and compare it with last month.
4. 04Read the user feedback, and sort it by cause.
5. 05Decide on fixes, and add new failures to the test set.
6. 06Write three lines on what changed, and share them with the team.

Mistakes we try to avoid

• Relying on the green health light as proof of quality.
• Reviewing only the cases that users complained about, which hides the quiet failures.
• Updating the model without re-running the tests.
• Letting the documents behind an assistant go stale.
• Having no named owner, so that everyone assumes someone else is watching.

Frequently asked questions

How often should we review quality?

Monthly is a good start for most systems, and weekly in the first few months after launch or after any major change.

How do we measure accuracy without labels?

Take a regular random sample of outputs, and have a person who knows the subject mark them. A small, honest sample beats none.

What if we use a model from an outside provider?

Pin the version where you can, re-run your tests when it changes and watch for shifts in behaviour. Do not assume that an update is an improvement for your case.

Is it expensive to monitor?

A basic routine costs a few hours a month. It is small compared with the cost of acting on wrong results for months before anyone notices.

A trustworthy AI system is not one that never fails. It is one whose owners are watching, listening and ready to fix it. Build that routine at the start, and your system will stay as useful as it was on the day it launched.