Why “Set It and Forget It” is the Biggest Lie in AI Right Now

The urgency with which AI is evolving is staggering. Former Cisco CEO John Chambers put it best: “AI advances at five times the speed of the internet… traditional strategy refresh cycles of two to three years are no longer sufficient.” (Business Insider). So this isn’t hype – it’s a mandate.

In 2024, 78% of organizations reported using AI in at least one business function, and 71% were using generative AI, per recent McKinsey data. (Deliberate Directions). Yet despite rapid adoption, the real challenge lies not in deploying AI – but in maintaining and improving it.

Because model drift is real. A staggering 91% of ML models suffer performance degradation over time (McKinsey & Co.). Left unchecked, this can result in hallucinations, skewed outputs, and bad decisions being made on autopilot.

The pace of change is relentless – not just new models, but new benchmarks, APIs, and behaviors shift constantly, and AI doesn’t always self-correct.

The engineering team behind the A(i)scent platform is continuously immersed in code – tuning, validating, refining. The platform relies on real-time industry data in making its client-facing recommendations, so it truly is a living, breathing thing. Well, maybe not breathing, but you get the idea.

The point is, you can’t afford to turn critical decisions over to AI and think that you’re forever set. The pace of change is relentless – not just new models, but new benchmarks, APIs, and behaviors shift constantly, and AI doesn’t always self-correct. You have to be actively monitoring it. Even household name LLMs can get complacent and neglect model upkeep, despite claiming otherwise.

As the graph below illustrates, even when a model starts off accurate, error rates climb fast – and unpredictably – as time passes. A model’s error rate can rise sharply with age, even when it was top-tier at day zero. It’s a vivid reminder that AI doesn’t “set and forget’; it ages – and so must our vigilance.

Source: https://www.nannyml.com/blog/91-of-ml-performance-degrade-in-time

Continuous Optimization Isn’t Optional:

If your company is deploying AI – whether proprietary or off-the-shelf – this must become part of your DNA.

• Embed automated drift detection tools in your pipeline. They’re not obscure – many teams already use and benefit from them (Business Insider).
• Retrain on triggers, not calendars. Quarterly patching isn’t enough in many environments. Dynamic, data-driven retraining is smarter (Artificial Intelligence in Plain English).
• Guard against model collapse, especially when working with synthetic or recursively generated data. Left unchecked, errors compound (PMC, 3ft.com).

What This Means for You:

AI can be a force multiplier – if, and only if, you treat it as a living system. That means:

• Designing continuous learning loops, not one-off deployments.
• Embedding human review and validation at every stage.
• Recognizing that even industry leaders aren’t always as diligent as they say.

This isn’t theory – it’s something we see ourselves and something we take very seriously at NorthShift. Every improvement in the algorithm sparks new insight. Every validation layer builds resilience. Every human action to validate AI’s thinking mitigates risk.

In this era, your competitive edge isn’t adopting AI – it’s maintaining the discipline to keep it razor sharp

Subscribe to have “Elevated Insights” delivered to you every Tuesday