From Notebook to Production Without “Magic” MLOps

Teams often arrive with a notebook that beats the baseline on a held-out slice and a slide that says “MLOps next quarter.” The gap between that notebook and a system auditors and operators can trust is where most AI programmes stall. The fix is not a mythical mythical MLOps platform sold with a marketing slide — it is a minimum chain of evidence from raw data to scored output, owned by people who will be on call when the model misbehaves on a Monday morning.

We have seen the same pattern in retail demand forecasting, industrial defect detection and credit scoring: a brilliant offline metric, then six months of platform evaluation while production still runs on a cron job and a pickle file on a shared drive. The organisations that ship fastest start narrow. They pick one decision the business already makes manually, one metric that matters to that decision, and one rollback path when the model is wrong. Everything else is sequenced after proof, not before.

The minimum production chain

Production ML rests on five links. First, a data contract: schema, freshness SLA, PII rules, named owner and escalation when upstream feeds slip. Without a contract, “the data team will fix it” becomes the incident runbook. Second, a versioned training set — an immutable snapshot ID referenced in every model card and deployment record. If you cannot point to the exact rows that trained the model scoring today, you cannot explain a bad Tuesday.

Third, a reproducible training pipeline: container or locked environment, random seed, git SHA, dependency lockfile. Fourth, a model registry entry with metrics, known limitations, approver name and the artefact promoted to serving. Fifth, serving with observability: latency, error rate, input distribution, and a feedback loop when labels arrive late. Skip any link and you pay later — usually during an incident when no one knows which data trained the model currently affecting customers.

A logistics client shipped with this chain in eleven weeks: nightly features in Parquet, training in a scheduled pipeline, promotion gated on hold-out performance and schema checks, serving behind an API with dashboards for input drift. No feature store, no Kubernetes for training. The model was boring; the business trusted it because operators could answer basic forensic questions.

What to postpone on purpose

Full feature stores, real-time streaming pipelines and AutoML marketplaces can wait. Build them when you have two or more models genuinely reusing features, or sub-minute latency requirements with a measured cost of batch scoring that exceeds the engineering investment. Until then, a well-structured batch pipeline with nightly features and explicit promotion gates is often more reliable — and far easier to explain to risk committees who ask “what happens if this model is wrong?”

We counsel teams to write a deferral list with triggers: “feature store when model B shares more than thirty percent of features with model A” or “streaming when p95 batch latency blocks revenue.” That list prevents platform tourism — six months evaluating vendors while nothing reaches production — and keeps architects honest about reuse that does not yet exist.

Governance that engineers will not hate

Model cards do not need forty pages. One page is enough: intended use, out-of-scope uses, training data summary, fairness and limitation notes, owner, rollback procedure. Tie promotion to automated checks — schema validation, performance floor on hold-out, bias thresholds where applicable — plus a human sign-off for production namespaces. Engineers accept governance when it blocks bad deploys automatically instead of adding meetings that review what CI could have caught.

Governance also means who can press promote. Separation between trainer and approver mirrors production change control. A credit scoring team adopted this after a misconfigured threshold reached production because the same person trained, evaluated and deployed during a deadline crunch. The rollback took longer than the fix because nobody else had context.

Drift before users notice

Monitor covariate shift on top inputs and outcome delay where labels arrive late. Alert on distribution change even when accuracy looks stable — stale accuracy is a common trap when the world changed three weeks ago but labels lag. A retailer continued trusting a demand model through a product mix shift because weekly accuracy on old SKUs looked fine; overstocks appeared in categories the model had never seen at current volumes.

Pair technical drift signals with business KPIs: return rate, manual override rate, margin impact. When operators override the model more often, that is often a drift signal faster than any statistical test. Instrument overrides; they are free labels about model trust.

Scenario: fraud scoring vs predictive maintenance

Two models, same company, different maturity needs. Fraud scoring demands low latency, explainability for investigators and immediate rollback — the minimum chain includes shadow mode before full promotion and a kill switch wired to a feature flag. Predictive maintenance on batch telemetry tolerates nightly scoring, emphasises data contract with OT teams and documents that missed alerts have safety implications reviewed with operations.

Trying to run both through one generic “AI platform” RFP delayed both. Sequencing fraud first — higher regulatory visibility — built organisational muscle for registry, monitoring and access control that maintenance inherited six months later. One platform team, two staged deliveries, zero magic.

Promotion path we will sign in a model card

If you cannot fill this table, you are not ready for a second model — you are still in a notebook:

• Training snapshot ID — immutable, stored, referenced in the registry. Not “the latest extract from last Thursday.”
• Git SHA + lockfile — the exact code that produced the artefact. “It ran on my laptop” is not a promotion criterion.
• Offline metric + decision metric — AUC is not enough if the business cares about false blocks on shipments or missed fraud euros. Write both.
• Known failure modes — seasonality, a plant that just changed a sensor, a new product code with no history.
• Rollback — previous artefact still loadable in under 15 minutes, with who is allowed to press it.

We refuse to call a pickle file on a shared drive “production” in front of an auditor. The organisations that ship are boring: one decision, one API, one on-call, logs with prediction_id, feature hash and model version on every score. Feature stores come after the second model shares features — not as a year-one platform RFP. More on the modelling side: AI / machine learning and data-science models.

Drift checks that catch Tuesday before customers do

Uptime of the scoring pod is hygiene. The failures that hurt are silent: input schema still validates, but a vendor changed a unit; a plant added a line and the population shifted; a colleague “temporarily” hard-coded a default that became 40% of traffic. Cheap monitors we install on day one of serving:

1. Daily PSI or simple histogram compare on the five features that drive the decision — alert on a threshold you set with the owner, not a generic 0.2 from a blog.
2. Count of scores per hour vs last four same-weekdays. A 30% drop at 09:00 is often a broken upstream, not “low demand.”
3. Rate of fallback / null features. If it climbs, the model is guessing.
4. A weekly sample of ten scored cases reviewed by a human who owns the decision — fifteen minutes, written notes, not a committee.

If nobody is on call for “the scores look weird,” you have a demo with a nicer container.

What to do next

Pick one model that affects a real decision this quarter. Write the data contract, snapshot last month’s training data, containerise training, register the artefact and add three monitoring charts: latency, error rate, top-feature distribution. If you cannot complete that chain, pause the platform RFP. Our Data Science Models practice helps teams ship the minimum chain first, then grow platform capabilities when reuse justifies them — not when a vendor brochure says so.