From Notebook to Production Without “Magic” MLOps

Teams often arrive with a notebook that beats the baseline on a held-out slice and a slide that says “MLOps next quarter.” The gap between that notebook and a system auditors and operators can trust is where most AI programmes stall. The fix is not a mythical MLOps unicorn platform sold with a logo slide — it is a minimum chain of evidence from raw data to scored output, owned by people who will be on call when the model misbehaves on a Monday morning.

We have seen the same pattern in retail demand forecasting, industrial defect detection and credit scoring: a brilliant offline metric, then six months of platform evaluation while production still runs on a cron job and a pickle file on a shared drive. The organisations that ship fastest start narrow. They pick one decision the business already makes manually, one metric that matters to that decision, and one rollback path when the model is wrong. Everything else is sequenced after proof, not before.

The minimum production chain

Production ML rests on five links. First, a data contract: schema, freshness SLA, PII rules, named owner and escalation when upstream feeds slip. Without a contract, “the data team will fix it” becomes the incident runbook. Second, a versioned training set — an immutable snapshot ID referenced in every model card and deployment record. If you cannot point to the exact rows that trained the model scoring today, you cannot explain a bad Tuesday.

Third, a reproducible training pipeline: container or locked environment, random seed, git SHA, dependency lockfile. Fourth, a model registry entry with metrics, known limitations, approver name and the artefact promoted to serving. Fifth, serving with observability: latency, error rate, input distribution, and a feedback loop when labels arrive late. Skip any link and you pay later — usually during an incident when no one knows which data trained the model currently affecting customers.

A logistics client shipped with this chain in eleven weeks: nightly features in Parquet, training in a scheduled pipeline, promotion gated on hold-out performance and schema checks, serving behind an API with dashboards for input drift. No feature store, no Kubernetes for training. The model was boring; the business trusted it because operators could answer basic forensic questions.

What to postpone on purpose

Full feature stores, real-time streaming pipelines and AutoML marketplaces can wait. Build them when you have two or more models genuinely reusing features, or sub-minute latency requirements with a measured cost of batch scoring that exceeds the engineering investment. Until then, a well-structured batch pipeline with nightly features and explicit promotion gates is often more reliable — and far easier to explain to risk committees who ask “what happens if this model is wrong?”

We counsel teams to write a deferral list with triggers: “feature store when model B shares more than thirty percent of features with model A” or “streaming when p95 batch latency blocks revenue.” That list prevents platform tourism — six months evaluating vendors while nothing reaches production — and keeps architects honest about reuse that does not yet exist.

Governance that engineers will not hate

Model cards do not need forty pages. One page is enough: intended use, out-of-scope uses, training data summary, fairness and limitation notes, owner, rollback procedure. Tie promotion to automated checks — schema validation, performance floor on hold-out, bias thresholds where applicable — plus a human sign-off for production namespaces. Engineers accept governance when it blocks bad deploys automatically instead of adding meetings that review what CI could have caught.

Governance also means who can press promote. Separation between trainer and approver mirrors production change control. A credit scoring team adopted this after a misconfigured threshold reached production because the same person trained, evaluated and deployed during a deadline crunch. The rollback took longer than the fix because nobody else had context.

Drift before users notice

Monitor covariate shift on top inputs and outcome delay where labels arrive late. Alert on distribution change even when accuracy looks stable — stale accuracy is a common trap when the world changed three weeks ago but labels lag. A retailer continued trusting a demand model through a product mix shift because weekly accuracy on old SKUs looked fine; overstocks appeared in categories the model had never seen at current volumes.

Pair technical drift signals with business KPIs: return rate, manual override rate, margin impact. When operators override the model more often, that is often a drift signal faster than any statistical test. Instrument overrides; they are free labels about model trust.

Scenario: fraud scoring vs predictive maintenance

Two models, same company, different maturity needs. Fraud scoring demands low latency, explainability for investigators and immediate rollback — the minimum chain includes shadow mode before full promotion and a kill switch wired to a feature flag. Predictive maintenance on batch telemetry tolerates nightly scoring, emphasises data contract with OT teams and documents that missed alerts have safety implications reviewed with operations.

Trying to run both through one generic “AI platform” RFP delayed both. Sequencing fraud first — higher regulatory visibility — built organisational muscle for registry, monitoring and access control that maintenance inherited six months later. One platform team, two staged deliveries, zero magic.

What to do next

Pick one model that affects a real decision this quarter. Write the data contract, snapshot last month’s training data, containerise training, register the artefact and add three monitoring charts: latency, error rate, top-feature distribution. If you cannot complete that chain, pause the platform RFP. Our Data Science Models practice helps teams ship the minimum chain first, then grow platform capabilities when reuse justifies them — not when a vendor brochure says so.