Skip to content

MLOps and AI Operations

MLOps applies software, data, and operational engineering to learned systems. A release includes code, model, data assumptions, features, configuration, evaluation, and serving infrastructure. Reproducibility and monitoring must cover the whole decision path.

Lifecycle

problem → data → train/configure → evaluate → register → deploy → monitor → learn

Begin with a baseline and offline evaluation before building a platform. Automate repeated, error-prone work only after the workflow is understood.

Data and Experiments

Record dataset identity, extraction query, time range, schema, labels, exclusions, transforms, code revision, dependency environment, random seeds, parameters, hardware, metrics, and artifacts. Sensitive raw data may require controlled references rather than copies.

Split data by the deployment reality—often time, user, device, or entity—not randomly when that leaks related examples. Test slice performance and label quality. An experiment tracker records evidence; it does not make experiments comparable unless inputs and evaluation agree.

Model Registry and Release

A registry should bind an immutable model artifact to provenance, evaluation, intended use, limitations, owner, approval, and deployment status. A stage name such as “production” is movable; deployment should resolve it to immutable content.

Promotion gates should compare against the current baseline on quality, critical slices, safety, latency, resource use, and cost. Human approval belongs where risk or policy requires judgment.

Serving

Choose batch prediction when results can arrive later; online serving when the workflow needs immediate output; edge serving for offline, privacy, or latency constraints. Define input schema, preprocessing parity, timeout, concurrency, fallback, and output validation.

Deploy canaries or shadows for risky changes. Shadow output must not cause side effects and still needs privacy controls. A rollback may require restoring model, feature logic, prompts, index, and configuration together.

Monitoring

Monitor four layers:

  • system: availability, latency, saturation, errors, and cost;
  • data: schema, missingness, ranges, categories, freshness, and drift;
  • model: output distribution, confidence calibration where meaningful, and slice behavior;
  • outcome: delayed labels, business value, harmful effects, overrides, and complaints.

Drift is a diagnostic signal, not proof of quality loss. Performance can degrade without obvious input drift, and benign seasonal change can trigger drift. Re-evaluate with trustworthy outcome labels.

Generative and Agent Systems

Version model, prompt, retrieval corpus/index, tools, schemas, and policy. Store representative traces with sensitive data minimized. Evaluate factual support, citations, refusal, prompt injection, tool trajectories, token use, and side effects. See RAG, Agents, and Prompt Engineering.

Training and Supply Chain

Pin code and dependencies, verify data and model provenance, isolate training jobs, protect checkpoints, scan serving images, and use short-lived workload identity. Third-party model files and custom loaders are untrusted artifacts and may execute code depending on the framework.

Accelerators improve selected workloads but introduce queueing, memory, topology, and utilization constraints. Size with representative sequence lengths, batches, precision, and concurrency rather than parameter count alone.

Retraining

Retraining should respond to validated need, not a blind schedule. Every candidate repeats evaluation and release controls. Prevent feedback loops in which model decisions determine future labels without independent observation. Preserve prior artifacts and data lineage for rollback and audit.

Operations

Define owners for data, model, platform, product outcomes, and incidents. Test missing features, stale sources, provider failure, quota exhaustion, corrupt artifacts, rollback, and delayed labels. Track cost per useful prediction or completed task, not only infrastructure spend.

Checklist

  • Can a release be reproduced from immutable code, data references, and configuration?
  • Are evaluation splits and slices representative without leakage?
  • Is every deployed artifact tied to provenance and approval?
  • Are system, data, model, and outcome behavior monitored separately?
  • Can all coupled components roll back coherently?
  • Are privacy, security, bias, and human recourse owned throughout operation?
  • Is retraining triggered and validated by evidence?