AI Engineering¶
AI engineering builds systems whose behavior is partly learned or generated rather than fully specified in code. The central challenge is not calling a model; it is defining useful behavior, evaluating uncertain outputs, controlling risk, and operating the complete system.
When AI Fits¶
AI is useful when inputs are unstructured, acceptable outputs have a range, examples or feedback exist, and occasional uncertainty can be detected or safely handled. Prefer deterministic code, search, rules, or ordinary statistics when the requirement is exact, easily specified, or high-stakes errors cannot be contained.
Start with the workflow and baseline:
- define users, task, decision, and harm from failure;
- build the simplest non-AI baseline;
- collect representative evaluation cases;
- test a model with a narrow interface;
- add retrieval, tools, or tuning only for measured gaps;
- deploy progressively with monitoring and fallback.
System Layers¶
product policy → input/context → model → validation/tools → output → feedback
Quality depends on every layer. A stronger model cannot repair missing source data, an unsafe tool boundary, or an invalid success metric.
Learning Types¶
- supervised learning maps labeled examples to predictions;
- unsupervised and self-supervised learning derive structure or targets from data;
- reinforcement learning optimizes behavior from rewards and interaction;
- generative models produce new content conditioned on context;
- discriminative models predict labels, scores, or boundaries.
Training, validation, and test data must represent the intended population and remain independent enough to measure generalization. Leakage makes evaluation optimistic.
Evaluation¶
Use task-specific outcome metrics plus slice analysis, adversarial cases, human review where judgment matters, latency, cost, and safety. Compare against the current production system and simple baselines. Keep a held-out regression set and periodically refresh it without silently rewriting history.
Offline scores do not guarantee workflow value. Validate with controlled production experiments and monitor downstream decisions.
Risk¶
Consider invalid or fabricated output, harmful bias, privacy leakage, security abuse, automation bias, model or data provenance, intellectual-property constraints, environmental and financial cost, and loss of human recourse. Risk grows when output is personalized, consequential, autonomous, or difficult to reverse.
The living NIST AI Risk Management Framework organizes risk work around governing, mapping, measuring, and managing. Apply controls according to the use case rather than as a generic checklist.
Choosing the Next Technique¶
| Observed need | Start with |
|---|---|
| Generate or classify text/images | Prompted foundation model |
| Use current private sources | RAG |
| Perform bounded external actions | Fixed workflow, then agents if necessary |
| Change stable style or task behavior | Examples, then consider tuning |
| Operate learned models | MLOps |
See LLMs and Prompt Engineering for mechanics and interface design.
Checklist¶
- Is AI better than a deterministic baseline?
- Is success measured on representative cases and slices?
- Are uncertainty, abuse, privacy, and human recourse designed?
- Are permissions and side effects bounded?
- Can the system fall back, roll back, and be audited?
- Does production feedback reveal real outcomes rather than engagement alone?