Designing Machine Learning Infrastructure
1. Designing Model Serving Architecture
| Server | Detail |
|---|---|
| TensorFlow Serving | TF SavedModel |
| TorchServe | PyTorch |
| Triton Inference Server | Multi-framework, GPU |
| BentoML / KServe / Seldon | K8s-native |
| vLLM / TGI | LLM serving |
| SageMaker / Vertex AI | Managed |
2. Designing Feature Store Architecture
| Element | Detail |
|---|---|
| Online store | Low-latency (Redis, DynamoDB) |
| Offline store | For training (Parquet, BigQuery) |
| Definition | Single source of truth (DSL) |
| Train/serve consistency | Same transformation logic |
| Tools | Feast, Tecton, Hopsworks, Vertex FS |
3. Designing Training Pipeline
| Stage | Detail |
|---|---|
| Data prep | Validate, split, transform |
| Train | Distributed, GPU |
| Eval | Hold-out / cross-validation |
| Register | Track artifacts + metrics |
| Tools | Kubeflow, MLflow, Metaflow, ZenML |
4. Designing Model Versioning and Registry
| Element | Detail |
|---|---|
| Artifact storage | Object store + metadata |
| Lineage | Code + data + params → model |
| Stage labels | staging / production / archived |
| Tools | MLflow Model Registry, Vertex Model Registry |
5. Designing A/B Testing for ML Models
| Pattern | Detail |
|---|---|
| Shadow | Score in parallel; don't act |
| Champion-challenger | % traffic to challenger |
| Bandit | Adaptive allocation to better variant |
| Holdout | Measure incremental lift |
| Online metrics | Per-variant business KPIs |
6. Designing Model Monitoring and Drift Detection
| Type | Detail |
|---|---|
| Data drift | Feature distribution shift (PSI, KS test) |
| Concept drift | Label distribution / relationship change |
| Performance drop | Accuracy / AUC over time |
| Tools | Evidently, WhyLabs, Arize, Fiddler |
| Action | Alert → retrain |
7. Designing Batch vs Real-Time Inference
| Mode | Detail |
|---|---|
| Batch | Score all rows nightly; precompute |
| Online (real-time) | Per-request inference |
| Streaming | Score events as they arrive |
| Choose by | Latency, freshness, cost |
8. Designing ML Data Pipeline
| Stage | Detail |
|---|---|
| Ingest | Raw → lake |
| Validation | Schema + statistical checks (TFDV) |
| Transformation | Spark / Beam / dbt |
| Feature compute | Push to feature store |
| Labeling | Manual / weak / programmatic |
9. Designing Model Retraining Strategy
| Trigger | Detail |
|---|---|
| Scheduled | Weekly / monthly |
| Drift-triggered | On detection |
| Performance-triggered | Below threshold |
| Continuous (online) | Streaming updates |
| Validation gate | Auto-promote only if better |
10. Designing Model Explainability
| Method | Detail |
|---|---|
| SHAP | Per-prediction feature attribution |
| LIME | Local linear approximation |
| Integrated Gradients | Deep models |
| Global importance | Permutation importance |
| Required for | Regulated domains (finance, health) |
11. Designing MLOps Pipeline
| Practice | Detail |
|---|---|
| CI/CD for models | Test, validate, deploy |
| Reproducibility | Versioned data + code + config |
| Automated retraining | Pipeline on trigger |
| Governance | Approval, audit, rollback |
| Tools | Kubeflow, MLflow, SageMaker Pipelines, Vertex AI |
12. Designing GPU Resource Management
| Aspect | Detail |
|---|---|
| K8s GPU scheduling | nvidia.com/gpu resource |
| MIG (A100/H100) | Slice GPU into instances |
| Time-sharing | Multiple pods share GPU |
| Spot GPUs | For training; checkpointing |
| Inference batching | Maximize GPU utilization |
| Quantization | INT8/FP8 for cost reduction |