Managed model-serving endpoints with autoscaling, A/B testing, and integrated quota.

Endpoint types¶

Type	Use	Scale-to-zero
realtime	Synchronous HTTP, low-latency	Yes (optional)
streaming	Token-streaming for LLM completions	No (warm pool)
batch	Async, queue-based, large input	N/A
serverless-async	Async, callback-based	Yes

IAM¶

Role	Can do
inference.viewer	List endpoints, view metrics
inference.invoker	Call endpoints (via API token)
inference.deployer	Create / update endpoints
inference.admin	Above + quota changes, model registry management

Model registry¶

Endpoints reference a model version in the registry — not a raw image:

bash cd inference model register \ --name acme-summarizer \ --version v1.4 \ --artifact s3://acme-models/summarizer-v1.4/ \ --framework pytorch \ --runtime cuda12.6

The model version is immutable. Deploys reference acme-summarizer:v1.4. Promotion = create a new version, point endpoint at it.

Endpoint config¶

bash cd inference endpoint create \ --name acme-summarize-prod \ --model acme-summarizer:v1.4 \ --instance-type gpu-a10-1x \ --min-replicas 2 --max-replicas 20 \ --target-concurrency 8 \ --timeout 30s

Cost shape¶

Billed per replica-hour while warm. Scale-to-zero possible but cold-start = model load time (often 10–30 s for LLMs).

For latency-sensitive endpoints, min-replicas >= 1.

Audit¶

Every inference call logged (request-id, principal, latency, status). Optional input/output logging (for accuracy debugging) — usually disabled in prod due to PII.

Related¶

• Inference — Operation
• Inference — Troubleshooting
• LLM-as-a-Service

Metrics¶

Metric	Healthy	Alert
inference.requests_per_sec	varies
inference.latency_ms p95	within SLO	breach
inference.error_rate	< 0.1%	> 1%
inference.gpu_utilization_pct	60–80%	< 30% (over-provisioned) or > 90% (under)
inference.queue_depth	< target	spikes (under-capacity)
inference.replica_count	scales with load	stuck at max

Autoscaling tuning¶

target-concurrency is the knob. Set to the per-replica throughput at which latency is acceptable:

bash cd inference endpoint update --name acme-summarize-prod --target-concurrency 6

Lower → more replicas, lower latency, more cost. Higher → fewer replicas, higher latency, lower cost.

A/B and canary deploys¶

```bash

Deploy v1.5 alongside v1.4¶

cd inference endpoint update --name acme-summarize-prod \ --traffic-split 'v1.4=90,v1.5=10'

Promote if metrics look good¶

cd inference endpoint update --name acme-summarize-prod \ --traffic-split 'v1.5=100' ```

A/B logs include the model-version label, so dashboards split metrics correctly.

Quantization and optimization¶

For LLMs / vision models, often-applied optimizations:

• INT8 / FP8 quantization (TensorRT, vLLM) — 2-4× throughput, minimal accuracy loss
• Continuous batching (vLLM, TGI) — 5-10× throughput for LLMs
• Speculative decoding — 1.5-2× for some workloads

Cloud Digit ships pre-optimized servers (vLLM, Triton); use those rather than raw model.predict().

Cold start¶

For scale-to-zero or new replica spin-up:

• Model load from S3 → GPU: 5–30 s typical (depends on model size)
• For instant scale-from-zero, min-replicas >= 1
• Snapshot-based fast scale supported on H100 SKUs (sub-second from snapshot)

Quotas¶

GPU inference shares quota with training. Plan separately if both happen in the same project:

bash cd inference quota set --project acme --instance-type gpu-a10-1x --max-replicas 20

Related¶

• Inference — Administration
• Inference — Troubleshooting

High p95 latency¶

Symptom	Likely cause
All requests slow	Model load misconfig (CPU instead of GPU)
Most fast, some slow	Queue depth spikes; under-provisioned
Slow during scale-up	Cold start of new replica
Slow only on certain inputs	Input-length-sensitive (LLM with long context)

bash cd inference logs --endpoint acme-summarize-prod --filter "latency>1000ms"

Errors after model deploy¶

Symptoms: 5xx rate spikes right after a new model version went live.

• Model artifact corrupted in S3 — check checksum
• Framework/runtime mismatch (PyTorch 2.5 model on PyTorch 2.4 server)
• Quantization broke (bug in the optimization pipeline)

Roll back:

bash cd inference endpoint update --name acme-summarize-prod \ --traffic-split 'v1.4=100,v1.5=0'

GPU under-utilized despite load¶

Symptom	Cause
GPU util 30% with full queue	Batch size too small; enable continuous batching
GPU util fluctuates 0-100%	Dataloader bottleneck (rare for inference)
GPU util 95% but throughput low	Model is compute-bound — quantize or shard

vLLM, TGI, Triton all support dynamic batching — enable in the server config.

Replica scale-up stuck¶

inference.replica_count not climbing despite high queue:

• Max replicas reached
• GPU capacity exhausted in the project / region — request quota
• Cold start failing (model load fails) — check replica logs

bash cd inference replicas --endpoint acme-summarize-prod

Memory leak¶

Replicas die after N requests:

• Memory leak in the model server (vLLM, custom) — find via memory metrics over time
• Caching too aggressively (KV cache for LLMs) — tune gpu_memory_utilization
• Restart policy: set --max-requests-per-replica 10000 to force periodic restart

Quantized model accuracy regression¶

You quantized a model and quality degraded:

• INT4 / FP4 quantization is more aggressive; some models can't tolerate
• Calibration dataset wasn't representative
• Backbone of the model is quantization-sensitive

Run a quality eval suite on the quantized version before promoting.

API token rejected¶

ERROR 401: invalid token

• Token expired (TTL ≤ 90 days)
• Token wrong scope (issued for a different project)
• Wrong endpoint URL (region mismatch — inference endpoints are regional)

Related¶

• Inference — Administration
• Inference — Operation