Most organizations that depend on LLMs now use more than one provider, whether for availability, capability, price, data residency or negotiating leverage. Doing that well is an architecture problem. APIs differ in message formats, tool-call schemas, structured-output guarantees, caching semantics and rate-limit behavior; a fallback model that is technically available can still break downstream parsers or change answers users rely on; and cost controls only work if every call is attributed to a tenant, feature and budget.
This expert course builds a reference architecture layer by layer. You will design a canonical request schema and adapters, then routing strategies from static rules through classifier routing and cost-quality cascades, constrained by residency, tenant policy and context length. You will build a failure taxonomy and map each failure to the right response (retry, hedge, circuit-break, fail over or degrade), including mid-stream and mid-tool-call failures, and gate every fallback model on regression evals. You will plan capacity across committed options such as Azure provisioned throughput, Bedrock reserved and priority tiers and Google provisioned throughput, and enforce budgets, quotas, caching and batch offload. You will compare gateways (LiteLLM, Portkey, OpenRouter, cloud-native gateways and build-your-own), instrument with the OpenTelemetry GenAI semantic conventions (still marked Development rather than Stable as of October 2026, so attribute names can change), and finish with contracts and exit plans, outage game days and a hypothetical capstone design for a regulated enterprise. Product facts are as of October 2026.