Reference architectures · Solutions Architecture · AI Infrastructure

How I design on-premises AI datacenters for enterprises in Myanmar

Sovereign GPU infrastructure for banks, conglomerates and regulated industries — where the data cannot leave the building, H100-class supply is uncertain, and the operations team is six people. These are the reference designs I build from.

0
GPU procurement paths, costed — because supply, not budget, is the real constraint
0
API transactions per day sustained on production Kubernetes
0
Annual uptime on regulated multi-cloud Kubernetes workloads
0
Cloud, Kubernetes and GPU platform engineering · CKA · CKS · MCT
Why on-premises, why here

In Myanmar, "just use the cloud" is not an available answer

Every on-prem AI engagement I run starts from the same four constraints. The architecture is downstream of them.

Regulator / CISO
"Customer data cannot leave the country, and we must be able to audit every inference."

Banking and telco data residency rules make hosted LLM APIs a non-starter for core workflows. Sovereignty is the requirement, not a preference.

Head of Infrastructure
"We can't source H100s through our channels, and lead times are quoted in quarters."

Supply, not capital, is the binding constraint. A design that assumes one specific GPU is a design that doesn't ship.

Platform team lead
"We know Docker. We don't have anyone who has run a GPU cluster before."

Six-person teams. Driver and CUDA lifecycle management is the number-one operational fear, and it is entirely legitimate.

CFO
"What does this actually cost per token, and when does it beat the API?"

On-prem AI is a capital decision that must be defended with a unit economic, not a benchmark screenshot.

Decision one · hardware

Three GPU paths, same platform on top

I don't present a single recommendation for GPU hardware. I present entry points chosen by budget, timeline and what is actually procurable — with an identical software platform above them, so the decision can change without rework.

Path A · Production target

Dedicated AI server

2× enterprise GPUs (96 GB class), dual EPYC/Xeon 32c+, 384–512 GB ECC DDR5, NVMe RAID1 boot + NVMe data, dual 25 GbE, redundant PSU ~2.5 kW

  • Runs 3–5 concurrent models with headroom
  • Scales by adding cards, not servers
  • Second node → HA pair, zero-downtime upgrades
  • No impact on existing workloads — isolated
Path B · Lowest capex

GPU retrofit into an existing server

Single 48–96 GB passive server card into a validated host. Requires PCIe Gen4/5 x16 double-width, +350–600 W PSU headroom, front-to-back airflow, Above-4G decoding

  • Entry production for one or two quantized models
  • Shares host CPU, RAM and I/O — must be capped
  • No HA story on a single chassis
  • Validate vendor thermal support before purchase
Path C · Available today

Desktop-class AI superchip

Grace-Blackwell class, 128 GB coherent unified memory, ~1 PFLOP FP4, up to 4 TB NVMe, pairs over ConnectX for larger models

  • Procurable off the shelf — proves the platform now
  • Pilot and departmental serving, fine-tuning
  • Becomes the dev/test tier once Path A lands
  • Fully isolated from production infrastructure
The reference design

Four independent tiers on one fabric

Web, GPU, data and storage each scale without touching the others. This is the diagram I draw on a whiteboard in the first architecture workshop.

CHANNELS Web / chat UIinternal users Core bankingAPIs · service bus Workflow / RPAticketing · ITSM BI & analystsdashboards Ingress · TLS · WAFrate limit · mTLS LLM gatewayauth · routing · quotas · audit WEB & BACKEND CLUSTER · RKE2 FrontendReact · static Backend APIsbusiness logic Agent orchestratortools · RAG · guardrails Cilium eBPF CNI · Gateway API · cert-manager · Kyverno policyhorizontal pod autoscaling — grow by adding worker nodes GPU AI CLUSTER · RKE2 ON GPU NODES vLLM · chat model27–35B MoE · FP8 vLLM · expert model70B quantized Embeddings+ vision / OCR NVIDIA GPU Operator — driver · CUDA toolkit · device plugin · DCGM · MIGgrow by adding GPU nodes or cards — driver lifecycle fully automated DATA CLUSTER · HIGH AVAILABILITY PostgreSQL HAstreaming replication Qdrantvector store · RAG ClickHouse · Redisaudit trail · cache Scale reads with replicas · point-in-time recovery · encrypted at rest STORAGE · DISTRIBUTED OBJECT + BLOCK MinIOerasure coded Model storeversioned weights Longhornblock PVs NVMelocal scratch Datasets · checkpoints · artefacts · backup — add nodes for capacity PLATFORM OPERATIONS · SPANS EVERY TIER ProvisioningAnsible · PXE · images GitOps deliveryArgo CD · Git as truth ObservabilityPrometheus · Grafana · Loki GPU telemetryDCGM exporter · alerts LLM tracingLangfuse · cost · audit SecurityVault · Trivy · CIS CIS-hardened Kubernetes · RBAC and namespace isolation · MIG partitioning for multi-tenant inference · full audit trail for regulator review
User request · application path Gateway · auth, routing, audit GPU inference · token generation Retrieval & persistence

Live trace: watch a single request enter through the gateway, retrieve context from the vector store, run inference on the GPU tier, and return — while model weights load from object storage and every call is written to the audit trail. Scales without rework: add web nodes for traffic, GPU nodes for models, database replicas for reads, object nodes for capacity.

Decision two · sizing

What actually fits — before you buy anything

Model weights are the number everyone quotes. KV cache is the number that decides whether the deployment works. I size both, then validate against the customer's real traffic in the pilot.

Model classWeights (Q4 / FP8)Typical roleFits on
4–14B SLM4–12 GBRouting, classification, extraction, draftsAny path
27–35B / MoE18–35 GBPrimary chat and agent reasoningAny path
70B class40–75 GBExpert and deep-reasoning agents96 GB card · 128 GB unified
120B+ MoE~65–80 GBStrategy, long-context analysis96 GB card · 128 GB unified
200B+ frontier100 GB+Future headroomPaired units · multi-GPU server
KV cache per token = 2 × layers × kv_heads × head_dim × bytes_per_element

70B-class model — 80 layers, GQA 8 KV heads, head_dim 128, FP16:
  2 × 80 × 8 × 128 × 2 = ~320 KB per token

8k context    → 320 KB × 8,192 ≈ 2.6 GB per sequence
32 concurrent → ≈ 84 GB of KV cache alone
+ FP8 weights (70 GB) = ~154 GB → needs tensor parallelism across two GPUs

Grouped-query attention is what makes this affordable. With full multi-head attention the cache would be 8× larger.

What that actually looks like on a single 141 GB accelerator

70B model, FP8 weights, 8k context, 32 concurrent users — drawn to scale

Weights · 70 GB KV cache · 84 GB
0 GB77 GB154 GB required
Model weights — fixed KV cache — grows with users × context
Doesn't fit. 154 GB required against 141 GB available — and that's before activation headroom. This deployment needs tensor parallelism across two accelerators, or a shorter context budget. The KV cache is bigger than the model. That is the number customers never plan for, and it is why I size before anyone signs a purchase order.

Prefill is compute-bound

Processes the whole prompt in parallel. Determines time-to-first-token — the metric users perceive as "responsiveness."

Decode is bandwidth-bound

One token at a time, limited by memory bandwidth over model size. Determines inter-token latency — perceived as "typing speed."

Batching raises throughput, not latency

Continuous batching keeps the GPU fed as sequences finish at different times. It does not make any single user's response faster.

Decision three · the serving platform

Kubernetes with the GPU Operator, not VMs with plain Docker

Almost every customer arrives planning two VMs running Docker, and almost every one raises driver management as their top operational worry. Those two facts resolve each other.

ConcernTwo VMs + plain DockerRKE2 + NVIDIA GPU Operator Recommended
Driver & CUDA lifecycleManual version matrix per VM — the exact risk the customer namedGPU Operator installs, version-matches and upgrades the whole NVIDIA stack on every node, automatically
FailoverDIY scripts. No orchestrationSelf-healing — failed model pods restart, node loss reschedules
GPU sharingManual, no isolationMIG for hardware-isolated multi-tenancy; time-slicing for dev
ReproducibilitySnowflake servers, config drift, hard to auditDeclarative and version-controlled — rebuildable, auditable
ScalingDoesn't scale past the first nodeAdd GPU nodes or cards; scheduler places workloads
Day-one learning curveFamiliar — fast first deploymentSteeper — covered by runbooks, enablement and handover

Docker remains the container runtime underneath. Nothing the team already knows is thrown away — the platform automates what two VMs would force them to script by hand.

vLLM for production

Continuous batching and PagedAttention, OpenAI-compatible API, AWQ/GPTQ/FP8 and multi-LoRA. The right default for an LLM-only fleet with a small operations team.

Triton when the zoo is mixed

Multi-framework serving with dynamic batching, concurrent model execution and server-side ensembles. It earns its complexity when ONNX or TensorFlow models must co-serve with LLMs.

MIG for multi-tenancy

Hardware partitioning with dedicated SMs, cache, memory and bandwidth. The only sharing mechanism I put under a production SLA — time-slicing and MPS give no real isolation.

How the engagement runs

Assessment to production, in four phases

The pilot exists to replace assumptions with the customer's own numbers before capital is committed.

Phase 1Assess

Environment review, workload discovery, data-residency and compliance constraints, existing stack audit, success criteria agreed in writing.

Phase 2Pilot / POV

Procurable GPU path stood up. Benchmark the customer's real prompts — tokens per second, VRAM under concurrency, cost per million tokens.

Phase 3Production build

HA cluster, CIS hardening, GitOps delivery, observability, MIG partitioning, governance and audit layer, disaster recovery tested.

Phase 4Handover

Runbooks, architecture documentation, and hands-on enablement so the customer's own engineers operate it. Delivery is not done until they can.