TechNexus — AI Infrastructure Reference Architectures

Start here

This is not a datacenter project. It is a first step.

Most published AI infrastructure guidance assumes a hyperscaler budget and an H100 purchase order. That guidance is useless in Yangon. The constraints below are what actually shape the design.

Regulator / CISO

"Customer data cannot leave the country, and every inference must be auditable."

Residency rules rule out hosted APIs for core workflows. Sovereignty is a requirement, not a preference — and the strongest argument for local compute.

Head of Infrastructure

"We cannot source current-generation GPUs, and lead times are quoted in quarters."

Supply is the binding constraint, not capital. A design that assumes one specific accelerator never ships. The platform must survive a hardware substitution.

Platform team lead

"We know Docker. Nobody here has run a GPU cluster before."

Driver and CUDA lifecycle is the number one operational fear, and a legitimate one — which is why the platform layer matters more than the hardware.

CEO / CFO

"What does this change about how we work — and when does it pay for itself?"

A capital decision defended with a unit economic, not a benchmark. When renting wins, it wins — which is why GPU-as-a-Service leads the four paths rather than trailing them.

The reasoning I apply first

Every AI application is still three layers on three foundations

Before any hardware conversation, decompose the use case. Strip away the novelty and an AI system is an interface, some logic and stored state — on compute, network and storage. That turns a vague ambition into something sized, buildable and defensible.

Commercially: two of the three layers are things the team already builds. Only the logic layer is new. That turns "we need an AI platform" — unbounded, frightening — into "we need GPU compute behind an interface we already understand". Scopeable, sizeable, fundable.

And every one of them does one of three things

Three patterns cover every AI use case I have ever scoped

A use case that lands in none of these is usually a technology looking for a problem — worth saying before anyone spends money.

Pattern 1 · Volume

Remove repeated human work

The same task performed over and over against different inputs — classifying tickets, extracting fields from documents, drafting the same category of reply. High volume, low variance, enormous cumulative human cost.

Sizing signal: throughput and cost per task. A small routed model usually beats a large one here, and the business case is measured in hours returned per week.

Pattern 2 · Judgement

Support decisions and fact-checking

A human still decides, but faster and better informed — surfacing the relevant policy, cross-checking a claim against source documents, summarising a case file, flagging what looks anomalous. The AI does not act; it hands over evidence.

Sizing signal: retrieval quality and citation grounding, not raw model size. This is where RAG earns its place — and where an ungrounded answer is worse than no answer.

Pattern 3 · Process

Compress operational overhead

Fewer hand-offs, fewer people in the loop, shorter elapsed time. An agent that reads the request, checks the system, performs the safe action and escalates the rest — replacing a chain of four humans and two days with one exchange.

Sizing signal: tool integration and guardrails dominate, not the model. This pattern carries the most risk, because it is the only one that acts.

Decision one · where the compute comes from

Four paths to GPU capacity, same platform above

Entry points, not a single recommendation — chosen by budget, timeline, residency and what can actually be procured. The platform layer is identical across all four, so the hardware decision stays reversible.

Path 1 · Rent

GPU-as-a-Service

Hourly GPU rental from a specialist provider — RunPod, Lambda, Vast.ai class. Container in, endpoint out.

Running the same day; no capex, no procurement cycle
Right answer for pilots, benchmarking and bursty fine-tuning
Data leaves the country — usually disqualifying for regulated workloads
Cost crosses over against owning at sustained utilisation

CommitmentHourly · cancel anytime

Path 2 · Desktop class

Grace-Blackwell developer system

128 GB coherent unified memory, ~1 PFLOP FP4, up to 4 TB NVMe. Pairs over ConnectX for larger models.

Procurable off the shelf — proves the platform now
Pilot and departmental serving; fine-tuning up to 70B class
Data stays on-premise from day one
Becomes the dev/test tier once production lands

Entry pointSingle unit, low five figures

Path 3 · Lowest capex

GPU retrofit into an existing server

One 48–96 GB passive server card into a validated host. Needs PCIe Gen4/5 x16 double-width and PSU headroom.

Cheapest route to on-premise inference
Entry production for one or two quantized models
Shares host CPU, RAM and I/O — must be capped
No high-availability story on a single chassis

Entry pointCard only — validate host first

Path 4 · Production target

Dedicated GPU server

2× enterprise 96 GB-class GPUs, dual 32-core CPUs, 384–512 GB ECC, NVMe RAID, dual 25 GbE, redundant PSU.

Runs 3–5 concurrent models with real headroom
Scales by adding cards before adding servers
Second node makes it an HA pair
The only path with a credible production SLA

CommitmentCapex + facility readiness

Architecture 1 of 6

Infrastructure base — three-tier, with a GPU tier bolted on

A conventional segmented enterprise design: DMZ, application tier, data tier — plus a dedicated GPU Kubernetes cluster on three nodes. Nothing exotic, which is precisely the point. A security team can review this in an afternoon because it looks like every other three-tier estate they have already approved.

North-south traffic Gateway · auth and audit GPU inference path Persistence

Three deliberate decisions. Database outside Kubernetes — stateful workloads on a young cluster is not a risk for a six-person team's first project. GPU nodes in a separate tainted pool, so a runaway deployment can never evict a model that takes four minutes to load. Data tier accepts no DMZ ingress — usually the first thing a security review checks.

Architecture 3 of 6

The GPU platform — one pool, many agents

This diagram answers “what is the GPU tier actually made of?” — the layering inside the box the previous diagram drew. The commercial argument lives here: one GPU investment does not buy a chatbot, it buys a serving platform hosting a fleet of specialist agents. Each new agent costs prompt engineering and retrieval work, not more hardware.

Why this wins the budget conversation. The expensive layer is the bottom one, and it is shared. Agent one justifies the hardware; agents two through six are configuration. You are not buying a help desk bot — you are buying the platform every later use case runs on.

Architecture 5 of 6

Guardrails — defense in depth, seven layers

Not in-process versus gateway — each layer owns the trust boundary it protects. For any AI service touching regulated data the question is placement, never whether.

"Model alignment is a floor, not a policy engine"

With a modified open-weight model that floor is missing entirely. Fine-tuning an open model for sovereignty removes L0 without anyone noticing — which makes L2 through L5 mandatory, not optional.

"The gateway is the only place you can prove compliance"

In-process rails are the only place you can understand the conversation. You need both — an application controls its own logs, so app-level audit alone never satisfies a regulator.

Decision two · sizing

What actually fits — before you buy anything

Frontier open models arrive monthly, each with a headline "runs on 8× H200". That figure is weights only, at batch size one. Taken at face value it produces hardware that benchmarks beautifully and falls over in week two.

Open model	Total / active	Architecture	Licence	Modality	API $/1M in·out	Published deployment
Inkling · Thinking Machines	975B / 41B	MoE + SWA-GA	Apache 2.0	text · image · audio	TBD	8× H200
Nemotron 3 Ultra · NVIDIA	550B / 55B	Mamba-MoE hybrid	OpenMDW	text	$0.50 · $2.20	4× B200
GLM-5.2 · Z.ai	753B / 40B	MoE + DSA	MIT	text	$1.40 · $4.40	8× H200
Kimi K2.6 · Moonshot	1T / 32B	MoE + MLA	MIT*	text · image · video	$0.70 · $3.00	8× H200
DeepSeek-V4 Pro · DeepSeek	1.6T / 49B	MoE + CSA-HCA	MIT	text	$0.44 · $0.87	8× H200
Qwen3.6-35B · Alibaba	35B / 3B	MoE + DeltaNet	Apache 2.0	text · image	$0.14 · $1.00	one 24 GB GPU
Gemma 4 31B · Google	31B dense	Dense + SWA hybrid	Apache 2.0	text · image · audio	$0.12 · $0.35	one gaming GPU
MiniMax M3 · MiniMax	428B / 23B	MoE + MSA	restricted	text · image · video	$0.30 · $1.20	8× H200

Specifications as published by ByteByteGo. ⚠️ Verify against the model card before quoting — this segment moves monthly and licences change.

What that last column does not tell you

"Runs on 8× H200" is a weights-only claim at batch size one

Three things it silently omits — and each one is the difference between a working deployment and an expensive one that doesn't serve anybody.

Omission 1 · users

No concurrency budget

The figure assumes one request in flight. Each additional user adds a full KV allocation. At 32 users and 8k context a 70B model needs roughly 80 GB of KV cache — more than its own FP8 weights.

Omission 2 · context

No context budget

KV scales linearly with context. Moving one user from 8k to 128k multiplies their footprint by sixteen. A model that "fits on one GPU" at 4k does not at 128k — and the figure never says which was assumed.

Omission 3 · MoE

Active params are not resident params

All expert weights must sit in VRAM even though only a fraction activate per token. DeepSeek-V4 Pro is "49B active" — but it is 1.6T resident. Active params tell you compute cost; total params tell you how much memory you must buy.

Published claim, checked	Resident weights	+ KV at 32 users · 8k	vs. 8× H200 = 1,128 GB	Verdict
DeepSeek-V4 Pro · 1.6T	FP8 1,600 GB FP4 800 GB	+61 GB	FP4 → 990 GB · 88%	FP4 only. FP8 needs 12 GPUs for weights alone. Room for ~95 users at 8k before headroom is gone
Kimi K2.6 · 1T	FP8 1,000 GB FP4 500 GB	+61 GB	FP8 → 1,220 GB · over	FP8 does not fit once users are added. The claim silently requires FP4
GLM-5.2 · 753B	FP8 753 GB	+92 GB	FP8 → 972 GB · 86%	Works at FP8 — the most honest of the 8× H200 claims. But no room to grow
Qwen3.6-35B · "one 24 GB GPU"	FP4 17.5 GB	+48 GB	needs 75 GB	True for one user. At 32 users it will not fit a 48 GB card, let alone 24

Every figure came from the planner below at 32 users and 8k context, so tool and table cannot drift apart. Four minutes of arithmetic that has changed the hardware line on more than one proposal.

Planning shorthand

By model class, once you stop reading marketing

Model class	Weights (Q4 / FP8)	Typical role	Realistically serves
4–14B SLM	4–12 GB	Routing, classification, extraction, drafts	High concurrency on one card
27–35B / MoE	18–35 GB	Primary chat and agent reasoning	Dozens of users on one 96 GB card
70B class	40–75 GB	Expert and deep-reasoning agents	2 GPUs for a real concurrency budget
120B+ MoE	~65–80 GB active tier	Strategy, long-context analysis	Resident size decides, not active size
500B+ frontier	500 GB+ resident	Frontier capability on-prem	Multi-GPU server minimum — 8 cards and up

KV cache per token = 2 × layers × kv_heads × head_dim × bytes_per_element

70B-class model — 80 layers, GQA 8 KV heads, head_dim 128, FP16:
2 × 80 × 8 × 128 × 2 = ~320 KB per token

8k context → 320 KB × 8,192 ≈ 2.6 GB per sequence
32 concurrent → ≈ 80 GB of KV cache alone
+ FP8 weights (70 GB) = ~150 GB → exceeds a single 141 GB accelerator

Grouped-query attention is what makes this affordable — full multi-head would be 8× larger. That is why MLA, DSA and the other variants above exist: KV-cache economics, not accuracy features.

Prefill is compute-bound

Processes the whole prompt in parallel. Determines time-to-first-token — what users perceive as responsiveness.

Decode is bandwidth-bound

One token at a time, limited by memory bandwidth over model size. Determines inter-token latency — perceived as typing speed.

Batching raises throughput, not latency

Continuous batching keeps the GPU fed as sequences finish at different times. It does not make any single user's response faster.

Interactive · capacity planner

Will it fit? Drag the sliders and find out

Set the model, precision, context and concurrency you actually need. The planner returns what the marketing figure left out.

GPU memory planner resident weights + hot KV + headroom vs. VRAM · MoE counts all experts · KV tiering supported

Model 70B dense

Precision FP8 / INT8

Context length 8,192

Concurrent users 32

KV cache tier HBM only

Accelerator H200 NVL · 141 GB

GPU count 1 GPU · 1 server

0 GB——

Weights

—

KV cache

—

Total needed

—

incl. 15% headroom

Available VRAM

—

HBM only

Everything resident

Every cache sits in GPU memory — fastest, and the assumption behind every "8× H200" claim. Also the most expensive way to buy concurrency: HBM prices for cache belonging to users who are reading, not generating.

+ CPU RAM

Offload to host memory

LMCache and vLLM CPU offloading tier idle sessions to host RAM, paging back on the next turn. Most chat sessions are idle between turns, so only a fraction stays resident. ~512 GB per server at PCIe speed — 75× slower than HBM, but only paid on resume.

+ NVMe

Offload to storage

Multi-terabyte pools on local NVMe, ideally over GPUDirect Storage so page-back skips the CPU bounce buffer. Far more capacity, bigger penalty. Right for long agent sessions and histories that must survive but are rarely hot.

The conclusion · making the case

Cost against outcome — the four numbers that decide it

Return is not measured in GPU utilisation. It is measured in work completed, mistakes avoided, people freed and time saved. Pick one process, measure it before, measure it after, and the case writes itself.

Because four people are in the room

The same project answers four different questions

A case built only for finance dies in the technical review. A case built only for engineering never gets funded. Each executive is asking something different, and each needs a different number.

Who	What they are actually asking	The number that answers it
CEO	Does this change what we can offer, or only what it costs us? And what happens if a competitor does it first?	Capacity created · new capability · sovereignty risk removed
CTO	Can we actually run this, and what does it break? Who maintains it when the vendor leaves?	Operational burden · integration surface · skills gap · exit cost
COO	Does the work genuinely change, or do my people now supervise a machine as well as doing the job?	Cycle time · hand-offs removed · people per process
CFO	When does it pay back, and what is the downside if adoption is half what was promised?	Cost per task before vs after · payback period · sensitivity at 40% adoption
CISO / Risk	What new exposure does this create, and can we prove control to a regulator?	Guardrail layers in place · audit completeness · data residency

Pattern 1 · Volume

Work per unit of effort

tasks/month × minutes saved × loaded hourly cost × adoption rate. The only pattern with a clean financial number — and the one where adoption rate quietly decides everything.

Pattern 2 · Judgement

Mistakes avoided

error rate before vs after × cost per error. Banks already price their error rates, so this is usually easier to source than people expect. Include errors introduced by the system — that is the honest version.

Pattern 3 · Process

Manpower and speed

hand-offs removed × cycle time saved. Value shows up as elapsed time and people redeployed, not people removed — and saying "redeployed" out loud is what keeps the operations team on side.

First contact

Where a conversation usually starts

Most engagements begin with one of three questions — usually from a CTO or COO, sometimes from a CEO who has been asked by the board. None require a budget to ask.

30 minutes

"Is this even feasible for us?"

A use case, the residency constraints, and a straight answer on whether AI is the right tool — including when it is not. No deck.

Half a day

"What would it cost?"

Pattern identified, model class sized, the four procurement paths costed against your token volume, and a defensible cost per million tokens.

2–4 weeks

"Prove it on our data"

A pilot on rented or procurable GPU, benchmarked on your real prompts — tokens/sec, VRAM under concurrency, cost per token. Assumptions replaced with measurements before capital is committed.