Architectural notes from building a multi-tenant, AI-driven call centre platform.
A carrier once sent us a one-line integration request: use 10.1.0.0/24, and we'll reach your SIP endpoint at 10.1.0.34.
It looked like a formality. It was actually the most consequential decision in the whole integration, and getting it wrong would have taken down an unrelated Kubernetes cluster in a way that would have been genuinely difficult to diagnose.
That range sat entirely inside the pod CIDR of one of our clusters. The cluster used Azure CNI, so pods and nodes draw addresses from that range dynamically. There was no way to reserve .34. The next scale-up would have handed it to a pod, and from that moment two different machines would have believed they owned the same address. Worse, the platform's own route for that range would always have beaten anything we pointed at the tunnel, so traffic destined for the carrier would have been silently delivered into the cluster and dropped. No error. No alert. Just calls that didn't connect.
We said no and issued a different range. Ten minutes of checking saved a multi-day outage investigation.
That episode is a decent lens on the whole problem space. Voice platforms fail in ways that web platforms don't, and the decisions that matter are rarely the ones that look architectural on a slide.
1. Address planning is a first-class architectural concern
In a normal SaaS, private IP ranges are plumbing. Nobody argues about them.
In a voice platform you interconnect with carriers, and each carrier arrives with their own addressing plan and their own assumptions. You accumulate peers, and every one of them is a chance to collide with a range you're already using, or one you'll want in two years.
So we treat the address plan as a maintained artefact, not an accident. One table, covering every VNet, every cluster CIDR, every service range, across every subscription. Before we accept any carrier prefix, we check it against that table.
Two practices earned their keep.
Reserve ranges you aren't using yet. Our environments don't sit adjacent to each other. The gaps are deliberate: room for a future environment that won't need renumbering to exist.
Know your fallbacks before you need them. If a carrier's plan genuinely collides with ours, we have answers ready: the upper half of the 172.16/12 block, and RFC 6598 carrier-grade NAT space, which enterprise RFC 1918 plans essentially never touch. Having those ready turns a two-week negotiation into a one-email answer.
The failure mode here isn't a crash. It's a routing table that quietly disagrees with your intent, and those are among the most expensive bugs to find.
2. Don't NAT SIP if you can possibly avoid it
Here's the thing that catches teams coming from web infrastructure: SIP carries IP addresses inside the message body.
When a call is set up, the SDP payload says, in effect, send the audio to this address on this port. That address is written by the endpoint. A NAT device rewrites packet headers; it does not, by default, rewrite the addresses inside the payload. So the packets arrive fine and the media goes nowhere.
This is the true cause of most "one-way audio" tickets in the industry. Everything looks healthy. Signalling completes. The call connects. Nobody can hear anybody.
You can solve it: SIP-aware NAT, an ALG, a session border controller rewriting SDP. All of these work. All of them are additional stateful components in the media path that must be scaled, monitored, and debugged at 3am.
We took the other route: design the topology so no address is ever rewritten. We advertise the media server's actual subnet to the carrier as our traffic selector, and the carrier sends signalling to the media server's real address. The tunnel gateway is a pure crypto and routing hop. It never terminates SIP and never appears in SDP.
The result is that SDP is truthful by construction. An entire category of bug simply cannot occur, and there's one less stateful thing in the media path.
The general principle: when a protocol embeds addresses in its payload, spend your architectural budget on making addresses stable rather than on machinery to rewrite them.
3. Carrier interconnect belongs on private transport
We terminate carrier traffic over IPsec rather than exposing SIP to the internet.
Let me be precise about why, because there's a lot of folklore here.
The reason is not a blanket legal prohibition on internet SIP. If you operate in Sri Lanka, the rule that actually exists is narrower and different: the Telecommunications Act No. 25 of 1991, as amended by Act No. 27 of 1996, restricts VoIP used to bypass licensed international telecommunications gateways. That is a licensing and revenue-protection rule about who may terminate international traffic, not a technical mandate about transport. Licensed operators sell regulated SIP trunking to enterprises precisely within that framework. If you're building here, the compliance question to answer is "are we terminating international voice through a licensed gateway?", not "is SIP on the internet allowed?" Talk to a telecoms lawyer, not a network engineer, and check your carrier's licence conditions, which frequently do specify private interconnect contractually.
The engineering reasons stand on their own and are, frankly, sufficient:
- A public SIP port is scanned continuously. Within hours of exposure you'll see registration attempts from everywhere. This isn't hypothetical; it's the baseline condition of the internet.
- Toll fraud is the monetised attack. A compromised SIP endpoint gets used to originate expensive international calls. The loss is measured in real currency and accrues by the minute.
- Media quality is contractual. A private path with an agreed provider is a path you can reason about when you owe someone an SLA.
So the architecture: carrier traffic arrives encrypted, is decrypted at a gateway that does nothing but route, and reaches a media server whose SIP ports accept traffic from the carrier's prefixes and nothing else. After cutover, the SIP surface has no public exposure at all.
Crucially, WebRTC stays public. Browser and mobile clients need to reach the platform directly. Those are different trust boundaries with different threat models, and conflating them costs you either security or usability.
4. Isolation boundaries decide your multi-tenancy story
"Multi-tenant" describes a spectrum, and the honest question is: what is the blast radius of one tenant's worst day?
Our boundaries, from hardest to softest:
Network. Environments live in separate virtual networks that peer to a shared hub, not to each other. Non-transitive peering isn't a limitation here; it's the property we want. Compromising one environment's network doesn't grant a path to another's.
Compute. Tenant workloads are namespaced within a cluster, with resource limits that stop a single tenant's traffic spike from starving its neighbours.
Media. This is the voice-specific one. Media servers are stateful and CPU-bound in ways web servers aren't. A call is a long-lived, latency-sensitive session, not a request you can retry. Media capacity is planned per environment rather than pooled globally.
Identity and secrets. Every tenant's credentials resolve through a central secret store, injected at runtime. Nothing tenant-specific is baked into an image.
The pattern: isolate hardest where recovery is slowest. Restarting a stateless API pod costs nothing. Dropping 200 live calls costs you a customer.
5. Reversibility is an architectural property
The cutover from public SIP to private transport had a genuinely unpleasant characteristic: it rebuilt the media server. Changing the instance's boot configuration is a replacement operation on our cloud, so calls would drop.
Our first design coupled that flip to the tunnel being configured. Elegant, on paper: a single source of truth, with no window where SIP is locked down before the replacement path exists.
It was also wrong, and code review caught it. Coupling them meant you could never test the tunnel before committing to it. You'd build the tunnel and cut over production voice in the same operation, and discover any problem with both hands already full.
We added an explicit override. Now it's two phases: build the tunnel and prove it end to end while SIP still runs publicly and the media server is untouched, then flip, in a window you choose, with one variable.
The broader lesson, and the one I'd most want to pass on: a migration you can't stage is a migration you don't control. Ask of any cutover design: can I test this before I need it, and can I get back? If either answer is no, the design isn't finished, however clean it looks.
6. Spend where the risk is, not where the brochure is
The managed VPN gateway would have cost roughly an order of magnitude more per month than the small burstable VM we used, and none of its scale or dynamic routing features were relevant to a single carrier tunnel.
That's not penny-pinching, it's proportionality. The risk in this integration was never throughput. Modern CPUs do AES in hardware, and a hundred concurrent G.711 calls is under 10 Mbps each way. The risk was address collision, NAT breaking SDP, and an untestable cutover. Every one of those is a design problem that money cannot solve.
Put the budget where the failure modes actually are.
What I'd tell someone starting this
Voice is not HTTP. Sessions are long-lived, latency is perceptible to humans, the protocol embeds addresses in its payload, and failures are often silent rather than loud. Instincts calibrated on stateless web services will mislead you.
Write down the address plan. Then check every integration against it, every time. It is the cheapest possible insurance.
Verify the regulation, don't repeat the folklore. I've heard confident claims about what regulators require that turn out, on inspection, to be a different rule about a different thing. Read the actual statute or ask someone qualified. Build on the engineering rationale. It's usually sufficient on its own, and it doesn't evaporate when someone checks.
Design the rollback before the rollout. Every time.
If you're working on similar problems (carrier interconnect, media scaling, multi-tenant isolation), I'd genuinely like to compare notes.