Streaming a Live Map to Every Client

01A map that never reloads

A map is only useful if it's current. The moment a new place, post, or marker appears, every open map should show it, with no refresh, no poll, no full re-fetch. Doing that for a dataset of over a million features, across many concurrent viewers, is a streaming problem more than a rendering one.

Atlas is the service that solves it for Layers. It sits as a read-optimized, in-memory streaming tier in front of MongoDB: on connect, a browser receives the tenant's features in chunks, and from then on a live push for every change. The database is the source of truth, but no client read ever touches it. Reads are served entirely from RAM, and the socket stays open.

02Architecture at a glance

Atlas is a single Go binary. MongoDB feeds it two ways, a bulk read on boot and a live change stream thereafter, into an in-memory engine partitioned by tenant. The engine emits every change to a hub, which fans it out to that tenant's connected clients. Google Cloud Storage holds periodic binary snapshots so the process can restart in seconds instead of re-reading the database.

03The in-memory engine

The engine is the datastore, and its shape is simple: a single geodata collection of partitions keyed by tenant, each holding a slice of documents, all guarded by one sync.RWMutex. A document wraps a GeoJSON feature alongside its cached encoded bytes.

It stays fresh without polling. On startup the engine hydrates from snapshots (§7); if it comes up empty, it falls back to a one-time MongoDB read of the tenant's full corpus, over a million recent, visible locations. A background goroutine then opens a MongoDB change stream, a live feed of inserts and updates, and applies each event to the right partition with UpdateLookup so the full document travels with the change. Reads never hit Mongo again; the socket does.

04Encode once, send to everyone

The key design decision in Atlas is where serialization happens. Instead of marshaling a feature separately for each recipient, every document is encoded to MessagePack exactly once, at ingest, and its bytes are cached on the document (RawMsgPack). When a change is broadcast, those pre-encoded bytes are written verbatim into every client's stream.

A hand-rolled envelope encoder writes just the small response header (action, total, event, and the FeatureCollection wrapper), then appends the cached feature bytes directly after a length prefix. No per-client re-marshal, no intermediate objects. The cost of fanning a change out to N clients is one encode plus N buffer copies, not N encodes. MessagePack is what makes this splice-able: a compact, length-prefixed binary format you can concatenate without re-parsing.

05The hub: broadcasting without blocking

The hub is the centerpiece. It follows an actor model: all mutable subscription state is owned by exactly one goroutine, Hub.Run, and every other goroutine interacts with it only by sending on channels: Subscribe, Unsubscribe, Broadcast. Single ownership means there is no lock to contend on and no interleaving to reason about. The hub processes one intent at a time.

Each connection gets a buffered channel (depth 256) and a dedicated writer goroutine. Subscription is a handshake: the caller sends a request and blocks until Run has created its channel and handed it back, so the connection is registered before any broadcast can race it.

The broadcast is a non-blocking send with a drop:

// Hub.Run fans one event out to every subscriber of a tenant.
for _, ch := range partition.All() {
    select {
    case ch <- event:
        // delivered
    default:
        // buffer (256) is full, drop for this one slow
        // client rather than block the hub for everyone.
    }
}

If a client's buffer is full, its update is dropped instead of blocking Run. Because the hub never waits on a slow consumer, one wedged connection can't stall fan-out to the rest, and the hub can't deadlock behind a stuck writer. It trades a guarantee of delivery for a guarantee of liveness. For a live map that's the right call: a dropped frame is fixed by the next update, but a frozen hub freezes everyone.

A deadlock test covers this. It subscribes a client that never reads, fires 300 events at its 256-deep buffer, and asserts that the broadcast still completes in time and that a brand-new subscriber is still serviced. Without the default drop, the 257th send would block Run forever and freeze the whole service, so the test exists to keep that from regressing. Companion tests exercise 100,000 concurrent clients and rapid subscribe/unsubscribe churn under Go's race detector.

06The wire protocol

One WebSocket per client, compressed, binary MessagePack in both directions. The client opens the socket and sends a small request; the server answers with feature frames and, from then on, unsolicited live updates.

// client → server
{ event: "InitialLoad", chunkSize: 250 }

// server → client
{
  action:   "Create",
  total:    <feature count>,
  event:    "InitialLoad" | "RealtimeUpdate",
  document: { type: "FeatureCollection", features: [ … ] }
}

The initial load is chunked and flow-controlled by hand: the server sends a chunk of features, then waits on an acknowledgement from its own writer before preparing the next. That bounds memory and lets it reuse buffers safely under load. Once the client is caught up, realtime updates arrive unprompted as each MongoDB change flows engine → hub → socket. A keepalive ping every 5 seconds holds the connection open.

07Fast restarts

Because the entire working set lives in RAM, a naive restart would mean re-reading well over a million documents from MongoDB before the service could serve anyone. Atlas avoids that with snapshots: the engine periodically serializes each partition to a binary blob in Google Cloud Storage, chunked at 50,000 documents, one goroutine per partition.

On boot it does the reverse. It hydrates first, loading snapshots back into memory with a concurrency limit of 20, and only touches MongoDB if the snapshots are empty. A cold start becomes a fast parallel blob read instead of a database scan, and the primary database is spared the load. An admin endpoint can trigger a snapshot on demand.

08Multi-tenant access control

Atlas is multi-tenant, and isolation is enforced at the door. Three layers gate a connection:

• Origin allowlist. The WebSocket upgrader only accepts connections from known Layers domains.
• Tenant resolution. The request host (e.g. a tenant subdomain) is looked up against the instances collection; an unknown host never gets a socket.
• Membership. For a private tenant, a signed passport cookie carries a nested JWT that resolves to a user, and that user's joined tenants must include this one.

The partitioning reinforces this. Because the engine and hub are split by tenant, a connection is only ever wired to its own partition's channel; no code path fans one tenant's features to another's clients.

09Scale & numbers

The performance model is features per second: because a feature is encoded once and copied to many sockets, aggregate delivered throughput scales with connection count, and the in-repo load generator measures exactly that: total features received across thousands of simultaneous connections. In production, Atlas has streamed a tenant's full set of over a million points to a fresh client in roughly five seconds.

The honest ceiling is that Atlas is scale-up, not scale-out: state is process-local, so it runs as a single stateful replica. That keeps the design simple and strongly consistent, at the cost of a per-node memory bound and no horizontal spread. It's a fair trade for a service whose whole advantage is holding everything in one place, in memory.

10Roadmap: scaling out

The single-replica ceiling from the previous section is the most interesting thing left to solve, and the current design points straight at the answer. Because state is already partitioned by tenant, the tenant is a natural shard key: instead of one node holding everyone, a fleet of Atlas nodes can each own a disjoint subset of tenants.

A thin routing tier would map each tenant to the node that owns it, via a consistent hash ring or a small coordination service, and pin every WebSocket upgrade for that tenant to the same node. The property that matters is that a tenant lives entirely on one node, so its fan-out stays in-process. The encode-once broadcast model carries over unchanged, with no cross-node message bus to add latency or a delivery problem.

The snapshot machinery from §7 is what makes this cheap to operate. Because any tenant's partition can be rehydrated on any node straight from GCS, moving a tenant, whether to rebalance a hot node or fail over a lost one, is a matter of re-pointing new connections and hydrating the target, not a bespoke migration. Horizontal scale and high availability then fall out of the same mechanism: the per-node memory bound disappears, and no single node stays a single point of failure.

11Stack