Detecting Trends at the Inflection Point

01The inflection point

Most trend analytics are retrospective. By the time a dashboard shows a spike, the trend has already crested. The interesting window, where a signal is real but not yet obvious, has closed. M.A.C.H.I.N.A is built around a single premise: the valuable moment is the inflection point, the transition from noise to signal, and detecting it requires watching velocity and semantics together, in real time.

Two observations shape the whole design. First, a trend is a cluster: many people independently producing semantically similar content in a short window. Second, trends are born locally, inside a specific community, topic, or region, long before they register in any global average. A global view hides early signals by drowning them in aggregate volume. So rather than a monolithic batch pipeline, M.A.C.H.I.N.A runs as a fleet of isolated, per-segment stream processors, each maintaining its own live memory of what "normal" looks like for its slice of the world.

02Architecture at a glance

The system ships as a single Rust crate exposing three binaries (a gateway, a worker, and an aggregator) that compose into a horizontally-sharded cluster. Events flow through a routing tier into per-partition actors, which lean on two purpose-built datastores and escalate confirmed anomalies to a cross-partition synthesis tier.

The remainder of this paper walks the path a single event takes: how it is made durable on arrival (§3), how it is routed to an isolated actor (§4), how that actor turns a stream of vectors into clusters (§5) and scores them (§6), how the cluster of nodes stays balanced under load (§7), and how local anomalies are fused into systemic ones (§8).

03Ingestion & durability

The ingest path has one hard requirement: an accepted event must never be lost, even if the process crashes mid-spike. M.A.C.H.I.N.A meets it at the ingest node itself, with a local-first write-ahead log, rather than relying on an upstream queue to hold events safely.

Every worker's ingestion boundary is an Axum HTTP handler backed by a Sled embedded key-value store used as a write-ahead log. The ordering is the important part:

The event is flushed to the WAL before the request is acknowledged, so a crash mid-spike can never lose an accepted event. On boot, a log monitor sweeps the WAL and replays any unacknowledged entries to re-hydrate the in-memory partition actors, and only then opens the HTTP port. Once ClickHouse confirms a batch has been persisted, the corresponding WAL entries are truncated to reclaim disk.

04The partitioned actor model

Behind the ingestion boundary sits the Pulse Dispatcher. It owns a concurrent map of partition actors and routes each event to the actor that owns its partition_id (a tenant, topic, region, or community). An unrecognized partition spawns a new actor: a dedicated Tokio task with its own isolated "hot state."

This is the core concurrency decision. Because each partition's rolling window of events and vectors lives inside a single task, there is no shared mutable state across partitions and therefore no lock contention on the hot path. Thousands of independent segments can run on one node without fighting over a global mutex. It also matches the domain: isolation is what lets a small community's early signal survive instead of being averaged away.

Each actor enforces bounded memory with sliding rolling windows (capped counts of the most recent records and events), truncating oldest-first as new data arrives. Every ~60 seconds the actor runs its analysis loop over the current window; that loop is the subject of the next two sections.

05The clustering pipeline

Clustering 1536-dimensional embeddings is expensive, and it has to finish inside a 60-second budget while scaling toward hundreds of thousands of vectors per partition. Running HDBSCAN directly on raw 1536-D vectors caps out around ~10k points at 30–60 s per tick. The pipeline earns its speed with two ideas: an adaptive nearest-neighbor stage and an aggressive dimensionality reduction before the density clustering.

Adaptive nearest neighbors

The neighbor graph is built one of two ways depending on window size. Below ~25,000 vectors, M.A.C.H.I.N.A computes an exact brute-force cosine graph, parallelized across every pair on a Rayon pool. Above the threshold it switches to an approximate HNSW index (hnsw_rs) with parallel insert and search, trading exactness for O(N log N) scaling.

Choosing the "worse" O(N²) algorithm at small N is intentional. Below the crossover (empirically ~22,000 points at 1536-D), brute force's tighter constant factors and the absence of any index-build cost make it faster than HNSW, so the algorithm picks its method from the window size.

Reduce, then cluster

The window is projected from 1536-D down to 32-D with UMAP (umap-rs) before HDBSCAN runs on the reduced vectors under a Euclidean metric. Thirty-two dimensions preserve the great majority of the embeddings' explained variance while making HDBSCAN's pairwise work cheap. This reduction is ephemeral: it exists only in memory for the duration of the clustering pass. Qdrant continues to store the full 1536-D vectors; cluster labels map straight back to the original record IDs.

The whole pipeline runs on a shared global Rayon pool, dispatched via spawn_blocking so CPU-bound clustering never starves the Tokio async runtime. umap-rs avoids a BLAS/LAPACK dependency chain, keeping the build native and the binary lean.

Reduction turns an intractable clustering job into one that fits the 60-second loop, at every scale the pipeline currently targets.

06Momentum: the math

A cluster on its own is not a trend. It has to be growing abnormally fast for its context. M.A.C.H.I.N.A defines the emergence of a cluster C in a partition p as its density scaled by how many standard deviations its engagement velocity sits above that partition's own historical baseline:

The baseline mean μp and standard deviation σp are computed per partition from ClickHouse, whose SummingMergeTree tables retain a trailing 30-day history of every engagement event. Actors hydrate their baseline on boot and flush aggregated buffers back on a 10–60 s cadence.

Weighted engagement

Not all engagement carries the same information. A cluster's velocity is a sum of events weighted by "mass", the cumulative momentum CM:

The weights encode a hierarchy of intent: an act that costs the user something (a tip) is a far stronger signal than a passive view. The weighting reduces to a single match on the event type:

// Each engagement event contributes weighted "mass"
// to a partition's cumulative momentum.
impl EventType {
    fn weight(&self) -> f64 {
        match self {
            EventType::Tip { amount }     => amount * 10.0,
            EventType::Award              => 20.0,
            EventType::Vote { score }     => *score,
            EventType::Comment { s }      => match s {
                Sentiment::Positive =>  5.0,
                Sentiment::Neutral  =>  1.0,
                Sentiment::Negative => -5.0,
            },
            EventType::Reaction { value } => *value,
            EventType::View               => 1.0,
        }
    }
}

Two axes, and a noise filter

The Z-score is computed on two independent axes: volume (are more records being produced than usual?) and gravity (are people reacting harder than usual?). Separating them yields a small taxonomy of how a trend is emerging (Record-Driven, Event-Driven, or Hybrid) that travels with the anomaly downstream.

Finally, a logarithmic volume modifier tunes the alert threshold to a partition's size. On a quiet partition it loosens the threshold (up to 2×) so a small community's surge isn't dismissed; on a very loud partition it tightens toward 0.5× so normal churn doesn't spam alerts. Final momentum is the product of cluster density, Z-score, and this modifier.

07Scaling out: the Smart Gateway

Per-partition actors give vertical isolation on one node; the Smart Gateway gives horizontal scale across many. The hard constraint is partition affinity: every event for a partition_id must reach the same physical worker that holds its temporal memory, or the rolling windows are meaningless. The gateway enforces that with a consistent hash ring: hashing the partition onto a continuous ring of worker sectors gives O(1) routing with no lookup table.

To find hot partitions without tracking every one of up to ~500,000 ids in a hashmap, each gateway replica maintains a Count-Min Sketch: a probabilistic counter that tracks per-partition volume for millions of partitions in a few megabytes at ~99.9% accuracy, from which it extracts the top heavy hitters.

Zero-RPC migration

When a worker drifts out of balance, the gateway doesn't call anyone to move a partition. It writes a route override to etcd and lets the control plane converge:

{ "route_override": { "partition_123": "machina-5" } }

Every gateway ring and every worker mirrors the same etcd watch stream. When an override lands, gateways reroute and the previous owner autonomously calls evict_partition() to drop the orphaned state from memory. The migration is fully decoupled, with no direct worker-to-worker RPC. Gateways run many replicas behind the ingress but elect a single leader through a 10-second etcd lease, so only one writes overrides while the rest serve as read-only proxies. Workers publish their total post counts back into etcd under a lease TTL, which is what the rebalancer reads to decide what to move.

08Cross-partition synthesis

A trend that erupts across many partitions at once (a systemic event) would otherwise surface as a scattering of disconnected local alerts. The Global Aggregator is the second-level correlator that fuses them, and suppresses the duplicate localized webhooks that cause alert fatigue.

It is a small singleton daemon. Actors emit only cheap mathematical summaries (an EmittedCentroid carrying the anomaly id, its centroid vector, combined Z-score, and core entity ids) into an in-memory buffer holding the last hour of centroids. Because it processes summaries rather than raw data, its footprint stays near zero, which is what lets it run as a singleton.

On each arrival the aggregator prunes anything older than the window, then compares the incoming centroid against every resident vector by cosine similarity. If similarity exceeds 0.95 and the partitions are distinct, it has found a confirmed cross-partition event: it merges the two anomalies' core entity ids, compounds their Z-scores into an elevated severity, halts the individual local webhooks, and fires a single aggregated systemic webhook to the downstream synthesis stage, where the top representative records and growth metrics are turned into a human-readable alert.

09Domain-agnostic by design

Nothing in the core engine is specific to social media. M.A.C.H.I.N.A makes only two assumptions about its input: that each item can be represented as a vector, and that activity around it arrives as a stream of weighted signals. Everything downstream (the density clustering, the baseline Z-scoring, the momentum threshold, the cross-partition correlation) operates on abstract vectors and scalar weights. Swap the embedding model and the weight table, and the same machinery detects emergence in an entirely different domain.

It is also highly configurable. The partition_id can be any segmentation that matters for the problem; the event weights, reduction dimensionality, window sizes, similarity threshold, and detection cadence are all tunable. The engine adapts to the data rather than the reverse. If it can be embedded as a vector, it can be watched for emergence.

Example: cybersecurity

Point it at a firehose of security telemetry (authentication attempts, network flows, log lines) embedded as vectors and partitioned by subnet, host, or service. Weight events by severity instead of social intent, and the same density-and-momentum logic surfaces a coordinated attack at its inflection point: a cluster of semantically similar anomalies growing abnormally fast for a given segment (credential stuffing, lateral movement, a nascent botnet) before it escalates into an incident. The Global Aggregator, unchanged, becomes a detector for distributed attacks that span many hosts at once but look benign on any single one.

Example: financial fraud

Point it at a stream of transactions (card authorizations, transfers, account actions) embedded as vectors and partitioned by merchant, account, or payment corridor. Weight events by amount or risk score instead of social intent, and the same logic surfaces an emerging fraud ring at its inflection point: a tight cluster of anomalous transactions growing abnormally fast for a segment (card-testing runs, a bust-out scheme, coordinated account takeover) before the losses compound. The Global Aggregator links fraud that fans out across many accounts or merchants while staying under each one's individual radar.

The same shape fits observability over metrics and traces, market-signal analysis, or anomaly detection over IoT sensor data. Anywhere "many similar things happening abnormally fast within one segment" is worth knowing about, the engine transfers with configuration alone.

10Roadmap

Several subsystems are designed and specified but not yet shipped. They are included here as the intended trajectory, not current behavior.

Polars acceleration — planned

Under viral load (>20k events/min) the sequential momentum math would begin to starve Tokio workers. The plan is a dual-path design gated on a threshold (~2,000 records): above it, offload the Z-score and momentum computation to a SIMD-accelerated Polars LazyFrame graph via spawn_blocking; below it, keep the sequential iterators, since Arrow serialization overhead outweighs the gain at low volume.

Phase-2 clustering — planned

To push past 100k vectors per partition, a Mini-Batch K-Means stage (K = √N) is inserted between UMAP and HDBSCAN: cluster the K centroids, then propagate labels back to the full set. The target is ~1,000,000 vectors in ~2 minutes.

Trend prediction — design

Beyond detection, three non-exclusive approaches are specified: velocity-and-acceleration flags that catch exponential risers before Z > 3.0; ClickHouse-native forecasting; and a "lookalike" model that stores the centroids of massively viral trends in a dedicated Qdrant collection and matches small, quiet clusters against them to predict explosions before they happen.

11Stack