Nexus

Nexus is a real-time event clustering platform: it turns a firehose of social posts into a continuously updated map of the stories happening in it. Each incoming post is embedded, matched against existing story hubs, and either assigned, parked as an ambiguous candidate, or rejected as noise. New stories emerge on their own through community detection over buffered posts, duplicate content is folded, and every decision is delivered to downstream applications through a transactional event log.

It is also a system that is meant to be correctable. Every automatic decision is journaled with the numbers that produced it. An operator can confirm or unlink any assignment, and those human corrections are the only signal the system learns from. A daily autotuner replays that history and proposes threshold revisions; a monthly job fine-tunes a LoRA adapter on the same corrections and promotes it only if it beats the current model on a held-out set.

What makes the project interesting is not the component list. It is that every meaningful decision came from measuring the actual system rather than trusting a reasonable default. The thresholds were recalibrated because the originals sat above the embedder's ceiling and produced silent failure. The parallelization was deleted because concurrency made throughput 10x worse. The noise filter was rewritten from a keyword list into a structural test after adversarial examples leaked through it. And the learning loop was restructured after an audit showed it was training on a positive it had curated itself, which meant it could never teach the model what the correct hub was.

Stack: PostgreSQL 16 with pgvector (HNSW), Celery with Redis, FastAPI, spaCy for entity extraction, a MiniLM transformer with a LoRA adapter, and a server-rendered operator panel. Everything runs on CPU.

The signal is sparse and distributed. The posts that matter to a single story are scattered across different people, different threads, and different hours. No individual reader assembles the full picture, so nobody can answer the only question that matters: what is actually going on here?
The obvious approach fails on identity. A similarity search over embeddings treats an Apple product launch and a Samsung product launch as near-identical text. Cluster a "phone launch" thread on embeddings alone and it fuses into one confident, wrong hub. The embedder has no concept of who is speaking, and cosine similarity cannot recover it.

Naive filtering throws away the important stuff. The usual approach to noise is a stopword list or a length threshold, but real feeds are full of text that looks substantive and isn't, and the important posts are often the ones that name nothing. "The two-year yield had its biggest one-day drop" is a genuine market event with no entity in it at all, and a noun-phrase filter discards it.
Automatic clustering cannot self-correct. A threshold that is slightly too loose accumulates errors invisibly. Worse, a system that learns from its own confident assignments learns to agree with itself, and its reported precision becomes fiction.

So the problem reduces to two questions that are much harder than they sound: recover identity from text, and decide what deserves to exist reliably enough to trust either with automatic action.

Measure the embedder before trusting it. The original thresholds were conventional (0.80–0.95 cosine) and produced zero hubs and 0% coverage: not misclustering, complete silence, because the constants sat above the model's actual ceiling. Profiling real content showed same-topic similarities peak around 0.71 (mean ~0.50) while cross-topic text sits near 0.1. Every threshold was recalibrated against that measured scale.

The same discipline killed other defaults windowed embeddings stay at a 256-token head because tiling measured worse on live content, and near-duplicate folding sits at 0.98 cosine because folding is destructive and must be near-certain.

Resolve identity by evidence overlap, not string equality. The naive rule "posts sharing an entity belong together" breaks on a single cross-mention post, where "Apple's reveal is compared to Samsung's event" welds two disjoint stories together. Instead, two entity strings are the same actor only when their post sets overlap substantially (≥ 0.6 of the smaller set): aliases overlap near 1.0, rivals only on comparison posts at ≤ 0.56. What counts as identity is deliberately asymmetric generic collective nouns ("insiders", "observers") are stripped because they appeared across every topic and would make all hubs look related; places are excluded because a hurricane legitimately spans several states; and people are included for vetoes but excluded from merge evidence, because "Musk" spans distinct Tesla and SpaceX threads.

Filter on structure rather than keywords. The gate rejects cheap cases (empty, too short, chatter, first-person diary) before spending anything on parsing, then admits anything carrying any independent route to substance: entities, proper nouns, debate structure, modality, a topical event lexicon, a claim hashtag, or a question. "Diversity drives innovation, period." clears on the verb drives with zero entities. Universal quantifiers are explicitly not evidence, because they appear constantly in filler.

Learn from operators, never from itself. Only explicit human corrections become training signal; system bookkeeping is stored and displayed but never graded. The learner was rewritten after an audit showed it had been training on a positive it curated itself, which meant it could not teach what the correct hub was. Positives now come from confirmed pairs, removed posts become hard negatives, and a post rejected after a merge moved it there counts as an even stronger negative. Promotion is gated on a holdout scored with the same metric, with the split hashing each post to exactly one side so nothing leaks.

The autotuner proposes rather than applies, and scans the entire precision curve because precision is non-monotone a first-fit scan silently picks a worse point.

Ingest. A durable row insert plus a bounded Redis hint queue. Redis is a latency fast path, never the source of truth, so an outage degrades to slower processing rather than lost posts. The row carries a content fingerprint computed at ingest, normalized conservatively so only byte-identical bodies collapse exactly; near-copies fall through to an embedding pass.

Process. Work flows through Celery to a single-flight CPU embedder, then the discourse gate, so chatter never costs an embedding. A worker claims posts with a compare-and-swap status transition, so concurrent workers can never process the same post twice.
Assign. A bulk chunk-level SQL pass: HNSW top-2 against fresh centroids in one statement, with a margin check between best and runner-up. An ambiguous post is parked as a candidate rather than guessed at. The HNSW index is partial, scoped to active non-null centroids, keeping the freshness window inside the index.
Discover. Hourly, buffered posts are partitioned into temporal slices, joined into a k-NN graph with exponential time decay (12h half-life), and clustered with Louvain community detection, guarded by multi-author and cohesion requirements. Because community detection over-splits (measured up to 34 hubs for 10 topics), a merge reconciliation to fixpoint runs immediately after birth.

Deliver. A transactional outbox with lease-based claiming via FOR UPDATE SKIP LOCKED, a dead-letter queue, and TTL garbage collection, so a downstream outage becomes a recoverable backlog instead of a stalled pipeline. Every event carries a per-post monotone sequence, letting consumers drop stale deliveries and order events per post despite at-least-once delivery.
Merge and correct. All merge initiators funnel through one guarded transaction, so the outcome is identical regardless of who asked. Concurrent merges of the same pair serialize on an advisory lock taken on the ordered hub id pair, making deadlock structurally impossible. Merges snapshot their member set so reopening is exact rather than heuristic, and reopening recomputes centroids for both hubs from live members instead of restoring the frozen snapshot.

Every mutation is versioned, journaled, and reversible.
Operate. Versioned advisory-locked migrations, structured JSON logging, a 49-route operator panel for curation, calibration, merges, and observability, and a fail-closed production posture where an unset auth token refuses to boot rather than serving open.