Architecture for 4 billion records · Dokumentasi

Dokumentasi teknis ini tersedia dalam bahasa Inggris.

Planning reference: volumes and infrastructure figures in this document are hypothetical sizing scenarios, not measured product capacity or deployed installations. Validate workload, costs, security and recognition performance before procurement. The current feature overview is available at /features.

This page describes the reference architecture for a national deployment that holds more than 4 billion fingerprint templates, 4 billion face records and billions of vehicle sightings, and receives plate reads from over 4 million cameras. Everything is hosted inside the agency, on its own physical servers or virtual machines, as operating-system services with no outside access.

1. The problem in numbers

Workload	Size	Why it is hard
Fingerprints	4,000,000,000 templates (e.g. 400 million people × 10 fingers)	An exhaustive 1:N search compares the probe with every template
Template memory	1.7 KB per template → 6.8 TB per copy, ~13.6 TB for two copies	Must sit in RAM for fast matching
Faces	4,000,000,000 × 128-number descriptors → ~2 TB raw	Too big for one server; kept compact (~180 bytes a face) and split over shards
Camera reads	4 million cameras × 2,000 reads/day → 8 billion reads a day	~90,000 reads a second on average, peaks about 3×
Sightings storage	~600 bytes per read → ~4.8 TB a day before compression	Needs time-based retention and archiving

2. The design principles

Narrow first, then compare exactly. Never compare a probe with all 4 billion templates if a cheaper step can shortlist the likely ones.
Shard everything that is big. Fingerprint galleries, face indexes and sightings are split across servers. Adding servers adds capacity at the same speed.
Keep hot data in memory. Matchers hold their shard of templates in RAM; watch-lists for camera reads are held in memory on ingest servers.
Stateless front tiers. Application and ingest servers keep no state of their own, so any number can sit behind the load balancers.
Two sites. A primary and a disaster-recovery data centre, with data replicated between them.
Separate networks. Edge, application, matching, data and management each live in their own network zone.

3. Topology

                     Internet / government WAN / camera VPNs
                                      |
                  +-------------------+--------------------+
                  |  EDGE ZONE   HAProxy ×2 + keepalived    |  TLS 1.3, one virtual IP
                  |              VPN concentrators (cameras)|
                  +-------------------+--------------------+
                                      |
        +-----------------------------+------------------------------+
        |  APPLICATION ZONE                                          |
        |   API / web servers ×8-16      Ingest servers ×~60 (peak)  |
        +---------+------------------------------+-------------------+
                  |                              |
   +--------------+-------------+    +-----------+-------------------+
   | MATCHING ZONE              |    | DATA ZONE                     |
   |  Fingerprint shortlist ×10 |    |  Oracle or MongoDB cluster    |
   |  Native C++ matchers ×40   |    |   24 shards × 3 replicas      |
   |  Face vector index ×8      |    |   3 config servers            |
   +----------------------------+    |  NFS or object storage ×8+    |
                                     +-------------------------------+
        MANAGEMENT ZONE: bastion, monitoring, log collection, backup, key management

4. How a fingerprint search works

The probe arrives at an API server (from the web app, the field app or a partner).
If the finger position is known, only that position is searched: about 10× less work.
The shortlist index returns the most likely candidate templates (about 2,000 from each shortlist server, ~20,000 in all) out of billions in milliseconds.
The API server sends the probe and the candidates to the matcher shards that hold them (MATCH_NODES). Each shard scores only its candidates with native C++ matching and returns its top results. A single 64-core matcher server scans more than 10 million templates in under 1 second.
The API server merges the per-shard top lists and returns the best matches.
If the shortlist does not produce a match, or a shortlist server cannot be reached, an exhaustive search across all shards runs automatically: the shortlist can only make a search faster, never miss a match the full search finds.

Matchers and search times for 4 billion templates. 40 matcher servers (64 cores, 512 GB RAM each) hold two copies of the gallery: 20 servers per copy, each with ~340 GB of templates. One search runs on one copy while the other copy serves other searches, or takes over when a server fails.

Search	Time
Full search of all 4 billion (finger position unknown)	~21 s (16–32 s)
Finger position known (scanners)	~2.4 s (1.9–3.4 s)
With the shortlist index	~1 s target

Both copies together handle about 72,000 finger-known or 8,000 full searches a day. Twice the matchers halves the time. See Sizing, or the server calculator for other sizes from 40 million to 6 billion.

All of this is built into Sentinel ID and tested with a full installation split over servers. Every server runs the same software in its role (api, jobs, matcher, shortlist, face index). Each matcher loads only its shard (SHARD_ID of SHARD_COUNT) and keeps it in memory; each shard can have two copies (MATCH_NODES=a|b,...), so a search goes on when one copy fails, and a search that could not reach a whole shard says so. New enrolments go to every copy over the private internal network, protected by CLUSTER_KEY. The api servers share live alerts, settings, calls and live camera views with each other, so users can be on any of them.

5. How a face search works

Faces are split over face index servers (4 shards × 2 copies for 4 billion). Each face is kept in about 180 bytes (its 128 numbers as one byte each, plus its ids), so a 256 GB server holds a billion. Every search is exact: each shard compares the probe with all its faces on every core and the API server keeps the nearest face per person. Measured at 10.3 million faces a second per thread, that is about 1.5 s for a billion faces on 64 cores; three times the face shards bring it to about 0.5 s.

6. How camera reads are handled

Cameras read plates at the edge: the camera (or an edge box on the pole) sends the plate text, a small crop and its location, not a video stream.
Reads reach the ingest servers through the camera VPN concentrators and the load balancers, using POST /api/v1/sightings or POST /api/v1/cameras/ingest with the camera's own API key.
Each read is checked against the wanted and flagged lists held in memory and written to the sightings collection. A hit raises an alert in under two seconds, with the camera's location.
Full-resolution images are kept only for hits and evidence. Ordinary reads keep the small crop for a short retention period.

7. Data tier

Database: Oracle or MongoDB. With MongoDB (7 or 8): a sharded cluster of 24 shards of 3-member replica sets for the reference size, plus 3 config servers. mongos routers run on the application and ingest servers.
Shard keys: hashed _id for people, templates and vehicles; { cameraId: 1, capturedAt: 1 } for sightings, with time-based archiving.
With Oracle Database (23ai / 26ai), for agencies that standardise on Oracle: partitioned tables, RAC within the site and Data Guard to the recovery site give the same scale-out and recovery.
Central file storage for photos, evidence, video and camera crops: an NFS (Network File System) export from an enterprise NAS cluster, or, better at national size, S3-compatible object storage with erasure coding. Both stay inside the agency.
The tamper-evident audit log lives in the database, hash-chained, and is also forwarded to the agency SIEM.

8. Availability and recovery

Component	Within a site	Between sites
Load balancers	Two HAProxy servers sharing a keepalived virtual IP	DNS or GSLB fail-over
API and ingest	N+2 servers	Same build at the recovery site
Matchers	Two in-memory copies of every shard	Recovery site loads from the replicated database
Database	3-member replica set per shard	Delayed / hidden members at the recovery site
Files	NAS cluster (NFS) or erasure-coded object storage	Asynchronous replication

Target recovery point: under 15 minutes. Target recovery time for the recovery site: under 4 hours (agreed in the contract).

9. Next pages

Sizing and hardware: servers, CPU, memory and storage for each tier.
Installation: step by step.
What is built today and what the project delivers.

← Sebelumnya Berikutnya →