یہ تکنیکی دستاویز انگریزی میں دستیاب ہے۔
Running Sentinel ID day to day: monitoring, backups, disaster recovery, upgrades and capacity.
Monitoring
Collect metrics from every server (for example with Prometheus node exporters, installed as system services) and watch:
| What | Warn when |
|---|---|
API health (/api/v1/health) on each server |
Not answering for 30 s |
| Response time of identify calls | 95th percentile above the agreed target |
| Matcher memory and template counts per shard | A shard's count differs from the database |
| Matcher engine | Any matcher not running the native engine |
| Ingest rate and backlog | Reads per second falling suddenly, or a growing queue |
| MongoDB replication lag | Above 10 s |
| Disk, memory, CPU | Above 80% for 15 minutes |
| Certificates | Expiring within 30 days |
| Cameras | Offline, low battery or tamper alarms (also raised as alerts in Sentinel ID) |
Backups
| Data | Method | Frequency | Kept |
|---|---|---|---|
| Database | Filesystem snapshots of a hidden secondary per shard, plus continuous oplog backup | Snapshot daily, oplog continuous | 35 days online, 1 year offline |
| Files (photos, evidence, video) | Replication to the recovery site, plus weekly offline copy | Continuous / weekly | By retention policy |
Configuration (/etc/sentinel, load balancers, CA) |
Encrypted archive to the management zone | On every change | 1 year |
Encrypt every backup and store the keys separately. Test a restore every month, on a test environment.
Disaster recovery
- The recovery site runs the same release, with database members replicated from the primary and files replicated asynchronously.
- Matchers at the recovery site load their shards from the replicated database on start.
- Fail over with DNS or the global load balancer once the database has elected primaries at the recovery site.
- Run a full recovery drill twice a year and record the recovery time achieved.
Upgrades
- Read the release notes; verify the release signature.
- Upgrade the test environment and run the acceptance checks.
- Back up the configuration and take a database snapshot.
- Upgrade application and ingest servers one at a time, taking each out of the load balancer first.
- Upgrade matchers one at a time; each reloads its shard before the next starts.
- Watch the dashboards for an hour; roll back by reinstalling the previous release if needed.
Capacity
- Add application or ingest servers as users and cameras grow: install, then add to the load balancer.
- Add database shards when disks pass 60% or write latency rises; MongoDB rebalances chunks online.
- Add matcher shards in a maintenance window: update
SHARD_COUNTandMATCH_NODESeverywhere, then restart matchers so each loads its new share. - Re-run the acceptance benchmark after any large change.