Operations

Running Sentinel ID day to day: monitoring, backups, disaster recovery, upgrades and capacity.

Monitoring

Collect metrics from every server (for example with Prometheus node exporters, installed as system services) and watch:

What	Warn when
API health (/api/v1/health) on each server	Not answering for 30 s
Response time of identify calls	95th percentile above the agreed target
Matcher memory and template counts per shard	A shard's count differs from the database
Matcher engine	Any matcher not running the native engine
Ingest rate and backlog	Reads per second falling suddenly, or a growing queue
MongoDB replication lag	Above 10 s
Disk, memory, CPU	Above 80% for 15 minutes
Certificates	Expiring within 30 days
Cameras	Offline, low battery or tamper alarms (also raised as alerts in Sentinel ID)

Backups

Data	Method	Frequency	Kept
Database	Filesystem snapshots of a hidden secondary per shard, plus continuous oplog backup	Snapshot daily, oplog continuous	35 days online, 1 year offline
Files (photos, evidence, video)	Replication to the recovery site, plus weekly offline copy	Continuous / weekly	By retention policy
Configuration (/etc/sentinel, load balancers, CA)	Encrypted archive to the management zone	On every change	1 year

Encrypt every backup and store the keys separately. Test a restore every month, on a test environment.

Disaster recovery

• The recovery site runs the same release, with database members replicated from the primary and files replicated asynchronously.
• Matchers at the recovery site load their shards from the replicated database on start.
• Fail over with DNS or the global load balancer once the database has elected primaries at the recovery site.
• Run a full recovery drill twice a year and record the recovery time achieved.

Upgrades

1. Read the release notes; verify the release signature.
2. Upgrade the test environment and run the acceptance checks.
3. Back up the configuration and take a database snapshot.
4. Upgrade application and ingest servers one at a time, taking each out of the load balancer first.
5. Upgrade matchers one at a time; each reloads its shard before the next starts.
6. Watch the dashboards for an hour; roll back by reinstalling the previous release if needed.

Capacity

• Add application or ingest servers as users and cameras grow: install, then add to the load balancer.
• Add database shards when disks pass 60% or write latency rises; MongoDB rebalances chunks online.
• Add matcher shards in a maintenance window: update SHARD_COUNT and MATCH_NODES everywhere, then restart matchers so each loads its new share.
• Re-run the acceptance benchmark after any large change.