3. Application tier (API, web and ingest servers)

The application tier runs Sentinel ID itself: the web application, the field-app API, the REST API, live alerts and voice-call signalling. Ingest servers run the same software and take the camera traffic. All of them are stateless and sit behind the load balancers.

Install Sentinel ID (every application and ingest server)

# release delivered as a signed archive
sudo mkdir -p /opt/sentinel /etc/sentinel /var/lib/sentinel/storage
sudo tar -C /opt/sentinel -xzf sentinel-id-<version>.tar.gz
cd /opt/sentinel
sudo -u sentinel npm ci --omit=dev
sudo chown -R root:sentinel /opt/sentinel && sudo chmod -R o-rwx /opt/sentinel
sudo chown sentinel:sentinel /var/lib/sentinel/storage

Verify the archive's signature before unpacking it. The application files are owned by root and only readable by the service account, so the service cannot modify its own code.

Configuration

/etc/sentinel/sentinel.env, mode 0600, owner sentinel:

NODE_ENV=production
PORT=4000
MONGO_URI=mongodb://sentinel_app:<password>@127.0.0.1:27017/sentinel_id?authSource=admin&tls=true
JWT_SECRET=<64 random bytes, hex>
UPLOAD_DIR=/var/lib/sentinel/storage
CORS_ORIGINS=https://sentinel.agency.gov

# fingerprint matching is done by the matcher shards
MATCH_NODES=https://m00.match.internal:4000,https://m01.match.internal:4000,...,https://m31.match.internal:4000
MATCH_NODE_KEY=<long random secret shared with the matchers>
FP_THRESHOLD=25
FACE_THRESHOLD=0.5

Generate secrets with openssl rand -hex 64. Never reuse a secret between environments.

systemd service

/etc/systemd/system/sentinel.service:

[Unit]
Description=Sentinel ID
After=network-online.target mongos.service
Wants=network-online.target

[Service]
User=sentinel
Group=sentinel
WorkingDirectory=/opt/sentinel
EnvironmentFile=/etc/sentinel/sentinel.env
ExecStart=/usr/local/bin/node src/server.js
Restart=always
RestartSec=3
LimitNOFILE=1048576

# hardening
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
PrivateDevices=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
RestrictSUIDSGID=yes
LockPersonality=yes
CapabilityBoundingSet=
ReadWritePaths=/var/lib/sentinel/storage

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable --now sentinel
journalctl -u sentinel -f      # expect: "Sentinel ID listening on http://localhost:4000"

If the port is already taken, the service stops with port 4000 is already in use instead of running without serving anything.

Shared files

Photos, evidence and video must be visible to every application server. Files live on central storage. Mount the NFS (Network File System) export from the highly available NAS cluster, on the isolated data network (NFS v4.2 with Kerberos sec=krb5p for encryption and authentication), at /var/lib/sentinel/storage on each server. At national size, S3-compatible object storage (erasure-coded, inside the agency) scales further; its adapter is delivered with the national-scale components (see What is built today).

Load balancers (HAProxy + keepalived)

Two HAProxy servers in the edge zone share a virtual IP with keepalived. HAProxy terminates public TLS and forwards to the application servers over TLS.

/etc/haproxy/haproxy.cfg (essentials):

global
  ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets
  ssl-default-bind-ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256

defaults
  mode http
  timeout connect 5s
  timeout client  60s
  timeout server  60s
  timeout tunnel  1h          # WebSockets (voice-call signalling, phone camera) and live alert streams

frontend public
  bind :443 ssl crt /etc/haproxy/certs/sentinel.pem alpn h2,http/1.1
  http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains"
  acl is_camera path_beg /api/v1/cameras/ingest /api/v1/sightings
  use_backend ingest if is_camera
  default_backend app

backend app
  balance leastconn
  option httpchk GET /api/v1/health
  server api01 10.20.0.11:4000 ssl verify required ca-file /etc/haproxy/ca.pem check
  server api02 10.20.0.12:4000 ssl verify required ca-file /etc/haproxy/ca.pem check
  # ...

backend ingest
  balance leastconn
  option httpchk GET /api/v1/health
  server in01 10.20.1.11:4000 ssl verify required ca-file /etc/haproxy/ca.pem check
  # ...

When Sentinel ID listens on plain HTTP behind the balancer, put nginx on each application server to terminate the internal TLS, or keep the application servers on the isolated application network and restrict port 4000 to the load balancers' addresses.

/etc/keepalived/keepalived.conf shares the virtual IP between the two balancers (VRRP, with an authentication password and a health check on HAProxy).

Firewall rules (application zone)

• Port 4000 only from the load balancers.
• Outgoing: 27017 to the local mongos only (loopback), 27018/27019 to the data zone, 4000 to the matcher zone, SMTP and SMS gateways for alert notifications.

Windows Server

On Windows Server the same release runs as a Windows service (for example with NSSM or WinSW), with IIS + ARR or HAProxy in front, the configuration in a file readable only by the service account, and BitLocker for the disks.

Check

curl -s https://sentinel.agency.gov/api/v1/health
# {"ok":true,"time":"...","started":"..."}

Next: Install the matching tier.