AI Video Monitoring for Existing Cameras

Operate · Recovery

Diagnose first, recover with proof

This page is the routing and recovery sequence for operators. Executable scripts and the complete incident playbooks ship only in the verified source kit. This website never operates your node or asks for an API key.

Use the matching revision first. Run the commands below only from a source kit whose SHA-256 and revision you verified. The node's /v1/openapi.json plus docs/api/operator-playbooks.md and deploy/api/recovery/README.md in that kit are authoritative for the release. Never paste these commands into an unknown checkout.

Route by stable error code

Record HTTP status, error.code, request_id, Retry-After, and aggregate health first. Never record API keys, RTSP URLs, request bodies, or environment dumps. Exact diagnosis and escalation criteria live in docs/api/operator-playbooks.md inside the source kit.

Stable signal	First playbook
source.authentication_failed · source_unreachable · rtsp_*	RTSP auth, disconnect, or admission
model_unavailable · inference.capacity_exceeded	Provider availability and capacity
database_capacity_exceeded · evidence_capacity_exceeded	Storage pressure and recovery point
delivery retrying/dead_letter	Webhook outage and bounded replay
invalid_api_key · bad webhook signature	API/signing-key compromise

Create a quiesced, verifiable backup

Enter deploy/api from a trusted POSIX shell. On Windows, use a separate checkout inside Ubuntu/WSL; these recovery scripts are not PowerShell commands. --stop-writers creates a brief monitoring gap that must be approved and recorded.

install -d -m 700 /secure/sentinel-backups
cd deploy/api
./recovery/backup.sh \
  --output-dir /secure/sentinel-backups \
  --stop-writers

The script stops only writers it proves belong to the Compose project, uses SQLite online backup, and preserves data, evidence, and model cache. It excludes .env; disaster recovery also requires the exact API image by immutable digest and separately managed secrets.

Verify every stored copy

./recovery/verify.sh \
  --backup-dir /secure/sentinel-backups/sentinel-backup-ID

Verification recalculates sizes and SHA-256, rejects unsafe archive paths, and runs SQLite quick_check. Run it again after copying to encrypted, access-controlled off-node storage. Manifest hashes detect corruption; they are not signatures or immutable storage.

Restore only into new volumes, then validate offline

BACKUP=/secure/sentinel-backups/sentinel-backup-ID
./recovery/restore.sh \
  --backup-dir "$BACKUP" \
  --target-data-volume sentinel-restore-data-ID \
  --target-model-volume sentinel-restore-models-ID

# Review the dry-run, then authorize creation of only those new volumes.
./recovery/restore.sh \
  --backup-dir "$BACKUP" \
  --target-data-volume sentinel-restore-data-ID \
  --target-model-volume sentinel-restore-models-ID \
  --execute

./recovery/validate-restore.sh \
  --backup-dir "$BACKUP" \
  --data-volume sentinel-restore-data-ID \
  --model-volume sentinel-restore-models-ID

Never overwrite a production volume in place. After offline byte-for-byte validation, start a candidate runtime with a different project, network, port, and the restored volumes. Re-run readiness, Source/Monitor reconciliation, evidence download, Webhook backlog, and real-model acceptance.

Roll back without deleting either generation

./recovery/rollback.sh --backup-dir "$BACKUP"
# Only after the read-only preflight proves volumes and exact image identity:
./recovery/rollback.sh --backup-dir "$BACKUP" --execute

Rollback loses writes made only after cutover. Stop new writes and preserve the candidate first when those writes matter. The script deletes neither generation; never add --volumes or run docker system prune --volumes.