AI Video Monitoring for Existing Cameras

Health & observability

Separate liveness, readiness, and workload health

/healthz answers whether the process event loop is alive; /readyz covers database access and startup reconciliation. One optional camera going offline should not make the whole runtime unready.

GET /healthz

Liveness

Unauthenticated. Use only to decide whether the process should restart.

GET /readyz

Readiness

Unauthenticated and secret-free. Fails when the database is unusable or reconciliation is incomplete.

Authenticated system state

GET /v1/system
GET /v1/system/resources
GET /v1/system/models

Use GET /v1/system/resources to query the current host, inference, webhook, database, SSE, RTSP, and evidence capacities. It returns a point-in-time JSON snapshot, not Prometheus metrics.

System summarizes application/contract versions, boot time, database state, and desired/running/failed monitor counts. Resources covers host capacity and workload; Models reports inference data boundary and readiness.

Minimum fields to monitor

Per-Source runtime_state, last_frame_at, and sanitized status.
Provider kind, sanitized target, data boundary, and readiness.
CPU load, RAM, process memory, and disk capacity.
GPU provider and device when available; unmeasured VRAM is null.
Inference queue depth and active work; latency percentiles are null when not measured.
Pending/retrying and dead-letter webhook delivery counts.

Container probe

healthcheck:
  test: ["CMD", "python", "/opt/sentinel/healthcheck.py"]
  interval: 10s
  timeout: 5s
  retries: 12
  start_period: 45s