Hardening

This guide is for operators running the open-source edition in production. It is the checklist an enterprise evaluator should work through before exposing Ony.ai to an untrusted network. It complements the self-hosting guide (how to run it) and the deploy runbook (deploy/RUNBOOK.md in the repo, for day-2 ops) by focusing on the security posture. The reasoning behind these controls is in the threat model.

Ony.ai sits on a sensitive path: it decides whether an AI agent may take an action it cannot undo. The threat model therefore assumes an attacker who will try to (a) forge an allow verdict, (b) impersonate a webhook from your carrier or a chat platform, (c) bypass rate limits to brute-force auth, or (d) tamper with the audit trail. Every control below closes one of those.

What the server enforces for you

Ony.ai fails fast at boot. With ONY_ENV=production, the verify_security_config boot guard refuses to start unless:

ONY_SIGNING_SECRET is strong (at least 32 chars, not the dev default). This one secret roots the entire trust chain: it signs auth, device, invite, and reset tokens; it signs the decision verdicts the connector honors; it keys the tamper-evident audit chain; and it derives the encryption key for stored TOTP secrets.
ONY_SIGNALWIRE_AUTH_TOKEN (or the active carrier’s equivalent) is set, so telephony webhooks are signature-verified rather than anonymous.
ONY_STRIPE_WEBHOOK_SECRET is set whenever ONY_STRIPE_SECRET_KEY is (a blank webhook secret would let anyone forge a “subscription paid” event).
When ONY_ESCALATION_ENABLED=true and a chat channel’s bot token is configured, its inbound webhook secret is set too (ONY_TELEGRAM_WEBHOOK_SECRET for ONY_TELEGRAM_BOT_TOKEN, ONY_SLACK_SIGNING_SECRET for ONY_SLACK_BOT_TOKEN). Otherwise inbound approve/deny callbacks would be unauthenticated.

In any non-production ONY_ENV (local, dev, test) these become warnings and dev conveniences stay on (passwordless reset-token echo, the shared dev login, webhook-signature skip when unconfigured).

Pre-deployment checklist

Secrets

Generate strong values. Use openssl rand -hex 32 for ONY_SIGNING_SECRET and any per-service password. Use a different secret per environment (dev is not staging is not prod). Keep them in a secrets manager (Vault, AWS or GCP Secrets Manager, 1Password) and inject at runtime; the .env file, if used, must be chmod 0600 and never committed (it is already in .gitignore).

Rotating ONY_SIGNING_SECRET has staged blast radius, so plan a short maintenance window:

Bearer and device tokens signed with the old secret stop validating, so users re-authenticate and enrolled devices re-enroll. This is the desired effect if you rotate because of a suspected compromise.
Stored TOTP secrets are encrypted with a key derived from the signing secret and are re-wrapped under a versioned envelope; audit rows carry a key_version so prior entries still verify.

Never log a secret. The logging pipeline redacts secret-looking keys, but the discipline is: don’t put raw request bodies, query params, or provider tokens into log messages.

Webhook authentication

Every inbound webhook is authenticated by the sender’s own signature scheme and rejected (403) on mismatch. Rejections increment the ony_webhook_signature_failures_total{channel} metric. Alert on a spike, which indicates tampering or a misconfigured secret.

Channel	Secret	Scheme
SignalWire / Twilio	`ONY_SIGNALWIRE_AUTH_TOKEN` (the space Signing Key, `PSK_...`) / `ONY_TWILIO_AUTH_TOKEN`	HMAC-SHA1 over URL + sorted params
Telnyx	`ONY_TELNYX_PUBLIC_KEY`	Ed25519 over `timestamp\|body`, 300s tolerance
Plivo	`ONY_PLIVO_AUTH_TOKEN`	HMAC-SHA256 (V3) over URL + nonce
Stripe	`ONY_STRIPE_WEBHOOK_SECRET`	HMAC over the raw body
Telegram	`ONY_TELEGRAM_WEBHOOK_SECRET`	secret-token header echoed by Telegram
Slack	`ONY_SLACK_SIGNING_SECRET`	HMAC-SHA256 over `v0:ts:body`, 5-min replay window

Trust boundary note: a chat approve/deny (a Telegram or Slack button) is authenticated by the platform’s signature and by the linked user’s identity, but it is not separately signed by Ony.ai. Only linked users (telegram_chat_id or slack_user_id on their account) can decide, and a weak channel cannot approve a high or critical action - those still require the phone call (critical stays dashboard-only). For approval calls, answering the org’s verified number is the possession factor; a per-rule require_pin opt-in additionally demands the caller’s authenticator (TOTP) code before an approval. Enable it on high-risk rules if you require the strongest assurance, and keep critical actions on the dashboard. See escalation and fallbacks for the chat channels.

Network exposure

TLS and reverse proxy. Ony.ai listens on an unprivileged HTTP port; terminate TLS at nginx, Caddy, or Traefik in front of it, add HSTS, and proxy to the app. The public hostname must equal ONY_PUBLIC_BASE_URL or carrier callbacks will not resolve.

Trusted proxy hops. ONY_TRUSTED_PROXY_HOPS (default 0) controls how the per-IP rate limiter derives the client IP from X-Forwarded-For. 0 ignores XFF entirely and uses the socket peer, so a client cannot spoof the header to rotate its rate-limit key. Behind one reverse proxy set it to 1; behind a load balancer plus proxy, 2. Setting it too low makes every request look like it comes from the proxy (limits become useless); too high trusts a client-supplied left-most entry (spoofable). Match it to your real hop count.

/metrics. The Prometheus endpoint is unauthenticated by design (scrapers carry no bearer token) and exposes only aggregate, PII-free counters. Do not expose it to the internet - scope it to your monitoring network or block it at the reverse proxy.

Data stores

Postgres. Use a strong ONY_DB_PASSWORD, keep the database on a private network, and enable encryption at rest at the storage or instance layer. The app does not encrypt the whole DB; sensitive fields are individually encrypted: TOTP secrets, and any telephony-carrier or model credentials entered from the dashboard, are all Fernet-encrypted, keyed from ONY_SIGNING_SECRET via HKDF, and never stored in plaintext or written to logs or audit metadata. Migrations apply automatically on API start.

Backups and DR. Run nightly logical dumps (pg_dump custom format) to off-host storage (S3, GCS). The runbook recommends at least 30 daily plus 12 monthly and a single-host-loss guarantee (the DB and its backup are never on the same host). Test a restore into a fresh database quarterly; consider WAL archiving for point-in-time recovery.

Valkey (a drop-in, BSD-licensed Redis replacement) is optional for a single instance (rate limiting and decision fan-out fall back to in-memory), but required for multi-instance deployments so decisions confirmed on one worker reach the worker holding the device socket. If you run Valkey, keep it private and set requirepass (or bind to loopback); an open Valkey lets an attacker flush the rate-limiter state and bypass auth limits.

Least privilege

The API and web containers run as unprivileged users and cannot bind ports below 1024. Do not run them as root. Mount secrets read-only (or inject via the environment), restrict egress to only what you use (LLM APIs, your carrier), and in Kubernetes apply a restrictive security context or Pod Security admission.

Rate limiting

Per-IP limits protect the public auth surface (signup, login, OTP, password reset, place-call, agent-handoff). Keep ONY_RATE_LIMIT_ENABLED=true in production. Monitor 429s to spot brute-force attempts, and remember the limiter needs a shared Valkey (and a correct ONY_TRUSTED_PROXY_HOPS) to work across multiple instances.

Observability for security

Set ONY_LOG_FORMAT=json and ship logs to a SIEM. Each line carries timestamp, level, logger, request_id, version, and env; access-log lines add http.method, http.route, http.status, and http.duration_ms. Correlate a request across the API and its webhook fan-out by request_id (also returned in the X-Request-ID response header).
Scrape /metrics and alert on: a rising ony_webhook_signature_failures_total (forged webhooks), authentication failures, ony_escalation_runs_finished_total{status="exhausted"} (nobody is answering), and unusual ony_decisions_total{result="approve"} rates.
The audit chain is tamper-evident (hash and MAC chained with the signing secret). Archive it off-system for compliance and re-verify periodically; a break indicates tampering or a key-version mismatch.

Incident response starters

Suspected token or device compromise: rotate ONY_SIGNING_SECRET (this invalidates all tokens; see the rotation notes above) and revoke the affected device from the dashboard.
Suspected webhook secret leak: rotate that channel’s secret and update the sender’s configuration; watch ony_webhook_signature_failures_total return to baseline.
Suspected signing-secret leak: treat audit integrity from that point as unverifiable, rotate immediately, and force re-authentication.

For the full list of environment variables referenced here, see the configuration reference; for carrier webhook setup, see carrier setup.