Incident log
Newest first. Every production incident gets an entry: what broke, why, what fixed it, what changed so it cannot repeat.
2026-09-08 — No email left production, ever
- Symptom: portal signups stored, but no confirmation or notification email arrived, with the Mailjet API returning success.
- Cause: every sender in the Mailjet account was Inactive: the sender domain was never ownership-validated after the move to API sending, and Mailjet silently drops mail from unvalidated senders.
- Fix: added the Mailjet ownership TXT record to DigitalOcean DNS, validated the
wildcard sender
*@keferboeck.comvia the API; messages flowed within minutes. - Prevention: ADR 0005 documents the wildcard validation; any new sender domain must be validated in the account before first use. The uptime monitor cannot see this failure, so newsletter signups double as the canary.
2026-09-07 — Website deploys failing at registry push
- Symptom:
kamal deployfor the website failed repeatedly during the image push ("use of closed network connection", then buildkit DNS refusingauth.docker.io), while plain-container DNS was healthy. - Cause: the buildx buildkit builder's embedded DNS wedged; recreating the builder did not help.
- Fix: switched the Kamal builder to the plain
dockerdriver inconfig/deploy.yml; deploys have been clean since. - Prevention: the driver choice is committed with an explanatory comment; long local builds also survive laptop sleep poorly — deploys run in background tasks.
2026-08 — SMTP silently timing out after the App Platform migration
- Symptom: portal mail hung after the move to the droplet.
- Cause: DigitalOcean blocks outbound SMTP ports on droplets.
- Fix: moved all delivery to the Mailjet HTTP API on 443 (ADR 0005).