Skip to content

Incident log

Newest first. Every production incident gets an entry: what broke, why, what fixed it, what changed so it cannot repeat.

2026-09-08 — No email left production, ever

  • Symptom: portal signups stored, but no confirmation or notification email arrived, with the Mailjet API returning success.
  • Cause: every sender in the Mailjet account was Inactive: the sender domain was never ownership-validated after the move to API sending, and Mailjet silently drops mail from unvalidated senders.
  • Fix: added the Mailjet ownership TXT record to DigitalOcean DNS, validated the wildcard sender *@keferboeck.com via the API; messages flowed within minutes.
  • Prevention: ADR 0005 documents the wildcard validation; any new sender domain must be validated in the account before first use. The uptime monitor cannot see this failure, so newsletter signups double as the canary.

2026-09-07 — Website deploys failing at registry push

  • Symptom: kamal deploy for the website failed repeatedly during the image push ("use of closed network connection", then buildkit DNS refusing auth.docker.io), while plain-container DNS was healthy.
  • Cause: the buildx buildkit builder's embedded DNS wedged; recreating the builder did not help.
  • Fix: switched the Kamal builder to the plain docker driver in config/deploy.yml; deploys have been clean since.
  • Prevention: the driver choice is committed with an explanatory comment; long local builds also survive laptop sleep poorly — deploys run in background tasks.

2026-08 — SMTP silently timing out after the App Platform migration

  • Symptom: portal mail hung after the move to the droplet.
  • Cause: DigitalOcean blocks outbound SMTP ports on droplets.
  • Fix: moved all delivery to the Mailjet HTTP API on 443 (ADR 0005).