Alarms for a whole AWS organisation

A Python data source finds resources across every account at plan time, and Terraform generates their CloudWatch alarms from per-service specs. A daily GitHub Actions run keeps it current.

The specs are per service, not per resource. One spec says what is worth alarming on for a service, it is written once, and everything of that kind in the organisation inherits it. There are 109 alarm modules behind them, covering compute, databases, networking and the rest of what the estate runs. Nobody writes an alarm by hand.

The resources are found rather than listed. A Python data source runs at plan time, walks every account with boto3 and returns what is actually there. Terraform then pairs the discovered resources with their specs and generates the alarms. Because the discovery happens on every plan, the plan diff is the reconciliation: anything created since the last run turns up with its alarms attached, anything deleted takes its alarms with it, and no alarm is left pointing at a resource that has gone.

A GitHub Actions workflow runs that daily, so a team that launched a database in the morning has alarms on it without filing a request. What the alarms produce goes to the monitoring and alerting team that owns the platform's alerting, next to the Grafana dashboards and Sensu checks covering the same estate.