[{"data":1,"prerenderedAt":71},["Reactive",2],{"work-alarm-framework":3,"work-next-alarm-framework":68},{"_path":4,"_dir":5,"_draft":6,"_partial":6,"_locale":7,"title":8,"description":7,"kicker":9,"summary":10,"cover":11,"coverSmall":12,"coverAlt":13,"where":14,"stack":15,"stats":20,"next":27,"order":28,"body":29,"_type":62,"_id":63,"_source":64,"_file":65,"_stem":66,"_extension":67},"/work/alarm-framework","work",false,"","Alarms for a whole AWS organisation","Monitoring · UK national lottery operator","A Python data source finds resources across every account at plan time, and Terraform generates their CloudWatch alarms from per-service specs. A daily GitHub Actions run keeps it current.","/3d/cover-alarm-framework.webp","/3d/cover-alarm-framework-640.webp","Clay model of a grid of small blocks with pin lights on top, two of them lit teal and one dark","UK national lottery operator",[16,17,18,19],"Terraform","Python","CloudWatch","GitHub Actions",[21,24],{"value":22,"label":23},"109","alarm modules",{"value":25,"label":26},"Every account","compute, databases, networking","terraform-guard-rails",3,{"type":30,"children":31,"toc":59},"root",[32],{"type":33,"tag":34,"props":35,"children":37},"element","case-section",{"label":36},"Overview",[38,44,49,54],{"type":33,"tag":39,"props":40,"children":41},"p",{},[42],{"type":43,"value":10},"text",{"type":33,"tag":39,"props":45,"children":46},{},[47],{"type":43,"value":48},"The specs are per service, not per resource. One spec says what is worth alarming on for a service, it is written once, and everything of that kind in the organisation inherits it. There are 109 alarm modules behind them, covering compute, databases, networking and the rest of what the estate runs. Nobody writes an alarm by hand.",{"type":33,"tag":39,"props":50,"children":51},{},[52],{"type":43,"value":53},"The resources are found rather than listed. A Python data source runs at plan time, walks every account with boto3 and returns what is actually there. Terraform then pairs the discovered resources with their specs and generates the alarms. Because the discovery happens on every plan, the plan diff is the reconciliation: anything created since the last run turns up with its alarms attached, anything deleted takes its alarms with it, and no alarm is left pointing at a resource that has gone.",{"type":33,"tag":39,"props":55,"children":56},{},[57],{"type":43,"value":58},"A GitHub Actions workflow runs that daily, so a team that launched a database in the morning has alarms on it without filing a request. What the alarms produce goes to the monitoring and alerting team that owns the platform's alerting, next to the Grafana dashboards and Sensu checks covering the same estate.",{"title":7,"searchDepth":60,"depth":60,"links":61},2,[],"markdown","content:work:alarm-framework.md","content","work/alarm-framework.md","work/alarm-framework","md",{"_path":69,"title":70},"/work/terraform-guard-rails","Terraform with guard rails",1789325965356]