---
name: datadog-observability
description: "Generate Datadog dashboards, monitors, SLOs, and APM traces. Production observability as code. Builds Datadog dashboards via Terraform, writes monitors with smart thresholds, sets up SLOs with realistic budgets, and instruments APM with the right tracer hooks for your stack."
license: MIT
---

# Datadog Observability

This skill scaffolds production observability for Datadog as version-controlled code: dashboards, monitors, SLOs, and APM instrumentation. It targets backend and platform engineers who want their telemetry to live in Terraform and Git rather than being hand-clicked in the UI and lost on the next reorg. Output is reviewable in PRs and reproducible across staging and prod.

## When to use it

- You're standing up monitoring for a new service and want dashboards, alerts, and an SLO from day one.
- You have dozens of click-configured monitors drifting out of sync and want to codify them.
- You're adding APM to an app and aren't sure which tracer hooks and env vars your stack needs.
- You need an error-budget-based SLO with an alert that fires on burn rate, not on raw error count.

## What it does

It generates HCL for the `DataDog/datadog` Terraform provider — `datadog_dashboard_json` (or `datadog_dashboard`) for dashboards, `datadog_monitor` for alerts, `datadog_service_level_objective` for SLOs, and `datadog_synthetics_test` where you want black-box checks. Monitor queries use real Datadog metric syntax (`avg(last_5m):...`, `anomalies()`, `forecast()`), and thresholds are split into `critical`/`warning` with sensible `evaluation_delay` and `no_data_timeframe` so alerts don't flap on sparse metrics.

For APM it wires up the matching `dd-trace` library — `ddtrace` (Python, `ddtrace-run`), `dd-trace` (Node, required before app code), `dd-java-agent.jar` via `-javaagent`, or the Go `dd-trace-go` `tracer.Start()` call — plus the `DD_ENV`, `DD_SERVICE`, and `DD_VERSION` unified-service-tagging vars.

## Usage

```bash
curl -fsSL --create-dirs -o ~/.claude/skills/datadog-observability/SKILL.md https://claudeskil.com/skills/datadog-observability/SKILL.md

# then, in your project:
"Create a Terraform monitor for p99 latency > 500ms on the checkout service,
 plus a 99.9% availability SLO over 30 days, and instrument the Node app with APM."
```

It emits `monitors.tf`, `slo.tf`, a dashboard JSON, and the tracer bootstrap, ready for `terraform plan`.

## Tips & gotchas

- Set `validate = false` on `datadog_dashboard_json` only if a widget type lags the provider schema; otherwise leave validation on to catch typos.
- SLO monitor IDs must already exist — create the `datadog_monitor` first and reference it, or Terraform applies in the wrong order.
- Unified service tagging (`env`/`service`/`version`) must match between your tracer config and your monitor `tags`, or APM-to-metric correlation breaks silently.
- `evaluation_delay` matters for metrics with ingestion lag (e.g. AWS integration metrics ~10min); too short and you get false no-data alerts.

## FAQ

**Does it need API and APP keys?** Yes — the Terraform provider authenticates with `DD_API_KEY` and `DD_APP_KEY` (or `api_key`/`app_key` provider args). The skill generates config, not credentials.

**Can it import existing dashboards?** It writes new resources; for existing ones use `terraform import` against the resource ID, then let the skill reshape the HCL.

**Which SLO types?** Both metric-based and monitor-based SLOs, with rolling or calendar-aligned time windows.