Datadog Observability

This skill scaffolds production observability for Datadog as version-controlled code: dashboards, monitors, SLOs, and APM instrumentation. It targets backend and platform engineers who want their telemetry to live in Terraform and Git rather than being hand-clicked in the UI and lost on the next reorg. Output is reviewable in PRs and reproducible across staging and prod.

When to use it

• You're standing up monitoring for a new service and want dashboards, alerts, and an SLO from day one.
• You have dozens of click-configured monitors drifting out of sync and want to codify them.
• You're adding APM to an app and aren't sure which tracer hooks and env vars your stack needs.
• You need an error-budget-based SLO with an alert that fires on burn rate, not on raw error count.

What it does

It generates HCL for the DataDog/datadog Terraform provider — datadog_dashboard_json (or datadog_dashboard) for dashboards, datadog_monitor for alerts, datadog_service_level_objective for SLOs, and datadog_synthetics_test where you want black-box checks. Monitor queries use real Datadog metric syntax (avg(last_5m):..., anomalies(), forecast()), and thresholds are split into critical/warning with sensible evaluation_delay and no_data_timeframe so alerts don't flap on sparse metrics.

For APM it wires up the matching dd-trace library — ddtrace (Python, ddtrace-run), dd-trace (Node, required before app code), dd-java-agent.jar via -javaagent, or the Go dd-trace-go tracer.Start() call — plus the DD_ENV, DD_SERVICE, and DD_VERSION unified-service-tagging vars.

Usage

curl -fsSL --create-dirs -o ~/.claude/skills/datadog-observability/SKILL.md https://claudeskil.com/skills/datadog-observability/SKILL.md

# then, in your project:
"Create a Terraform monitor for p99 latency > 500ms on the checkout service,
 plus a 99.9% availability SLO over 30 days, and instrument the Node app with APM."

It emits monitors.tf, slo.tf, a dashboard JSON, and the tracer bootstrap, ready for terraform plan.

Tips & gotchas

• Set validate = false on datadog_dashboard_json only if a widget type lags the provider schema; otherwise leave validation on to catch typos.
• SLO monitor IDs must already exist — create the datadog_monitor first and reference it, or Terraform applies in the wrong order.
• Unified service tagging (env/service/version) must match between your tracer config and your monitor tags, or APM-to-metric correlation breaks silently.
• evaluation_delay matters for metrics with ingestion lag (e.g. AWS integration metrics ~10min); too short and you get false no-data alerts.

FAQ

Does it need API and APP keys? Yes — the Terraform provider authenticates with DD_API_KEY and DD_APP_KEY (or api_key/app_key provider args). The skill generates config, not credentials.

Can it import existing dashboards? It writes new resources; for existing ones use terraform import against the resource ID, then let the skill reshape the HCL.

Which SLO types? Both metric-based and monitor-based SLOs, with rolling or calendar-aligned time windows.