Metric Configuration

Files: metrics/*.yml

Basic Structure

# Metric identification
name: cpu_usage
description: |                   # Optional: multi-line, surfaces as {description}
  CPU usage monitoring metric.
  Tracks system load over time.
tags: [critical, infrastructure] # Optional: drives `dtk run --select tag:critical`
ai_context:                      # Optional: OSI-compatible grounding (descriptive only)
  synonyms: ["cpu load"]         #   alt names → opt-in {synonyms} alert var + tune cockpit
profile: prod                   # Optional: override default_profile
enabled: true                   # Optional: disable metric

# Data loading
interval: 1min
query: |
  SELECT timestamp, cpu_percent AS value
  FROM system_metrics
  WHERE timestamp >= '{{ dtk_start_time }}'
    AND timestamp < '{{ dtk_end_time }}'
  ORDER BY timestamp

# Or use external SQL file
# query_file: sql/cpu_usage.sql

# Column mapping (optional)
query_columns:
  timestamp: timestamp
  metric: value

# Data loading options
loading_start_time: "2024-01-01 00:00:00"
loading_batch_size: 1440         # Load 1 day at a time

# Seasonality extraction (auto-extracted from timestamps)
seasonality_columns:
  - hour
  - day_of_week

# Detectors
detectors:
  - type: mad
    params:
      threshold: 3.0
      window_size: 1440
      min_samples: 100

# Alerting
alerting:
  enabled: true
  channels:
    - mattermost_ops
  consecutive_anomalies: 3

# Custom table names (optional)
tables:
  datapoints: _dtk_datapoints_cpu
  detections: _dtk_detections_cpu

Editing a metric after it already has data? A detector’s identity is a hash of its parameters, and each alerting block’s state is keyed by a hash of its functional fields. So changing a detector parameter (or seasonality_components), removing a detector, or changing/removing an alerting block leaves the old rows behind in _dtk_detections / _dtk_alert_states — the pipeline simply stops writing to them. Run dtk clean --select <metric> to preview and prune that orphaned data. Renamed or deleted the metric entirely? Use dtk clean --orphaned-metrics. (Datapoints are not orphaned by a parameter edit — they are keyed only by timestamp; use --full-refresh to reload those.)

Metric Identification

name (string, required)

Unique metric identifier. Used in:

• CLI selectors (dtk run --select cpu_usage)
• Database queries (WHERE metric_name = ‘cpu_usage’)
• Logs and alerts

Must be unique across all metrics in the project.

description (string, optional)

Free-form description of the metric. Supports multi-line text (use a YAML block scalar |). Surfaced in alert templates as the {description} and {description_line} variables.

ai_context (string or mapping, optional)

OSI-compatible grounding for the metric — its business meaning, alternative names and example values — mirroring the Open Semantic Interchange ai_context shape so a metric’s meaning is portable to and from an OSI semantic model. Accepts a bare string (lifted to instructions) or the full mapping:

ai_context:
  instructions: "Revenue recognized at order completion, net of refunds (UTC)."
  synonyms: ["total revenue", "gross sales"]
  examples: ["12030.50", "9821.00"]

It is purely descriptive — it never affects load/detect/alert or the detector id, and it does not change any default-rendered alert message. The synonyms are exposed to alert templates as the opt-in {synonyms} / {synonyms_line} variables (so a custom template can add an “Also known as: …” line), and the whole block is carried in the dtk tune cockpit payload as read-only grounding. A metric with no ai_context behaves exactly as before.

tags (list of strings, optional)

Labels for selecting metrics on the command line. Run all metrics carrying a tag with dtk run --select tag:<tag> (e.g., dtk run --select tag:critical). Tags allow alphanumeric characters, underscores, and dashes; duplicates are rejected.

profile (string, optional)

Database profile to use for this metric. Overrides default_profile from project config.

source_profile (string, optional)

Hybrid mode: the profiles.yml profile whose database runs this metric’s SQL query, while all _dtk_* state (datapoints, detections, task locks, alert state) stays in the active state profile — the one dtk run is already connected to for everything else. Overrides the project-level source_profile; unset on both means this metric’s SQL runs through the state profile itself, like every other step.

name: api_errors
source_profile: warehouse   # only this metric's load step reads from `warehouse`

Not to be confused with profile above — source_profile is the live hybrid-mode field; profile is unrelated and not applied at runtime by dtk run today. See the Hybrid Mode guide for the full config example, what runs where, and error semantics.

enabled (boolean, default: true)

Whether metric is active. Disabled metrics are skipped by dtk run.

Data Loading

interval (string or int, required)

Time interval between data points.

String format:

• "1min", "5min", "10min"
• "1hour", "2hours"
• "1day", "7days"

Integer format (seconds):

• 60 = 1 minute
• 600 = 10 minutes
• 3600 = 1 hour

query (string, optional)

Inline SQL query to load data.

Built-in template variables (Jinja2, substituted by detectkit for every loading batch):

• {{ dtk_start_time }} - Start of time range (inclusive), rendered as YYYY-MM-DD HH:MM:SS
• {{ dtk_end_time }} - End of time range (exclusive), same format
• {{ interval_seconds }} - Metric interval in seconds

Every query must constrain its time range using {{ dtk_start_time }} and {{ dtk_end_time }} — otherwise incremental and batched loading cannot work. The rendered values are plain datetime strings, so wrap them in quotes in SQL.

Required columns:

• Timestamp column (default name: timestamp)
• Metric value column (default name: value)
• Optional seasonality columns (declare them in query_columns.seasonality)

Returned timestamps must already be bucketed to the metric’s interval grid (toStartOfInterval / time_bucket / date_trunc, matching interval) — gap filling maps each row onto the grid by exact timestamp match, and the grid’s phase comes from loading_start_time (or the first --from). A query that returns unbucketed or differently-phased timestamps (e.g. :28 past the hour against a grid phased at :00) can land every row off-grid, which loads as a silently all-NULL batch; if a whole batch misses the grid like this, detectkit logs a one-time warning naming both phases.

Example:

SELECT
  timestamp,
  AVG(response_time_ms) AS value,
  EXTRACT(HOUR FROM timestamp) AS hour_of_day
FROM api_logs
WHERE timestamp >= '{{ dtk_start_time }}'
  AND timestamp < '{{ dtk_end_time }}'
GROUP BY timestamp, hour_of_day
ORDER BY timestamp

query_file (string, optional)

Path to external SQL file (relative to sql_dir).

Mutually exclusive with query.

Example:

query_file: sql/complex_metric.sql

query_columns (object, optional)

Map query column names to internal names.

query_columns:
  timestamp: time_interval      # Query has "time_interval" column
  metric: metric_value          # Query has "metric_value" column
  seasonality:                  # Query has these seasonality columns
    - hour_of_day
    - day_of_week

Defaults:

• timestamp: "timestamp"
• metric: "value"
• seasonality: null

loading_start_time (string, optional)

Start timestamp for initial data load (UTC).

Format: "YYYY-MM-DD HH:MM:SS"

Used only when the metric has no saved datapoints yet. If it is not set and no --from date is passed on the command line, the initial load fails with an error — detectkit does not guess where your data begins. Once datapoints exist, subsequent runs resume from the last saved timestamp and this setting is ignored.

Example:

loading_start_time: "2024-01-01 00:00:00"  # Start from Jan 1, 2024

loading_batch_size (int, optional)

Number of rows to load per batch. Useful for large datasets.

Example:

interval: 10min
loading_batch_size: 2160  # 15 days of 10-min intervals

loading_delay (string or int, optional)

Data-maturity delay: how long after an interval closes the upstream source needs before its data for that interval is complete. Use it when a dbt model (or any other source) finishes writing a few minutes after the interval boundary — without it, a dtk run scheduled right on the boundary loads and permanently persists a partially-written bucket (load only ever resumes forward from the last saved timestamp, so that partial value is never revisited). This causes false “drop” anomaly alerts, false no-data alerts, and a skewed trailing window. Filtering in the SQL itself (WHERE ts < now() - INTERVAL X) does not fix this: the loader decides the upper bound outside SQL and gap-fills the still-missing bucket as NULL, which still trips a no-data alert.

Format: duration string ("10min", "1h") or integer seconds.

Example — a 10-minute metric fed by a dbt model that finishes 5-8 minutes after each interval closes:

interval: 10min
loading_delay: "10min"   # hold back the newest interval until dbt has caught up

Behavior:

• Applies only when the load end bound is implicit (i.e. “now”); an explicit dtk run --to <timestamp> is trusted verbatim and bypasses it.
• The no-data alert expectation shifts back in lockstep, so the deliberately withheld newest interval never fires a false no-data alert.
• Resolution is metric → project-wide default → 0 (see Project Configuration); an explicit loading_delay: 0 on the metric opts it out of a project-wide default.

See “Schedule around upstream data maturity” under Best Practices below for the caveats to weigh before setting this.

Seasonality Extraction

seasonality_columns (list of strings, optional)

Seasonality features auto-extracted from the timestamp for seasonal detection.

Available features:

• hour: Hour of day (0-23)
• day_of_week: Day of week (0=Monday, 6=Sunday)
• day_of_month: Day of month (1-31)
• month: Month (1-12)
• is_weekend: Boolean (Saturday/Sunday)
• is_holiday: Boolean (holiday calendar not implemented yet — always false)

Example:

seasonality_columns:
  - hour
  - day_of_week

These features are stored with each datapoint and can be referenced in detector seasonality_components.

Alternatively, return custom seasonality columns directly from the query and declare them in query_columns.seasonality — query-provided columns take precedence over seasonality_columns.

Detectors

detectors (list, required)

List of detector configurations. Each detector independently analyzes the metric.

Full parameter set for the windowed statistical detectors (mad, zscore, iqr — they share one implementation and accept identical parameters):

detectors:
  - type: mad                     # mad, zscore, iqr, manual_bounds, autoreg
    params:
      # Algorithm parameters (all participate in the detector ID)
      threshold: 3.0              # defaults: mad 3.0, zscore 3.0, iqr 1.5
      window_size: 100            # trailing window in points (current point excluded)
      min_samples: 30             # min valid points in window to run detection
      seasonality_components:     # default: null
        - "hour"                  # single component
        - ["hour", "day_of_week"] # or combined grouping
      min_samples_per_group: 10   # defaults: mad 10, zscore 3, iqr 4
      input_type: values          # values | changes | absolute_changes | log_changes
      smoothing: null             # null | ema | sma
      smoothing_alpha: 0.3        # EMA factor (0, 1]
      smoothing_window: 10        # SMA window in points
      window_weights: null        # null (uniform) | exponential | linear
      half_life: null             # for exponential weights: age at which a point's
                                  # weight halves; int = points or duration string ("3d")
                                  # (default when unset: max(window_size / 20, min_samples / 2))
      detrend: null               # null | linear (robust in-window detrending)

      # Execution parameters (not part of the detector ID)
      start_time: "2024-01-01 00:00:00"   # when detection starts
      batch_size: 500                     # detection batch size

Parameter semantics — the σ-equivalent threshold scaling (MAD is scaled by 1.4826), detector-identity hashing, the deprecated weight_decay alias, and fail-fast validation timing — are documented once in Shared Detector Parameters.

See the Detectors Guide for choosing and tuning a detector.

Alerting

alerting (object, optional)

Alert configuration for the metric.

alerting:
  enabled: true                  # Enable/disable alerting
  suppress_until: null           # Suppress alerts until UTC datetime (default: null)
  timezone: "Europe/Moscow"      # Display timezone (default: UTC)
  channels:                      # List of channel names from profiles.yml
    - mattermost_ops
    - slack_critical

  # Dashboard / runbook links
  dashboard_url: null            # Optional dashboard/runbook URL (default: null)
  links: {}                      # Extra "label: url" links (default: {})

  # Anomaly filtering
  min_detectors: 1               # Detectors that must satisfy the quorum per point (default: 1)
  direction: "same"              # "same", "any", "up", "down" (default: "same")
  consecutive_anomalies: 3       # Consecutive quorum points to trigger (default: 3)

  # Fraction-based alert window (optional; OR-ed with consecutive_anomalies)
  anomaly_window: null           # trailing window, e.g. "30min" (default: null; set with min_anomaly_share)
  min_anomaly_share: null        # fraction in (0, 1] of window points required (default: null)

  # Alert cooldown - Prevent spam from persistent anomalies
  alert_cooldown: "30min"        # Minimum time between alerts
                                 # (default: null = re-alert on EVERY run!)
  cooldown_reset_on_recovery: true  # Reset cooldown when metric recovers (default: true)

  # Recovery notifications
  notify_on_recovery: false      # Send notification when metric stabilizes (default: false)
  template_recovery: null        # Custom recovery message template (default: null)

  # Mentions — tag users/groups in alerts
  mentions: []                   # Plain usernames without @, e.g., ["oncall", "here"]

  # Missing data alert
  no_data_alert: false           # Fire alert when last interval has no row (default: false)
  template_no_data: null         # Custom no-data message template

  # Custom templates
  template_single: null          # Used when consecutive_count <= 1
  template_consecutive: null     # Used for streaks (falls back to template_single)

Multiple alerting blocks

alerting: also accepts a list of blocks. Each block is dispatched independently and carries its own cooldown and alert state, so you can route the same metric to different channels with different rules (e.g., a noisy warning stream plus a strict on-call page).

alerting:
  - enabled: true
    channels: [slack_ops]
    consecutive_anomalies: 1     # warn early
    alert_cooldown: "15min"
  - enabled: true
    channels: [telegram_oncall]
    min_detectors: 2             # page only on a stronger signal
    consecutive_anomalies: 3
    alert_cooldown: "2h"

The single-dict form shown above is still supported and is treated as a list of one block. See the Alerting Guide for full details.

Alert filtering options (see the Alerting Guide for the full contract):

• min_detectors: How many detectors must satisfy the direction policy at every point in the consecutive chain

  • 1 = One qualifying detector is enough
  • 2 = At least 2 detectors must qualify at each point

• direction: Which anomalies count toward the quorum

  • "same" (default) = At least min_detectors detectors must agree on ONE direction at the latest point (up and down counted separately — disagreement is not consensus). The winning direction is locked for the whole consecutive chain.
  • "any" = Every anomaly counts regardless of direction (1 up + 1 down satisfies min_detectors: 2)
  • "up" = Only anomalies above the confidence interval count; “down” anomalies are ignored (they neither help nor block)
  • "down" = Only anomalies below the confidence interval count

• consecutive_anomalies: Consecutive quorum points required

  • 1 = Alert on first anomaly
  • 3 = Alert after 3 consecutive anomalies (reduces false positives)
  • Points must be exactly one metric interval apart — a gap in the detection grid breaks the chain

• anomaly_window / min_anomaly_share: an optional second rule, OR-ed with consecutive_anomalies, for a flapping incident that a strict consecutive chain would reset on one clean point

  • null / null (default) = disabled — must be set together
  • Fires when the latest point meets the quorum AND at least min_anomaly_share (a fraction in (0, 1]) of the trailing anomaly_window grid points do too; missing/no-data points count only in the denominator (an outage makes it harder to fire, not easier)
  • See Alerting → Fraction-Based Alert Window for the full contract

• alert_cooldown: Minimum time between alerts (e.g., "2h", 1800)

  • null (default) = no cooldown: a persisting anomaly re-alerts on every dtk run. Set a cooldown for production metrics.
  • No-data alerts and anomaly alerts share the same cooldown state per alert config block.

• notify_on_recovery: Send notification when metric returns to normal

  • false = No recovery notifications (default)
  • true = Send one recovery notification per incident

• template_recovery: Custom recovery message template

  • Supports the same variables as anomaly templates (incl. {expected_range} and the rule echo {min_detectors} / {direction_policy} / {consecutive_required}), plus {status}
  • Default template (alert-centric): "🟢 Alert cleared: {metric_name}\nThe alert condition no longer holds — the metric is back within expected bounds.\nRule: ...\n..."

• suppress_until: Temporarily suppress alerts until a UTC datetime

  • null = No suppression (default)
  • "2026-04-11 18:00:00" = Suppress alerts until this UTC time
  • Also accepts "2026-04-11 18:00", "2026-04-11T18:00:00" and a bare date "2026-04-11" (midnight UTC); anything else is rejected when the config loads
  • Load and detect steps continue running; only alerting is paused
  • Alerts auto-resume after the specified time — no need to edit config again

• mentions: Users/groups to mention in alerts

  • Plain usernames without @ prefix (e.g., ["oncall_user", "here"])
  • Special keywords: here, channel, all for broadcast mentions
  • Each channel formats mentions in its native syntax
  • Available as {mentions} and {mentions_line} template variables

• dashboard_url: Optional dashboard/runbook URL

  • null (default) — no dashboard link
  • Surfaced as a first-class action on every channel: a clickable attachment title on Slack/Mattermost, an inline “Open dashboard” link on Telegram, and an “Open dashboard” button in email
  • Also available to custom templates as {dashboard_url} and {dashboard_line}

• links: Extra label: url links shown alongside dashboard_url

  • {} (default) — no extra links
  • Each entry is appended as a labelled link, e.g. {Runbook: 'https://...', Grafana: 'https://...'}
  • On webhook channels (Slack/Mattermost/generic) these render as compact clickable labels in one Links line — never raw URLs

• {help_url} / {help_line} (template variables): the “How to read this alert” stakeholder link carried on every alert. This is not a per-metric field — it is set project-wide via alert_help_url in detectkit_project.yml (tri-state: unset → official guide, URL → your runbook, false → hidden); see Configuration → alert_help_url. Available to custom templates as {help_url} / {help_line}, mirroring {dashboard_url} / {dashboard_line}.

alerting:
  channels: [mattermost_ops]
  dashboard_url: https://grafana.ops/d/api-errors
  links:
    Runbook: https://runbooks.ops/api-errors

• no_data_alert: Alert when the latest expected interval has no datapoint

  • false (default) — disabled
  • true — at the alert step, checks _dtk_datapoints for the last complete interval. If no row exists OR the row’s value is NULL / NaN, fires a dedicated alert with status=NO_DATA through the same channels. Honours alert_cooldown and suppress_until.
  • min_detectors and consecutive_anomalies deliberately do not apply — missing data is a single binary signal, not a per-detector vote.
  • Webhook channels render no-data alerts in amber (#F0AD4E) instead of red.

• template_no_data: Custom message body for no-data alerts

  • Default: "No data for metric: {metric_name}\n...Time: {timestamp}\nStatus: query returned no datapoint for the latest interval"
  • Variables: {metric_name}, {timestamp}, {timezone}, {description}, {description_line}, {mentions}, {mentions_line}, {status} (always "NO_DATA")
  • Avoid {value:.2f} / {confidence_interval} — there is no value for no-data alerts. The formatter falls back to the default template if your template uses a numeric format spec on a non-numeric value, but it’s cleaner not to rely on the fallback.

Custom Table Names

tables (object, optional)

Override default table names for this metric.

tables:
  datapoints: _dtk_datapoints_sales
  detections: _dtk_detections_sales

Use cases:

• Separate critical metrics into dedicated tables
• Organize metrics by team or service
• Apply different retention policies

Note: tasks table cannot be overridden (shared across all metrics).

See the Internal Tables reference for the full schema (columns, primary keys, engines) of every _dtk_* table.

Tuning aids

false_alert_budget (float, optional)

A target false-alert rate (FDR) for this metric — a fraction in (0, 1], e.g. 0.3 for “at most 30% of fired alerts should be false”. The dtk tune cockpit flags, gently, when the metric’s false-alert rate exceeds this budget.

false_alert_budget: 0.3

Overrides the project-wide false_alert_budget; unset, the project default (then a built-in 0.5) applies. Tuning-only — it never affects the load/detect/alert pipeline, and labeling stays optional.

Complete Examples

Simple Metric

name: api_errors
interval: 1min
query: |
  SELECT
    timestamp,
    error_count AS value
  FROM logs
  WHERE timestamp >= '{{ dtk_start_time }}'
    AND timestamp < '{{ dtk_end_time }}'
  ORDER BY timestamp

detectors:
  - type: manual_bounds
    params:
      upper_bound: 10

alerting:
  enabled: true
  channels:
    - slack_critical
  consecutive_anomalies: 1  # Alert immediately

Advanced Metric with Seasonality

name: website_traffic
interval: 10min
query_file: sql/traffic.sql

# The query itself returns the seasonality columns
query_columns:
  timestamp: period_time
  metric: visitor_count
  seasonality:
    - hour_of_day
    - day_of_week

loading_start_time: "2024-01-01 00:00:00"
loading_batch_size: 2160  # 15 days

detectors:
  - type: mad
    params:
      threshold: 3.0
      window_size: 8640  # 60 days
      min_samples: 1000
      start_time: "2024-03-01 00:00:00"
      seasonality_components:
        - ["hour_of_day", "day_of_week"]
      min_samples_per_group: 10
      window_weights: exponential  # favor recent data...
      half_life: "3d"              # ...so gradual trends don't cause alert spam

alerting:
  enabled: true
  timezone: "Europe/Moscow"
  channels:
    - mattermost_ops
  min_detectors: 1
  direction: "same"
  consecutive_anomalies: 3

Multiple Detectors

name: cpu_usage
interval: 30s
query: |
  SELECT timestamp, cpu_percent AS value
  FROM system_metrics
  WHERE timestamp >= '{{ dtk_start_time }}'
    AND timestamp < '{{ dtk_end_time }}'
  ORDER BY timestamp

detectors:
  # Hard limit: CPU should never exceed 95%
  - type: manual_bounds
    params:
      upper_bound: 95.0

  # Statistical: detect unusual patterns
  - type: mad
    params:
      threshold: 3.0
      window_size: 2880  # 1 day
      min_samples: 100

alerting:
  enabled: true
  channels:
    - slack_ops
  min_detectors: 1  # Alert if ANY detector triggers
  consecutive_anomalies: 2

Best Practices

1. Use External SQL Files for Complex Queries

# Good: Readable, maintainable
query_file: sql/daily_revenue.sql

# Avoid: Hard to read and maintain
query: |
  WITH daily_sales AS (
    SELECT ...
    FROM ...
    -- 50 lines of SQL
  )
  SELECT ...

2. Set Appropriate Batch Sizes

# 10-minute interval, load 15 days at a time
interval: 10min
loading_batch_size: 2160  # 15 days × 144 intervals/day

Rule of thumb: 7-30 days worth of data per batch.

3. Use loading_start_time for Historical Metrics

# Don't load years of old data unnecessarily
loading_start_time: "2024-01-01 00:00:00"

4. Group Related Metrics

metrics/
├── api_errors.yml
├── api_latency.yml
├── api_throughput.yml
└── database_cpu.yml

5. Use Descriptive Metric Names

# Good
name: api_p95_latency_ms

# Avoid
name: metric1

6. Test Queries Manually First

Before adding to detectkit, test SQL queries in your database client to ensure they return expected data.

7. Document Custom Configurations

Add comments explaining non-obvious settings:

detectors:
  - type: mad
    params:
      threshold: 4.0  # Higher threshold due to noisy metric
      window_size: 8640  # 60 days to smooth seasonality

8. Schedule Around Upstream Data Maturity

Schedule dtk run right after the metric’s interval boundary, and set loading_delay to cover how long the upstream ETL/dbt job typically takes to finish writing that interval’s data:

interval: 10min
loading_delay: "10min"   # dbt model finishes ~5-8 min after the interval closes

This is a trade-off, not a free lunch:

• Every second of delay adds the same amount to real-outage detection time (a genuine no-data condition now fires loading_delay later). Set it to the observed worst-case (p99) upstream latency, not a generous round number.
• It reduces but does not eliminate the race — if the upstream job occasionally takes longer than the configured delay, a partial bucket can still be persisted. Repair it with dtk run --from <date before the bad bucket>, which reloads and overwrites the bad value; no --full-refresh needed.
• Enabling loading_delay does not retroactively fix a bucket already persisted before the setting existed — pair the config change with a one-time dtk run --from <date> over the recent tail if you know a bad value is already there.

See Also

• Detectors Guide - Detector-specific configuration
• Alerting Guide - Alert channels and templates
• Hybrid Mode - Reading metric SQL from a different profile than _dtk_* state
• CLI Reference - Command-line options