Dagvane

Designing Airflow Retry and Alert Policy So On-Call Stops Ignoring the Channel

The usual state of an Airflow alert channel is a few hundred messages a week, almost all of them noise, and one real failure buried among them. On-call learns to ignore it. This post describes a policy that makes the channel worth reading again. It assumes you already know how to set retries; the work is in deciding what each failure should do.

Start by classifying failures

Before changing configuration, pull ninety days of task failures from your logs or alerting history and sort them into three groups.

  1. Transient. Network resets, throttled API calls, a warehouse queue timeout, a pod evicted. A retry usually fixes these. Nobody should be told unless retries run out.
  2. Deterministic. A schema change, an expired credential, a bad config value, a code bug. Retrying wastes time and slots. These should fail at once and alert once.
  3. Late or missing. Nothing failed, but the data did not arrive when a consumer needed it. No task-failure alert catches this.

In most estates we read, group 1 is the largest by count and group 2 is the largest by cost. Group 3 is the one that reaches the business.

Set retries per failure class, not globally

A global retries: 3 in default_args treats a throttled API and a dropped table the same. Set defaults conservatively, then override where the failure mode is known.

default_args = {
    "retries": 2,
    "retry_delay": timedelta(minutes=2),
    "retry_exponential_backoff": True,
    "max_retry_delay": timedelta(minutes=30),
    "execution_timeout": timedelta(hours=2),
}

Two details matter. First, exponential backoff only helps if the delay is longer than the outage you are waiting out. Two minutes is fine for a throttled API and useless for a database failover. Second, always set execution_timeout. Without it, a hung task holds a slot until someone notices, and it never reaches the retry logic at all.

For deterministic failures, raise AirflowFailException from the task. It marks the task failed without consuming remaining retries. Catch the specific exceptions you know are permanent (authentication errors, missing objects, validation failures) and convert them. Do not wrap everything in a broad except; that hides transient errors you want retried.

Alert on the final outcome, not each attempt

on_failure_callback fires when a task reaches its failed state, after retries are exhausted. on_retry_callback fires on every retry. Most noisy channels are wired to the second one, or to email-on-retry. Remove that. A task that fails once and succeeds on retry is the system working.

If you want visibility into retry rates, count them in metrics and review the trend weekly. A task that retries on every run is a defect to fix, but it is not a 3am event.

At the DAG level, prefer one alert per DAG run over one per task. When an upstream task fails, downstream tasks end as upstream_failed. Alerting on those produces ten messages for one fault. Alert on the root failure, and include the DAG, run date, task ID, try number and a log link in the message so the reader does not need to open the UI to triage.

Route by consequence

Not every DAG deserves the same channel. Tag each DAG with a tier, in tags or in your own metadata:

  • Tier 1: feeds billing, finance reporting or customer-facing data. Failure pages on-call.
  • Tier 2: feeds internal dashboards. Failure posts to a channel, read next business morning.
  • Tier 3: experiments and one-off loads. Failure goes to the owning team only, or nowhere.

The tiering exercise is also where you find DAGs nobody owns. Those are candidates for retirement, not for an alert route.

Catch late data separately

Task-failure alerts cannot tell you a DAG never started because the scheduler was down, or that a sensor is still waiting. You need a check that asks whether the expected output exists by a deadline.

Airflow 2 had a sla parameter and sla_miss_callback. It was unreliable: it was evaluated by the scheduler, produced duplicate callbacks, and did not behave well with manual runs. Airflow 3 removed it. Newer 3.x releases add deadline alerts as a replacement, but check the exact release you run, because that feature is recent and its behaviour has changed between minors. If you are mid-migration, do not assume your SLA alerting carried over. It did not.

A dependable alternative is independent of Airflow's own state: a small scheduled check, outside the orchestrator if possible, that queries the target table for the latest load timestamp and alerts if it is older than the consumer's deadline. It also catches the case where Airflow itself is down. The caveat is that you now maintain one more thing, and the deadline values drift out of date unless someone owns them.

Add a rate limit and a summary

Two cheap measures finish the job. Deduplicate alerts by DAG and task within a window, so a flapping task sends one message an hour. And send a daily digest of Tier 2 and Tier 3 failures, so those get read without interrupting anyone.

Roll it out in order

  1. Remove alerts on retry and on upstream_failed.
  2. Add execution_timeout everywhere.
  3. Convert known permanent errors to AirflowFailException.
  4. Tier the DAGs and route by tier.
  5. Add the late-data check for Tier 1.

Measure alert volume per week before and after. A good result is a drop of an order of magnitude in messages with no increase in incidents found by a business user first.

The caveat

This policy depends on someone owning the tiers. On an estate where the original authors left, you will not know which DAGs are Tier 1 without asking the consumers, and the answers will be inconsistent. Do that conversation first. Also, tightening retries can expose failures that retries were silently covering, so expect the first two weeks to show more real failures, not fewer. That is the policy working, but warn the team.