Look at a busy Airflow estate at 2am and a large share of the running tasks are often doing nothing: sensors waiting for a file to land, a partition to appear or an upstream DAG to finish. In the default mode, each of those sensors holds a worker slot while it waits. On estates with many sensors, worker capacity is sized for waiting, not working.
Deferrable operators fix this by handing the wait to the triggerer, a separate component that runs many lightweight async waits in one process. The task frees its worker slot while deferred and resumes when the trigger fires.
Why not just use reschedule mode?
Sensors have long supported mode="reschedule", which releases the worker slot between pokes. It helps, but each poke is still a full task execution: schedule, queue, start a process or pod, run the check, exit. With short poke intervals across many sensors, that churn becomes its own load on the scheduler and executor, and on the Kubernetes executor it means a pod start per poke. Deferral avoids that: the trigger polls inside the triggerer without involving the scheduler until the condition is met.
Which sensors to convert first
Rank sensors by total time spent waiting, not by count. The run history tells you: duration of sensor task instances, summed over a week. Typically the list is led by:
- External task sensors waiting on another DAG.
- Object storage key sensors waiting for a file to land.
- SQL sensors waiting for a partition or a row.
- Long timeouts: anything configured to wait for hours.
Many built-in and provider sensors accept deferrable=True. For those, the conversion is one argument plus testing. Custom sensors need a trigger written for them, which is more work; convert the high-wait ones and leave short, rare custom sensors in reschedule mode.
Run the triggerer properly
A deferred task is only as reliable as the triggerer it is waiting in.
- Run at least two triggerers in production. Triggers are redistributed if one dies, but with one triggerer a restart pauses every deferred wait.
- Size it for its workload. Triggers are async coroutines; well-behaved ones are cheap, but a trigger that does blocking I/O stalls every other trigger in the process. Watch the triggerer's own metrics for blocked loops.
- Check managed platform support. Amazon MWAA and Google Cloud Composer both support the triggerer, but how it is enabled and sized differs; check your environment configuration rather than assuming.
What it does to worker count
The effect is easiest to see by comparing peak concurrent running tasks before and after conversion. On estates where sensors dominate the overnight window, the peak falls substantially and worker autoscaling settles at a lower level. On the Kubernetes executor, the pod churn from reschedule-mode pokes disappears too. The saving is real money on managed environments billed by worker size and count, but measure your own estate: an estate with few long waits will see little change.
Things that go wrong
- Timeouts behave differently. Read how the timeout is enforced for the deferrable version of each sensor; do not assume it matches the old poke-based timeout.
- Custom triggers that block. A synchronous database call inside a trigger blocks the event loop for every other trigger. Use async clients or move the check out of the trigger.
- Silent waits. A deferred task looks quiet. Make sure alerting covers tasks that have been deferred far longer than normal, or a missing upstream file will not page anyone until the downstream report is late.
A conversion checklist
For each sensor you convert:
- Confirm the deferrable version exists for your provider version, and pin that version.
- Read its timeout, retry and soft-fail behaviour, and set them explicitly rather than relying on defaults.
- Run it in a non-production environment against a condition that is met, one that is met late, and one that is never met.
- Watch the triggerer's logs and metrics during that test, not only the task's.
- Compare peak running tasks for the DAG before and after, so the saving is measured rather than assumed.
The real-estate caveat
Before converting a sensor, ask whether the wait should exist at all. Many sensors that poll for an upstream DAG are better replaced by asset-based scheduling, where the downstream DAG simply runs when its input is produced. That removes the wait entirely, which beats making it cheaper.