On a small estate nobody thinks about DAG parsing. On an estate with several hundred DAG files, parsing quietly becomes the thing that limits everything else: edits take minutes to appear, DAG import errors linger, and the processor's CPU stays pinned. Airflow re-parses DAG files continuously, so a file that takes ten seconds to import costs ten seconds every cycle, forever.
The good news is that parse time is usually concentrated in a small number of files, and the fixes are well understood.
Measure first
Airflow records how long each file takes to process. The airflow dags report command lists parse duration per file, and the DAG processor emits per-file timing metrics if you ship them to your metrics system. Sort by duration. On most estates we look at, a handful of files account for most of the total.
Also note the total: the sum of all file parse times divided by the number of parsing processes is roughly the minimum time for one full pass over the folder. If that is longer than your file processing interval, the processor can never catch up.
Cause 1: Work at module level
Anything at the top level of a DAG file runs on every parse. The usual offenders:
- Calling external systems: querying a database, calling an API, listing a bucket to decide which tasks to generate.
- Reading Variables at import time: each
Variable.getat module level is a round trip on every parse. - Heavy imports: data libraries and SDKs that take seconds to import, used only inside a task.
Fix: move the work into the task. Imports used only inside a callable go inside the callable. Values from Variables go into templated fields, which are resolved at run time, not parse time. If a DAG's shape genuinely depends on external data, generate a static config file from that data on a schedule and have the DAG read the file, which is fast and deterministic.
Cause 2: Dynamic DAG factories that do too much
One file that generates many DAGs from a config is a reasonable pattern, until the generator reads dozens of YAML files, validates them against a remote schema and builds hundreds of DAGs on every parse.
Fix: keep the generator, but make it cheap: local files only, no network, and cache anything expensive. Consider splitting one giant generator file into several, so a change to one team's config does not force a slow re-parse of everyone's DAGs.
Cause 3: Files that are not DAGs
Helper modules, tests, SQL files and notebooks sitting in the DAG folder all get scanned.
Fix: use .airflowignore to exclude anything that is not a DAG definition, and keep shared code in a proper package outside the DAG folder.
Tuning the processor
Once the files themselves are fast, the processor settings matter:
- Parsing processes: more processes parse more files in parallel, up to the CPU you give the processor.
- File processing interval: how often each file is re-parsed. On a large, stable estate, a longer interval reduces load at the cost of slower pickup of changes.
- Folder scan interval: how often the folder is listed for new files.
In Airflow 3, the DAG processor runs as its own component, so it can be sized and scaled independently of the scheduler. That is worth using: give it the CPU it needs instead of letting it compete with scheduling.
Watch the scheduler, too
Parsing problems often masquerade as scheduler problems: tasks sit in the scheduled state, and the instinct is to add scheduler replicas. Check parse duration and the DAG processor's CPU before you scale the scheduler. If the processor is saturated, more schedulers only add contention on the metadata database while the real bottleneck stays where it was.
What good looks like
There is no universal target, but a healthy large estate parses every file in well under a second, with total pass time comfortably below the processing interval. New and edited DAGs appear within a minute or two, and the import errors page is empty.
The real-estate caveat
Parse-time work is easy to undo. A month after a cleanup, someone adds a module-level API call and the problem returns. The durable fix is a check in CI that imports each changed DAG file and fails if it takes longer than a threshold you agree, plus a review standard that says why. And if the slowest file belongs to a DAG nobody runs any more, deleting it is the fastest optimisation available.