The upgrade lint tells you whether a DAG will import and parse on Airflow 3. It does not tell you whether the DAG still produces the same data. A context key that resolves to a different timestamp at a month boundary, a provider operator with a changed default, a sensor that now times out differently: none of these show up statically. The only way to find them is to run both versions and compare what they write.
This is how we build that comparison.
The shape
Two Airflow environments run the same DAGs over the same input data. The Airflow 2 environment is production, writing to the real targets. The Airflow 3 environment writes to isolated shadow targets: a separate schema, bucket prefix or database. A comparison job runs after each pair of runs and reports differences.
The rule that makes this safe: the Airflow 3 side must never write to a production target or call a side-effecting external system. That means its connections point at shadow schemas, outbound APIs are stubbed or pointed at sandboxes, and email and notification connections are disabled.
Step 1: Classify every DAG by side effect
Before building anything, sort DAGs into three groups:
- Pure transforms: read from sources, write to tables or files you control. These can be dual-run directly.
- External writers: post to APIs, send email, push files to partners, trigger billing. These need stubs or sandbox endpoints on the Airflow 3 side, or they are validated by comparing the payload that would have been sent rather than sending it.
- Destructive maintenance: deletes, truncates, vacuum jobs. These are validated in a separate test environment, never dual-run against anything shared.
Step 2: Parameterise targets
Most estates hard-code schema names or bucket paths somewhere. Move them behind a connection, variable or environment setting so the same DAG code writes to analytics in one environment and analytics_shadow in the other. This is worth doing anyway; it is also the change that makes the dual run possible.
Step 3: Pin the inputs
Both sides need to read the same data. For warehouse sources, that usually means both runs read the same snapshot or the same bounded date range. For sources that move under you, such as APIs and streaming tables, the Airflow 2 run can land its raw extract and the Airflow 3 run reads that landed copy instead of calling the source again. Otherwise you will spend days chasing differences that are just timing.
Step 4: Decide what "the same" means
Per target table, choose comparisons that fit the data:
- Row counts per partition: cheap, catches gross errors.
- Aggregate checksums over key columns, for example a sum of a hash of the business key and the measures, per partition. Catches most real differences without moving data.
- Key-level diffs for tables where a count can match while contents differ: an anti-join on the business key in both directions, plus a column-by-column comparison on matching keys.
- Run-state comparison: did the same tasks succeed, fail, skip and retry? A task that skips on Airflow 3 but ran on Airflow 2 often points at a changed trigger rule or context value.
Exclude columns that legitimately differ, such as load timestamps and run identifiers, explicitly and in writing, so nobody later wonders why they were ignored.
Step 5: Run it as a DAG
The comparison itself is just another DAG, running on the Airflow 3 side after each pair of runs, with its results written to a small report table: DAG, target, partition, check, expected, actual, status. A dashboard over that table becomes the cutover readiness view.
Step 6: Run long enough
A dual run needs to cover the cycles that matter to the business: at least one month end for anything financial, a full weekly cycle for anything with weekend behaviour, and any quarterly or annual job you can trigger with a test date. A week of clean nightly runs proves less than it seems if the bug is at a period boundary.
Reading the differences
Most differences fall into a few buckets: date semantics (the data interval versus the old execution date), changed operator defaults in newer provider versions, timezone handling, and ordering that was never guaranteed but happened to be stable. Each one should be either fixed or explicitly accepted with a note. An unexplained difference is a cutover blocker.
The real-estate caveat
A dual run costs compute and attention for weeks, and on a large estate you will not dual-run everything with the same rigour. Rank DAGs by business impact and give the top of the list full key-level diffs, the middle row counts and checksums, and the long tail a run-state comparison. And if a DAG produces data nobody can say they use, the dual run is a good moment to switch it off on both sides and see who notices.