Dagvane

What Breaks When You Move an Airflow 2 DAG to the Task SDK

Run the Airflow 3 upgrade lint rules over a production DAG folder and you will usually get a long list. On estates of a few hundred DAGs, findings in the high hundreds are normal. The number is less alarming than it looks, because most findings are the same handful of problems repeated. What matters is sorting them early into three piles: mechanical edits, real rewrites, and DAGs that should be retired instead of fixed.

This piece walks through the findings we see most, and which pile each belongs in.

Start with the lint, not the upgrade

The Airflow project publishes upgrade rules for the Ruff linter (the AIR3 family). Run them against the DAG folder, the plugins folder and any internal library your DAGs import, not only the DAG files themselves. Shared helper modules are where the surprising findings hide, because a single deprecated import in a helper shows up as one finding but breaks every DAG that uses it.

Write the output to a file and group it by rule and by file. You want two views: which rules fire most (that is your mechanical work), and which files have the most distinct rules firing (that is where the rewrites are).

The mechanical pile

These are safe to fix in bulk, ideally with automated fixes plus review:

  • Imports that moved. DAG authoring now comes from the Task SDK (airflow.sdk): DAG, the @dag and @task decorators, Variable, Connection and friends. Old import paths from airflow.models and similar modules are flagged.
  • Standard operators moved to a provider. BashOperator, PythonOperator and the other everyday operators now live in the standard provider package. The import path changes and the package must be installed.
  • schedule_interval and timetable arguments. Both are replaced by a single schedule argument.
  • Renamed context keys. execution_date and the derived keys built on it (next_ds, prev_ds, tomorrow_ds and so on) are gone from the task context. Most uses map onto logical_date or the data interval, but check the semantics, not just the name: a DAG that used next_execution_date to mean "the end of the period I am processing" should use data_interval_end.
  • Datasets are now Assets. The concept is the same; the names changed.

A mechanical finding in a DAG with no other findings is genuinely a quick fix. The same finding in a DAG with five other rules firing is a sign the DAG needs a closer look.

The rewrite pile

These need a human who understands what the DAG is for.

Task code that talks to the metadata database

In Airflow 3, tasks run through the Task Execution API and no longer get a database session. Any task that imports the ORM models, opens a Session, or queries DagRun or TaskInstance directly will fail. These are rarely flagged as cleanly as an import, because the access is often buried in a helper. Search for provide_session, create_session, settings.Session and imports from airflow.models inside callables, not just at module level. We wrote a separate piece on replacing these patterns.

SubDAGs

SubDAGs were removed. The replacement is usually a task group, sometimes a separate DAG triggered by an asset, sometimes nothing because the SubDAG existed to work around a limitation that no longer applies. Each one is a design decision.

Custom operators that reach into internals

Custom operators often subclass something internal, read attributes that were never public, or call private methods on the task instance. These break in ways the lint cannot see. Read every custom operator and hook in the plugins folder before estimating.

SLAs

The 2.x SLA mechanism is not part of Airflow 3. If on-call depends on SLA miss emails, that alerting has to be replaced before cutover, not discovered missing afterwards.

REST API clients

Anything outside Airflow that calls the stable REST API (internal tooling, CI jobs, a dashboard someone built) needs to move to the new API version. These clients are not in the DAG folder, so the lint never sees them. Grep your other repositories for the API path.

The retire pile

Every finding list contains DAGs nobody has looked at in a year. Before fixing anything, cross-reference the findings with run history: DAGs that are paused, that have not succeeded in months, or whose output tables nobody queries. Porting them is pure cost. Retiring them shrinks the migration and the estate at the same time.

The order we fix things

  1. Retire what can be retired.
  2. Fix shared helpers and plugins first, because every DAG depends on them.
  3. Apply mechanical fixes in bulk, one rule at a time, with a review per rule rather than per file.
  4. Work the rewrite pile DAG by DAG, starting with the ones the business depends on most, so they get the longest dual-run time.
  5. Re-run the lint until it is clean, then run every DAG in a non-production Airflow 3 environment, because the lint only catches what it can see statically.

The real-estate caveat

A clean lint is a starting line, not a finish line. Lint cannot tell you whether a DAG still produces the same data, whether a renamed context key now resolves to a different date at a period boundary, or whether a custom operator relied on behaviour that changed. That is why we never cut over without a dual run that compares outputs. And on some estates, the honest finding from this exercise is that a large share of the DAGs should not move at all.