Airflow 2 to Airflow 3 Migration

Airflow 3 changed the contract between task code and the platform. Tasks talk to the API server instead of the metadata database, DAG authoring moved to the Task SDK, and a long list of deprecated imports and context variables is gone. On a real estate that means rewriting code, not editing a requirements file.

The sequence

1. Clear the upgrade checks. Run the Airflow 3 lint rules across the DAG folder and plugins, and triage every finding into mechanical fixes, real rewrites and DAGs that should be retired instead.
2. Remove direct metadata-database access from task code. Anything that opens a session against the metadata database from inside a task has to move to the supported API or be redesigned. This is usually the hardest part of the estate.
3. Move DAGs to the Task SDK and current providers. Update imports, replace deprecated operators, and move the standard operators to their provider package.
4. Adapt deployment and executor configuration. The API server, DAG processor and triggerer are deployed and sized for the new architecture, whether that is self-managed Kubernetes, Amazon MWAA or Google Cloud Composer.
5. Adopt new features only where they earn their keep. DAG versioning and asset-based scheduling are useful on some estates and noise on others. We decide per DAG.
6. Dual-run before cutover. Old and new environments run the same DAGs against isolated targets, and a comparison harness diffs row counts, checksums and run state before anyone switches traffic.
7. Cut over with a rollback plan, then stabilise. Cutover is scheduled around your batch calendar, with a written rollback path. We stay through the first weeks of production runs.

What we need from you

Access to the repository and a non-production environment, a person who knows which DAGs the business actually depends on, and the batch calendar so cutover avoids month-end.

If your estate is on Airflow 2 today, read what end of life means for it first. Billed hourly; talk to us about scope.