Airflow Consulting Services
Four rungs, each usable on its own.
What we do on Apache Airflow
Most teams that call us have one of three problems: an estate still on Airflow 2, an upgrade that started and stalled, or a scheduler that misses its windows and pages someone at 3am. The work below is ordered the way it usually happens, but every rung stands on its own. If you only need the assessment, you get a plan you can execute yourselves.
Airflow Estate Assessment
A structured read of the whole estate: DAG inventory and dependency map, scheduler and executor topology, provider and operator drift, deprecated API usage, Airflow 2 end-of-life exposure, secrets and connection handling, and SLA and failure history. The output is a written findings document with a prioritised, sequenced remediation and upgrade plan, plus an honest call on which DAGs should be migrated, rewritten or retired.
Airflow 2 to Airflow 3 Migration
Hands-on migration of a production estate to a supported 3.x line. We clear the upgrade checks, remove direct metadata-database access from task code, move DAGs to the Task SDK and current provider packages, adapt deployment and executor configuration, and then dual-run old and new side by side with output comparison before cutover. Rollback plan and post-cutover stabilisation are part of the job.
Pipeline Reliability and Cost Engineering
For estates already on a supported version that still hurt: scheduler and parsing-time tuning, concurrency and pool design, deferrable operators and sensor cleanup, backfill and catchup strategy, retry and alerting policy that separates real failures from noise, Kubernetes executor sizing, and right-sizing of managed environments.
Retained Airflow Support and Staff Augmentation
An embedded senior engineer alongside your data team: on-call backup for nightly DAG failures, routine minor-version and provider upgrades, review of new DAGs against agreed standards, and onboarding for your engineers.
How engagements are billed
Hourly time and materials. Senior US engineers billed by the hour, as project engagements or staff augmentation; scope, staffing and timeline are agreed through an initial conversation. We do not quote per DAG.