Pipeline Reliability and Cost Engineering
For estates on a supported version that still page you.
When the upgrade is done but the estate still hurts
Being on a supported version does not make an estate reliable. These are the problems we are usually called in for, and what we do about them.
Scheduler and parsing time
Hundreds of DAG files with heavy top-level code make the DAG processor the bottleneck. We profile parse times per file, move work out of module scope, tune the processor settings, and use ignore files to keep non-DAG code out of the parse loop.
Concurrency and pools
Pools, per-DAG concurrency and per-task limits decide whether a warehouse gets hammered at 2am. We design them from what the downstream systems can actually absorb.
Sensors and deferrable operators
Polling sensors hold a worker slot while they wait. Replacing them with deferrable operators moves the wait to the triggerer and frees worker capacity, often enough to shrink the worker fleet.
Backfill and catchup
A backfill strategy that does not take the warehouse down: bounded concurrency, explicit date ranges, and idempotent tasks that can be re-run safely.
Retry and alerting policy
Retries that hide real failures and alerts that fire on every transient blip both teach on-call to ignore the channel. We separate the two, so a page means something.
Executor and environment sizing
Kubernetes executor pod templates and resource requests, worker autoscaling, and right-sizing of Amazon MWAA and Google Cloud Composer environments so the orchestration bill matches the workload.
Billed hourly. Tell us what is paging you.