Technical Tutorials

The landscape of data orchestration is evolving. While Apache Airflow has long been the de facto standard for scheduling complex data pipelines, its Python-centric DAG model can introduce significant cognitive load and versioning complexity for teams. Enter Kestra, a modern, code-first orchestration tool that shifts the paradigm from imperative logic to declarative, data-driven workflows. This post explores the key differences and provides a roadmap for migrating legacy Airflow DAGs to a Kestra environment.

Understanding the Paradigm Shift

Airflow requires users to write Python code to define DAGs. While powerful, this means that the logic for how to run tasks is interleaved with what the tasks are. Changes to task logic often require code deploys, and the static nature of DAGs can limit dynamic branching capabilities without complex workarounds.

Kestra, conversely, uses YAML to define workflows. This separation of concerns allows business users and engineers to modify pipeline logic without touching the underlying task implementations. In Kestra, tasks are first-class citizens that can be reused across multiple workflows, and the engine handles the orchestration logic natively. This "code-first" approach doesn't mean writing low-level code for every node; it means writing clear, maintainable configurations that map directly to your data flow.

Key Technical Differences

Definition Language: Airflow uses Python; Kestra uses YAML.
Dynamic Tasks: Airflow requires specific patterns (like DynamicTaskMap) for dynamic expansion; Kestra handles dynamic tasks natively via loops and rendering.
State Management: Kestra has built-in state management and output handling, reducing the need for external databases or S3 buckets to pass data between tasks.
Multi-tenancy: Kestra offers native namespace and multi-tenancy support, making it easier to manage isolated environments for different teams or projects.

Migrating a Legacy DAG

Consider a typical Airflow DAG that extracts data from a S3 bucket, transforms it using a Python script, and loads it into a PostgreSQL database.

# legacy_dag.py
from airflow import DAG
from airflow.providers.amazon.aws.operators.s3 import S3ToRedshiftOperator
from airflow.operators.python import PythonOperator
from datetime import datetime

def transform_data(**kwargs):
    # Custom logic here
    pass

with DAG(
    's3_to_pg_pipeline',
    start_date=datetime(2023, 1, 1),
    schedule_interval='@daily',
) as dag:
    extract = S3ToRedshiftOperator(
        task_id='extract_s3',
        aws_conn_id='aws_default',
        s3_key='data/file.csv',
        redshift_table='public.staging_data',
    )
    
    transform = PythonOperator(
        task_id='transform_data',
        python_callable=transform_data,
    )
    
    extract >> transform

Here is the equivalent Kestra YAML workflow:

id: s3_to_pg_pipeline
namespace: com.company.team

tasks:
  - id: extract_s3
    type: io.kestra.plugin.aws.s3.DownloadFile
    url: s3://bucket/data/file.csv
    output:
      - /tmp/file.csv

  - id: transform_data
    type: io.kestra.plugin.shell.exec.ShellTask
    input:
      - /tmp/file.csv
    output:
      - /tmp/transformed.csv

  - id: load_pg
    type: io.kestra.plugin.postgres.Insert
    url: "{{ secrets('POSTGRES_URL') }}"
    input: /tmp/transformed.csv

Practical Migration Steps

Inventory Your DAGs: Identify dependencies, custom operators, and external triggers.
Map Tasks to Kestra Plugins: Leverage Kestra's extensive plugin library. If a custom Python task exists, wrap it in a simple shell or Java/Kotlin plugin if high performance is needed.
Refactor Logic: Move conditional logic from Python code into Kestra's conditional tasks (if, switch). This makes the control flow visible in the UI.
Handle Data Flow: Replace external state stores (like Redis or S3 for intermediate data) with Kestra's native output/input mechanisms.
Test in Parallel: Run both Airflow and Kestra for a period to ensure consistency.

Conclusion

Migrating from Airflow to Kestra is not just a tool swap; it's an opportunity to modernize your orchestration strategy. By embracing a declarative, YAML-based model, teams can reduce boilerplate code, improve visibility into data lineage, and enable faster iterations. While the initial migration requires careful planning, the long-term benefits in maintainability and scalability make Kestra a compelling choice for organizations looking to move beyond the limitations of traditional Python DAGs.