Mastering Data Lineage and State Management in Kestra for Production ETL

In the modern data engineering landscape, reliability is paramount. While many orchestration tools focus primarily on task execution, Kestra distinguishes itself by treating workflow state and data lineage as first-class citizens. For production-grade Extract, Transform, Load (ETL) pipelines, understanding how data flows through your system and managing the state of your workflow execution are critical for debugging, auditing, and maintaining data integrity. This post explores how to leverage Kestra’s native capabilities to build resilient, observable, and state-aware ETL pipelines.

The Core Challenge: State and Lineage in Distributed Systems

Traditional ETL pipelines often suffer from "black box" syndrome. When a job fails, engineers struggle to pinpoint whether the issue lies in the source data, the transformation logic, or the sink configuration. Furthermore, without clear lineage, impact analysis becomes a manual, error-prone process. Kestra addresses these challenges by providing built-in support for tracing every input and output variable across tasks, while simultaneously managing the lifecycle of the workflow execution state.

Implementing Granular Data Lineage

Kestra automatically tracks the lineage of variables. However, to maximize this feature, you must design your flows with explicit input and output definitions. Consider a typical ETL scenario where we extract data from a database, transform it using Python, and load it into a data warehouse.

By utilizing Kestra’s triggers and task outputs, we can visualize exactly how data moves through our pipeline. Here is a practical example of a flow that demonstrates explicit variable passing:

id: production_etl_pipeline
namespace: company.data

tasks:
  - id: extract
    type: io.kestra.plugin.jdbc.postgresql.Query
    url: "{{ secrets.POSTGRES_URL }}"
    sql: "SELECT * FROM raw_events WHERE created_at > '{{ prevRunDate.date('yyyy-MM-dd') }}'"
    
  - id: transform
    type: io.kestra.plugin.scripts.python.Script
    runner: DOCKER
    script: |
      import json
      import sys
      # Extract input data
      data = json.loads(inputs.raw_data)
      # Transform logic
      cleaned = [x for x in data if x['value'] is not None]
      # Output as a new variable for lineage tracking
      outputs.cleaned_data = json.dumps(cleaned)
    inputs:
      raw_data: "{{ taskrun.extract.outputs.rows }}"

  - id: load
    type: io.kestra.plugin.jdbc.postgresql.BulkOutput
    url: "{{ secrets.POSTGRES_URL }}"
    schema: public
    table: cleaned_events
    columns:
      - id
      - value
      - created_at
    from: "{{ outputs.transform.cleaned_data }}"

In this example, Kestra automatically traces the lineage from the extract task to the transform task, and finally to the load task. This allows you to click through the UI and see exactly which rows influenced a specific output, which is invaluable for debugging data discrepancies.

Advanced State Management and Retry Logic

Production pipelines are subject to transient failures—network timeouts, API rate limits, or temporary database locks. Kestra’s state management allows you to define granular retry policies and error handling strategies without cluttering your main workflow logic.

Instead of relying on external tools like Airflow for state persistence, Kestra stores execution state in its native storage backend (Postgres or S3). You can leverage errors blocks to handle failures gracefully:

- id: load_retry
  type: io.kestra.plugin.jdbc.postgresql.BulkOutput
  # ... configuration ...
  retry:
    type: FIXED
    interval: 30s
    maxDuration: 5m
    maxAttempt: 3
  errors:
    - type: io.kestra.plugin.core.log.Log
      message: "Final load attempt failed. Notifying Slack."

This approach ensures that your pipeline is resilient by default. The state is updated atomically, meaning you can pause, resume, or kill a workflow mid-execution, and Kestra will maintain a consistent record of what has been processed.

Conclusion

Implementing complex data lineage and state management in Kestra transforms your ETL pipelines from fragile scripts into robust, enterprise-grade systems. By explicitly defining inputs, leveraging automatic lineage tracing, and configuring granular retry policies, you gain the observability needed to maintain trust in your data infrastructure. As your data needs grow, Kestra’s declarative YAML approach and powerful plugin ecosystem ensure that your workflows remain maintainable and scalable.