What is Kestra? This beginner’s guide explains Kestra’s YAML-based orchestration model, core concepts, architecture, and how it compares to Airflow and Dagster.

If you’ve looked into Airflow, Dagster, or Temporal and found yourself thinking “there has to be a simpler way to describe a workflow than writing this much Python,” Kestra might be exactly what you’re after. So what is Kestra, and why has it been picking up momentum among data and platform teams over the past couple of years? This guide walks through everything a beginner needs to know, from the core concept to a working example you can run yourself.

What is Kestra?

Kestra is an open-source orchestration platform for building, running, and monitoring workflows, using YAML instead of a general-purpose programming language as its primary interface. Where tools like Airflow, Dagster, and Temporal ask you to write Python or Go code to define a pipeline, Kestra lets you describe a workflow as a declarative YAML file, listing the tasks, their order, and any conditions or triggers involved.

This might sound like a small difference, but it changes who can actually build and maintain workflows. A YAML file is readable by data analysts, platform engineers, and even non-technical stakeholders who’d never touch a Python codebase. Kestra calls its core unit of work a “flow,” and a flow can orchestrate almost anything: data pipelines, infrastructure automation, business processes, or scheduled scripts, all defined the same declarative way.

Kestra is built by a company of the same name, and the project has grown quickly within the broader data orchestration space, positioning itself as an alternative to Airflow that trades some of Python’s flexibility for simplicity, readability, and a lower barrier to entry.

The Problem Kestra Solves

To understand why Kestra exists, it helps to look at what teams were running into with earlier orchestration tools.

Airflow, for years the default choice for scheduling data pipelines, requires you to write DAGs in Python. That’s fine for engineers, but it creates a real barrier for anyone else on a data team who wants to understand, adjust, or troubleshoot a pipeline. A one-line schedule change often means opening a Python file, understanding decorators and imports, and redeploying code, even for what’s conceptually a trivial edit.

There’s also the operational overhead. Airflow’s scheduler, webserver, and metadata database setup has a reputation for being genuinely complex to run well at scale, with plenty of teams spending real engineering time just keeping Airflow itself healthy, separate from the actual pipelines running on top of it.

Kestra’s answer to both of these problems is to make the workflow definition language itself simpler (YAML instead of Python) and to build the platform around a modern, API-first architecture from the start, rather than one that grew organically over a decade like Airflow’s did. The result is a tool that’s genuinely approachable for someone writing their first workflow, while still scaling to handle complex, production-grade automation.

How Kestra Works: Core Architecture

Kestra’s architecture is built around a handful of core pieces, each with a specific job.

Flows

A flow is Kestra’s term for a workflow, defined entirely in YAML. A flow specifies an identifier, a namespace (used for organizing flows, similar to a folder structure), and a list of tasks to execute, along with any triggers that should kick it off automatically. Because flows are just YAML files, they can be version-controlled in Git, reviewed like any other code change, and deployed through standard CI/CD pipelines.

Tasks

Tasks are the individual steps inside a flow, the actual units of work. Kestra ships with a large library of built-in tasks and plugins covering things like running shell scripts, querying databases, calling APIs, moving files between cloud storage providers, and orchestrating containers directly. Tasks execute in the order defined in the flow, or according to whatever dependency structure you specify.

Triggers

Triggers determine when a flow runs. Kestra supports several trigger types: schedule triggers based on cron expressions, similar to a traditional cron job; flow triggers that fire when another flow completes, letting you chain pipelines together; and webhook or event-based triggers that fire in response to an external signal, like a file arriving in cloud storage or a message on a queue.

The Executor

The executor is the component responsible for actually running tasks, whether that’s executing a shell command directly, spinning up a Docker container, or delegating to a Kubernetes pod. Kestra supports multiple execution models, so the same flow definition can run tasks locally during development and in isolated containers in production, without changing the flow itself.

The Kestra Server and UI

The Kestra server coordinates everything: storing flow definitions, scheduling triggers, tracking execution history, and serving the web UI. The UI is one of Kestra’s stronger beginner-friendly features, showing a visual timeline of every execution, the exact output and logs of each task, and a topology view showing how tasks in a flow relate to each other.

Key Concepts You’ll Run Into

A few additional terms come up constantly once you start building with Kestra.

Namespaces organize flows into logical groups, similar to folders or projects, which matters once you have more than a handful of flows running.

Inputs let a flow accept parameters at execution time, so the same flow definition can behave differently depending on what’s passed in when it runs, similar to function arguments.

Outputs let a task pass data forward to later tasks in the same flow, allowing information (a computed value, a file path, an API response) to flow through the pipeline.

Subflows let you call one flow from within another, useful for breaking a large, complex process into smaller, reusable pieces, the same idea as functions calling other functions.

Plugins extend Kestra with integrations for specific tools and services, databases, cloud providers, messaging systems, and more. Kestra maintains an extensive plugin ecosystem, which is a large part of why teams can adopt it without writing custom integration code for common systems they already use.

Backfills let you re-run a flow across a historical date range, useful when you’ve fixed a bug and need to reprocess data that was previously handled incorrectly.

A Simple Example of a Kestra Flow

Here’s what a basic flow looks like in practice, a simple pipeline that downloads a file, processes it, and sends a notification:

id: process_daily_orders
namespace: company.orders

tasks:
  - id: download_orders
    type: io.kestra.plugin.core.http.Download
    uri: https://example.com/orders/daily.csv

  - id: process_orders
    type: io.kestra.plugin.scripts.python.Script
    script: |
      import pandas as pd
      df = pd.read_csv("{{ outputs.download_orders.uri }}")
      summary = df.groupby("region")["amount"].sum()
      print(summary)

  - id: notify_completion
    type: io.kestra.plugin.notifications.slack.SlackIncomingWebhook
    url: "{{ secret('SLACK_WEBHOOK_URL') }}"
    payload: |
      {"text": "Daily order processing completed successfully"}

triggers:
  - id: daily_schedule
    type: io.kestra.plugin.core.trigger.Schedule
    cron: "0 6 * * *"

Notice how much is expressed without writing a traditional program. The download_orders task fetches a file, process_orders runs a Python script against it (Kestra can execute Python, even though the flow itself is YAML), and notify_completion sends a Slack message once everything succeeds. The triggers section schedules this entire flow to run daily at 6 a.m., using the same cron syntax you’d find in traditional scheduling tools.

The {{ outputs.download_orders.uri }} syntax is Kestra’s templating system, referencing the output of an earlier task, which is how data flows between steps without you manually wiring variables together in code.

Kestra vs Other Orchestration Tools

Since Kestra enters a space with several established players, it’s worth comparing directly.

Kestra vs Airflow: Airflow requires Python for every DAG, which means every workflow change is a code change. Kestra’s YAML-first model lowers that barrier substantially, letting more people on a team read, write, and modify flows without deep Python knowledge. Airflow has a longer track record and a larger existing plugin ecosystem built up over a decade, which is worth weighing against Kestra’s simpler learning curve.

Kestra vs Dagster: Dagster is asset-centric and deeply Python-based, built specifically around the idea of data assets with strong lineage and testing support. Kestra is more general-purpose, covering data pipelines but also infrastructure automation and business process workflows, with less emphasis on the asset abstraction Dagster is built around. Teams focused specifically on data asset lineage and testing may prefer Dagster; teams wanting a broader, simpler automation tool across many kinds of workflows may prefer Kestra.

Kestra vs Temporal: Temporal is a code-first, durable execution engine built for long-running business processes, with workflows written in a full programming language and a strong emphasis on determinism and fault tolerance for complex state machines. Kestra’s YAML model is far simpler to pick up but doesn’t provide the same level of fine-grained programmatic control that Temporal offers for genuinely complex, code-heavy business logic. If you want to go deeper on that comparison, our What is Temporal guide covers Temporal’s model in detail.

Kestra vs n8n or Zapier-style tools: Visual, no-code automation tools are great for simple integrations between SaaS apps, but they tend to struggle with anything involving heavier data processing, custom scripting, or complex conditional logic. Kestra sits a level above these tools in terms of power, while still remaining far more approachable than a code-first orchestrator.

Common Use Cases for Kestra

Kestra shows up across a fairly wide range of automation needs, given its general-purpose design.

• ETL and data pipeline orchestration, moving and transforming data between databases, APIs, and cloud storage on a schedule
• Infrastructure automation, running provisioning scripts, cleanup jobs, or deployment steps in response to triggers
• Business process automation, chaining together approval steps, notifications, and system updates without writing custom application code
• Data quality monitoring, running scheduled checks against production data and alerting when something looks wrong
• Machine learning pipeline orchestration, coordinating training jobs, evaluation steps, and deployment triggers
• Event-driven automation, reacting to files landing in storage, messages on a queue, or webhooks from external systems

The common thread is teams who want workflow automation without committing to writing and maintaining a full codebase just to describe “do this, then this, then notify someone.”

Deployment Options: Self-Hosted vs Kestra Cloud

Like most modern orchestration platforms, Kestra offers more than one way to run it.

Self-hosted Kestra can be run via Docker Compose for local development and small deployments, or on Kubernetes for production-scale workloads. Kestra provides official Helm charts, and the self-hosted, open-source version includes the full core feature set, flows, tasks, triggers, and the UI, at no cost.

Kestra Cloud (Kestra EE / managed offering) is the hosted, managed version, removing the operational burden of running the Kestra server yourself. It typically adds enterprise features like more granular access control, dedicated support, and additional scalability options, aimed at larger organizations running Kestra as critical infrastructure.

For a beginner or a small project, self-hosting through Docker Compose is the fastest way to get started, and it’s genuinely simple to spin up given Kestra’s containerized-first design philosophy.

Getting Started: Running Kestra Locally

For anyone wanting to try Kestra hands-on, the fastest path is Docker. A minimal setup looks like this:

docker run --pull=always --rm -it -p 8080:8080 \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /tmp:/tmp kestra/kestra:latest server local

This single command starts a fully functional local Kestra instance, including the UI, accessible at http://localhost:8080. From there, you can create a new flow directly in the UI’s built-in editor, paste in a YAML definition like the example above, and execute it immediately, watching logs and outputs update in real time.

This low-friction setup is part of what makes Kestra appealing for evaluation. There’s no need to configure a database, a message broker, or a scheduler separately before you can see something working, a meaningful difference compared to setting up a first Airflow instance from scratch.

Benefits of Using Kestra

Pulling together what makes Kestra distinct:

Low barrier to entry. YAML is dramatically easier for non-engineers to read and modify compared to a full programming language, opening up workflow maintenance to a wider group of people on a team.

Version-controllable by design. Since flows are plain text YAML files, they fit naturally into existing Git workflows, code review processes, and CI/CD pipelines without any special tooling.

Broad plugin ecosystem. Kestra’s built-in plugins cover a wide range of common integrations out of the box, reducing the amount of custom glue code teams need to write.

Flexible execution model. The same flow can run tasks locally, in Docker containers, or on Kubernetes, letting teams scale execution without rewriting flow definitions.

Strong UI-driven observability. Execution history, logs, and a visual topology view are available out of the box, without needing to configure separate monitoring tools.

General-purpose scope. Unlike tools narrowly focused on data pipelines, Kestra applies equally well to infrastructure automation and business process workflows, giving teams one tool for multiple kinds of automation needs.

Challenges and Things to Consider

Kestra isn’t without tradeoffs, and it’s worth being upfront about them.

YAML’s simplicity is also its limitation. Complex conditional logic, intricate branching, or anything that would naturally benefit from loops and functions in a real programming language can become awkward to express purely in YAML, even with Kestra’s templating support. Very complex workflows sometimes end up embedding actual scripts (Python, in the earlier example) inside YAML tasks anyway, which is a reasonable middle ground but worth knowing about upfront.

Kestra is younger than Airflow, with a smaller long-term track record in large-scale production environments. Teams evaluating it for mission-critical infrastructure should weigh that against the maturity of more established alternatives.

Self-hosting still requires real operational investment for production-scale deployments, even though local development is notably easier to set up than with older tools. Running Kestra reliably at scale on Kubernetes still means managing that infrastructure, monitoring it, and keeping it upgraded.

Frequently Asked Questions About Kestra

Is Kestra open source? Yes. Kestra’s core platform is open source, and the project is actively developed with a growing plugin ecosystem. A separate enterprise edition and managed cloud offering exist for teams wanting additional features and support.

Do I need to know Python to use Kestra? No, not for basic flows. Flows are defined in YAML, and many built-in tasks handle common operations (downloading files, calling APIs, moving data between systems) without any custom code. Python or other scripting can be embedded inside specific tasks when you need custom logic, but it isn’t required to get started.

How does Kestra compare to writing a custom Python script with a cron job? A cron job with a Python script gives you no visibility into what happened, no retry handling, and no easy way to backfill historical runs. Kestra provides scheduling, execution history, retries, and a UI showing exactly what happened at each step, all without extra code to build that functionality yourself.

Can Kestra run tasks in Kubernetes? Yes. Kestra supports a Kubernetes-based execution model where individual tasks run as isolated pods, letting you scale execution horizontally and match Kestra’s runtime environment to your existing infrastructure.

Is Kestra a replacement for Airflow? For many teams, yes, particularly ones that value a lower barrier to entry and simpler operational overhead. Some organizations run Kestra alongside an existing Airflow deployment during a gradual migration, given how large some legacy Airflow codebases can be to migrate all at once.

What’s the best way to start learning Kestra? Running the local Docker setup covered earlier and building a small flow yourself is the fastest way to get a feel for it. From there, exploring Kestra’s plugin catalog for the specific systems you already use (databases, cloud providers, notification tools) will show you how much can be accomplished without custom code.

Wrapping Up

So, what is Kestra, in a single sentence? It’s an orchestration platform that lets you define workflows, from data pipelines to business automation, as readable YAML files rather than application code, with a modern architecture and a genuinely approachable local setup behind it.

For teams tired of Python being a prerequisite just to change a schedule, or looking for something simpler to operate than a decade-old scheduler, Kestra is a strong option worth evaluating. It won’t replace every use case for tools like Dagster or Temporal, which offer deeper programmatic control for asset lineage or complex durable business logic respectively, but for teams that want approachable, version-controlled automation without a steep coding requirement, it fills a real gap in the orchestration landscape.

If you’re comparing orchestration tools for your team, it’s worth reading our guides on what Temporal is and our Dagster tutorial for beginners alongside this one, since the right choice often comes down to how much of your workflow logic genuinely needs a full programming language versus how much can be expressed cleanly in YAML.

For deeper technical reference as you get started, the official Kestra documentation covers the full plugin catalog, flow syntax, and deployment guides in more depth than a single beginner’s guide can.