Kestra Workflow Generator

Kestra uses declarative YAML to define workflows. Every flow requires id, namespace, and tasks.

Core YAML Structure

id: flow-name              # Required: unique within namespace, lowercase with hyphens
namespace: company.team    # Required: dot-separated hierarchy
description: |             # Optional: supports Markdown
  Flow description
labels:                    # Optional: key-value pairs for organization
  env: prod
  team: data-engineering
inputs:                    # Optional: typed parameters
  - id: my_input
    type: STRING
    required: false
    defaults: "default value"
variables:                 # Optional: reusable values
  base_url: "https://api.example.com"
tasks:                     # Required: list of tasks to execute
  - id: task-name
    type: io.kestra.plugin...
triggers:                  # Optional: automatic execution
  - id: schedule
    type: io.kestra.plugin.core.trigger.Schedule
    cron: "0 9 * * *"
errors:                    # Optional: error handling tasks
  - id: notify
    type: io.kestra.plugin.core.log.Log
    message: "Flow failed!"

Task Generation Workflow

1. •Identify the task type needed (script, HTTP, flowable, etc.)
2. •Use fully qualified plugin type: io.kestra.plugin.[category].[TaskName]
3. •Include all required properties for the task type
4. •Use Pebble templating for dynamic values: {{ inputs.name }}, {{ outputs.taskId.property }}

Input Types

Common Task Types

Logging & Debug

- id: log_message
  type: io.kestra.plugin.core.log.Log
  message: "Hello {{ inputs.name }}!"
  level: INFO  # DEBUG, INFO, WARN, ERROR

HTTP Requests

- id: api_call
  type: io.kestra.plugin.core.http.Request
  uri: "https://api.example.com/data"
  method: GET
  headers:
    Authorization: "Bearer {{ secret('API_TOKEN') }}"

Python Scripts

- id: python_task
  type: io.kestra.plugin.scripts.python.Script
  containerImage: python:3.11-slim
  beforeCommands:
    - pip install pandas requests
  script: |
    import pandas as pd
    from kestra import Kestra
    
    data = {"result": "success"}
    Kestra.outputs(data)  # Pass outputs to next tasks
  outputFiles:
    - "*.csv"  # Files to capture

Shell Commands

- id: shell_task
  type: io.kestra.plugin.scripts.shell.Commands
  commands:
    - echo "Processing {{ inputs.filename }}"
    - cat {{ outputs.download.uri }}

Detailed References

Core Workflow Components

• •Core Plugins (HTTP, KV Store, Storage, Logging): See references/core.md
• •Script Tasks (Python, Shell, Node.js): See references/scripts.md
• •Triggers (Schedule, Webhook, Flow): See references/triggers.md
• •Flowable Tasks (Parallel, If, Switch, ForEach): See references/flowable.md
• •Error Handling & Retries: See references/error-handling.md
• •Pebble Templating & Expressions: See references/templating.md
• •Internal Storage & File Access: See references/storage.md - URI schemes (kestra:///, nsfile:///, file:///), fetchType modes, Ion format, read() function, inputFiles/outputFiles, and storage management tasks

Plugins

• •Elasticsearch: See references/elasticsearch.md
• •AI & LLM (OpenAI, Agents, RAG): See references/ai-llm.md
• •SerDes (CSV, JSON, Avro, Parquet): See references/serdes.md

Architecture & Patterns

• •Best Practices (Task Runners, Subflows, Python): See references/best-practices.md
• •Common Patterns & Examples: See references/patterns.md
• •Gotchas & Hard-Won Lessons: See references/gotchas.md - Critical pitfalls including kestra:/// URI auto-resolution, ForEachItem merge outputs, shell quoting, and debugging techniques

Additional Resources

• •Ask Kestra AI API: See references/ask-kestra-ai.md - Query Kestra's documentation AI when you need more specific answers or the latest documentation

Tip: If a Kestra MCP server is configured, use it to validate flows, list executions, or trigger runs directly. This skill focuses on generating correct YAML; the MCP server handles runtime interaction.

Tip: If you can't find an answer in the static references, use the Ask Kestra AI API to query their documentation assistant. It's especially useful for:

• •Edge cases not covered in common patterns
• •Latest plugin features or changes
• •Complex output structures (e.g., ForEachItem merging)
• •Troubleshooting specific error messages

Quick Debugging Checklist

• • Indentation: Exactly 2 spaces (no tabs)
• • Task IDs: Unique within flow, use lowercase with hyphens
• • Plugin types: Fully qualified (io.kestra.plugin.core.log.Log)
• • Expressions: Use {{ }} for Pebble templating
• • Secrets: Use {{ secret('SECRET_NAME') }}
• • Outputs: Reference as {{ outputs.taskId.property }}
• • Inputs: Reference as {{ inputs.inputId }}
• • Strings with colons: Quote them (message: "Time: 10:00")
• • Multi-line scripts: Use | for literal block scalar

Common Errors & Fixes