Training course · AI Agent Development · Data Engineering

AI Agent Development for Data Engineers

Build AI agents professionally: architectures, tool use, agent SDKs, LangGraph, memory, evals, guardrails and deployment.

Duration
2 days (14 hrs)
Format
Online or on-site
Language
German or English
Participants
open dates or exclusive for your team
Certificate
Certificate of attendance

Overview

“AI Agent Development for Data Engineers” is a hands-on course for roles such as Data engineers and analytics engineers, BI developers responsible for ETL and the data warehouse and Platform teams running lakehouse and streaming infrastructure. Rather than learning AI Agent Development in the abstract, you work on tasks from your daily routine – with examples, exercises and templates from Data Engineering.

AI agents go beyond conventional chatbots: they break tasks down, call tools and APIs, evaluate the results and keep iterating until a goal is met. In this two-day course you learn to build such systems so that they work reliably, transparently and safely in day-to-day business. You compare architectural patterns ranging from simple workflows through the agent loop to multi-agent systems, and you implement them hands-on with the Claude Agent SDK, the OpenAI Agents SDK and LangGraph.

The focus is not on a quick demo but on the path to production. You design tool interfaces and context deliberately, connect tools via the Model Context Protocol, plan short-term and long-term memory and build evaluations that surface regressions early. On top of that come guardrails against prompt injection and harmful actions, human-in-the-loop approvals, tracing, cost control and deployment. By the end you can judge when an agent is worthwhile, when a deterministic workflow is enough and which framework fits your project.

Data engineers build and run the pipelines that reporting, analytics and AI applications depend on: ingestion from source systems, transformations in dbt or Spark, orchestration with Airflow or Dagster, and data quality and governance in the warehouse or lakehouse. Much of the work involves SQL, Python, YAML and a great deal of detailed knowledge about schemas and dependencies. AI coding assistants and agents accelerate this work considerably: they write dbt models with tests and documentation, optimise slow queries, create Airflow DAGs, migrate legacy ETL and explain complex lineage. Because data is often sensitive, handling schemas, sample data and credentials responsibly is at the heart of this course.

The course is delivered live – either online in a virtual classroom or on-site in our training rooms in Munich, Salzburg, Berlin, Augsburg, Hanover, Bremen, Stuttgart and Kaiserslautern. We can also run it exclusively for your team.

Who is this course for?

  • Data engineers and analytics engineers
  • BI developers responsible for ETL and the data warehouse
  • Platform teams running lakehouse and streaming infrastructure
  • Developers modernising legacy ETL processes

Typical challenges we address

  • Ageing ETL built on stored procedures or proprietary tools with no tests
  • Slow, expensive queries and inefficient Spark jobs
  • Missing data quality checks and unclear ownership of datasets
  • Patchy documentation of data models, lineage and business logic

What you will be able to do

  • You choose the right pattern for a task, from a fixed workflow to an autonomous agent.
  • You design tools and context so that agents work reliably and cost-effectively.
  • You implement agents with the Claude Agent SDK, the OpenAI Agents SDK or LangGraph.
  • You set up evaluations and tracing to measure quality and detect regressions.
  • Use AI agents productively for dbt models, pipelines and SQL optimisation.
  • Modernise legacy ETL with AI support and safeguard results with reconciliation tests.
  • Protect sensitive data by working with schemas, synthetic samples and minimal access rights.

Course outline

Day 1

Module 1 Agent architectures and design patterns
  • Workflows versus agents: prompt chaining, routing, parallelisation, orchestrator-worker
  • The agent loop: plan, act, observe, reflect
  • Multi-agent systems, subagents and handoffs between agents
  • Criteria for choosing model, framework and level of autonomy
Module 2 Tool use and context engineering
  • Function calling and structured outputs with JSON Schema
  • Designing good tools: granularity, naming, error messages, idempotency
  • Connecting and reusing tools via MCP servers
  • Managing the context window: summarisation, compaction, retrieval and prompt caching
Module 3 Frameworks in practice
  • Claude Agent SDK: agent loop, tools, subagents, permissions and hooks
  • OpenAI Agents SDK: agents, handoffs, guardrails and tracing
  • LangGraph: stateful graphs, checkpointing and human-in-the-loop
  • Side-by-side comparison on the same task and criteria for choosing a framework
Module 4 Memory, evals and observability
  • Short-term and long-term memory, storage strategies and forgetting
  • Evaluation datasets, LLM-as-a-judge and task-based success metrics
  • Tracing tool calls, token usage and latency
  • Regression testing when prompts, models or tools change

Day 2

Module 5 Guardrails, security and deployment
  • Protecting against prompt injection, data exfiltration and over-privileged tools
  • Input and output checks, approval steps and sandboxing
  • Deploying as a service, asynchronous execution, retries and cost limits
  • Governance, audit logging and requirements under GDPR and the EU AI Act
Module 6 Practice lab: Data Engineering
  • dbt models with tests
  • Optimising SQL queries
  • Airflow and Dagster pipelines
  • Migrating legacy ETL
  • Automating data quality
  • Analysing Spark jobs
Module 7 Data protection, quality & transfer to daily work
  • Data engineering prompts should contain schemas and synthetic sample data rather than real personal records, and database credentials or connection strings must stay out of context. Apply GDPR principles such as purpose limitation and data minimisation, document data flows transparently and check new Python packages and connectors for licensing and supply-chain risk.
  • Ground rules for safe AI use in your team
  • Checking results: facts, sources, tests and the four-eyes principle
  • Personal action plan for the next 30 days
  • Open questions, outlook and resources for further learning

Practical scenarios from your field

dbt models with tests

The agent writes staging, intermediate and mart models in dbt, adds schema tests, source definitions and descriptions, and runs dbt build iteratively until models and tests pass.

Optimising SQL queries

The AI analyses execution plans, spots expensive joins and missing partitioning or clustering, and rewrites queries so they scan less data and run measurably faster.

Airflow and Dagster pipelines

Requirements become DAGs or assets with a clean dependency structure, retries, sensors, alerting and backfill support, along with tests for the transformation logic. Existing naming conventions are respected.

Migrating legacy ETL

Stored procedures or graphical ETL jobs are analysed, the business logic documented and moved step by step into dbt or PySpark code, safeguarded by reconciliation queries comparing old and new.

Automating data quality

The assistant defines checks for completeness, uniqueness, value ranges and freshness, implements them with dbt tests or Great Expectations and sets up notifications for violations.

Analysing Spark jobs

Spark UI information and logs are interpreted, skew, shuffles and memory issues identified and optimisations such as broadcast joins, repartitioning or caching proposed. Every optimisation is backed by runtime comparisons.

Possible hands-on projects

Data quality dashboard

A small web app that reads results from dbt tests or data quality checks and presents trends, outliers and affected tables at a glance.

Ingestion pipeline for a REST API

A Python pipeline that loads data incrementally from an API, detects schema changes and writes the results as partitioned tables to the lakehouse.

Lineage explorer

A tool that parses the dbt manifest and displays dependencies between sources, models and dashboards as an interactive graph. The impact of changes becomes visible at a glance.

Prerequisites

Solid programming skills in Python or TypeScript, experience with REST APIs and some hands-on experience with LLM APIs or AI assistants. Confident SQL and Python skills plus practical experience with a data warehouse, dbt, Spark or an orchestration tool.

Technical requirements

A laptop with Python or Node.js, Git and a code editor, plus API access to at least one LLM provider such as Anthropic or OpenAI. Credentials for the exercises can be provided if needed.

Data protection & compliance

Data engineering prompts should contain schemas and synthetic sample data rather than real personal records, and database credentials or connection strings must stay out of context. Apply GDPR principles such as purpose limitation and data minimisation, document data flows transparently and check new Python packages and connectors for licensing and supply-chain risk.

Suggested dates

All dates are suggestions – your preferred date is always possible.

  • Tue, 24 November 2026 – Wed, 25 November 2026 Online Request
  • Wed, 2 December 2026 – Thu, 3 December 2026 Munich Request
  • Thu, 10 December 2026 – Fri, 11 December 2026 Online Request
  • Tue, 12 January 2027 – Wed, 13 January 2027 Hanover Request
  • Thu, 28 January 2027 – Fri, 29 January 2027 Online Request
  • Mon, 1 February 2027 – Tue, 2 February 2027 Berlin Request

Frequently asked questions

Which agent framework should we use?

It depends on your model strategy, the level of control you need and your team’s skills. Vendor SDKs offer a lean start with tightly integrated features, while LangGraph gives you finer control over state and flow. In the course you solve the same task with several frameworks and derive clear selection criteria for your own project.

How do you stop an agent from taking wrong or dangerous actions?

With several layers of protection: narrowly scoped tools with minimal permissions, input and output validation, approval steps for critical actions, sandboxing and continuous evaluation. Prompt injection via external content is treated as a realistic threat. You practise these measures directly on your own agent.

Does every automation need an agent?

No. Many tasks are handled more reliably, cheaply and transparently by a fixed workflow of a few LLM calls. Agents pay off when the solution path cannot be planned in advance and the system must react flexibly to intermediate results. We train exactly this trade-off using practical examples.

Does the AI need access to our production data?

No, and it usually makes little sense. For the vast majority of tasks, schemas, metadata, execution plans and synthetic sample data are sufficient. On the course you learn to give the agent exactly this context, restrict read access to development environments and still obtain reliable results.

Will this work with our data warehouse?

The working methods are platform-independent and apply equally to Snowflake, BigQuery, Databricks, Redshift, Synapse or PostgreSQL. What matters is telling the assistant about the SQL dialect and platform specifics such as partitioning or clustering. We show you how to capture this in instruction files.

How much does “AI Agent Development for Data Engineers” cost?

The course costs a flat €949 plus VAT per participant – whether online or on-site. This includes 2 training day(s) with a live trainer, course materials, a lab environment and a certificate of attendance.

Can I suggest my own date?

Yes. Our suggested dates are just a guide. In the request form you can always describe your preferred date or period in free text – we check availability and confirm within one working day.

Is the course online or on-site?

Both are possible: live online in a virtual classroom or on-site in Munich, Salzburg, Berlin, Augsburg, Hanover, Bremen, Stuttgart or Kaiserslautern. We can also run the course exclusively for your team.

More courses for Data Engineers

More AI Agent Development courses

All 30 AI Agent Development courses

AI Agent Development · MunichAI Agent Development · SalzburgAI Agent Development · BerlinAI Agent Development · AugsburgAI Agent Development · HanoverAI Agent Development · BremenAI Agent Development · StuttgartAI Agent Development · KaiserslauternAI Agent Development · Online