Development & technology · Data Engineering

AI training for Data Engineers

Data engineers build and run the pipelines that reporting, analytics and AI applications depend on: ingestion from source systems, transformations in dbt or Spark, orchestration with Airflow or Dagster, and data quality and governance in the warehouse or lakehouse. Much of the work involves SQL, Python, YAML and a great deal of detailed knowledge about schemas and dependencies. AI coding assistants and agents accelerate this work considerably: they write dbt models with tests and documentation, optimise slow queries, create Airflow DAGs, migrate legacy ETL and explain complex lineage. Because data is often sensitive, handling schemas, sample data and credentials responsibly is at the heart of this course.

Who is this course for?

  • Data engineers and analytics engineers
  • BI developers responsible for ETL and the data warehouse
  • Platform teams running lakehouse and streaming infrastructure
  • Developers modernising legacy ETL processes

Typical challenges we address

  • Ageing ETL built on stored procedures or proprietary tools with no tests
  • Slow, expensive queries and inefficient Spark jobs
  • Missing data quality checks and unclear ownership of datasets
  • Patchy documentation of data models, lineage and business logic

Practical scenarios from your field

dbt models with tests

The agent writes staging, intermediate and mart models in dbt, adds schema tests, source definitions and descriptions, and runs dbt build iteratively until models and tests pass.

Optimising SQL queries

The AI analyses execution plans, spots expensive joins and missing partitioning or clustering, and rewrites queries so they scan less data and run measurably faster.

Airflow and Dagster pipelines

Requirements become DAGs or assets with a clean dependency structure, retries, sensors, alerting and backfill support, along with tests for the transformation logic. Existing naming conventions are respected.

Migrating legacy ETL

Stored procedures or graphical ETL jobs are analysed, the business logic documented and moved step by step into dbt or PySpark code, safeguarded by reconciliation queries comparing old and new.

Automating data quality

The assistant defines checks for completeness, uniqueness, value ranges and freshness, implements them with dbt tests or Great Expectations and sets up notifications for violations.

Analysing Spark jobs

Spark UI information and logs are interpreted, skew, shuffles and memory issues identified and optimisations such as broadcast joins, repartitioning or caching proposed. Every optimisation is backed by runtime comparisons.

Possible hands-on projects

Data quality dashboard

A small web app that reads results from dbt tests or data quality checks and presents trends, outliers and affected tables at a glance.

Ingestion pipeline for a REST API

A Python pipeline that loads data incrementally from an API, detects schema changes and writes the results as partitioned tables to the lakehouse.

Lineage explorer

A tool that parses the dbt manifest and displays dependencies between sources, models and dashboards as an interactive graph. The impact of changes becomes visible at a glance.

All 10 courses for Data Engineers

Vibe & agentic coding (10)

Data protection & compliance

Data engineering prompts should contain schemas and synthetic sample data rather than real personal records, and database credentials or connection strings must stay out of context. Apply GDPR principles such as purpose limitation and data minimisation, document data flows transparently and check new Python packages and connectors for licensing and supply-chain risk.

Prerequisites

Confident SQL and Python skills plus practical experience with a data warehouse, dbt, Spark or an orchestration tool.

Frequently asked questions

Does the AI need access to our production data?

No, and it usually makes little sense. For the vast majority of tasks, schemas, metadata, execution plans and synthetic sample data are sufficient. On the course you learn to give the agent exactly this context, restrict read access to development environments and still obtain reliable results.

Will this work with our data warehouse?

The working methods are platform-independent and apply equally to Snowflake, BigQuery, Databricks, Redshift, Synapse or PostgreSQL. What matters is telling the assistant about the SQL dialect and platform specifics such as partitioning or clustering. We show you how to capture this in instruction files.

Request without obligation

We will get back to you within one working day with a confirmation or an alternative suggestion.

Format *
Total €949 plus VAT