ADaSci Academy

    The Modern Data Engineering Playbook

    Data engineering has never been more consequential. The rise of LLMs, RAG systems, and AI-native applications hasn't made pipelines less important; it's made them the foundation on which everything else depends. Across f…

    2,871+ enrolled
    10 hours
    Intermediate
    The Modern Data Engineering Playbook
    01 — Overview

    About this course

    Data engineering has never been more consequential. The rise of LLMs, RAG systems, and AI-native applications hasn't made pipelines less important; it's made them the foundation on which everything else depends. Across four intensive, hands-on workshops, you'll rebuild your understanding of the data stack from the ground up. Lakehouse architecture, semantic layers, DataOps practices like CI/CD and observability, and the full pipeline behind real AI systems: RAG ingestion, embedding lifecycles, and vector store delivery.


    You'll also learn what happens after you ship. How to instrument LLM pipelines, catch dataset drift and prompt regressions before they become incidents, and build feedback loops that make your systems smarter over time. Whether you're a data engineer stepping into AI or a seasoned practitioner modernising your stack, this course gives you the blueprints to build data infrastructure that's reliable, observable, and ready for the AI era.

    02 — Outcomes

    What you'll learn

    Design and implement a modern lakehouse architecture using Delta Lake, with semantic layers that make data accessible, trustworthy, and ready for both analytics and AI workloads.
    Apply DataOps principles to build reliable, observable data pipelines with CI/CD, automated testing, data quality checks, and lineage tracking using orchestration tools.
    Build end-to-end data pipelines that power AI systems, covering RAG ingestion, embedding generation, vector store delivery, and knowledge base management.
    Instrument and monitor production LLM pipelines using tools like LangSmith and Weights & Biases, tracking prompt telemetry, retrieval accuracy, and embedding quality across the full stack.
    Build automated feedback loops that detect dataset drift, prompt regressions, and retrieval degradation, keeping AI systems reliable and continuously improving long after deployment.
    03 — Curriculum

    Course content

    4 sections · 8 lectures

    04 — Prerequisites

    Requirements

    • Data Platforms & Storage - Databricks or Snowflake (trial/free accounts suffice), Delta Lake, and Apache Iceberg.
    • Processing & Orchestration - Apache Spark for data processing, and pipeline orchestration.
    • AI & Vector Infrastructure - OpenAI APIs (or equivalent), and a vector database (Pinecone or Weaviate).
    • Observability & Experiment Tracking - LangSmith and Weights & Biases for monitoring and evaluating LLM pipelines.
    • Development Setup - VS Code or any equivalent IDE, GitHub or GitLab for version control and CI/CD, and Docker (optional but recommended).
    05 — Audience

    Who should take this course

    • Data Engineers looking to modernise their stack and build infrastructure that supports AI and ML workloads.
    • Analytics Engineers who want to go beyond transformation layers and understand the full data platform picture.
    • ML Engineers who work at the intersection of data pipelines and model deployment and want stronger data engineering foundations.
    • Data Architects designing scalable, AI-ready platforms for their organisations.
    • DataOps / Platform Engineers responsible for pipeline reliability, observability, and operational excellence.
    • Senior Data Scientists / Forward Deployed Engineers transitioning into more engineering-heavy roles and looking to upskil in modern data infrastructure.

    Save 50% with Membership

    Get unlimited access to all courses and exclusive member discounts on certifications.

    50% off all courses30% off certificationsExclusive content
    The Modern Data Engineering Playbook

    $199.00

    Member Pricing

    FREE with Premium

    50% off with General

    Duration10h
    LevelIntermediate
    CertificateYes

    $199.00