Wirewalk Instituteof AI
dat-201 Intermediate Waitlist

Data Engineering for AI

For data and ML engineers who feed models: build reproducible pipelines that collect, store, clean, deduplicate, version and serve data for training, fine-tuning, retrieval and evaluation.

The curriculum is being prepared and reviewed. Enrollment is not open yet.

About this course

Models are bounded by the data they see, and AI data work has requirements that analytics pipelines rarely face: unstructured and multimodal inputs, corpus-scale near-duplicate detection, exact reproducibility of a training set months later, and loaders that must keep expensive GPUs busy. This course builds the full path. You will design storage on object stores using Parquet with Apache Iceberg or Delta Lake tables, choose between Spark, Ray Data, Daft and DuckDB for a processing job, and orchestrate pipelines with Airflow or Dagster using data-aware scheduling.

The core of the course is the preparation pipeline used for pretraining and fine-tuning corpora: language identification, heuristic and model-based quality filtering, exact and MinHash-LSH near-duplicate removal, PII scrubbing and decontamination against evaluation sets, with reference to open tooling such as Hugging Face datatrove and NVIDIA NeMo Curator. You will then version datasets with table snapshots or lakeFS, package them in Hugging Face datasets, WebDataset or MosaicML Streaming formats, and measure loader throughput on GPU nodes. Streaming ingestion with Kafka for retrieval freshness rounds out the course. Governance and lineage are covered in depth in dat-320. The course ends in an automated exam; passing earns a Wirewalk Institute of AI certificate of completion and assessment.

What you will be able to do

  • Explain how data requirements differ for pretraining, fine-tuning, retrieval and evaluation, and design a pipeline for each
  • Design object-store lakehouse storage with Parquet and Iceberg or Delta tables, including partitioning and schema evolution
  • Choose Spark, Ray Data, Daft or DuckDB for a processing job based on data size, data type and GPU involvement
  • Build a text preparation pipeline with language identification, quality filtering, MinHash-LSH deduplication and PII scrubbing
  • Decontaminate training data against evaluation sets and document the method
  • Version a dataset so a training run can be reproduced exactly, using table snapshots or lakeFS commits
  • Package and stream training data in shard-based formats and measure whether the loader keeps GPUs saturated

Syllabus

6 modules · 18 lessons · about 15 hours

Module 1: Data Requirements for AI Workloads

What each stage of the model lifecycle needs from data, and how that changes pipeline design.
3 lessons
  1. Pretraining, Fine-Tuning, Retrieval and Eval Data: Different Jobs, Different Pipelines
  2. Sourcing Data: Licenses, Terms of Use, Crawling and Provenance Records
  3. Reference Architecture for an AI Data Platform

Module 2: Storage and Table Formats

Lakehouse storage that is cheap, scalable and reproducible.
3 lessons
  1. Object Storage, Parquet and Columnar Layout for AI Data
  2. Apache Iceberg and Delta Lake: Snapshots, Time Travel and Schema Evolution
  3. Storing Unstructured and Multimodal Data Alongside Tables

Module 3: Processing Engines and Orchestration

Picking the engine for the job and running pipelines on a schedule or on data arrival.
3 lessons
  1. Spark, Ray Data, Daft and DuckDB: Choosing a Processing Engine
  2. GPU-Accelerated Data Processing: Batch Inference and Embedding Jobs
  3. Orchestration with Airflow and Dagster: Assets, Lineage and Data-Aware Scheduling

Module 4: Corpus Preparation at Scale

The cleaning steps that decide training data quality.
4 lessons
  1. Extraction, Language Identification and Heuristic Filters
  2. Exact and MinHash-LSH Near-Duplicate Removal
  3. Model-Based Quality Classifiers and PII Scrubbing
  4. Decontamination Against Benchmarks and Held-Out Sets

Module 5: Versioning and Reproducibility

Being able to say exactly what a model was trained on.
2 lessons
  1. Dataset Versioning with Table Snapshots and lakeFS
  2. Dataset Cards, Manifests and Hugging Face Datasets

Module 6: Feeding Models and Keeping Data Fresh

High-throughput loading for training and streaming updates for retrieval.
3 lessons
  1. Shard Formats: WebDataset, MosaicML Streaming and Arrow
  2. Data Loader Throughput: Measuring and Fixing GPU Starvation
  3. Streaming Ingestion with Kafka and Change Data Capture for Retrieval Freshness

Exam and certificate

The exam unlocks once you have completed every lesson. It draws 20 questions from a larger bank, spread across the modules, with the answer options shuffled for each attempt. You have 40 minutes and need 80% to pass. After a failed attempt you can try again after 24 hours, up to 3 attempts in total. Feedback is given per module, never as the answer key.

Passing earns a certificate of completion and assessment from the Wirewalk Institute of AI with a unique ID that anyone can check at institute.wirewalk.com/verify. It is not an accredited qualification or a vendor certification. How certification works.

The Wirewalk Institute of AI is independent. This course teaches publicly documented products and is not affiliated with or endorsed by Apache Spark, Ray, Apache Iceberg, Delta Lake, DuckDB, Apache Airflow, Dagster, Apache Kafka, Hugging Face, NVIDIA, or lakeFS. Product and company names are trademarks of their respective owners.

dat-210 Intermediate Waitlist

Document Processing and Unstructured Data Pipelines

Build production pipelines for unstructured documents: format detection and text extraction, layout analysis and reading order, OCR and vision-language parsers, table and form extraction, schema-driven field extraction with LLMs, human review loops, and running it all at scale with measured accuracy and cost.

14 h · 19 lessons Join the waitlist
dat-220 Intermediate Waitlist

Data Labeling and Annotation Operations

Run annotation as an operation with measurable quality: task design and guidelines, labeling tools such as Label Studio, CVAT and Argilla, in-house versus vendor workforces, inter-annotator agreement, gold questions and adjudication, model-assisted and active-learning workflows, and the human preference and rubric labeling used for LLM post-training and evaluation.

12 h · 18 lessons Join the waitlist
dat-310 Expert Waitlist

Synthetic Data Generation

An expert course on generating and validating synthetic data: LLM-generated instruction, reasoning and tool-use data (Self-Instruct, Evol-Instruct, persona-driven and verifier-filtered generation), distillation, synthetic evaluation sets, tabular synthesis with SDV and differential privacy, fidelity, utility and privacy metrics, model-collapse risk, and the license terms that govern using model outputs.

18 h · 19 lessons Join the waitlist