For data and ML engineers who feed models: build reproducible pipelines that collect, store, clean, deduplicate, version and serve data for training, fine-tuning, retrieval and evaluation.
The curriculum is being prepared and reviewed. Enrollment is not open yet.
About this course
Models are bounded by the data they see, and AI data work has requirements that analytics pipelines rarely face: unstructured and multimodal inputs, corpus-scale near-duplicate detection, exact reproducibility of a training set months later, and loaders that must keep expensive GPUs busy. This course builds the full path. You will design storage on object stores using Parquet with Apache Iceberg or Delta Lake tables, choose between Spark, Ray Data, Daft and DuckDB for a processing job, and orchestrate pipelines with Airflow or Dagster using data-aware scheduling.
The core of the course is the preparation pipeline used for pretraining and fine-tuning corpora: language identification, heuristic and model-based quality filtering, exact and MinHash-LSH near-duplicate removal, PII scrubbing and decontamination against evaluation sets, with reference to open tooling such as Hugging Face datatrove and NVIDIA NeMo Curator. You will then version datasets with table snapshots or lakeFS, package them in Hugging Face datasets, WebDataset or MosaicML Streaming formats, and measure loader throughput on GPU nodes. Streaming ingestion with Kafka for retrieval freshness rounds out the course. Governance and lineage are covered in depth in dat-320. The course ends in an automated exam; passing earns a Wirewalk Institute of AI certificate of completion and assessment.
What you will be able to do
Explain how data requirements differ for pretraining, fine-tuning, retrieval and evaluation, and design a pipeline for each
Design object-store lakehouse storage with Parquet and Iceberg or Delta tables, including partitioning and schema evolution
Choose Spark, Ray Data, Daft or DuckDB for a processing job based on data size, data type and GPU involvement
Build a text preparation pipeline with language identification, quality filtering, MinHash-LSH deduplication and PII scrubbing
Decontaminate training data against evaluation sets and document the method
Version a dataset so a training run can be reproduced exactly, using table snapshots or lakeFS commits
Package and stream training data in shard-based formats and measure whether the loader keeps GPUs saturated
Syllabus
6 modules · 18 lessons · about 15 hours
1
Module 1: Data Requirements for AI Workloads
What each stage of the model lifecycle needs from data, and how that changes pipeline design.3 lessons
Pretraining, Fine-Tuning, Retrieval and Eval Data: Different Jobs, Different Pipelines
Sourcing Data: Licenses, Terms of Use, Crawling and Provenance Records
Reference Architecture for an AI Data Platform
2
Module 2: Storage and Table Formats
Lakehouse storage that is cheap, scalable and reproducible.3 lessons
Object Storage, Parquet and Columnar Layout for AI Data
Apache Iceberg and Delta Lake: Snapshots, Time Travel and Schema Evolution
Storing Unstructured and Multimodal Data Alongside Tables
3
Module 3: Processing Engines and Orchestration
Picking the engine for the job and running pipelines on a schedule or on data arrival.3 lessons
Spark, Ray Data, Daft and DuckDB: Choosing a Processing Engine
GPU-Accelerated Data Processing: Batch Inference and Embedding Jobs
Orchestration with Airflow and Dagster: Assets, Lineage and Data-Aware Scheduling
4
Module 4: Corpus Preparation at Scale
The cleaning steps that decide training data quality.4 lessons
Extraction, Language Identification and Heuristic Filters
Exact and MinHash-LSH Near-Duplicate Removal
Model-Based Quality Classifiers and PII Scrubbing
Decontamination Against Benchmarks and Held-Out Sets
5
Module 5: Versioning and Reproducibility
Being able to say exactly what a model was trained on.2 lessons
Dataset Versioning with Table Snapshots and lakeFS
Dataset Cards, Manifests and Hugging Face Datasets
6
Module 6: Feeding Models and Keeping Data Fresh
High-throughput loading for training and streaming updates for retrieval.3 lessons
Shard Formats: WebDataset, MosaicML Streaming and Arrow
Data Loader Throughput: Measuring and Fixing GPU Starvation
Streaming Ingestion with Kafka and Change Data Capture for Retrieval Freshness
Exam and certificate
The exam unlocks once you have completed every lesson. It draws 20 questions from a larger bank, spread across the modules, with the answer options shuffled for each attempt. You have 40 minutes and need 80% to pass. After a failed attempt you can try again after 24 hours, up to 3 attempts in total. Feedback is given per module, never as the answer key.
Passing earns a certificate of completion and assessment from the Wirewalk Institute of AI with a unique ID that anyone can check at institute.wirewalk.com/verify. It is not an accredited qualification or a vendor certification. How certification works.
The Wirewalk Institute of AI is independent. This course teaches publicly documented products and is not affiliated with or endorsed by Apache Spark, Ray, Apache Iceberg, Delta Lake, DuckDB, Apache Airflow, Dagster, Apache Kafka, Hugging Face, NVIDIA, or lakeFS. Product and company names are trademarks of their respective owners.
Build production pipelines for unstructured documents: format detection and text extraction, layout analysis and reading order, OCR and vision-language parsers, table and form extraction, schema-driven field extraction with LLMs, human review loops, and running it all at scale with measured accuracy and cost.
Run annotation as an operation with measurable quality: task design and guidelines, labeling tools such as Label Studio, CVAT and Argilla, in-house versus vendor workforces, inter-annotator agreement, gold questions and adjudication, model-assisted and active-learning workflows, and the human preference and rubric labeling used for LLM post-training and evaluation.
An expert course on generating and validating synthetic data: LLM-generated instruction, reasoning and tool-use data (Self-Instruct, Evol-Instruct, persona-driven and verifier-filtered generation), distillation, synthetic evaluation sets, tabular synthesis with SDV and differential privacy, fidelity, utility and privacy metrics, model-collapse risk, and the license terms that govern using model outputs.