Labeled data is still the bottleneck for supervised models, fine-tuning, evaluation and preference optimization, and most labeling problems are specification problems. This course starts with task design: turning a fuzzy requirement into label definitions, edge-case rules and a guideline document that annotators can apply consistently, then piloting and revising it. It covers the main task types (classification, span and entity annotation, bounding boxes, polygons and segmentation, audio and video, and LLM-specific pairwise preference and rubric scoring) and tools to run them: open-source Label Studio, CVAT and Argilla, Prodigy, and managed platforms and vendors such as Labelbox, Scale AI, SuperAnnotate and Encord.
Quality gets the most attention: inter-annotator agreement with Cohen's and Fleiss' kappa and Krippendorff's alpha, gold and honeypot items, review and adjudication workflows, and annotator feedback. You will also cut cost with model-assisted pre-labeling, active learning and LLM-as-annotator with human verification, while measuring the bias these introduce. The final module covers workforce ethics and pay, data protection for annotators handling sensitive data, documentation for EU AI Act data-governance expectations, and budgeting a labeling program. Regulatory notes are educational, not legal advice. The course ends in an automated exam; passing earns a Wirewalk Institute of AI certificate of completion and assessment.