Wirewalk Instituteof AI
dev-305 Expert Waitlist

Event-Driven AI Integration: Kafka, Queues and Streaming

For backend and data engineers who put LLMs and ML models into event-driven systems and need them to be reliable, idempotent, cost-controlled and observable at stream scale.

The curriculum is being prepared and reviewed. Enrollment is not open yet.

About this course

Model calls are slow, expensive, rate-limited and non-deterministic compared with the rest of a stream-processing pipeline. Putting them into event-driven systems naively produces duplicate charges, poison messages, consumer-lag spirals and results nobody can reproduce. Done well, events are the most natural way to run AI at scale: enrichment, classification, embedding refresh, document processing and agent workflows all become asynchronous, replayable and observable.

This course starts from messaging fundamentals as they apply to AI: delivery semantics, partitioning and ordering, consumer groups versus Kafka share groups (Queues for Kafka, production-ready in Kafka 4.2), and when a classic queue such as RabbitMQ, SQS, Pub/Sub, Service Bus or NATS JetStream fits better. You then build AI consumers with idempotency keys, retries, dead-letter topics, backpressure and provider rate-limit handling, and use batch APIs where latency allows. You run model inference inside stream processors with Flink SQL ML_PREDICT and managed Flink AI functions, keep vector indexes fresh with Debezium CDC, and design schemas and contracts with Schema Registry, CloudEvents and AsyncAPI. The final modules cover agentic and long-running work (durable workflows with Temporal, event-driven agents, human-in-the-loop steps), replay and reprocessing when models change, observability and cost control, and data protection in streams. Product capabilities are stated as of October 2026.

What you will be able to do

  • Choose between Kafka consumer groups, Kafka share groups and classic queues for an AI workload based on ordering, parallelism, acknowledgement and replay needs
  • Implement an AI-calling consumer with idempotency keys, bounded retries, dead-letter handling, provider rate-limit backoff and backpressure
  • Run model inference inside a stream processor using Flink SQL model functions, and decide when to call out to an external service instead
  • Build a CDC-driven pipeline with Debezium that keeps embeddings and vector indexes consistent with source systems
  • Design event contracts for AI outputs with Schema Registry, CloudEvents and AsyncAPI, including model and prompt version metadata for replay
  • Orchestrate long-running and agentic AI work with durable workflows and human-approval steps, and recover correctly from partial failure
  • Instrument an event-driven AI system for lag, latency, error rate, token spend and output quality, and plan safe reprocessing after model changes

Syllabus

6 modules · 19 lessons · about 20 hours

Module 1: Messaging Foundations for AI Workloads

Delivery semantics, ordering and parallelism when the consumer is a model call.
3 lessons
  1. Why AI Calls Break Naive Consumers: Latency, Cost, Rate Limits and Non-Determinism
  2. Kafka Partitions, Consumer Groups and Share Groups (Queues for Kafka)
  3. Classic Queues and Pub/Sub: RabbitMQ, SQS, Pub/Sub, Service Bus and NATS JetStream

Module 2: Reliable AI Consumers

The patterns that stop duplicate spend, poison messages and lag spirals.
4 lessons
  1. Idempotency Keys, Exactly-Once Myths and the Outbox Pattern
  2. Retries, Dead-Letter Topics and Poison-Message Triage
  3. Backpressure, Concurrency Limits and Provider Rate-Limit Budgets
  4. Synchronous, Streaming and Batch API Calls: Choosing per Workload

Module 3: Inference Inside Stream Processing

Running models where the data already flows.
3 lessons
  1. Flink SQL Model Inference: CREATE MODEL and ML_PREDICT
  2. Managed Stream AI: Confluent Flink AI Functions and Streaming Agents
  3. Kafka Streams and Consumer-Side Enrichment with External Model Services

Module 4: Data Contracts and Freshness

Schemas, CDC and keeping derived AI data consistent with sources.
3 lessons
  1. Event Contracts for AI Outputs: Schema Registry, CloudEvents and AsyncAPI
  2. CDC with Debezium: Keeping Embeddings and Vector Indexes in Sync
  3. Replay and Reprocessing When Models or Prompts Change

Module 5: Event-Driven Agents and Long-Running Work

Agents that react to events and workflows that survive failure.
3 lessons
  1. Event-Driven Agent Architectures: Triggers, Tools and Shared State
  2. Durable Workflows with Temporal for Multi-Step AI Processes
  3. Human-in-the-Loop Approvals and Compensating Actions

Module 6: Operating Event-Driven AI

Observability, cost, security and capacity at stream scale.
3 lessons
  1. Observability: Lag, Latency, Error Rates, Token Spend and Quality Signals
  2. Sensitive Data in Streams: Redaction, Encryption and Retention
  3. Capacity Planning and Load Testing for Streams with AI Consumers

Exam and certificate

The exam unlocks once you have completed every lesson. It draws 25 questions from a larger bank, spread across the modules, with the answer options shuffled for each attempt. You have 60 minutes and need 80% to pass. After a failed attempt you can try again after 24 hours, up to 3 attempts in total. Feedback is given per module, never as the answer key.

Passing earns a certificate of completion and assessment from the Wirewalk Institute of AI with a unique ID that anyone can check at institute.wirewalk.com/verify. It is not an accredited qualification or a vendor certification. How certification works.

The Wirewalk Institute of AI is independent. This course teaches publicly documented products and is not affiliated with or endorsed by Apache Kafka, Apache Flink, Confluent (Confluent Cloud, Schema Registry, Flink AI functions, Streaming Agents), Debezium, RabbitMQ, NATS, Amazon (SQS, SNS, EventBridge, Kinesis), Google (Cloud Pub/Sub), Microsoft (Azure Event Hubs, Service Bus), Temporal, or CloudEvents and AsyncAPI. Product and company names are trademarks of their respective owners.

More in AI in Software Development

All 19 courses in this track
dev-001 Free intro Available

Your First AI API Call: A One-Hour Hands-On Start

A free, one-hour, hands-on start built on one realistic job: sorting a property manager's 1,150 monthly maintenance requests. You get an API key the safe way, make real calls with curl and an official Python SDK, run them over forty work orders, read stop reasons, usage and errors, estimate the monthly cost, and decide whether the script is ready. Exercises run on your own provider account for well under a dollar.

1 h · 7 lessons Free intro
dev-201 Intermediate Available

Python AI Integration: SDKs, Async, Pydantic, Packaging and Testing

A code-first course on integrating LLMs into Python the way production code needs it: uv projects and secrets, the OpenAI, Anthropic and Google GenAI SDKs side by side, async concurrency and streaming, Pydantic structured outputs and tools, retries, local models, embeddings and RAG, FastAPI and pandas integration, testing and evals with pytest, observability, cost tracking and deployment.

15 h · 19 lessons See price at checkout
dev-202 Intermediate Waitlist

Perl and AI: Modernizing Scripts and Text Pipelines with LLM APIs

Call OpenAI, Anthropic, Gemini and local Ollama models from Perl with real CPAN modules, add streaming, concurrency and validated JSON output to text pipelines, and use AI coding assistants to modernize legacy Perl behind characterization tests.

12 h · 18 lessons Join the waitlist