Dockerized Data & AI Stacks is a collection of ready-to-run development and learning environments I created for data engineering, distributed systems, machine learning, and deep learning.
The goal was simple: let developers start experimenting with a complete stack without spending hours installing runtimes, resolving dependency conflicts, or wiring services together by hand.
Why I built them
Tools such as Kafka and Spark are much easier to understand when developers can run a realistic multi-service environment, inspect its behavior, and write code against it. At the time, getting that environment onto a laptop often meant a long list of manual installation and configuration steps.
I packaged the infrastructure, developer tools, sample applications, data mounts, and startup scripts into reusable Docker environments. That made workshops, training, demos, and local experimentation more consistent across machines.
The stacks
Kafka in Docker
A local Apache Kafka environment that can scale from a minimal single-broker setup to a three-broker cluster with monitoring.
It includes configurations for ZooKeeper, Kafka brokers, a Kafka management UI, Java and Python development environments, Prometheus, Grafana, and supporting metrics services. The repository also includes sample applications and troubleshooting utilities for testing broker connectivity.
Spark in Docker
A Docker Compose environment for running an Apache Spark master and multiple workers on one machine.
The stack provides Java, Scala, Python, PySpark, and Jupyter tooling without requiring those runtimes to be installed on the host. It includes persistent workspace and data mounts, Spark web UIs, configurable worker counts, and sample applications in Java, Scala, and Python.
Data & AI Training Sandbox
An all-in-one teaching image designed to reduce setup time in machine learning, deep learning, and big-data courses.
The environment brings together Java, Scala, Spark, Kafka, ZooKeeper, Anaconda, TensorFlow, PyTorch, scikit-learn, Jupyter, and browser-accessible desktop tooling. Students can start with a consistent environment while keeping their project files and datasets mounted from the host.
View the Training Sandbox on Docker Hub →
BigDL and Analytics Zoo in Docker
A containerized environment for Intel BigDL and Analytics Zoo tutorials. It packages the framework and its dependencies with Jupyter, provides launch scripts, and mounts a persistent working directory so developers can run notebooks without assembling the full stack locally.
View BigDL and Analytics Zoo in Docker →
What I built across the collection
- Multi-container development environments with Docker Compose
- One-command startup and shutdown scripts
- Persistent mounts for source code, notebooks, and datasets
- Sample applications in Java, Scala, and Python
- Local web interfaces for Spark, Kafka, Jupyter, Grafana, and Prometheus
- Monitoring and metrics for multi-broker Kafka environments
- Workshop- and classroom-friendly environments that behave consistently across machines
- Documentation, quick starts, architecture notes, and troubleshooting guides
Developer experience focus
These projects were infrastructure work, but their purpose was developer experience. They turned complex distributed systems into something developers could launch, inspect, break, reset, and learn from on a single machine.
They also made technical training more reliable: everyone could begin with the same tools and configuration, leaving more time for code and concepts instead of environment setup.
Project status: These are earlier open-source projects and several images pin older component versions. They are best used as learning resources and reference implementations rather than production deployment templates.
Apache Kafka, Apache Spark, Kafka, Spark, and their logos are trademarks of The Apache Software Foundation. Docker and the Docker logo are trademarks of Docker, Inc. BigDL and Intel marks belong to their respective owners. The marks are used here only to identify the technologies; no affiliation or endorsement is implied.