E2E Real-Time Data Pipeline

📖 Overview

This project demonstrates a real-time end-to-end (E2E) data pipeline designed to handle clickstream data. It shows how to ingest, process, store, query, and visualize streaming data using open-source tools, all containerized with Docker for easy deployment.

🔎 Technologies Used:

• Data Ingestion: Apache Kafka

• Stream Processing: Apache Flink

• Object Storage: MinIO (S3-compatible)

• Data Lake Table Format: Apache Iceberg

• Query Engine: Trino

• Visualization: Apache Superset

Flow

1. Clickstream Data Generator simulates real-time user events and pushes them to Kafka topic.

2. Apache Flink processes Kafka streams and writes clean data to Iceberg tables stored on MinIO.

3. Trino connects to Iceberg for querying the processed data.

4. Apache Superset visualizes the data by connecting to Trino.

🏆 Key Features

🔄 Real-Time Data Processing

• Stream processing with Apache Flink.

• Clickstream events are transformed and filtered in real-time.

📂 Modern Data Lakehouse

• Data is stored in Apache Iceberg on MinIO, S3 compatible, supporting schema evolution and time travel.

⚡ Fast SQL Analytics

• Trino provides fast, distributed SQL queries on Iceberg data.

📊 Interactive Dashboards

• Apache Superset delivers real-time visual analytics.

📦 Fully Containerized Setup

• Simplified deployment using Docker and Docker Compose for seamless integration across all services.