WikiStream Event Data Analytics Pipeline in AWS
Content monitoring analytics service using latest AWS S3 Tables along with MSK, EMR (SLA=20 mins)
0.0
0 reviews•
Apache Spark·
AWS·
deequ·
SQL·
Apache Iceberg
AWS native data pipeline based on lambda architecture that captures live Wikipedia edits from Wikimedia EventStreams, processes them through a medallion architecture, and surfaces insights via dashboa...
About this project
AWS native data pipeline based on lambda architecture that captures live Wikipedia edits from Wikimedia EventStreams, processes them through a medallion architecture, and surfaces insights via dashboards for content monitoring and risk detection with pipeline observability. It has streaming EMR jobs to continuously collect data real-time and batch EMR jobs to perform data processing for further analytics in loop. It has data quality gates for each data layer. You also have runbook and scripts to easily config and start the whole pipeline end-to-end
Stack:
Apache Spark