The Digital Shelf Automated ETL Data Pipeline
An end-to-end automated ETL (Extract, Transform, Load) data pipeline that processes raw book collect
0.0
0 reviews•
Apache Airflow·
PostgreSQL·
Python·
Snowflake·
SQL
Data Cleaning & Normalization: Automated Python pipeline using Pandas for text standardization, missing value imputation, and schema enforcement.Cloud Data Warehousing: Automated data loading to Snowf...
About this project
Data Cleaning & Normalization: Automated Python pipeline using Pandas for text standardization, missing value imputation, and schema enforcement.
Cloud Data Warehousing: Automated data loading to Snowflake using snowflake-connector-python and optimized bulk operations via write_pandas.
Workflow Orchestration: Managed DAG execution with Apache Airflow including dependency management, failure handling, and scheduled batch runs.
Containerized Infrastructure: Fully isolated environment running Airflow and PostgreSQL services via Docker Compose.
Security & Best Practices: Environment variables management (.env) preventing credential exposure and preventing redundant backfills (catchup=False).
Stack:
Apache Airflow