Best viewed on a larger screen

This workbook (checklist, animated pipeline, and the interactive SQL/Python lab) needs more screen space to work properly. Please open this link on a tablet or computer.

https://data-engineering-workbook.vercel.app/

Your progress is saved automatically in this browser.
To take it with you, share it, or start fresh, use the buttons below.
LEGEND
MUSTEssential
SHOULDImportant
NICEOptional

DATA ENGINEERING ROADMAP 2026-2027

PROGRESSIVE DATA ENGINEERING LEARNING PATH

TARGET
Portfolio Ready
skills + practice + resources
1
THE BIG
PICTURE

How a real project is built

BUSINESS REQ
->
SOURCES
->
INGESTION
->
STORAGE
->
TRANSFORM
->
QUALITY
->
ORCHESTRATION
->
CONSUMPTION
LAB
PIPELINE
SIMULATOR

Real Estate Data Platform

Horizon Estates — Live Data JourneyWatch files, events and tables change form from source to business value
SOURCESWhere data is born
INGESTIONExtract & capture
BRONZERaw & immutable
PROCESSINGClean & transform
SILVERValidated & standard
GOLDBusiness-ready
CONSUMPTIONDecisions & products
TRANSFORMATION & DATA STORAGERaw data → cleaned data → modeled business data
APIREST / SaaS APIsJSON · OAuth · pagination · rate limits
PostgreSQLOperational DBsPostgreSQL · MySQL · SQL Server
Partner filesCSV · Excel · JSON · XML
PythonPython ingestionrequests · auth · pages · retry
AirbyteAirbyte / FivetranConnectors · CDC · incremental
File loadervalidate · checksum · manifest
Apache KafkaApache Kafkaevents · topics · partitions
S3 DATA LAKE — Bronzeraw · immutable · replayable
PythonPython / Polarsclean · enrich · normalize
Apache SparkApache Sparkdistributed batch + stream
Quality gatetests · quarantine · retry
QUARANTINE / DLQinvalid rows and events for replay
S3 DATA LAKE — Silverclean · typed · deduplicated
AMAZON ATHENASQL query engine over Silver Parquet
dbtdbt + SQL TRANSFORMATIONAthena adapter · CTEs · tests · incremental models
REDSHIFT DATA WAREHOUSESQL facts · dimensions · marts
Microsoft Power BIdashboards · governed KPIs
Data APIstrusted metrics for apps
ML / Notebooksforecast · features · AI
SQLSQL ANALYTICSSELECT · JOIN · CTE · window functions
File schema + checksum
Event schema + duplicates + late data
Nulls + types + business rules
dbt SQL tests
Gold reconciliation
ORCHESTRATION & DELIVERY CONTROL — commands travel; business data does notAirflowApache Airflow
schedule · DAG · retryGitHub ActionsGitHub Actions
test · build · deploySecrets
credentials · config
FOUNDATIONS & CROSS-CUTTING CONCERNSLinux · Git · Docker · SQLCatalog · Lineage · ContractsLogs · Metrics · AlertsIAM · Encryption · Governance
OBSERVABILITYCloudWatch / OpenTelemetry — task logs, row counts, latency, freshness, failures and alerts
JSON CDC CSV RAW.JSON SNAPSHOT FILE.CSV KAFKA EVENT CSV/JSON PARQUET CLEAN.PQ INVALID EVENT SILVER.PQ SQL VIEW FACT+DIMS DATASET KPI.JSON FEATURE SET SQL RESULT RUN JOB SCHEDULE
STEP 1/7 — EXTRACT: Python requests authenticates and fetches the next JSON page from the listings API.
Swipe to explore
Horizon Estates Data PlatformProperty listings, leasing, finance and market intelligence - AWS production architecture
All systems operational
ORCHESTRATION & CONTROL PLANECommands, dependencies, schedules and recovery
1. DATA SOURCESReal estate operations
2. INGESTIONCollect and capture
3. EVENT STREAMReal-time property events
4. PROCESSINGTransform and model
PASS -> publish trusted data
FAIL -> alert, quarantine, retry
5. AWS LAKEHOUSES3 + Parquet layers
6. WAREHOUSEAnalytics and semantics
7. CONSUMPTIONDecisions and products
GOVERNANCE, OBSERVABILITY & RECOVERY
ENGINEERING FOUNDATIONS FROM THE ROADMAPThey support every stage; data does not flow through all of them.
Pipeline status
Current rundag_2026_07_11_0600
Rows processed0
Quality pass rate--
Live activity
Ready - Airflow is waiting for the daily schedule or an event trigger.
DataControlMetadata / lineageAlert / recovery
Select a tool: see when it acts, what it receives, and what it sends to the next system.
2
FOUNDATION
SKILLS

Used throughout the entire pipeline

SQL - Universal Data Language
  • SELECT / WHERE / GROUP BY
  • JOINs (INNER, LEFT, FULL)
  • CTEs & Subqueries
  • Window Functions
  • Query Optimization
  • DDL / DML
  • Indexing Strategy
  • Transactions & ACID
  • Views & Materialized Views
  • Analytics Queries
Python - Automation Engine
  • Data structures
  • File I/O (CSV, JSON, Parquet)
  • requests & API handling
  • Virtual environments
  • Error handling & logging
  • OOP for pipelines
  • Type hints & testing
  • Package management
  • pandas / Polars
  • Automation scripts
Data Modeling - Business Structure
  • OLTP vs OLAP
  • Star / Snowflake Schema
  • Fact & Dimension Tables
  • SCD Types 1, 2, 3
  • Grain & Keys
  • Kimball / Inmon
  • ERD Design
  • Naming Conventions
3
PIPELINE
LAYERS

Detailed breakdown

1. SOURCES
MUST LEARN
  • REST APIs / JSON
  • Authentication
  • Pagination
  • Rate limits
  • CSV / Excel / Parquet
  • PostgreSQL / MySQL
SHOULD KNOW
  • Salesforce
  • SAP / HubSpot
NICE TO KNOW
  • Kafka
  • Event streams
2. INGESTION
MUST LEARN
  • ETL / ELT
  • Batch & Stream
  • CDC
  • Retry logic
  • Incremental loads
  • Error handling
SHOULD KNOW
  • Airbyte
  • Fivetran
  • Meltano
NICE TO KNOW
  • Informatica
  • Talend
3. STORAGE
MUST LEARN
  • PostgreSQL / S3
  • Partitioning
  • Row vs Columnar
  • Compression
  • Lifecycle policies
SHOULD KNOW
  • Snowflake
  • BigQuery
  • Delta / Iceberg
NICE TO KNOW
  • Redshift
  • Synapse
4. TRANSFORMATION
MUST LEARN
  • SQL CTEs / Windows
  • DataFrames
  • dbt Models & Tests
  • Data cleaning
  • Medallion layers
SHOULD KNOW
  • Spark / PySpark
  • Databricks
NICE TO KNOW
  • Flink / Beam
5. ORCHESTRATION
MUST LEARN
  • Airflow DAGs
  • Scheduling
  • Retries & Alerts
  • Connections
  • Monitoring
SHOULD KNOW
  • Dagster
  • Prefect
6. DATA QUALITY
MUST LEARN
  • Null / Duplicate checks
  • Schema validation
  • Business rules
  • dbt tests
SHOULD KNOW
  • Great Expectations
NICE TO KNOW
  • Soda
  • Monte Carlo
CONSUMPTION
MUST LEARN
  • SQL analytics
  • Business queries
SHOULD KNOW
  • Power BI
  • Tableau
  • Looker
NICE TO KNOW
  • ML pipelines
  • GenAI / RAG
  • Streaming
4
ENGINEERING
LAYER

Cross-cutting skills

7. DEVOPS & GIT
  • Git / Branching
  • Linux / Bash
  • Docker / Compose
  • CI/CD
  • Kubernetes
8. CLOUD
  • AWS IAM / S3
  • RDS / Lambda
  • CloudWatch
  • Secrets Manager
  • Azure / GCP
9. GOVERNANCE
  • Data ownership
  • Data contracts
  • Catalog & Lineage
  • Retention policies
10. OBSERVABILITY
  • Logging & Metrics
  • Alerts & SLAs
  • Grafana / Datadog
  • OpenTelemetry
1.1. ARCHITECTURE
  • Batch / Incremental
  • Event-driven
  • Lakehouse
  • Modern Data Stack
5
LEARNING
PATH

12+ months journey

0. BUSINESS REQUIREMENTS
  • Stakeholder mapping
  • Data needs
->
1. FOUNDATION
  • SQL + Python
  • Modeling basics
->
2. SOURCES
  • APIs & Connectors
  • Files & Data
->
3. INGESTION
  • ETL / ELT
  • Batch & CDC
->
4. STORAGE
  • PostgreSQL & S3
  • Partitioning
->
5. TRANSFORMATION
  • SQL Advanced
  • dbt & Modeling
->
6. QUALITY
  • Data Quality
  • Testing
->
7. ORCHESTRATION
  • Airflow
  • Scheduling
->
8. DEVOPS
  • Git & Docker
  • CI/CD
->
9. CLOUD
  • AWS S3 & RDS
  • Secrets
->
10. GOVERNANCE
  • Ownership
  • Contracts
->
11. OBSERVABILITY
  • Monitoring
  • Alerts
->
12. ARCHITECTURE
  • Patterns
  • Design
->
13. CONSUMPTION
  • Analytics
  • BI Tools

INTERACTIVE CHECKLIST - Track Your Progress

0%
Must Learn (green) Should Learn (blue) Nice to Know (white)