If I had to start learning #dataengineering all over again, I’d follow this plan, mostly in order:
- Learn SQL
— Aggregations with GROUP BY
— Joins (INNER, LEFT, FULL OUTER)
— Window functions
— Common table expressions
- Learn about data modeling
— read about data normalization and 3rd normal form
— read about fact, dimension, and aggregate tables
— read about efficient table designs such as cumulative
— learn about complex data types like MAP, ARRAY, and STRUCT
- Learn Python
— Learn loops, if statements
— try out some really solid libraries like Pandas, numpy, scikit-learn, and Great Expectations
- Learn about data quality
— how do you write a good data check?
— how can you implement the write-audit-publish pattern in your pipelines?
- Learn about distributed compute
— Read about MapReduce and see how it guided the design of todays distributed computing
— learn about partitioning, skew, and spilling to disk
- learn about job orchestration
— read about CRON
— try out a scheduler like Airflow or Prefect
- Apply distributed compute principles
— a free trial with Snowflake or BigQuery is a good place to start
— or try out Spark + S3 if you want to be adventurous
If you follow this, you’ll feel way more confident in interviews.
173091.8K428K
