#dataengineering

Getting into #dataengineering is actually pretty easy - learn SQL - learn Python - learn Snowflake/BigQuery/DataBricks - learn data modeling - learn data pipelines with Airflow If you learn these 5 things, you’ll be interview-ready for a junior position for sure

Level 1 data engineers: I use SQL Level 2 data engineers: SQL is hard to test, you need TDD in your pipelines, data frames only! Level 3 data engineers: I use SQL and dbt #dataengineering

If I had to start learning #dataengineering all over again, I’d follow this plan, mostly in order: - Learn SQL — Aggregations with GROUP BY — Joins (INNER, LEFT, FULL OUTER) — Window functions — Common table expressions - Learn about data modeling — read about data normalization and 3rd normal form — read about fact, dimension, and aggregate tables — read about efficient table designs such as cumulative — learn about complex data types like MAP, ARRAY, and STRUCT - Learn Python — Learn loops, if statements — try out some really solid libraries like Pandas, numpy, scikit-learn, and Great Expectations - Learn about data quality — how do you write a good data check? — how can you implement the write-audit-publish pattern in your pipelines? - Learn about distributed compute — Read about MapReduce and see how it guided the design of todays distributed computing — learn about partitioning, skew, and spilling to disk - learn about job orchestration — read about CRON — try out a scheduler like Airflow or Prefect - Apply distributed compute principles — a free trial with Snowflake or BigQuery is a good place to start — or try out Spark + S3 if you want to be adventurous If you follow this, you’ll feel way more confident in interviews.

Python offers many powerful libraries for data visualization, including Matplotlib, Seaborn, Plotly, and many more. #DataScience #pythonprogramming #DataEngineering #dataviz #DataScientist #Database #codinglife

Here's how I would learn #dataengineering basics in 2023: - Find a data source you care about (examples: gaming APIs, stock market, web scraping, etc) - Use Python to interact and ingest your source. Initially just write the data to a CSV. 1/3

Quick guide to go from 0 to #dataengineering hero: - learn SQL Data Lemur is a great resource here - learn Python Do like… 30-40 leetcode easy and medium questions - distributed compute Get a trial of Databricks or Snowflake and find a training to learn about it 1/3

Here’s the LLM-driven data engineering lecture I did. Completely free. Code based linked as well! Here’s the lab too Almost 2 hours of content here! #dataengineering

SQL interviews are common in data engineering. They’re even more common in big tech. I wrote an article today revealing everything I know about them in my nine years of data engineering experience! Link in my bio since Elon would bury it otherwise! #dataengineering

The following books will change your #dataengineering career forever: Fundamentals of Data Engineering by Joe Reis and Matt Housley Designing Data Intensive Applications by Martin Kleppman

The data modeling round in big tech interviews weeds out the DEs who can't solve vague business problems! I wrote a free article about everything you need to know to pass these interviews! Link in my bio since Elon would downrank otherwise! #dataengineering

When I worked at Netflix, I built pipelines that processed over 2000 terabytes per day, data pipelines play by different rules when you get to this scale. I go into more detail here in this 2 min YouTube video you should check out! #dataengineering

My data engineer handbook made it to the top spot of the trending page on GitHub today! Let’s go! I’m so happy my hard work contributing to open source is paying off! #dataengineering

Window functions are very important in data engineering interviews! The “second highest salary” question is notorious in industry. I’ve seen it 3-4 times in interviews! If you like this content, please follow my IG at #dataengineering

Don’t stop at SQL and Python when learning #dataengineering Add: - distributed computation - data modeling - bash/docker/dev ops - a statically typed language like Java you’ll make a lot more money if you do this

Switching into data engineering? The interview won't stop at one SQL question. Start each one free on 1. SQL & PySpark in your browser 2. Data modeling canvas 3. 45-min AI mock interview 4. Kafka & Spark cohorts #dataengineering

My favorite stack to build a data analytics product - Apache Spark (for processing) - Amazon S3 (for storage) - Apache Iceberg (for metadata) - Apache Airflow (for scheduling) - Apache Superset (for visualization) - Great Expectations (for data quality) #dataengineering

Please stop using sub queries in your pipelines! #dataengineering

When you get to make data engineering content with Joe Reis for a course featuring Andrew Ng too This stuff is going to 🌙! #dataengineering

Things you should never see in production SQL pipelines: - SELECT * - RIGHT JOIN - GROUP BY 1,2,3 / ORDER BY 1,2,3 - Derived columns without aliases - Nested subqueries What would you add? #dataengineering

Please never use COUNT(*) in your SQL. It’s bad and unnecessarily selects all the columns. Use COUNT(1) for a basic row count. Or COUNT(column) for the count of a specific column. #dataengineering