#dataengineer

If you want to get up to speed on data engineering, start with what the community voted for. These are my articles that hit the Hacker News front page or the top of r/dataengineering: 1) 101 Concepts Every Data Engineer Should Know (915+ upvotes on r/dataengineering) 2) Data Engineering Vault: the map of the entire field (HN front page) 3) The Data Engineering Toolkit: Essential Tools for Your Machine (221+ upvotes) 4) Building a Data Engineering Project in 20 Minutes (211+ upvotes) 5) A Diary of a Data Engineer (55+ upvotes) 6) Data Modeling, The Unsung Hero of Data Engineering: Architecture Pattern, Tools and the Future (HN front page) 7) Data Modeling Guide for Real-Time Analytics with ClickHouse (HN front page) 8) Why Semantic Layers Matter, and How to Build One with DuckDB (HN front page) 9) Why Pivot Tables Never Die (HN front page) 10) A Beginner's Guide to Geospatial with DuckDB All free, no signup. If you liked this post, repost & help others find it.

572 - Hard to name what a materialized lake view even is. Closest thing: a stored procedure that builds its own table when it finishes. Update bronze and everything downstream catches up. #microsoftfabric #lakehouse #sql #dataengineering

الحمدلله 🤍 اخذت شهاده Fabric Data Engineering من Microsoft في ضمن رحله تغيير مجالي الى هندسه بيانات صرت مهندس بيانات معتمد من مايكروسوفت في الـ Fabric. خطوة جديدة في عالم البيانات، والطريق مكمّل للي بعده بإذن الله #Microsoft #DataEngineer

Ask an AI system for monthly revenue. It generates valid SQL. The query runs. The numbers look reasonable. But did it count orders created, payments captured, or revenue after refunds? What timezone defines the month? The database schema might tell the model which tables to query. It rarely tells the model what your business means by “revenue.” That distinction matters. A better prompt cannot reliably compensate for undefined business semantics. Text-to-SQL needs more than schema awareness. It needs trusted metric definitions, appropriate access controls, and a way to verify that the query matches the actual question. Generating SQL is the easy part. Encoding business meaning is the harder one. #TextToSQL #DataEngineering #EnterpriseAI

This repo has all the resources you need to become a data engineer. 👇

The Analyst reveals the truth. The Engineer builds the backbone. The Scientist shapes the future. The role you choose defines the impact you make. #DataAnalyst #DataEngineer #DataScientist

If you had the chance to master Databricks or Snowflake and the related stacks for Data Engineering, which one would you go for? #DataAnalytics #DataEngineering #DataArchitecture #Databricks #Snowflake

570 - Work IQ, web IQ, foundry IQ, fabric IQ. Mike counted the branding and still can't get one agent to talk to his data without a fight. The friction is the whole problem. #microsoftfabric #powerbi #aiagents #dataengineering

Switching into data engineering? The interview won't stop at one SQL question. Start each one free on 1. SQL & PySpark in your browser 2. Data modeling canvas 3. 45-min AI mock interview 4. Kafka & Spark cohorts #dataengineering

Have expertise in modern #dataarchitecture, #cloudengineering or data transformation? We’ve got the job & benefits you’ve been waiting for at NSA. #dataengineer #datascience #datasciencejobs

Can’t believe it’s been 4 months already in the telecommunications space. Really nice stuff! What does growth mean to you? #RadioFrequencyEngineer #DataEngineer #DataAnalysis #Telecoms

What’s next for #AI, #ML & #DataEngineering in 2026? 🔹 Smaller models, smarter reasoning 🔹 AI agents scaling from individuals to entire organizations 🔹 AI evaluations ensuring agents do the right things 🔹 & more 🔗 Full #TrendsReport 👉 #InfoQ

#infographic: #DataEngineer Career Path Via @ingliguori #MachineLearning #DataScience #SQL #Cybersecurity #BigData #Analytics #AI #IIoT #Python #RStats #TensorFlow #FullStackDev #ReactJS #CloudComputing #DataScientist #Programming #Coding #IoT #100DaysofCode #NodeJS #GitHub

Master the SQL concepts used in everyday data analysis, from filtering, JOINs, and aggregations to CTEs, window functions, subqueries, and conditional logic. Learn practical techniques to write efficient queries and solve real-world data problems. #SQL #DataAnalytics #DataScience #SQLTips #DataEngineering #Database #DataAnalyst

572 - Microsoft's materialized view example gives a CSV file type, then quietly hands it a folder path instead of a file path. Read between the lines and you get incremental loading for free. #microsoftfabric #lakehouse #csv #dataengineering

Hiring remote: Data Platform Engineer United States | Data Engineer | Full-time Interested? Reply here with your email and we'll contact you. For more remote roles, check Hidden Jobs. #Hiring #DataEngineering

Day 7 Progressing through the @dotsetxyz Data Engineering cohort! • Today completed the task 3 • Try to complete task 4 #DataEngineering

Day 5, Day 6 Progressing through the @dotsetxyz Data Engineering cohort! On the 5th day, I couldn't make any significant progress because the workload was heavy.I am very far behind, and I am trying to move forward. Day 6: • Completed the first and second tasks.

Day 20 of my Data Engineering journey Completed Week 1 tasks. Finished 4 tasks of Week 2 . Received the GitHub Pull Shark badge. Created a key table for our project. #DataEngineering #GitHub #Snowflake #Dotset

Week 2: Loaded business data into Snowflake and started Python basics. Practiced string methods like len(), count(), upper(), lower(), find(), replace(), and format(), then created Python files and pushed them to the repo. Making progress with SQL, Snowflake, Python, and Git.

Machine Learning Model-Serving Patterns & Best Practices — Definitive guide to deploying, monitoring, & providing accessibility to ML models in production: —— #DataScience #DataScientist #MLOps #CDO #DataEngineering

#DataEngineering with Google Cloud Platform #GCP — A Guide to leveling up as a #DataEngineer by building a scalable data platform with Google Cloud: [2nd Edition] v/ @PacktDataML ————— #DataAnalytics #Analytics #CDO #CTO #AI #ML #MLOps #DataScience