#SparkSQL

Learn how to connect #SparkSQL and #ApacheKafka with ease using Structured Streaming #ApacheSpark

Introducing New Built-in Functions and Higher-Order Functions for Complex Data Types in Apache Spark 2.4 #ApacheSpark #SparkSQL

Big data made easy - An intro to #SparkSQL. Tune in: by @AskTheData

Learn about SQL Pivot: Converting Rows to Columns in Apache Spark 2.4 #ApacheSark #SparkSQL

Found this filter logical optimization in #ApacheSpark #SparkSQL quite interesting 🤔 Quiz: 1️⃣(easy) Can you explain how the value attribute got this extra filter condition? 2️⃣(hard) What's the name of the optimization?

Analyzing data in seconds with Python, SparklineSNAP and @nteractio Exploratory data analysis and drag and drop B.I can exist side by side. See how you can multi-dimensionally explore a 100 million row dataset. #spark #sparksql #python #fastbi

One of the new features in #ApacheSpark #SparkSQL 3.0 is "Observable Metrics" for batch and streaming queries. 👉 Thanks @Westerflyer 👏 Highlighted: 👉 👉

Learning #SparkSQL! #BigData #Analytics #DataScience #IoT #IIoT #PyTorch #Python #RStats #TensorFlow #Java #JavaScript #ReactJS #GoLang #CloudComputing #Serverless #DataScientist #Linux #Books #Programming #Coding #100DaysofCode

You can use multi-id aliases in #SparkSQL 👍 👉 Exactly the reason why I **love** exploring the sources of #ApacheSpark (and reading docs of various SQL engines, e.g. Snowflake) ❤️ 👉

It's hard to believe that #ApacheSpark was first released as a research project 10 years ago! My @SparkAISummit keynote (live now) goes through the lessons in the past 10 years and what's new in #ApacheSpark 3.0.

#ApacheSpark today is very different from the research version released in 2010. First of all, the most widely used programming language for Spark is now Python. On @Databricks, we see that 68% of commands in notebooks are in Python, which is 6x more than Scala.

Also, most computations on Spark today run through the #SparkSQL engine, added in 2014. Even in Python and Scala, 90% of API calls go through the DataFrame API that runs on Spark SQL. Thus the community has invested a lot in Spark SQL recently, with Spark 3.0 giving a 2x speedup.

I’m sold to the idea of recording videos about #ApacheSpark #SparkSQL that’d be no longer than 5-10 minutes (such short videos have always worked best for me). Anything in particular you’d like to watch first?

Learn how to use SQL Pivot: Converting Rows to Columns. #ApacheSpark #SparkSQL #SQL

Watch this! » #ApacheSpark Performance Tuning - Part 1 #SparkSQL

#BigQuery Storage and #SparkSQL. #BigData #Analytics #DataScience #AI #MachineLearning #ApacheSpark #Jupyter #IoT #IIoT #PyTorch #Python #RStats #JavaScript #ReactJS #CloudComputing #Serverless #DataScientist #Linux #Programming #Coding #100DaysofCode

Adaptive Query Execution in #SparkSQL is a set of features that can optimize the query execution at runtime. Today it's time to discover the first of them called shuffle partitions coalesce. If you want to learn more about it 👉

What a day! It’s only now when I can consider myself able to explain conversions between #SparkSQL abstractions (LogicalPlan, QueryExecution and Dataset/DataFrame). The talk at @SparkAISummit in London is going to be very in-depth! #ApacheSpark

Some time ago, I was analyzing the query plan generated by my #ApacheSpark app and found a mysterious partial operations. I decided then to investigate 🕵️‍♀️ and today's post starts a series about aggregations in #SparkSQL

Get a deep dive into the technical details of #SparkSQL at #SparkSummit East

(new in Spark 2.2) Loading datasets from Dataset[String] for CSV/JSON #SparkSQL #ApacheSpark

In March I wrote about partition-based sort in #ApacheSpark #SparkSQL. This feature is used in partitionBy method I described in my today's post