ViewTube

Azure Databricks Learning: Performance Optimization: Spark/Databricks Interview Question Series - II ...

Apache Spark™️ has become the de-facto open-source standard for big data processing due to its ease of use and performance ...

The SQL tab in the Spark UI provides a lot of information for analysing your spark queries, ranging from the query plan, to all ...

In data warehouse area, it is common to use one or more columns in complex type, such as map, and put many subfields into it.

Over the years, there has been extensive and continuous effort on improving Spark SQL's query optimizer and planner, in order to ...

You've seen the technical deep dives on Spark's Catalyst query optimizer. You understand how to fix joins, how to find common ...

Uneven distribution of input (or intermediate) data can often cause skew in joins. In Spark, this leads to very slow join stages ...

User Defined Functions is an important feature of Spark SQL which helps extend the language by adding custom constructs.

Nowadays, Spark is widely adopted in the big enterprise by handling the large volume of data. In PayPal, more and more complex ...

This talk will break down merge in Delta Lake—what is actually happening under the hood—and then explain about how you can ...

The Delta Architecture pattern has made the lives of data engineers much simpler, but what about improving query performance ...

Bucketing is commonly used in Hive and Spark SQL to improve performance by eliminating Shuffle in Join or group-by-aggregate ...

Parquet is a very popular column based format. Spark can automatically filter useless data using parquet file statistical data by ...

One of the most significant benefits provided by Databricks Delta is the ability to use z-ordering and dynamic file pruning to ...

Learn PySpark, an interface for Apache Spark in Python. PySpark is often used for large-scale data processing and machine ...

Defining customized scalable aggregation logic is one of Apache Spark's most powerful features. User Defined Aggregate ...

John submits a query and expects it to run smoothly. Based on his prior experience, he anticipates the query to finish in 20 mins.

Join us for a four part learning series: Introduction to Data Analysis for Aspiring Data Scientists. This is the fourth of four online ...

This talk will introduce TeraCache, a new scalable cache for Spark that avoids both garbage collection (GC) and serialization ...

It's very easy to be distracted by the latest and greatest approaches with technology, but sometimes there's a reason old ...