Databricks announced that the company will contribute all features and enhancements it has made to Delta Lake to the Linux Foundation and open source all Delta Lake APIs as part of the Delta Lake 2.0 release. In addition, the company announced MLflow 2.0, which includes MLflow Pipelines, a new feature to accelerate and simplify ML model deployments. Finally, the company introduced Spark Connect, to enable the use of Spark on virtually any device, and Project Lightspeed, a next generation Spark Structured Streaming engine for data streaming on the lakehouse.
Databricks Launches SQL Analytics to Enable Cloud Data Warehousing on Data Lakes
November 14, 2020 by
Databricks, the data and AI company, announced the launch of SQL Analytics, which for the first time enables data analysts to perform workloads previously meant only for a data warehouse on a data lake. This expands the traditional scope of the data lake from data science and machine learning to include all data workloads including Business Intelligence (BI) and SQL.
Top 5 Mistakes When Writing Spark Applications
January 7, 2018 by
In the presentation below from Spark Summit 2016, Mark Grover goes over the top 5 things that he’s seen in the field that prevent people from getting the most out of their Spark clusters. When some of these issues are addressed, it is not uncommon to see the same job running 10x or 100x faster with the same clusters, the same data, just a different approach.
The Data Scientist’s Guide to Apache Spark
December 27, 2017 by
Looking to dive deeper into the more cutting edge machine learning use cases in Apache Spark? To successfully use Spark’s advanced analytics capabilities including large scale machine learning and graph analysis, check out The Data Scientist’s Guide to Apache Spark, from our friends over at Databricks.
Structuring Apache Spark 2.0: SQL, DataFrames, Datasets And Streaming
December 2, 2016 by
In the talk below, Michael Armbrust, gives an overview of some of the exciting new API’s available in Spark 2.0, namely Datasets and Structured Streaming. Together, these APIs are bringing the power of Catalyst, Spark SQL’s query optimizer, to all users of Spark.
Apache Spark MLlib 2.0 Preview: Data Science and Production
August 6, 2016 by
From the recent Spark Summit 2016 in San Francisco, the video presentation below by Joseph K. Bradley of Databricks give focus to “Apache Spark MLlib 2.0 Preview: Data Science and Production.”
Large-Scale Deep Learning with TensorFlow
June 18, 2016 by
We bring you the keynote presentation below from the recent Spark Summit 2016 held in San Francisco on June 6-8. Speaker Jeff Dean joined Google in 1999 and is currently a Google Senior Fellow.
Spark MLlib: Making Practical Machine Learning Easy and Scalable
November 23, 2015 by
In this talk, Xiangrui Meng of Databricks shares his experience in developing MLlib. The talk covers both higher-level APIs, ML pipelines, that make MLlib easy to use, as well as lower-level optimizations that make MLlib scale to massive data sets.
Advanced Apache Spark
November 13, 2015 by
Big data is going Spark crazy! Here’s a whopping 6 hour intensive, fast-paced and vendor agnostic look at Spark Core presented by Sameer Farooqui, a client services engineer at Databricks.
Apache Spark is the Smartphone of Big Data
November 9, 2015 by
In this special guest feature, Denny Lee of Databricks, talks about the versatility of Spark – essentially comparing it to the Swiss Army Knife of on your camping trip, called Big Data/Analytics.
