The 2022 Data Engineering Survey, from our friends over at Immuta, examined the changing landscape of data engineering and operations challenges, tools, and opportunities. The modern data engineering technology market is dynamic, driven by the tectonic shift from on-premise databases and BI tools to modern, cloud-based data platforms built on lakehouse architectures.
More than the on-premises market that preceded it, the cloud data technology market is evolving rapidly, and spans a vast set of open source and commercial data technologies, tools, and products. At the same time, organizations are adopting multiple technologies to keep up with the scale, speed, and use cases that today’s data environment demands. To remain competitive and maximize the value of their data – including sensitive data – organizations are developing DataOps functions and frameworks to varying degrees. DataOps tools and processes enable continuous and automated delivery of data to power BI, analytics, data science, and data-powered products.
This comprehensive survey examines the changing landscape of data engineering and emerging challenges such as data privacy, security, and quality as companies shift data management from on-premises databases and BI tools to cloud-based data platforms.
The survey probed data leaders, architects, and engineers about the trends they’re seeing, challenges they face, and what they expect for the coming years, including:
- The most and least challenging tasks that data teams face with data pipelines
- Why the modern data landscape and data use are causing new challenges to emerge
- The top cloud data platforms and tools that data teams are adopting to maximize their data’s value
This report aims to help data teams understand and learn from the choices others are making, and the challenges they face as the data landscape continues its rapid evolution.
Download the full report HERE.
Sign up for the free insideAI News newsletter.
Join us on Twitter: @InsideBigData1 – https://twitter.com/InsideBigData1
