Data Warehouses Ascend to the Cloud

Data warehouses have been in use since the mid to late 1970s. Substantial investments have since been made in technology and labor to develop, deploy, manage, and maintain data warehouses of various sizes. IDC research indicates that, during the past quarter century, about $100 billion in aggregate has been invested in data warehouse software alone. Should these investments continue? Have they added value, and do they continue to do so? These are critical questions that today's IT and business decision makers must consider.

Obviously, the relational database software that underpins data warehouses (as well as hardware infrastructure) has changed over the past decades but most of the concepts of data warehousing have remained the same. In other words, the warehouse metaphor works: Various tools are used to move data to a large storage location where the data is organized in a manner that optimizes its retrieval for shipment to other tools that present the data to decision makers. From this perspective, data warehousing has been an unquestionable success.

For many organizations, the relational database has become the foundation of their analytics architecture — a highly performant, governed, trusted, and secure source of integrated enterprise data that supports a range of business intelligence and analytics workloads and use cases. Yet some data warehouses have also become victims of their own success due to these characteristics. IT's tight control over the data warehouse makes it a less than ideal data processing environment for rapid experimentation including the addition of new types of internal log data (either from clickstream or sensor activity) or external data of various types. A recent IDC survey found that, in the past 24 months, 77% of large and midsize organizations in North America expanded the data types and sources they analyze.