In a recently published research called Ascent Journey 2016, the Atos Scientific Community considers the massive growth in data and storage as an important trend in IT.
Whilst the concept of Big Data has been around for a number of years and is relatively well understood, it is now becoming clear that everything we do is leaving a trail of data that can be analyzed and used.
Examples include the payments we make on a credit card, the books we read on an e-reader and our energy use by driving an electric car. This will lead to a new era of Total Data that, in turn, will lead to new business models, services and economic growth.
We don't yet understand all the implications of this – for businesses and society – but organizations that are able to harness and make sense of the vast quantities of heterogeneous data from disparate sources will gain valuable insights into market trends and opportunities.
An 'Ecosystem' of new management tools is taking shape, covering the various layers of the data stack in the enterprise and capable of delivering a 'Total Data' approach.
The technology that supports the Information Management Lifecycle in the enterprise is going through a profound change, due to the emergence of new solutions, many from open source background (NoSQL databases, Hadoop, analytical tools like R, visualization tools). To enable the 'Total Data' environment, the new technologies need to connect into and partly replace traditional technologies.
- In some scenarios, data must be obtained, processed and correlated with insights being derived and actions initiated as close to real time as possible.
Yesterday's data is not interesting unless it helps predict tomorrow. Yesterday's traffic report isn't helpful in plotting a journey today unless it is known to represent today's pattern as well, and combined with other data can improve future congestion. Pattern Based Strategy enables huge amounts of historical data to be analyzed for previously invisible patterns. These patterns give us the power to start predicting what is likely to happen in the future, so we can plan and improve, both in real-time and in non-real time scenario-planning. For example, real time predictive analysis will plot the route for transporting donor organs across a city safely and quickly, continuously adapting the route to changes in the traffic patterns as they are happening. Another example is a country to make compliance recommendations (and potentially becoming a legal requirement) to companies for maintenance regimes for their infrastructures or industry plants using analytics on historic data and thus establishing an automated "what are the lessons learned" process.
- Everything will be digital and everything will be connected.
Everything will be captured; "your life is becoming a video" – you can even replay your actions, thoughts and analyse in various forms and for multiple purposes (see http://quantifiedself.com/ for example); this is not only becoming possible for peoples life's, but anything that can be measured can be tracked, traced and put in a digital context for analysis. The ability of businesses to process this wealth of information is still unclear. What these developments – and others related such as 3-D printers and cognitive computers that will be able to replicate smell and touch for their users – mean for society, laws and concepts such as individual privacy need to be reassessed and will prove a huge challenge to governments, businesses and individuals in the 21st Century; for example long-established laws and concepts such as individual privacy need to be reassessed.
- After an initial confusion phase, traditional and 'Big Data' orientated approaches to analytics will converge in a unified 'Total Data' platform.
Big data relies on its sister technologies of optimized IT networks, rapid mobilization communication tools and cloud computing. Data Analytics as a Service could emerge from a combination of Big Data, Pattern Based Strategy and Cloud technologies. Business performance can improve in areas such as increased forecasting and enhanced automation capabilities and buildt new business propositions upon the discoveries they can do using Total Data as a source of undiscovered information.
[This blog post is a rewrite of http://blog.atos.net/blog/2013/02/22/the-data-explosion-is-real/?preview=true&preview_id=1555&preview_nonce=6df2f23c80 ]