Your Future; Now available in real-time

cobblestones (road)Imagine you have an automatically and real-time updated agenda – it continuously adapts your schedule to meetings taking longer, predicts and updates in real-time your travel-time to the next meetings and will adapt your schedule because it ‘knows’ that typically any meeting with your best client always takes 30 minutes longer than you originally plan it for.

A proof of concept conducted by the Atos Scientific Community looked at this aspect of predictability and took the data of the traffic in the city of Berlin to see if it was possible to do real time traffic forecasting (RTTF). The result is in a recently published white paper.

  “RTTF enables a prediction (within 1 minute) of sensor data streams for the immediate future (up to four hours) and provides traffic condition classification for the upcoming time period based on the forecasted data.”

“The forecast provides a suitable time span for proactively managing upcoming incidents even before they appear.”

The team took a radical different approach to the challenges of today’s traffic management. Instead of proposing another reactive traffic management IT system with some smart analytics, the team targeted successfully a proactive traffic management approach which provides analytics solutions to predict critical events in advance before they appear.  Using historic data and artificial neuron network technology, predictions are created for the intermediate future and utilized to determine the traffic status of the upcoming next four hours. Based on that information, actions can be taken proactively to mitigate or avoid future upcoming events. Utilizing the software and bringing in data scientists with an understanding of the context was the next step. This helped in defining the right parameters and a pattern based strategy (PBS) in place.

“Being able to identify patterns out of the existing data, model them into patterns and come up with a system that can provide reliable predictions is a remarkable achievement in itself, but the true value of PBS is being able to apply such capabilities to strategy definition and decision making.”

Working with the subject matter experts the team identified multiple models that were then consequently implemented in the software. The models are important, they avoid that you are trapped into simplification; when a car is driving slowly, it can be because of a traffic jam, but it can also be an older person driving more carefully.

By introducing the concept of ‘flow’ – the number of vehicles passing a sensor each hour – the team could identify 4 different states, which were in themselves also parameterized by looking at road capacity, speed limits, etc. This information is then fed into a look-up table based complex event processing engine in order to predict, within 1 minute, the traffic situation at given locations.

Because in real-life the historic data is continuously refreshed with the actual events of the past time, the system will be able to predict in real-time the situation on the road.

The proof of concept clearly showed that a self-learning system, combined with a complex event processing unit and the help of some subject matter expert data scientist can accurately predict the future – the white paper shows this in some great details.

  “Real Time Traffic Forecasting is an excellent example of how data sources and identified patterns can be exploited to gain insights and to develop proactive strategies to deal with upcoming events and incidents. It enables a short term view into the future which is long enough to act on predicted incidents rather than react on occurring ones”

For me this proof of concept shows the benefits of data analytics in everyday life, and I am looking forward to this future.

This blog post was previously published at 

Curiosity drives cloud computing

I like asking questions and I like getting good answers even better. It is because of that, I now have a love / hate relationship with search engines. Most of the time they give me a 50% answer, a kind of direction, a suggestion, a kind of coaching to the real answer. It is like the joke about the consultant; “the right answer must be in there somewhere, because he or she gives me so many responses”.

PH03797IIn spite of all kind of promises, search engines have not really increased their intelligence. Complex questions with multiple variables are still nearly impossible to get answered and the suggestions to improve my question are mostly about my spelling or because the search engine would have liked a different subject to be questioned on.

So nothing really good is coming from search engines then? Well most arguably search engines have brought us cloud computing and a very powerful access to lots and lots and lots of data, otherwise known as ‘the world wide web’.

No wonder I envision that powerful access and cloud computing are the two most important values we want to keep while increasing the capacity and intelligence to do real analytics on large data sets.

In a whitepaper of the Atos Scientific Community, these 2 elements are explored in great depth:

  • Data Analytics needs cloud computing to create an “Analytics as a Service” – model because that model addresses in the best way how people and organizations want to use analytics.
  • This Data Analytics as a Service – model (DAaaS) should not behave as an application, but it should be available as a platform for application development.

The first statement on the cloud computing needs suggests we can expect analytics to become easily deployed, widely accessible and not depending on deep investments by single organizations; ‘as a service’ implies relatively low cost and certainly a flexible usage model.

The second statement about the platform capability of data analytics however, has far reaching consequences for the way we implement and build the analytic capabilities for large data collections.

Architecturally, and due to the intrinsic complexities of analytical processes, the implementation of DAaaS represents an important set of challenges, as it is more similar to a flexible Platform as a Service (PaaS) solution than a more “fixed” Software as a Service (SaaS) application

It is relatively easy to implement a single application that will give you an answer to a complex question; many of the applications for mobile devices are built on this model (take for example the many applications for public transport departure, arrival times and connections).

This “1-application-1-question” approach is in my opinion not a sustainable business model for business environments; we need some kind of workbench and toolkit that is based on a stable and well defined service.

The white paper describes a proof of concept that has explored such an environment for re-usability, cloud aspects and flexibility. It also points to the technology used and how the technology can work together to create ‘Data Analytics as a Service’.

This blog post was previously published at