In what seems to be a major step forward in solving big data problems and advancing research capabilities, IBM announced today that it was awarded by the U.S. Department of Energy a $325 million contract for building the world’s most advanced data-centric super-computer, by 2017.

The data-centric concept is explained by IBM as one which “moves much of the processing to the places where the data is stored, whether that’s within a single computing system, in a network of computers, or far away in sensors tracking the weather or monitoring an energy pipeline”. This is needed because, particularly in the context of big data, one of the main challenges is not the actual processing power, but the speed at which vast quantities of data can be transported to and from the processing facilities.

Until now, most hopes for revolutionizing big data processing were with quantum computing. Quantum computers are particularly suitable to tackle big data because of their ability to simultaneously process different approaches to the same problem. Progress in the area of quantum computing has been slower than expected, because among other issues, problems need to be defined in a whole new way in order to be solved using this technology.

For quantum computing, the Canadian company D-Wave leads the game with high profile clients such as Google, NASA and (reportedly) NSA

In the new contract with DoE, IBM brings the data-centric computing approach, and has joined efforts with NVidia (GPU and interconnectivity) and Mellanox Technologies (interconnectivity).

The level of investment, effort and risk situates these projects well out of reach for the majority of regular companies and research institutions. However, with more options developing, we are likely to see new methods and approaches for dealing with big data, which will will be gradually mainstreamed.

A few links:

– the concept of data-centric computing explained in (rather) simple terms:

http://www.engadget.com/2014/11/14/ibm-data-centric-computing/

– IBM’s data-centric design vision:

http://research.ibm.com/articles/datacentricdesign/

– a video (trying) to describe some quantum computing terms:

http://mashable.com/2013/10/13/google-quantum-computing-video/