Global Sources
EE Times-India
Stay in touch with EE Times India
EE Times-India > Embedded

Open-source tool matches Watson's prowess

Posted: 15 Dec 2014     Print Version  Bookmark and Share

Keywords:DARPA  Watson  DeepDive  artificial intelligence 

If you are desperate to get our hands on a Watson-level artificial intelligence tool, then your prayers have been answered. DeepDive is a free version of IBM's Watson developed in the same Defence Advanced Research Projects Agency (DARPA).

Although it's never been pitted against IBM's Watson, open-source DeepDive has gone up against a more fleshy foe: the human being. Result: DeepDive beat or at least equalled humans in the time it took to complete an arduous cataloguing task. These were no ordinary humans, but expert human cataloguers tackling the same task as DeepDive, to read technical journal articles and catalogue them by understanding their content.


DeepDive was developed in the same DARPA program as IBM's Watson, but is being made available free by its programmers at University of Wisconsin-Madison. (Image: University of Wisconsin-Madison)

"We tested DeepDive against humans performing the same tasks, and DeepDive came out ahead or at least equalled the efforts of the humans," stated professor Shanan Peters, who supervised the testing.

DeepDive is free and open-source, which was the idea of its primary programmer, Christopher Re.

"We started out as part of a machine-reading project funded by DARPA in which Watson also participated," noted Re, a professor at the University of Wisconsin. "Watson is a question-answering engine (although now it seems to be much bigger). [In contrast] DeepDive's goal is to extract lots of structured data" from unstructured data sources.

DeepDive incorporates probability-based learning algorithms as well as open-source tools such as MADlib, Impala (from Oracle), and low-level techniques such as Hogwild, some of which have also been included in Microsoft's Adam. To build DeepDive into your application, you should be familiar with SQL and Python.

"Underneath the covers, DeepDive is based on a probability model; this is a very principled, academic approach to build these systems, but the question for use was, 'Could it actually scale in practice?' Our biggest innovations in Deep Dive have to do with giving it this ability to scale," Re responded.


DeepDive came out ahead or at least equalled the efforts of the humans.

For the future, DeepDive aims to be proven in other domains.

"We hope to have similar results in those domains soon, but it's too early to be very specific about our plans here," Re stated. "We use a RISC processor right now, we're trying to make a compiler, and we think machine learning will let us make it much easier to program in the next generation of DeepDive. We also plan to get more data types into DeepDive: images, figures, tables, charts, spreadsheets, a sort of 'Data Omnivore' to borrow a line from Oren Etzioni."

- R. Colin Johnson
  EE Times

Comment on "Open-source tool matches Watson's pr..."
*  You can enter [0] more charecters.
*Verify code:


Visit Asia Webinars to learn about the latest in technology and get practical design tips.


Go to top             Connect on Facebook      Follow us on Twitter      Follow us on Orkut

Back to Top