AI teaching assistant answers inquiries with 97% accuracy

That's an ideal situation for the Watson platform, which specialises in answering questions with distinct, clear solutions. The team wrote code that allows Jill to field routine questions that are asked every semester. For example, students consistently ask where they can find particular assignments and readings.

Jill wasn't very good for the first few weeks after she started in January, often giving odd and irrelevant answers. Her responses were posted in a forum that wasn't visible to students.

"Initially her answers weren't good enough because she would get stuck on keywords," said Lalith Polepeddi, one of the graduate students who co-developed the virtual TA. "For example, a student asked about organising a meet-up to go over video lessons with others, and Jill gave an answer referencing a textbook that could supplement the video lessons—same keywords—but different context. So we learned from mistakes like this one, and gradually made Jill smarter."

After some tinkering by the research team, Jill found her groove and soon was answering questions with 97% certainty. When she did, the human TAs would upload her responses to the students. By the end of March, Jill didn't need any assistance: She wrote the class directly if she was 97% positive her answer was correct.

Figure 2: AI technology for modelling and simulation of complex phenomena can lead to much deeper learning than simply reading about it or hearing a lecture on it.

The students, who were studying artificial intelligence, were unknowingly interacting with it. Goel didn't inform them about Jill's true identity until April 26. The student response was uniformly positive. One admitted her mind was blown. Another asked if Jill could "come out and play." Since then some students have organised a KBAI alumni forum to learn about new developments with Jill after the class ends, and another group of students has launched an open source project to replicate her.

Back in February, student Tyson Bailey began to wonder if Jill was a computer and posted his suspicions on Piazza.

"We were taking an AI course, so I had to imagine that it was possible there might be an AI lurking around," said Bailey, who lives in Albuquerque, New Mexico. "Then again, I asked Dr. Goel if he was a computer in one of my first email interactions with him. I think it's a great idea and hope that they continue to improve it."

Jill ended the semester able to answer many routine questions asked. She'll return—with a different name—next semester. The goal is to have the virtual teaching assistant answer 40% of all questions by the end of year.

   1 • 2