Have you ever found that you gained a deeper understanding of a subject or came up with new ideas after listening to another person’s opinion and gaining their perspective?

“In the field of cognitive science, it is known that interacting with others who have different viewpoints or knowledge is effective in promoting metacognition and critical thinking. It is through these repeated interactions that we realize how one piece of knowledge relates to another, and we gain the ability to grasp knowledge more objectively,” Yugo Hayashi explains.

How do people acquire new perspectives and understanding through their communication and interaction with others? In addition to exploring this question, Hayashi has been looking into ways of using computers to support inter-person communication and person-to-system or robot communications and collaborative learning. According to Hayashi, his field of study has become quite popular in recent years and is called human-computer interaction (HCI). “However, it has gradually become clear to me that an approach based on information engineering alone is insufficient to capture social interactions,” Hayashi says. He is currently attempting to discover the mechanism of knowledge acquisition by integrating the fields of HCI and cognitive science and plans to apply his findings to create new collaborative learning systems and robots.

His work revolves around a collaborative learning system that uses an embodied conversational agent, that is, a computer-based pedagogical instructor, to promote effective interaction when multiple people learn together. “Not everybody can instantly put into practice communication methods and techniques that lead to effective learning. That’s why the facilitation of classes by instructors or teachers’ aids plays an important role in the field of education. Computer-based instructors that play the role of educators are called Pedagogical Conversational Agents (PCAs),” explains Hayashi. There are many studies on the use of PCAs in intellectual learning support systems. However, Hayashi notes that when it comes to the specifics of how best to facilitate and stimulate learners’ interactions and promote collaborative learning, an optimal method is yet to be established.

Hayashi first conducted dialogue experiments between people using the utterance analysis method (protocol analysis) of cognitive science to clarify the types of words that promote interaction. He analyzed the data from these experiments and extracted utterances related to acquiring another person’s perspective and those concerning metacognitions. Following this, he created an utterance model and applied it to PCAs. He then conducted controlled laboratory experiments to verify the effectiveness of using PCAs in collaborative learning activities.

One of the PCAs he worked on assists collaborative learning by having a pair of learners explain to each other concepts they have just learned, to deepen their understanding. This PCA uses voice recognition and natural language processing technology to detect and analyze utterances made by the pair. When the conversation comes to a standstill or when conversation unrelated to learning occurs for a while, the PCA presents some keywords expected to promote effective discussion.

Hayashi also examined factors other than words, that could increase learning in any situation. This effort was for developing a multimodal interface that can sense various pieces of information being conveyed by the human body and use multiple communication channels during collaborative learning. For example, one such factor is emotional expressions. Hayashi proved that when words of encouragement and other positive words are used in addition to an image that shows positive emotion, such as a happy face, learners’ motivations to learn and their level of understanding increase.

Hayashi further focused on the line of sight or gaze. He developed an interface for two people who are engaged in remote collaborative learning via computer monitors. Each person’s interface has an eye-tracker that detects eye movements in real-time and displays them on the other person’s monitor. “It is said that joint attention is vital to establishing effective communication between remote learners. We developed a method that helps learners see their partner’s gaze in their own display, which allows them to see where the other partner is looking. We verified that, compared with when this interface is not used, this method helps guide learners’ gazes towards a shared object on their screens and facilitates joint attention, which promotes more lively conversation.”