Information Technology
stimulates People-to-People

Yugo Hayashi

College of Comprehensive Psychology

Applying insights from cognitive science
to develop an interface
that promotes social interaction.

Have you ever found that you gained a deeper understanding of a subject or came up with new ideas after listening to another person’s opinion and gaining their perspective?

“In the field of cognitive science, it is known that interacting with others who have different viewpoints or knowledge is effective in promoting metacognition and critical thinking. It is through these repeated interactions that we realize how one piece of knowledge relates to another, and we gain the ability to grasp knowledge more objectively,” Yugo Hayashi explains.

How do people acquire new perspectives and understanding through their communication and interaction with others? In addition to exploring this question, Hayashi has been looking into ways of using computers to support inter-person communication and person-to-system or robot communications and collaborative learning. According to Hayashi, his field of study has become quite popular in recent years and is called human-computer interaction (HCI). “However, it has gradually become clear to me that an approach based on information engineering alone is insufficient to capture social interactions,” Hayashi says. He is currently attempting to discover the mechanism of knowledge acquisition by integrating the fields of HCI and cognitive science and plans to apply his findings to create new collaborative learning systems and robots.

His work revolves around a collaborative learning system that uses an embodied conversational agent, that is, a computer-based pedagogical instructor, to promote effective interaction when multiple people learn together. “Not everybody can instantly put into practice communication methods and techniques that lead to effective learning. That’s why the facilitation of classes by instructors or teachers’ aids plays an important role in the field of education. Computer-based instructors that play the role of educators are called Pedagogical Conversational Agents (PCAs),” explains Hayashi. There are many studies on the use of PCAs in intellectual learning support systems. However, Hayashi notes that when it comes to the specifics of how best to facilitate and stimulate learners’ interactions and promote collaborative learning, an optimal method is yet to be established.

Hayashi first conducted dialogue experiments between people using the utterance analysis method (protocol analysis) of cognitive science to clarify the types of words that promote interaction. He analyzed the data from these experiments and extracted utterances related to acquiring another person’s perspective and those concerning metacognitions. Following this, he created an utterance model and applied it to PCAs. He then conducted controlled laboratory experiments to verify the effectiveness of using PCAs in collaborative learning activities.

One of the PCAs he worked on assists collaborative learning by having a pair of learners explain to each other concepts they have just learned, to deepen their understanding. This PCA uses voice recognition and natural language processing technology to detect and analyze utterances made by the pair. When the conversation comes to a standstill or when conversation unrelated to learning occurs for a while, the PCA presents some keywords expected to promote effective discussion.

Hayashi also examined factors other than words, that could increase learning in any situation. This effort was for developing a multimodal interface that can sense various pieces of information being conveyed by the human body and use multiple communication channels during collaborative learning. For example, one such factor is emotional expressions. Hayashi proved that when words of encouragement and other positive words are used in addition to an image that shows positive emotion, such as a happy face, learners’ motivations to learn and their level of understanding increase.

Hayashi further focused on the line of sight or gaze. He developed an interface for two people who are engaged in remote collaborative learning via computer monitors. Each person’s interface has an eye-tracker that detects eye movements in real-time and displays them on the other person’s monitor. “It is said that joint attention is vital to establishing effective communication between remote learners. We developed a method that helps learners see their partner’s gaze in their own display, which allows them to see where the other partner is looking. We verified that, compared with when this interface is not used, this method helps guide learners’ gazes towards a shared object on their screens and facilitates joint attention, which promotes more lively conversation.”

Realtime display of the eye movements of each learner on the other’s monitor

“The literature suggests that controversial or conflicting discussions, such as critiques based on varying evidence, are effective in collaborative learning. Thus, in our latest study, we have been developing a collaborative learning system that induces arguments leading to constructive conflict,” Hayashi says. Currently, Hayashi and the students in his lab are working on a project to investigate ways of presenting prompts so as to induce conflict. Their aim is to create an instructor, a PCA, that can train learners who have not developed debating skills yet, so they may eventually engage in constructive arguments. He explains that a model of argumentation will be constructed by first developing a protocol to deepen one’s learning from ideal controversial debates among those who are experienced in debates and discussions. The plan is to integrate that protocol with a PCA and examine its effect.

For example, in future classrooms, we may find an autonomous PCA that Hayashi developed taking on the role of instructor. The applicability of Hayashi’s PCA could expand into group therapy or counseling. We eagerly wait for the day when the fruits of Hayashi’s research will be seen in society.

Yugo Hayashi
Yugo Hayashi
Professor, College of Comprehensive Psychology
Research Themes: Collaborative problem solving, communication with a conversational agent/robot, media equation, developing embodied conversational agents for pedagogy, study of analysis and application of textual and behavioral data on the SNS
Specialties: Cognitive science, human interface and interaction, Kansei informatics, web informatics, service informatics, learning support system, educational technology, social psychology

storageResearchers database

Photo Cooperation: DMM.make

May 10, 2021