Development of a New Audio-Visuo-Haptic Training System: Learning requires collaboration between an expert and a novice. To collaborate efficiently, these partners must construct a Common Frame-of-Reference (CFR) system. The CFR is a representation of the shared situation, which is updated continuously by an information exchange between a teacher and a learner. The CFR allows the teacher and the learner to understand each other and to organize their action in a common framework. In everyday life, this paradigm is observed when an expert takes the hand of a novice to demonstrate a movement. A grounding process occurs when the team partners exchange information and share signs of understanding to develop their CFR. Depending on the task, the partners may use different communication channels (voice, vision, and haptics) to develop their CFR. The main goal of this project is to develop a new audio-visuo-haptic training system based on the What-You-Feel-Is-What-I-Feel (WYFIWIF) paradigm (see Figure 1). WYFIWIF will allow teachers to guide novice surgeons using voice, visuals, and haptics. Using this novel Minimally Invasive Surgery (MIS) training paradigm, the two hands of an expert surgeon will feel and correct haptically the motion and forces applied by a training surgeon hands during a MIS procedure. In WYFIWIF, the teacher perceives the student action by guiding him through verbal suggestions, visual, and haptics as he performs a procedure involving ambidextrous tool use in a real-situation. This comprehensive perception of task forces where the learner can construct movement coordination through practice, and hence facilitates skill acquisition. This new paradigm could replace traditional surgical simulators, which rely on inaccurate virtual simulations which are known to be un-realistic. In our system, the teacher/student surgeons will interact with the real surgical environment as if the surgical tools were not connected to the haptic devices. This will be done by carefully designing a surgical tools holder which will be mechanically linked to the haptic devices and will be compatible with operating room conditions. In this way, the haptic devices will be able to digitize the location/orientation of the tools and be able to apply guiding forces without interfering with their normal usages.

Figure 1: WYFIWIF Concept

Figure 2: Haptic Guidance System

WYFIWIF with Gaze Training: By tracking eye-hand coordination patterns, gaze analysis can separate experts from novices. Gaze can also serve as a teaching tool in surgical training. For years, the method “see one, do one, teach one” was generally used to learn surgical skills. To learn how to manipulate the endoscope, trainees are encouraged to watch procedures directly or indirectly (videos), and then perform these procedures under supervision. Recent advances in our understanding of skills acquisition by athletes suggests that learners build sophisticated motor skills by observing, analyzing, and copying the gaze strategy of experts. In order to improve our audio-visuo-haptic training system, we will investigate if gaze training is useful at improving teaching of sophisticated surgical skills like MIS. Specifically, we will use eye tracking in addition to the haptic interfaces as a feedback to teach junior surgeons how to manage their gaze strategy and eye-hand coordination while learning an MIS procedure using WYFIWIF. Junior surgeons’ eye gaze will be digitized and displayed compared to expert surgeons’ gaze in real-time (see Figure 1). Various strategies will be explored to display these differences.

Through these three sensorimotor pathways, it is expected that junior surgeons will construct eye-hand coordination naturally and with less cognitive demand compared to traditional motion-focused learning methods that rely on copying actions of others.