Industrial - Bachelors
Haptikos is an interface that connects several technologies to help surgeons and trainees develop skills in otherwise rare or difficult surgeries, with surgeons developing models using digital twin data that trainees can practice on providing a range of data and information to help develop skills and assess trainees.
This project was initiated by the Herston Biofabrication Institute (HBI) to support their digital twin surgical training (DTST) initiative to be developed with medical health professionals including doctors, academics, specialists and surgeons. The goal of this project was to help educate and train surgical trainees on surgical procedures of 2 or more hours using digital twin and XR technologies.
Secondary research was done initially to inform later primary research focussing on the impacts of XR technologies on surgery and medical education. An expert interview was conducted with a surgeon with decades of experience over a recorded Zoom call for half an hour, a survey was also deployed online and distributed to clinics through email, personal medical contacts to pass on to their colleagues, and various moderated surgical groups online, however, there was only 1 survey respondent over 3 weeks so it was instead analysed as a separate interview with a medical student.
Surgeons at Metro North Hospital who want to prepare a trainee for an operation or recall a previous operational experience that would benefit others would use digital twin data previously collected to develop anatomical models for trainees to practice. This data would be moved through Digital Anatomy™ Creator a 3D slicing software designed for the Stratasys Digital Anatomy™ J750 printer which uses composite printed materials to mimic the mechanical properties of tissues. Fiber optic shape sensors are imbedded between layers that would receive deformation or strain below the area of surgical interest and using the FBG-Scan 915-EP™ measuring device and the ILLumiSense™ visualisation software forces applied to the model through surgical instruments can be mapped. The surgeon would first work with HBI to calibrate the model for future use by operating on the model this would set a benchmark score/base procedure that would need to be imitated by trainees.
The training apparatus integrates, the Intel® RealSense™ D435 depth camera and projector positioned above the surgical table. Below, the FBG-Scan 915-EP™, coupled with a computer, collects data. The Haptikos serves as the central hub, ensuring a deep learning experience for trainees and a refined practice environment for surgeons.
The training mat is divided into four distinct zones using fiducial markers. The depth camera cross-verifies the user’s tool list, identifying any missing items as ‘active tools.’ The ‘surgical table’ hosts the anatomical model, securely mounted through the board’s tapped base, with cables connected to the computer and the FGB measuring device. The projection screen displays performance metrics, procedural guidance, and useful tips based on the anatomical model and user input. In the ‘input area,’ trainees can select models and navigate the computer peripherals.
Notably, the projector can also project directly onto the anatomical model, offering real-time guidance for procedures, including depth in bone drilling, tool speed, torque, and precise cutting guidelines. This comprehensive approach to surgical training exemplifies our commitment to innovation and excellence in medical education.
James is an Industrial Designer, studying Design and Engineering, and enjoys model making, rapid prototyping, CAD, and mechanical design.