Augmented reality technology in hepatopancreatobiliary surgery

Aim. To summarize the experience of applying augmented reality technology in hepatopancreatobiliary surgery.

Materials and methods. From November 2021 to January 2024, 43 patients underwent surgery using augmented reality technology. These procedures included pancreatoduodenectomy, distal pancreatectomy, irreversible electroporation of neoplasms, pancreatic pseudocyst removal, resection of the common bile duct with biliodigestive anastomosis formation, atypical liver resections, right hemihepatectomy, and transarterial chemoembolization of hepatic tumor arteries.

Results. The use of augmented reality technology showed no significant increase in operative time. In minimally invasive liver procedures, augmented reality proved to be a convenient navigation tool, contributing to reduced fluoroscopy duration and overall operative time. Analysis of Likert scales completed as feedback forms for intraoperative use by the surgical team demonstrated that augmented reality technology was convenient and beneficial for enhancing visualization and navigation.

Conclusion. Augmented reality has proven to be an effective, reliable, and promising tool for hepatopancreatobiliary surgery. However, further technological advancements are required to fully realize its potential. Enhancing the performance of augmented reality systems – including their accuracy, stability, and adaptability to various clinical scenarios – will make them more dependable and versatile for a wide range of surgical procedures.

1. Cleary K., Peters T.M. Image-guided interventions: technology review and clinical applications. Annu Rev. Biomed. Eng. 2010; 12: 119–142. https://doi.org/10.1146/annurev-bioeng-070909-105249

2. Mezger U., Jendrewski C., Bartels M. Navigation in surgery. Langenbecks Arch. Surg. 2013; 398 (4): 501–514. https://doi.org/10.1007/s00423-013-1059-4

3. Azuma R.T. A Survey of augmented reality. Presence: Teleoperators and Virtual Environments. 1997; 6 (4): 355–385. https://doi.org/10.1162/pres.1997.6.4.355

4. Sielhorst T., Feuerstein M., Navab N. Advanced medical displays: a literature review of augmented reality. J. Display Technol. 2008; 4 (4): 451–467. https://doi.org/10.1109/jdt.2008.2001575

5. Kalkofen D., Mendez E., Schmalstieg D. Comprehensible visualization for augmented reality. IEEE Trans. Vis. Comp. Graph. 2009; 15 (2): 193–204. https://doi.org/10.1109/tvcg.2008.96

6. Marescaux J., Clément J.M., Tassetti V., Koehl C., Cotin S., Russier Y., Mutter D., Delingette H., Ayache N. Virtual reality applied to hepatic surgery simulation: the next revolution. Ann. Surg. 1998; 228 (5): 627–634. https://doi.org/10.1097/00000658-199811000-00001

7. Wang J., Suenaga H., Liao H., Hoshi K., Yang L., Kobayashi E., Sakuma I. Real-time computer-generated integral imaging and 3D image calibration for augmented reality surgical navigation. Comp. Med. Imag. Graph. 2015; 40 (40): 147–159. https://doi.org/10.1016/j.compmedimag.2014.11.003

8. Tagaytayan R., Kelemen A., Sik-Lanyi C. Augmented reality in neurosurgery. Arch. Med. Sci. 2018; 14 (3): 572–578. https://doi.org/10.5114/aoms.2016.58690

9. Shirai R., Xiaoshuai C., Sase K., Komizunai Sh., Tsujita T., Konno A. AR brain-shift display for computer-assisted neurosurgery. 58th Annual Conference of the Society of Instrument and Control Engineers of Japan (SICE). 2019: 1113–1118. https://doi.org/10.23919/sice.2019.8859884

10. Zhang X., Fan Z., Wang J., Liao H. 3D Augmented reality based orthopaedic interventions. Comp. Radiol. Orthopaedic Intervent. 2016: 71–90. https://doi.org/10.1007/978-3-319-23482-3_4

11. Bernhardt S., Nicolau S.A., Soler L., Doignon C. The status of augmented reality in laparoscopic surgery as of 2016. Med. Image Anal. 2017; 37: 66–90. https://doi.org/10.1016/j.media.2017.01.007

12. Tang R., Ma L.F., Rong Z.X., Li M.D., Zeng J.P., Wang X.D., Liao H.E., Dong J.H. Augmented reality technology for preoperative planning and intraoperative navigation during hepatobiliary surgery: a review of current methods. Hepatobiliary Pancreat. Dis. Int. 2018; 17 (2): 101–112. https://doi.org/10.1016/j.hbpd.2018.02.002

13. Zhang F., Zhang S., Zhong K., Yu L., Sun L.N. Design of navigation system for liver surgery guided by augmented reality. IEEE Access. 2020; 8: 126687–126699. https://doi.org/10.1109/access.2020.3008690

14. Gavriilidis P., Edwin B., Pelanis E., Hidalgo E., de'Angelis N., Memeo R., Aldrighetti L., Sutcliffe R.P. Navigated liver surgery: state of the art and future perspectives. Hepatobiliary Pancreat. Dis. Int. 2021; 21 (3): 226–233. https://doi.org/10.1016/j.hbpd.2021.09.002

15. Dai J., Qi W., Qiu Z., Li C. The application and prospection of augmented reality in hepato-pancreato-biliary surgery. BioSci. Trends. 2023; 17 (3): 193–202. https://doi.org/10.5582/bst.2023.01086

16. Ma C., Chen G., Zhang X., Ning G., Liao H. Moving-tolerant augmented reality surgical navigation system using autostereoscopic three-dimensional image overlay. IEEE J. Biomed. Health Inform. 2019; 23 (6): 2483–2493. https://doi.org/10.1109/jbhi.2018.2885378

17. Marescaux J., Clément J.M., Tassetti V., Koehl C., Cotin S., Russier Y., Mutter D., Delingette H., Ayache N. Virtual reality applied to hepatic surgery simulation: the next revolution. Ann. Surg. 1998; 228 (5): 627–634. https://doi.org/10.1097/00000658-199811000-00001

18. Krummel T.M. Surgical simulation and virtual reality: the coming revolution. Ann. Surg. 1998; 228 (5): 635–637. https://doi.org/10.1097/00000658-199811000-00002

19. Liao H., Hata N., Nakajima S., Iwahara M., Sakuma I., Dohi T. Surgical navigation by autostereoscopic image overlay of integral videography. IEEE Trans. Inf. Technol. Biomed. 2004; 8 (2): 114–121. https://doi.org/10.1109/titb.2004.826734

20. Stüdeli T., Kalkofen D., Risholm P., Ali W., Freudenthal A., Samset E. Visualization tool for improved accuracy in needle placement during percutaneous radio-frequency ablation of liver tumors. Medical Imaging 2008: Visualization, ImageGuided Procedures, and Modeling. 2008; 6918: 116–127. https://doi.org/10.1117/12.769399

21. Sugimoto M., Yasuda H., Koda K., Suzuki M., Yamazaki M., Tezuka T., Kosugi C., Higuchi R., Watayo Y., Yagawa Y., Uemura S., Tsuchiya H., Azuma T. Image overlay navigation by markerless surface registration in gastrointestinal, hepatobiliary and pancreatic surgery. J. Hepatobiliary Pancreat. Sci. 2010; 17 (5): 629–636. https://doi.org/10.1007/s00534-009-0199-y

22. Panchenkov D.N., Abdulkerimov Z.A., Semeniakin I.V., Gabdullin A.F., Grigorieva E.V., Klimov D.D., Prokhorenko L.S., Gritsaenko A.I., Liskevich R.V., Tupikin K.A. First experience of using augmented reality technology in liver and pancreas laparoscopy. Annaly khirurgicheskoy gepatologii = Annals of HPB Surgery. 2023; 28 (1): 62–70. https://doi.org/10.16931/1995-5464.2023-1-62-70 (In Russian)

23. Breder V.V., Bazin I.S., Balakhnin P.V., Virshke E.R., Kosyrev V. Yu., Ledin E.V., Medvedeva B.M., Moiseenko F.V., Moroz E.A., Petkau V.V., Pokataev I.A. Malignant tumors of the liver and biliary tract. Practical recommendations by Russian Society of Clinical Oncology (RUSSCO), Part 1. Malignant tumours. 2023; 13 (3s2–1): 494–538. https://doi.org/10.18027/2224-5057-2023-13-3s2-1-494-538 (In Russian)

24. Ducreux M., Abou-Alfa G.K., Bekaii-Saab T., Berlin J., Cervantes A., de Baere T., Eng C., Galle P., Gill S., Gruenberger T., Haustermans K., Lamarca A., Laurent-Puig P., Llovet J.M., Lordick F., Macarulla T., Mukherji D., Muro K., Obermannova R., O'Connor J.M., O'Reilly E.M., Osterlund P., Philip P., Prager G., Ruiz-Garcia E., Sangro B., Seufferlein T., Tabernero J., Verslype C., Wasan H., Van Cutsem E. The management of hepatocellular carcinoma. Current expert opinion and recommendations derived from the 24th ESMO/ World Congress on Gastrointestinal Cancer, Barcelona, 2022. ESMO Open. 2023; 8 (3): 101567–101567. https://doi.org/10.1016/j.esmoop.2023.101567

25. Yoshino T., Cervantes A., Bando H., Martinelli E., Oki E., Xu R.H., Mulansari N.A., Govind Babu K., Lee M.A., Tan C.K., Cornelio G., Chong D.Q., Chen L.T., Tanasanvimon S., Prasongsook N., Yeh K.H., Chua C., Sacdalan M.D., Sow Jenson W.J., Kim S.T., Chacko R.T., Syaiful R.A., Zhang S.Z., Curigliano G., Mishima S., Nakamura Y., Ebi H., Sunakawa Y., Takahashi M., Baba E., Peters S., Ishioka C., Pentheroudakis G. Pan-Asian adapted ESMO Clinical Practice Guidelines for the diagnosis, treatment and follow-up of patients with metastatic colorectal cancer. ESMO Open. 2023; 8 (3): 101558–101558. https://doi.org/10.1016/j.esmoop.2023.101558

26. Chen F., Cui X., Liu J., Han B., Zhang X., Zhang D., Liao H. Tissue structure updating for in situ augmented reality navigation using calibrated ultrasound and two-level surface warping. IEEE Trans. Biomed. Eng. 2020; 67 (11): 3211–3222. https://doi.org/10.1109/tbme.2020.2979535

27. Dixon B.J., Daly M.J., Chan H., Vescan A.D., Witterick I.J., Irish J.C. Surgeons blinded by enhanced navigation: the effect of augmented reality on attention. Surg. Endosc. 2012; 27 (2): 454–461. https://doi.org/10.1007/s00464-012-2457-3.