Application of artificial intelligence visual image segmentation in surgical training

doi:10.3877/cma.j.issn.1674-6899.2024.03.001

Abstract

Abstract:

Surgical training has progressed with the development of medical technology. The in-depth intervention of artificial intelligence will continue to influence the medical industry. "Data" is the basis for AI exploration, and the massive video and image data generated by laparoscopic surgery is highly suitable for the development of AI models. Artificial intelligence visual segmentation is the basis of machine learning for situational awareness, and has already achieved the segmentation and identification of automated intraoperative surgical instruments, the identification of surgical processes in some surgical procedures, and the identification of key surgical areas. With further improvement in accuracy, fully automated recognition and annotation of surgical videos will help junior surgeons learn surgery efficiently and autonomously. This may change the traditional teaching mode of surgery.

Key words: Artificial intelligence, Laparoscopic surgery, Teaching

Mengyang Li, Enli Zhang, Junjie Wu, Zhiming Zhao. Application of artificial intelligence visual image segmentation in surgical training[J]. Chinese Journal of Laparoscopic Surgery(Electronic Edition), 2024, 17(03): 129-134.

/ / Recommend

Add to citation manager EndNote|Ris|BibTeX

URL: https://zhqjwkzz.cma-cmc.com.cn/EN/10.3877/cma.j.issn.1674-6899.2024.03.001

https://zhqjwkzz.cma-cmc.com.cn/EN/Y2024/V17/I03/129

Figures/Tables 2

References 38

1	Maier-Hein L, Vedula SS, Speidel S, et al. Surgical data science for next-generation interventions[J]. Nat Biomed Eng, 2017, 1(9)：691-696.
2	胡明根，陈况，马奔，等. 微创外科时代肝胆外科医师教学培养模式的探讨[J/CD]. 中华腔镜外科杂志(电子版), 2020, 13(2)：65-68.
3	Bailey RW, Imbembo AL, Zucker KA. Establishment of a laparoscopic cholecystectomy training program[J]. Am Surg, 1991, 57(4)：231-236.
4	Madion MP, Kastenmeier A, Goldblatt MI, et al. Robotic surgery training curricula: prevalence, perceptions, and educational experiences in general surgery residency programs[J]. Surg Endosc, 2022, 36(9)：6638-6646.
5	Bjerrum F, Thomsen ASS, Nayahangan LJ, et al. Surgical simulation: current practices and future perspectives for technical skills training[J]. Med Teach, 2018, 40(7)：668-675.
6	Cook DA, Hatala R, Brydges R, et al. Technology-enhanced simulation for health professions education: a systematic review and meta-analysis[J]. JAMA, 2011, 306(9)：978-988.
7	Chen G, Jin S, Xia Q, et al. Insight into the history and trends of surgical simulation training in education: a bibliometric analysis[J]. Int J Surg, 2023, 109(8)：2204-2213.
8	Costello DM, Huntington I, Burke G, et al. A review of simulation training and new 3D computer-generated synthetic organs for robotic surgery education[J]. J Robot Surg, 2022, 16(4)：749-763.
9	Aydin A, Raison N, Khan MS, et al. Simulation-based training and assessment in urological surgery[J]. Nat Rev Urol, 2016, 13(9)：503-519.
10	Chen IA, Ghazi A, Sridhar A, et al. Evolving robotic surgery training and improving patient safety, with the integration of novel technologies[J]. World J Urol, 2021, 39(8)：2883-2893.
11	Zheng, S. Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. in computer vision and pattern recognition (2021).
12	Shvets A, Rakhlin A, Kalinin AA, et al. Automatic instrument segmentation in robot-assisted surgery using deep learning[C]. 2018.
13	Jin Y, Cheng K, Dou Q, et al. Incorporating temporal prior from motion flow for instrument segmentation in minimally invasive surgery video[C]//International Conference on Medical Image Computing and Computor-Assisted Intervention. Springer, Cham, 2019.
14	Ni ZL, Bian GB, Hou ZG, et al.Attention-guided lightweight network for real-time segmentation of robotic surgical instruments[C].IEEE, 2020.
15	Shen W, Wang Y, Liu M, et al. Branch aggregation attention network for robotic surgical instrument segmentation[J]. IEEE Trans Med Imaging, 2023, 42(11)：3408-3419.
16	Madad Zadeh S, Francois T, Calvet L, et al. SurgAI: deep learning for computerized laparoscopic image understanding in gynaecology[J]. Surg Endosc, 2020, 34(12)：5377-5383.
17	Kumazu Y, Kobayashi N, Kitamura N, et al. Automated segmentation by deep learning of loose connective tissue fibers to define safe dissection planes in robot-assisted gastrectomy[J]. Sci Rep, 2021, 11(1)：21198.
18	Koo B, Robu MR, Allam M, et al. Automatic, global registration in laparoscopic liver surgery[J]. Int J Comput Assist Radiol Surg, 2022, 17(1)：167-176.
19	Une N, Kobayashi S, Kitaguchi D, et al. Intraoperative artificial intelligence system identifying liver vessels in laparoscopic liver resection: a retrospective experimental study[J]. Surg Endosc, 2024, 38(2)：1088-1095.
20	Kitaguchi D, Takeshita N, Matsuzaki H, et al. Real-time vascular anatomical image navigation for laparoscopic surgery: experimental study[J]. Surg Endosc, 2022, 36(8)：6105-6112.
21	Chen J, Li M, Han H, et al. Surgnet: self-supervised pretraining with semantic consistency for vessel and instrument segmentation in surgical images[J]. IEEE Trans Med Imaging, 2024, 43(4)：1513-1525.
22	Bodenstedt S, Allan M, Agustinos A, et al.Comparative evaluation of instrument segmentation and tracking methods in minimally invasive surgery[C]. 2018.
23	Allan, M．，Shvets, A．，Kurmann, T．，Zhang, Z. & Azizian, M. 2017 Robotic Instrument Segmentation Challenge. (2019).
24	Allan M, Kondo S, Bodenstedt S, et al. 2018 Robotic Scene Segmentation Challenge[C]. 2021.
25	Roβ T, Reinke A, Full PM, et al. Comparative validation of multi-instance instrument segmentation in endoscopy: results of the ROBUST-MIS 2019 challenge[J]. Med Image Anal, 2021, 70：101920.
26	Pfeiffer M, Funke I, Robu MR, et al. Generating large labeled data sets for laparoscopic image processing tasks using unpaired image-to-image translation[C]. 2019.
27	Jha D, Ali S, Emanuelsen K, et al. Kvasir-Instrument: diagnostic and therapeutic tool segmentation dataset in gastrointestinal endoscopy[C]//International Conference on Multimedia Modeling. Springer, Cham, 2021.
28	Maier-Hein L, Mersmann S, Kondermann D, et al. Can masses of non-experts train highly accurate image classifiers? A crowdsourcing approach to instrument segmentation in laparoscopic images[J]. Med Image Comput Comput Assist Interv, 2014, 17(Pt 2)：438-445.
29	Bouget D, Benenson R, Omran M, et al. Detecting surgical tools by modelling local appearance and global shape[J]. IEEE Trans Med Imaging, 2015, 34(12)：2603-2617.
30	Hong WY, Kao CL, Kuo YH, et al. CholecSeg8k: A semantic segmentation dataset for laparoscopic cholecystectomy based on cholec80[C]. 2020.
31	Grammatikopoulou, M．，et al. CaDIS: Cataract dataset for surgical RGB-image segmentation. 71 (2021).
32	Carstens M, Rinner FM, Bodenstedt S, et al. The dresden surgical anatomy dataset for abdominal organ segmentation in surgical data science[J]. Sci Data, 2023, 10(1)：3.
33	Madani A, Namazi B, Altieri MS, et al. Artificial intelligence for intraoperative guidance: using semantic segmentation to identify surgical anatomy during laparoscopic cholecystectomy[J]. Ann Surg, 2022, 276(2)：363-369.
34	Mascagni P, Vardazaryan A, Alapatt D, et al. Artificial intelligence for surgical safety: automatic assessment of the critical view of safety in laparoscopic cholecystectomy using deep learning[J]. Ann Surg, 2022, 275(5)：955-961.
35	Liu Y, Zhao S, Zhang G, et al. Multilevel effective surgical workflow recognition in robotic left lateral sectionectomy with deep learning: experimental research[J]. Int J Surg, 2023, 109(10)：2941-2952.
36	Kiyasseh D, Ma R, Haque TF, et al. A vision transformer for decoding surgeon activity from surgical videos[J]. Nat Biomed Eng, 2023, 7(6)：780-796.
37	Le Roy B, Ozgur E, Koo B, et al. Augmented reality guidance in laparoscopic hepatectomy with deformable semi-automatic computed tomography alignment (with video)[J]. J Visc Surg, 2019, 156(3)：261-262.
38	Tukra S, Marcus HJ, Giannarou S. See-through vision with unsupervised scene occlusion reconstruction[J]. IEEE Trans Pattern Anal Mach Intell, 2022, 44(7)：3779-3790.

数据名称	手术名称	目标类别	图像分辨率	图像数目
EndoVis 2015^[22]	腹腔镜手术	器械	640×480	300
EndoVis 2017^[23]	腹腔	器械	1 920×1 080	3 000
EndoVis 2018^[24]	肾切除术	器械、器官、组织	1 280×1 024	2 831
EndoVis 2019^[25]	腹腔镜手术	器械	960×540	10 040
Lap. I2I Translation^[26]	腹腔镜手术	器械、器官	452×256	20 000
Kvasir Instrument^[27]	胃镜、结肠镜检查	器械	1 280×1 024	590
InstrumentCrowd^[28]	腹腔镜手术	器械	640×480	120
NeuroSurgicalTools^[29]	神经外科手术	器械	612×460	2 476
CholecSeg8k^[30]	腹腔镜手术	器械、器官	854×480	8 080
HeiSurF	腹腔镜手术	器械、器官、组织	1 920×1 080	829
CaDIS^[31]	白内障手术	器械、组织	960×540	4 670
SAR-RARP50	前列腺切除术	器械	1 920×1 080	16 250
Dresden Surgical Anatomy^[32]	腹腔镜手术	器械，器官、组织、血管	1 920×1 080	13 195

数据名称	手术名称	目标类别	图像分辨率	图像数目
EndoVis 2015^[22]	腹腔镜手术	器械	640×480	300
EndoVis 2017^[23]	腹腔	器械	1 920×1 080	3 000
EndoVis 2018^[24]	肾切除术	器械、器官、组织	1 280×1 024	2 831
EndoVis 2019^[25]	腹腔镜手术	器械	960×540	10 040
Lap. I2I Translation^[26]	腹腔镜手术	器械、器官	452×256	20 000
Kvasir Instrument^[27]	胃镜、结肠镜检查	器械	1 280×1 024	590
InstrumentCrowd^[28]	腹腔镜手术	器械	640×480	120
NeuroSurgicalTools^[29]	神经外科手术	器械	612×460	2 476
CholecSeg8k^[30]	腹腔镜手术	器械、器官	854×480	8 080
HeiSurF	腹腔镜手术	器械、器官、组织	1 920×1 080	829
CaDIS^[31]	白内障手术	器械、组织	960×540	4 670
SAR-RARP50	前列腺切除术	器械	1 920×1 080	16 250
Dresden Surgical Anatomy^[32]	腹腔镜手术	器械，器官、组织、血管	1 920×1 080	13 195

[1]	Yang Li, Jinyu Cai, Xiaozhi Dang, Wanying Chang, Yan Ju, Yi Gao, Hongping Song. Application value of a deep learning-based model for generating strain elastography images using breast grayscale ultrasound images [J]. Chinese Journal of Medical Ultrasound (Electronic Edition), 2024, 21(06): 563-570.
[2]	Zihan Niu, Wenbo Li, Mengsu Xiao, Qing Zhang, Yixiu Zhang, Qingli Zhu. Integrating "curriculum ideology and politics" into ultrasonic medicine teaching: practice and effect evaluation [J]. Chinese Journal of Medical Ultrasound (Electronic Edition), 2024, 21(06): 597-601.
[3]	Yuxin Cao, Zhuojun Mao, Jiahe Liang, Jiangpu Yi, Zekai Zhang, Wenshuai Ma, Yuntao Chen, Xiaoqian Li, Yuxin Zhang, Tiesheng Cao, Lijun Yuan. Application of three-dimensional printed heart model in simulating clinical teaching of left atrial appendage closure [J]. Chinese Journal of Medical Ultrasound (Electronic Edition), 2024, 21(06): 602-607.
[4]	Jingwu Yang, Meijun Zhou, Yufan Chen, Sushu Li, Yanni He, Nan Cui, Hongmei Liu. Artificial intelligence ultrasound combined with quality control circle activity comprehensively improves ability of junior sonographers to assess risk of thyroid nodules [J]. Chinese Journal of Medical Ultrasound (Electronic Edition), 2024, 21(05): 522-526.
[5]	Gang Luo, Silin Pan, Lingyu Sun, Zhixin Li, Taotao Chen, Sibo Qiao, Shanchen Pang. Classification of right ventricular hypoplasia in fetuses diagnosed with pulmonary atresia with an intact ventricular septum or critical pulmonary stenosis via a new semantic parsing network model [J]. Chinese Journal of Medical Ultrasound (Electronic Edition), 2024, 21(04): 377-383.
[6]	Mengxiao Feng, Yuanqiang Lu. Analysis and reflection on current situation of emergency medicine teaching for international students coming to China—taking Zhejiang University as an example [J]. Chinese Journal of Critical Care Medicine(Electronic Edition), 2024, 17(03): 177-179.
[7]	Boxing Su, Bo Xiao, Jianxing Li. Hot research and interpretation of surgical therapy for stone disease in the annual meeting of American Urology Association in 2024 [J]. Chinese Journal of Endourology(Electronic Edition), 2024, 18(04): 303-308.
[8]	Linjian Mo, Shubo Yang, Weiyun Nong, Jiwen Cheng. Application of artificial intelligence virtual digital doctors in patient education management for day surgery holmium laser enucleation of the prostate [J]. Chinese Journal of Endourology(Electronic Edition), 2024, 18(04): 318-322.
[9]	Xingxing Ruan, Zhiyuan Huang, Fuxiang Liu, Jinming Di. Clinical research paper writing based on the perspective of doctors in diagnosing and treating patients [J]. Chinese Journal of Endourology(Electronic Edition), 2024, 18(04): 397-401.
[10]	Hai Huang, Bisheng Cheng, Jian Huang. 2024 Annual Meeting of European Association of Urology: frontier exploration and future trends in prostate cancer research [J]. Chinese Journal of Endourology(Electronic Edition), 2024, 18(03): 202-207.
[11]	Zixu Zhang, Junjiong Zheng, Yun Luo, Tianxin Lin. Development of isolated porcine kidney training model for laparoscopic partial nephrectomy [J]. Chinese Journal of Endourology(Electronic Edition), 2024, 18(03): 277-283.
[12]	Ziyang Peng, Zhibo Wang, Dan Wang, Haoqian Peng, Lei Wang, Wei Peng, Juanjuan Wang, Yu Li, Xuemin Liu, Rongqian Wu, Junxi Xiang, Yi Lyu. Application of intelligent auxiliary real-time image defogging technology in laparoscopic cholecystectomy [J]. Chinese Journal of Hepatic Surgery(Electronic Edition), 2024, 13(03): 328-333.
[13]	Meng Cao, Jiedong Guo, Can Zhu, Teng Xu, Ruizhi Fan, Tao Jiang, Jun Song, Yixin Xu. Efficacy and safety evaluation of Overlap anastomosis in complete laparoscopic right hemicolectomy [J]. Chinese Journal of Digestion and Medical Imageology(Electronic Edition), 2024, 14(04): 315-319.
[14]	Ming Lu, Biying Zhang, Zhonghua Deng, Xiaoguang Li, Ning Shen. Application of scene teaching for standardized training of residents in fever clinics [J]. Chinese Journal of Clinicians(Electronic Edition), 2024, 18(04): 388-391.
[15]	Xiaofei Hou, Wei Wang, Shibao Lu. Application of case presentation-based preoperative discussion model in teaching degenerative spinal diseases [J]. Chinese Journal of Clinicians(Electronic Edition), 2024, 18(03): 295-298.

Please choose a citation manager

Content to export