关键词: 眼科视频画像, 多任务学习, 多模态融合, 知识图谱, 深度学习

Abstract: To address challenges in profiling ophthalmic videos on social networks,such as the low discriminability of visual features,the colloquial nature of text descriptions,and multimodal semantic heterogeneity,this paper proposes an OVP(Ophthalmic Video Profiling) method based on multi-task learning.The proposed method aims to mine multi-dimensional features with medical semantic value from unstructured video and text streams to facilitate precise video representation.In the OVP framework,a pre-trained deep residual network is employed to extract high-dimensional visual representations from keyframes,capturing fine-grained features specific to ophthalmic imagery.To overcome the sparsity of professional semantics in social media text,a method for extracting textual features from ophthalmic videos based on an ophthalmic knowledge graph is proposed,which retrieves and fuses external entity annotations and related knowledge before encoding via BERT.Subsequently,a cross-modal attention fusion mechanism is designed to dynamically calculate interaction weights between visual and textual features,achieving deep alignment between visual information and medical semantics.Furthermore,a multi-task joint optimization and ophthalmic multidimensional profiling is constructed to jointly train three sub-tasks:disease classification,popularity prediction,and content quality assessment,utilizing shared information to enhance model generalization.Experiments conducted on a real ophthalmic video dataset demonstrate that the OVP method significantly outperforms existing baseline methods in terms of disease classification accuracy,heatmap prediction,and quality assessment performance for ophthalmic videos.The experimental results validate the effectiveness of the OVP method in feature fusion and multidimensional profiling of complex ophthalmic videos.

Key words: Ophthalmic video profiling, Multi-task learning, Multi-modal fusion, Knowledge graph, Deep learning