2023/05/24 by Yuxuan Sun, Chenglu Zhu, Sun, Yuxuan +14 · 8 citations
Biochemistry, Genetics and Molecular Biology · Computer Science · Decision Sciences · #Biomedical Text Mining and Ontologies #Computer Vision and Pattern Recognition (cs.CV) #Data Mining Algorithms and Applications #FOS: Computer and information sciences #Multimedia (cs.MM) #Scientific Computing and Data Management
paper · pdf · doi:10.48550/arxiv.2305.15072
openalex publication_date 2023/05/24 · openalex created_date 2023/05/27 · openalex updated_date 2026/07/28
As advances in large language models (LLMs) and multimodal techniques continue to mature, the development of general-purpose multimodal large language models (MLLMs) has surged, offering significant applications in interpreting natural images. However, the field of pathology has largely remained untapped, particularly in gathering high-quality data and designing comprehensive model frameworks. To bridge the gap in pathology MLLMs, we present PathAsst, a multimodal generative foundation AI assistant to revolutionize diagnostic and predictive analytics in pathology. The development of PathAsst involves three pivotal steps: data acquisition, CLIP model adaptation, and the training of PathAsst's multimodal generative capabilities. Firstly, we collect over 207K high-quality pathology image-text pairs from authoritative sources. Leveraging the advanced power of ChatGPT, we generate over 180K instruction-following samples. Furthermore, we devise additional instruction-following data specifically tailored for invoking eight pathology-specific sub-models we prepared, allowing the PathAsst to effectively collaborate with these models, enhancing its diagnostic ability. Secondly, by leveraging the collected data, we construct PathCLIP, a pathology-dedicated CLIP, to enhance PathAsst's capabilities in interpreting pathology images. Finally, we integrate PathCLIP with the Vicuna-13b and utilize pathology-specific instruction-tuning data to enhance the multimodal generation capacity of PathAsst and bolster its synergistic interactions with sub-models. The experimental results of PathAsst show the potential of harnessing AI-powered generative foundation model to improve pathology diagnosis and treatment processes.