2024/04/04 by Nishimoto, Tomohiro, Taichi Nishimura, Koki Yamamoto +21
Biochemistry, Genetics and Molecular Biology · #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Genomics and Phylogenetic Studies #Machine Learning in Bioinformatics #Multimedia (cs.MM)
paper · pdf · doi:10.48550/arxiv.2404.03161
openalex publication_date 2024/04/04 · openalex created_date 2024/04/06 · openalex updated_date 2026/07/28
This paper introduces BioVL-QR, a biochemical vision-and-language dataset comprising 23 egocentric experiment videos, corresponding protocols, and vision-and-language alignments. A major challenge in understanding biochemical videos is detecting equipment, reagents, and containers because of the cluttered environment and indistinguishable objects. Previous studies assumed manual object annotation, which is costly and time-consuming. To address the issue, we focus on Micro QR Codes. However, detecting objects using only Micro QR Codes is still difficult due to blur and occlusion caused by object manipulation. To overcome this, we propose an object labeling method combining a Micro QR Code detector with an off-the-shelf hand object detector. As an application of the method and BioVL-QR, we tackled the task of localizing the procedural steps in an instructional video. The experimental results show that using Micro QR Codes and our method improves biochemical video understanding. Data and code are available through https://nishi10mo.github.io/BioVL-QR/