2022/06/16 by Ghadah Alabduljabbar, Alabduljabbar, Ghadah, Hafida Benhidour +3 · 2 citations
Computer Science · #Advanced Image and Video Retrieval Techniques #Algorithm #Artificial intelligence #Closed captioning #Computer Vision and Pattern Recognition (cs.CV) #Computer science #Computer vision #Decoding methods #ENCODE #Encoder #Encoding (memory) #FOS: Computer and information sciences #Feature (linguistics) #Image (mathematics) #Image Retrieval and Classification Techniques #Linguistics #Multimodal Machine Learning Applications #Natural language #Natural language processing #Pattern recognition (psychology) #Process (computing) #Salient #Speech recognition #cs.CV
paper · pdf · doi:10.48550/arxiv.2206.07986
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2022/06/16 · arxiv created 2022/07/25 · arxiv updated 2022/07/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/08
Image captioning is the process of automatically generating a description of an image in natural language. Image captioning is one of the significant challenges in image understanding since it requires not only recognizing salient objects in the image but also their attributes and the way they interact. The system must then generate a syntactically and semantically correct caption that describes the image content in natural language. With the significant progress in deep learning models and their ability to effectively encode large sets of images and generate correct sentences, several neural-based captioning approaches have been proposed recently, each trying to achieve better accuracy and caption quality. This paper introduces an encoder-decoder-based image captioning system in which the encoder extracts spatial features from the image using ResNet-101. This stage is followed by a refining model, which uses an attention-on-attention mechanism to extract the visual features of the target image objects, then determine their interactions. The decoder consists of an attention-based recurrent module and a reflective attention module, which collaboratively apply attention to the visual and textual features to enhance the decoder's ability to model long-term sequential dependencies. Extensive experiments performed on Flickr30K, show the effectiveness of the proposed approach and the high quality of the generated captions.