Zhe Gan
- MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training
2024/03/14 by Brandon McKinzie, Zhe Gan, McKinzie, Brandon +61 · 9 voices · 13 citations
#cs.CV #cs.CL #cs.LG
- GIT: A Generative Image-to-text Transformer for Vision and Language
2022/05/27 by Jianfeng Wang, Zhengyuan Yang, Wang, Jianfeng +15 · 2 voices · 47 citations
Computer Science · #Handwritten Text Recognition Techniques #Multimodal Machine Learning Applications #Natural Language Processing Techniques #cs.CV
- Pico-Banana-400K: A Large-Scale Dataset for Text-Guided Image Editing
2025/10/22 by Yusu Qian, Eli Bocek-Rivele, Qian, Yusu +13 · 5 voices · 8 citations
#cs.CV #cs.CL #cs.LG
- AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks
2017/11/28 by Tao Xu, Xu, Tao, Pengchuan Zhang +11 · 1 voice · 24 citations
Computer Science · #Generative Adversarial Networks and Image Synthesis #Human Pose and Action Recognition #Multimodal Machine Learning Applications #cs.CV
- Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs
2024/04/08 by Keen You, You, Keen, Haotian Zhang +13 · 4 voices · 26 citations
Computer Science · #Speech and dialogue systems #cs.CL #cs.CV #cs.HC
- UNITER: UNiversal Image-TExt Representation Learning
2020/01/01 by Yen-Chun Chen, Linjie Li, Licheng Yu +5 · 87 citations
Computer Science · #Advanced Image and Video Retrieval Techniques #Domain Adaptation and Few-Shot Learning #Multimodal Machine Learning Applications
- Generalized Decoding for Pixel, Image, and Language
2022/12/21 by Xueyan Zou, Zi-Yi Dou, Zou, Xueyan +25 · 32 citations
Computer Science · #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling
- Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling
2021/02/11 by Jie Lei, Lei, Jie, Linjie Li +11 · 25 citations
Computer Science · #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Human Pose and Action Recognition #Multimodal Machine Learning Applications
- Variational Autoencoder for Deep Learning of Images, Labels and Captions
2016/09/28 by Yunchen Pu, Pu, Yunchen, Zhe Gan +11 · 20 citations
Computer Science · #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Multimodal Machine Learning Applications
- MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning
2024/09/30 by Haotian Zhang, Mingfei Gao, Zhang, Haotian +43 · 1 voice · 13 citations
Computer Science · #cs.CV #cs.CL #cs.LG
- Multimodal Foundation Models: From Specialists to General-Purpose Assistants
2023/09/18 by Chunyuan Li, Zhe Gan, Li, Chunyuan +11 · 28 citations
Computer Science · #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling
- StoryGAN: A Sequential Conditional GAN for Story Visualization
2018/12/06 by Yitong Li, Zhe Gan, Li, Yitong +15 · 13 citations
Computer Science · #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Human Pose and Action Recognition #Multimodal Machine Learning Applications #Video Analysis and Summarization
- ReCo: Region-Controlled Text-to-Image Generation
2022/11/23 by Zhengyuan Yang, Yang, Zhengyuan, Jianfeng Wang +19 · 19 citations
Computer Science · #Multimodal Machine Learning Applications #Image Retrieval and Classification Techniques #Generative Adversarial Networks and Image Synthesis
- HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training
2020/05/01 by Linjie Li, Yen-Chun Chen, Li, Linjie +9 · 14 citations
Computer Science · #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Human Pose and Action Recognition #Machine Learning (cs.LG) #Multimodal Machine Learning Applications
- An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA
2021/09/10 by Zhengyuan Yang, Zhe Gan, Yang, Zhengyuan +11 · 15 citations
Computer Science · #Multimodal Machine Learning Applications #Advanced Image and Video Retrieval Techniques #Domain Adaptation and Few-Shot Learning
- Guiding Instruction-based Image Editing via Multimodal Large Language Models
2023/09/29 by Tsu-Jui Fu, Wenze Hu, Fu, Tsu-Jui +9 · 21 citations
Computer Science · #Multimodal Machine Learning Applications #Domain Adaptation and Few-Shot Learning #Advanced Image and Video Retrieval Techniques
- Improve Vision Language Model Chain-of-thought Reasoning
2024/10/21 by Ruohong Zhang, Zhang, Ruohong, Bowen Zhang +15 · 30 citations
Computer Science · #Semantic Web and Ontologies
- VeCLIP: Improving CLIP Training via Visual-enriched Captions
2023/10/11 by Zhengfeng Lai, Lai, Zhengfeng, Haotian Zhang +20 · 18 citations
Computer Science · #Multimodal Machine Learning Applications #Advanced Image and Video Retrieval Techniques #Domain Adaptation and Few-Shot Learning
- Scaling Up Vision-Language Pre-training for Image Captioning
2021/11/24 by Xiaowei Hu, Zhe Gan, Hu, Xiaowei +11 · 12 citations
Computer Science · #Advanced Image and Video Retrieval Techniques #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Multimodal Machine Learning Applications
- Multimodal Autoregressive Pre-training of Large Vision Encoders
2024/11/21 by Enrico Fini, Mustafa Shukor, Fini, Enrico +29 · 27 citations
Computer Science · #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Neural Networks and Applications
- Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Models
2024/04/11 by Haotian Zhang, Haoxuan You, Zhang, Haotian +19 · 20 citations
Computer Science · Medicine · #Multimodal Machine Learning Applications #COVID-19 diagnosis using AI #Domain Adaptation and Few-Shot Learning
- An Empirical Study of Training End-to-End Vision-and-Language Transformers
2021/11/03 by Zi-Yi Dou, Dou, Zi-Yi, Yichong Xu +21 · 12 citations
Computer Science · #Multimodal Machine Learning Applications #Domain Adaptation and Few-Shot Learning #Advanced Neural Network Applications
- SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Models
2024/07/22 by Mingze Xu, Mingfei Gao, Xu, Mingze +13 · 20 citations
Computer Science · Medicine · #COVID-19 diagnosis using AI #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Human Pose and Action Recognition #Multimodal Machine Learning Applications
- Generating Informative and Diverse Conversational Responses via\n Adversarial Information Maximization
2018/09/16 by Yizhe Zhang, Zhang, Yizhe, Michel Galley +11 · 11 citations
Computer Science · Physics and Astronomy · #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Model Reduction and Neural Networks #Topic Modeling
- Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models
2021/06/01 by Linjie Li, Li, Linjie, Jie Lei +5 · 8 citations
Computer Science · Medicine · #COVID-19 diagnosis using AI #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Multimodal Machine Learning Applications
- Relation-Aware Graph Attention Network for Visual Question Answering
2019/03/29 by Linjie Li, Li, Linjie, Zhe Gan +5 · 7 citations
Computer Science · #Advanced Image and Video Retrieval Techniques #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Multimodal Machine Learning Applications
- MIA-Bench: Towards Better Instruction Following Evaluation of Multimodal LLMs
2024/07/01 by Yusu Qian, Qian, Yusu, Hanrong Ye +9 · 15 citations
Arts and Humanities · Computer Science · #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Natural Language Processing Techniques #Translation Studies and Practices #linguistics and terminology studies
- NUWA-Infinity: Autoregressive over Autoregressive Generation for Infinite Visual Synthesis
2022/07/20 by Chenfei Wu, Jian Liang, Wu, Chenfei +15 · 8 citations
Computer Science · #Advanced Image and Video Retrieval Techniques #Advanced Vision and Imaging #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis
- GRiT: A Generative Region-to-text Transformer for Object Understanding
2022/12/01 by Jialian Wu, Jianfeng Wang, Wu, Jialian +11 · 8 citations
Computer Science · #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling
- Contrastive Distillation on Intermediate Representations for Language Model Compression
2020/09/29 by Siqi Sun, Zhe Gan, Sun, Siqi +9 · 6 citations
Computer Science · #Topic Modeling #Natural Language Processing Techniques #Multimodal Machine Learning Applications
- InfoBERT: Improving Robustness of Language Models from An Information\n Theoretic Perspective
2020/10/05 by Boxin Wang, Wang, Boxin, Shuohang Wang +11 · 5 citations
Computer Science · #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Topic Modeling
- VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation
2021/06/08 by Linjie Li, Jie Lei, Li, Linjie +27 · 6 citations
Computer Science · #Multimodal Machine Learning Applications #Topic Modeling #Domain Adaptation and Few-Shot Learning
- Chasing Sparsity in Vision Transformers: An End-to-End Exploration
2021/06/08 by Tianlong Chen, Yu Cheng, Chen, Tianlong +9 · 5 citations
Computer Science · Engineering · #Advanced Memory and Neural Computing #Advanced Neural Network Applications #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Visual Attention and Saliency Detection
- Tactical Rewind: Self-Correction via Backtracking in Vision-and-Language Navigation
2019/03/06 by Liyiming Ke, Ke, Liyiming, Xiujun Li +15 · 4 citations
Computer Science · #Multimodal Machine Learning Applications #Advanced Image and Video Retrieval Techniques #Domain Adaptation and Few-Shot Learning
- How Easy is It to Fool Your Multimodal LLMs? An Empirical Analysis on Deceptive Prompts
2024/02/20 by Yusu Qian, Haotian Zhang, Qian, Yusu +5 · 8 citations
Social Sciences · #Artificial Intelligence in Law #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences
- An Empirical Study of Multimodal Model Merging
2023/04/28 by Yi-Lin Sung, Sung, Yi-Lin, Linjie Li +9 · 6 citations
Computer Science · #Multimodal Machine Learning Applications #Topic Modeling #Domain Adaptation and Few-Shot Learning
- Ferret-UI 2: Mastering Universal User Interface Understanding Across Platforms
2024/10/24 by Zhangheng Li, Keen You, Li, Zhangheng +17 · 11 citations
Computer Science · #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Usability and User Interface Design
- Contrastive Localized Language-Image Pre-Training
2024/10/03 by Hong-You Chen, Zhengfeng Lai, Chen, Hong-You +17 · 1 voice · 7 citations
Arts and Humanities · Computer Science · #Computer Vision and Pattern Recognition (cs.CV) #EFL/ESL Teaching and Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Second Language Learning and Teaching #cs.CV #cs.LG
- Sequential Attention GAN for Interactive Image Editing
2018/12/20 by Yu Cheng, Zhe Gan, Cheng, Yu +7 · 3 citations
Computer Science · #Multimodal Machine Learning Applications #Generative Adversarial Networks and Image Synthesis #Video Analysis and Summarization
- An Empirical Study of End-to-End Video-Language Transformers with Masked Visual Modeling
2022/09/04 by Tsu-Jui Fu, Linjie Li, Fu, Tsu-Jui +11 · 4 citations
Computer Science · #Multimodal Machine Learning Applications #Advanced Neural Network Applications #Domain Adaptation and Few-Shot Learning
- MM-Ego: Towards Building Egocentric Multimodal LLMs for Video QA
2024/10/09 by Haihong Ye, Ye, Hanrong, Haotian Zhang +19 · 7 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Semantic Web and Ontologies #Speech and dialogue systems
- What Makes A Good Story? Designing Composite Rewards for Visual Storytelling
2019/09/11 by Junjie Hu, Yu Cheng, Hu, Junjie +9 · 3 citations
Computer Science · #Computation and Language (cs.CL) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Human Pose and Action Recognition #Multimodal Machine Learning Applications
- EarlyBERT: Efficient BERT Training via Early-bird Lottery Tickets
2020/12/31 by Xiaohan Chen, Yu Cheng, Chen, Xiaohan +9 · 5 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling
- VIOLIN: A Large-Scale Dataset for Video-and-Language Inference
2020/03/25 by Jingzhou Liu, Wenhu Chen, Liu, Jingzhou +11 · 3 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling
- Deep Temporal Sigmoid Belief Networks for Sequence Modeling
2015/09/23 by Zhe Gan, Gan, Zhe, Chunyuan Li +7 · 2 citations
Computer Science · #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Music and Audio Processing #Time Series Analysis and Forecasting
- Understanding Alignment in Multimodal LLMs: A Comprehensive Study
2024/07/02 by Elmira Amirloo, Jean-Philippe Fauconnier, Amirloo, Elmira +19 · 4 citations
Computer Science · Arts and Humanities · #Natural Language Processing Techniques #linguistics and terminology studies #Translation Studies and Practices
- Improving Zero-shot Voice Style Transfer via Disentangled Representation\n Learning
2021/03/16 by Siyang Yuan, Pengyu Cheng, Yuan, Siyang +9 · 2 citations
Computer Science · #Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Music and Audio Processing #Sound (cs.SD) #Speech Recognition and Synthesis #Speech and Audio Processing #electronic engineering #information engineering
- Contrastively Smoothed Class Alignment for Unsupervised Domain Adaptation
2019/09/11 by Shuyang Dai, Yu Cheng, Dai, Shuyang +9 · 2 citations
Biochemistry, Genetics and Molecular Biology · Computer Science · #Cancer-related molecular mechanisms research #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Multimodal Machine Learning Applications
- Adversarial Symmetric Variational Autoencoder
2017/11/14 by Yunchen Pu, Pu, Yunchen, Weiyao Wang +11 · 2 citations
Computer Science · #Advanced Image Processing Techniques #Adversarial Robustness in Machine Learning #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG)
- Deconvolutional Paragraph Representation Learning
2017/08/16 by Yizhe Zhang, Dinghan Shen, Zhang, Yizhe +9 · 2 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling
- UltraCUA: A Foundation Model for Computer Use Agents with Hybrid Action
2025/10/20 by Yuhao Yang, Yang, Yuhao, Zhen Yang +20 · 5 citations
Computer Science · #Advanced Software Engineering Methodologies #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Reinforcement Learning in Robotics
- The Elastic Lottery Ticket Hypothesis
2021/03/30 by Xiaohan Chen, Yu Cheng, Chen, Xiaohan +9 · 1 citation
Computer Science · #Advanced Neural Network Applications #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Multimodal Machine Learning Applications
- Revisit Large-Scale Image-Caption Data in Pre-training Multimodal Foundation Models
2024/10/03 by Zhengfeng Lai, Vasileios Saveris, Lai, Zhengfeng +21 · 2 citations
Computer Science · #Advanced Image and Video Retrieval Techniques #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Human Pose and Action Recognition #Machine Learning (cs.LG) #Multimodal Machine Learning Applications
- Exploring Discrete Diffusion Models for Image Captioning
2022/11/21 by Zixin Zhu, Yixuan Wei, Zhu, Zixin +17 · 1 citation
Computer Science · #Advanced Image and Video Retrieval Techniques #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Video Analysis and Summarization
- Learning Generic Sentence Representations Using Convolutional Neural Networks
2016/11/23 by Zhe Gan, Gan, Zhe, Yunchen Pu +9 · 3 citations
Computer Science · #Topic Modeling #Natural Language Processing Techniques #Multimodal Machine Learning Applications
- Improving Sequence-to-Sequence Learning via Optimal Transport
2019/01/18 by Li‐Qun Chen, Yizhe Zhang, Chen, Liqun +17 · 1 citation
Computer Science · #Topic Modeling #Multimodal Machine Learning Applications #Natural Language Processing Techniques
- JointGAN: Multi-Domain Joint Distribution Learning with Generative Adversarial Nets
2018/06/08 by Yunchen Pu, Shuyang Dai, Pu, Yunchen +13 · 1 citation
Computer Science · #Anomaly Detection Techniques and Applications #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Machine Learning (stat.ML)
- DeepMMSearch-R1: Empowering Multimodal LLMs in Multimodal Web Search
2025/10/14 by Kartik Narayan, Yang Xu, Narayan, Kartik +16 · 4 citations
Biochemistry, Genetics and Molecular Biology · Computer Science · #Biomedical Text Mining and Ontologies #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Information Retrieval (cs.IR) #Natural Language Processing Techniques #Semantic Web and Ontologies