Louis‐Philippe Morency

1. Multimodal Machine Learning: A Survey and Taxonomy
   2017/05/26 by Tadas Baltrušaitis, Baltrušaitis, Tadas, Chaitanya Ahuja +4 · 1 voice · 161 citations
   Computer Science · #Advanced Text Analysis Techniques #FOS: Computer and information sciences #Machine Learning (cs.LG) #Music and Audio Processing #Text and Document Classification Technologies #cs.LG
2. Efficient Low-rank Multimodal Fusion with Modality-Specific Factors
   2018/05/31 by Zhun Liu, Ying Shen, Liu, Zhun +9 · 53 citations
   Computer Science · Psychology · #Sentiment Analysis and Opinion Mining #Emotion and Mood Recognition #Multimodal Machine Learning Applications
3. SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents
   2023/10/18 by Xuhui Zhou, Hao Zhu, Zhou, Xuhui +19 · 84 citations
   Computer Science · #Topic Modeling #Multi-Agent Systems and Negotiation #Natural Language Processing Techniques
4. Speaker-Follower Models for Vision-and-Language Navigation
   2018/06/07 by Daniel Fried, Ronghang Hu, Fried, Daniel +17 · 36 citations
   Computer Science · #Speech and dialogue systems #Natural Language Processing Techniques #Speech Recognition and Synthesis
5. Memory Fusion Network for Multi-view Sequential Learning
   2018/02/03 by Amir Zadeh, Zadeh, Amir, Paul Pu Liang +9 · 35 citations
   Computer Science · #Artificial Intelligence (cs.AI) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Face and Expression Recognition #Machine Learning (cs.LG) #Machine Learning and ELM
6. MultiBench: Multiscale Benchmarks for Multimodal Representation Learning
   2021/07/15 by Paul Pu Liang, Liang, Paul Pu, Yiwei Lyu +21 · 36 citations
   Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimedia (cs.MM) #Multimodal Machine Learning Applications #Topic Modeling
7. Think Locally, Act Globally: Federated Learning with Local and Global Representations
   2020/01/06 by Paul Pu Liang, Liang, Paul Pu, Terrance Liu +13 · 27 citations
   Computer Science · Social Sciences · #Distributed #FOS: Computer and information sciences #Human Mobility and Location-Based Analysis #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Parallel #Privacy, Security, and Data Protection #Privacy-Preserving Technologies in Data #and Cluster Computing (cs.DC)
8. Learning Factorized Multimodal Representations
   2018/06/16 by Yao-Hung Hubert Tsai, Tsai, Yao-Hung Hubert, Paul Pu Liang +7 · 29 citations
   Computer Science · #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Multimodal Machine Learning Applications #Sentiment Analysis and Opinion Mining #Topic Modeling
9. Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions
   2022/09/07 by Paul Pu Liang, Liang, Paul Pu, Amir Zadeh +3 · 36 citations
   Computer Science · #Speech and dialogue systems #Topic Modeling
10. Towards Understanding and Mitigating Social Biases in Language Models
    2021/06/24 by Paul Pu Liang, Chiyu Wu, Liang, Paul Pu +5 · 27 citations
    Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Computers and Society (cs.CY) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Text Readability and Simplification #Topic Modeling
11. Transformer Dissection: A Unified Understanding of Transformer's\n Attention via the Lens of Kernel
    2019/08/30 by Yao-Hung Hubert Tsai, Tsai, Yao-Hung Hubert, Shaojie Bai +7 · 21 citations
    Computer Science · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Natural Language Processing Techniques #Speech Recognition and Synthesis #Topic Modeling
12. Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
    2018/12/19 by Hai Pham, Paul Pu Liang, Pham, Hai +7 · 20 citations
    Computer Science · #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Human-Computer Interaction (cs.HC) #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Multimodal Machine Learning Applications #Sentiment Analysis and Opinion Mining #Topic Modeling
13. Towards Debiasing Sentence Representations
    2020/07/16 by Paul Pu Liang, Irene Li, Liang, Paul Pu +9 · 17 citations
    Computer Science · #Topic Modeling #Hate Speech and Cyberbullying Detection #Natural Language Processing Techniques
14. Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework
    2023/02/23 by Paul Pu Liang, Yun Cheng, Liang, Paul Pu +19 · 22 citations
    Computer Science · #Topic Modeling #Machine Learning in Healthcare #Explainable Artificial Intelligence (XAI)
15. Think Twice: Perspective-Taking Improves Large Language Models' Theory-of-Mind Capabilities
    2023/11/16 by Alex Wilf, Wilf, Alex, Sihyun Shawn Lee +5 · 26 citations
    Computer Science · Social Sciences · #Topic Modeling #Computational and Text Analysis Methods #Explainable Artificial Intelligence (XAI)
16. Uncertainty Quantification with Pre-trained Language Models: A Large-Scale Empirical Analysis
    2022/10/10 by Yuxin Xiao, Paul Pu Liang, Xiao, Yuxin +9 · 13 citations
    Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Natural Language Processing Techniques #Text Readability and Simplification #Topic Modeling
17. Deep Gamblers: Learning to Abstain with Portfolio Theory
    2019/06/29 by Liu Ziyin, Ziyin, Liu, Zhikang Wang +9 · 9 citations
    Economics, Econometrics and Finance · Engineering · #FOS: Computer and information sciences #Financial Markets and Investment Strategies #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Reservoir Engineering and Simulation Methods #Sports Analytics and Performance
18. DIME: Fine-grained Interpretations of Multimodal Models via Disentangled Local Explanations
    2022/03/03 by Yiwei Lyu, Lyu, Yiwei, Paul Pu Liang +7 · 10 citations
    Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning in Healthcare #Topic Modeling
19. Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal\n Sentiment Analysis
    2021/07/28 by Wei Han, Han, Wei, Hui Chen +9 · 7 citations
    Computer Science · Social Sciences · #Advanced Computing and Algorithms #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Rough Sets and Fuzzy Logic #Sentiment Analysis and Opinion Mining
20. Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviors
    2018/11/23 by Yansen Wang, Wang, Yansen, Ying Shen +9 · 7 citations
    Computer Science · Psychology · #Sentiment Analysis and Opinion Mining #Emotion and Mood Recognition #Topic Modeling
21. MMoE: Enhancing Multimodal Models with Mixtures of Multimodal Interaction Experts
    2023/11/16 by Haofei Yu, Yu, Haofei, Qi, Zhengyang +7 · 10 citations
    Computer Science · Psychology · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Language, Metaphor, and Cognition #Multimodal Machine Learning Applications #Topic Modeling
22. Understanding Masked Autoencoders via Hierarchical Latent Variable Models
    2023/06/08 by Lingjing Kong, Martin Q. Ma, Kong, Lingjing +11 · 9 citations
    Computer Science · Engineering · #Generative Adversarial Networks and Image Synthesis #Domain Adaptation and Few-Shot Learning #Image Processing Techniques and Applications
23. Advancing Social Intelligence in AI Agents: Technical Challenges and Open Questions
    2024/04/17 by Leena Mathur, Paul Pu Liang, Mathur, Leena +3 · 12 citations
    Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Human-Computer Interaction (cs.HC) #Machine Learning (cs.LG) #Online Learning and Analytics
24. Learning Representations from Imperfect Time Series Data via Tensor Rank\n Regularization
    2019/07/01 by Paul Pu Liang, Zhun Liu, Liang, Paul Pu +9 · 5 citations
    Computer Science · Mathematics · #Computation and Language (cs.CL) #Computational Physics and Python Applications #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Tensor decomposition and applications #Topic Modeling
25. Cross-modal Attention Congruence Regularization for Vision-Language Relation Alignment
    2022/12/20 by Rohan Pandey, Pandey, Rohan, Rulin Shao +7 · 6 citations
    Computer Science · #Advanced Image and Video Retrieval Techniques #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimodal Machine Learning Applications
26. High-Modality Multimodal Transformer: Quantifying Modality & Interaction Heterogeneity for High-Modality Representation Learning
    2022/03/02 by Paul Pu Liang, Yiwei Lyu, Liang, Paul Pu +13 · 6 citations
    Computer Science · #Speech and dialogue systems #Multimodal Machine Learning Applications #Music and Audio Processing
27. Neural Methods for Point-wise Dependency Estimation
    2020/06/09 by Yao-Hung Hubert Tsai, Han Zhao, Tsai, Yao-Hung Hubert +7 · 4 citations
    Computer Science · Materials Science · #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning in Materials Science #Methodology (stat.ME) #Neural Networks and Applications
28. Conditional Contrastive Learning for Improving Fairness in Self-Supervised Learning
    2021/06/05 by Martin Q. Ma, Yao-Hung Hubert Tsai, Ma, Martin Q. +11 · 4 citations
    Computer Science · #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG)
29. MultiViz: Towards Visualizing and Understanding Multimodal Models
    2022/06/30 by Paul Pu Liang, Yiwei Lyu, Liang, Paul Pu +13 · 4 citations
    Computer Science · #Explainable Artificial Intelligence (XAI) #Software Engineering Research #Anomaly Detection Techniques and Applications
30. Multimodal Learning Without Labeled Multimodal Data: Guarantees and Applications
    2023/06/07 by Paul Pu Liang, Chun Kai Ling, Liang, Paul Pu +15 · 5 citations
    Computer Science · #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Information Theory (cs.IT) #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Music and Audio Processing
31. HEMM: Holistic Evaluation of Multimodal Foundation Models
    2024/07/03 by Paul Pu Liang, Liang, Paul Pu, Akshay Goindani +11 · 9 citations
    Engineering · #BIM and Construction Integration
32. Select-Additive Learning: Improving Generalization in Multimodal Sentiment Analysis
    2016/09/16 by Haohan Wang, Wang, Haohan, Aaksha Meghawat +5 · 2 citations
    Computer Science · Psychology · #Computation and Language (cs.CL) #Emotion and Mood Recognition #FOS: Computer and information sciences #Information Retrieval (cs.IR) #Sentiment Analysis and Opinion Mining #Topic Modeling
33. Understanding the Tradeoffs in Client-side Privacy for Downstream Speech Tasks
    2021/01/22 by Peter Wu, Paul Pu Liang, Wu, Peter +9 · 2 citations
    Computer Science · #Adversarial Robustness in Machine Learning #Audio and Speech Processing (eess.AS) #Cryptography and Security (cs.CR) #FOS: Computer and information sciences #FOS: Electrical engineering #Privacy-Preserving Technologies in Data #Sound (cs.SD) #Speech Recognition and Synthesis #electronic engineering #information engineering
34. Conditional Contrastive Learning with Kernel
    2022/02/11 by Yao-Hung Hubert Tsai, Tsai, Yao-Hung Hubert, Tianqin Li +11 · 2 citations
    Computer Science · Health Professions · #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Interpreting and Communication in Healthcare #Machine Learning (cs.LG)
35. PACS: A Dataset for Physical Audiovisual CommonSense Reasoning
    2022/03/21 by Samuel Yu, Peter Wu, Yu, Samuel +7 · 2 citations
    Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimedia (cs.MM) #Multimodal Machine Learning Applications #Music and Audio Processing #Speech and dialogue systems
36. Text-Transport: Toward Learning Causal Effects of Natural Language
    2023/10/31 by Victoria Lin, Louis‐Philippe Morency, Lin, Victoria +3 · 2 citations
    Computer Science · Social Sciences · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Hate Speech and Cyberbullying Detection #Media Influence and Politics #Methodology (stat.ME) #Social Media and Politics
37. LLAMADRS: Evaluating Open-Source LLMs on Real Clinical Interviews--To Reason or Not to Reason?
    2025/01/07 by Gaoussou Youssouf Kebe, Jeffrey M. Girard, Kebe, Gaoussou Youssouf +9 · 4 citations
    Psychology · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Human-Computer Interaction (cs.HC) #I.2.7 #Mental Health via Writing
38. IoT-LM: Large Multisensory Language Models for the Internet of Things
    2024/07/13 by Shentong Mo, Mo, Shentong, Russ R. Salakhutdinov +5 · 3 citations
    Agricultural and Biological Sciences · Social Sciences · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Food Supply Chain Traceability #Machine Learning (cs.LG) #Multimedia (cs.MM) #Public Relations and Crisis Communication
39. Diverse and Admissible Trajectory Forecasting through Multimodal Context\n Understanding
    2020/03/06 by Seong Hyeon Park, Gyubok Lee, Park, Seong Hyeon +15 · 1 citation
    Computer Science · Engineering · #Autonomous Vehicle Technology and Safety #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Time Series Analysis and Forecasting #Video Surveillance and Tracking Methods
40. Visual Referring Expression Recognition: What Do Systems Actually Learn?
    2018/05/30 by Volkan Cirik, Louis‐Philippe Morency, Cirik, Volkan +3 · 2 citations
    Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Neural and Evolutionary Computing (cs.NE) #Topic Modeling
41. A Note on Connecting Barlow Twins with Negative-Sample-Free Contrastive Learning
    2021/04/28 by Yao-Hung Hubert Tsai, Shaojie Bai, Tsai, Yao-Hung Hubert +5 · 1 citation
    Computer Science · #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and Algorithms #Machine Learning and ELM
42. StylePTB: A Compositional Benchmark for Fine-grained Controllable Text\n Style Transfer
    2021/04/12 by Yiwei Lyu, Lyu, Yiwei, Paul Pu Liang +11 · 1 citation
    Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Speech Recognition and Synthesis #Topic Modeling
43. Using Syntax to Ground Referring Expressions in Natural Images
    2018/05/26 by Volkan Cirik, Cirik, Volkan, Taylor Berg-Kirkpatrick +3 · 2 citations
    Computer Science · #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Human Pose and Action Recognition #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Neural and Evolutionary Computing (cs.NE)
44. Learning Weakly-Supervised Contrastive Representations
    2022/02/14 by Yao-Hung Hubert Tsai, Tianqin Li, Tsai, Yao-Hung Hubert +9 · 1 citation
    Computer Science · #Artificial Intelligence (cs.AI) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Music and Audio Processing #Text and Document Classification Technologies
45. Multimodal Lecture Presentations Dataset: Understanding Multimodality in Educational Slides
    2022/08/17 by Dong Won Lee, Chaitanya Ahuja, Lee, Dong Won +7 · 1 citation
    Arts and Humanities · Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimedia (cs.MM) #Multimodal Machine Learning Applications #Subtitles and Audiovisual Media #Video Analysis and Summarization
46. SeedBERT: Recovering Annotator Rating Distributions from an Aggregated Label
    2022/11/23 by Aneesha Sampath, Sampath, Aneesha, Victoria Lin +3 · 1 citation
    Computer Science · Psychology · Social Sciences · #Computation and Language (cs.CL) #Emotion and Mood Recognition #FOS: Computer and information sciences #Machine Learning (cs.LG) #Misinformation and Its Impacts #Sentiment Analysis and Opinion Mining
47. Difference-Masking: Choosing What to Mask in Continued Pretraining
    2023/05/23 by Alex Wilf, Syeda Nahida Akter, Wilf, Alex +13 · 1 citation
    Computer Science · #Computation and Language (cs.CL) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimodal Machine Learning Applications
48. Multimodal Fusion Interactions: A Study of Human and Automatic Quantification
    2023/06/07 by Paul Pu Liang, Yun Cheng, Liang, Paul Pu +5 · 1 citation
    Computer Science · Decision Sciences · #Advanced Text Analysis Techniques #Computation and Language (cs.CL) #FOS: Computer and information sciences #Human-Computer Interaction (cs.HC) #Machine Learning (cs.LG) #Multi-Criteria Decision Making #Speech and dialogue systems
49. MultiZoo & MultiBench: A Standardized Toolkit for Multimodal Deep Learning
    2023/06/28 by Paul Pu Liang, Yiwei Lyu, Liang, Paul Pu +11 · 1 citation
    Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimedia (cs.MM) #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling
50. From Reproduction to Replication: Evaluating Research Agents with Progressive Code Masking
    2025/06/24 by G.J. Kim, Kim, Gyeongwon James, Alex Wilf +5 · 2 citations
    Computer Science · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Model-Driven Software Engineering Techniques #Semantic Web and Ontologies #Software Engineering Research
51. Factorized Multimodal Transformer for Multimodal Sequential Learning
    2019/11/22 by Amir Zadeh, Zadeh, Amir, Chengfeng Mao +11 · 1 citation
    Computer Science · #Speech and dialogue systems #Speech Recognition and Synthesis #Topic Modeling
52. Learning Not to Learn in the Presence of Noisy Labels
    2020/02/16 by Liu Ziyin, Blair Chen, Ziyin, Liu +11 · 3 citations
    Computer Science · #FOS: Computer and information sciences #Information Theory (cs.IT) #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Algorithms #Machine Learning and Data Classification #Text and Document Classification Technologies