Set-to-Sequence Methods in Machine Learning: A Review

2021/03/31 by Mateusz Jurewicz, Leon Derczynski, Leon Strømberg-Derczynski · 1 citation
Computer Science · Engineering · Mathematics · #Artificial intelligence #Computer science #Engineering #Field (mathematics) #Grid #Invariant (physics) #Key (lock) #Machine Learning and Algorithms #Machine learning #Mathematics #Multi-task learning #Natural Language Processing Techniques #Permutation (music) #Programming language #Representation (politics) #Sequence (biology) #Set (abstract data type) #Structured prediction #Task (project management) #Theoretical computer science #Topic Modeling #acm:68T01 #acm:68T07 #cs.AI #cs.LG #msc:68T01 #msc:68T07

paper · pdf · doi:10.1613/jair.1.12839

published in Journal of Artificial Intelligence Research 71, 885-924 (AI Access Foundation) · 46 pages of text, with 10 pages of references. Contains 2 tables and 4 figures. Updated version includes expanded notes on method comparison

openalex publication_date 2021/08/12 · arxiv created 2021/08/16 · arxiv updated 2021/09/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Machine learning on sets towards sequential output is an important and ubiquitous task, with applications ranging from language modelling and meta-learning to multi-agent strategy games and power grid optimization. Combining elements of representation learning and structured prediction, its two primary challenges include obtaining a meaningful, permutation invariant set representation and subsequently utilizing this representation to output a complex target permutation. This paper provides a comprehensive introduction to the _eld as well as an overview of important machine learning methods tackling both of these key challenges, with a detailed qualitative comparison of selected model architectures.

Citations

• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Axiomatic Set Theory
• The Proof and Measurement of Association between Two Things
• Reasoning about Entailment with Neural Attention
• Sequence to Sequence Learning with Neural Networks
• Neural Machine Translation by Jointly Learning to Align and Translate
• Adaptive Computation Time for Recurrent Neural Networks
• Learning Convolutional Neural Networks for Graphs
• Sequence-to-Sequence Learning as Beam-Search Optimization
• Learning Combinatorial Optimization Algorithms over Graphs
• Inductive Representation Learning on Large Graphs
• Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning
• CondenseNet: An Efficient DenseNet using Learned Group Convolutions
• Reinforcement Learning for Solving the Vehicle Routing Problem
• Deep Architectures for Learning Context-dependent Ranking Functions
• Natural Language to Structured Query Generation via Meta-Learning
• Learning Permutations with Sinkhorn Policy Gradient
• Universal Transformers
• Unsupervised Adversarial Invariance
• Actor-Attention-Critic for Multi-Agent Reinforcement Learning
• An Introductory Survey on Attention Mechanisms in NLP Problems
• Stochastic Optimization of Sorting Networks via Continuous Relaxations
• Attention, Learn to Solve Routing Problems!
• On the Limitations of Representing Functions on Sets
• FSPool: Learning Set Representations with Featurewise Sort Pooling
• PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation
• Theoretical Limitations of Self-Attention in Neural Sequence Models
• Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks
• Sentence Ordering and Coherence Modeling using Recurrent Neural Networks
• SetRank: Learning a Permutation-Invariant Ranking Model for Information Retrieval
• Deep Sets for Generalization in RL
• Language Models are Few-Shot Learners
• Conditional Set Generation with Transformers
• Machine learning for combinatorial optimization: A methodological tour d’horizon
• Dynamic Neural Networks: A Survey
• Meta-Learning for One-Class Classification with Few Examples using Order-Equivariant Network