2023/05/27 by Kaize Ding, Albert Jiongqian Liang, Ding, Kaize +15 · 2 citations
Computer Science · #Advanced Graph Neural Networks #FOS: Computer and information sciences #Information Retrieval (cs.IR) #Machine Learning (cs.LG) #Text and Document Classification Technologies #Topic Modeling
paper · pdf · doi:10.48550/arxiv.2305.17386
openalex publication_date 2023/05/27 · openalex created_date 2023/05/31 · openalex updated_date 2026/07/28
Learning expressive representations for high-dimensional yet sparse features has been a longstanding problem in information retrieval. Though recent deep learning methods can partially solve the problem, they often fail to handle the numerous sparse features, particularly those tail feature values with infrequent occurrences in the training data. Worse still, existing methods cannot explicitly leverage the correlations among different instances to help further improve the representation learning on sparse features since such relational prior knowledge is not provided. To address these challenges, in this paper, we tackle the problem of representation learning on feature-sparse data from a graph learning perspective. Specifically, we propose to model the sparse features of different instances using hypergraphs where each node represents a data instance and each hyperedge denotes a distinct feature value. By passing messages on the constructed hypergraphs based on our Hypergraph Transformer (HyperFormer), the learned feature representations capture not only the correlations among different instances but also the correlations among features. Our experiments demonstrate that the proposed approach can effectively improve feature representation learning on sparse features.