2021/06/01 by Mark Anderson, Anders Søgaard, Anderson, Mark +3
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Software Engineering Research #Topic Modeling
paper · pdf · doi:10.48550/arxiv.2106.00352
openalex publication_date 2021/06/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
S ogaard (2020) obtained results suggesting the fraction of trees occurring\nin the test data isomorphic to trees in the training set accounts for a\nnon-trivial variation in parser performance. Similar to other statistical\nanalyses in NLP, the results were based on evaluating linear regressions.\nHowever, the study had methodological issues and was undertaken using a small\nsample size leading to unreliable results. We present a replication study in\nwhich we also bin sentences by length and find that only a small subset of\nsentences vary in performance with respect to graph isomorphism. Further, the\ncorrelation observed between parser performance and graph isomorphism in the\nwild disappears when controlling for covariants. However, in a controlled\nexperiment, where covariants are kept fixed, we do observe a strong\ncorrelation. We suggest that conclusions drawn from statistical analyses like\nthis need to be tempered and that controlled experiments can complement them by\nmore readily teasing factors apart.\n