BERT: A Review of Applications in Natural Language Processing and Understanding

2021/03/22 by Mikhail Koroteev, Koroteev, M. V. · 31 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Semantic Web and Ontologies

paper · pdf · doi:10.48550/arxiv.2103.11943

openalex publication_date 2021/03/22 · openalex created_date 2021/04/13 · openalex updated_date 2026/07/28

Abstract

In this review, we describe the application of one of the most popular deep learning-based language models - BERT. The paper describes the mechanism of operation of this model, the main areas of its application to the tasks of text analytics, comparisons with similar models in each task, as well as a description of some proprietary models. In preparing this review, the data of several dozen original scientific articles published over the past few years, which attracted the most attention in the scientific community, were systematized. This survey will be useful to all students and researchers who want to get acquainted with the latest advances in the field of natural language text analysis.

Cited by

• NepEMO: A Multi-Label Emotion and Sentiment Analysis on Nepali Reddit with Linguistic Insights and Temporal Trends
• Fast SAM2 with Text-Driven Token Pruning
• QE-Catalytic: A Graph-Language Multimodal Base Model for Relaxed-Energy Prediction in Catalytic Adsorption
• Rethinking Knowledge Distillation in Collaborative Machine Learning: Memory, Knowledge, and Their Interactions
• FlashFuser: Expanding the Scale of Kernel Fusion for Compute-Intensive Operators via Inter-Core Connection
• Prompting-in-a-Series: Psychology-Informed Contents and Embeddings for Personality Recognition With Decoder-Only Models
• ChipMind: Retrieval-Augmented Reasoning for Long-Context Circuit Design Specifications
• LLMs Know More Than Words: A Genre Study with Syntax, Metaphor & Phonetics
• MSME: A Multi-Stage Multi-Expert Framework for Zero-Shot Stance Detection
• StockMem: An Event-Reflection Memory Framework for Stock Forecasting
• HuggingR⁴: A Progressive Reasoning Framework for Discovering Optimal Model Companions
• Zero-Fi: Zero-Shot Wi-Fi-Based Human Activity Recognition via Contrastive Signal-Language Alignment
• Artificial intelligence in bioinformatics: a survey
• Beyond One-Size-Fits-All: Personalized Harmful Content Detection with In-Context Learning
• Activating Visual Context and Commonsense Reasoning through Masked Prediction in VLMs
• Metacognitive Self-Correction for Multi-Agent System via Prototype-Guided Next-Execution Reconstruction
• Synergistic Integration and Discrepancy Resolution of Contextualized Knowledge for Personalized Recommendation
• Indoor Localization using Compact, Telemetry-Agnostic, Transfer-Learning Enabled Decoder-Only Transformer
• MonitorVLM:A Vision Language Framework for Safety Violation Detection in Mining Operations
• TAP: Two-Stage Adaptive Personalization of Multi-task and Multi-Modal Foundation Models in Federated Learning
• Robust Fine-Tuning from Non-Robust Pretrained Models: Mitigating Suboptimal Transfer With Adversarial Scheduling
• Dendritic Resonate-and-Fire Neuron for Effective and Efficient Long Sequence Modeling
• OVITA: Open-Vocabulary Interpretable Trajectory Adaptations
• TableMind: An Autonomous Programmatic Agent for Tool-Augmented Table Reasoning
• LLM-GUARD: Large Language Model-Based Detection and Repair of Bugs and Security Vulnerabilities in C++ and Python
• CaresAI at BioCreative IX Track 1 -- LLM for Biomedical QA
• PRISM: A Framework Harnessing Unsupervised Visual Representations and Textual Prompts for Explainable MACE Survival Prediction from Cardiac Cine MRI
• Better Language Model-Based Judging Reward Modeling through Scaling Comprehension Boundaries
• Overcoming Low-Resource Barriers in Tulu: Neural Models and Corpus Creation for OffensiveLanguage Identification
• Remote Sensing Image Intelligent Interpretation with the Language-Centered Perspective: Principles, Methods and Challenges
• Breaking Obfuscation: Cluster-Aware Graph with LLM-Aided Recovery for Malicious JavaScript Detection

Related

• What the [MASK]? Making Sense of Language-Specific BERT Models
• A Primer in BERTology: What we know about how BERT works
• BERT for Sentiment Analysis: Pre-trained and Fine-Tuned Alternatives
• Advancements in Natural Language Processing: Exploring Transformer-Based Architectures for Text Understanding
• Universal Text Representation from BERT: An Empirical Study