2025/12/15 by Du, Junye, Li, Zhenghao, Gu, Zhutong +1
#FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)
paper · doi:10.48550/arxiv.2512.13565
This paper tackles the problem of feature selection in a highly challenging setting: 𝔼(y | \boldsymbolx) = G(\boldsymbolxS0), where S0 is the set of relevant features and G is an unknown, potentially nonlinear function subject to mild smoothness conditions. Our approach begins with feature selection in deep neural networks, then generalizes the results to Hölder smooth functions by exploiting the strong approximation capabilities of neural networks. Unlike conventional optimization-based deep learning methods, we reformulate neural networks as index models and estimate S0 using the second-order Stein's formula. This gradient-descent-free strategy guarantees feature selection consistency with a sample size requirement of n = Ω(p2), where p is the feature dimension. To handle high-dimensional scenarios, we further introduce a screening-and-selection mechanism that achieves nonlinear selection consistency when n = Ω(s log p), with s representing the sparsity level. Additionally, we refit a neural network on the selected features for prediction and establish performance guarantees under a relaxed sparsity assumption. Extensive simulations and real-data analyses demonstrate the strong performance of our method even in the presence of complex feature interactions.