Zeroth order GreedyLR: An adaptive learning rate scheduler for deep neural network training

Deep neural networks are a powerful tool for a wide range of applications, including natural language processing (NLP) and computer vision (CV). However, training these networks can be a challenging task, as it requires careful selection of hyperparameters such as learning rates and scheduling strategies. Despite significant advances in designing dynamic (and adaptive) learning rate schedulers, choosing the right learning rate/schedule for a machine learning task is still more art than science. In this paper, we introduce Zeroth order GreedyLR, a novel scheduler that adaptively adjusts the learning rate during training based on the current loss and gradient information. To validate the effectiveness of our proposed method, we conduct experiments on several NLP and CV tasks. The results show that our approach outperforms several state-of-the-art schedulers in terms of accuracy, speed, and convergence. Furthermore, our method is easy to implement, computationally efficient, and requires minimal hyperparameter tuning. Overall, our study provides a useful tool for researchers and practitioners in the field of deep learning.