This my TIL today. I just found out that momentum is already very good at finding good local minima. It can be better than adaptive (and more sophisticated) optimisers if carefully tuned. Adaptive optimisers are designed to be more robust, where we don’t need to use good hyperparameters to start with. However, adaptive optimisers prone to assume they are in a correct direction to local minima, which causes them to adjust the learning rate so that they will converge faster. The optimisers will reduce the learning rate if the minimum is close (so that it won’t overshoot) and vice versa. With that said, they are more likely to converge to sharp local minima.
On the other hand, momentum doesn’t suffer this.
That’s why momentum is used almost everywhere.
|Fast R-CNN ||Momentum|
|Faster R-CNN ||Momentum|
 Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. “Imagenet classification with deep convolutional neural networks.” Advances in neural information processing systems. 2012.
 Zeiler, Matthew D., and Rob Fergus. “Visualizing and understanding convolutional networks.” European conference on computer vision. Springer, Cham, 2014.
 Simonyan, Karen, and Andrew Zisserman. “Very deep convolutional networks for large-scale image recognition.” arXiv preprint arXiv:1409.1556 (2014).
 Szegedy, Christian, et al. “Going deeper with convolutions.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2015.
 He, Kaiming, et al. “Deep residual learning for image recognition.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.
 Girshick, Ross, et al. “Rich feature hierarchies for accurate object detection and semantic segmentation.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2014.
 Girshick, Ross. “Fast r-cnn.” Proceedings of the IEEE international conference on computer vision. 2015.
 Ren, Shaoqing, et al. “Faster r-cnn: Towards real-time object detection with region proposal networks.” Advances in neural information processing systems. 2015.