非线性单元：

Maxout Ian J Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio. Maxout networks. arXiv preprint arXiv:1302.4389, 2013.

dropout Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1): 1929–1958, 2014.

LReLU Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. arXiv preprint arXiv:1502.01852, 2015.

目前非线性单元一般不破坏ReLU的结构而用非线性的运算方法接入网络层与层之间来产生非线性表达能力。

增加模型深度：

NIN Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. 12 2013. URL http://arxiv.org/abs/1312.4400.

Inception/GoogLeNet Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. arXiv preprint arXiv:1409.4842, 2014.

这里指的“深度”不光指层数，仅靠增加层数会带来训练困难。这里是指在有限层增加网络复杂程度。也可以说是非线性单元的一种变体。

训练过程中的效率：

LReLU Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. arXiv preprint arXiv:1502.01852, 2015.

BatchNorm Sergey Ioffe, Christian Szegedy,. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift.

前者为参数提供了更加容易收敛初始值，后者防止训练过程中的梯度发散，两者都是解决同类问题，vanishing gradients（前）和exploding gradients（后）。

Detection

RCNN Ross Girshick, Jeff Donahue, Trevor Darrell, Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation.
CV proposal + CNN feature extraction + SVM classifier

Fast RCNN Ross Girshick. Fast R-CNN.
CV proposal + CNN feature extraction + Regression Network Prediction

Faster RCNN Shaoqing Ren, Kaiming He, Ross Girshick, Jian Sun. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks.
CNN Regression Network Proposal + CNN feature extraction + Regression Network Prediction