Make l(1) regularization effective in training sparse CNN

doi:10.1007/s10589-020-00202-1

CSpace

	Make l(1) regularization effective in training sparse CNN
	He, Juncai 1; Jia, Xiaodong 2; Xu, Jinchao 1; Zhang, Lian 1; Zhao, Liang 3,4
	2020-09-01
发表期刊	COMPUTATIONAL OPTIMIZATION AND APPLICATIONS
ISSN	0926-6003
卷号	77 期号:1 页码:163-182
摘要	Compressed Sensing using l(1) regularization is among the most powerful and popular sparsification technique in many applications, but why has it not been used to obtain sparse deep learning model such as convolutional neural network (CNN)? This paper is aimed to provide an answer to this question and to show how to make it work. Following Xiao (J Mach Learn Res 11(Oct):2543-2596, 2010), We first demonstrate that the commonly used stochastic gradient decent and variants training algorithm is not an appropriate match with l(1) regularization and then replace it with a different training algorithm based on a regularized dual averaging (RDA) method. The RDA method of Xiao (J Mach Learn Res 11(Oct):2543-2596, 2010) was originally designed specifically for convex problem, but with new theoretical insight and algorithmic modifications (using proper initialization and adaptivity), we have made it an effective match with l(1) regularization to achieve a state-of-the-art sparsity for the highly non-convex CNN compared to other weight pruning methods without compromising accuracy (achieving 95% sparsity for ResNet-18 on CIFAR-10, for example).
关键词	Sparse optimization l(1) regularization Dual averaging CNN
DOI	10.1007/s10589-020-00202-1
收录类别	SCI
语种	英语
资助项目	Peking University Joint Center for Computational Mathematics and Applications ; Beijing International Center for Mathematical Research from Peking University ; Verne M. William Professorship Fund from Penn State University ; China Scholarship Council ; Hong Kong RGC Competitive Earmarked Research Grant[HKUST16301218] ; Penn State University Joint Center for Computational Mathematics and Applications
WOS研究方向	Operations Research & Management Science ; Mathematics
WOS类目	Operations Research & Management Science ; Mathematics, Applied
WOS记录号	WOS:000545777500001
出版者	SPRINGER
引用统计
文献类型	期刊论文
条目标识符	http://ir.amss.ac.cn/handle/2S8OKBNM/51757
专题	中国科学院数学与系统科学研究院
通讯作者	Xu, Jinchao
作者单位	1.Penn State Univ, Dept Math, University Pk, PA 16802 USA 2.Penn State Univ, Dept Comp Sci & Engn, University Pk, PA 16802 USA 3.Chinese Acad Sci, State Key Lab Sci & Engn Comp, Acad Math & Syst Sci, Beijing 100190, Peoples R China 4.Univ Chinese Acad Sci, Beijing 100190, Peoples R China
推荐引用方式 GB/T 7714	He, Juncai,Jia, Xiaodong,Xu, Jinchao,et al. Make l(1) regularization effective in training sparse CNN[J]. COMPUTATIONAL OPTIMIZATION AND APPLICATIONS,2020,77(1):163-182.
APA	He, Juncai,Jia, Xiaodong,Xu, Jinchao,Zhang, Lian,&Zhao, Liang.(2020).Make l(1) regularization effective in training sparse CNN.COMPUTATIONAL OPTIMIZATION AND APPLICATIONS,77(1),163-182.
MLA	He, Juncai,et al."Make l(1) regularization effective in training sparse CNN".COMPUTATIONAL OPTIMIZATION AND APPLICATIONS 77.1(2020):163-182.