Distributed stochastic gradient tracking methods with momentum acceleration for non-convex optimization

doi:10.1007/s10589-022-00432-5

CSpace

	Distributed stochastic gradient tracking methods with momentum acceleration for non-convex optimization
	Gao, Juan 1; Liu, Xin-Wei 2; Dai, Yu-Hong 3,4; Huang, Yakui 2; Gu, Junhua 1
	2022-11-23
发表期刊	COMPUTATIONAL OPTIMIZATION AND APPLICATIONS
ISSN	0926-6003
页码	42
摘要	We consider a distributed non-convex optimization problem of minimizing the sum of all local cost functions over a network of agents. This problem often appears in large-scale distributed machine learning, known as non-convex empirical risk minimization. In this paper, we propose two accelerated algorithms, named DSGT-HB and DSGT-NAG, which combine the distributed stochastic gradient tracking (DSGT) method with momentum accelerated techniques. Under appropriate assumptions, we prove that both algorithms sublinearly converge to a neighborhood of a first-order stationary point of the distributed non-convex optimization. Moreover, we derive the conditions under which DSGT-HB and DSGT-NAG achieve a network-independent linear speedup. Numerical experiments for a distributed non-convex logistic regression problem on real data sets and a deep neural network on the MNIST database show the superiorities of DSGT-HB and DSGT-NAG compared with DSGT.
关键词	Distributed non-convex optimization Machine learning Momentum methods Optimization algorithms Convergence rate
DOI	10.1007/s10589-022-00432-5
收录类别	SCI
语种	英语
资助项目	National Natural Science Foundation of China[12071108] ; National Natural Science Foundation of China[11671116] ; National Natural Science Foundation of China[12021001] ; National Natural Science Foundation of China[11991021] ; National Natural Science Foundation of China[11991020] ; National Natural Science Foundation of China[11971372] ; National Natural Science Foundation of China[11701137] ; Strategic Priority Research Program of CAS[XDA27000000]
WOS研究方向	Operations Research & Management Science ; Mathematics
WOS类目	Operations Research & Management Science ; Mathematics, Applied
WOS记录号	WOS:000886800900001
出版者	SPRINGER
引用统计
文献类型	期刊论文
条目标识符	http://ir.amss.ac.cn/handle/2S8OKBNM/60593
专题	中国科学院数学与系统科学研究院
通讯作者	Liu, Xin-Wei
作者单位	1.Hebei Univ Technol, Sch Artificial Intelligence, Tianjin 300401, Peoples R China 2.Hebei Univ Technol, Inst Math, Tianjin 300401, Peoples R China 3.Chinese Acad Sci, Acad Math & Syst Sci, ICMSEC, LSEC, Beijing 100190, Peoples R China 4.Univ Chinese Acad Sci, Sch Math Sci, Beijing 100049, Peoples R China
推荐引用方式 GB/T 7714	Gao, Juan,Liu, Xin-Wei,Dai, Yu-Hong,et al. Distributed stochastic gradient tracking methods with momentum acceleration for non-convex optimization[J]. COMPUTATIONAL OPTIMIZATION AND APPLICATIONS,2022:42.
APA	Gao, Juan,Liu, Xin-Wei,Dai, Yu-Hong,Huang, Yakui,&Gu, Junhua.(2022).Distributed stochastic gradient tracking methods with momentum acceleration for non-convex optimization.COMPUTATIONAL OPTIMIZATION AND APPLICATIONS,42.
MLA	Gao, Juan,et al."Distributed stochastic gradient tracking methods with momentum acceleration for non-convex optimization".COMPUTATIONAL OPTIMIZATION AND APPLICATIONS (2022):42.