Stochastic Variance Reduced Gradient Methods Using a Trust-Region-Like Scheme
Yu, Tengteng1; Liu, Xin-Wei2; Dai, Yu-Hong3,4; Sun, Jie2,5
AbstractStochastic variance reduced gradient (SVRG) methods are important approaches to minimize the average of a large number of cost functions frequently arising in machine learning and many other applications. In this paper, based on SVRG, we propose a SVRG-TR method which employs a trust-region-like scheme for selecting stepsizes. It is proved that the SVRG-TR method is linearly convergent in expectation for smooth strongly convex functions and enjoys a faster convergence rate than SVRG methods. In order to overcome the difficulty of tuning stepsizes by hand, we propose to combine the Barzilai-Borwein (BB) method to automatically compute stepsizes for the SVRG-TR method, named as the SVRG-TR-BB method. By incorporating mini-batching scheme with SVRG-TR and SVRG-TR-BB, respectively, we further propose two extended methods mSVRG-TR and mSVRG-TR-BB. Linear convergence and complexity of mSVRG-TR are given. Numerical experiments on some standard datasets show that SVRG-TR and SVRG-TR-BB are generally better than or comparable to SVRG with best-tuned stepsizes and some modern stochastic gradient methods, while mSVRG-TR and mSVRG-TR-BB are very competitive with mini-batch variants of recent successful stochastic gradient methods.
KeywordStochastic variance reduced gradient Trust region Barzilai-Borwein stepsizes Mini-batches Empirical risk minimization 90C06 90C30 90C90 90C25
Indexed BySCI
Funding ProjectChinese NSF[11671116] ; Chinese NSF[11701137] ; Chinese NSF[12071108] ; Chinese NSF[11631013] ; Chinese NSF[11991020] ; Chinese NSF[12021001] ; Major Research Plan of the NSFC[91630202] ; Beijing Academy of Artificial Intelligence (BAAI)
WOS Research AreaMathematics
WOS SubjectMathematics, Applied
WOS IDWOS:000620641500005
Citation statistics
Document Type期刊论文
Corresponding AuthorLiu, Xin-Wei
Affiliation1.Hebei Univ Technol, Sch Artificial Intelligence, Tianjin 300401, Peoples R China
2.Hebei Univ Technol, Inst Math, Tianjin 300401, Peoples R China
3.Chinese Acad Sci, Acad Math & Syst Sci, Inst Computat Math & Sci Engn Comp, State Key Lab Sci & Engn Comp, Beijing 100190, Peoples R China
4.Univ Chinese Acad Sci, Sch Math Sci, Beijing 100049, Peoples R China
5.Natl Univ Singapore, Sch Business, Singapore 119245, Singapore
Recommended Citation
GB/T 7714
Yu, Tengteng,Liu, Xin-Wei,Dai, Yu-Hong,et al. Stochastic Variance Reduced Gradient Methods Using a Trust-Region-Like Scheme[J]. JOURNAL OF SCIENTIFIC COMPUTING,2021,87(1):24.
APA Yu, Tengteng,Liu, Xin-Wei,Dai, Yu-Hong,&Sun, Jie.(2021).Stochastic Variance Reduced Gradient Methods Using a Trust-Region-Like Scheme.JOURNAL OF SCIENTIFIC COMPUTING,87(1),24.
MLA Yu, Tengteng,et al."Stochastic Variance Reduced Gradient Methods Using a Trust-Region-Like Scheme".JOURNAL OF SCIENTIFIC COMPUTING 87.1(2021):24.
Files in This Item:
There are no files associated with this item.
Related Services
Recommend this item
Usage statistics
Export to Endnote
Google Scholar
Similar articles in Google Scholar
[Yu, Tengteng]'s Articles
[Liu, Xin-Wei]'s Articles
[Dai, Yu-Hong]'s Articles
Baidu academic
Similar articles in Baidu academic
[Yu, Tengteng]'s Articles
[Liu, Xin-Wei]'s Articles
[Dai, Yu-Hong]'s Articles
Bing Scholar
Similar articles in Bing Scholar
[Yu, Tengteng]'s Articles
[Liu, Xin-Wei]'s Articles
[Dai, Yu-Hong]'s Articles
Terms of Use
No data!
Social Bookmark/Share
All comments (0)
No comment.

Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.