KMS Of Academy of mathematics and systems sciences, CAS
| Stochastic Variance Reduced Gradient Methods Using a Trust-Region-Like Scheme | |
| Yu, Tengteng1; Liu, Xin-Wei2; Dai, Yu-Hong3,4; Sun, Jie2,5 | |
| 2021-02-17 | |
| Source Publication | JOURNAL OF SCIENTIFIC COMPUTING
 |
| ISSN | 0885-7474 |
| Volume | 87Issue:1Pages:24 |
| Abstract | Stochastic variance reduced gradient (SVRG) methods are important approaches to minimize the average of a large number of cost functions frequently arising in machine learning and many other applications. In this paper, based on SVRG, we propose a SVRG-TR method which employs a trust-region-like scheme for selecting stepsizes. It is proved that the SVRG-TR method is linearly convergent in expectation for smooth strongly convex functions and enjoys a faster convergence rate than SVRG methods. In order to overcome the difficulty of tuning stepsizes by hand, we propose to combine the Barzilai-Borwein (BB) method to automatically compute stepsizes for the SVRG-TR method, named as the SVRG-TR-BB method. By incorporating mini-batching scheme with SVRG-TR and SVRG-TR-BB, respectively, we further propose two extended methods mSVRG-TR and mSVRG-TR-BB. Linear convergence and complexity of mSVRG-TR are given. Numerical experiments on some standard datasets show that SVRG-TR and SVRG-TR-BB are generally better than or comparable to SVRG with best-tuned stepsizes and some modern stochastic gradient methods, while mSVRG-TR and mSVRG-TR-BB are very competitive with mini-batch variants of recent successful stochastic gradient methods. |
| Keyword | Stochastic variance reduced gradient Trust region Barzilai-Borwein stepsizes Mini-batches Empirical risk minimization 90C06 90C30 90C90 90C25 |
| DOI | 10.1007/s10915-020-01402-x |
| Indexed By | SCI |
| Language | 英语 |
| Funding Project | Chinese NSF[11671116] ; Chinese NSF[11701137] ; Chinese NSF[12071108] ; Chinese NSF[11631013] ; Chinese NSF[11991020] ; Chinese NSF[12021001] ; Major Research Plan of the NSFC[91630202] ; Beijing Academy of Artificial Intelligence (BAAI) |
| WOS Research Area | Mathematics |
| WOS Subject | Mathematics, Applied |
| WOS ID | WOS:000620641500005 |
| Publisher | SPRINGER/PLENUM PUBLISHERS |
| Citation statistics | |
| Document Type | 期刊论文 |
| Identifier | http://ir.amss.ac.cn/handle/2S8OKBNM/58195 |
| Collection | 中国科学院数学与系统科学研究院 |
| Corresponding Author | Liu, Xin-Wei |
| Affiliation | 1.Hebei Univ Technol, Sch Artificial Intelligence, Tianjin 300401, Peoples R China 2.Hebei Univ Technol, Inst Math, Tianjin 300401, Peoples R China 3.Chinese Acad Sci, Acad Math & Syst Sci, Inst Computat Math & Sci Engn Comp, State Key Lab Sci & Engn Comp, Beijing 100190, Peoples R China 4.Univ Chinese Acad Sci, Sch Math Sci, Beijing 100049, Peoples R China 5.Natl Univ Singapore, Sch Business, Singapore 119245, Singapore |
| Recommended Citation GB/T 7714 | Yu, Tengteng,Liu, Xin-Wei,Dai, Yu-Hong,et al. Stochastic Variance Reduced Gradient Methods Using a Trust-Region-Like Scheme[J]. JOURNAL OF SCIENTIFIC COMPUTING,2021,87(1):24. |
| APA | Yu, Tengteng,Liu, Xin-Wei,Dai, Yu-Hong,&Sun, Jie.(2021).Stochastic Variance Reduced Gradient Methods Using a Trust-Region-Like Scheme.JOURNAL OF SCIENTIFIC COMPUTING,87(1),24. |
| MLA | Yu, Tengteng,et al."Stochastic Variance Reduced Gradient Methods Using a Trust-Region-Like Scheme".JOURNAL OF SCIENTIFIC COMPUTING 87.1(2021):24. |
| Files in This Item: | There are no files associated with this item. | |||||
Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.
Edit Comment