KMS Of Academy of mathematics and systems sciences, CAS
| OptQuant: Distributed training of neural networks with optimized quantization mechanisms | |
He, Li1,2; Zheng, Shuxin3; Chen, Wei4; Ma, Zhi-Ming1,2 ; Liu, Tie-Yan4
| |
| 2019-05-07 | |
| Source Publication | NEUROCOMPUTING
 |
| ISSN | 0925-2312 |
| Volume | 340Pages:233-244 |
| Abstract | Nowadays, it has been a common practice to speed up the training of deep neural networks by utilizing multiple computational nodes. It is non-trivial to achieve desirable speed-up due to the potentially large communication overhead in distributed training. To reduce the communication cost, several unbiased random quantization mechanisms were proposed, in which the local workers, i.e., computational nodes, quantize their local gradients before communications with other workers. Most previous quantization mechanisms are static, i.e., the gradients are quantized in the same way during the training process. However, for different neural network models, the distributions of gradients might be very different even after normalization. To minimize the quantization loss better, we design the parameterized unbiased quantization mechanisms and dynamically optimize the quantization mechanism during the training process, using the aggregated information of the gradients. We call the distributed deep learning algorithms with our new quantization method (unbiased) OptQuant algorithms. Theoretically, we show that the unbiased OptQuant algorithms converge faster than the static unbiased quantization. In addition, if we trade-off the bias and the variance in the quantization, the algorithm converges faster. Motivated by this theoretical result, we further design the parameterized biased quantization mechanisms and the biased OptQuant algorithms. We evaluate our algorithms for different deep neural networks with benchmark datasets. Experimental results indicate that the OptQuant algorithms train the neural network models faster than previous quantization algorithms and much faster than the float version. (C) 2019 Elsevier B.V. All rights reserved. |
| Keyword | Distributed machine learning Deep learning Stochastic gradient descent Gradient quantization |
| DOI | 10.1016/j.neucom.2019.02.049 |
| Language | 英语 |
| Funding Project | National Center for Mathematics and Interdisciplinary Sciences of Chinese Academy of Sciences |
| WOS Research Area | Computer Science |
| WOS Subject | Computer Science, Artificial Intelligence |
| WOS ID | WOS:000461360000020 |
| Publisher | ELSEVIER SCIENCE BV |
| Citation statistics | |
| Document Type | 期刊论文 |
| Identifier | http://ir.amss.ac.cn/handle/2S8OKBNM/33374 |
| Collection | 应用数学研究所 |
| Corresponding Author | He, Li |
| Affiliation | 1.Univ Chinese Acad Sci, 19 A Yuquan Rd, Beijing, Shijingshan Dis, Peoples R China 2.Chinese Acad Sci, Acad Math & Syst Sci, 55 Zhongguancun East Rd, Beijing, Haidian Distric, Peoples R China 3.Univ Sci & Technol China, 96 Jinzhai Rd, Hefei, Anhui, Peoples R China 4.Microsoft Res, 5 Dan Ling St, Beijing, Haidian Distric, Peoples R China |
| Recommended Citation GB/T 7714 | He, Li,Zheng, Shuxin,Chen, Wei,et al. OptQuant: Distributed training of neural networks with optimized quantization mechanisms[J]. NEUROCOMPUTING,2019,340:233-244. |
| APA | He, Li,Zheng, Shuxin,Chen, Wei,Ma, Zhi-Ming,&Liu, Tie-Yan.(2019).OptQuant: Distributed training of neural networks with optimized quantization mechanisms.NEUROCOMPUTING,340,233-244. |
| MLA | He, Li,et al."OptQuant: Distributed training of neural networks with optimized quantization mechanisms".NEUROCOMPUTING 340(2019):233-244. |
| Files in This Item: | There are no files associated with this item. | |||||
Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.
Edit Comment