CSpace  > 应用数学研究所
OptQuant: Distributed training of neural networks with optimized quantization mechanisms
He, Li1,2; Zheng, Shuxin3; Chen, Wei4; Ma, Zhi-Ming1,2; Liu, Tie-Yan4
2019-05-07
发表期刊NEUROCOMPUTING
ISSN0925-2312
卷号340页码:233-244
摘要Nowadays, it has been a common practice to speed up the training of deep neural networks by utilizing multiple computational nodes. It is non-trivial to achieve desirable speed-up due to the potentially large communication overhead in distributed training. To reduce the communication cost, several unbiased random quantization mechanisms were proposed, in which the local workers, i.e., computational nodes, quantize their local gradients before communications with other workers. Most previous quantization mechanisms are static, i.e., the gradients are quantized in the same way during the training process. However, for different neural network models, the distributions of gradients might be very different even after normalization. To minimize the quantization loss better, we design the parameterized unbiased quantization mechanisms and dynamically optimize the quantization mechanism during the training process, using the aggregated information of the gradients. We call the distributed deep learning algorithms with our new quantization method (unbiased) OptQuant algorithms. Theoretically, we show that the unbiased OptQuant algorithms converge faster than the static unbiased quantization. In addition, if we trade-off the bias and the variance in the quantization, the algorithm converges faster. Motivated by this theoretical result, we further design the parameterized biased quantization mechanisms and the biased OptQuant algorithms. We evaluate our algorithms for different deep neural networks with benchmark datasets. Experimental results indicate that the OptQuant algorithms train the neural network models faster than previous quantization algorithms and much faster than the float version. (C) 2019 Elsevier B.V. All rights reserved.
关键词Distributed machine learning Deep learning Stochastic gradient descent Gradient quantization
DOI10.1016/j.neucom.2019.02.049
语种英语
资助项目National Center for Mathematics and Interdisciplinary Sciences of Chinese Academy of Sciences
WOS研究方向Computer Science
WOS类目Computer Science, Artificial Intelligence
WOS记录号WOS:000461360000020
出版者ELSEVIER SCIENCE BV
引用统计
文献类型期刊论文
条目标识符http://ir.amss.ac.cn/handle/2S8OKBNM/33374
专题应用数学研究所
通讯作者He, Li
作者单位1.Univ Chinese Acad Sci, 19 A Yuquan Rd, Beijing, Shijingshan Dis, Peoples R China
2.Chinese Acad Sci, Acad Math & Syst Sci, 55 Zhongguancun East Rd, Beijing, Haidian Distric, Peoples R China
3.Univ Sci & Technol China, 96 Jinzhai Rd, Hefei, Anhui, Peoples R China
4.Microsoft Res, 5 Dan Ling St, Beijing, Haidian Distric, Peoples R China
推荐引用方式
GB/T 7714
He, Li,Zheng, Shuxin,Chen, Wei,et al. OptQuant: Distributed training of neural networks with optimized quantization mechanisms[J]. NEUROCOMPUTING,2019,340:233-244.
APA He, Li,Zheng, Shuxin,Chen, Wei,Ma, Zhi-Ming,&Liu, Tie-Yan.(2019).OptQuant: Distributed training of neural networks with optimized quantization mechanisms.NEUROCOMPUTING,340,233-244.
MLA He, Li,et al."OptQuant: Distributed training of neural networks with optimized quantization mechanisms".NEUROCOMPUTING 340(2019):233-244.
条目包含的文件
条目无相关文件。
个性服务
推荐该条目
保存到收藏夹
查看访问统计
导出为Endnote文件
谷歌学术
谷歌学术中相似的文章
[He, Li]的文章
[Zheng, Shuxin]的文章
[Chen, Wei]的文章
百度学术
百度学术中相似的文章
[He, Li]的文章
[Zheng, Shuxin]的文章
[Chen, Wei]的文章
必应学术
必应学术中相似的文章
[He, Li]的文章
[Zheng, Shuxin]的文章
[Chen, Wei]的文章
相关权益政策
暂无数据
收藏/分享
所有评论 (0)
暂无评论
 

除非特别说明,本系统中所有内容都受版权保护,并保留所有权利。