Conference paper

 
Mikolov, T., Deoras, A., Povey, D., Burget, L., Cernocký, J.: Strategies for Training Large Scale Neural Network Language Models, In: Proceedings of ASRU 2011, Hilton Waikoloa Village, Big Island, Hawaii, US, IEEESP, 2011, p. 196-201, ISBN 978-1-4673-0366-8
Publication language:english
Original title:Strategies for Training Large Scale Neural Network Language Models
Title (cs):Strategie pro trénování velkých jazykových modelu zalozených na neuronových sítích
Pages:196-201
Proceedings:Proceedings of ASRU 2011
Conference:IEEE 2011 Workshop on Automatic Speech Recognition and Understanding
Place:Hilton Waikoloa Village, Big Island, Hawaii, US
Year:2011
ISBN:978-1-4673-0366-8
Publisher:IEEE Signal Processing Society
URL:http://www.fit.vutbr.cz/research/groups/speech/publi/2011/mikolov_asru2011_00196.pdf [PDF]
Keywords
recurrent neural network, language model, speech recognition, maximum entropy
Annotation
Techniques for effective training of recurrent neural network based language models are described, and new state-of-the-art results on standard speech recognition task are reported.
Abstract
We describe how to effectively train neural network based language models on large data sets. Fast convergence during training and better overall performance is observed when the training data are sorted by their relevance. We introduce hash-based implementation of a maximum entropy model, that can be trained as a part of the neural network model. This leads to significant reduction of computational complexity. We achieved around 10% relative reduction of word error rate on English Broadcast News speech recognition task, against large 4-gram model trained on 400M tokens.
BibTeX:
@INPROCEEDINGS{
   author = {Tomás Mikolov and Anoop Deoras and Daniel Povey and Lukás
	Burget and Jan Cernocký},
   title = {Strategies for Training Large Scale Neural Network Language
	Models},
   pages = {196--201},
   booktitle = {Proceedings of ASRU 2011},
   year = {2011},
   location = {Hilton Waikoloa Village, Big Island, Hawaii, US},
   publisher = {IEEE Signal Processing Society},
   ISBN = {978-1-4673-0366-8},
   language = {english},
   url = {http://www.fit.vutbr.cz/research/view_pub.php?id=9775}
}