Conference paper

BENEŠ Karel, BASKAR Murali K. and BURGET Lukáš. Residual Memory Networks in Language Modeling: Improving the Reputation of Feed-Forward Networks. In: Proceedings of Interspeeech 2017. Stockholm: International Speech Communication Association, 2017, pp. 284-288. ISSN 1990-9772. Available from:
Publication language:english
Original title:Residual Memory Networks in Language Modeling: Improving the Reputation of Feed-Forward Networks
Title (cs):Sítě s reziduální pamětí pro jazykové modelování: zlepšení reputace dopředných sítí
Proceedings:Proceedings of Interspeeech 2017
Conference:Interspeech 2017
Place:Stockholm, SE
Journal:Proceedings of Interspeech, Vol. 2017, No. 08, FR
Publisher:International Speech Communication Association
+Type Name Title Size Last modified
iconbenes_interspeech2017_IS171442.pdf288 KB2018-01-02 18:51:12
^ Select all
With selected:
residual memory networks, feed-forward networks, language modeling
The paper describes the residual memory networks in language modeling: Improving the Reputation of Feed-Forward Networks.
We introduce the Residual Memory Network (RMN) architecture to language modeling. RMN is an architecture of feedforward neural networks that incorporates residual connections and time-delay connections that allow us to naturally incorporate information from a substantial time context. As this is the first time RMNs are applied for language modeling, we thoroughly investigate their behaviour on the well studied Penn Treebank corpus. We change the model slightly for the needs of language modeling, reducing both its time and memory consumption. Our results show that RMN is a suitable choice for small-sized neural language models: With test perplexity 112.7 and as few as 2.3M parameters, they out-perform both a much larger vanilla RNN (PPL 124, 8M parameters) and a similarly sized LSTM (PPL 115, 2.08M parameters), while being only by less than 3 perplexity points worse than twice as big LSTM.
   author = {Karel Bene{\v{s}} and K. Murali Baskar and Luk{\'{a}}{\v{s}}
   title = {Residual Memory Networks in Language Modeling: Improving the
	Reputation of Feed-Forward Networks},
   pages = {284--288},
   booktitle = {Proceedings of Interspeeech 2017},
   journal = {Proceedings of Interspeech},
   volume = {2017},
   number = {08},
   year = {2017},
   location = {Stockholm, SE},
   publisher = {International Speech Communication Association},
   ISSN = {1990-9772},
   language = {english},
   url = {}

Your IPv4 address:
Switch to IPv6 connection

DNSSEC [dnssec]