Publication Details

i-vectors in language modeling: An efficient way of domain adaptation for feed-forward models

BENEŠ Karel, KESIRAJU Santosh and BURGET Lukáš. i-vectors in language modeling: An efficient way of domain adaptation for feed-forward models. In: Proceedings of Interspeech 2018. Hyderabad: International Speech Communication Association, 2018, pp. 3383-3387. ISSN 1990-9772. Available from: https://www.isca-speech.org/archive/Interspeech_2018/abstracts/1070.html
Czech title
i-vektory pro jazykové modelování: efektivní způsob doménové adaptace s dopřednými modely
Type
conference paper
Language
english
Authors
Beneš Karel, Ing. (DCGM FIT BUT)
Kesiraju Santosh (DCGM FIT BUT)
Burget Lukáš, doc. Ing., Ph.D. (DCGM FIT BUT)
URL
Keywords

language modeling, feed-forward models, subspace multinomial model, domain adaptation

Abstract

We show an effective way of adding context information to shallow neural language models. We propose to use Subspace Multinomial Model (SMM) for context modeling and we add the extracted i-vectors in a computationally efficient way. By adding this information, we shrink the gap between shallow feed-forward network and an LSTM from 65 to 31 points of perplexity on the Wikitext-2 corpus (in the case of neural 5-gram model). Furthermore, we show that SMM i-vectors are suitable for domain adaptation and a very small amount of adaptation data (e.g. endmost 5% of a Wikipedia article) brings a substantial improvement. Our proposed changes are compatible with most optimization techniques used for shallow feedforward LMs.

Published
2018
Pages
3383-3387
Journal
Proceedings of Interspeech - on-line, vol. 2018, no. 9, ISSN 1990-9772
Proceedings
Proceedings of Interspeech 2018
Conference
Interspeech Conference, Hyderabad, India, IN
Publisher
International Speech Communication Association
Place
Hyderabad, IN
DOI
UT WoS
000465363900706
EID Scopus
BibTeX
@INPROCEEDINGS{FITPUB11842,
   author = "Karel Bene\v{s} and Santosh Kesiraju and Luk\'{a}\v{s} Burget",
   title = "i-vectors in language modeling: An efficient way of domain adaptation for feed-forward models",
   pages = "3383--3387",
   booktitle = "Proceedings of Interspeech 2018",
   journal = "Proceedings of Interspeech - on-line",
   volume = 2018,
   number = 9,
   year = 2018,
   location = "Hyderabad, IN",
   publisher = "International Speech Communication Association",
   ISSN = "1990-9772",
   doi = "10.21437/Interspeech.2018-1070",
   language = "english",
   url = "https://www.fit.vut.cz/research/publication/11842"
}
Back to top