Neural Language Models 2015


Notes:

  • Bigram neural language model
  • Class-based neural language model
  • Conditional neural language model
  • Context-aware neural language model
  • Factored neural language model
  • Feature-based neural language model
  • Hierarchal neural language model
  • Log-bilinear neural language model
  • Multimodal neural language model
  • Neural language modeLing
  • Neural language modeLLing
  • Neural network language model
  • Probabilistic neural language model
  • Recurrent neural language model
  • Structure-content neural language model

Resources:

  • rwthlm .. a toolkit for feedforward and long short-term memory neural network language modeling

Wikipedia:

  • Convolutional neural network .. a type of feed-forward artificial neural network, and variation of multilayer perceptron, designed to minmize preprocessing.

See also:

100 Best Deep Learning Videos | 100 Best GitHub: Deep Learning | Deep Learning & Dialog Systems | DNLP (Deep Natural Language Processing)Natural Language Image Recognition | SkipGrams 2013 | Word2vec Neural Network


Character-aware neural language models Y Kim, Y Jernite, D Sontag, AM Rush – arXiv preprint arXiv:1508.06615, 2015 – arxiv.org Abstract: We describe a simple neural language model that relies only on character-level inputs. Predictions are still made at the word-level. Our model employs a convolutional neural network (CNN) over characters, whose output is given to a long short-term memory … Cited by 87 Related articles All 10 versions

Hierarchical neural language models for joint representation of streaming documents and their content N Djuric, H Wu, V Radosavljevic, M Grbovic… – Proceedings of the 24th …, 2015 – dl.acm.org Abstract We consider the problem of learning distributed representations for documents in data streams. The documents are represented as low-dimensional vectors and are jointly learned with distributed vector representations of word tokens using a hierarchical … Cited by 21 Related articles All 9 versions

Scaling recurrent neural network language models W Williams, N Prasad, D Mrva, T Ash… – … on Acoustics, Speech …, 2015 – ieeexplore.ieee.org ABSTRACT This paper investigates the scaling properties of Recurrent Neural Network Language Models (RNNLMs). We discuss how to train very large RNNs on GPUs and address the questions of how RNNLMs scale with respect to model size, training-set size, … Cited by 20 Related articles All 5 versions

Strategies for Training Large Vocabulary Neural Language Models W Chen, D Grangier, M Auli – arXiv preprint arXiv:1512.04906, 2015 – arxiv.org Abstract: Training neural network language models over large vocabularies is still computationally very costly compared to count-based models such as Kneser-Ney. At the same time, neural language models are gaining popularity for many applications such as … Cited by 8 Related articles All 3 versions

Recurrent neural network language model training with noise contrastive estimation for speech recognition X Chen, X Liu, MJF Gales… – 2015 IEEE International …, 2015 – ieeexplore.ieee.org ABSTRACT In recent years recurrent neural network language models (RNNLMs) have been successfully applied to a range of tasks including speech recognition. However, an important issue that limits the quantity of data used, and their possible application areas, is … Cited by 22 Related articles All 9 versions

Recurrent neural network language model adaptation for multi-genre broadcast speech recognition X Chen, T Tan, X Liu, P Lanchantin… – Proceedings of …, 2015 – pdfs.semanticscholar.org Abstract Recurrent neural network language models (RNNLMs) have recently become increasingly popular for many applications including speech recognition. In previous research RNNLMs have normally been trained on well-matched in-domain data. The … Cited by 14 Related articles All 6 versions

Improving the training and evaluation efficiency of recurrent neural network language models X Chen, X Liu, MJF Gales… – 2015 IEEE International …, 2015 – ieeexplore.ieee.org ABSTRACT Recurrent neural network language models (RNNLMs) are be coming increasingly popular for speech recognition. Previously, we have shown that RNNLMs with a full (non-classed) output layer (F-RNNLMs) can be trained efficiently using a GPU giving a … Cited by 11 Related articles All 10 versions

A primer on neural network models for natural language processing Y Goldberg – arXiv preprint arXiv:1510.00726, 2015 – arxiv.org Abstract: Over the past few years, neural networks have re-emerged as powerful machine- learning models, yielding state-of-the-art results in fields such as image recognition and speech processing. More recently, neural network models started to be applied also to … Cited by 27 Related articles All 8 versions

Recurrent neural network language model adaptation with curriculum learning Y Shi, M Larson, CM Jonker – Computer Speech & Language, 2015 – Elsevier Abstract This paper addresses the issue of language model adaptation for Recurrent Neural Network Language Models (rnnlm s), which have recently emerged as a state-of-the-art method for language modeling in the area of speech recognition. Curriculum learning is … Cited by 7 Related articles All 3 versions

FPGA acceleration of recurrent neural network based language model S Li, C Wu, H Li, B Li, Y Wang… – … Machines (FCCM), 2015 …, 2015 – ieeexplore.ieee.org Abstract—Recurrent neural network (RNN) based language model (RNNLM) is a biologically inspired model for natural language processing. It records the historical information through additional recurrent connections and therefore is very effective in … Cited by 6 Related articles All 8 versions

Texts in, meaning out: neural language models in semantic similarity task for Russian A Kutuzov, I Andreev – arXiv preprint arXiv:1504.08183, 2015 – arxiv.org Abstract: Distributed vector representations for natural language vocabulary get a lot of attention in contemporary computational linguistics. This paper summarizes the experience of applying neural network language models to the task of calculating semantic similarity … Cited by 5 Related articles All 3 versions

Bidirectional recurrent neural network language models for automatic speech recognition E Arisoy, A Sethy, B Ramabhadran… – 2015 IEEE International …, 2015 – ieeexplore.ieee.org ABSTRACT Recurrent neural network language models have enjoyed great success in speech recognition, partially due to their ability to model longer-distance context than word n- gram models. In recurrent neural networks (RNNs), contextual information from past inputs … Cited by 7 Related articles

The fixed-size ordinally-forgetting encoding method for neural network language models S Zhang, H Jiang, M Xu, J Hou, L Dai – Proceedings of ACL, 2015 – anthology.aclweb.org Abstract In this paper, we propose the new fixedsize ordinally-forgetting encoding (FOFE) method, which can almost uniquely encode any variable-length sequence of words into a fixed-size representation. FOFE can model the word order in a sequence using a simple … Cited by 8 Related articles All 7 versions

Pre-computable multi-layer neural network language models J Devlin, C Quirk, A Menezes – … Methods in Natural Language …, 2015 – emnlp2015.org Abstract In the last several years, neural network models have significantly improved accuracy in a number of NLP tasks. However, one serious drawback that has impeded their adoption in production systems is the slow runtime speed of neural network models … Cited by 4 Related articles All 9 versions

Multi-language image description with neural sequence models D Elliott, S Frank, E Hasler – arXiv preprint arXiv:1510.04709, 2015 – arxiv.org Abstract: In this paper we present an approach to multi-language image description bringing together insights from neural machine translation and neural image description. To create a description of an image for a given target language, our sequence generation models … Cited by 10 Related articles All 4 versions

Investigations on phrase-based decoding with recurrent neural network language and translation models T Alkhouli, F Rietig, H Ney – Proc. WMT, 2015 – aclweb.org Abstract This work explores the application of recurrent neural network (RNN) language and translation models during phrasebased decoding. Due to their use of unbounded context, the decoder integration of RNNs is more challenging compared to the integration of … Cited by 6 Related articles All 15 versions

Unnormalized exponential and neural network language models A Sethy, S Chen, E Arisoy… – 2015 IEEE International …, 2015 – ieeexplore.ieee.org ABSTRACT Model M, an exponential class-based language model, and neu ral network language models (NNLM’s) have outperformed word n-gram language models over a wide range of tasks. However, these gains come at the cost of vastly increased computation … Cited by 4 Related articles All 3 versions

Paraphrastic recurrent neural network language models X Liu, X Chen, MJF Gales… – 2015 IEEE International …, 2015 – ieeexplore.ieee.org ABSTRACT Recurrent neural network language models (RNNLM) have become an increasingly popular choice for state-of-the-art speech recognition systems. Linguistic factors influencing the realization of surface word sequences, for example, expressive richness, … Cited by 4 Related articles All 9 versions

Dependency recurrent neural language models for sentence completion P Mirowski, A Vlachos – arXiv preprint arXiv:1507.01193, 2015 – arxiv.org Abstract: Recent work on language modelling has shifted focus from count-based models to neural models. In these works, the words in each sentence are always considered in a left-to- right order. In this paper we show how we can improve the performance of the recurrent … Cited by 6 Related articles All 15 versions

Auto-Sizing Neural Networks: With Applications to n-gram Language Models K Murray, D Chiang – arXiv preprint arXiv:1508.05051, 2015 – arxiv.org Abstract: Neural networks have been shown to improve performance across a range of natural-language tasks. However, designing and training them can be complicated. Frequently, researchers resort to repeated experimentation to pick optimal settings. In this … Cited by 5 Related articles All 12 versions

Recurrent neural network language model with structured word embeddings for speech recognition T He, X Xiang, Y Qian, K Yu – 2015 IEEE International …, 2015 – ieeexplore.ieee.org ABSTRACT Due to effective word context encoding and long-term context preserving, recurrent neural network language model (RNNLM) has attracted great interest by showing better performance over back-off n-gram models and feed-forward neural network … Cited by 3 Related articles

Discriminative method for recurrent neural network language models Y Tachioka, S Watanabe – 2015 IEEE International Conference …, 2015 – ieeexplore.ieee.org ABSTRACT A recurrent neural network language model (RNN-LM) can use a long word context more than can an n-gram language model, and its effective has recently been shown in its accomplishment of auto matic speech recognition (ASR) tasks. However, the training … Cited by 4 Related articles All 8 versions

Evaluation of neural network language models in handwritten Chinese text recognition YC Wu, F Yin, CL Liu – Document Analysis and Recognition ( …, 2015 – ieeexplore.ieee.org Abstract-Handwritten Chinese text recognition based on over-segmentation and path search integrating contexts has been demonstrated successful, where language models play an impor tant role. Recently, neural network language models (NNLMs) have shown … Cited by 2 Related articles All 3 versions

Prosodically-enhanced recurrent neural network language models SR Gangireddy, S Renals… – … Annual Conference of …, 2015 – homepages.inf.ed.ac.uk Abstract Recurrent neural network language models have been shown to consistently reduce the word error rates (WERs) of large vocabulary speech recognition tasks. In this work we propose to enhance the RNNLMs with prosodic features computed using the … Cited by 3 Related articles All 5 versions

A Fixed-Size Encoding Method for Variable-Length Sequences with its Application to Neural Network Language Models S Zhang, H Jiang, M Xu, J Hou, LR Dai – arXiv preprint arXiv:1505.01504, 2015 – arxiv.org Abstract: In this paper, we propose the new fixed-size ordinally-forgetting encoding (FOFE) method, which can almost uniquely encode any variable-length sequence of words into a fixed-size representation. FOFE can model the word order in a sequence using a simple … Cited by 2 Related articles All 5 versions

Random walks and neural network language models on knowledge bases J Goikoetxea, A Soroa, E Agirre… – … : Human Language …, 2015 – aclweb.org Abstract Random walks over large knowledge bases like WordNet have been successfully used in word similarity, relatedness and disambiguation tasks. Unfortunately, those algorithms are relatively slow for large repositories, with significant memory footprints. In … Cited by 4 Related articles All 5 versions

Bag-of-Words Input for Long History Representation in Neural Network-Based Language Models for Speech Recognition K Irie, R Schlüter, H Ney – Proceedings of …, 2015 – www-i6.informatik.rwth-aachen.de Abstract In most of previous works on neural network based language models (NNLMs), the words are represented as 1-of-N encoded feature vectors. In this paper we investigate an alternative encoding of the word history, known as bag-of-words (BOW) representation of … Cited by 3 Related articles All 3 versions

A Bidirectional Recurrent Neural Language Model for Machine Translation A Peris, F Casacuberta – Procesamiento del Lenguaje Natural, 2015 – journal.sepln.org Resumen A language model based in continuous representations of words is presented, which has been applied to a statistical machine translation task. This model is implemented by means of a bidirectional recurrent neural network, which is able to take into account … Cited by 1 Related articles All 6 versions

Online Representation Learning in Recurrent Neural Language Models M Rei – arXiv preprint arXiv:1508.03854, 2015 – arxiv.org Abstract: We investigate an extension of continuous online learning in recurrent neural network language models. The model keeps a separate vector representation of the current unit of text being processed and adaptively adjusts it after each prediction. The initial … Cited by 1 Related articles All 12 versions

Enhanced Word Embeddings from a Hierarchical Neural Language Model X Wang, K Sudoh, M Nagata – … of the 24th ACM International on …, 2015 – kecl.ntt.co.jp ABSTRACT This paper proposes a neural language model to capture the interaction of text units of different levels, ie., documents, paragraphs, sentences, words in an hierarchical structure. At each paralleled level, the model incorporates Markov property while each … Related articles All 2 versions

Deep Neural Language Models for Machine Translation MT Luong, M Kayser, CD Manning – CoNLL 2015, 2015 – aclweb.org Abstract Neural language models (NLMs) have been able to improve machine translation (MT) thanks to their ability to generalize well to long contexts. Despite recent successes of deep neural networks in speech and vision, the general practice in MT is to incorporate … Cited by 1 Related articles All 16 versions

Domain adaptation in semantic role labeling using a neural language model and linguistic resources QTN Do, S Bethard, MF Moens – … , Speech, and Language …, 2015 – ieeexplore.ieee.org Abstract—We propose a method for adapting Semantic Role Labeling (SRL) systems from a source domain to a target domain by combining a neural language model and linguistic resources to generate additional training examples. We primarily aim to improve the … Cited by 2 Related articles All 4 versions

Neural Network Language Model for Chinese Pinyin Input Method Engine SY Chen, R Wang, H Zhao – 2015 – pdfs.semanticscholar.org Abstract Neural network language models (NNLMs) have been shown to outperform traditional n-gram language model. However, too high computational cost of NNLMs becomes the main obstacle of directly integrating it into pinyin IME that normally requires a … Related articles All 8 versions

Myanmar Language Speech Recognition with Hybrid Artificial Neural Network and Hidden Markov Model TT Nwe, T Myint – urst.org Abstract: There are many artificial intelligence approaches used in the development of Automatic Speech Recognition (ASR), hybrid approach is one of them. The common hybrid method in speech recognition is the combination of Artificial Neural Network (ANN) and … Related articles

Personalizing a Universal Recurrent Neural Network Language Model with User Characteristic Features by Crowdsouring over Social Networks BH Tseng, HY Lee, LS Lee – arXiv preprint arXiv:1506.01192, 2015 – arxiv.org Abstract: With the popularity of mobile devices, personalized speech recognizer becomes more realizable today and highly attractive. Each mobile device is primarily used by a single user, so it’s possible to have a personalized recognizer well matching to the … Related articles All 3 versions

Recurrent neural network language model for English-Indonesian Machine Translation: Experimental study A Hermanto, TB Adji… – … Conference on Science in …, 2015 – ieeexplore.ieee.org Abstract—At recent time, the statistical based language model and neural based language model are still dominating the researches in the field of machine translation. The statistical based machine translation today is the fastest one but it has a weakness in term of … Related articles

Parameterized Neural Network Language Models for Information Retrieval B Piwowarski, S Lamprier, N Despres – arXiv preprint arXiv:1510.01562, 2015 – arxiv.org Abstract: Information Retrieval (IR) models need to deal with two difficult issues, vocabulary mismatch and term dependencies. Vocabulary mismatch corresponds to the difficulty of retrieving relevant documents that do not contain exact query terms but semantically … Related articles All 4 versions

Deep neural network language model research and application overview FL Yin, XY Pan, XW Liu, HX Liu – Wavelet Active Media …, 2015 – ieeexplore.ieee.org Abstract: Research of the neural network language model in NLP is reviewed. In this paper, the neural network language models are classified into early shallow language models and deep neural network models based on deep learning. This paper emphatically introduces …

Integrating meta-information into recurrent neural network language models Y Shi, M Larson, J Pelemans, CM Jonker… – Speech …, 2015 – Elsevier Abstract Due to their advantages over conventional n-gram language models, recurrent neural network language models (rnnlm s) recently have attracted a fair amount of research attention in the speech recognition community. In this paper, we explore one advantage of … Related articles All 7 versions

Investigation of back-off based interpolation between recurrent neural network and n-gram language models X Chen, X Liu, MJF Gales… – 2015 IEEE Workshop on …, 2015 – ieeexplore.ieee.org ABSTRACT Recurrent neural network language models (RNNLMs) have become an increasingly popular choice for speech and language processing tasks including automatic speech recognition (ASR). As the generalization patterns of RNNLMs and n-gram LMs are … Cited by 1 Related articles All 8 versions

Integrating prosodic information into recurrent neural network language model for speech recognition T Fu, Y Han, X Li, Y Liu, X Wu – 2015 Asia-Pacific Signal and …, 2015 – ieeexplore.ieee.org Abstract—Prosody is a kind of cues that are critical to human speech perception and comprehension, so it is plausible to integrate prosodic information into machine speech recognition. However, as a result of the supra-segmental nature, it is hard to integrate … Related articles All 2 versions

Scalable Recurrent Neural Network Language Models for Speech Recognition X Chen – interspeech2015.org Abstract Language model is a vital part in modern state-ofart ASR systems. N-Gram LMs have been dominating the area of language modelling during last several decades. Recently, recurrent neural network language models (RNNLMs) present promising … Related articles

On Efficient Training of Word Classes and Their Application to Recurrent Neural Network Language Models R Botros, K Irie, M Sundermeyer… – Sixteenth …, 2015 – www-i6.informatik.rwth-aachen.de Abstract In this paper, we investigated various word clustering methods, by studying two clustering algorithms: Brown clustering and exchange algorithm, and three objective functions derived from different class-based language models (CBLM): two-sided, … Cited by 3 Related articles All 3 versions

Morphological Analysis for Unsegmented Languages using Recurrent Neural Network Language Model H Morita, D Kawahara, S Kurohashi – 2015 – aclweb.org Abstract We present a new morphological analysis model that considers semantic plausibility of word sequences by using a recurrent neural network language model (RNNLM). In unsegmented languages, since language models are learned from … Cited by 2 Related articles All 9 versions

Input Dimension Reduction based on Continuous Word Vector for Deep Neural Network Language Model KH Kim, D Lee, M Lim, JH Kim – Journal of the Korean society of …, 2015 – koreascience.or.kr Abstract In this paper, we investigate an input dimension reduction method using continuous word vector in deep neural network language model. In the proposed method, continuous word vectors were generated by using Google’s Word2Vec from a large training corpus to … Cited by 1

Forecasting language test performance with a back propagation neural network model M Dong – Natural Computation (ICNC), 2015 11th International …, 2015 – ieeexplore.ieee.org Abstract—Though researchers agree that test anxiety is one of the most debilitating factors that deteriorates the test performance of test-takers who are often anxious during examinations, it is generally neglected when calculating test scores by both classical and … Related articles

Blackout: Speeding Up Recurrent Neural Net-Work Language Models With Very Large Vo S Ji, SVN Vishwanathan, N Satish, MJ Anderson… – pdfs.semanticscholar.org ABSTRACT We propose BlackOut, an approximation algorithm to efficiently train massive recurrent neural network language models (RNNLMs) with million word vocabularies. BlackOut is motivated by using a discriminative loss, and we describe a weighted … Related articles

Personalizing universal recurrent neural network language model with user characteristic features by social network crowdsourcing BH Tseng, H Lee, LS Lee – 2015 IEEE Workshop on Automatic …, 2015 – ieeexplore.ieee.org ABSTRACT With the popularity of mobile devices, personalized speech recognizer becomes more realizable today and highly attractive. Each mobile device is primarily used by a single user, so it’s possible to have a personalized recognizer well matching to the … Cited by 1 Related articles All 2 versions

A Cognitive Neural Model of Executive Functions in Natural Language Processing B Golosio, A Cangelosi, O Gamotina… – Procedia Computer …, 2015 – Elsevier Abstract Although extensive research has been devoted to cognitive models of human language, the role of executive functions in language processing has little been explored. In this work we present a neural-network-based cognitive architecture which models the … Related articles All 2 versions

Simulating intervention to support compensatory strategies in an artificial neural network model of atypical language development J Yang, MSC Thomas – Citeseer Abstract Artificial neural networks have been used to model developmental deficits in cognitive and language development, most often by including sub-optimal inputoutput representations or computational parameters in these learning systems. The next step is … Related articles All 4 versions

BlackOut: Speeding up Recurrent Neural Network Language Models With Very Large Vocabularies S Ji, SVN Vishwanathan, N Satish… – arXiv preprint arXiv: …, 2015 – arxiv.org Abstract: We propose BlackOut, an approximation algorithm to efficiently train massive recurrent neural network language models (RNNLMs) with million word vocabularies. BlackOut is motivated by using a discriminative loss, and we describe a new sampling … Cited by 6 Related articles All 2 versions

Semantic Memory Association, Procedural Grammar Syntax and Episodic Modality Coordination as Three Interactive Neural Processes Organizing Language: A Model ZJ Cai – 2015 – researchgate.net Abstract In the declarative/procedural model of language, it assumes the semantic words as declarative memories while the grammatical syntax as procedural rules, whereas it is herein suggested that different words associate with different cortical modalities, so that it is … Related articles

A real-time American Sign Language word recognition system based on neural networks and a probabilistic model N Sarawate, MC Leu, C ÖZ – Turkish Journal of Electrical …, 2015 – journals.tubitak.gov.tr Abstract: The development of an American Sign Language (ASL) word recognition system based on neural networks and a probabilistic model is presented. We use a CyberGlove and a Flock of Birds motion tracker to extract the gesture data. The finger joint angle data … Related articles All 4 versions