Understanding Keras Lstm ( Lstm_text_generation.py ) - Ram Memory Issues

I'm diving into LSTM RNN with Keras and Theano backend. While trying to use lstm examples from keras' repo whole code of lstm_text_generation.py on github, I've got one thing that

Solution 1:

There are at least two optimizations in Keras which you could use in order to decrease amount of memory which is need in this case:

1. An Embedding layer which makes it possible to accept only a single integer intead of full one hot vector. Moreover - this layer could be pretrained before the final stage of network training - so you could inject some prior knowledge into your model (and even finetune it during the network fitting).

2. A fit_generator method makes it possible to train a network using a predefinied generator which would produce pairs (x, y) need in network fitting. You could e.g. save the whole dataset to disk and read it part by part using a generator interface.

Of course - both of this methods could be mixed. I think that simplicity was the reason behind this kind of implementation in the example you provided.