Keras Exit Code -1073741819 (0xc0000005) After Running Training 2 Models

I use Pycharm to run my script. I have a script that loops. Each loop: 1. Select a dataset. 2. Trains a new Keras model. 3. Evaluate that model. So the code works perfectly for 2 w

Solution 1:

Below is the explanation for the things I suggested in the comments that worked, in case anyone faces the same issue.

Manually setting session for keras rather than using the default one at the start of each loop.

sess = tf.Session()  
K.set_session(sess) 
#..... train your model
K.clear_session()

Deleting loader variable as this object must be having reference to the original model object as I can see you are calling the train() on it.

Explicitly collecting all the memory released by deleting these the variable using gc.collect() after each loop so that we have enough memory for building our new model.

So, the gist is when running multiple independent model in a loop like this make sure you have explicitly set the tensorflow session so that you can clear this session after loop finishes, releasing all the resources uses by this session. Delete all the references that might be tied to tensorflow objects in that loop and collect the free memory.