Check Failed: Error == Cudasuccess (2 Vs. 0) Out Of Memory

I am trying to run a neural network with pycaffe on gpu. This works when I call the script for the first time. When I run the same script for the second time, CUDA throws the error

Solution 1:

Your GPU memory is not getting freed. This happens when the previous process is stopped but not terminated. See my answer here.

Solution 2:

This happens when you run out of memory in the GPU. Are you sure you stopped the first script properly? Check the running processes on your system (ps -A in ubuntu) and see if the python script is still running. Kill it if it is. You should also check the memory being used in your GPU (nvidia-smi).