Memory Not Getting Reclaimed On Restart Of A Process

I have a python job that runs a caffe net for image processing on nvidia GPUs. The job takes images from a rabbitmq queue, processes it and then writes the result in another queue.

Solution 1:

It's your GPU memory which is not getting freed. Get the process id

$ nvidia-smi

and then

$ kill -9 <process id>