Check failed: error == cudaSuccess (2 vs. 0) out of memory


I am trying to run a neural network with pycaffe on gpu.

This works when I call the script for the first time. When I run the same script for the second time, CUDA throws the error in the title.

Batch size is 1, image size at this moment is 243x322, the gpu has 8gb RAM.

I guess I am missing a command that resets the memory?

Thank you very much!


Maybe I should clarify a few things: I am running caffe on windows.

When i call the script with python script.py, the process terminates and the gpu memory gets freed, so this works.

With ipython, which I use for debugging, the GPU memory indeed does not get freed (after one pass, 6 of the 8 bg are in use, thanks for the nvidia-smi suggestion!)

So, what I am looking for is a command I can call from within pyhton, along the lines of:

run network

process image output

<strong>free gpu memory</strong>


Your GPU memory is not getting freed. This happens when the previous process is stopped but not terminated. See my answer <a href="https://stackoverflow.com/a/35748621/3579977" rel="nofollow">here</a>.


This happens when you run out of memory in the GPU. Are you sure you stopped the first script properly? Check the running processes on your system (ps -A in ubuntu) and see if the python script is still running. Kill it if it is. You should also check the memory being used in your GPU (nvidia-smi).


  • How to train Caffe with only G and B channels
  • How to train and test LeNet using caffe using python
  • How to build CLM in android
  • ImportError: dlopen(…) library not open
  • VGG 16/19 Slow Runtimes
  • Tensorflow's asymmetric padding assumptions
  • Caffe: How to get `solver.prototxt` parameters by code?
  • How many images can you pass to Caffe at a time?
  • May i install caffe or caffe2 on Google Colaboratory?
  • scala.MatchError: java.lang.StackOverflowError (of class java.lang.StackOverflowError)
  • Eclipse won't start
  • PE file - what's missing?
  • Cannot convert a char value to money. The char value has incorrect syntax
  • UIImagePickerControllerDelegate Methods Not Called When Delegate Not UIViewController
  • @Autowired for @ModelAttribute
  • Is there a chance to get -splash: work for SWT applications that require -XstartOnFirstThread?
  • Program crashes when run outside IDE
  • Programatically open file in visual studio
  • include dlls in visual studio c++ 2008
  • netsh acl setting (need alternative method - registry settings?)
  • how to set variables in a php include file?
  • R Split data.frame using a column that represents and on/off switch
  • Memory error in python- how to use more memory
  • Center align outputs in ipython notebook
  • How can I enlarge video fullscreen without the affected interface project in as3?
  • How to run “Deployd” on port 80 instead of port 5000 in webserver.
  • Python urlparse: small issue
  • FileReader+canvas image loading problem
  • Recording logins for password protected directories
  • Optimizing database types to compact database (SQLite)
  • JavaScriptCore crash on iOS9
  • ActionScript 2 vs ActionScript 3 performance
  • Build own AppleScript numerical error handling
  • How can I estimate amount of memory left with calling System.gc()?
  • Apache 2.4 - remove | delete | uninstall
  • Why joiner is not used after Sequence generator or Update statergy
  • Exception on Android 4.0 `android.os.StrictMode$AndroidBlockGuardPolicy.onNetwork(StrictMode)`
  • File not found error Google Drive API
  • python draw pie shapes with colour filled
  • Is it possible to post an object from jquery to bottle.py?