50932

How can I estimate amount of memory left with calling System.gc()?

I have some data processing code which uses the following recipe:

    <li>Read in as much data as will fit in memory (call this a 'chunk')</li> <li>Perform processing on the chunk</li> <li>Write out processed chunk to disk</li> <li>Repeat</li> <li>...</li> <li>Merge all the processed chunks to get the final answer.</li> </ul>

    This last stage is most efficient when there are as few chunks as possible, so I want the first stage to read in as much data as will fit in memory. I can do this by querying Runtime.freeMemory().

    However, this means I need to call System.gc(), or the number returned by Runtime.freeMemory() is much smaller than the amount of memory I could safely allocate.

    I have heard a number of authorities say that calling System.gc() explicitly is a bad idea. Is there any way I can avoid this?

    Answer1:

    Even if you call System.gc() right before checking how much memory you have, there is no guarantee that a garbage collection will actually have occurred. Myself I really wouldn't bother, I'd set a fixed chunk size (preferably configured through a property or similar) and always use that. If the rest of your program is simple enough, you can just use the chunk size plus a fixed number of megs as the heap size. If the size of your program is too uncertain due to other reasons, you could look into running two programs side by side and using an IPC mechanism.

    Of course it could well be that your code needs more fine grained control over memory, but I'd humbly suggest you're using the wrong language then; or at least the wrong runtime (there's RT java offerings out there, I assume they're more geared toward this sort of thing).

    I'm sorry if this doesn't seem like the most useful answer, but basically I'm wondering whether you really need this?

    Answer2:

    The reason calling System.gc() is a bad idea is most likely because it does not guarantee anything.

    The <strong>real</strong> reason that calling System.gc() is a bad idea is that the JVM is best at knowing the optimal time to run the GC; i.e. when the heap is full. If you call System.gc() at some other time you are telling the JVM to do something expensive and wasteful.

    Back to the original question, I think that the best solution is to not try to code the application to second guess the memory allocator. Instead, code the application so that the chunk size is command line parameter / system property / whatever, and manually tune the chunk size versus the JVM memory size. You probably also want to ensure that the JVM initial and max memory sizes are the same.

    Answer3:

    Cache the first value of freememory, reuse it and let the VM do the work.

    Answer4:

    Very nice timing. I asked this earlier today and got some useful answers, hope it helps.

    EDIT: this doesn't really answer your question but it refers to the calling the System.gc() not being a good idea.

    Answer5:

    use JConsole or something like that

    Answer6:

    The reason calling System.gc() is a bad idea is most likely because it does not guarantee anything.

    If you really want to be certain that the JVM does garbage collection, you must tell it to. One way is the same way as JConsole, namely through JMX.

    See http://java.sun.com/j2se/1.5.0/docs/guide/management/agent.html#local

    Answer7:

    what about using JMX? In particular the MemoryMXbean:

    MemoryMXBean memoryMXBean = ManagementFactory.getMemoryMXBean();

    check also the MemoryUsage class.

    Answer8:

    The JVM Tool INterface (jvmti) has a method named ForceGarbageCollection. You could write some JNI to call it.

    something like

    #include "jvmti.h" #include "jni.h" jvmtiEnv *jvmti; JNIEXPORT jint JNICALL Agent_OnLoad(JavaVM *vm, char *options, void *reserved) { (*vm)->GetEnv(vm, (void **)&jvmti, JVMTI_VERSION_1); return JNI_OK; } JNIEXPORT void JNICALL my_managled_function_name_that_is_entirely_too_long_to_be_easy_to_use (JNIEnv *env) { error = (*jvmti)->ForceGarbageCollection(jvmti); // you can trap the error if you want; }

    BTW, this is a bad idea. I only use this code for debugging (to ensure that certain class, like listeners, have no more reachable references.)

    My bet is that the VM will gc all possible data before throwing Memory Errors.

Recommend

  • about the braces in case statement in switch
  • How to communicate two android applications?
  • IRC related help
  • jsTree show/hide nodes
  • Eclipse > Javascript > Code highlighting not working with Object Notation
  • Can a MEF application built on .NET 4 import types built against .NET 3.5?
  • How does CoreLocation locate the device?
  • How to set infinite shared access signature policy in azure?
  • Manually Timing out a C# Thread
  • What is the default HTTP verb in WebApi ? GET or POST?
  • Java Application vs. Java Desktop Application in Netbeans [duplicate]
  • Is it possible to get the word under the mouse cursor in a ``?
  • Can my PDF ping my server when it is opened?
  • import scipy.sparse failed
  • Access user's phone number on iOS 7
  • Center align outputs in ipython notebook
  • GAE: Way to get reference to an HttpSession from its ID?
  • Ember.js model to be organised as a tree structure
  • Redirect STDERR in OPEN pipe comand. Perl Linux
  • Cannot upload to OneDrive using the new SDK
  • Differences in dis-assembled C code of GCC and Borland?
  • Graphics.CopyFromScreen [Web application] + The handle is invalid
  • nonblocking BIO_do_connect blocked when there is no internet connected
  • Android application: how to use the camera and grab the image bytes?
  • How to run “Deployd” on port 80 instead of port 5000 in webserver.
  • Jackson Parser: ignore deserializing for type mismatch
  • Rails Find when some params will be blank
  • OpenGL ES texture problem, 4 duplicate columns and horizontal lines (Android)
  • Lost migrations and Azure database is now out of sync
  • Get one-time binding to work for ng-if
  • Dynamically accessing properties of knockoutjs observable array
  • Jenkins: How To Build multiple projects from a TFS repository?
  • Deselecting radio buttons while keeping the View Model in synch
  • Bug in WPF DataGrid
  • Sony Xperia Z Tablet not found by adb
  • How to recover from a Spring Social ExpiredAuthorizationException
  • ActionScript 2 vs ActionScript 3 performance
  • jQuery tmpl and DataLink beta
  • Apache 2.4 - remove | delete | uninstall
  • Hits per day in Google Big Query