57152

Node Async Looping - Why does the memory grow and just drop all of the sudden?

Question:

I'm writing a program in Node that uses an async loop. The goal is to get this program to run on Heroku for an extended period of time. It grows in memory, as expected. But then once the memory usage hits about 57MiB, it drops back down to 22MiB (where it started). What causes the memory usage to drop out of nowhere like that?

Here is my code, if it helps at all. database.read is just a simplification of http.request.

var http = require("http"); var util = require('util'); var fnstraj = require("./predictors/fnstraj.js"); var database = require("./library/database.js"); var COUNT = 0; //////////////// // Queue Loop // //////////////// var worker = function() { setTimeout(function() { COUNT++; console.log("Worker Clock: " + COUNT + "."); console.log(util.inspect(process.memoryUsage())); database.read('/queue/', function( results, error ) { if ( typeof error !== "undefined" && error ) { process.nextTick( worker ); } else { var queue = results; database.read('/flights/', function ( results, error ) { if ( typeof error !== "undefined" && error ) { process.nextTick( worker ); } else { var flights = results; if ( !flights.error && typeof queue.rows[0] !== "undefined" ) { for ( flight in flights.rows ) { if ( flights.rows[flight].doc._id === queue.rows[0].doc._id ) { var thisFlight = flights.rows[flight].doc; console.log("Flight " + thisFlight._id + " started"); thisFlight.duration = fnstraj.vertPred(thisFlight.launch.altitude, thisFlight.balloon.burst, thisFlight.balloon.radius, thisFlight.balloon.lift); fnstraj.predict(thisFlight, function() { database.remove('/queue/' + thisFlight._id); console.log("Flight " + thisFlight._id + " completed"); process.nextTick( worker ); }); var found = true; } } if ( !found ) { process.nextTick( worker ); } } } }); } }); }, 25); };

Answer1:

This is related to the V8 Garbage collection. You may tweak it with the '--gc_interval ' option of node but note that this is an advanced parameter.

node --gc_interval <allocation count interval>

This may also be related to the heap compression of the GC. Tis is the process of gathering all the previously freed space and eventually give is back to the Operating System.

for more tweaks, you can experiments with V8 specific options:

node --v8-options

Recommend

  • Android FileProvider delete file
  • Celery + Redis - .get() hangs indefinitely after running smoothly for ~70 hours
  • why lock_guard can get an already locked mutex by unique_lock?
  • Client-Side: Accessing Windows Azure Drive?
  • Seamless scrolling text
  • IE8 stops network access after 5 long polling request
  • setTimeOut and local function
  • Error “Reflect.defineMetadata” while trying to load a transient web worker
  • Python: sending key press events over SSH
  • Spring Integration Bridge with poller not working as expected for JMS
  • Android Database Error - getWriteableDatabase
  • Using Generics on right hand side in Java 6?
  • How to make Twilio api Post request with the help of AFNetworking?
  • Using Sax parsing to edit and write XML in VB6
  • What is the purpose of TaskExecutor in spring?
  • how to find common suffix in java by using method
  • rspec simple example getting error on request variable in integration test
  • Mysterious problem with floating point in LISP - time axis generation
  • How to know when stdin is empty if it contains EOF?
  • Combining SpatialPolygonsDataFrame of two neighbour countries
  • How to attach a node.js readable stream to a Sendgrid email?
  • Exception “firebase.functions() takes … no argument …” when specifying a region for a Cloud Function
  • Django: Count of Group Elements
  • Spray.io: When (not) to use non-blocking route handling?
  • Menu Color Fade on Hover with Jquery
  • Does CUDA 5 support STL or THRUST inside the device code?
  • Sending data from AppleScript to FileMaker records
  • MySQL WHERE-condition in procedure ignored
  • Timeout for blocking function call, i.e., how to stop waiting for user input after X seconds?
  • Why winpcap requires both .lib and .dll to run?
  • How to delete a row from a dynamic generate table using jquery?
  • using HTMLImports.whenReady not working in chrome
  • Authorize attributes not working in MVC 4
  • EntityFramework adding new object to nested object collection
  • reshape alternating columns in less time and using less memory
  • Reading document lines to the user (python)
  • Binding checkboxes to object values in AngularJs
  • Net Present Value in Excel for Grouped Recurring CF
  • jQuery Masonry / Isotope and fluid images: Momentary overlap on window resize
  • How to load view controller without button in storyboard?