13850

Continuously INFO JobScheduler:59 - Added jobs for time *** ms in my Spark Standalone Cluster

Question:

We are working with Spark Standalone Cluster with 8 Cores and 32GB Ram, with 3 nodes cluster with same configuration.

Some times streaming batch completed in less than 1sec. some times it takes more than 10 secs at that time below log will appears in console.

2016-03-29 11:35:25,044 INFO TaskSchedulerImpl:59 - Removed TaskSet 18.0, whose tasks have all completed, from pool 2016-03-29 11:35:25,044 INFO DAGScheduler:59 - Job 18 finished: foreachRDD at EventProcessor.java:87, took 1.128755 s 2016-03-29 11:35:31,471 INFO JobScheduler:59 - Added jobs for time 1459231530000 ms 2016-03-29 11:35:35,004 INFO JobScheduler:59 - Added jobs for time 1459231535000 ms 2016-03-29 11:35:40,004 INFO JobScheduler:59 - Added jobs for time 1459231540000 ms 2016-03-29 11:35:45,136 INFO JobScheduler:59 - Added jobs for time 1459231545000 ms 2016-03-29 11:35:50,011 INFO JobScheduler:59 - Added jobs for time 1459231550000 ms 2016-03-29 11:35:55,004 INFO JobScheduler:59 - Added jobs for time 1459231555000 ms 2016-03-29 11:36:00,014 INFO JobScheduler:59 - Added jobs for time 1459231560000 ms 2016-03-29 11:36:05,003 INFO JobScheduler:59 - Added jobs for time 1459231565000 ms 2016-03-29 11:36:10,087 INFO JobScheduler:59 - Added jobs for time 1459231570000 ms 2016-03-29 11:36:15,004 INFO JobScheduler:59 - Added jobs for time 1459231575000 ms 2016-03-29 11:36:20,004 INFO JobScheduler:59 - Added jobs for time 1459231580000 ms 2016-03-29 11:36:25,139 INFO JobScheduler:59 - Added jobs for time 1459231585000 ms

Can you please help, how to solve this problem.

Answer1:

Change the spark-submit master from local to local[2]

spark-submit --master local[2] --class YOURPROGRAM YOUR.jar

Or set

new SparkConf().setAppName("SparkStreamingExample").setMaster("local[2]")

If you still facing the same problem after changing the number to 2, maybe you should just change it to a bigger number.

Reference: <a href="http://spark.apache.org/docs/latest/streaming-programming-guide.html" rel="nofollow">http://spark.apache.org/docs/latest/streaming-programming-guide.html</a>

When running a Spark Streaming program locally, do not use “local” or “local[1]” as the master URL. Either of these means that only one thread will be used for running tasks locally. If you are using an input DStream based on a receiver (e.g. sockets, Kafka, Flume, etc.), then the single thread will be used to run the receiver, leaving no thread for processing the received data. Hence, when running locally, always use “local[n]” as the master URL, where n > the number of receivers to run (see Spark Properties for information on how to set the master).

Extending the logic to running on a cluster, the number of cores allocated to the Spark Streaming application must be more than the number of receivers. Otherwise, the system will receive data, but not be able to process them.

Credit to bit1129: <a href="http://bit1129.iteye.com/blog/2174751" rel="nofollow">http://bit1129.iteye.com/blog/2174751</a>

Answer2:

I solved this problem by setting master from local to local[2]. Following related quote is from spark streaming doc:

<blockquote>

But note that a Spark worker/executor is a long-running task, hence it occupies one of the cores allocated to the Spark Streaming application. Therefore, it is important to remember that a Spark Streaming application needs to be allocated enough cores (or threads, if running locally) to process the received data, as well as to run the receiver(s).

</blockquote>

Answer3:

It's not a problem indeed, these INFOs are just log messages which you can avoid by changing log level from INFO to WARN or ERROR in conf/log4j.properties.

Spark Streaming would buffer your input data into small batches and submit the batch of input for execution periodically, therefore not a problem here.

Recommend

  • Get exit(1) from Python in shell
  • How to solve 'Attempted to read or write protected memory…' error with programmatic approa
  • WF4 Workflow under AppFabric not resuming properly after IISreset
  • Set job scheduler when screen is on (device NOT idle)?
  • new spark.sql.shuffle.partitions value not used after checkpointing
  • detecting connection lost in spark streaming
  • ckeditor and jquery UI dialog not working
  • Vuejs: Lifecycle hooks of child routerview components using keep alive
  • What causes the runtime difference in this trivial fortran code?
  • Retrieve IP address of device
  • using System.Speech.Synthesis with Windows10 universal app (XAML-C#)
  • Implicit joins and Where in Doctrine - how?
  • Android Activity.onWindowFocusChanged doesn't get called from within TabHost
  • Adding elements to a huge XML file
  • Button click event not firing in jQuery
  • Parsing a CSV string while ignoring commas inside the individual columns
  • chrome.tabs.executeScript only fires when the Developer Console is open
  • How to define and use opencv mat of user type
  • JSON response opens as a file, but I can't access it with JavaScript
  • Spark fat jar to run multiple versions on YARN
  • Ajax Loaded meta Tags
  • req.body is undefined - nodejs
  • Cassandra Data Model
  • Join two tables and save into third-sql
  • How to model a transition system with SPIN
  • ORA-29908: missing primary invocation for ancillary operator
  • Calling of Constructors in a Java
  • How do I rollback to a specific git commit
  • Traverse Array and Display in markup
  • Transpose CSV data with awk (pivot transformation)
  • Why can't I rebase on to an ancestor of source changesets if on a different branch?
  • Getting Messege Twice Using IMvxMessenger
  • Java static initializers and reflection
  • Are Kotlin's Float, Int etc optimised to built-in types in the JVM? [duplicate]
  • unknown Exception android
  • costura.fody for a dll that references another dll
  • Observable and ngFor in Angular 2
  • Unable to use reactive element in my shiny app
  • Converting MP3 duration time
  • Python/Django TangoWithDjango Models and Databases