44441

Futures somehow slower then agents?

The following code does essentially just let you execute something like (function (range n)) in parallel.

(experiment-with-agents 10000 10 #(filter prime? %))

This for example finds the prime numbers between 0 and 10000 with 10 agents.

(experiment-with-futures 10000 10 #(filter prime? %))

Same just with futures.

Now the problem is that the solution with futures doesn't run faster with more futures. Example:

; Futures (time (experiment-with-futures 10000 1 #(filter prime? %))) "Elapsed time: 33417.524634 msecs" (time (experiment-with-futures 10000 10 #(filter prime? %))) "Elapsed time: 33891.495702 msecs" ; Agents (time (experiment-with-agents 10000 1 #(filter prime? %))) "Elapsed time: 33048.80492 msecs" (time (experiment-with-agents 10000 10 #(filter prime? %))) "Elapsed time: 9211.864133 msecs"

Why? Did I do something wrong (probably, new to Clojure and just playing around with stuff^^)? Because I thought that futures are actually prefered in that scenario.

Source:

(defn setup-agents [coll-size num-agents] (let [step (/ coll-size num-agents) parts (partition step (range coll-size)) agents (for [_ (range num-agents)] (agent []) ) vect (map #(into [] [%1 %2]) agents parts)] (vec vect))) (defn start-agents [coll f] (for [[agent part] coll] (send agent into (f part)))) (defn results [agents] (apply await agents) (vec (flatten (map deref agents)))) (defn experiment-with-agents [coll-size num-agents f] (-> (setup-agents coll-size num-agents) (start-agents f) (results))) (defn experiment-with-futures [coll-size num-futures f] (let [step (/ coll-size num-futures) parts (partition step (range coll-size)) futures (for [index (range num-futures)] (future (f (nth parts index))))] (vec (flatten (map deref futures)))))

Answer1:

You're getting tripped up by the fact that for produces a lazy sequence inside of experiment-with-futures. In particular, this piece of code:

(for [index (range num-futures)] (future (f (nth parts index))))

does not immediately create all of the futures; it returns a lazy sequence that will not create the futures until the contents of the sequence are realized. The code that realizes the lazy sequence is:

(vec (flatten (map deref futures)))

Here, map returns a lazy sequence of the dereferenced future results, backed by the lazy sequence of futures. As vec consumes results from the sequence produced by map, each new future is not submitted for processing until the previous one completes.

To get parallel processing, you need to not create the futures lazily. Try wrapping the for loop where you create the futures inside a doall.

The reason you're seeing an improvement with agents is the call to (apply await agents) immediately before you gather the agent results. Your start-agents function also returns a lazy sequence and does not actually dispatch the agent actions. An implementation detail of apply is that it completely realizes small sequences (under 20 items or so) passed to it. A side effect of passing agents to apply is that the sequence is realized and all agent actions are dispatched before it is handed off to await.

Recommend

  • how can you interleave two vectors of differing lengths in clojure
  • Using midje provided in a let clause doesn't stub methods
  • Ajax / jQuery to PHP and back again?
  • StackOverflowError on tail-recursive function
  • Clojure: how do I require a class and call a static method?
  • How to use prepared statements in lua-dbi?
  • Mongo collection query and Operators
  • Should I prefer namespaces or classes with static functions?
  • Increase the speed of redrawing contour plot in matplotlib
  • 'Could not find us.bpsm:edn-java:0.4.3' error with Gradle for Clojure (Clojuresque)
  • ObjectID generated by server on pymongo
  • Plot a decision tree with R
  • Clojure embed var to String, a alternative way? Like sprintf way?
  • C# foreach - Is collection computed with each iteration? [duplicate]
  • stringify/parse edn in clojure/ClojureScript
  • Locating unmatched delimiters in Clojurescript
  • Spring security - same page to deliver different content based on user role
  • Merge Module leaving files during uninstall
  • Migration to HRD - How to convert string-encoded keys to new application
  • Prevent page break in text block with iText, XMLWorker
  • how to populate a SQLite database and use that database in phonegap?
  • Python 3.2.2, error(scripts to exe)
  • NSIS decompiler
  • Symfony 2 error page response
  • How can I tell a form not to dispose a particular control when it closes?
  • Hash Code in SQL Server?
  • Intel-64 and ia32 atomic operations acquire-release semantics and GCC 5+
  • Read text file that is not in the main package in a runnable jar
  • Problem while Building a Setup Project for a windows Service?
  • Allowing both email and username for authentication
  • Meteor: Do Something On Email Verification Confirmation
  • Handling un-mapped Rest path
  • When to use `image` and when to use `Matrix` in Emgu CV?
  • Counter field in MS Access, how to generate?
  • Where to put my custom functions in Wordpress?
  • Javascript + PHP Encryption with pidCrypt
  • Websockets service method fails during R startup
  • RestKit - RKRequestDelegate does not exist
  • WPF Applying a trigger on binding failure
  • Why can't I rebase on to an ancestor of source changesets if on a different branch?