22734

Mesos Framework

Question:

I want to distribute Docker containers on a cluster by using java and define dependencies, so that when one container is finished I can parse the produced output in my java code. There are many mesos frameworks that can achieve that and I'm not sure which one to take: Marathon, Singularity, Chronos, Aurora.

What I have learned so far:

Marathon has a really nice java client api but is used for long term tasks (don't know if that is a problem, because my tasks will not run that long) and I dont think that I can define that kind of dependency. If I would use marathon I would have to poll for the status of the app. Chronos has no java api (At least I could not find any). So I can choose between Aurora and Singularity.

Can anyone help and recommend me one of them for my use case.

Answer1:

<a href="https://mesos.github.io/chronos/" rel="nofollow">Chronos</a> (like Marathon) is written in Scala, so it should be easy to integrate with any JVM-based app. Even better, they both (probably all 4 actually) have REST APIs, so you can just curl your actions/queries, regardless of each app/service's implementation language.

For your use case, you'll probably prefer Chronos over Marathon, since you expect your tasks to complete (and not be restarted). Marathon is intended for long-running (marathon, running.. get it?) tasks, so it will restart a failed/completed task. Chronos is like a distributed cron with dependencies, so you can schedule your jobs, and then add dependent jobs to start upon completion of previous jobs, building arbitrary DAGs of jobs.

I'll let others chime in about Aurora and Singularity. I think they both support cron-like jobs.

Answer2:

<a href="http://aurora.apache.org" rel="nofollow">Apache Aurora</a> added Docker support in <a href="http://aurora.apache.org/blog/aurora-0-7-0-incubating-released/" rel="nofollow">February's 0.7.0 release</a>, and can manage both long-running services as well as cron using the same scheduler. It's also written in Java.

There's a great tech talk from the developer who contributed this feature about <a href="https://www.youtube.com/watch?v=ZZXtXLvTXAE" rel="nofollow">how they schedule Docker containers using Aurora</a>. Although the examples provided in the talk are for long-running services, there's just a minor change your job configuration to specify a service vs cron.

Answer3:

Chronos is good for that. You can create dependency DAGs in Chronos. However, it doesn't provide you mechanisms to pass the parameters from one job to other. You have to serialize and deserialize the stuff in DB or shared storage across the cluster.

Recommend

  • How to best manage SMTP clients
  • Swift iOS don't recognize GCM library imports?
  • Deduce parent class of inherited method in C++
  • Get the number 18437736874454810627
  • NHibernate manually control fetching
  • JSR-330 support in Picocontainer : @Inject … @Named(\"xxx)
  • SyntaxError: (irb):26: both block arg and actual block given
  • Can't remove headers after they are sent
  • Hide HTML elements without javascript, only CSS
  • How do I display a dialog that asks the user multi-choice questıon using tkInter?
  • How to use JavaScript to determine whether a file exists in a directory?
  • Insert new calendar with SyncAdapter- Calendar API Android
  • Ensure fsync did its job
  • Declaring variable dynamically in VB.net
  • Transactional Create with Validation in ServiceStack Redis Client
  • How do I pass the string value parameter of the selected list item from an auto-populated dropdown l
  • Yii2: Config params vs. const/define
  • Scrapy recursive link crawler
  • Ajax Loaded meta Tags
  • How to rebase a series of branches?
  • Exchange data b/w iOS devices using Bluetooth 4.0
  • req.body is undefined - nodejs
  • Can I have the cursor start on a particular column by default in jqgrid's edit mode?
  • Symfony2: How to get request parameter
  • How to make Safari send if-modified-since header?
  • Akka Routing: Reply's send to router ends up as dead letters
  • How to pass list parameters for each object using Spring MVC?
  • SVN: Merging two branches together
  • Hibernate gives error error as “Access to DialectResolutionInfo cannot be null when 'hibernate.
  • Is there a mandatory requirement to switch app.yaml?
  • VB.net deserialize, JSON Conversion from type 'Dictionary(Of String,Object)' to type '
  • KeystoneJS: Relationships in Admin UI not updating
  • How to CLICK on IE download dialog box i.e.(Open, Save, Save As…)
  • Setting background image for body element in xhtml (for different monitors and resolutions)
  • Can Visual Studio XAML designer handle font family names with spaces as a resource?
  • How can I remove ASP.NET Designer.cs files?
  • python draw pie shapes with colour filled
  • JaxB to read class hierarchy
  • Is there any way to bind data to data.frame by some index?
  • How can i traverse a binary tree from right to left in java?