19599

linux worker script/queue (php)

Question:

I need a binary/script (php) that does the following.

Start n process of X in the background and maintain the number processes.

An example:

<ul><li>n = 50</li> <li>initially 50 processes are started</li> <li>a process exits</li> <li>49 are still running</li> <li>so 1 should be started again.</li> </ul>

P.S.: I posted the same question on SV, which makes me probably very unpopular.

Answer1:

Can you use the crontab linux and write to a db or file the number of current process?. If DB, the advantage is that you can use to procedure and lock the table, and write the number of process.

But to backgroun you should use & at the end of the call to script

# php-f pro.php &

Answer2:

Pseudocode:

for (i=1; i<=50; i++) myprocess endfor while true while ( $(ps --no-headers -C myprocess|wc -l) < 50 ) myprocess endwhile endwhile

If you translate this to php and fix its flaws, it might just do what you want.

Answer3:

I would go in the direction that andres suggested. Just put something like this at the top of your pro.php file...

$this_file = __FILE__; $final_count = 50; $processes = `ps auwx | grep "php -f $this_file"`; $processes = explode("\n", $processes); if (count($processes)>$final_count+3) { exit; } //... Remaining code goes here

Answer4:

Have you tried making a PHP Daemon before?

<a href="http://kevin.vanzonneveld.net/techblog/article/create_daemons_in_php/" rel="nofollow">http://kevin.vanzonneveld.net/techblog/article/create_daemons_in_php/</a>

Answer5:

Here's something in Perl I have in my library (and hey, let's be honest, I'm not going to rig this up in PHP just to give you something working in that language this moment. I'm just using what I can copy / paste).

#!/usr/bin/perl use threads; use Thread::Queue; my @workers; my $num_threads = shift; my $dbname = shift; my $queue = new Thread::Queue; for (0..$num_threads-1) { $workers[$_] = new threads(\&worker); print "TEST!\n"; } while ($_ = shift @ARGV) { $queue->enqueue($_); } sub worker() { while ($file = $queue->dequeue) { system ('./4parser.pl', $dbname, $file); } } for (0..$num_threads-1) { $queue->enqueue(undef); } for (0..$num_threads-1) { $workers[$_]->join; }

Whenever one of those systems calls finishes up, it moves on dequeing. Oh, and damn if I know hwy I did 0..$numthreads instead of the normal my $i = 0; $i < ... idiom, but I did it that way that time.

Answer6:

I have to solutions to propose. Both do child process reboot on exit, do child process reloading on USR1 signal, wait for the children exit on SIGTERM and so on.

The first is based on <a href="https://github.com/swoole/swoole-src" rel="nofollow">swoole php extension</a>. It is very performant, async, non-blocking. Here's the usage example code:

<?php use Symfony\Component\Process\PhpExecutableFinder; require_once __DIR__.'/../vendor/autoload.php'; $phpBin = (new PhpExecutableFinder)->find(); if (false === $phpBin) { throw new \LogicException('Php executable could not be found'); } $daemon = new \App\Infra\Swoole\Daemon(); $daemon->addWorker(1, $phpBin, [__DIR__ . '/console', 'quartz:scheduler', '-vvv']); $daemon->addWorker(3, $phpBin, [__DIR__ . '/console', 'enqueue:consume', '--setup-broker', '-vvv']); $daemon->run();

The daemon code is <a href="https://gist.github.com/makasim/6f0b5c7ebbc1f70c2375d31d61460675" rel="nofollow">here</a>

Another is based on Symfony <a href="http://symfony.com/doc/current/components/process.html" rel="nofollow">process library</a>. It does not require any extra extensions. The usage example and daemon code could be found <a href="https://github.com/php-enqueue/enqueue-dev/pull/51" rel="nofollow">here</a>

Recommend

  • jQuery: Searching textarea value for words using regex
  • use grep and awk to transfer data from .srt to .csv/xls
  • How to draw cross background via css which is responsive
  • How to access list of email accounts with cPanel API?
  • How can I fill all space of a div 100% with 3 divs with the same size?
  • Easy Way to Get Averages Based on Names in List
  • Is there a way to get the process ID of a console program I've just started in the background?
  • TCameraComponent and TVideoCaptureDevice do not initialize in Win32
  • cordova build android throws error on Ubuntu 12.04
  • ASP.NET MVC - Detect Time Spent on Page
  • How to separate filename from path? basename() versus preg_split() with array_pop()
  • How do we generate stack trace in TOMCAT?
  • Excel 2007: Format of email address from Outlook 2007
  • How to debug Shell command after customization
  • Unicorn and Rails eat up 2x MySQL connections
  • pygame.init() shows as undefined variable after installing Pygame
  • Pythons argparse default value doesn't work
  • Are there any issues with placing image as background on an extended JFrame?
  • How to enable large page memory for the JVM?
  • Java making confirming exit
  • Criterion causing memory consumption to explode, no CAFs in sight
  • Magento get URL before current
  • Ubuntu and bcrypt
  • Azure table store snapshot/backup capability
  • Using Sax parsing to edit and write XML in VB6
  • Using JRuby with Rails 3.2
  • Code in Job's Script Block after Start-Process Does not Execute
  • xtable package: Skipping some rows in the output
  • Django: Count of Group Elements
  • Change JButton Shape while respecting Look And Feel
  • Splitting given String into two variables - php
  • Obtain ObjectIdHex value from mgo query
  • Sending data from AppleScript to FileMaker records
  • MySQL WHERE-condition in procedure ignored
  • How to CLICK on IE download dialog box i.e.(Open, Save, Save As…)
  • How can I remove ASP.NET Designer.cs files?
  • How to Embed XSL into XML
  • UserPrincipal.Current returns apppool on IIS
  • Conditional In-Line CSS for IE and Others?
  • java string with new operator and a literal