39779

Executing same Python program with different arguments in parallel

How can i execute the same python program in parallel (I am thinking of x10) with the only difference being two input arguments representing a time-range? I need it for some data-processing, which otherwise will take too long to finish

I know i can do it manually in shell by starting 10 scripts one by one, but it does not seem to be the most "elegant" solution + I would also love to define arguments for each of those programs dynamically in the "main" python program.

Is there a way to do it?

Answer1:

Enclose your script in a main method, like so:

def main(args): a, b = args # do something if __name__ == '__main__': args = parse_arguments() main(args)

Then you can use a second script together with multiprocessing.Pool, to run the main method with different arguments.

from myscript import main from multiprocessing import Pool a = [1, 2, 3, 4, 5] b = [6, 7, 8, 9, 10] if __name__ == '__main__': pool = Pool(4) # four parallel jobs results = pool.map(main, zip(a, b))

Answer2:

You can try the following in shell:

python main.py [1, 2, 3, 4, 5] & python main.py [6, 7, 8, 9, 10]

Recommend

  • Scheduling a Python Script via Windows Task Scheduler
  • How to debug office add-in for mac?
  • Composer error [UnexpectedValueException]
  • powershell: script to start a program with parameters?
  • Additional scripts in PATH
  • How can I include javascript in content loaded with jquery using $.ajax
  • get src attribute of an image based on class or id
  • how to make vpython .exe using pyinstaller
  • How to change the default configuration files used in bootstrapping of reactJs through npm
  • c++ search a vector for element first seen position
  • Is there any way of quickening monkeyrunner script execution?
  • Is there any purpose for h2-h6 headings in HTML5?
  • py4j.protocol.Py4JJavaError when selecting nested column in dataframe using select statetment
  • python: forcing relative imports to search from script file
  • Python PIL to extract number from image
  • Available space left on drive - WinAPI - Windows CE
  • why calling cd shell command through system() or execvp() from a child process won't work?
  • Python 3.2.2, error(scripts to exe)
  • Examples of how to a STS in .Net 4.5 using WCF
  • Installing Perl6 and Panda on Ubuntu 15.10. Problems with bootstrap.pl
  • How can I get the full list of running processes on a Mac from a python app
  • Optimizing the print function in Matlab
  • How do I display a dialog that asks the user multi-choice questıon using tkInter?
  • how does System.Web.HttpRequest::PathInfo work?
  • Security issues with PHP's Readfile method
  • Android application: how to use the camera and grab the image bytes?
  • Stop Bash Script if Hive Fails
  • Content-Length header not returned from Pylons response
  • iOS: Detect app start via notification press
  • Test if a set exists before trying to drop it
  • Play WS (2.2.1): post/put large request
  • Chrome doesn't support silverlight anymore? How to solve this?
  • Spark fat jar to run multiple versions on YARN
  • How to access EntityManager inside Entity class in EJB3
  • formatting the colorbar ticklabels with SymLogNorm normalization in matplotlib
  • What is the “return” in scheme?
  • vba code to select only visible cells in specific column except heading
  • Which linear programming package should I use for high numbers of constraints and “warm starts” [clo
  • How can I estimate amount of memory left with calling System.gc()?
  • Append folder name and increment by 1 using batch script