How can i execute the same python program in parallel (I am thinking of x10) with the only difference being two input arguments representing a time-range? I need it for some data-processing, which otherwise will take too long to finish
I know i can do it manually in shell by starting 10 scripts one by one, but it does not seem to be the most "elegant" solution + I would also love to define arguments for each of those programs dynamically in the "main" python program.
Is there a way to do it?
Enclose your script in a main method, like so:
def main(args): a, b = args # do something if __name__ == '__main__': args = parse_arguments() main(args)
Then you can use a second script together with
multiprocessing.Pool, to run the main method with different arguments.
from myscript import main from multiprocessing import Pool a = [1, 2, 3, 4, 5] b = [6, 7, 8, 9, 10] if __name__ == '__main__': pool = Pool(4) # four parallel jobs results = pool.map(main, zip(a, b))
You can try the following in shell:
python main.py [1, 2, 3, 4, 5] & python main.py [6, 7, 8, 9, 10]