python - Simple parallel scientific programming -

python - Simple parallel scientific programming -

i quite write simple optimization routines this:

def createinstance(n): while(true): #create instance called instance yield instance loopno = 100000 n= 100 min = 0 in xrange(loopno): inst in createinstance(n): value = foo(inst) if (value < min): min = value print min

i able utilize of cores on machine this.

a simple method split range parts , farm them out cores , collect results @ end.

a improve method have cores request batch of instances when idle.

what's nice way solve problem maximum efficiency in python? maybe standard plenty community question?

this question seems simpler version of solving embarassingly parallel problems using python multiprocessing .

with regards standard python install, multiprocessing module seems you'd want, illustration case in http://docs.python.org/2/library/multiprocessing.html looks adapted case. if memory isn't issue, splitting range on n processors efficient option, have no communication between processes.

there seem many ways of parallelising python, either using mpi or smp. there's list of potential candidates on python wiki http://wiki.python.org/moin/parallelprocessing

parallel python http://www.parallelpython.com supports smp, want in instance (single machine, multiple processors, rather cluster of machines).

python multiprocessing