How to parallelise a function from another module using python multithreading without pickling it? -

How to parallelise a function from another module using python multithreading without pickling it? -

in implementation of svm prediction model, create execution of function svmutil.svm_train multi-threaded. although i'm new implementation of multi-threaded programs, have knowledge of parallel programming concepts , believe training multiple models different sets of parameters simultaneously theoretically possible.

setup:

import svmutil import multiprocessing mp problem = svm_util.svm_read_problem('my_problem') # have list of svm_param objects want train params = mycode.svm_param_list() # calculate number of worker threads processes = mp.cpu_count() * 2

split training multiple threads of execution:

pool = mp.pool(processes) param in params: pool.apply(svmutil.svm_train, args=(problem, param,)) pool.close() pool.join()

however, problem i'm having svmutil.svm_train cannot pickled contains c-type pointer. python interpreter gives me error:

valueerror: ctypes objects containing pointers cannot pickled

i'd rather adjust implementation somehow pickle function in module. therefore, know, there way in can parallelise function without pickling it?

also, how can gather results of function? ideally, list of trained models (output of function svmutil.svm_train each time called function).

i managed utilize top answer, in question linked in comment above, build solution. able parallelise phone call python function in external module spawning pipes , processes , listening results. function pass pipe follows: pipe.send(svmutil.svm_train(problem,param)))

python python-2.7 multiprocessing libsvm pool