joblib-htcondor

joblib-htcondor is a joblib backend that sends each task to an HTCondor cluster as a separate job. Code that already uses joblib, such as scikit-learn’s cross-validation and hyperparameter search, can scale out to a whole cluster with very few changes:

from operator import neg
from joblib import Parallel, delayed, parallel_config
from joblib_htcondor import register_htcondor

register_htcondor()

with parallel_config(
    backend="htcondor",
    n_jobs=-1,
    request_cpus=1,
    request_memory="1Gb",
):
    Parallel()(delayed(neg)(i + 1) for i in range(5))

Tasks and results go through a disk shared by all the nodes, so the backend works best for many compute-heavy tasks rather than short or I/O-bound ones.

pip install joblib-htcondor

• Source code on GitHub