Metadata-Version: 2.1
Name: minionize
Version: 0.1.2
Summary: Massively Parallel operation made easy
Home-page: https://gitlab.inria.fr/msimonin/minionize
Keywords: packaging,dependency,poetry
Author: msimonin
Author-email: matthieu.simonin@inria.fr
Requires-Python: >=3.7,<4.0
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Requires-Dist: execo (>=2.6.4,<3.0.0)
Requires-Dist: google-cloud-pubsub (>=1.7.0,<2.0.0)
Project-URL: Repository, https://gitlab.inria.fr/msimonin/minionize
Description-Content-Type: text/x-rst


Rationale
---------

- You wrote a program ``a.out`` with some parameters
- You need to explore the space of parameters

Minionize is a solution to spawn a legion of ``a.out`` in a massively
parallel manner.

.. note::

    .. code-block:: bash

        $) minionize a.out

    will wait for inputs coming from a source and invoke ``a.out`` upon
    reception. Depending on the command outcome, the inputs may be acked or
    redelivered to another minion.

How does it work
----------------

A classical pattern to do the above is to apply the master/worker pattern
where a master give tasks to workers. Workers repeatedly fetch a new task
from a queue , run it and report back somewhere its status.

Minionize encapsulates ``a.out`` so that it can takes its inputs from a queue.

Currently we support:

- ``execo`` based queue: the queue is stored in a shared file system in your cluster
- ``Google pub/sub`` based queue: the queue is hosted by Google.

Some examples
-------------

- Simplest use:
    - with `Execo` engine:

.. code-block:: bash

    # generate some input parameters
    $) python -c "from execo_engine.sweep import ParamSweeper; ParamSweeper('sweeps', sweeps=range(10), save_sweeps=True)"

    # start your minions
    $) minionize echo hello
    hello 0
    hello 1
    hello 2
    hello 3
    hello 4
    hello 5
    hello 6
    hello 7
    hello 8
    hello 9


- ``examples/process.py``: run your program as a subprocess each time a new parameter comes / shows how to transform the input parameters
    - use it with `Execo` engine:


    .. code-block:: bash

        # generate the queue of task
        python -c "from execo_engine.sweep import ParamSweeper, sweep; ParamSweeper('sweeps', sweeps=sweep({'a': [0, 1], 'b': ['x', 't"]}), save_sweeps=True)"

        # start your minions
        MINION_ENGINE=execo python process.py


    - use it with `GooglePubSub` engine:

    .. code-block:: bash

        # start your minions
        MINION_ENGINE=google \
        GOOGLE_PROJECT_ID=gleaming-store-288314  \
        GOOGLE_TOPIC_ID=TEST \
        GOOGLE_SUBSCRIPTION=tada \
        GOOGLE_APPLICATION_CREDENTIALS=``/.gcp/gleaming-store-288314-2444b0d20a52.json \
        python process.py

Roadmap
-------

- Easy integration as docker entrypoint
- Support new queues (Redis stream, RabbitMQ, Kakfa ...)
- Support new abstractions to run container based application (docker, singularity...)
- Automatic encapsulation using a .minionize.yml
- Keep in touch (matthieu dot simonin at inria dot fr)
