Metadata-Version: 2.1
Name: d3rlpy
Version: 0.61
Summary: Data-driven Deep Reinforcement Learning Library as an Out-of-the-box Tool
Home-page: https://github.com/takuseno/d3rlpy
Author: Takuma Seno
Author-email: takuma.seno@gmail.com
License: MIT License
Description: <p align="center"><img align="center" width="300px" src="assets/logo.png"></p>
        
        # d3rlpy: A data-driven deep reinforcement learning library as an out-of-the-box tool
        
        ![test](https://github.com/takuseno/d3rlpy/workflows/test/badge.svg)
        ![build](https://github.com/takuseno/d3rlpy/workflows/build/badge.svg)
        [![Documentation Status](https://readthedocs.org/projects/d3rlpy/badge/?version=latest)](https://d3rlpy.readthedocs.io/en/latest/?badge=latest)
        [![codecov](https://codecov.io/gh/takuseno/d3rlpy/branch/master/graph/badge.svg?token=AQ02USKN6Y)](https://codecov.io/gh/takuseno/d3rlpy)
        [![Maintainability](https://api.codeclimate.com/v1/badges/c9162eb736d0b0f612d8/maintainability)](https://codeclimate.com/github/takuseno/d3rlpy/maintainability)
        [![Gitter](https://img.shields.io/gitter/room/d3rlpy/d3rlpy)](https://gitter.im/d3rlpy/d3rlpy)
        ![MIT](https://img.shields.io/badge/license-MIT-blue)
        
        d3rlpy is a data-driven deep reinforcement learning library as an out-of-the-box tool.
        
        ```py
        from d3rlpy.dataset import MDPDataset
        from d3rlpy.algos import CQL
        
        # MDPDataset takes arrays of state transitions
        dataset = MDPDataset(observations, actions, rewards, terminals)
        
        # train data-driven deep RL
        cql = CQL()
        cql.fit(dataset.episodes)
        
        # ready to control
        actions = cql.predict(x)
        ```
        
        Documentation: https://d3rlpy.readthedocs.io
        
        ## key features
        
        ### :zap: Most Practical RL Library Ever
        - **offline RL**: d3rlpy supports state-of-the-art offline RL algorithms. Offline RL is extremely powerful when the online interaction is not feasible during training (e.g. robotics, medical).
        - **online RL**: d3rlpy also supports conventional state-of-the-art online training algorithms without any compromising, which means that you can solve any kinds of RL problems only with `d3rlpy`.
        - **advanced engineering**: d3rlpy is designed to implement the faster and efficient training algorithms. For example, you can train Atari environments with x4 less memory space and as fast as the fastest RL library.
        
        ### :beginner: Easy-To-Use API
        - **zero-knowledge of DL library**: d3rlpy provides many state-of-the-art algorithms through intuitive APIs. You can become a RL engineer even without knowing how to use deep learning libraries.
        - **scikit-learn compatibility**: d3rlpy is not only easy, but also completely compatible with scikit-learn API, which means that you can maximize your productivity with the useful scikit-learn's utilities.
        
        ### :rocket: Beyond State-Of-The-Art
        - **distributional Q function**: d3rlpy is the first library that supports distributional Q functions in the all algorithms. The distributional Q function is known as the very powerful method to achieve the state-of-the-performance.
        - **many tweek options**: d3rlpy is also the first to support N-step TD backup, ensemble value functions and data augmentation in the all algorithms, which lead you to the place no one ever reached yet.
        
        
        ## installation
        d3rlpy supports Linux, macOS and Windows.
        ### PyPI
        [![PyPI version](https://badge.fury.io/py/d3rlpy.svg)](https://badge.fury.io/py/d3rlpy)
        ![PyPI - Downloads](https://img.shields.io/pypi/dm/d3rlpy)
        ```
        $ pip install d3rlpy
        ```
        ### Anaconda
        [![Anaconda-Server Badge](https://anaconda.org/conda-forge/d3rlpy/badges/version.svg)](https://anaconda.org/conda-forge/d3rlpy)
        [![Anaconda-Server Badge](https://anaconda.org/conda-forge/d3rlpy/badges/platforms.svg)](https://anaconda.org/conda-forge/d3rlpy)
        [![Anaconda-Server Badge](https://anaconda.org/conda-forge/d3rlpy/badges/downloads.svg)](https://anaconda.org/conda-forge/d3rlpy)
        ```
        $ conda install -c conda-forge d3rlpy
        ```
        
        ### Docker
        ![Docker Pulls](https://img.shields.io/docker/pulls/takuseno/d3rlpy)
        ```
        $ docker run -it --gpus all --name d3rlpy takuseno/d3rlpy:latest bash
        ```
        
        ## supported algorithms
        | algorithm | discrete control | continuous control | data-driven RL? |
        |:-|:-:|:-:|:-:|
        | Behavior Cloning (supervised learning) | :white_check_mark: | :white_check_mark: | |
        | [Deep Q-Network (DQN)](https://www.nature.com/articles/nature14236) | :white_check_mark: | :no_entry: | |
        | [Double DQN](https://arxiv.org/abs/1509.06461) | :white_check_mark: | :no_entry: | |
        | [Deep Deterministic Policy Gradients (DDPG)](https://arxiv.org/abs/1509.02971) | :no_entry: | :white_check_mark: | |
        | [Twin Delayed Deep Deterministic Policy Gradients (TD3)](https://arxiv.org/abs/1802.09477) | :no_entry: | :white_check_mark: | |
        | [Soft Actor-Critic (SAC)](https://arxiv.org/abs/1812.05905) | :white_check_mark: | :white_check_mark: | |
        | [Batch Constrained Q-learning (BCQ)](https://arxiv.org/abs/1812.02900) | :white_check_mark: | :white_check_mark: | :white_check_mark: |
        | [Bootstrapping Error Accumulation Reduction (BEAR)](https://arxiv.org/abs/1906.00949) | :no_entry: | :white_check_mark: | :white_check_mark: |
        | [Advantage-Weighted Regression (AWR)](https://arxiv.org/abs/1910.00177) | :white_check_mark: | :white_check_mark: | :white_check_mark: |
        | [Conservative Q-Learning (CQL)](https://arxiv.org/abs/2006.04779) (recommended) | :white_check_mark: | :white_check_mark: | :white_check_mark: |
        | [Advantage Weighted Actor-Critic (AWAC)](https://arxiv.org/abs/2006.09359) | :no_entry: | :white_check_mark: | :white_check_mark: |
        | [Policy in Latent Action Space (PLAS)](https://arxiv.org/abs/2011.07213) | :no_entry: | :white_check_mark: | :white_check_mark: |
        
        ## supported Q functions
        - [x] standard Q function
        - [x] [Quantile Regression](https://arxiv.org/abs/1710.10044)
        - [x] [Implicit Quantile Network](https://arxiv.org/abs/1806.06923)
        - [x] [Fully parametrized Quantile Function](https://arxiv.org/abs/1911.02140) (experimental)
        
        ## other features
        Basically, all features are available with every algorithm.
        
        - [x] evaluation metrics in a scikit-learn scorer function style
        - [x] export greedy-policy as TorchScript or ONNX
        - [x] parallel cross validation with multiple GPU
        - [x] [model-based algorithm](https://arxiv.org/abs/2005.13239)
        
        ## examples
        ### Atari 2600
        <p align="center"><img align="center" width="160px" src="assets/breakout.gif"></p>
        
        ```py
        from d3rlpy.datasets import get_atari
        from d3rlpy.algos import DiscreteCQL
        from d3rlpy.metrics.scorer import evaluate_on_environment
        from d3rlpy.metrics.scorer import discounted_sum_of_advantage_scorer
        from sklearn.model_selection import train_test_split
        
        # get data-driven RL dataset
        dataset, env = get_atari('breakout-expert-v0')
        
        # split dataset
        train_episodes, test_episodes = train_test_split(dataset, test_size=0.2)
        
        # setup algorithm
        cql = DiscreteCQL(n_frames=4, q_func_factory='qr', scaler='pixel', use_gpu=True)
        
        # start training
        cql.fit(train_episodes,
                eval_episodes=test_episodes,
                n_epochs=100,
                scorers={
                    'environment': evaluate_on_environment(env),
                    'advantage': discounted_sum_of_advantage_scorer
                })
        ```
        
        See more Atari datasets at [d4rl-atari](https://github.com/takuseno/d4rl-atari).
        
        ### PyBullet
        <p align="center"><img align="center" width="160px" src="assets/hopper.gif"></p>
        
        ```py
        from d3rlpy.datasets import get_pybullet
        from d3rlpy.algos import CQL
        from d3rlpy.metrics.scorer import evaluate_on_environment
        from d3rlpy.metrics.scorer import discounted_sum_of_advantage_scorer
        from sklearn.model_selection import train_test_split
        
        # get data-driven RL dataset
        dataset, env = get_pybullet('hopper-bullet-mixed-v0')
        
        # split dataset
        train_episodes, test_episodes = train_test_split(dataset, test_size=0.2)
        
        # setup algorithm
        cql = CQL(q_func_factory='qr', use_gpu=True)
        
        # start training
        cql.fit(train_episodes,
                eval_episodes=test_episodes,
                n_epochs=300,
                scorers={
                    'environment': evaluate_on_environment(env),
                    'advantage': discounted_sum_of_advantage_scorer
                })
        ```
        
        See more PyBullet datasets at [d4rl-pybullet](https://github.com/takuseno/d4rl-pybullet).
        
        ### Online Training
        ```py
        import gym
        
        from d3rlpy.algos import SAC
        from d3rlpy.online.buffers import ReplayBuffer
        
        # setup environment
        env = gym.make('HopperBulletEnv-v0')
        eval_env = gym.make('HopperBulletEnv-v0')
        
        # setup algorithm
        sac = SAC(use_gpu=True)
        
        # setup replay buffer
        buffer = ReplayBuffer(maxlen=1000000, env=env)
        
        # start training
        sac.fit_online(env, buffer, n_steps=1000000, eval_env=eval_env)
        ```
        
        ## tutorials
        Try a cartpole example on Google Colaboratory!
        
        - offline RL tutorial: [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/takuseno/d3rlpy/blob/master/tutorials/cartpole.ipynb)
        - online RL tutorial: [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/takuseno/d3rlpy/blob/master/tutorials/online.ipynb)
        
        ## scikit-learn compatibility
        This library is designed as if born from scikit-learn.
        You can fully utilize scikit-learn's utilities to increase your productivity.
        ```py
        from sklearn.model_selection import train_test_split
        from d3rlpy.metrics.scorer import td_error_scorer
        
        train_episodes, test_episodes = train_test_split(dataset)
        
        cql.fit(train_episodes,
                eval_episodes=test_episodes,
                scorers={'td_error': td_error_scorer})
        ```
        
        You can naturally perform cross-validation.
        ```py
        from sklearn.model_selection import cross_validate
        
        scores = cross_validate(cql, dataset, scoring={'td_error': td_error_scorer})
        ```
        
        And more.
        ```py
        from sklearn.model_selection import GridSearchCV
        
        gscv = GridSearchCV(estimator=cql,
                            param_grid={'actor_learning_rate': [3e-3, 3e-4, 3e-5]},
                            scoring={'td_error': td_error_scorer},
                            refit=False)
        gscv.fit(train_episodes)
        ```
        
        ## contributions
        Any kind of contribution to d3rlpy would be highly appreciated!
        Please check the [contribution guide](CONTRIBUTING.md).
        
        ## citation
        ```
        @misc{seno2020d3rlpy,
          author = {Takuma Seno},
          title = {d3rlpy: A data-driven deep reinforcement library as an out-of-the-box tool},
          year = {2020},
          publisher = {GitHub},
          journal = {GitHub repository},
          howpublished = {\url{https://github.com/takuseno/d3rlpy}}
        }
        ```
        
        ## acknowledgement
        This work is supported by Information-technology Promotion Agency, Japan
        (IPA), Exploratory IT Human Resources Project (MITOU Program) in the fiscal
        year 2020.
        
Platform: UNKNOWN
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Education
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Operating System :: POSIX :: Linux
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: MacOS :: MacOS X
Requires-Python: >=3.6.0
Description-Content-Type: text/markdown
