Metadata-Version: 2.3
Name: gen3utils
Version: 0.10.8
Summary: Utils for Gen3 Commons management
License: Apache-2.0
Author: CTDS UChicago
Author-email: cdis@uchicago.edu
Requires-Python: >=3.9,<4.0
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Provides-Extra: s3log
Requires-Dist: cdislogging (>=1,<2)
Requires-Dist: click (>=8,<9)
Requires-Dist: dictionaryutils (>=3,<4)
Requires-Dist: gen3datamodel (>=3,<4)
Requires-Dist: gen3dictionary (>=2,<3)
Requires-Dist: gen3git (>=0.7.0,<1.0.0)
Requires-Dist: packaging (>=20,<24)
Requires-Dist: psqlgraph (>=3,<4)
Requires-Dist: pyyaml (<7)
Project-URL: Repository, https://github.com/uc-cdis/gen3utils
Description-Content-Type: text/markdown

# gen3utils

Utils for Gen3 commons management. Gen3utils includes a CLI and is intended to be used as an
external facing tool for validating configuration, generating release notes, posting deployment changes,
logging parser information, and more.

## Install with pip
pip install gen3utils

## manifest.json validation

Validate one or more `manifest.json` files:
```
gen3utils validate_manifest cdis-manifest/*/manifest.json
```

The validation settings can be updated by modifying [this file](gen3utils/manifest/validation_config.yaml).

## etlMapping.yaml validation

Validate an `etlMapping.yaml` file against the dictionary URL specified in a `manifest.json` file:
```
gen3utils validate_etl_mapping etlMapping.yaml manifest.json
```

## Portal Configuration (gitops.json) validation

Validate a `gitops.json` file against the dictionary URL specified in a `manifest.json` file and an etlMapping.yaml file. Adds a comment to a pull request listing all the errors encountered when validating against etlMapping.yaml
```
gen3utils validate_portal_config etlMapping.yaml manifest.json gitops.json <username>/<repository> <pull request number>
```
To run without making a pull request comment
```
gen3utils validate_portal_config etlMapping.yaml manifest.json gitops.json
```


## Comment on a PR with any deployment changes when updating manifest services

The command requires the name of the repository, the pull request number and **a `GITHUB_TOKEN` environment variable** containing a token with read and write access to the repository. It also comments a warning if a service is pinned on a branch.
```
pip install gen3utils
gen3utils post-deployment-changes <username>/<repository> <pull request number>
```

## Log parser for CTDS log pipeline

```
pip install gen3utils
gen3utils s3log --help
gen3utils s3log [OPTIONS] BUCKET PREFIX SCRIPT
```

Run `SCRIPT` in Gen3 logs under S3 `BUCKET:PREFIX`. The `SCRIPT` should be importable defining a method like this:
```
def handle_row(obj, line):
    if 1 + 1 == 2:
        return line
```

For example, to process logs in bucket `my-commons-logs` at prefix `my-logs` with a `gen3utils/script.py` file:
```
pip install gen3utils
gen3utils s3log my-commons-logs my-logs gen3utils.script
```

## Running tests locally

```
poetry install -vv
poetry run pytest -vv ./tests
```

