Resolwe Runtime Utilities¶

A project that provides convenience utilities for writing processes for the Resolwe dataflow engine.

Contents:

Install¶

Prerequisites¶

Make sure you have Python (3.4+) installed on your system. If you don’t have it yet, follow these instructions.

Installing via pip¶

pip install resolwe-runtime-utils

Installing from source¶

Download the latest release of resolwe-runtime-utils and extract it.

Go to the directory with the extracted source code and install it:

pip install .

API reference¶

Utility functions that make it easier to write a Resolwe process.

class resolwe_runtime_utils.OutputType¶: An enumeration.

resolwe_runtime_utils.command(name, data)¶: Create a command from command name and payload.

resolwe_runtime_utils.send_message(data, header_size=8)¶: Send data over socket.

resolwe_runtime_utils.collect_entry(entry, references)¶

Get the size of the entry and its references and upload them.

The entry and its references are uploaded to the chosen storage connector.

NOTE: This process may take considerable amount of time.

Args references:
Args entry:	file or directory that is being collected.
	references belonging to the entry.

resolwe_runtime_utils.save_list(key, *values)¶: Construct save_list command.

resolwe_runtime_utils.annotate_entity(key, value)¶: Construct annotate entity command.

resolwe_runtime_utils.save(key, value)¶: Construct save command.

resolwe_runtime_utils.save_file(key, file_path, *refs)¶

Construct save file command.

Data is of the form: { key: {“file”: file_path, “size”: file_size, “total_size”: total_size }}

with added references

“refs”: [refs[0], refs[1], … ]

when applicable.

resolwe_runtime_utils.save_file_list(key, *files_refs)¶

Construct the save files command.

Each parameter is a file-refs specification of the form: <file-path>:<reference1>,<reference2>, …, where the colon ‘:’ and the list of references are optional.

Data object is of the form: { key: {“file”: file_path}}, or { key: {“file”: file_path, “refs”: [refs[0], refs[1], … ]}}

resolwe_runtime_utils.save_dir(key, dir_path, *refs)¶

Construct save dir command.

Data object is of the form: { key: {“dir”: dir_path}}, or { key: {“dir”: dir_path, “refs”: [refs[0], refs[1], … ]}}

resolwe_runtime_utils.save_dir_list(key, *dirs_refs)¶

Construct save dirs command.

Each parameter is a dir-refs specification of the form: <dir-path>:<reference1>,<reference2>, …, where the colon ‘:’ and the list of references are optional.

Data object is of the form: { key: {“dir”: dir_path}}, or { key: {“dir”: dir_path, “refs”: [refs[0], refs[1], … ]}}

resolwe_runtime_utils.info(value)¶: Construct info command.

resolwe_runtime_utils.warning(value)¶: Construct warning command.

resolwe_runtime_utils.error(value)¶: Construct error command.

resolwe_runtime_utils.progress(progress)¶

Construct progress command.

Progress is reported as float between 0 and 1 and sent to listener as int between 0 and 100.

resolwe_runtime_utils.checkrc(rc, *args)¶

Check if rc (return code) meets requirements.

Check if rc is 0 or is in args list that contains acceptable return codes. Last argument of args can optionally be error message that is printed if rc doesn’t meet requirements.

Output is JSON of the form:

{“proc.rc”: <rc>,

“proc.error”: “<error_msg>”},

where “proc.error” entry is omitted if empty.

resolwe_runtime_utils.export_file(file_path)¶: Construct export command.

resolwe_runtime_utils.run(process_slug, run)¶

Run process with the given slug with the given inputs.

The argument run must be a valid JSON string representing the inputs.

class resolwe_runtime_utils.ImportedFormat¶: Import destination file format.

resolwe_runtime_utils.import_file(src, file_name, imported_format='both', progress_from=0.0, progress_to=None)¶

Import file to working directory.

Parameters:	src – Source file path or URL file_name – Source file name imported_format – Import file format (extracted, compressed or both) progress_from – Initial progress value progress_to – Final progress value
Returns:	Destination file path (if extracted and compressed, extracted path given)

Console reference¶

List of console commands that can be used inside a Bash Resolwe process.

re-save: A console wrapper for resolwe_runtime_utils.save().

re-export: A console wrapper for resolwe_runtime_utils.export_file().

re-save-list: A console wrapper for resolwe_runtime_utils.save_list().

re-save-file: A console wrapper for resolwe_runtime_utils.save_file().

re-save-file-list: A console wrapper for resolwe_runtime_utils.save_file_list().

re-save-dir: A console wrapper for resolwe_runtime_utils.save_dir().

re-save-dir-list: A console wrapper for resolwe_runtime_utils.save_dir_list().

re-info: A console wrapper for resolwe_runtime_utils.info().

re-warning: A console wrapper for resolwe_runtime_utils.warning().

re-error: A console wrapper for resolwe_runtime_utils.error().

re-progress: A console wrapper for resolwe_runtime_utils.progress().

_re-checkrc

A console wrapper for resolwe_runtime_utils.checkrc().

Note

It should not be used directly. Instead, create a wrapper Bash function that calls it with the return value of the previously executed Bash command:

re-checkrc() { _re-checkrc $? "$@"; }

Add the following snippet to your ~/.bash_profile and/or ~/.bashrc file to make it easily accessible.

Change Log¶

All notable changes to the Resolwe Runtime Utilities project will be documented in this file. This project adheres to Semantic Versioning.

Unreleased¶

Added¶

Add support for resolving urls

3.1.1 - 2021-07-21¶

Changed¶

Fix requests version due to compatibility with Python 3.4

Added¶

Add support for Python 3.9

Changed¶

Migrate from Travis to Github Actions

3.1.0 - 2021-02-09¶

Added¶

Signal processing container to upload referenced files
Send JSON fields over socket instead of using files

3.0.0 - 2020-12-09¶

Added¶

Add re-run command
Send commands over socket instead of printing JSON to stdout

2.1.0 - 2020-02-11¶

Added¶

Add support for Python 3.8
Add re-annotate-entity command

2.0.1 - 2019-04-10¶

Changed¶

Loosen requests requirement to >=2.20.1

2.0.0 - 2019-04-09¶

Changed¶

The export function was renamed to export_file to be consistent with import_file

Added¶

Add import_file that imports compressed (or not) files of various formats to working directory

1.2.0 - 2017-08-08¶

Changed¶

Handle unexpected errors in re-* functions and print an error with a descriptive message
Escape double quotes in values of re-* commands

1.1.0 - 2016-07-25¶

Added¶

Add tests for all console commands (TestConsoleCommands)
Add export function and console command

1.0.0 - 2016-06-16¶

Added¶

Use Travis CI to run the tests
Add test coverage and track it with Codecov
Start writing the Change Log and include it in the Documentation
Add docs and packaging Tox testing environments
Add dev, a list of extra requirements for development
Add save_list and save_file_list functions and console commands
Add save_dir and save_dir_list functions and console commands

Changed¶

Consistently use Resolwe Runtime Utilities as the project name/title
Improve documentation
Use py.test as the test runner since its pytest-cov plugin enables to easily compute the test coverage while running the tests
Create _get_json auxiliary function use it in save and save_list functions
Check if files exist before saving them
Save Resolwe errors instead of raising Python exceptions
Make checkrc and progress functions more robust to improper input

Contributing¶

Preparing environment¶

Fork the main Resolwe Runtime Utilities’ git repository.

If you don’t have Git installed on your system, follow these instructions.

Clone your fork (replace <username> with your GitHub account name) and change directory:

git clone https://github.com/<username>/resolwe-runtime-utils.git
cd resolwe-runtime-utils

Prepare Resolwe Runtime Utilities for development:

pip install -e .[dev,docs,package,test]

Note

We recommend using pyvenv to create an isolated Python environement for resolwe-runtime-utils.

Running tests¶

Using Tox ¶

To run the tests, use:

tox

To re-create the virtual environment before running the tests, use:

tox -r

To only run the tests of a given Tox environment, use:

tox -e <tox-environment>

For example, to only run the packaging tests, use

tox -e packaging

Note

To see the list of available Tox environments, see tox.ini.

Manually¶

To run the tests, use:

py.test

Coverage report¶

To see the tests’ code coverage, use:

py.test --cov=resolwe_runtime_utils

To generate a HTML with tests’ code coverage, use:

py.test --cov=resolwe_runtime_utils --cov-report=html

Building documentation¶

python setup.py build_sphinx

Preparing release¶

Checkout the latest code and create a release branch:

git checkout master
git pull
git checkout -b release-<new-version>

Replace the Unreleased heading in docs/CHANGELOG.rst with the new version, followed by release’s date (e.g. 13.2.0 - 2018-10-23).

Note

Use Semantic versioning.

Commit changes to git:

git commit -a -m "Prepare release <new-version>"

Push changes to your fork and open a pull request:

git push --set-upstream <fork-name> release-<new-version>

Wait for the tests to pass and the pull request to be approved. Merge the code to master:

git checkout master
git merge --ff-only release-<new-version>

Tag the new release from the latest commit:

git checkout master
git tag -m "Version <new-version>" <new-version>

Note

Project’s version will be automatically inferred from the git tag using setuptools_scm.

Push the tag to the main Resolwe Runtime Utilities git repository:

git push <upstream-name> master <new-version>

The tagged code will we be released to PyPI automatically. Inspect Travis logs of the Release step if errors occur.

Preparing pre-release¶

When preparing a pre-release (i.e. an alpha release), one can skip the “release” commit that updates the change log and just tag the desired commit with a pre-release tag (e.g. 13.3.0a1). By pushing it to GitHub, the tagged code will be automatically tested by Travis CI and then released to PyPI.

Resolwe Runtime Utilities¶

Install¶

Prerequisites¶

Installing via pip¶

Installing from source¶

API reference¶

Console reference¶

Change Log¶

Unreleased¶

Added¶

3.1.1 - 2021-07-21¶

Changed¶

Added¶

Changed¶

3.1.0 - 2021-02-09¶

Added¶

3.0.0 - 2020-12-09¶

Added¶

2.1.0 - 2020-02-11¶

Added¶

2.0.1 - 2019-04-10¶

Changed¶

2.0.0 - 2019-04-09¶

Changed¶

Added¶

1.2.0 - 2017-08-08¶

Changed¶

1.1.0 - 2016-07-25¶

Added¶

1.0.0 - 2016-06-16¶

Added¶

Changed¶

Contributing¶

Preparing environment¶

Running tests¶

Using Tox¶

Manually¶

Coverage report¶

Building documentation¶

Preparing release¶

Preparing pre-release¶

Using Tox ¶