Welcome to Picasso’s documentation!

Contents:

Picasso

A CNN model visualizer

Quickstart

Picasso uses Python 3.5+ so use a virtual environment if necessary (e.g. virtualenv env --python=python3) and activate it!

  1. Install with pip or from source.

    With pip:

    pip install picasso-viz
    

    From the repository:

    git clone git@github.com:merantix/picasso.git
    cd picasso
    pip install -e .
    

    Note: you’ll need the Tensorflow backend for Keras for these examples to work. Make sure your ~/.keras/keras.json file looks like:

    {
        "backend": "tensorflow",
        "image_dim_ordering": "tf",
        "floatx": "float32",
        "epsilon": 1e-07
    }
    
  2. Optional (untested!): install Tensorflow with GPU support

    pip uninstall tensorflow
    pip install --upgrade tensorflow-gpu
    
  3. Start the Flask server

    export FLASK_APP=picasso
    flask run
    

    Point your browser to 127.0.0.1:5000 and you should see the landing page! When you’re done, Ctrl+C in the terminal to kill your Flask server.

  4. By default, the visualizer starts a Keras MNIST example. We’ve also included a Keras VGG16 example. To use, it you’ll need to get the VGG16 graph and weights. We’ve included a small script to do this.

    1. Setup VGG16:

      python picasso/examples/keras-vgg16/prepare_model.py
      

      NOTE: if you installed with pip, you’ll need to find the location of this file in the site packages. pip show picasso_viz will tell you the location. For instance, if pip show picasso_viz shows you /home/ryan/test/env/lib/python3.5/site-packages, then the above command should be:

      python /home/ryan/test/env/lib/python3.5/site-packages/picasso/examples/keras-vgg16/prepare_model.py
      

      If this script fails, you might be behind a proxy. You can download the necessary files manually.

      mkdir ~/.keras/models # If directory doesn't exist
      wget --no-check-certificate -P ~/.keras/models/ https://github.com/fchollet/deep-learning-models/releases/download/v0.1/vgg16_weights_tf_dim_ordering_tf_kernels.h5
      wget --no-check-certificate -P ~/.keras/models/ https://s3.amazonaws.com/deep-learning-models/image-models/imagenet_class_index.json
      

      Run the script again, and you should be good to go!

    2. Point to the correct configuration (making sure to use the correct path to your directory):

      export PICASSO_SETTINGS=/absolute/path/to/repo/picasso/picasso/examples/keras-vgg16/config.py
      

      Again, if you installed with pip install picasso-viz, this will look something like:

      export PICASSO_SETTINGS=/home/ryan/test/env/lib/python3.5/site-packages/picasso/examples/keras-vgg16/config.py
      

      You can check the pip show picasso_viz command for the base directory.

    3. Start Flask flask run. If it worked, the “Current checkpoint” label should have changed on the landing page.

Building the docs

Assuming you’ve cloned the repository, install the required packages:

pip install -e .[docs]

Then build them:

cd docs/
make html

Then you can open _build/html/index.html in your browser of choice.

Running the tests

Install the test requirements:

pip install -e .[test]

Then run with:

py.test

Notes

  1. This should be considered alpha software. You will encounter bugs and issues. Don’t deploy this to a live server, probably…
  2. Models generated on Keras using the Theano backend should in principle be supported. The only difference is the array ordering of convolutions. I haven’t tried this yet though, so an extra config parameter may be needed.

Credits

This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template.

Making your own visualizations

Picasso is made with ease of adding new models in mind. This tutorial will show you how to make a new visualization from scratch. Our visualization will be based on the very simple ClassProbabilities (see ClassProbabilities code) visualization, along with its HTML template.

Setup

Every visualization requires a class defining its behavior and an HTML template defining its layout. You can put them in the visualizations and templates folder respectively. It’s important that the class name and HTML template name are the same.

For our example, FunViz, we’ll need picasso/visualizations/fun_viz.py:

from picasso.visualizations.base import BaseVisualization


class FunViz(BaseVisualization):

    DESCRIPTION = 'A fun visualization!'

    def make_visualization(self, inputs, output_dir, settings=None):
        pass

and picasso/templates/FunViz.html:

{% extends "result.html" %}
{% block vis %}
your visualization html goes here
{% endblock %}

Some explanation for the FunViz class in fun_viz.py: All visualizations should inherit from BaseVisualization. You can also add a description which will display on the landing page.

Some explanation for FunViz.html: The web app is uses Flask, which uses Jinja2 templating. This explains the funny {% %} delimiters. The {% extends "result.html" %} just tells the your page to inherit from a boilerplate. All your html should sit within the vis block.

You can even start the app at this point (see Quickstart). You should see your visualization in the menu.

_images/menu.png

Are we having fun yet? ☆(◒‿◒)☆ YES

If you try to upload images, you will get an error. This is because the visualization doesn’t actually return anything to visualize. Let’s fix that.

Add visualization logic

Our visualization should actually do something. It’s just going to compute the class probabilities and pass them back along to the web app. So we’ll add:

from picasso.visualizations.base import BaseVisualization


class FunViz(BaseVisualization):

    DESCRIPTION = 'A fun visualization!'

    def make_visualization(self, inputs, output_dir):
        pre_processed_arrays = self.model.preprocess([example['data']
                                                     for example in inputs])
        predictions = self.model.sess.run(self.model.tf_predict_var,
                                          feed_dict={self.model.tf_input_var:
                                                     pre_processed_arrays})
        filtered_predictions = self.model.decode_prob(predictions)
        results = []
        for i, inp in enumerate(inputs):
            results.append({'input_file_name': inp['filename'],
                            'predict_probs': filtered_predictions[i]})
        return results

Let’s go line by line:

...

class FunViz(BaseVisualization):
    ...

    def make_visualization(self, inputs, output_dir):
        pre_processed_arrays = self.model.preprocess([example['data']
                                                     for example in inputs])
        ...

inputs are sent to the visualization class as a list of {'filename': ... , 'data': ...} dictionaries. The data are PIL Images created from raw data that the user has uploaded to the webapp. The preprocess method of model simply turns the input images into appropriately-sized arrays for the input of whichever computational graph you are using. Therefore, pre_processed_arrays is an array with the first dimension equal to the number of inputs, and subsequent dimensions determined by the preprocess function.

...

class FunViz(BaseVisualization):
    ...

    def make_visualization(self, inputs, output_dir):
        pre_processed_arrays = self.model.preprocess([example['data']
                                                     for example in inputs])
        predictions = self.model.sess.run(self.model.tf_predict_var,
                                          feed_dict={self.model.tf_input_var:
                                                     pre_processed_arrays})
        ...

Here’s where we actually do some computation to be used in the visualization. Note that the model object exposes the Tensorflow session (regardless of if the backend is Keras or Tensorflow). We also store the input and output tensors with the model members tf_input_var and tf_predict_var respectively. Thus this is just a standard Tensorflow run which will return an array of dimension n x c where n is the number of inputs, and c is the number of classes.

...

class FunViz(BaseVisualization):
    ...

    def make_visualization(self, inputs, output_dir):
        pre_processed_arrays = self.model.preprocess([example['data']
                                                     for example in inputs])
        predictions = self.model.sess.run(self.model.tf_predict_var,
                                          feed_dict={self.model.tf_input_var:
                                                     pre_processed_arrays})
        filtered_predictions = self.model.decode_prob(predictions)
        ...

decode_prob is another model-specific method. It gives us back the class labels from the predictions array. The format will be list of dictionaries in the format [{'index': class_index, 'name': class_name, 'prob': class_probability}, ...]. It will also only return the top class predictions (this comes in handy when using models like VGG16, which has 1000 classes).

...

class FunViz(BaseVisualization):
    ...

    def make_visualization(self, inputs, output_dir):
        pre_processed_arrays = self.model.preprocess([example['data']
                                                     for example in inputs])
        predictions = self.model.sess.run(self.model.tf_predict_var,
                                          feed_dict={self.model.tf_input_var:
                                                     pre_processed_arrays})
        filtered_predictions = self.model.decode_prob(predictions)
        results = []
        for i, inp in enumerate(inputs):
            results.append({'input_file_name': inp['filename'],
                            'predict_probs': filtered_predictions[i]})
        return results

Here we arrange the results to pass back to the webapp. In our case, we just return a list of dictionaries which hold the original filename, and the formatted prediction results. The exact structure isn’t so important, but you’ll have to deal with it when you write your HTML template, so try to keep it manageable. Now you’ll be able to see your result page from earlier.

_images/result_nohtml.png

At least it’s fast, right?

Of course, we haven’t told the template how to display the results yet. Let’s get down to it.

Configure the HTML template

We need to specify how to layout our visualization. Here are the lines we’ll add:

{% extends "result.html" %}
{% block vis %}
<table>
{% for result in results %}
    <tr>
      <td><b> {{ result.filename }} </b></td>
      {% for predict_prob in result.predict_probs %}
        <td><b> {{ predict_prob.name }} </b></td>
      {% endfor %}
    </tr>
    <tr>
      <td>
        <img src="inputs/{{ result.filename }}" style="width:244px;height:244px;"/>
      </td>
      {% for predict_prob in result.predict_probs %}
        <td> {{ predict_prob.prob }} </td>
      {% endfor %}
    </tr>
{% endfor %}
</table>
{% endblock %}

Let’s look at the pieces separately again:

{% extends "result.html" %}
{% block vis %}
<table>
{% for result in results %}
    <tr>
      <td><b> {{ result.filename }} </b></td>
      {% for predict_prob in result.predict_probs %}
        <td><b> {{ predict_prob.name }} </b></td>
      {% endfor %}
    </tr>
    <tr>
      <td>
        <img src="inputs/{{ result.filename }}" style="width:244px;height:244px;"/>
      </td>
      {% for predict_prob in result.predict_probs %}
        <td> {{ predict_prob.prob }} </td>
      {% endfor %}
    </tr>
{% endfor %}
</table>
{% endblock %}

Every visualization gets a results object from the web app. The results object will have the exact same structure as the return value of the make_visualization method of your visualization class. Since we returned a list, we iterate over it with this for-loop to generate the rows of the table.

{% extends "result.html" %}
{% block vis %}
<table>
{% for result in results %}
    <tr>
      <td><b> {{ result.filename }} </b></td>
      {% for predict_prob in result.predict_probs %}
        <td><b> {{ predict_prob.name }} </b></td>
      {% endfor %}
    </tr>
    <tr>
      <td>
        <img src="inputs/{{ result.filename }}" style="width:244px;height:244px;"/>
      </td>
      {% for predict_prob in result.predict_probs %}
        <td> {{ predict_prob.prob }} </td>
      {% endfor %}
    </tr>
{% endfor %}
</table>
{% endblock %}

There are actually two rows per result. One with the filename and class labels, and one with the input image and class probabilities. Let’s look at each in turn.

{% extends "result.html" %}
{% block vis %}
<table>
{% for result in results %}
    <tr>
      <td><b> {{ result.filename }} </b></td>
      {% for predict_prob in result.predict_probs %}
        <td><b> {{ predict_prob.name }} </b></td>
      {% endfor %}
    </tr>
    <tr>
      <td>
        <img src="inputs/{{ result.filename }}" style="width:244px;height:244px;"/>
      </td>
      {% for predict_prob in result.predict_probs %}
        <td> {{ predict_prob.prob }} </td>
      {% endfor %}
    </tr>
{% endfor %}
</table>
{% endblock %}

The first column has the filename and the class name headers. The for-loop loops over the result.predict_prob list of predictions (which we generated in make_visualization) and puts each class header in a cell.

{% extends "result.html" %}
{% block vis %}
<table>
{% for result in results %}
    <tr>
      <td><b> {{ result.filename }} </b></td>
      {% for predict_prob in result.predict_probs %}
        <td><b> {{ predict_prob.name }} </b></td>
      {% endfor %}
    </tr>
    <tr>
      <td>
        <img src="inputs/{{ result.filename }}" style="width:244px;height:244px;"/>
      </td>
      {% for predict_prob in result.predict_probs %}
        <td> {{ predict_prob.prob }} </td>
      {% endfor %}
    </tr>
{% endfor %}
</table>
{% endblock %}

The second row contains the input image and the actual numerical probabilities. Note the inputs/ in the img tag. All input images are stored here by the web app.

_images/basic_vis.png

Sooo beautiful ⊂◉‿◉つ

Similarly, there is an outputs/ folder (not shown in this example). Its path is passed to the visualization class as output_dir. Anything the visualization stores there is also available to the template (for example, additional images needed for the visualization).

Add some settings

Maybe we’d like the user to be able to limit the number of classes shown. We can easily do this by adding an ALLOWED_SETTINGS property to the FunViz class.

from picasso.visualizations import BaseVisualization


class FunViz(BaseVisualization):

    ALLOWED_SETTINGS = {'Display': ['1', '2', '3']}

    DESCRIPTION = 'A fun visualization!'

    @property
    def display(self):
        return int(self._display)

    def make_visualization(self, inputs, output_dir, settings=None):
        pre_processed_arrays = self.model.preprocess([example['data']
                                                     for example in inputs])
        predictions = self.model.sess.run(self.model.tf_predict_var,
                                          feed_dict={self.model.tf_input_var:
                                                     pre_processed_arrays})
        filtered_predictions = self.model.decode_prob(predictions)
        results = []
        for i, inp in enumerate(inputs):
            results.append({'input_file_name': inp['filename'],
                            'predict_probs': filtered_predictions[i][:self.display})
        return results

The ALLOWED_SETTINGS dict tells the web app what to display on the settings page. The names of these settings will be turned into lowercase properties preceeded by an underscore. Thus, “Display” becomes _display. You should implement a property function to cast the string to the correct type.

A page to select the settings will automatically be generated.

_images/setting.png

The automatically generated settings page

_images/with_settings.png

It works! ヽ(^◇^*)/

Add some styling

The template that FunViz.html derives from imports Bootstrap, so you can add some fancier styling if you like!

{% extends "result.html" %}
{% block vis %}
<table class="table table-sm table-striped">
     <tbody>
     {% for result in results %}
    <tr>
      <td align="center"><b> {{ result.filename }} </b></td>
      {% for predict_prob in result.predict_probs %}
        <td align="center"><b> {{ predict_prob.name }} </b></td>
      {% endfor %}
    </tr>
    <tr>
      <td align="center">
        <img src="inputs/{{ result.filename }}" style="width:244px;height:244px;"/>
      </td>
      {% for predict_prob in result.predict_probs %}
        <td class="vert-align" align="center"> {{ predict_prob.prob }} </td>
      {% endfor %}
    </tr>
     {% endfor %}
     </tbody>
</table>
{% endblock %}

Further Reading

For more complex visualizations, see the examples in the visualizations module.

Using your own models

We include three examples for you to try: a model trained on the MNIST dataset for both Keras and Tensorflow, and a Keras VGG16 model. We’ve tried to make it as simple as possible, but we do make a few assumptions:

  1. Your graph has a definitive entry and exit point. Specifically, a placeholder tensor for some kind of input and output operation (typically image arrays and class probabilities, respectively).
  2. These placeholders are of unspecified length.
  3. The portion of the computational graph you’re interested in requires no other inputs.

If you built your model with Keras using a Sequential model, you should be more or less good to go. If you used Tensorflow, you’ll need to manually specify the entry and exit points [1].

Your model data

You can specify the data directory with the MODEL_LOAD_ARGS.data_dir setting (see Settings). This directory should contain the Keras or Tensorflow checkpoint files. If multiple checkpoints are found, the latest one will be used (see example Keras model code).

Utility functions

In addition to the graph and weight information of the model itself, you’ll need to define a few functions to help the visualization interact with user input, and interpret raw output from your computational graph. These are arbitrary python functions, and their locations can be specified in the Settings.

We’ll draw from the Keras MNIST example for this guide. All custom models from the relevant model: either KerasModel or TensorflowModel.

Preprocessor

The preprocessor takes images uploaded to the webapp and converts them into arrays that can be used as inputs to your model. The Flask app will haved converted them to PIL Image objects.

import numpy as np
from PIL import Image

from picasso.models.keras import KerasModel

MNIST_DIM = (28, 28)

class KerasMNISTModel(KerasModel):

    def preprocess(self, raw_inputs):
        image_arrays = []
        for target in targets:
            im = target.convert('L')
            im = im.resize(MNIST_DIM, Image.ANTIALIAS)
            arr = np.array(im)
            image_arrays.append(arr)

        all_targets = np.array(image_arrays)
        return all_targets.reshape(len(all_targets),
                                   MNIST_DIM[0],
                                   MNIST_DIM[1], 1).astype('float32') / 255

Specifically, we have to convert an arbitrary input color image to a float array of the input size specified with MNIST_DIM.

Class Decoder

Class probabilities are usually returned in an array. For any visualization where we use classification, it’s much nicer to have the class labels available. This method simply attaches the labels to computed probabilities.

class KerasMNISTModel(KerasModel):

    ...

    def decode_prob(self, class_probabilities):
        results = []
        for row in class_probabilities:
            entries = []
            for i, prob in enumerate(row):
                entries.append({'index': i,
                                'name': str(i),
                                'prob': prob})

            entries = sorted(entries,
                             key=itemgetter('prob'),
                             reverse=True)[:self.top_probs]

            for entry in entries:
                entry['prob'] = '{:.3f}'.format(entry['prob'])
            results.append(entries)
        return results

results is then a list of dicts in the format [{'index': class_index, 'name': class_name, 'prob': class_probability}, ...]. In the case of the MNIST dataset, the index is the same as the class name (digits 0-9).

[1]We hope to remove these limitations in the future to accomodate a wider variety of possible graph topologies while still maintaining separation between the visualization and model implementation as much as possible.

Settings

Application settings are managed by Flask. This means you can use environment variables or a configuration file.

To specify your own configuration, use the PICASSO_SETTINGS environment variable. Let’s look at the Tensorflow MNIST example.

export PICASSO_SETTINGS=/absolute/path/to/repo/picasso/picasso/examples/tensorflow/config.py

Tells the app to use this configuration instead of the default one. Inside config.py, we have:

import os

base_dir = os.path.split(os.path.abspath(__file__))[0]

MODEL_CLS_PATH = os.path.join(base_dir, 'model.py')
MODEL_CLS_NAME = 'TensorflowMNISTModel'
MODEL_LOAD_ARGS = {
    'data_dir': os.path.join(base_dir, 'data-volume'),
    'tf_input_var': 'convolution2d_input_1:0',
    'tf_predict_var': 'Softmax:0',
}

Any lowercase line is ignored for the purposes of determining a setting. MODEL_LOAD_ARGS will pass the arguments along to the model’s load function.

For explanations of each setting, see picasso.config.

picasso

picasso package

Subpackages

picasso.interfaces package
Submodules
picasso.interfaces.rest module
picasso.interfaces.web module
Module contents
picasso.models package
Submodules
picasso.models.base module
picasso.models.keras module
picasso.models.tensorflow module
Module contents
picasso.visualizations package
Submodules
picasso.visualizations.base module
picasso.visualizations.class_probabilities module
picasso.visualizations.partial_occlusion module
picasso.visualizations.saliency_maps module
Module contents

Submodules

picasso.config module

picasso.utils module

Module contents

API

Since v0.2.0, Picasso allows you to call parts of its functionality via an API. The API is intended to be RESTful and provides responses as JSON. The following chapter gives you some insight on how to use the API.

Currently the session is stored in a cookie to allow reuse of uploaded images and separate the user space on the server.

All files referenced in the API can be directly accessed via /inputs/<filename> and /outputs/<filename>.

GET /api/

Right now this is only a placeholder. It could potentially be used to start a session to authenticate a user and to display the API version.

curl "localhost:5000/api/"

Output:

{
  "message": "Picasso v0.2.0. Developer documentation coming soon!",
  "version": "v0.2.0"
}

GET /api/app_state

Get information about the current backend and app information.

app_title Name of the App (e.g. Picasso)
model_name Name of the model
latest_ckpt_name Name of the latest model checkpoint
latest_ckpt_time Time of last update of the model
curl localhost:5000/api/app_state -b /path/to/cookie -c /path/to/cookie

Output:

{
  "app_title": "Picasso Visualizer",
  "model_name": "KerasMNISTModel",
  "latest_ckpt_name": "MNIST-weights.hdf5",
  "latest_ckpt_time": "2017-05-31 23:29:48"
}

POST /api/images

Upload an image. It returns the filename and a UID in JSON

curl -F "file=@/path/to/image.png" localhost:5000/api/images -b /path/to/cookie -c /path/to/cookie

Output:

{
  "file": "image.png",
  "ok": "true",
  "uid": 0
}

GET /api/images

List all images uploaded via this API

curl "localhost:5000/api/images" -b /path/to/cookie -c /path/to/cookie

Output:

{
  "images": [
    {
      "filename": "Screen_Shot_2016-11-08_at_22.57.51.png",
      "uid": 0
    },
    {
      "filename": "Image.png",
      "uid": 1
    }
  ]
}

GET /api/visualizers

List all available visualizers

curl "localhost:5000/api/visualizers" -b /path/to/cookie -c /path/to/cookie

Output:

{
  "visualizers": [
    {
      "name": "ClassProbabilities"
    },
    {
      "name": "PartialOcclusion"
    },
    {
      "name": "SaliencyMaps"
    }
  ]
}

GET /api/visualizers/<vis_name>

List all available settings for visualizer <viz_name>

curl "localhost:5000/api/visualizers/PartialOcclusion" -b /path/to/cookie -c /path/to/cookie

Output:

{
  "settings": {
    "Occlusion": [
      "grey",
      "black",
      "white"
    ],
    "Strides": [
      "2",
      "5",
      "10",
      "20",
      "30"
    ],
    "Window": [
      "0.50",
      "0.40",
      "0.30",
      "0.20",
      "0.10",
      "0.05"
    ]
  }
}

returns an empty settings object when no settings available:

{
  "settings": {}
}

GET /api/visualize

This endpoint needs at least 2 arguments (image=X and visualizer=Y) in the query string. Each response is guaranteed to have at least the following attributes:

input_file_name String  
predict_probs List of probabilities  
has_output boolean if this is True the output will also have a list output_file_names
has_processed_input boolean if this is True the output will also have an attribute processed_input_file_name
curl "localhost:5000/api/visualize?image=0&visualizer=PartialOcclusion" -b /path/to/cookie -c /path/to/cookie

output:

{
  "has_output": true,
  "has_processed_input": true,
  "input_file_name": "test.png",
  "output_file_names": [
    "1504440185.6014730_test.png",
    "1504440185.6964661_test.png",
    "1504440185.7823882_test.png",
    "1504440185.86981823_test.png",
    "1504440185.9575094_test.png"
  ],
  "predict_probs": [
    {
      "index": 8,
      "name": "8",
      "prob": "0.171"
    },
    {
      "index": 6,
      "name": "6",
      "prob": "0.125"
    },
    {
      "index": 2,
      "name": "2",
      "prob": "0.122"
    },
    {
      "index": 5,
      "name": "5",
      "prob": "0.119"
    },
    {
      "index": 0,
      "name": "0",
      "prob": "0.098"
    }
  ],
  "processed_input_file_name": "1504440185.5588531test.png"
}

Contributing

Contributions are welcome, and they are greatly appreciated! Every little bit helps, and credit will always be given.

All contributors (submitting code) must agree to the Eclipse Foundation Contributor License Agreement. We can instruct you how to do this in your pull request.

You can contribute in many ways:

Types of Contributions

Report Bugs

Report bugs at https://github.com/merantix/picasso/issues.

If you are reporting a bug, please include:

  • Your operating system name and version.
  • Any details about your local setup that might be helpful in troubleshooting.
  • Detailed steps to reproduce the bug.

Fix Bugs

Look through the GitHub issues for bugs. Anything tagged with “bug” and “help wanted” is open to whoever wants to implement it.

Implement Features

Look through the GitHub issues for features. Anything tagged with “enhancement” and “help wanted” is open to whoever wants to implement it.

Write Documentation

picasso could always use more documentation, whether as part of the official picasso docs, in docstrings, or even on the web in blog posts, articles, and such.

Submit Feedback

The best way to send feedback is to file an issue at https://github.com/merantix/picasso/issues.

If you are proposing a feature:

  • Explain in detail how it would work.
  • Keep the scope as narrow as possible, to make it easier to implement.
  • Remember that this is a volunteer-driven project, and that contributions are welcome :)

Credits

Development Lead

Contributors

History

0.2.0 (2017-06-07)

  • Add RESTful API
  • Rework configuration loaders
  • Major refactor

0.1.2 (2017-06-07)

  • Fix Keras loading issues
  • Check tensorflow installation before installing

0.1.1 (2017-05-16)

  • First release on PyPI.

Indices and tables