Welcome to voeventdb.server’s documentation!

Version 1.3.4

Contents:

Overview

Introduction

voeventdb.server is a database-store and accompanying RESTful query API for archiving and retrieving VOEvent packets.

There is an accompanying Python client-library for end-users (i.e. astronomers), voeventdb.remote.

A what?

(If you’ve never heard of a VOEvent, you may want to read this first).

This package contains code for building a database of transient astronomical events, for which alerts or follow-up data have been sent via the VOEvent network. It also provides a RESTful API, which means you can query the database remotely via the internet, either using your web-browser or your programming language of choice.

This serves two main purposes:

  • It allows people distributing or monitoring VOEvent packets to ‘catch up’ with any missed data, in the event of a network or systems outage.
  • It allows astronomers to search through the archive of VOEvents. This can be useful for planning future observations, or looking for related events in a particular region of sky, or mapping the distribution of detected events, etc.

Key Features

The key features of voeventdb, as far as the end-user is concerned, can be summarised as follows:

  • Ready-made Python client-library
    If you’ve never used a ‘REST’ interface before, don’t worry. Voeventdb comes with a ready-made Python client-library, voeventdb.remote, which simplifies querying the database to calling a few Python functions, as laid out in the accompanying tutorial.
  • Full XML storage
    You can always retrieve the original packet-data if you know the IVORN identifier (and assuming there’s a copy in the database).
  • Spatial filtering
    You can limit your search query to a ‘cone’, i.e. only returning events near to a given location. This functionality is powered by Sergey Koposov’s q3c.
  • Web of citations
    The database contains the information needed to determine reference / citation links between different VOEvent packets, so for example you can check if an exciting new transient announced in one packet has already been followed up by another VOEvent broadcasting observatory.
  • Various summary statistics at your fingertips
    View the number of VOEvents broken down by observatory, by month, by type, etc.
  • Extensive filters, applicable throughout
    Narrow your query to a particular type of alert, limit it to packets authored between certain dates, only return events which are cited by others, return all events citing one particular packet, etc etc. Filters and endpoints can be thought of as orthogonal, i.e. the same filter-set can be applied to any endpoint.
  • Loading / dumping of XML packets in plain-old-tarballs
    The package contains routines for handling BZ2-compressed tarballs containing the original XML packets. This allows for version migration, or exporting the archive to another tool entirely. It also keeps the backups remarkably compact - all 750,000 or so packets for the past 12 months weigh in at just under 100MB in compressed form.

Getting started

If you just want to query our copy of the database rather than running your own, then you should probably head over to the Python-client-package, voeventdb.remote, or jump straight to the relevant documentation.

If you would like to access the REST API directly, our copy of voeventdb is hosted and documented at http://voeventdb.4pisky.org/. You can find reference information on how to form your http-queries here.

If you want to run a local copy of voeventdb, or contribute to voeventdb’s development, see Installation.

Installation

Note

This page is intended for those who wish to install their own local copy of voeventdb. If you just want to query our copy of the database, things are much simpler - see Getting started.

All-in-one installation scripts for the DevOps enthusiast

If you’d like to install your own copy of voeventdb, you may be interested in the accompanying Ansible role that provides a pre-configured installation of the voeventdb database and REST API, optionally served via Apache, available here: https://github.com/timstaley/ansible-voeventdb.

The voeventdb role is demonstrated in conjunction with another role installing the Comet broker (to pull in the latest VOEvent packets) here: https://github.com/4pisky/voeventdb-deploy - in fact, those are the scripts which build http://voeventdb.4pisky.org/.

Note that, at time of writing, I’ve not had time to document these packages much - if you’d be interested in making use of them please drop me a line and I may be able to provide help / improve the associated docs.

Development installation

Alternately, if you want a local installation for development and testing purposes (or you just prefer the hands-on approach) then read on for some tips regarding manual installation.

Postgres Database Setup

Before the unit-tests or database setup will run, we need a PostgreSQL installation, and a database login with createdb rights. On a typical Debian system, install postgres with something like:

sudo apt-get install libpq-dev postgresql-9.3 postgresql-server-dev-9.3

You can check if you have database login and creation rights by running e.g.

createdb

Which will attempt to create a database named after your username. On a fresh Postgres installation you will get an access rights error - you need to login and create your user like so:

userfoo@machine: sudo su postgres
postgres@machine: psql
postgres=# create user userfoo with superuser;
postgres=# alter user userfoo with password 'userfoo';

For full details, consult the postgres docs.

Note the unit-tests assume by default a database username/password which are both the same as your login username, but this can be easily changed, see the module voeventdb.server.database.config.

A final word on Postgres setup - I recommend setting

timezone = 'UTC'

in postgresql.conf, this ensures that all timestamps are displayed in UTC both when querying the database via the command line, and when returning datetime objects via the SQLAlchemy API (full docs here).

Checking out and building q3c

We pull in q3c as a submodule, so unless you used a recursive git clone you will need to:

git submodule init
git submodule update

Then, to build q3c and install it into the PostgreSQL contrib folder:

cd external/q3c
make
sudo make install
Package Installation

Once all that’s out of the way, installation is trivial, simply run:

pip install .[all]

from the repository root, or (if you plan on development work):

pip install -e .[all]

Testing

voeventdb is accompanied by an extensive test-suite, powered by pytest.

To run the tests, you can run:

py.test -sv

From the repository root (see py.test -h for details), or you can use:

./runtests.py -sv

Which is a wrapper arount pytest which sets up some sensible logging defaults.

The tests can be run via tox, which additionally confirms a correct build-and-install process. Commits uploaded to github are automatically tested via Travis.

REST API v1 (apiv1) reference

Querying the REST API

The voeventdb web-interface is designed around widely used RESTful concepts, which means (simplifying grossly) that all the details of your data query are encoded in an HTTP URL. By making requests at that URL, you get back the data matching your query. You can try this out by following links and editing the URL in your browser, but typically you’ll want to grab data using a scripting library such as Python’s requests. [1]

[1]A ready-made Python-client library which wraps requests in a convenient fashion can be found at https://github.com/timstaley/voeventdb.remote.

Finding and using endpoints

The base URLs which represent different queries are known as endpoints - full listings for voeventdb can be found on the Endpoints page. Some useful places to start are the root endpoint, which provides a concise listing of the endpoints available, and the stream count endpoint, which serves as a sort of ‘contents’ page for the database.

Narrowing your query

By default, most endpoints return data on all VOEvents currently stored in the database. [2] To narrow down your query to a specific subset of the VOEvents, you can apply a selection of the available filters listed on the Query filters page. Filters are applied by adding key-value pairs as part of the query-string in your HTTP request.

For example, to return a count of the packets stored since the start of November 2015, which have been assigned the ‘observation’ role, you can form an HTTP address like:

http://voeventdb.4pisky.org/apiv1/count?authored_since=2015-11&role=observation

Though typically you would let your scripting library do the job of stitching together the various parts. See the Query filters page for more details.

Note

You can apply any filter (or combination of filters) to any endpoint, so (for example)

http://voeventdb.4pisky.org/apiv1/map/stream_count?authored_since=2015-11&role=observation

is also a valid query-URL (where we have replaced the /apiv1/count endpoint with /apiv1/map/stream_count).

[2]The exceptions are the single-packet endpoints:, which are intended to retrieve data pertaining to a single VOEvent.

URL-Encoding

Note that if you are accessing the single-packet endpoints:, or specifying a query-filter value which contains the # character, then you will need to use URL-encoding (because otherwise the query-value is indistinguishable from an incorrectly-formed URL). It’s simple to URL-encode the value using a web-based tool, or e.g. in Python:

import urllib
s = urllib.quote_plus("ivo://foo.bar/baz#quux")
print(s)

List-pagination controls

The database can easily handle millions of entries, so it makes sense to break up the return-data for queries which return lists of data. You can use pagination-keys in the same manner as query-keys (i.e. in the query-string) to control this:

class voeventdb.server.restapi.v1.definitions.PaginationKeys[source]

These query-keys control the ordering and subset (‘page’) of results.

Use limit and offset values in your querystring to control slicing, i.e. the subset of an ordered list returned in the current query.

(Only applies to list views, e.g. the IVORN listing endpoint.)

The keywords are adopted from SQL, cf http://www.postgresql.org/docs/9.3/static/queries-limit.html

Note that if no values are supplied, a default limit value is applied. (You can still check what it was, by inspecting the relevant value in the result-dict.)

limit = 'limit'

The maximum number of results returned for a single request.

offset = 'offset'

The number of rows ‘skipped’ before returning results for a request.

order = 'order'

Controls the ordering of results, before limit and offset are applied. Valid values are enumerated by the OrderValues class.

class voeventdb.server.restapi.v1.definitions.OrderValues[source]

Values that may be used in a querystring with the ‘order’ key.

E.g. By specifying order=author_datetime in a querystring, results are returned ordered by author_datetime (ascending, i.e. oldest first). By default, results are returned in database-id (ascending) order, which in effect means that the first results to be loaded into the database are returned first.

Each value has a pairing with a ‘-‘ prefix, implying reverse (descending) ordering.

author_datetime = 'author_datetime'
author_datetime_desc = '-author_datetime'

Order results by author_datetime (timestamp from the Who section). Default (‘ascending’) implies oldest-first.

id = 'id'
id_desc = '-id'

Order results by database-id (assigned as events are added to the database). This is the default setting.

ivorn = 'ivorn'
ivorn_desc = '-ivorn'

Order results by ivorn (alphabetically).

Returned content

class voeventdb.server.restapi.v1.viewbase.ResultKeys[source]

Most endpoints return a JSON-encoded dictionary. [3]

At the top level, the dictionary will contain some or all of the following keys:

[3](The exception is the XML-retrieval endpoint, obviously.)

Note

The key-strings can be imported and used in autocomplete-friendly fashion, for example:

from voeventdb.server.restapi.v1 import ResultKeys as rkeys
print rkeys.querystring
endpoint = 'endpoint'

The endpoint the query was made against.

limit = 'limit'

The maximum number of entries returned in a single request (Only applies to list-view endpoints.)

querystring = 'querystring'

A dictionary displaying the query-string values applied. (With urlencode-decoding applied as necessary.)

Note that each entry contains a list, as a query-key may be applied multiple times.

result = 'result'

The data returned by your query, either in dictionary or list format according to the endpoint.

See endpoint listings for detail.

url = 'url'

The complete URL the query was made against.

Endpoints

The apiv1 endpoints are listed below.

Most endpoints return a JSON-encoded dictionary; the ‘Result’ values documented below refer to the contents of the result entry in the JSON-dict. See Returned content for details.

Query endpoints

GET /apiv1/list/ivorn_ncites

(Link)

Result (list of 2-element lists):
[[ivorn, n_cited], ...]

Get rows containing citation counts. Row entries are:

  • IVORN of packet
  • Number of times this packet is cited by others
GET /apiv1/list/ivorn_nrefs

(Link)

Result (list of 2-element lists):
[[ivorn, n_refs], ...]

Get rows containing reference counts. Row entries are

  • IVORN of packet
  • Number of references to other packets, in this packet.
GET /apiv1/list/ivorn

(Link)

Result (list of strings):
[ ivorn1, ivorn2, ... ]

List of ivorns matching querystring. Number returned is limited by the limit parameter, which defaults to 100 (see List-pagination controls).

GET /apiv1/map/authored_month_count

(Link)

Result:
Dict: Mapping month -> packet counts per-month.

Here, ‘month’ refers to the month of the ‘authoring’ DateTime, i.e. the Who.Date element of the VOEvent. NB, may be None.

GET /apiv1/map/stream_role_count

(Link)

Result:
Nested dict: Mapping stream -> role -> packet counts per-stream-and-role.
GET /apiv1/map/stream_count

(Link)

Result:
Dict: Mapping stream -> packet counts per-stream.
GET /apiv1/map/role_count

(Link)

Result:
Dict: Mapping role -> packet counts per-role.
GET /apiv1/count

(Link)

Result (int):
Number of packets matching querystring.

Returns total number of packets in database if the querystring is blank.

GET /apiv1/

(Link)

API root url. Shows a list of active endpoints.

Single-packet endpoints

Unlike the query-endpoints, these return data on a single packet.

The IVORN of the desired packet should be appended to the endpoint-URL in URL-encoded form.

GET /apiv1/packet/synopsis/(path: url_encoded_ivorn)
GET /apiv1/packet/synopsis/

(Link)

Result:

Nested dict providing key details, e.g.:

{"coords": [
                {
                    "dec": 10.9712,
                    "error": 0.05,
                    "ra": 233.7307,
                    "time": "2015-10-01T15:04:22.930000+00:00"
                },
                ...
            ],
 "refs":   [
                {
                    "cite_type": u"followup",
                    "description": "This is the XRT Position ...",
                    "ref_ivorn": "ivo://nasa.gsfc.gcn/SWIFT#BAT_..."
                },
                ...
            ],
 "voevent": {
                "author_datetime": "2015-10-01T15:04:46+00:00",
                "author_ivorn": "ivo://nasa.gsfc.tan/gcn",
                "ivorn": "ivo://nasa.gsfc.gcn/SWIFT#BAT_GRB_Pos_657286-112",
                "received": "2015-11-19T20:41:38.226431+00:00",
                "role": "observation",
                "stream": "nasa.gsfc.gcn/SWIFT",
                "version": "2.0"
            }
 "relevant_urls": [ "http://address1.foo.bar",
                    "http://address2.foo.bar"
                    ]
}

Returns some key details for the packet specified by IVORN.

The required IVORN should be appended to the URL after /synopsis/ in URL-encoded form.

GET /apiv1/packet/xml/(path: url_encoded_ivorn)
GET /apiv1/packet/xml/

(Link)

Returns the XML packet contents stored for a given IVORN.

The required IVORN should be appended to the URL after /xml/ in URL-encoded form.

Query filters

These filters can be applied by providing key-value pairs as part of the query string in your HTTP request, where the key is equal to the querystring_key entry below, and the value is formatted like one of the example_values. See Narrowing your query for examples.

class voeventdb.server.restapi.v1.filters.AuthoredSince[source]

Return only VOEvents with a Who.Date entry dated after the given time.

(Time-range is inclusive, i.e. >=)

Date-time strings passed should be in a format parseable by the iso8601.parse_date() function (see example_values).

querystring_key = u'authored_since'
example_values = [u'2015-10-09T21:34:19', u'2015-10-09', u'2015-10']
class voeventdb.server.restapi.v1.filters.AuthoredUntil[source]

Return only VOEvents with a Who.Date entry dated before the given time.

(Time-range is inclusive, i.e. <=)

Date-time strings passed should be in a format parseable by the iso8601.parse_date() function (see example_values).

querystring_key = u'authored_until'
example_values = [u'2015-10-09T21:34:19', u'2015-10-09', u'2015-10']
class voeventdb.server.restapi.v1.filters.CitedByAny[source]

Return only VOEvents which are cited by another VOEvent in the database.

Applied via query-strings cited=true or cited=false

querystring_key = u'cited'
example_values = [u'true', u'false']
class voeventdb.server.restapi.v1.filters.ConeSearch[source]

Return only VOEvents with co-ords in the given cone.

Cone specified as a 3-element list in JSON format:

[ra,dec,radius]

(values in decimal degrees).

querystring_key = u'cone'
example_values = [u'[10,20,5]', u'[359.9,-30,5]']
class voeventdb.server.restapi.v1.filters.CoordsAny[source]

Return only VOEvents which have / do not have associated co-ord positions.

Applied via query-strings coord=true or coord=false

querystring_key = u'coord'
example_values = [u'true', u'false']
class voeventdb.server.restapi.v1.filters.DecGreaterThan[source]

Return VOEvents with position with Dec greater than given value.

Dec should be specified in decimal degrees.

querystring_key = u'dec_gt'
example_values = [u'0', u'-45.123']
class voeventdb.server.restapi.v1.filters.DecLessThan[source]

Return VOEvents with position with Dec less than given value.

Dec should be specified in decimal degrees.

querystring_key = u'dec_lt'
example_values = [u'0', u'-45.123']
class voeventdb.server.restapi.v1.filters.IvornContains[source]

Return only VOEvents which have the given substring in their IVORN.

querystring_key = u'ivorn_contains'
example_values = [u'BAT_GRB_Pos', u'XRT']
combinator(filters)[source]

AND

class voeventdb.server.restapi.v1.filters.IvornPrefix[source]

Return only VOEvents where the IVORN begins with the given value.

Note that the value passed should be URL-encoded if it contains the # character e.g.:

quote_plus('ivo://nvo.caltech/voeventnet/catot#1404')
querystring_key = u'ivorn_prefix'
example_values = [u'ivo://nasa.gsfc.gcn', 'ivo%3A%2F%2Fnvo.caltech%2Fvoeventnet%2Fcatot%231404']
combinator(filters)[source]

OR

class voeventdb.server.restapi.v1.filters.RefAny[source]

Return only VOEvents which make / don’t make reference to any other VOEvents.

Applied via query-strings ref_any=true or ref_any=false. NB ‘true’/’false’ string-values are case-insensitive, so e.g. ‘true’, ‘True’, ‘TRUE’, ‘tRUe’ are all valid.

querystring_key = u'ref_any'
example_values = [u'true', u'True', u'false']
class voeventdb.server.restapi.v1.filters.RefContains[source]

Return VOEvents which reference an IVORN containing the given substring.

querystring_key = u'ref_contains'
example_values = [u'BAT_GRB_Pos', u'GBM_Alert']
combinator(filters)[source]

OR

class voeventdb.server.restapi.v1.filters.RefExact[source]

Return only VOEvents which contain a ref to the given (url-encoded) IVORN.

querystring_key = u'ref_exact'
example_values = ['ivo%3A%2F%2Fnasa.gsfc.gcn%2FSWIFT%23BAT_GRB_Pos_649113-680', 'ivo%3A%2F%2Fnasa.gsfc.gcn%2FFermi%23GBM_Alert_2015-08-10T14%3A49%3A38.83_460910982_1-814']
combinator(filters)[source]

OR

class voeventdb.server.restapi.v1.filters.RoleEquals[source]
querystring_key = u'role'
example_values = [u'observation', u'utility', u'test']
combinator(filters)[source]

OR

class voeventdb.server.restapi.v1.filters.StreamEquals[source]
querystring_key = u'stream'
example_values = [u'nasa.gsfc.gcn#SWIFT', u'nvo.caltech/voeventnet/catot']
combinator(filters)[source]

OR

Change history

1.3.2 (2016/11/28) - Mirror upstream changes in voevent-parse

Minor update to mirror API changes in voevent-parse 1.0.

1.3.1 (2016/11/14) - Fix issue with UTF-8 encoded unicode chars in VOEvents

Fix an issue where non-ascii characters in XML packets broke tarball-dumping, because Postgres returns the data as unicode, and we need to explicitly encode that to a UTF-8 bytestring before the tarball code will accept it.

1.3.0 (2016/11/09) - Add parsing of barycentric event co-ords

Make use of voevent-parse 0.9.8 (and hence astropy 1.2.x) to apply TDB->UTC time conversion where relevant (e.g. GAIA event timestamps).

Also relaxes list of allowed position formats.

Note

VOEventDB does not currently apply any conversion to co-ordinates as read in from the packets**, this may produce small inaccuracies for close objects. Those needing high-precision ICRS co-ordinates should use coarse spatial queries and perform their own position parsing / transformation from packet-XML directly. See docstring and source under voeventdb.server.database.models.Coord.from_etree for more details and justification.

1.2.1 (2016/09/19) - Minor bugfix

Authored_month_count could bin entries incorrectly if postgres timezone was not configured to UTC.

If Postgres is configured correctly, then all was fine. But, if default timezone was configured to say, timezone=GB, (UTC+1 in summer), then datetime is fetched and truncated in the default (UTC+1) timezone. Therefore the month endpoints were off by the timezone offset compared to the regular ‘author_datetime’ queries which always enforce timezone evaluation, and so the ‘authored_month_count’ endpoint produced conflicting results to ‘count’ with relevant datetime filters applied.

1.2.0 (2016/09/15) - Tarball-dump-script enhancements

Add datetime-range filtering options to tarball-dump script. Also change default tarball size to avoid unreasonable RAM requirements, document typical RAM usage.

Indices and tables