Welcome to voeventdb.remote’s documentation!

Version 1.1.0

Contents:

Introduction

voeventdb.remote is a Python library for performing remote queries against a voeventdb REST interface, like the one hosted by the 4 Pi Sky research group at http://voeventdb.4pisky.org.

You can use this library to easily retrieve information about transient astronomical events, if that information was sent via the public VOEvent network.

For a fuller introduction including a list of features, see the main voeventdb.server docs. Alternatively, jump right in to the tutorial to learn by example.

Installation

voeventdb.remote is available via PyPi, which means that if you are using a virtualenv then you should be able to simply type:

pip install voeventdb.remote

at the command line. Alternatively, you can install into your user-area using:

pip install --user voeventdb.remote

Tutorial

The tutorials take the form of Jupyter notebooks; you can either read them online or download them to work with interactively. The notebook files can be found in the notebooks folder of the source repository, or downloaded using the links on each tutorial page.

Quickstart guide

Combined with the API reference docs, this notebook gives an overview of ``voeventdb.remote``, for those who are already familiar with Python, VOEvents, and the basic ideas around web-based API usage. A gentler introduction can be found in the rest of the tutorial notebooks.

In [ ]:
from __future__ import print_function
import voeventdb.remote as vr
import voeventdb.remote.apiv1 as apiv1
from voeventdb.remote.apiv1 import FilterKeys, OrderValues
from voeventdb.remote.helpers import Synopsis
from datetime import datetime
import pytz

In voeventdb.remote endpoints are represented by functions, and can be found under the apiv1 namespace:

In [ ]:
#Get a count of VOEvents in the database
apiv1.count()
In [ ]:
#List the IVORNs of the 2 VOEvents most recently inserted into the database
apiv1.list_ivorn(order=OrderValues.id_desc,
                 n_max=2)

We can apply filters to all endpoints except those returning a single packet. If no filters are passed then the result-set is effectively the entire database.

Filter-sets are defined as dictionaries:

In [ ]:
#Filter to only 'observation' packets authored since 2015/12/01.
my_filters = {
    FilterKeys.role:'observation',
    FilterKeys.authored_since:datetime(2015,12,1,tzinfo=pytz.UTC),
}
In [ ]:
#The `map_` endpoints return dictionaries summarising the matching result-set:
apiv1.map_stream_count(filters=my_filters)
In [ ]:
#Get the IVORN of the most recently authored SWIFT GRB alert:
my_filters = {
    FilterKeys.role:'observation',
    FilterKeys.ivorn_contains:'BAT_GRB', #case sensitive
}

swift_bat_grb_list = apiv1.list_ivorn(
    filters=my_filters,
    order=OrderValues.author_datetime_desc,
    n_max=1
)
swift_bat_grb_ivorn = swift_bat_grb_list[0]
swift_bat_grb_ivorn

To get more detail on a VOEvent packet, you can use the ‘packet_synopsis’ function:

In [ ]:
# Retrieve a 'synopsis' (nested dictionary) for the VOEvent packet,
grb_nested_dict = apiv1.packet_synopsis(swift_bat_grb_ivorn)
# And convert it to a user-friendly class-object:
grb_synopsis = Synopsis(grb_nested_dict)
print(grb_synopsis.ivorn)
print(grb_synopsis.author_ivorn)
print(grb_synopsis.author_datetime)
print(grb_synopsis.sky_events)

voeventdb.remote makes use of Astropy coordinates . So for example, the Synopsis helper class parses the event-position co-ordinates into an Astropy SkyCoord:

In [ ]:
sky_event=grb_synopsis.sky_events[0]

sky_event.position
In [ ]:
print(sky_event.position.ra.deg)
print(sky_event.position.ra.hms)

We can search for VOEvents with nearby co-ordinates to the Swift event we just looked up:

In [ ]:
from astropy.coordinates import SkyCoord, Angle
my_filters = {
    FilterKeys.role:'observation',
    FilterKeys.cone: (sky_event.position, Angle(1.0, unit='deg'))
    }
apiv1.map_stream_count(my_filters)

Or we run a search at co-ordinates of our choice (see the SkyCoord docs for parseable formats):

In [ ]:
from astropy.coordinates import SkyCoord, Angle

chosen_position = SkyCoord(45,45, unit='deg')
radius = Angle(1.0, unit='deg')
my_filters = {
    FilterKeys.role:'observation',
    FilterKeys.cone: (chosen_position,radius)
    }
apiv1.map_stream_count(my_filters)

Note we can also retrieve the original XML for the packet (and save it to disk, etc):

In [ ]:
raw_xml = apiv1.packet_xml(swift_bat_grb_ivorn)
print(raw_xml[:72])
print(' ... ')
with open('swift_voevent.xml','wb') as f:
    f.write(raw_xml)

Tutorial 1 - basic use of endpoints

** This notebook covers the basics of working with voeventdb.remote, demonstrating simple endpoint calls and the data-structures they return. **

(It also includes a few tips on using IPython’s autocomplete functionality, for newcomers to IPython / Jupyter Notebooks.)

First, let’s set up logging - by switching on ‘DEBUG’ level logging, we get to see the HTTP requests that are being made behind the scenes every time you run a query using voeventdb.remote. (NB we’ll be using the Python-3 style print operator.)

In [ ]:
from __future__ import print_function
import logging
logging.basicConfig(level=logging.DEBUG)

Next, library imports: Most of the user-facing code lives in voeventdb.remote.apiv1, but we’ll also alias voeventdb.remote to the shorthand vr:

In [ ]:
import voeventdb.remote as vr
import voeventdb.remote.apiv1 as apiv1
### Aside: Autocomple tion tricks For those new to IPython / notebooks: note you can use tab-comple tion to help you explore / save on typing (as long as you’ve already imported the relevant packages). For example, type:
apiv1.
in a notebook code-cell or IPython terminal and then hit Tab to see your available options. Likewise, in notebooks you can position your cursor on a function and hit Shift+Ta b to see a summary of function arguments, etc.

There are a couple of variables living in the top-level voeventdb.remote module; for example the default_host. You can set this to avoid specifying a host for every query. By default it’s set to point at the 4PiSky server:

In [ ]:
# For testing against a local-dev server:
# vr.default_host = 'http://localhost:5000'
print(vr.default_host)

OK, let’s query the database. We use different HTTP addresses, called endpoints, to give us different summaries or subsets of the database. We’ll start off by using apiv1.count to find out how many packets are stored in this database - if it’s empty, any other queries won’t do us much good:

In [ ]:
apiv1.count()

Getting on for a million packets, that’s quite lot! So there’s plenty here for us to work with.

IVORN’s and single-packet details

Next let’s try apiv1.list_ivorn, which returns a list of IVORN’s, unique idenitifiers which represent a VOEvent packet, analogous to how a URL represents a web-page.

In [ ]:
ivorn_list = apiv1.list_ivorn()
len(ivorn_list)
In [ ]:
#Take a look at the first 10
ivorn_list[:10]

We got back a list of IVORN’s as expected - but only 5000 of them, not a million. By default, voeventdb.remote applies a limit to the maximum number of entries returned by a list query, as defined in default_list_n_max:

In [ ]:
print(vr.default_list_n_max)

You can override this, or set it to 0 to simply fetch all matching results (though this can take a while).

In [ ]:
## How to override the default:
# vr.default_list_n_max = 2500
## Set n_max for a single query:
short_list = apiv1.list_ivorn(n_max=50)

If we know the IVORN of a packet, we can look up some details using the apiv1.packet_synopsis endpoint:

In [ ]:
ivorn = ivorn_list[0]
apiv1.packet_synopsis(ivorn)

And we can even retrieve the raw XML, (i.e. the original packet), so we can pull out any data specific to a given observatory, etc:

In [ ]:
xml = apiv1.packet_xml(ivorn)

A convenient way to inspect the xml-data is to use the voevent-parse library (which even has its very own tutorial).

In [ ]:
## A brief example, see the voevent-parse tutorial for more
# import voeventparse as vp
# voe_tree = vp.loads(xml)
# vp.pull_params(voe_tree)

Summary endpoints

So we can view individual packet IVORNS, and look up packet details, but that’s very fine-grained information - we don’t want to wade through all of them to figure out what’s in the database. We can use ‘summary’ endpoints to give us a higher-level overview. For example, we can organise VOEvents by ‘streams’ - basically the bit of the IVORN between the ivo:// prefix and the # symbol. The apiv1.map_stream_count endpoint tells us how many VOEvent packets belong to each stream:

In [ ]:
apiv1.map_stream_count()

Alternatively, we can slice up the packets by the authoring timestamp, binning by calendar month:

In [ ]:
apiv1.map_authored_month_count()

Or, we can divide them up by assigned role, which comes in three flavours:

In [ ]:
apiv1.map_role_count()

Similarly, apiv1.map_stream_role_count gives a breakdown of how many packets belong to each role, listed by stream.

Coming next …

So now that we have an idea of what’s in the database, how do we choose a category and drill-down to select specific VOEvents? For that we need filters, which are covered in tutorial 2.

Tutorial 2 - applying query-filters

We previously covered the basics of using data-endpoints with voeventdb.remote in tutorial 1.

** This notebook demonstrates use of filters to narrow down your query, and introduces a few convenient ‘helper classes’ for handling nested data-structures. **

As before, we’ll switch on ‘DEBUG’ level logging, to see the the HTTP requests go whizzing by.

In [ ]:
from __future__ import print_function
import logging
logging.basicConfig(level=logging.DEBUG)
In [ ]:
import voeventdb.remote as vr
import voeventdb.remote.apiv1 as apiv1

We’ve already briefly looked at the map_stream_count endpoint, and mentioned how VOEvents come in three flavours of role, ‘observation’, ‘utility’, and ‘test’. Let’s remind ourselves what the default map_stream_count output looks like:

In [ ]:
apiv1.map_stream_count()

Using filters

Quite obviously, a number of those streams are ‘junk’, they contain only test-packets used to verify that the VOEvent infrastructure is up and working correctly. For scientific work, we’ll want to filter those out.

Fortunately, we can ask the voeventdb server to do the filtering work for us. The voeventdb.remote library comes with an easy-to-use list of filters, stored as `voeventdb.remote.apiv1.FilterKeys <http://voeventdbremote.readthedocs.org/en/latest/reference/index.html#voeventdb.remote.apiv1.FilterKeys>`__. To see what’s available at a glance you can use the IPython tab-completion and doc-lookup tools, as in the cell below.

Full definitions of the filter-keys (and example filter-values) can be found in the voeventdb server docs, but we’ll cover most of them in these tutorial notebooks - read on.

In [ ]:
#Alias voeventdb.remote.apiv1.FilterKeys to just 'FilterKeys', for brevity
from voeventdb.remote.apiv1 import FilterKeys
In [ ]:
## To see the list of filters, you can use tab-completion:
## (Uncomment the following line and try it for yourself)
# FilterKeys.
## Or the ipython doc-lookup magic, by prefixing with ``??`` and running the cell:
# ??FilterKeys
Filtering by role

So: we were trying to filter out the test-packets. FilterKeys.role sounds promising. To apply a filter, or multiple filters, we simply define a dictionary with the filters we want to apply, and then pass it to the relevant query-function, like this:

In [ ]:
my_filters = { FilterKeys.role: 'observation' }
In [ ]:
apiv1.map_stream_count(my_filters)
Filtering by date

That results in a much shorter list, containing only scientifically interesting streams. Still, those numbers are pretty large (mainly for Swift). It might be useful to get a smaller representative sample. How many packets will we get if we limit our query to a single week?

In [ ]:
from datetime import datetime, timedelta
import pytz
start_date = datetime(2015,12,1,tzinfo=pytz.UTC)
my_filters = {
    FilterKeys.role: 'observation',
    FilterKeys.authored_since: start_date,
    FilterKeys.authored_until: start_date + timedelta(days=7)
    }
my_filters
In [ ]:
apiv1.map_stream_count(my_filters)
Filtering by stream

Ok, so there’s still a lot of Swift packets there. Let’s take a look at a sample of those, and see if we can break them up further. First, lets add another filter to limit our query to just Swift packets.

In [ ]:
my_filters[FilterKeys.stream] = 'nasa.gsfc.gcn/SWIFT'
my_filters

So now if we apply the filters to map_stream_count, we only get back one entry (the Swift stream):

In [ ]:
apiv1.map_stream_count(filters=my_filters)
Filters can be used across different query-endpoints

Not particularly helpful, but at least everything is working as expected. Now, the neat thing about the voeventdb filters is that they can be applied to any query-endpoint - we can just re-use the filter-dictionary with the apiv1.list_ivorn function to get back a list of IVORNs:

In [ ]:
swift_ivorns = apiv1.list_ivorn(filters=my_filters)
print("Retrieved",len(swift_ivorns),"IVORNs")
#Show just the first 10
swift_ivorns[:10]

That’s a long list, but there’s clearly a pattern to how the Swift IVORNs are formatted. We’ll use a little Python trickery (cf set, str.rsplit) to chop off the trailing ID numbers and sort them into sub-categories:

In [ ]:
swift_categories = set(ivorn.rsplit('_',1)[0] for ivorn in swift_ivorns)
swift_categories

Now we’re getting somewhere! We can clearly see the subcategories of Swift packets - BAT alerts, XRT positions, UVOT followup, etc.

Filtering by IVORN substring

We can use this knowledge to refine our filters, by filtering on a substring of the IVORN, using the ivorn_contains filter. For example, we might want to filter to just those IVORNs containing XRT positions (note this filter is case-sensitive):

In [ ]:
my_filters[FilterKeys.ivorn_contains] = 'XRT_Pos'
my_filters
In [ ]:
xrt_pos_ivorns = apiv1.list_ivorn(filters=my_filters)
print("Retrieved",len(xrt_pos_ivorns),"IVORNs")
xrt_pos_ivorns

As in tutorial 1, we can inspect the details of any given packet using the packet_synopsis endpoint - we’ll take a look at the first one. This packet makes a good example, as it includes details of the event co-ordinates and timestamp, and also references an earlier VOEvent:

In [ ]:
synopsis_dict = apiv1.packet_synopsis(xrt_pos_ivorns[0])
synopsis_dict

Ready-made ‘helper’ classes for parsing output

Nested dictionaries can be kind of a pain to work with. If you want, you can use voeventdb.remote’s `Synopsis <http://voeventdbremote.readthedocs.org/en/latest/reference/index.html#voeventdb.remote.helpers.Synopsis>`__ ‘helper’ class to parse this into an easy-to use object.

In [ ]:
from voeventdb.remote.helpers import Synopsis
In [ ]:
xrt_synopsis = Synopsis(synopsis_dict)
# Prints with nicer formatting, ordering of values:
print(xrt_synopsis)

Now we can easily access the values (with the ever-handy IPython autocompletion):

In [ ]:
xrt_synopsis.author_ivorn
In [ ]:
xrt_synopsis.references

One of the Synopsis class attributes is a list called sky_events. Each entry is a `SkyEvent <http://voeventdbremote.readthedocs.org/en/latest/reference/index.html#voeventdb.remote.helpers.SkyEvent>`__ class, which reprents a very basic set of information about an observed event: - estimated position, - error circle on the estimated position, - timestamp of the observed event.

The position coordinates and error-circle are represented by `astropy.coordinates <http://astropy.readthedocs.org/en/stable/coordinates/index.html>`__ classes, which come with a bunch of features related to formatting, distance calculations, frame-of-reference transformations, etc.

In [ ]:
xrt_synopsis.sky_events
In [ ]:
# List of 1, in this case. Grab the first (and only) element:
sky_event = xrt_synopsis.sky_events[0]
In [ ]:
print(type(sky_event.position))
sky_event.position
In [ ]:
print(type(sky_event.position_error))
sky_event.position_error.deg

Astropy coordinates come with all the usual weird and wonderful astronomical formatting options, see the astropy docs for details:

In [ ]:
print(sky_event.position.ra.deg)
print(sky_event.position.ra.hms)

Advanced usage: specifying multiple values for the same filter

Before we move on, it’s worth mentioning that some filters can take on multiple values. This is specified by defining the filter-value as a list - for example, to return all VOEvents with a role of ‘observation’ or ‘utility’ we can use the following:

In [ ]:
my_filters = {apiv1.FilterKeys.role: ['observation','utility']}
apiv1.map_stream_count(my_filters)

How does this work? Well, we can think of each entry in the list defining a separate filter. For the role value, these filters are combined in the logical ‘OR’ sense, so we get back combined counts for both ‘observation’ and ‘utility’ packets. You can check whether a filter accepts multiple values, and if they are combined via logical ‘OR’ or ‘AND’, by checking the filter-definitions page and looking for the combinator attribute.

Coming next …

We’ve seen how to narrow our search, locate packets of interest, and use helper-classes to easily access packet details. In tutorials 3 & 4, we’ll cover different ways of finding related VOEvents.

Tutorial 3 - Spatial filtering and Astropy coords

By now, you should have some idea of how to explore the various voeventdb.remote endpoints and apply filters.

** This notebook expands on the ‘cone-search’ filter, including how to use it with astropy.coordinates classes. **

In [ ]:
from __future__ import print_function
import logging
logging.basicConfig(level=logging.DEBUG)
In [ ]:
import voeventdb.remote as vr
import voeventdb.remote.apiv1 as apiv1
from voeventdb.remote.apiv1 import FilterKeys
from voeventdb.remote.helpers import Synopsis
from astropy.coordinates import Angle, SkyCoord

In the last tutorial, we inspected a Swift XRT event which gave us a position estimate and and error-circle. Let’s grab those details again:

In [ ]:
xrt_synopsis = Synopsis(apiv1.packet_synopsis('ivo://nasa.gsfc.gcn/SWIFT#XRT_Pos_666352-553'))
In [ ]:
sky_event = xrt_synopsis.sky_events[0]
print(sky_event)
print("Error-circle radius in degrees, arcminutes and arcseconds:", sky_event.position_error)

Filtering using a cone-search (AKA spatial queries)

Next, let’s see if there are any other recorded events with associated positions nearby. To do so, we’ll need to define a ‘cone’, a sky-position and circle around it to search in. For setting up a voeventdb.remote cone-filter, we can use a tuple of type

(astropy.coordinates.SkyCoord, astropy.coordinates.Angle)

A natural candidate is the position and position-error from the XRT event; like so:

In [ ]:
cone = (sky_event.position, sky_event.position_error)
cone

However, the XRT position has a really tight error-circle, about 5 arcseconds. Note that the cone-search will only return VOEvents with a best-estimate position within the cone - it does not take into account overlapping error-circles (at least for version 1!). This means that we could have a related event with a large error-circle, but which has a best-estimate position just outside the tiny XRT error-circle, and it wouldn’t be returned - so we have to use some judgement here. We’ll set the cone angular radius to half a degree, instead:

In [ ]:
cone = (sky_event.position, Angle(0.5, unit='deg'))
cone

OK, let’s see how that works:

In [ ]:
cone_filters = {
    FilterKeys.role: 'observation',
    FilterKeys.cone: cone
    }
In [ ]:
apiv1.map_stream_count(cone_filters)

A reasonable number. Let’s take a look:

In [ ]:
sorted(apiv1.list_ivorn(cone_filters))
Investigating ‘background’ event rates

So, we have a bunch of Swift packets related to the same event (ID 666352), some ‘BAT_SubSubThresh’ events, and a ‘Known_Pos’ event. It’s worth noting that ‘SubSubThresh’ events are extremely common and show up all over; in fact they currently make up about half the packets in the database:

In [ ]:
all_events_count = apiv1.count(filters={FilterKeys.role:'observation'})
ss_thresh_count = apiv1.count(filters={
        FilterKeys.role:'observation',
        FilterKeys.ivorn_contains:'BAT_SubSubThresh',
        })
print("Of {} observation packets in the database, {} are BAT_SubSubThresh packets".format(
        all_events_count, ss_thresh_count))

So it’s perhaps not surprising that we’d encounter a few of them co-incidentally lying in the search-cone. We can define a search-cone of the same radius at arbitrary co-ordinates and see what we get back, for comparison:

In [ ]:
cone2 = (SkyCoord(ra=0, dec=35., unit='deg'), Angle(0.5, unit='deg'))
cone2
In [ ]:
cone2_filters = {
    FilterKeys.role: 'observation',
    FilterKeys.cone: cone2
    }
In [ ]:
apiv1.map_stream_count(filters=cone2_filters)
In [ ]:
apiv1.list_ivorn(filters=cone2_filters)

Result: looks like we get a similar number of these ‘SubSubThresh’ events wherever we point our search-cone!

Restricting the time-period to a 60-day window

Fortunately, it’s easy to narrow things down a bit further, if we expect related events to occur within a reasonably short time-interval. Let’s modify our original filter-set a bit more:

In [ ]:
from datetime import timedelta
cone_filters[FilterKeys.authored_since] = sky_event.timestamp - timedelta(days=30)
cone_filters[FilterKeys.authored_until] = sky_event.timestamp + timedelta(days=30)
cone_filters
In [ ]:
sorted(apiv1.list_ivorn(cone_filters))

As a result of restricting our search to a window about the XRT-position timestamp, we’ve cut out all of those ‘SubSubThresh’ events that probably weren’t related.

A note of warning

What hasn’t been mentioned so far is that a query with a cone-search filter will only return VOEvents which contain a position in the `WhereWhen <http://voevent.readthedocs.org/en/latest/reading.html#overall-structure>`__ section of the packet. This sounds obvious, but on the other hand it’s worth being reminded that some related and interesting VOEvents may not contain any position information.

Most packets relating to follow-up or simultaneous observations of an event will contain position data, but we can imagine circumstances where this wouldn’t apply, e.g. an all-sky high-energy gamma-ray detector without good localization. Alternatively a VOEvent author may simply decide not to repeat co-ordinates that have been given in an earlier packet (perhaps when giving refined estimates about flux / lightcurve / spectral index, etc), but instead rely on the citation mechanism - see the next notebook!

Coming next …

In the search above for packets near to a Swift XRT position, many of the packets were clearly designated as relating to the same event - they all had the same ID number. Is that relation encoded into the packet-data? Can we just select the VOEvents which are already marked as being related to a particular packet?

The answer, of course, is yes - we make use of citation data.

Tutorial 4 - Reference and Citation lookups

In the previous notebook, we retrieved VOEvents related to a Swift XRT detection using the cone-search filter. A cone-search can bring to light new associations between VOEvents from different observatories, but we saw that it can also return unrelated events that just happen to lie nearby.

** This notebook shows how to explore relations between VOEvents which are already encoded by the packet author, using the ‘citation’ mechanism. **

It also includes a short example using networkx to visualise a citation-network.

In [ ]:
from __future__ import print_function
In [ ]:
import voeventdb.remote as vr
import voeventdb.remote.apiv1 as apiv1
from voeventdb.remote.apiv1 import FilterKeys
from voeventdb.remote.helpers import Synopsis

Once again, we’ll retrieve a handy example VOEvent representing a Swift XRT detection of a GRB:

In [ ]:
xrt_synopsis = Synopsis(apiv1.packet_synopsis('ivo://nasa.gsfc.gcn/SWIFT#XRT_Pos_666352-553'))

References and citations

Note on terminology:

In voeventdb, we use the terms ‘reference’ and ‘citation’ in their precise bibliographic sense, i.e. references are made (to another packet), and citations are received (from other packets).

References made by a packet

It’s easy to inspect the references made by a packet - it’s right there in the synopsis:

In [ ]:
print("This packet contains {} references.".format(len(xrt_synopsis.references)))
xrt_synopsis.references[0]
Citations received

We can also check if this packet receives any citations. To do so, we search for other VOEvents which list it as a reference:

In [ ]:
my_filters = {FilterKeys.ref_exact:xrt_synopsis.ivorn}
apiv1.list_ivorn(my_filters)

Nope! No citations. But what about the original BAT trigger, the one referenced by the XRT VOEvent?

In [ ]:
bat_trigger_ivorn = xrt_synopsis.references[0]['ref_ivorn']
my_filters = {FilterKeys.ref_exact:bat_trigger_ivorn}
citations_to_bat = apiv1.list_ivorn(my_filters)
citations_to_bat
Number of citations received, batch version:

Aha! So, the original BAT GRB trigger is the ‘anchor reference’ for all these other packets. Are any of them cited, in turn? There’s a quick way to find out; we can use the ivorn_cited_count endpoint to get a citation count for all the packets matching our current filter set:

In [ ]:
my_filters
In [ ]:
apiv1.list_ivorn_ncites(my_filters)

Again, a big fat nope - they all have zero citations.

Party trick - network mapping

So we have quite a boring citation network - several packets cite the BAT GRB position, then the trail ends. Nonetheless, we can use it to show off a party trick - voeventdb.remote contains an extra function that makes repeated calls to the server to perform a depth-first search (with configurable maximum recursion level) of the citation network:

In [ ]:
from voeventdb.remote.apiv1.convenience import citation_network_map
cite_map = citation_network_map(xrt_synopsis.ivorn)
In [ ]:
cite_map

If we use matplotlib and networkx, we can even draw the citation map. You can see the ‘BAT_GRB’ packet with many edges, representing citations, leading out from it

(To do: More elaborate plots, with readable labels / colouring, etc. Contributions welcome!)

In [ ]:
%matplotlib inline
import networkx as nx
G=nx.DiGraph()
for ivorn in cite_map.keys():
    G.add_node(ivorn)
for origin_ivorn, citing_ivorns in cite_map.items():
    for ci in citing_ivorns:
        G.add_edge(ivorn,ci)
nx.draw_networkx(G, arrows=True, with_labels=False)

Tutorial 5 - Ordering and n-results limits

We’ve now covered all the endpoints provided by voeventdb.

** This notebook briefly demonstrates how to apply list-orderings and number-of-results limits. **

We use a motivating example - displaying a short list of recent GRB alerts.

In [ ]:
from __future__ import print_function
import voeventdb.remote as vr
import voeventdb.remote.apiv1 as apiv1
from voeventdb.remote.apiv1 import FilterKeys
from voeventdb.remote.helpers import Synopsis
from datetime import datetime, timedelta
import pytz

import logging
logging.basicConfig(level=logging.DEBUG)

List ordering

When using the voeventdb list-query endpoints, we can control ordering using the order parameter. So for example, if we simply request the first 5 IVORNs with no modifiers, we get the first entries inserted into the database:

In [ ]:
apiv1.list_ivorn(n_max=5)

In this case, whatever happened to arrive just after the database was switched on. But, we can change the ordering, choosing from one of the order-values. For example, to retrieve the most recent ‘observation’ VOEvents, we’ll request the list in ‘author-datetime descending’ order:

In [ ]:
apiv1.list_ivorn(filters={FilterKeys.role:'observation'},
                 order=apiv1.OrderValues.author_datetime_desc,
                 n_max=5,
                )

Retrieving recent GRB events

With this last feature, we can start using voeventdb for applications such as reviewing the most recent alerts, and perhaps even planning our follow-up. Suppose we want to retrieve the 10 most-recent GRB-event alerts from the Swift BAT:

In [ ]:
filters = { FilterKeys.ivorn_contains: 'BAT_GRB',
            FilterKeys.role: 'observation'}
now = datetime.utcnow()
recent_swift_grb_ivorns = apiv1.list_ivorn(filters,
                                     order=apiv1.OrderValues.author_datetime_desc,
                                     n_max=10,
                                    )
recent_swift_grbs = [Synopsis(apiv1.packet_synopsis(i)) for i in recent_swift_grb_ivorns]

Let’s view a summary table, displaying just the dates and co-ords:

In [ ]:
print("Recent GRBs as of {}:".format(now))
print()
print ("Timestamp:\t\t\t RA, \tDec")
for grb in recent_swift_grbs:
    print("{}:\t {:06.2f}, {:+07.2f}".format(
            grb.author_datetime,
            grb.coords[0].ra.deg,
            grb.coords[0].dec.deg))

Wrapping up

That about covers all the major features of voeventdb, as demonstrated using the voeventdb.remote client-library. Comments, questions, bug-reports and other contributions are all welcomed - you can leave a note on the issue tracker or find more contact details at http://4pisky.org/voevents/.

And finally…

We’ve covered how to get data out of voeventdb - but not what to do with the data when you’ve got it. The examples notebook demonstrates a few basic ideas - see next!

Data-analysis and visualization examples

This notebook does not directly demonstrate voeventdb (see the tutorials!) but provides examples of what’s possible when data on astronomical transients is readily available:

Prelude - fetch 10 most-recent GRBs

In [ ]:
from __future__ import print_function
import voeventdb.remote as vr
import voeventdb.remote.apiv1 as apiv1
from voeventdb.remote.apiv1 import FilterKeys
from voeventdb.remote.helpers import Synopsis
from datetime import datetime, timedelta
import pytz
import logging
logging.basicConfig(level=logging.DEBUG)
In [ ]:
filters = { FilterKeys.ivorn_contains: 'BAT_GRB',
            FilterKeys.role: 'observation'}

recent_swift_grb_ivorns = apiv1.list_ivorn(
    filters,
    order=apiv1.OrderValues.author_datetime_desc,
    n_max=10,
)
recent_swift_grbs = [Synopsis(apiv1.packet_synopsis(i))
                     for i in recent_swift_grb_ivorns]

Plotting a timeline

In the last tutorial we retrieved and printed the timestamps of the most recent GRBs - but deciphering textual timestamps isn’t much fun. Let’s plot a timeline instead:

In [ ]:
%matplotlib inline
import matplotlib as mpl
import matplotlib.pyplot as plt

grb_dtimes=[s.author_datetime for s in recent_swift_grbs]

fonts = {'weight' : 'bold',
        'size'   : 14}

mpl.rc('font', **fonts)

now = pytz.UTC.localize((datetime.utcnow()))
week_markers = [now - timedelta(days=7)*w for w in range(0,5)]


markersize=95

plt.scatter(grb_dtimes, [1]*len(grb_dtimes),  marker='*',
            s=markersize, label='GRB')
plt.scatter(now, 1, marker='o', s=markersize, c='r')
for d in grb_dtimes:
    plt.axvline(d, ymax=0.5, ls='-')

first_label = True
for w in week_markers:
    plt.axvline(w, ymax=0.5, ls='--',c='r',
                label=('Week marker' if first_label else None))
    first_label=False

plt.xlim(min(grb_dtimes)-timedelta(days=2),
         max(max(grb_dtimes),now)+timedelta(days=2))
plt.gcf().autofmt_xdate()
ax = plt.gca()
ax.yaxis.set_visible(False)
ax.spines['right'].set_visible(False)
ax.spines['left'].set_visible(False)
ax.spines['top'].set_visible(False)
ax.xaxis.set_ticks_position('bottom')
ax.get_yaxis().set_ticklabels([])
plt.legend(loc='best')
plt.gcf().set_size_inches(12,4)
plt.gcf().suptitle("Recent Swift BAT alerts, as of {}".format(now),
                   fontsize=22)

Creating a sky-coordinate scatterplot

Let’s see where those GRBs lie on-sky (with credit to http://www.astropy.org/astropy-tutorials/plot-catalog.html):

In [ ]:
from astropy.coordinates import Angle
import astropy.units as u
grb_ra_coords = Angle([grb.coords[0].ra for grb in recent_swift_grbs])
grb_dec_coords = Angle([grb.coords[0].dec for grb in recent_swift_grbs])
grb_ra_coords = grb_ra_coords.wrap_at(180*u.degree)

print(grb_ra_coords.deg)
print(grb_dec_coords.deg)
In [ ]:
fig = plt.figure(figsize=(12,8))
ax = fig.add_subplot(111, projection="mollweide")
ax.scatter(grb_ra_coords.radian, grb_dec_coords.radian,
           marker='*', s=13**2, label='GRB')
ax.grid(True)
plt.legend()
plt.gcf().suptitle('Locations of 10 most recent Swift BAT GRB alerts',
                   fontsize=24)

Planning an observation with astropy

Let’s suppose we want to observe a recently discovered GRB from Cambridge, UK. We have the location of the event, already converted to astropy co-ordinates. This means we can use astropy’s Alt-Az conversions to calculate the altitude of the target at any given time.

(This section borrows heavily from http://docs.astropy.org/en/stable/coordinates/observing-example.html, consult the original for more detail and background)

In [ ]:
#Grab the latest IERS time-data first:
from astropy.utils.data import download_file
from astropy.utils import iers
iers.IERS.iers_table = iers.IERS_A.open(
    download_file(iers.IERS_A_URL, cache=True))
In [ ]:
import numpy as np
from astropy import units as u
from astropy.time import Time
from astropy.coordinates import SkyCoord, EarthLocation, AltAz

Let’s pick a GRB which will actually be observable from the UK, i.e. one in the Northern hemisphere:

In [ ]:
filters[FilterKeys.dec_greater_than]=0.0
filters
In [ ]:
northern_grb_list = apiv1.list_ivorn(
    filters,
    order=apiv1.OrderValues.author_datetime_desc,
    n_max=1,
    )

grb_target = Synopsis(apiv1.packet_synopsis(northern_grb_list[0]))
In [ ]:
print("Sample target GRB:")
print(grb_target)
In [ ]:
grb_target_posn = grb_target.coords[0]
cambridge = EarthLocation(lat=52.205*u.deg, lon=0.118*u.deg, height=6*u.m)
utcoffset = -0*u.hour  # GMT
time = Time.now() - utcoffset
grb_altaz = grb_target_posn.transform_to(AltAz(obstime=time,location=cambridge))
print("GRB's current altitude = {0.alt:.1f}".format(grb_altaz)  )
In [ ]:
from astropy.coordinates import get_sun
now = Time.now()
delta_24 = np.linspace(0, 24, 100)*u.hour
times = now + delta_24

altaz_frame = AltAz(obstime=times, location=cambridge)

grb_altaz_array = grb_altaz.transform_to(altaz_frame)
sun_altaz_array = get_sun(times).transform_to(altaz_frame)
In [ ]:
fig = plt.figure(figsize=(12,8))
plt.plot(delta_24, grb_altaz_array.alt, lw=10, label='GRB')
plt.plot(delta_24, sun_altaz_array.alt, lw=10, c='y', label='Sun')
# plt.fill_between(delta_midnight, 0, 90, sun_altaz_array.alt < -0*u.deg, color='0.5', zorder=0)
# plt.fill_between(delta_midnight, 0, 90, sun_altaz_array.alt < -18*u.deg, color='k', zorder=0)

plt.axhline(0, ls='--', lw=8, c='k',
           label='Horizon',)
plt.xlabel('Hours from {}'.format(now))
plt.ylabel('Source altitude')
plt.legend(loc='best')
plt.gcf().suptitle("GRB target altitude over next 24 hours", fontsize=24)

Note

Optional dependencies:

For trying out the basic functionality in the tutorials, all you need to install is the voeventdb.remote library itself (and optionally, the Jupyter notebook dependency, if you are running the code from the notebooks rather than pasting it in at the command line). However, there are a few sections where we make use of other libraries for generating plots, etc. If you want to run those cells, you can ensure you have all the required libraries by installing the list found in docs/requirements.txt, i.e.

sphinx
sphinx_rtd_theme
nbsphinx
jupyter
voevent-parse
matplotlib
networkx

API Reference

Note that all the functions in voeventdb.remote mirror the underlying server functionality. The server docs may be helpful when trying to understand the underlying data-queries and filters.

Package default variables

Define some package-level default values:

voeventdb.remote.default_host = 'http://voeventdb.4pisky.org'

The default host to query.

voeventdb.remote.default_list_n_max = 5000

Maximum number of rows to fetch from a list request, by default.

If you really want to fetch more rows in a single query, this can be changed for a single request by setting the n_max parameter-value. Setting a value of n_max=0 will fetch all available rows.

voeventdb.remote.default_pagesize = 1000

Number of rows fetched in each HTTP GET request (when querying list-endpoints).

When many rows are available, the top-level interface will make multiple GET requests behind the scenes, fetching pagesize rows in each GET. This variable can typically be left unchanged for casual usage - it’s aimed at advanced usage when trying to improve network performance.

API version 1 (apiv1)

Endpoints

Functions representing endpoints

Endpoint definitions can be found in the server-docs. Function calls use a consistent style throughout:

  • For all endpoints you can specify a particular host. If this is None, the default host will be used.
  • All endpoint-functions can be passed filters, a dictionary defining a set of filter-keys and values (except for the packet_ functions, which retrieve details on a single packet as specified by the IVORN).
  • List endpoints can be passed an order (see OrderValues), and an n_max parameter which limits the number of list-items returned. There is also a pagesize parameter, but this can typically be left at the default setting (see voeventdb.remote.default_pagesize).

Note

These are imported into the apiv1 namespace for brevity, so you can access them like voeventdb.remote.apiv1.count().

voeventdb.remote.apiv1.endpoints.count(filters=None, host=None)[source]
voeventdb.remote.apiv1.endpoints.list_ivorn(filters=None, order=None, pagesize=None, n_max=None, host=None)[source]
voeventdb.remote.apiv1.endpoints.list_ivorn_ncites(filters=None, order=None, pagesize=None, n_max=None, host=None)[source]
voeventdb.remote.apiv1.endpoints.list_ivorn_nrefs(filters=None, order=None, pagesize=None, n_max=None, host=None)[source]
voeventdb.remote.apiv1.endpoints.map_authored_month_count(filters=None, host=None)[source]
voeventdb.remote.apiv1.endpoints.map_role_count(filters=None, host=None)[source]
voeventdb.remote.apiv1.endpoints.map_stream_count(filters=None, host=None)[source]
voeventdb.remote.apiv1.endpoints.map_stream_role_count(filters=None, host=None)[source]
voeventdb.remote.apiv1.endpoints.packet_synopsis(ivorn, host=None)[source]
voeventdb.remote.apiv1.endpoints.packet_xml(ivorn, host=None)[source]

Enumeration classes

class voeventdb.remote.apiv1.FilterKeys[source]

Enumerates valid filters.

Useful for building dictionaries representing filter-sets, eg.:

filters = {
    FilterKeys.ivorn_contains: 'GRB',
    FilterKeys.role : 'observation',
}

For definitions of the various filters, and examples of valid values, see the voeventdb query-filters page.

authored_since = 'authored_since'
authored_until = 'authored_until'
cited = 'cited'
cone = 'cone'
coord = 'coord'
dec_greater_than = 'dec_gt'
dec_less_than = 'dec_lt'
ivorn_contains = 'ivorn_contains'
ivorn_prefix = 'ivorn_prefix'
ref_any = 'ref_any'
ref_contains = 'ref_contains'
ref_exact = 'ref_exact'
role = 'role'
stream = 'stream'
class voeventdb.remote.apiv1.OrderValues[source]

Enumerates valid orderings.

For details see this section in the voeventdb.server docs.

author_datetime = 'author_datetime'
author_datetime_desc = '-author_datetime'
id = 'id'
id_desc = '-id'
ivorn = 'ivorn'
ivorn_desc = '-ivorn'

Helper classes

Helper classes for providing convenience parsing of returned JSON content.

class voeventdb.remote.helpers.SkyEvent(skyevent_dict)[source]

Represents universal attributes of an observed event on sky.

I.e. the most basic details that we expect to find in all packets reporting events with sky-positions.

position

astropy.coordinates.SkyCoord – Best-estimate sky-coordinates of the event being reported.

position error

astropy.coordinates.Angle – Error-cone on the position estimate.

timestamp of event

datetime.datetime – Timestamp for the reported event (UTC timezone).

class voeventdb.remote.helpers.Synopsis(synopsis_dict, api_version_string='apiv1')[source]

Parses the output from the ‘synopsis’ endpoint into a class-object.

author_datetime

datetime.datetime – The Who.Date timestamp. (Parsed to datetime, UTC timezone). May be None if no timestamp present in the VOEvent packet.

author_ivorn

string – The Who.AuthorIVORN entry. May be None if no entry present in the VOEvent packet.

ivorn

string

received

datetime.datetime – Dates exactly when the packet was loaded into this instance of voeventdb.

role

string

stream

string

version

string

sky_events

list – A list of SkyEvent objects. These are parsed from the WhereWhen section of a VOEvent packet, if possible. (If the packet has no WhereWhen data, this is an empty list).

coords

listastropy.coordinates.SkyCoord positions. This is a syntax-shortcut, the positions are just those of the sky_events entries, if any are present.

references

list – References to other VOEvent packets. This is a list of dictionaries representing any references present in the Citations section of the VOEvent packet.

Additional convenience functions

Convenience functions for exploring / extracting more complex data-structures.

voeventdb.remote.apiv1.convenience.citation_network_map(ivorn, max_recursion_levels=5)[source]

Indices and tables