edeposit.amqp.aleph_link_export¶
Two-way communication system used to deliver update requests from E-deposit to Aleph used in Czech National library. Updates may contain new http links to other systems or identifiers, like URN:NBN, or UUID.
Links created in Aleph cannot be changed automatically by direct-manipulation, so this project defines AMQP to XML protocol bridge. XML then copied over SCP to Aleph, processed and resulting XML response is then translated back to AMQP messages received by E-deposit.
Package structure¶
File relations¶
API¶
link_export submodule¶
This module provides API wrapper over RequestDatabase.
- aleph_link_export.link_export.get_rd(fn)¶
- aleph_link_export.link_export.export(request)¶
Save the export request to the database.
Parameters: request (obj) – Instance of LinkUpdateRequest.
- aleph_link_export.link_export.collect_responses()¶
Collect processed resposes.
Returns: List of LinkUpdateResponse objects. Return type: list
request_database submodule¶
This module is used to keep incomming and outgoing structures and also for serialization/deserialization to/from XML.
- class aleph_link_export.request_database.RequestDatabase(log_fn='/home/aleph_export/link_export.log', db_fn='/home/aleph_export/request_datase.shelve', db_key='request_database', logging=True, req_fn='/home/aleph_export/edep2aleph/requests.xml', resp_fn='/home/aleph_export/aleph2edep/responses.xml', xsd_url='http://edeposit-aplikace.nkp.cz/link_export_notification.xsd')¶
Bases: aleph_link_export.shelvedb.ShelveDatabase
Keep the track of requests and resposes and their serialization and deserialization from/to XMl.
- req_fn = None¶
Path to the request XML.
- resp_fn = None¶
Path to the response XML
- add_request(request)¶
Add new request object to database.
Parameters: request (obj) – Object with defined session_id property and to_dict_xml() method.
- get_responses()¶
Process response queue, remove finished requests from request queue, return list of response objects.
Returns: List of LinkUpdateResponse objects. Return type: list
- to_xml()¶
Convert _req_queue to XML as defined in request XSD.
Returns: XML. Return type: unicode
- save()¶
Read the response XML, process it, save the database and request XML.
- static load(fn='/home/aleph_export/request_datase.shelve', db_key='request_database', creator=<function <lambda> at 0x7f138ab9aaa0>)¶
Load the database from the shelve fn.
Parameters: - fn (str) – Path to the database file. Default DATABASE_FN.
- db_key (str) – What database key to use. Default DATABASE_KEY.
- creator (fn reference) – Reference to the function, which will create new RequestDatabase if the old is not found. Default lambda, which expects fn parameter lambda fn: ...
Returns: RequestDatabase instance from the fn or newly created.
Return type: obj
shelvedb submodule¶
Module used to load/store object from/to shelve database.
- aleph_link_export.shelvedb.shelver(*args, **kwds)¶
In python 2.7, there is no context manager for shelve. So this is it.
- class aleph_link_export.shelvedb.ShelveDatabase(log_fn, db_fn, db_key, logging)¶
Bases: object
Class that can save and load itself to shelve.
- db_fn = None¶
Path to the database file.
- log_fn = None¶
Path to the log file.
- logging = None¶
Is the logging enabled?
- save()¶
Save this object to shelve.
- static load(fn, db_key, creator)¶
Load the database from the shelve fn.
Parameters: - fn (str) – Path to the database file.
- db_key (str) – What database key to use. Default DATABASE_KEY.
- creator (reference) – Reference to the function, which will create new RequestDatabase if the old is not found. Default lambda, which expects fn parameter lambda fn: ...
Returns: RequestDatabase instance from the fn or newly created.
Return type: obj
settings submodule¶
Module is containing all necessary global variables for the package.
Module also has the ability to read user-defined data from two paths:
- $HOME/_SETTINGS_PATH
- /etc/_SETTINGS_PATH
See _SETTINGS_PATH for details.
Note
If the first path is found, other is ignored.
Example of the configuration file ($HOME/edeposit/aleph_export.json):
{
"REQUEST_FN": "/home/whatever/req.xml"
}
Attributes¶
- aleph_link_export.settings.REQUEST_FN = '/home/aleph_export/edep2aleph/requests.xml'¶
Path to the XML file, where the requests will be stored.
- aleph_link_export.settings.RESPONSE_FN = '/home/aleph_export/aleph2edep/responses.xml'¶
Path to the file, where the Aleph will put the XML responses.
- aleph_link_export.settings.DATABASE_FN = '/home/aleph_export/request_datase.shelve'¶
- aleph_link_export.settings.LOG_FN = '/home/aleph_export/link_export.log'¶
Path to the file, where the logs will be stored.
- aleph_link_export.settings.DATABASE_KEY = 'request_database'¶
Don’t change this! Key for the database.
- aleph_link_export.settings.EXPORT_XSD_LINK = 'http://edeposit-aplikace.nkp.cz/link_export_notification.xsd'¶
Link to the export XSD
- aleph_link_export.settings.LOGGING_ENABLED = True¶
Logging enabled or not?
Request structures¶
- class aleph_link_export.structures.requests.LinkDescription¶
Bases: aleph_link_export.structures.requests.LinkDescription
Optional structure, which can be used instead of string to describe the format of the url.
- url¶
str – URL of the document.
- format¶
str – Format of the document.
- to_dict_xml()¶
Serialize the object to dictionary, which may be later used for conversion to XML.
- Retruns:
- OrderedDict: Itself as ordered dict.
- class aleph_link_export.structures.requests.LinkUpdateRequest¶
Bases: aleph_link_export.structures.requests.LinkUpdateRequest
Request to update metadata in Aleph.
- uuid¶
str – UUID for the doc_number you wish to update.
- doc_number¶
str – Document number of the document you wish to update. If there is a summary aleph record for this document at Aleph please send a doc_number of this summary aleph record.
- document_urls¶
list – Newly added public URL to the storage / whatever subsystem. List of strings or LinkDescription.
- kramerius_url¶
str, default None – Newly added URL to the Kramerius subsystem.
- to_dict_xml()¶
Convert the structure to nested ordered-dicts, which are later used for construction of the XML.
Returns: Itself as ordered dicts. Return type: OrderedDict
- class aleph_link_export.structures.requests.StatusRequest¶
Bases: aleph_link_export.structures.requests.StatusRequest
This structure is used to wake the daemon to go and check whether the files on the disc changed or not.
Response structures¶
- class aleph_link_export.structures.responses.LinkUpdateResponse¶
Bases: aleph_link_export.structures.responses.LinkUpdateResponse
Response to the LinkUpdateRequest request.
This object is returned only when the Aleph signals, that the record was really updated.
- status¶
str – Status of the update request. Either OK, or ERROR.
- session_id¶
str – Corresponding session id.
- reason¶
str, default None – Optional reason why the request was rejected.
AMQP protocol¶
Here is the list of Request -> Response pairs describing responses to AMQP communication:
LinkUpdateRequest ---> 0-N × LinkUpdateResponse
StatusRequest -------> 0-N × LinkUpdateResponse
Protocol is really simple - you can send the LinkUpdateRequest and you will get back all waiting LinkUpdateResponse responses. If there is none, you won’t get any.
You can also trigger the lookup for waiting responses by sending periodic StatusRequest messages.
Installation¶
Module is hosted at PYPI, and can be easily installed using PIP:
sudo pip install edeposit.amqp.aleph_link_export
Don’t forget to add proper paths into your configuration file (see settings for details) in /home/edeposit/aleph_export.json or /etc/edeposit/aleph_export.json.
Example:
{
"REQUEST_FN": "/home/aleph_export/edep2aleph.xml",
"RESPONSE_FN": "/home/aleph_export/aleph2edep.xml",
"DATABASE_FN": "/home/aleph_export/database.shelve",
"LOG_FN": "/home/aleph_export/log.txt"
}
- Warning:
- The directories have to be created before you try to run the project!
Source code¶
Project is released under the MIT license. Source code can be found at GitHub:
Unittests¶
Almost every feature of the project is tested by unittests. You can run those tests using provided run_tests.sh script, which can be found in the root of the project.
If you have any trouble, just add --pdb switch at the end of your run_tests.sh command like this: ./run_tests.sh --pdb. This will drop you to PDB shell.
Requirements¶
This script expects that package pytest is installed. In case you don’t have it yet, it can be easily installed using following command:
pip install --user pytest
or for all users:
sudo pip install pytest
Example¶
$ ./run_tests.sh
============================= test session starts ==============================
platform linux2 -- Python 2.7.6 -- py-1.4.30 -- pytest-2.7.2
rootdir: /home/bystrousak/Plocha/Dropbox/c0d3z/prace/edeposit.amqp.aleph_link_export, inifile:
plugins: cov
collected 17 items
tests/test_RequestDatabase.py .......
tests/test_amqp_chain.py ....
tests/structures/test_requests.py .....
tests/structures/test_responses.py .
========================== 17 passed in 0.25 seconds ===========================