Kaneda¶
Kaneda is a Python library that allows to report events and metrics of your applications. It provides a several builtin Metrics methods in order to store any amount of data that you want to then analyze it or for performance studies
Example:
from kaneda import Metrics
metrics = Metrics(...)
metrics.event('welcome', 'Kaneda is cool')
metrics.gauge('answer_of_life', 42)
Usage¶
You need to install Kaneda package:
pip install kaneda
Backend reporting¶
You need a backend in order to keep data in a persistent storage. You can use builtin Backends
or define your custom backend subclassing the BaseBackend
class.
The following example it shows how to send metrics with Elasticsearch as a backend:
from kaneda.backend import ElasticsearchBackend
from kaneda import Metrics
backend = ElasticsearchBackend(index_name='myindex', app_name='myapp', host='localhost',
port=9200, user='kaneda', password='kaneda')
metrics = Metrics(backend=backend)
metrics.gauge('answer_of_life', 42)
A backend class can also be instantiated passing a previously defined connection client. This is specially useful when you want to use a tuned connection:
client = Elasticsearch(['localhost'], port=9200, http_auth=('kaneda', 'kaneda'), timeout=0.3)
backend = ElasticsearchBackend(index_name='myindex', app_name='myapp', client=client)
Asynchronous reporting¶
Depending the selection of the backend the process of reporting metrics could be “slow” if the response time of your application is critical (e.g: a website). Furthermore if your application doesn’t need the see the reported metrics in real time you probably have to consider to using asynchronous reporting. With this system you are allowed to send a metric report in background without adding too much overhead.
To use this system you need to install a queue system and use one of the builtin Kaneda Queues classes. To setup Kaneda in async mode follow these steps.
- Install and configure your queue system (e.g: RQ).
pip install rq
- Setup your backend configuration in new file named
kanedasettings.py
.
BACKEND = 'kaneda.backends.ElasticsearchBackend'
ELASTIC_INDEX_NAME = 'myindex'
ELASTIC_APP_NAME = 'myapp'
ELASTIC_HOST = 'localhost'
ELASTIC_PORT = 9200
ELASTIC_USER = 'kaneda'
ELASTIC_PASSWORD = 'kaneda'
- Run the worker
rqworker
Now you can use Kaneda with the same Metrics API:
from kaneda.queues import RQQueue
from kaneda import Metrics
queue = RQQueue(redis_url='redis://localhost:6379/0')
metrics = Metrics(queue=queue)
metrics.gauge('answer_of_life', 42)
As in the backend example it can be used passing a queue client:
q = Queue(queue_name, connection=Redis())
queue = RQQueue(queue=q)
Also you are able to specify a Redis connection url (or a broker url if you use Celery). Notice this allows you to run the worker on a different server.
Metrics¶
-
class
kaneda.base.
Metrics
(backend=None, queue=None)[source]¶ Metrics reporting class
Parameters: - backend – instance of kaneda.backends. It is the responsible to store the reported data.
- queue – instance of kaneda.queues. It is the responsible to store the reported data asynchronously.
If none of the parameters are passed it tries get the backend from kaneda settings file.
-
custom
(name, metric, value, tags=None, id_=None)[source]¶ Send a custom metric report.
>>> metrics.custom('hotel.response_data', metric='xml', value={'status': 'ok', 'xml': ...}, id_='2B75D750')
-
event
(name, text, tags=None)[source]¶ Record an event.
>>> metrics.event('user.signup', 'New user registered')
-
gauge
(name, value, tags=None)[source]¶ Record the value of a gauge.
>>> metrics.gauge('users.notifications', 13, tags=['new_message', 'follow_request'])
-
increment
(name, tags=None)[source]¶ Increment a counter.
>>> metrics.increment('user.profile.views')
-
timed
(name=None, tags=None, use_ms=None)[source]¶ Measure the amount of time of a function (using a decorator) or a piece of code (using a context manager). If name is not provided while using the decorator it will be used the name of the module and the function.
# With decorator @metrics.timed('request.response_time') def perform_request(params): pass # With context manager with metrics.timed('request.response_time'): pass
Backends¶
Kaneda provides builtin backends to store metrics and events in a persistent storage. If you want to use your
custom backend you need to subclass BaseBackend
and implement your custom report
method which
is the responsible to store the metrics data.
Elasticsearch¶
Elasticsearch is a search based NoSQL database that works very well with metrics data. It provides powerful tools to analyze data and build real-time dashboards easily with Kibana.
Note
Before using Elasticesearch as backend you need to install Elasticsearch Python client:
pip install elasticsearch
-
class
kaneda.backends.
ElasticsearchBackend
(index_name, app_name, client=None, connection_url=None, host=None, port=None, user=None, password=None, timeout=0.3)[source]¶ Elasticsearch backend.
Parameters: - index_name – name of the Elasticsearch index used to store metrics data. Default name format will be index_name-YYYY.MM.DD.
- app_name – name of the app/project where metrics are used.
- client – client instance of Elasticsearch class.
- connection_url – Elasticsearch connection url (https://user:secret@localhost:9200). It can be used passing a single connection_url (a string) or passing multiple connection_urls (a list).
- host – server host. It can be used passing a single host (a string) or passing multiple hosts (a list).
- port – server port.
- user – HTTP auth username.
- password – HTTP auth password.
- timeout – Elasticsearch connection timeout (seconds).
MongoDB¶
MongoDB is a document oriented NoSQL database. Is a great tool to store metrics as it provides a powerful aggregation framework to perform data analysis.
Note
Before using MongoDB as backend you need to install MongoDB Python client:
pip install pymongo
-
class
kaneda.backends.
MongoBackend
(db_name, collection_name, client=None, connection_url=None, host=None, port=None, timeout=300)[source]¶ MongoDB backend.
Parameters: - db_name – name of the MongoDB database.
- collection_name – name of the MongoDB collection used to store metric data.
- client – client instance of MongoClient class.
- connection_url – Mongo connection url (mongodb://localhost:27017/).
- host – server host.
- port – server port.
- timeout – MongoDB connection timeout (milliseconds).
RethinkDB¶
RethinkDB is an open source scalable, distributed NoSQL database built for realtime applications.
Note
Before using RethinkDB as backend you need to install RethinkDB Python client:
pip install rethinkdb
-
class
kaneda.backends.
RethinkBackend
(db, table_name=None, connection=None, host=None, port=None, user=None, password=None, timeout=0.3)[source]¶ RethinkDB backend.
Parameters: - db – name of the RethinkDB database.
- table_name – name of the RethinkDB table. If this is not provided, it will be used the name of the metric.
- host – server host.
- port – server port.
- user – auth username.
- password – auth password.
- timeout – RethinkDB connection timeout (seconds).
InfluxDB¶
InfluxDB is an open source time series database with no external dependencies. It’s useful for recording metrics, events, and performing analytics.
Note
Before using InfluxDB as backend you need to install InfluxDB Python client:
pip install influxdb
Warning
InfluxDB can store other type of data besides time series. However it has some restrictions:
Metrics tags field can’t be a
list
only adict
:# bad metrics.timing('user.profile_load_time', 230, tags=['login', 'edit_profile']) # good metrics.timing('user.profile_load_time', 230, tags={'from': 'login', 'to': 'edit_profile'})
Custom
metric value field can’t be alist
nor a nesteddict
:# bad metrics.custom('zone.search', metric='query_time', value={'times': [120, 230]}) metrics.custom('zone.search', metric='query_time', value={'times': {'start': 120}, {'end': 230}}) # good metrics.custom('zone.search', metric='query_time', value={'start_time': 120, 'end_time': 230})
-
class
kaneda.backends.
InfluxBackend
(database, client=None, connection_url=None, host=None, port=None, username=None, password=None, timeout=0.3)[source]¶ InfluxDB backend.
Parameters: - database – name of the InfluxDB database.
- client – client instance of InfluxDBClient class.
- connection_url – InfluxDB connection url (influxdb://username:password@localhost:8086/databasename).
- host – server host.
- port – server port.
- username – auth username.
- password – auth password.
- timeout – InfluxDB connection timeout (seconds).
Logger¶
You can use a logger instance of the logging library from the Python standard lib. Useful for debugging.
Queues¶
Kaneda provides builtin queues to store metrics and events to perform asynchronous reporting. If you want to use your
custom asynchronous queue system you need to subclass BaseQueue
and implement your custom report
method
which is the responsible to pass metrics data to a job queue.
Celery¶
Celery is a simple, flexible and reliable distributed system to process vast amounts of messages. It can be configured using various broker systems such Redis or RabbitMQ.
Note
Before using Celery as async queue you need to install Celery library:
pip install celery
-
class
kaneda.queues.
CeleryQueue
(app=None, broker=None, queue_name='')[source]¶ Celery queue.
Parameters: - app – app instance of Celery class.
- broker – broker connection url where Celery will attend the async reporting requests.
- queue_name – name of the queue being used by the Celery worker process.
To run the worker execute this command:
celery -A kaneda.tasks.celery worker
RQ¶
RQ (Redis Queue) is a simple Python library for queueing jobs and processing them in the background with workers. It uses Redis as main broker system.
Note
Before using RQ as async queue you need to install RQ and Redis library:
pip install redis
pip install rq
To run the worker execute this command:
rqworker [queue]
The default queue is “kaneda”.
ZMQ¶
ZMQ (or ZeroMQ) is a library which extends the standard socket interfaces with features traditionally provided by specialised messaging middleware products. ZeroMQ sockets provide an abstraction of asynchronous message queues and much more.
Note
Before using ZMQ as async queue you need to install ZMQ library:
pip install pyzmq
To run the worker execute this command:
zmqworker --connection_url=<zmq_connection_url>
or define ZMQ settings in kanedasettings.py
and simply execute the worker command with:
zmqworker
-
class
kaneda.queues.
ZMQQueue
(connection_url, timeout=300)[source]¶ ZeroMQ queue
Parameters: - connection_url – ZMQ connection url (tcp://127.0.0.1:5555).
- timeout – ZMQ socket timeout (milliseconds).
Settings¶
Kaneda can be used with a settings file as the same way to use with Django. Simply define a
kanedasettings.py
file with the backend or queue settings. Alternatively you can define the environment variable
DEFAULT_SETTINGS_ENVAR pointing to the desired settings filename.
With this you will be able to use Metrics class without passing parameters:
from kaneda import Metrics
metrics = Metrics()
metrics.gauge('answer_of_life', 42)
Backends settings¶
General¶
- BACKEND
Class name of the backend. Available options are:
kaneda.backends.ElasticsearchBackend
kaneda.backends.MongoBackend
kaneda.backends.LoggerBackend
kaneda.backends.RethinkBackend
kaneda.backends.InfluxBackend
Elasticsearch¶
- ELASTIC_INDEX_NAME
- Name of the Elasticsearch index used to store metrics data. Default name format will be app_name-YYYY.MM.DD.
- ELASTIC_APP_NAME
- Name of the app/project where metrics are used.
- ELASTIC_CONNECTION_URL
- Elasticsearch connection url (https://user:secret@localhost:9200).
- ELASTIC_HOST
- Server host.
- ELASTIC_PORT
- Server port.
- ELASTIC_USER
- HTTP auth username.
- ELASTIC_PASSWORD
- HTTP auth password.
- ELASTIC_TIMEOUT
- Elasticsearch connection timeout (seconds).
MongoDB¶
- MONGO_DB_NAME
- Name of the MongoDB database.
- MONGO_COLLECTION_NAME
- Name of the MongoDB collection used to store metric data.
- MONGO_CONNECTION_URL
- Mongo connection url (mongodb://localhost:27017/).
- MONGO_HOST
- Server host.
- MONGO_PORT
- Server port.
- MONGO_TIMEOUT
- MongoDB connection timeout (milliseconds).
RethinkDB¶
- RETHINK_DB
- Name of the RethinkDB database
- RETHINK_TABLE_NAME
- Name of the RethinkDB table. If this is not provided, it will be used the name of the metric.
- RETHINK_HOST
- Server host.
- RETHINK_PORT
- Server port.
- RETHINK_USER
- Auth username.
- RETHINK_PASSWORD
- Auth password.
- RETHINK_TIMEOUT
- RethinkDB connection timeout (seconds).
InfluxDB¶
- INFLUX_DATABASE
- Name of the InfluxDB database.
- INFLUX_CONNECTION_URL
- InfluxDB connection url (influxdb://username:password@localhost:8086/databasename).
- INFLUX_HOST
- Server host.
- INFLUX_PORT
- Server port.
- INFLUX_USERNAME
- Auth username.
- INFLUX_PASSWORD
- Auth password.
- INFLUX_TIMEOUT
- InfluxDB connection timeout (seconds).
Logger¶
- LOGGER_FILENAME
- Name of the file where logger will store the metrics.
Queues settings¶
General¶
- QUEUE
Class name of the queue. Available options are:
kaneda.backends.CeleryQueue
kaneda.backends.RQQueue
kaneda.backends.ZMQQueue
Celery¶
- CELERY_BROKER
- Broker connection url.
- CELERY_QUEUE_NAME
- Name of the Celery queue.
RQ¶
- RQ_REDIS_URL
- Redis connection url.
- RQ_QUEUE_NAME
- Name of the RQ queue.
ZMQ¶
- ZMQ_CONNECTION_URL
- ZMQ connection url.
- ZMQ_TIMEOUT
- ZMQ socket timeout (milliseconds).
Django Setup¶
Kaneda can be use with Django as a mechanism to reporting metrics and events.
Add
django_kaneda
toINSTALLED_APPS
insettings.py
.Set
KANEDA_BACKEND
and the properly configuration of your selected backend insettings.py
. If you want to use Elasticsearch our configuration will be something like this:KANEDA_BACKEND = 'kaneda.backends.ElasticsearchBackend' KANEDA_ELASTIC_INDEX_NAME = 'kaneda' KANEDA_ELASTIC_APP_NAME = 'YouProject' KANEDA_ELASTIC_HOST = 'localhost' KANEDA_ELASTIC_PORT = 9200 KANEDA_ELASTIC_USER = 'user' KANEDA_ELASTIC_PASSWORD = 'pass'
Alternatively you can set KANEDA_QUEUE
to specify a queue configuration to use Kaneda in async mode:
KANEDA_BACKEND = 'kaneda.queues.CeleryQueue'
KANEDA_CELERY_BROKER = 'redis://localhost:6379/0'
With this, you can use Kaneda in everyplace of your Django project:
from django_kaneda import metrics
class UserProfileView(TemplateView):
template_name = 'user/profile.html'
@metrics.timed('user_profile.time')
def get(self, request, *args, **kwargs):
metrics.increment('user_profile.views')
return super(UserProfileView, self).get(request, *args, **kwargs)
Debug mode¶
You can use Kaneda in debug mode with a logger as backend. Simply set KANEDA_DEBUG
to True to report everything
to a logger instead a persistent backend. Furthermore, you can set a previously defined logger on settings.py
and use as
your debug logger:
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'formatters': {
'with_timestamp': {
'format': '%(asctime)s - %(name)s - %(message)s'
}
},
'handlers': {
'file': {
'level': 'INFO',
'class': 'logging.FileHandler',
'filename': '/tmp/kaneda-demo.log',
'formatter': 'with_timestamp'
},
},
'loggers': {
'kaneda.demo': {
'handlers': ['file'],
'level': 'INFO',
'propagate': True,
},
},
}
KANEDA_DEBUG = True
KANEDA_LOGGER = 'kaneda.demo'
Alternatively you can set KANEDA_LOGGER_FILENAME
instead of KANEDA_LOGGER
to store the reporting results
in a specific filename.
Available settings¶
Elasticsearch¶
- KANEDA_ELASTIC_INDEX_NAME (=’kaneda’)
- Name of the Elasticsearch index used to store metrics data. Default name format will be app_name-YYYY.MM.DD.
- KANEDA_ELASTIC_APP_NAME (=’default’)
- Name of the app/project where metrics are used.
- KANEDA_ELASTIC_CONNECTION_URL (=None)
- Elasticsearch connection url (https://user:secret@localhost:9200).
- KANEDA_ELASTIC_HOST (=None)
- Server host.
- KANEDA_ELASTIC_PORT (=None)
- Server port.
- KANEDA_ELASTIC_USER (=None)
- HTTP auth username.
- KANEDA_ELASTIC_PASSWORD (=None)
- HTTP auth password.
- KANEDA_ELASTIC_TIMEOUT (=0.3)
- Elasticsearch connection timeout (seconds).
MongoDB¶
- KANEDA_MONGO_DB_NAME (=’kaneda’)
- Name of the MongoDB database.
- KANEDA_MONGO_COLLECTION_NAME (=’default’)
- Name of the MongoDB collection used to store metric data.
- KANEDA_MONGO_CONNECTION_URL (=None)
- Mongo connection url (mongodb://localhost:27017/).
- KANEDA_MONGO_HOST (=None)
- Server host.
- KANEDA_MONGO_PORT (=None)
- Server port.
- KANEDA_MONGO_TIMEOUT (=300)
- MongoDB connection timeout (milliseconds).
RethinkDB¶
- KANEDA_RETHINK_DB (=’kaneda’)
- Name of the RethinkDB database
- KANEDA_RETHINK_TABLE_NAME (=None)
- Name of the RethinkDB table. If this is not provided, it will be used the name of the metric.
- KANEDA_RETHINK_HOST (=None)
- Server host.
- KANEDA_RETHINK_PORT (=None)
- Server port.
- KANEDA_RETHINK_USER (=None)
- Auth username.
- KANEDA_RETHINK_PASSWORD (=None)
- Auth password.
- KANEDA_RETHINK_TIMEOUT (=0.3)
- RethinkDB connection timeout (seconds).
InfluxDB¶
- KANEDA_INFLUX_DATABASE (=’kaneda’)
- Name of the InfluxDB database.
- KANEDA_INFLUX_CONNECTION_URL (=None)
- InfluxDB connection url (influxdb://username:password@localhost:8086/databasename).
- KANEDA_INFLUX_HOST (=None)
- Server host.
- KANEDA_INFLUX_PORT (=None)
- Server port.
- KANEDA_INFLUX_USERNAME (=None)
- Auth username.
- KANEDA_INFLUX_PASSWORD (=None)
- Auth password.
- KANEDA_INFLUX_TIMEOUT (=0.3)
- InfluxDB connection timeout (seconds).
Celery¶
- KANEDA_CELERY_BROKER (=’‘)
- Broker connection url.
- KANEDA_CELERY_QUEUE_NAME (=’‘)
- Name of the Celery queue.
RQ¶
- KANEDA_RQ_REDIS_URL (=’‘)
- Redis connection url.
- KANEDA_RQ_QUEUE_NAME (=’kaneda’)
- Name of the RQ queue.
ZMQ¶
- KANEDA_ZMQ_CONNECTION_URL (=’‘)
- ZMQ connection url.
- KANEDA_ZMQ_TIMEOUT (=300)
- ZMQ socket timeout (milliseconds).
Debug¶
- KANEDA_DEBUG (=True)
- Use Kaneda in debug mode.
- KANEDA_LOGGER (=None)
- Name of a previously defined logger, to use in debug mode.
- KANEDA_LOGGER_FILENAME (=None)
- Name of the file where logger will store the metrics, to use in debug mode.
Changelog¶
0.5 (2016-05-25)¶
- Add InfluxDB reporting backend.
- Add RethinkDB reporting backend
- Add ZMQ queue class and task for asynchronous reporting.
0.4 (2016-04-28)¶
Add support to run Kaneda in asynchronous mode.¶
- Add Celery and RQ queue classes and tasks with backend reporting.
- Add the option to configure Kaneda with a settings file.
- Add utils.py file to handle the settings.
- Add asynchronous reporting, settings file and queue classes documentation.
- Add integration and benchmark testing. Improve unit tests.
Add Django support.¶
- Move from django-kaneda project.
- Add integration tests.
- Improve Django documentation
0.3.1 (2016-03-29)¶
- ElasticBackend now can be configured passing a list of ‘hosts’ or a list of ‘connection_urls’.
0.3 (2016-03-28)¶
- Add LoggerBackend.
- Add exeception treatment on backend report methods.
- Add connection timeout for storage backends.
- Storage backends now support passing client or connection url as parameters instead all parameters.
- Add support to use django-kaneda on debug mode.
0.2 (2016-02-17)¶
- Refactor backend payload build method to unpack value field when is a dict.
0.1 (2016-01-13)¶
- Initial release.