PK 1S/B‡Ár«ÑL ÑL doqu-latest/ext_fields.html
New in version 0.23.
Note
This abstraction is by no means a complete replacement for the normal approach of semantic grouping. Please use it with care. Also note that the API can change. The class can even be removed in future versions of Doqu.
Representation of a document property. Syntax sugar for separate definitions of structure, validators, defaults and labels.
Usage:
class Book(Document):
title = Field(unicode, required=True, default=u'Hi', label='Title')
this is just another way to type:
class Book(Document):
structure = {
'title': unicode
}
validators = {
'title': [validators.Required()]
}
defaults = {
'title': u'Hi'
}
labels = {
'title': u'The Title'
}
Nice, eh? But be careful: the title definition in the first example barely fits its line. Multiple long definitions will turn your document class into an opaque mess of characters, while the semantically grouped definitions stay short and keep related things aligned together. “Semantic sugar” is sometimes pretty bitter, use it with care.
Complex validators still need to be specified by hand in the relevant dictionary. This can be worked around by creating specialized field classes (e.g. EmailField) as it is done e.g. in Django.
Parameters: |
|
---|
Handles externally stored files.
Warning
This field saves the file when process_outgoing() is triggered (see outgoing_processors in DocumentMetadata).
Outdated (replaced) files are not automatically removed.
Usage:
class Doc(Document):
attachment = FileField(base_path=MEDIA_ROOT+'attachments/')
d = Doc()
d.attachment = open('foo.txt')
d.save(db)
dd = Doc.objects(db)[0]
print dd.attachment.file.read()
Parameters: | base_path – A string or callable: the directory where the files should be stored. |
---|
alias of FileWrapper
A FileField that provides extended support for images. The ImageField.file is an ImageWrapper instance.
Usage:
class Photo(Document):
summary = Field(unicode)
image = ImageField(base_path='photos/')
p = Photo(summary='Fido', image=open('fido.jpg'))
p.save(db)
# playing with image
print "The photo is {0}×{1}px".format(*p.image.size)
p.image.rotate(90)
p.image.save()
alias of ImageWrapper
A validator simply takes an input and verifies it fulfills some criterion, such as a maximum length for a string. If the validation fails, a ValidationError is raised. This simple system allows chaining any number of validators on fields.
The module is heavily inspired by (and partially ripped off from) the WTForms validators. However, ours serve a bit different purpose. First, error messages are not needed here (the errors will not be displayed to end users). Second, these validators include query filtering capabilities.
Usage example:
class Person(Document):
validators = {
'first_name': [required(), length(min=2)],
'age': [number_range(min=18)],
}
This document will raise ValidationError if you attempt to save it with wrong values. You can call Document.is_valid() to ensure everything is OK.
Now let’s query the database for all objects of Person:
Person.objects(db)
Doqu does not deal with tables or collections, it follows the DRY (Don’t Repeat Yourself) principle and uses the same validators to determine what database records belong to given document class. The schema defined above is alone equivalent to the following query:
...where(first_name__exists=True, age__gte=18).where_not(first_name='')
This is actually the base query available as Person.objects(db).
Note
not all validators affect document-related queries. See detailed documentation on each validator.
Causes the validation chain to stop.
If StopValidation is raised, no more validators in the validation chain are called.
Raised when a validator fails to validate its input.
Validates an email address. Note that this uses a very primitive regular expression and should only be used in instances where you later verify by other means, such as email activation or lookups.
Adds conditions to the document-related queries: the field must match the pattern.
Compares the values of two fields.
Parameters: | name – The name of the other field to compare to. |
---|
Compares the value to another value.
Parameters: | other_value – The other value to compare to. |
---|
Adds conditions to the document-related queries.
Ensures given field exists in the record. This does not affect validation of a document with pre-defined structure but does affect queries.
Adds conditions to the document-related queries.
Validates an IP(v4) address.
Adds conditions to the document-related queries: the field must match the pattern.
Validates the length of a string.
Parameters: |
|
---|
Validates that a number is of a minimum and/or maximum value, inclusive. This will work with any comparable number type, such as floats and decimals, not just integers.
Parameters: |
|
---|
Adds conditions to the document-related queries.
alias of NumberRange
Allows empty value (i.e. bool(value) == False) and terminates the validation chain for this field (i.e. no more validators are applied to it). Note that errors raised prior to this validator are not suppressed.
Requires that the value is not empty, i.e. bool(value) returns True. The bool values can also be False (but not anything else).
Adds conditions to the document-related queries: the field must exist and be not equal to an empty string.
Validates the field against a user provided regexp.
Parameters: |
|
---|
Note
the pattern must be provided as string because compiled patterns cannot be used in database lookups.
Adds conditions to the document-related queries: the field must match the pattern.
Simple regexp based url validation. Much like the email validator, you probably want to validate the url later by other means if the url must resolve.
Parameters: | require_tld – If true, then the domain-name portion of the URL must contain a .tld suffix. Set this to false if you want to allow domains like localhost. |
---|
Adds conditions to the document-related queries: the field must match the pattern.
Compares the incoming data to a sequence of valid inputs.
Parameters: | choices – A sequence of valid inputs. |
---|
Adds conditions to the document-related queries.
Compares the incoming data to a sequence of invalid inputs.
Parameters: | choices – A sequence of invalid inputs. |
---|
Adds conditions to the document-related queries.
A storage/query backend for Tokyo Cabinet.
Allows direct access to the database and is thus extremely fast. However, it locks the database and is therefore not suitable for environments where concurrent access is required. Please use Tokyo Tyrant for such environments.
status: | beta |
---|---|
database: | Tokyo Cabinet |
dependencies: | tokyo-python, pyrant |
suitable for: | general purpose, embedded |
Warning
this module is not intended for production despite it may be stable. Bug reports and patches are welcome.
Note
this module should not depend on Pyrant; just needs some refactoring.
Note
support for metasearch is planned.
Usage:
>>> import os
>>> import doqu
>>> DB_SETTINGS = {
... 'backend': 'doqu.ext.tokyo_cabinet',
... 'path': '_tc_test.tct',
... }
>>> assert not os.path.exists(DB_SETTINGS['path']), 'test database must not exist'
>>> db = doqu.get_db(DB_SETTINGS)
>>> class Person(doqu.Document):
... structure = {'name': unicode}
... def __unicode__(self):
... u'%(name)s' % self
...
>>> Person.objects(db) # the database is expected to be empty
[]
>>> db.connection['john'] = {'name': 'John'}
>>> mary = Person(name='Mary')
>>> mary_pk = mary.save(db)
>>> q = Person.objects(db)
>>> q
[<Person John>, <Person Mary>]
>>> q.where(name__matches='^J')
[<Person John>]
>>> q # the original query was not modified by the descendant
[<Person John>, <Person Mary>]
>>> db.connection.close()
>>> os.unlink(DB_SETTINGS['path'])
Parameters: | path – relative or absolute path to the database file (e.g. test.tct) |
---|
Note
Currently only table flavour of Tokyo Cabinet databases is supported. It is uncertain whether it is worth supporting other flavours as they do not provide query mechanisms other than access by primary key.
Clears the whole storage from data, resets autoincrement counters.
Connects to the database. Raises RuntimeError if the connection is not closed yet. Use reconnect() to explicitly close the connection and open it again.
Deletes record with given primary key.
Closes internal store and removes the reference to it. If the backend works with a file, then all pending changes are saved now.
Returns instances of given class, optionally filtered by given conditions.
Parameters: |
|
---|
Note
By default this returns a tuple of (key, data_dict) per item. However, this can be changed if doc_class provides the method from_storage(). For example, Document has the notion of “saved state” so it can store the key within. Thus, only a single Document object is returned per item.
Returns document instance for given document class and primary key. Raises KeyError if there is no item with given key in the database.
Parameters: |
|
---|
Returns an iterator of documents with primary keys from given list. Basically this is just a simple wrapper around get() but some backends can reimplement the method in a much more efficient way.
Queries the database for records associated with given document class and conforming to given extra conditions. If such records exist, picks the first one (the order may be random depending on the database). If there are no such records, creates one.
Returns the document instance and a boolean value “created”.
alias of QueryAdapter
Gracefully closes current connection (if it’s not broken) and connects again to the database (e.g. reopens the file).
Saves given data with given primary key into the storage. Returns the primary key.
Parameters: |
|
---|
Note that you must provide current primary key for a record which is already in the database in order to update it instead of copying it.
Synchronizes the storage to disk immediately if the backend supports this operation. Normally the data is synchronized either on save(), or on timeout, or on disconnect(). This is strictly backend-specific. If a backend does not support the operation, NotImplementedError is raised.
The Query class.
Same as __len__ but without fetching the records (i.e. faster).
Deletes all records that match current query.
Returns a query object with same conditions but with results sorted by given field. By default the direction of sorting is ascending.
Parameters: |
|
---|
Returns an iterator that yields distinct values for given column name.
Note
this is currently highly inefficient because the underlying library does not support columns mode (tctdbiternext3). Moreover, even current implementation can be optimized by removing the overhead of creating full-blown document objects (though preserving data type is necessary).
Returns Query instance filtered by given conditions. The conditions are specified by backend’s underlying API.
Returns Query instance. Inverted version of where().
Doqu is a lightweight Python framework for document databases. It provides a uniform API for modeling, validation and queries across various kinds of storages.
It is not an ORM as it doesn’t map existing schemata to Python objects. Instead, it lets you define schemata on a higher layer built upon a schema-less storage (key/value or document-oriented). You define models as a valuable subset of the whole database and work with only certain parts of existing entities – the parts you need.
Topics:
Originally written by Andrey Mikhaylenko since 2009.
See the file AUTHORS for a complete authors list of this application.
Please feel free to submit patches, report bugs or request features:
Doqu is free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
Doqu is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public License along with Doqu. If not, see <http://gnu.org/licenses/>.
A storage/query backend for shove which is bundled with Python.
status: | beta |
---|---|
database: | any supported by shove: storage — Amazon S3 Web Service, Berkeley Source Database, Filesystem, Firebird, FTP, DBM, Durus, Memory, Microsoft SQL Server, MySQL, Oracle, PostgreSQL, SQLite, Subversion, Zope Object Database (ZODB); caching — Filesystem, Firebird, memcached, Memory, Microsoft SQL Server, MySQL, Oracle, PostgreSQL, SQLite |
dependencies: | shove |
suitable for: | “smart” interface to a key/value store; temporary memory storage |
This extension wraps the shove library and provides the uniform query API along with support for Document API.
Note
Regardless of the underlying storage, Shove serializes the records and only offers access by primary key. This means that efficient queries are impossible even with RDBMS; moreover, such databases are more likely to perform slower than simple key/value stores. The Docu queries with Shove involve iterating over the full set of records on client side and making per-row comparison without proper indexing.
That said, the backend is considered not suitable for applications that depend on queries and require decent speed of lookups by value. However, it can be very useful as a memory storage (e.g. to analyze a JSON dump or calculate some data on the fly) or as an improved interface to an existing pure key/value storage which is mostly used without advanced queries.
All parametres are optional. Here are the most common:
Parameters: |
|
---|
The URI format for a backend is documented in its module (see the shove documentation). The URI form is the same as SQLAlchemy’s.
Clears the whole storage from data, resets autoincrement counters.
Connects to the database. Raises RuntimeError if the connection is not closed yet. Use reconnect() to explicitly close the connection and open it again.
Deletes record with given primary key.
Closes internal store and removes the reference to it. If the backend works with a file, then all pending changes are saved now.
Returns instances of given class, optionally filtered by given conditions.
Parameters: |
|
---|
Note
By default this returns a tuple of (key, data_dict) per item. However, this can be changed if doc_class provides the method from_storage(). For example, Document has the notion of “saved state” so it can store the key within. Thus, only a single Document object is returned per item.
Returns document instance for given document class and primary key. Raises KeyError if there is no item with given key in the database.
Parameters: |
|
---|
Returns an iterator of documents with primary keys from given list. Basically this is just a simple wrapper around get() but some backends can reimplement the method in a much more efficient way.
Queries the database for records associated with given document class and conforming to given extra conditions. If such records exist, picks the first one (the order may be random depending on the database). If there are no such records, creates one.
Returns the document instance and a boolean value “created”.
alias of QueryAdapter
Gracefully closes current connection (if it’s not broken) and connects again to the database (e.g. reopens the file).
Saves given data with given primary key into the storage. Returns the primary key.
Parameters: |
|
---|
Note that you must provide current primary key for a record which is already in the database in order to update it instead of copying it.
Synchronizes the storage to disk immediately if the backend supports this operation. Normally the data is synchronized either on save(), or on timeout, or on disconnect(). This is strictly backend-specific. If a backend does not support the operation, NotImplementedError is raised.
The Query class.
Same as __len__ but a bit faster.
Deletes all records that match current query.
Defines order in which results should be retrieved.
Parameters: |
|
---|
Examples:
q.order_by('name') # ascending
q.order_by('name', reverse=True) # descending
If multiple names are provided, grouping is done from left to right.
Note
while you can specify the direction of sorting, it is not possible to do it on per-name basis due to backend limitations.
Warning
ordering implementation for this database is currently inefficient.
Returns an iterator that yields distinct values for given column name.
Supports date parts (i.e. date__month=7).
Note
this is currently highly inefficient because the underlying library does not support columns mode (tctdbiternext3). Moreover, even current implementation can be optimized by removing the overhead of creating full-blown document objects.
Note
unhashable values (like lists) are silently ignored.
Returns Query instance filtered by given conditions. The conditions are specified by backend’s underlying API.
Returns Query instance. Inverted version of where().
As easy as it can be:
$ pip install doqu
Another way is to use the Mercurial repo:
$ hg clone http://bitbucket.org/neithere/doqu
$ cd doqu
$ ./setup.py install
You may also need to install some other libraries (see Extensions).
A storage/query backend for MongoDB.
status: | beta |
---|---|
database: | MongoDB |
dependencies: | pymongo |
suitable for: | general purpose (mostly server-side) |
Warning
this module is not intended for production. It contains some hacks and should be refactored. However, it is actually used in a real project involving complex queries. Patches, improvements, rewrites are welcome.
Parameters: |
|
---|
Clears the whole storage from data, resets autoincrement counters.
Connects to the database. Raises RuntimeError if the connection is not closed yet. Use reconnect() to explicitly close the connection and open it again.
Deletes record with given primary key.
Closes internal store and removes the reference to it. If the backend works with a file, then all pending changes are saved now.
Returns instances of given class, optionally filtered by given conditions.
Parameters: |
|
---|
Note
By default this returns a tuple of (key, data_dict) per item. However, this can be changed if doc_class provides the method from_storage(). For example, Document has the notion of “saved state” so it can store the key within. Thus, only a single Document object is returned per item.
Returns document instance for given document class and primary key. Raises KeyError if there is no item with given key in the database.
Parameters: |
|
---|
Returns an iterator of documents with primary keys from given list. Basically this is just a simple wrapper around get() but some backends can reimplement the method in a much more efficient way.
Queries the database for records associated with given document class and conforming to given extra conditions. If such records exist, picks the first one (the order may be random depending on the database). If there are no such records, creates one.
Returns the document instance and a boolean value “created”.
Gracefully closes current connection (if it’s not broken) and connects again to the database (e.g. reopens the file).
Saves given data with given primary key into the storage. Returns the primary key.
Parameters: |
|
---|
Note that you must provide current primary key for a record which is already in the database in order to update it instead of copying it.
Synchronizes the storage to disk immediately if the backend supports this operation. Normally the data is synchronized either on save(), or on timeout, or on disconnect(). This is strictly backend-specific. If a backend does not support the operation, NotImplementedError is raised.
Returns the number of records that match given query. The result of q.count() is exactly equivalent to the result of len(q) but does not involve fetching of the records.
Deletes all records that match current query.
Returns a query object with same conditions but with results sorted by given field. By default the direction of sorting is ascending.
Parameters: |
|
---|
Returns a list of unique values for given field name.
Parameters: | name – the field name. |
---|
Note
A set is dynamically build on client side if the query contains conditions. If it doesn’t, a much more efficient approach is used. It is only available within current connection, not query.
Returns Query instance filtered by given conditions. The conditions are specified by backend’s underlying API.
Returns Query instance. Inverted version of where().
Please activate JavaScript to enable the search functionality.
From here you can search these documents. Enter your search words into the box below and click "search". Note that the search function will automatically search for all of the words. Pages containing fewer words won't appear in the result list.
A place where data is stored. Provides a single namespace. Key/value stores can be represented with a single storage object, some other databases will require multiple storage objects (e.g. each “database” of CouchDB or each “collection” of MongoDB). Docu does not use nested namespaces because in document databases they mean nothing anyway.
Doqu offers a uniform API for different databases by providing “storage adapters”. See Backend API for technical details and Extensions for a list of adapters bundled with Docu.
An dictionary with metadata. Can be associated with a record in a storage. The structure can be restricted by schema. Optional validators determine how should the document look before it can be saved into the storage, or what records can be associated with documents of given class. Special behaviour can abe added with methods of the Document subclass (see Document API).
The simplest document is just a dictionary with some metadata. The metadata can be empty or contain information about where the document comes from, what does its record look like, etc.
A document without schema or validators is equal to its record. A document with schema is only equal to the record if they have the same sets of fields and these fields are valid (i.e. have correct data types and pass certain tests).
As you see, there is a difference between documents and records but sometimes it’s very subtle.
Offers integration with WTForms.
status: | beta |
---|---|
dependencies: | wtforms |
The extension provides two new field classes: QuerySetSelectField and DocumentSelectField (inspired by wtforms.ext.django.*). They connect the forms with the Doqu API for queries. You can manually create forms with these fields.
The easiest way to create a Document-compliant form is using the function document_form_factory(). It returns a form class based on the document structure:
from doqu import Document
from doqu import validators
from doqu.ext.forms import document_form_factory
class Location(Document):
structure = {'name': unicode}
class Person(Document):
structure = {'name': unicode, 'age': int, 'location': Location}
labels = {'name': 'Full name', 'age': 'Age', 'location': 'Location'}
validators = {'name': [required()]}
PersonForm = document_form_factory(Person)
The last line does the same as this code:
from wtforms import TextField, IntegerField, validators
from doqu.ext.forms import DocumentSelectField
class PersonForm(wtforms.Form):
name = TextField('Full name', [validators.Required()])
age = IntegerField('Age')
location = DocumentSelectField('Location', [], Location)
Expects a Document instance, creates and returns a wtforms.Form class for this model.
The form fields are selected depending on the Python type declared by each property.
Parameters: |
|
---|
Caveat: the unicode type can be mapped to TextField and TextAreaField. It is impossible to guess which one should be used unless maximum length is defined for the property. TextAreaField is picked by default. It is a good idea to automatically shrink it with JavaScript so that its size always matches the contents.
Given a QuerySet either at initialization or inside a view, will display a select drop-down field of choices. The data property actually will store/keep an ORM model instance, not the ID. Submitting a choice which is not in the queryset will result in a validation error.
Specifying label_attr in the constructor will use that property of the model instance for display in the list, else the model object’s __str__ or __unicode__ will be used.
If allow_blank is set to True, then a blank choice will be added to the top of the list. Selecting this choice will result in the data property being None. The label for the blank choice can be set by specifying the blank_text parameter.
Populates obj.<name> with the field’s data.
Note : | This is a destructive operation. If obj.<name> already exists, it will be overridden. Use with caution. |
---|
Override if you need to run any field-level validation tasks after normal validation. This shouldn’t be needed in most cases.
Parameters: |
|
---|
Process incoming data, calling process_data, process_formdata as needed, and run filters.
If data is not provided, process_data will be called on the field’s default.
Field subclasses usually won’t override this, instead overriding the process_formdata and process_data methods. Only override this for special advanced processing, such as when a field encapsulates many inputs.
Process the Python data applied to this field and store the result.
This will be called during form construction by the form’s kwargs or obj argument.
Parameters: | value – The python object containing the value to process. |
---|
Validates the field and returns True or False. self.errors will contain any errors raised during validation. This is usually only called by Form.validate.
Subfields shouldn’t override this, but rather override either pre_validate, post_validate or both, depending on needs.
Parameters: |
|
---|
Like a QuerySetSelectField, except takes a document class instead of a queryset and lists everything in it.
Populates obj.<name> with the field’s data.
Note : | This is a destructive operation. If obj.<name> already exists, it will be overridden. Use with caution. |
---|
Override if you need to run any field-level validation tasks after normal validation. This shouldn’t be needed in most cases.
Parameters: |
|
---|
Process incoming data, calling process_data, process_formdata as needed, and run filters.
If data is not provided, process_data will be called on the field’s default.
Field subclasses usually won’t override this, instead overriding the process_formdata and process_data methods. Only override this for special advanced processing, such as when a field encapsulates many inputs.
Process the Python data applied to this field and store the result.
This will be called during form construction by the form’s kwargs or obj argument.
Parameters: | value – The python object containing the value to process. |
---|
Validates the field and returns True or False. self.errors will contain any errors raised during validation. This is usually only called by Form.validate.
Subfields shouldn’t override this, but rather override either pre_validate, post_validate or both, depending on needs.
Parameters: |
|
---|
Documents represent database records. Each document is a (in)complete subset of fields contained in a record. Available data types and query mechanisms are determined by the storage in use.
The API was inspired by Django, MongoKit, WTForms, Svarga and several other projects. It was important to KISS (keep it simple, stupid), DRY (do not repeat yourself) and to make the API as abstract as possible so that it did not depend on backends and yet did not get in the way.
A document/query object. Dict-like representation of a document stored in a database. Includes schema declaration, bi-directional validation (outgoing and query), handles relations and has the notion of the saved state, i.e. knows the storage and primary key of the corresponding record.
Returns given query filtered by schema and validators defined for this document.
Deletes the object from the associated storage.
Returns a document instance filled with given data and bound to given storage and key. The instance can be safely saved back using save(). If the concrete subclass defines the structure, then usused fields coming from the storage are hidden from the public API but nevertheless they will be saved back to the database as is.
Returns current primary key (if any) or None.
Saves instance to given storage.
Parameters: |
|
---|
Checks if instance data is valid. This involves a) checking whether all values correspond to the declated structure, and b) running all Validators against the data dictionary.
Raises ValidationError if something is wrong.
Note
if the data dictionary does not contain some items determined by structure or validators, these items are not checked.
Note
The document is checked as is. There are no side effects. That is, if some required values are empty, they will be considered invalid even if default values are defined for them. The save() method, however, fills in the default values before validating.
alias of OneToManyRelation