https://secure.travis-ci.org/yourlabs/django-cities-light.png?branch=master https://pypip.in/d/django-cities-light/badge.png https://pypip.in/v/django-cities-light/badge.png https://codecov.io/github/yourlabs/django-cities-light/coverage.svg?branch=stable/3.x.x

django-cities-light – Simple django-cities alternative

This add-on provides models and commands to import country, region/state, and city data in your database.

The data is pulled from GeoNames and contains cities, regions/states and countries.

Spatial query support is not required by this application.

This application is very simple and is useful if you want to make a simple address book for example. If you intend to build a fully featured spatial database, you should use django-cities.

Requirements:

  • Python 2.7 or 3.3,
  • Django >= 1.7
  • MySQL or PostgreSQL or SQLite.

Yes, for some reason, code that used to work on MySQL (not without pain xD) does not work anymore. So we’re now using django.db.transaction.atomic which comes from Django 1.6 just to support MySQL quacks.

Upgrade

See CHANGELOG.

Installation

Install django-cities-light:

pip install django-cities-light

Or the development version:

pip install -e git+git@github.com:yourlabs/django-cities-light.git#egg=cities_light

Add cities_light to your INSTALLED_APPS.

Configure filters to exclude data you don’t want, ie.:

CITIES_LIGHT_TRANSLATION_LANGUAGES = ['fr', 'en']
CITIES_LIGHT_INCLUDE_COUNTRIES = ['FR']
CITIES_LIGHT_INCLUDE_CITY_TYPES = ['PPL', 'PPLA', 'PPLA2', 'PPLA3', 'PPLA4', 'PPLC', 'PPLF', 'PPLG', 'PPLL', 'PPLR', 'PPLS', 'STLMT',]

Now, run migrations, it will only create tables for models that are not disabled:

./manage.py migrate

Data update

Finally, populate your database with command:

./manage.py cities_light

This command is well documented, consult the help with:

./manage.py help cities_light

Development

To build the docs use the following steps:

  1. mkvirtualenv dcl-doc
  2. pip install -e ./
  3. pip install -r docs/requirements.txt
  4. cd docs
  5. make html

Resources

You could subscribe to the mailing list ask questions or just be informed of package updates.

Contents:

Populating the database

Data install or update

Populate your database with command:

./manage.py cities_light

By default, this command attempts to do the least work possible, update what is necessary only. If you want to disable all these optimisations/skips, use –force-all.

This command is well documented, consult the help with:

./manage.py help cities_light

Signals

Signals for this application.

cities_light.signals.city_items_pre_import

Emited by city_import() in the cities_light command for each row parsed in the data file. If a signal reciever raises InvalidItems then it will be skipped.

An example is worth 1000 words: if you want to import only cities from France, USA and Belgium you could do as such:

import cities_light

def filter_city_import(sender, items, **kwargs):
    if items[8] not in ('FR', 'US', 'BE'):
        raise cities_light.InvalidItems()

cities_light.signals.city_items_pre_import.connect(filter_city_import)

Note: this signal gets a list rather than a City instance for performance reasons.

cities_light.signals.region_items_pre_import

Same as city_items_pre_import.

cities_light.signals.country_items_pre_import

Same as region_items_pre_import and cities_light.signals.city_items_pre_import.

cities_light.signals.city_items_post_import

Emited by city_import() in the cities_light command for each row parsed in the data file, right before saving City object. Along with City instance it pass items with geonames data. Will be useful, if you define custom cities models with settings.CITIES_LIGHT_APP_NAME.

Example:

import cities_light

def process_city_import(sender, instance, items, **kwargs):
    instance.timezone = items[17]

cities_light.signals.city_items_post_import.connect(process_city_import)
cities_light.signals.region_items_post_import

Same as city_items_post_import.

cities_light.signals.country_items_post_import

Same as region_items_post_import and cities_light.signals.city_items_post_import.

exception cities_light.exceptions.CitiesLightException[source]

Base exception class for this app’s exceptions.

exception cities_light.exceptions.InvalidItems[source]

The cities_light command will skip item if a city_items_pre_import signal reciever raises this exception.

exception cities_light.exceptions.SourceFileDoesNotExist(source)[source]

A source file could not be found.

Configure logging

This command is made to be compatible with background usage like from cron, to keep the database fresh. So it doesn’t do direct output. To get output from this command, simply configure a handler and formatter for cities_light logger. For example:

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'simple': {
            'format': '%(levelname)s %(message)s'
        },
    },
    'handlers': {
        'console':{
            'level':'DEBUG',
            'class':'logging.StreamHandler',
            'formatter': 'simple'
        },
    },
    'loggers': {
        'cities_light': {
            'handlers':['console'],
            'propagate': True,
            'level':'DEBUG',
        },
        # also use this one to see SQL queries
        'django': {
            'handlers':['console'],
            'propagate': True,
            'level':'DEBUG',
        },
    }
}

Simple django app

Settings

Settings for this application. The most important is TRANSLATION_LANGUAGES because it’s probably project specific.

cities_light.settings.TRANSLATION_LANGUAGES

List of language codes. It is used to generate the alternate_names property of cities_light models. You want to keep it as small as possible. By default, it includes the most popular languages according to wikipedia, which use a rather ascii-compatible alphabet. It also contains ‘abbr’ which stands for ‘abbreviation’, you might want to include this one as well.

See:

Example:

CITIES_LIGHT_TRANSLATION_LANGUAGES = ['es', 'en', 'fr', 'abbr']
cities_light.settings.INCLUDE_COUNTRIES

List of country codes to include. It’s None by default which lets all countries in the database. But if you only wanted French and Belgium countries/regions/cities, you could set it as such:

CITIES_LIGHT_INCLUDE_COUNTRIES = ['FR', 'BE']
cities_light.settings.INCLUDE_CITY_TYPES

List of city feature codes to include. They are described at http://www.geonames.org/export/codes.html, section “P city, village”.

CITIES_LIGHT_INCLUDE_CITY_TYPES = [
‘PPL’, ‘PPLA’, ‘PPLA2’, ‘PPLA3’, ‘PPLA4’, ‘PPLC’, ‘PPLF’, ‘PPLG’, ‘PPLL’, ‘PPLR’, ‘PPLS’, ‘STLMT’,

]

cities_light.settings.COUNTRY_SOURCES

A list of urls to download country info from. Default is countryInfo.txt from geonames download server. Overridable in settings.CITIES_LIGHT_COUNTRY_SOURCES.

cities_light.settings.REGION_SOURCES

A list of urls to download region info from. Default is admin1CodesASCII.txt from geonames download server. Overridable in settings.CITIES_LIGHT_REGION_SOURCES.

cities_light.settings.CITY_SOURCES

A list of urls to download city info from. Default is cities15000.zip from geonames download server. Overridable in settings.CITIES_LIGHT_CITY_SOURCES.

cities_light.settings.TRANSLATION_SOURCES

A list of urls to download alternate names info from. Default is alternateNames.zip from geonames download server. Overridable in settings.CITIES_LIGHT_TRANSLATION_SOURCES.

cities_light.settings.SOURCES

A list with all sources, auto-generated.

cities_light.settings.DATA_DIR

Absolute path to download and extract data into. Default is cities_light/data. Overridable in settings.CITIES_LIGHT_DATA_DIR

cities_light.settings.INDEX_SEARCH_NAMES

If your database engine for cities_light supports indexing TextFields (ie. it is not MySQL), then this should be set to True. You might have to override this setting with settings.CITIES_LIGHT_INDEX_SEARCH_NAMES if using several databases for your project.

cities_light.settings.CITIES_LIGHT_APP_NAME

Modify it only if you want to define your custom cities models, that are inherited from abstract models of this package. It must be equal to app name, where custom models are defined. For example, if they are in geo/models.py, then set settings.CITIES_LIGHT_APP_NAME = 'geo'. Note: you can’t define one custom model, you have to define all of cities_light models, even if you want to modify only one.

class cities_light.settings.ICountry[source]

Country field indexes in geonames.

class cities_light.settings.IRegion[source]

Region field indexes in geonames.

class cities_light.settings.ICity[source]

City field indexes in geonames. Description of fields: http://download.geonames.org/export/dump/readme.txt

class cities_light.settings.IAlternate[source]

Alternate names field indexes in geonames. Description of fields: http://download.geonames.org/export/dump/readme.txt

Models

See source for details.

By default, all models are taken from this package. But it is possible to customise these models to add some fields. For such purpose cities_light models are defined as abstract (without customisation they all inherit abstract versions automatically without changes).

Steps to customise cities_light models
  • Define all of cities abstract models in your app:
    # yourapp/models.py
    
    from cities_light.abstract_models import (AbstractCity, AbstractRegion,
        AbstractCountry)
    from cities_light.receivers import connect_default_signals
    
    
    class Country(AbstractCountry):
        pass
    connect_default_signals(Country)
    
    class Region(AbstractRegion):
        pass
    connect_default_signals(Region)
    
    
    class City(AbstractCity):
        timezone = models.CharField(max_length=40)
    connect_default_signals(City)
    
  • Add post import processing to you model [optional]:
    import cities_light
    from cities_light.settings import ICity
    
    def set_city_fields(sender, instance, items, **kwargs):
        instance.timezone = items[ICity.timezone]
    cities_light.signals.city_items_post_import.connect(set_city_fields)
    
  • Define settings.py:
    INSTALLED_APPS = [
        # ...
        'cities_light',
        'yourapp',
    ]
    
    CITIES_LIGHT_APP_NAME = 'yourapp'
    
  • Create tables:
    python manage.py syncdb
    

That’s all!

Notes:
  • model names can’t be modified, i.e. you have to use exactly City, Country, Region names and not MyCity, MyCountry, MyRegion.
  • Connect default signals for every custom model by calling connect_default_signals (or not, if you don’t want to trigger default signals).
  • if in further versions of cities_light abstract models will be updated (some fields will be added/removed), you have to deal with migrations by yourself, as models are on your own now.

Convert a string value into a string that is usable against City.search_names.

For example, ‘Paris Texas’ would become ‘paristexas’.

cities_light.models.filter_non_cities(sender, items, **kwargs)[source]

Exclude any city which feature code must not be included. By default, this receiver is connected to city_items_pre_import(), it raises InvalidItems if the row feature code is not in the INCLUDE_CITY_TYPES setting.

cities_light.models.filter_non_included_countries_country(sender, items, **kwargs)[source]

Exclude any country which country must not be included. This is slot is connected to the country_items_pre_import() signal and does nothing by default. To enable it, set the INCLUDE_COUNTRIES setting.

cities_light.models.filter_non_included_countries_region(sender, items, **kwargs)[source]

Exclude any region which country must not be included. This is slot is connected to the region_items_pre_import() signal and does nothing by default. To enable it, set the INCLUDE_COUNTRIES setting.

cities_light.models.filter_non_included_countries_city(sender, items, **kwargs)[source]

Exclude any city which country must not be included. This is slot is connected to the city_items_pre_import() signal and does nothing by default. To enable it, set the INCLUDE_COUNTRIES setting.

class cities_light.models.Country(id, name_ascii, slug, geoname_id, alternate_names, name, code2, code3, continent, tld, phone)[source]
class cities_light.models.Region(id, name_ascii, slug, geoname_id, alternate_names, name, display_name, geoname_code, country)[source]
class cities_light.models.City(id, name_ascii, slug, geoname_id, alternate_names, name, display_name, search_names, latitude, longitude, region, country, population, feature_code)[source]

Admin

See source for details.

class cities_light.admin.CityAdmin(model, admin_site)[source]

ModelAdmin for City.

form

alias of CityForm

class cities_light.admin.CountryAdmin(model, admin_site)[source]

ModelAdmin for Country.

form

alias of CountryForm

class cities_light.admin.RegionAdmin(model, admin_site)[source]

ModelAdmin for Region.

form

alias of RegionForm

cities_light.contrib

For django-ajax-selects

Couples cities_light and django-ajax-selects.

Register the lookups in settings.AJAX_LOOKUP_CHANNELS, add:

'cities_light_country': ('cities_light.lookups', 'CountryLookup'),
'cities_light_city': ('cities_light.lookups', 'CityLookup'),
class cities_light.contrib.ajax_selects_lookups.CityLookup[source]

Lookup channel for City, hits name and search_names.

model

alias of City

class cities_light.contrib.ajax_selects_lookups.CountryLookup[source]

Lookup channel for Country, hits name and name_ascii.

model

alias of Country

class cities_light.contrib.ajax_selects_lookups.RegionLookup[source]

Lookup channel for Region, hits name and name_ascii.

model

alias of Region

class cities_light.contrib.ajax_selects_lookups.StandardLookupChannel[source]

Honnestly I’m not sure why this is here.

format_item_display(obj)[source]

(HTML) formatted item for displaying item in the selected deck area

format_match(obj)[source]

(HTML) formatted item for displaying item in the dropdown

For djangorestframework

The contrib contains support for v1, v2 and v3 of django restframework.

Django REST framework 3

This contrib package defines list and detail endpoints for City, Region and Country. If rest_framework (v3) is installed, all you have to do is add this url include:

url(r'^cities_light/api/', include('cities_light.contrib.restframework3')),

This will configure six endpoints:

^cities/$ [name='cities-light-api-city-list']
^cities/(?P<pk>[^/]+)/$ [name='cities-light-api-city-detail']
^countries/$ [name='cities-light-api-country-list']
^countries/(?P<pk>[^/]+)/$ [name='cities-light-api-country-detail']
^regions/$ [name='cities-light-api-region-list']
^regions/(?P<pk>[^/]+)/$ [name='cities-light-api-region-detail']
All list endpoints support search with a query parameter q::
/cities/?q=london

For Region and Country endpoints, the search will be within name_ascii field while for City it will search in search_names field. HyperlinkedModelSerializer is used for these models and therefore every response object contains url to self field and urls for related models. You can configure pagination using the standard rest_framework pagination settings in your project settings.py.

Couple djangorestframework and cities_light.

It defines a urlpatterns variables, with the following urls:

  • cities-light-api-city-list
  • cities-light-api-city-detail
  • cities-light-api-region-list
  • cities-light-api-region-detail
  • cities-light-api-country-list
  • cities-light-api-country-detail

If rest_framework (v3) is installed, all you have to do is add this url include:

url(r'^cities_light/api/', include('cities_light.contrib.restframework3')),

And that’s all !

Ideas for contributions

  • templatetag to render a city’s map using some external service
  • flag images, maybe with django-countryflags
  • currencies
  • generate po files when parsing alternate names

FAQ

MySQL errors with special characters, how to fix it ?

The cities_light command is continuously tested on travis-ci on all supported databases: if it works there then it should work for you.

If you’re new to development in general, you might not be familiar with the concept of encodings and collations. Unless you have a good reason, you must have utf-8 database tables. See MySQL documentation for details.

We’re pointing to MySQL documentations because PostgreSQL users probably know what UTF-8 is and won’t have any problem with that.

Some data fail to import, how to skip them ?

GeoNames is not perfect and there might be some edge cases from time to time. We want the cities_light management command to work for everybody so you should open an issue in GitHub if you get a crash from that command.

However, we don’t want you to be blocked, so keep in mind that you can use Signals like cities_light.city_items_pre_import, cities_light.region_items_pre_import, cities_light.country_items_pre_import, to skip or fix items before they get inserted in the database by the normal process.

Indices and tables