Django Database Backup¶
This Django application provides management commands to help backup and restore your project database and media files with various storages such as Amazon S3, DropBox or local file system.
It is made to:
- Allow you to secure your backup with GPG signature and encryption
- Archive with compression
- Deal easily with remote archiving
- Keep your development database up to date
- Use Crontab or Celery to setup automated backups
Warning
Django DBBackup version 3 is very different to its predecessors. See Upgrade documentation to help to get up to date.
Contents:
Installation¶
Installing on your system¶
Getting the latest stable release¶
pip install django-dbbackup
Getting the latest release from trunk¶
In general, you should not be downloading and installing stuff directly from repositories – especially not if you are backing up sensitive data.
Security is important, bypassing PyPi repositories is a bad habit, because it will bypass the fragile key signatures authentication that are at least present when using PyPi repositories.
pip install -e git+https://github.com/mjs7231/django-dbbackup.git#egg=django-dbbackup
Add it in your project¶
In your settings.py
, make sure you have the following things:
INSTALLED_APPS = (
...
'dbbackup', # django-dbbackup
)
DBBACKUP_STORAGE = 'django.core.files.storage.FileSystemStorage'
DBBACKUP_STORAGE_OPTIONS = {'location': '/my/backup/dir/'}
Create the backup directory:
mkdir /var/backups
Note
This configuration uses filesystem storage, but you can use any storage supported by Django API. See Storage settings for more information about it.
Testing that everything worked¶
Now, you should be able to create your first backup by running:
$ python manage.py dbbackup
If your database was called default
which is the normal Django behaviour
of a single-database project, you should now see a new file in your backup
directory.
Configuration¶
General settings¶
DBBACKUP_DATABASES¶
List of key entries for settings.DATABASES
which shall be used to
connect and create database backups.
Default: list(settings.DATABASES.keys())
(keys of all entries listed)
DBBACKUP_TMP_DIR¶
Directory to be used in local filesystem for temporary files.
Default: tempfile.gettempdir()
DBBACKUP_TMP_FILE_MAX_SIZE¶
Maximum size in bytes for file handling in memory before a temporary
file is written in DBBACKUP_TMP_DIR
.
Default: 10*1024*1024
DBBACKUP_CLEANUP_KEEP and DBBACKUP_CLEANUP_KEEP_MEDIA¶
When issuing dbbackup
and mediabackup
with --clean
option, the
number of old backup files that are looked for and removed.
Default: 10
(backups)
DBBACKUP_CLEANUP_FILTER¶
A callable that takes a filename (of an old backup, to be cleaned) and returns
a boolean indicating whether the backup should be kept (True
) or deleted
(False
).
Default: lambda filename: False
This can be used to keep monthly backups, for example.
DBBACKUP_DATE_FORMAT¶
Date format to use for naming files. It must contain only alphanumerical
characters, '_'
, '-'
or '%'
.
Default: '%Y-%m-%d-%H%M%S'
DBBACKUP_HOSTNAME¶
Used to identify a backup by a server name in the file name.
Default: socket.gethostname()
DBBACKUP_FILENAME_TEMPLATE¶
The template to use when generating the backup filename. By default this is
'{databasename}-{servername}-{datetime}.{extension}'
. This setting can
also be made a function which takes the following keyword arguments:
def backup_filename(databasename, servername, datetime, extension, content_type):
pass
DBBACKUP_FILENAME_TEMPLATE = backup_filename
This allows you to modify the entire format of the filename, for example, if you want to take advantage of Amazon S3’s automatic expiry feature, you need to prefix your backups differently based on when you want them to expire.
{datetime}
is rendered with DBBACKUP_DATE_FORMAT
.
DBBACKUP_MEDIA_FILENAME_TEMPLATE¶
Same as DBBACKUP_FILENAME_TEMPLATE
but for media files backups.
Encrypting your backups¶
Considering that you might be putting secure data on external servers and perhaps untrusted servers where it gets forgotten over time, it’s always a good idea to encrypt backups.
Just remember to keep the encryption keys safe, too!
PGP¶
You can encrypt a backup with the --encrypt
option. The backup is done
using GPG.
python manage.py dbbackup --encrypt
…or when restoring from an encrypted backup:
python manage.py dbrestore --decrypt
Requirements:
- Install the python package python-gnupg:
pip install python-gnupg>=0.5.0
. - You need a GPG key. (GPG manual)
- Set the setting
DBBACKUP_GPG_RECIPIENT
to the name of the GPG key.
DBBACKUP_GPG_ALWAYS_TRUST¶
The encryption of the backup file fails if GPG does not trust the public
encryption key. The solution is to set the option ‘trust-model’ to ‘always’.
By default this value is False
. Set this to True
to enable this option.
DBBACKUP_GPG_RECIPIENT¶
The name of the key that is used for encryption. This setting is only used
when making a backup with the --encrypt
or --decrypt
option.
Email configuration¶
DBBACKUP_SEND_EMAIL¶
Controls whether or not django-dbbackup sends an error email when an uncaught exception is received.
Default: True
DBBACKUP_SERVER_EMAIL¶
The email address that error messages come from, such as those sent to
DBBACKUP_ADMINS
.
Default: django.conf.settings.SERVER_EMAIL
DBBACKUP_ADMINS¶
A list of all the people who get code error notifications. When DEBUG=False
and an operation raises an exception, DBBackup will email these people with the
full exception information. This should be a tuple of (Full name,
email address).
Default: django.conf.settings.ADMINS
Warning
DBBACKUP_FAILURE_RECIPIENTS
was used before and is now deprecated
DBBACKUP_EMAIL_SUBJECT_PREFIX¶
Subject-line prefix for email messages sent by DBBackup.
Default: '[dbbackup] '
Database configuration¶
By default, DBBackup uses parameters from settings.DATABASES
but you can
make an independent configuration, see Database settings
Storage configuration¶
You have to use a storage for your backups, see Storage settings for more.
Database settings¶
The following databases are supported by this application:
- SQLite
- MySQL
- PostgreSQL
- MongoDB
- …and any other that you might implement
By default, DBBackup will try to use your database settings in DATABASES
to handle the database, but some databases require custom options so you might
want to use different parameters for backups. That’s why we included a
DBBACKUP_CONNECTORS
setting; it follows the form of the django DATABASES
setting:
DBBACKUP_CONNECTORS = {
'default': {
'USER': 'backupuser',
'PASSWORD': 'backuppassword',
'HOST': 'replica-for-backup'
}
}
This configuration will allow you to use a replica with a different host and user, which is a great practice if you don’t want to overload your main database.
DBBackup uses Connectors
for creating and restoring backups; below you’ll see
specific parameters for the built-in ones.
Common¶
All connectors have the following parameters:
CONNECTOR¶
Absolute path to a connector class by default is:
dbbackup.db.sqlite.SqliteConnector
for'django.db.backends.sqlite3'
dbbackup.db.mysql.MysqlDumpConnector
fordjango.db.backends.mysql
dbbackup.db.postgresql.PgDumpConnector
fordjango.db.backends.postgresql
dbbackup.db.postgresql.PgDumpGisConnector
fordjango.contrib.gis.db.backends.postgis
dbbackup.db.mongodb.MongoDumpConnector
fordjango_mongodb_engine
All supported built-in connectors are described in more detail below.
Following database wrappers from django-prometheus
module are supported:
django_prometheus.db.backends.postgresql
fordbbackup.db.postgresql.PgDumpBinaryConnector
django_prometheus.db.backends.sqlite3
fordbbackup.db.sqlite.SqliteConnector
django_prometheus.db.backends.mysql
fordbbackup.db.mysql.MysqlDumpConnector
django_prometheus.db.backends.postgis
fordbbackup.db.postgresql.PgDumpGisConnector
EXCLUDE¶
Tables to exclude from backup as list. This option may be unavailable for connectors when making snapshots.
EXTENSION¶
Extension of backup file name, default 'dump'
.
Command connectors¶
Some connectors use a command line tool as a dump engine, mysqldump
for
example. These kinds of tools have common attributes:
DUMP_CMD¶
Path to the command used to create a backup; default is the appropriate
command supposed to be in your PATH, for example: 'mysqldump'
for MySQL.
This setting is useful only for connectors using command line tools (children
of dbbackup.db.base.BaseCommandDBConnector
)
RESTORE_CMD¶
Same as DUMP_CMD
but used when restoring.
DUMP_PREFIX and RESTORE_PREFIX¶
String to include as prefix of dump or restore command. It will be added with a space between the launched command and its prefix.
DUMP_SUFFIX and RESTORE_PREFIX¶
String to include as suffix of dump or restore command. It will be added with a space between the launched command and its suffix.
ENV, DUMP_ENV and RESTORE_ENV¶
Environment variables used during command running, default are {}
. ENV
is used for every command, DUMP_ENV
and RESTORE_ENV
override the
values defined in ENV
during the dedicated commands.
USE_PARENT_ENV¶
Specify if the connector will use its parent’s environment variables. By
default it is True
to keep PATH
.
SQLite¶
SQLite uses by default dbbackup.db.sqlite.SqliteConnector
.
SqliteConnector¶
It is in pure Python and copies the behavior of .dump
command for creating a
SQL dump.
SqliteCPConnector¶
You can also use dbbackup.db.sqlite.SqliteCPConnector
for making a
simple raw copy of your database file, like a snapshot.
In-memory database aren’t dumpable with it.
MySQL¶
MySQL uses by default dbbackup.db.mysql.MysqlDumpConnector
. It uses
mysqldump
and mysql
for its job.
PostgreSQL¶
Postgres uses by default dbbackup.db.postgresql.PgDumpConnector
, but
we advise you to use dbbackup.db.postgresql.PgDumpBinaryConnector
. The
first one uses pg_dump
and pqsl
for its job, creating RAW SQL files.
The second uses pg_restore
with binary dump files.
They can also use psql
for launching administration command.
SINGLE_TRANSACTION¶
When doing a restore, wrap everything in a single transaction, so errors cause a rollback.
This corresponds to --single-transaction
argument of psql
and
pg_restore
.
Default: True
DROP¶
With PgDumpConnector
, it includes tables dropping statements in dump file.
PgDumpBinaryConnector
drops at restoring.
This corresponds to --clean
argument of pg_dump
and pg_restore
.
Default: True
PostGIS¶
Set in dbbackup.db.postgresql.PgDumpGisConnector
, it does the same as
PostgreSQL but launches CREATE EXTENSION IF NOT EXISTS postgis;
before
restore database.
PSQL_CMD¶
Path to psql
command used for administration tasks like enable PostGIS
for example, default is psql
.
PASSWORD¶
If you fill this settings PGPASSWORD
environment variable will be used
with every commands. For security reason, we advise to use .pgpass
file.
ADMIN_USER¶
Username used for launch action with privileges, extension creation for example.
ADMIN_PASSWORD¶
Password used for launch action with privileges, extension creation for example.
MongoDB¶
MongoDB uses by default dbbackup.db.mongodb.MongoDumpConnector
. it
uses mongodump
and mongorestore
for its job.
For AuthEnabled MongoDB Connection, you need to add one custom option AUTH_SOURCE
in your DBBACKUP_CONNECTORS
.
DBBACKUP_CONNECTORS = {
'default': {
...
'AUTH_SOURCE': 'admin',
}
}
Or in DATABASES
one:
DATABASES = {
'default': {
...
'AUTH_SOURCE': 'admin',
}
}
OBJECT_CHECK¶
Validate documents before inserting in database (option --objcheck
in command line), default is True
.
DROP¶
Replace objects that are already in database, (option --drop
in command line), default is True
.
Custom connector¶
Creating your connector is easy; create a children class from
dbbackup.db.base.BaseDBConnector
and create _create_dump
and
_restore_dump
. If your connector uses a command line tool, inherit it from
dbbackup.db.base.BaseCommandDBConnector
Connecting a Custom connector¶
Here is an example, on how to easily connect a custom connector that you have created or even that you simply want to reuse:
DBBACKUP_CONNECTOR_MAPPING = {
'transaction_hooks.backends.postgis': 'dbbackup.db.postgresql.PgDumpGisConnector',
}
Obviously instead of dbbackup.db.postgresql.PgDumpGisConnector
you can
use the custom connector you have created yourself and transaction_hooks.backends.postgis
is simply the database engine name you are using.
Storage settings¶
One of the most helpful features of django-dbbackup is the ability to store and retrieve backups from a local or a remote storage. This functionality is mainly based on Django Storage API and extends its possibilities.
You can choose your backup storage backend by setting settings.DBBACKUP_STORAGE
,
it must be the full path of a storage class. For example:
django.core.files.storage.FileSystemStorage
to use file system storage.
Below, we’ll list some of the available solutions and their options.
The storage’s option are gathered in settings.DBBACKUP_STORAGE_OPTIONS
which
is a dictionary of keywords representing how to configure it.
Warning
Do not configure backup storage with the same configuration as your media files as you’ll risk sharing backups inside public directories.
By default DBBackup uses the built-in file system storage to manage files on a local directory. Feel free to use any Django storage, you can find a variety of them at Django Packages.
Note
Storing backups to local disk may also be useful for Dropbox if you already have the official Dropbox client installed on your system.
File system storage¶
Setup¶
To store your backups on the local file system, simply setup the required settings below.
DBBACKUP_STORAGE = 'django.core.files.storage.FileSystemStorage'
DBBACKUP_STORAGE_OPTIONS = {'location': '/my/backup/dir/'}
Available settings¶
location
Absolute path to the directory that will hold the files.
file_permissions_mode
The file system permissions that the file will receive when it is saved.
directory_permissions_mode
The file system permissions that the directory will receive when it is saved.
See FileSystemStorage’s documentation for a full list of available settings.
Google cloud storage¶
Our backend for Google cloud storage uses django-storages.
Setup¶
In order to backup to Google cloud storage, you’ll first need to create an account at google. Once that is complete, you can follow the required setup below.
pip install django-storages[google]
Add the following to your project’s settings. Strictly speaking only bucket_name is required, but we’d recommend to add the other two as well. You can also find more settings in the readme for django-storages
DBBACKUP_STORAGE = 'storages.backends.gcloud.GoogleCloudStorage'
DBBACKUP_STORAGE_OPTIONS = {
"bucket_name": "your_bucket_name",
"project_id": "your_project_id",
"blob_chunk_size": 1024 * 1024
}
Amazon S3¶
Our S3 backend uses Django Storages which uses boto3.
Setup¶
In order to backup to Amazon S3, you’ll first need to create an Amazon Webservices Account and setup your Amazon S3 bucket. Once that is complete, you can follow the required setup below.
pip install django-storages[boto3]
Add the following to your project’s settings:
DBBACKUP_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
DBBACKUP_STORAGE_OPTIONS = {
'access_key': 'my_id',
'secret_key': 'my_secret',
'bucket_name': 'my_bucket_name',
'default_acl': 'private',
}
Available settings¶
Note
See the Django Storage S3 storage official documentation for all options.
The options listed here are a selection of dictionary keys returned by
get_default_settings
in django-storages’ storages/backends/s3boto3.py,
which allows us to write nicer code compared to using the AWS_
prefixed
settings.
access_key - Required
Your AWS access key as string. This can be found on your Amazon Account Security Credentials page.
secret_key - Required
Your Amazon Web Services secret access key, as a string.
bucket_name - Required
Your Amazon Web Services storage bucket name, as a string. This directory must exist before attempting to create your first backup.
region_name - Optional
Specify the Amazon region, e.g. 'us-east-1'
endpoint_url - Optional
Set this to fully override the endpoint, e.g. to use an alternative S3 service,
which is compatible with AWS S3. The value must contain the protocol, e.g.
'https://compatible-s3-service.example.com'
.
default_acl - Required
This setting can either be 'private'
or 'public'
. Since you want your
backups to be secure you’ll want to set 'default_acl'
to 'private'
.
NOTE: This value will be removed in a future version of django-storages. See their CHANGELOG for details.
location - Optional
If you want to store your backups inside a particular folder in your bucket you need to specify the 'location'
.
The folder can be specified as 'folder_name/'
.
You can specify a longer path with 'location': 'root_folder/sub_folder/sub_sub_folder/'
.
Dropbox¶
In order to backup to Dropbox, you’ll first need to create a Dropbox account and set it up to communicate with the Django-DBBackup application. Don’t worry, all instructions are below.
Setup your Dropbox account¶
- Login to Dropbox and navigate to Developers » MyApps. https://www.dropbox.com/developers/apps
- Click the button to create a new app and name it whatever you like. As an example, I named mine ‘Website Backups’.
- After your app is created, note the options button and more importantly the ‘App Key’ and ‘App Secret’ values inside. You’ll need those later.
Setup your Django project¶
pip install dropbox django-storages
…And make sure you have the following required settings:
DBBACKUP_STORAGE = 'storages.backends.dropbox.DropBoxStorage'
DBBACKUP_STORAGE_OPTIONS = {
'oauth2_access_token': 'my_token',
}
Available settings¶
Note
See django-storages dropbox official documentation for more details.
oauth2_access_token - Required
Your OAuth access token
root_path
Jail storage to this directory
FTP¶
To store your database backups on a remote filesystem via [a]FTP, simply setup the required settings below.
Setup¶
pip install django-storages
Warning
This storage doesn’t use a private connection for communication so don’t use it if you’re not certain about the security of the link between the client and the server.
DBBACKUP_STORAGE = 'storages.backends.ftp.FTPStorage
DBBACKUP_STORAGE_OPTIONS = {
'location': 'ftp://user:pass@server:21'
}
Settings¶
location - Required
A FTP URI with optional user, password and port. example: 'ftp://anonymous@myftp.net'
Setup¶
We use FTP backend from Django-Storages (again).
pip install django-storages
Here a simple configuration:
DBBACKUP_STORAGE = 'storages.backends.ftp.FTPStorage'
DBBACKUP_STORAGE_OPTIONS = {'location': ftp://myftpserver/}
SFTP¶
To store your database backups on a remote filesystem via SFTP, simply setup the required settings below.
Setup¶
This backend is from Django-Storages with the paramiko backend.
pip install paramiko django-storages
The following configuration grants SSH server access to the local user:
DBBACKUP_STORAGE = 'storages.backends.sftpstorage.SFTPStorage'
DBBACKUP_STORAGE_OPTIONS = {'host': 'myserver'}
Available settings¶
host - Required
Hostname or address of the SSH server
root_path - Default ~/
Jail storage to this directory
params - Default {}
Argument used by method:paramikor.SSHClient.connect(). See paramiko SSHClient.connect() documentation for details.
interactive - Default False
A boolean indicating whether to prompt for a password if the connection cannot
be made using keys, and there is not already a password in params
.
file_mode
UID of the account that should be set as owner of the files on the remote.
dir_mode
GID of the group that should be set on the files on the remote host.
known_host_file
Absolute path of known_hosts file, if it isn’t set "~/.ssh/known_hosts"
will be used.
Commands¶
The primary usage of DBBackup is with command line tools. By default, commands will create backups and upload to your defined storage or download and restore the latest backup.
Arguments can be passed to commands to compress/uncompress and encrypt/decrypt.
dbbackup¶
Backup of database.
$ ./manage.py dbbackup
Backing Up Database: /tmp/tmp.x0kN9sYSqk
Backup size: 3.3 KiB
Writing file to tmp-zuluvm-2016-07-29-100954.dump
Help¶
usage: manage.py dbbackup [-h] [--noinput] [-q] [-c] [-d DATABASE] [-s SERVERNAME] [-z] [-e] [-o OUTPUT_FILENAME] [-O OUTPUT_PATH] [-x EXCLUDE_TABLES] [--version] [-v {0,1,2,3}] [--settings SETTINGS] [--pythonpath PYTHONPATH] [--traceback] [--no-color] [--force-color] [--skip-checks] Backup a database, encrypt and/or compress and write to storage. optional arguments: -h, --help show this help message and exit --noinput Tells Django to NOT prompt the user for input of any kind. -q, --quiet Tells Django to NOT output other text than errors. -c, --clean Clean up old backup files -d DATABASE, --database DATABASE Database(s) to backup specified by key separated by commas(default: all) -s SERVERNAME, --servername SERVERNAME Specify server name to include in backup filename -z, --compress Compress the backup files -e, --encrypt Encrypt the backup files -o OUTPUT_FILENAME, --output-filename OUTPUT_FILENAME Specify filename on storage -O OUTPUT_PATH, --output-path OUTPUT_PATH Specify where to store on local filesystem -x EXCLUDE_TABLES, --exclude-tables EXCLUDE_TABLES Exclude tables from backup --version show program's version number and exit -v {0,1,2,3}, --verbosity {0,1,2,3} Verbosity level; 0=minimal output, 1=normal output, 2=verbose output, 3=very verbose output --settings SETTINGS The Python path to a settings module, e.g. "myproject.settings.main". If this isn't provided, the DJANGO_SETTINGS_MODULE environment variable will be used. --pythonpath PYTHONPATH A directory to add to the Python path, e.g. "/home/djangoprojects/myproject". --traceback Raise on CommandError exceptions --no-color Don't colorize the command output. --force-color Force colorization of the command output. --skip-checks Skip system checks.
dbrestore¶
- Restore a database. ::
- $ ./manage.py dbrestore Restoring backup for database: /tmp/tmp.x0kN9sYSqk Finding latest backup Restoring: tmp-zuluvm-2016-07-29-100954.dump Restore tempfile created: 3.3 KiB
Help¶
usage: manage.py dbrestore [-h] [--noinput] [-q] [-d DATABASE] [-i INPUT_FILENAME] [-I INPUT_PATH] [-s SERVERNAME] [-c] [-p PASSPHRASE] [-z] [--version] [-v {0,1,2,3}] [--settings SETTINGS] [--pythonpath PYTHONPATH] [--traceback] [--no-color] [--force-color] [--skip-checks] Restore a database backup from storage, encrypted and/or compressed. optional arguments: -h, --help show this help message and exit --noinput Tells Django to NOT prompt the user for input of any kind. -q, --quiet Tells Django to NOT output other text than errors. -d DATABASE, --database DATABASE Database to restore -i INPUT_FILENAME, --input-filename INPUT_FILENAME Specify filename to backup from -I INPUT_PATH, --input-path INPUT_PATH Specify path on local filesystem to backup from -s SERVERNAME, --servername SERVERNAME If backup file is not specified, filter the existing ones with the given servername -c, --decrypt Decrypt data before restoring -p PASSPHRASE, --passphrase PASSPHRASE Passphrase for decrypt file -z, --uncompress Uncompress gzip data before restoring --version show program's version number and exit -v {0,1,2,3}, --verbosity {0,1,2,3} Verbosity level; 0=minimal output, 1=normal output, 2=verbose output, 3=very verbose output --settings SETTINGS The Python path to a settings module, e.g. "myproject.settings.main". If this isn't provided, the DJANGO_SETTINGS_MODULE environment variable will be used. --pythonpath PYTHONPATH A directory to add to the Python path, e.g. "/home/djangoprojects/myproject". --traceback Raise on CommandError exceptions --no-color Don't colorize the command output. --force-color Force colorization of the command output. --skip-checks Skip system checks.
mediabackup¶
Backup media files, gather all in a tarball and encrypt or compress.
$ ./manage.py mediabackup
Backup size: 10.0 KiB
Writing file to zuluvm-2016-07-04-081612.tar
Help¶
usage: manage.py mediabackup [-h] [--noinput] [-q] [-c] [-s SERVERNAME] [-z] [-e] [-o OUTPUT_FILENAME] [-O OUTPUT_PATH] [--version] [-v {0,1,2,3}] [--settings SETTINGS] [--pythonpath PYTHONPATH] [--traceback] [--no-color] [--force-color] [--skip-checks] Backup media files, gather all in a tarball and encrypt or compress. optional arguments: -h, --help show this help message and exit --noinput Tells Django to NOT prompt the user for input of any kind. -q, --quiet Tells Django to NOT output other text than errors. -c, --clean Clean up old backup files -s SERVERNAME, --servername SERVERNAME Specify server name to include in backup filename -z, --compress Compress the archive -e, --encrypt Encrypt the backup files -o OUTPUT_FILENAME, --output-filename OUTPUT_FILENAME Specify filename on storage -O OUTPUT_PATH, --output-path OUTPUT_PATH Specify where to store on local filesystem --version show program's version number and exit -v {0,1,2,3}, --verbosity {0,1,2,3} Verbosity level; 0=minimal output, 1=normal output, 2=verbose output, 3=very verbose output --settings SETTINGS The Python path to a settings module, e.g. "myproject.settings.main". If this isn't provided, the DJANGO_SETTINGS_MODULE environment variable will be used. --pythonpath PYTHONPATH A directory to add to the Python path, e.g. "/home/djangoprojects/myproject". --traceback Raise on CommandError exceptions --no-color Don't colorize the command output. --force-color Force colorization of the command output. --skip-checks Skip system checks.
mediarestore¶
Restore media files, extract files from archive and put into media storage.
$ ./manage.py mediarestore
Restoring backup for media files
Finding latest backup
Reading file zuluvm-2016-07-04-082551.tar
Restoring: zuluvm-2016-07-04-082551.tar
Backup size: 10.0 KiB
Are you sure you want to continue? [Y/n]
2 file(s) restored
Help¶
usage: manage.py mediarestore [-h] [--noinput] [-q] [-i INPUT_FILENAME] [-I INPUT_PATH] [-s SERVERNAME] [-e] [-p PASSPHRASE] [-z] [-r] [--version] [-v {0,1,2,3}] [--settings SETTINGS] [--pythonpath PYTHONPATH] [--traceback] [--no-color] [--force-color] [--skip-checks] Restore a media backup from storage, encrypted and/or compressed. optional arguments: -h, --help show this help message and exit --noinput Tells Django to NOT prompt the user for input of any kind. -q, --quiet Tells Django to NOT output other text than errors. -i INPUT_FILENAME, --input-filename INPUT_FILENAME Specify filename to backup from -I INPUT_PATH, --input-path INPUT_PATH Specify path on local filesystem to backup from -s SERVERNAME, --servername SERVERNAME If backup file is not specified, filter the existing ones with the given servername -e, --decrypt Decrypt data before restoring -p PASSPHRASE, --passphrase PASSPHRASE Passphrase for decrypt file -z, --uncompress Uncompress gzip data before restoring -r, --replace Replace existing files --version show program's version number and exit -v {0,1,2,3}, --verbosity {0,1,2,3} Verbosity level; 0=minimal output, 1=normal output, 2=verbose output, 3=very verbose output --settings SETTINGS The Python path to a settings module, e.g. "myproject.settings.main". If this isn't provided, the DJANGO_SETTINGS_MODULE environment variable will be used. --pythonpath PYTHONPATH A directory to add to the Python path, e.g. "/home/djangoprojects/myproject". --traceback Raise on CommandError exceptions --no-color Don't colorize the command output. --force-color Force colorization of the command output. --skip-checks Skip system checks.
listbackups¶
This command helps to list backups filtered by type ('media'
or 'db'
),
by compression or encryption.
Help¶
usage: manage.py listbackups [-h] [--noinput] [-q] [-d DATABASE] [-z] [-Z] [-e] [-E] [-c CONTENT_TYPE] [--version] [-v {0,1,2,3}] [--settings SETTINGS] [--pythonpath PYTHONPATH] [--traceback] [--no-color] [--force-color] [--skip-checks] optional arguments: -h, --help show this help message and exit --noinput Tells Django to NOT prompt the user for input of any kind. -q, --quiet Tells Django to NOT output other text than errors. -d DATABASE, --database DATABASE Filter by database name -z, --compressed Exclude non-compressed -Z, --not-compressed Exclude compressed -e, --encrypted Exclude non-encrypted -E, --not-encrypted Exclude encrypted -c CONTENT_TYPE, --content-type CONTENT_TYPE Filter by content type 'db' or 'media' --version show program's version number and exit -v {0,1,2,3}, --verbosity {0,1,2,3} Verbosity level; 0=minimal output, 1=normal output, 2=verbose output, 3=very verbose output --settings SETTINGS The Python path to a settings module, e.g. "myproject.settings.main". If this isn't provided, the DJANGO_SETTINGS_MODULE environment variable will be used. --pythonpath PYTHONPATH A directory to add to the Python path, e.g. "/home/djangoprojects/myproject". --traceback Raise on CommandError exceptions --no-color Don't colorize the command output. --force-color Force colorization of the command output. --skip-checks Skip system checks.
Integration tutorials¶
Note
If you have a custom and/or interesting way of use DBBackup, do not hesitate to make a pull request.
Django-cron¶
Example of cron job with django-cron with file system storage:
import os
from django.core import management
from django.conf import settings
from django_cron import CronJobBase, Schedule
class Backup(CronJobBase):
RUN_AT_TIMES = ['6:00', '18:00']
schedule = Schedule(run_at_times=RUN_AT_TIMES)
code = 'my_app.Backup'
def do(self):
management.call_command('dbbackup')
Upgrade from 2.5.x¶
Settings¶
The following settings are now useless, you can remove them:
DBBACKUP_BACKUP_ENVIRONMENT
: Must be set inCONNECTORS['dbname']['env']
DBBACKUP_RESTORE_ENVIRONMENT
: Same asBACKUP_ENVIRONMENT
DBBACKUP_FORCE_ENGINE
DBBACKUP_READ_FILE
DBBACKUP_WRITE_FILE
DBBACKUP_BACKUP_DIRECTORY
: Was used by Filesystem storage, uselocation
parameterDBBACKUP_SQLITE_BACKUP_COMMANDS
: Was used by SQLite database, useCONNECTORS
’s parameters.DBBACKUP_SQLITE_RESTORE_COMMANDS
: Same asSQLITE_BACKUP_COMMANDS
DBBACKUP_MYSQL_BACKUP_COMMANDS
: Same asSQLITE_BACKUP_COMMANDS
but for MySQLDBBACKUP_MYSQL_RESTORE_COMMANDS
: Same asMYSQL_BACKUP_COMMANDS
DBBACKUP_POSTGRESQL_BACKUP_COMMANDS
Same asMYSQL_BACKUP_COMMANDS
but for PostgreSQLDBBACKUP_POSTGRESQL_RESTORE_COMMANDS
: Same asDBBACKUP_POSTGRESQL_BACKUP_COMMANDS
: Was used to activate PostGIS, usePgDumpGisConnector
connector to enable this optionDBBACKUP_POSTGRESQL_RESTORE_SINGLE_TRANSACTION
: Must be set inCONNTECTORS['dbname']['single_transaction']
DBBACKUP_BUILTIN_STORAGE
Commands¶
dbrestore¶
--backup-extension
has been removed, DBBackup should automatically
know the appropriate file.
Listing from this command, --list
, has been removed in favor of
listbackups
command.
mediabackup¶
mediabackup
’s --no-compress
option has been replaced by --compress
to maintain consistency with other commands.
This command can now backup remote storage, not only the local filesystem
DBBACKUP_BACKUP_DIRECTORY
.
mediarestore¶
You are now able to restore your media files backups. Unfortunately you’ll not be able to restore backup files from previous versions of dbbackup.
Database connector¶
We made a total refactoring of DBCommands system. It is now easier to use, configure and implement a custom connector.
All database configuration for backups are defined in settings
DBBACKUP_CONNECTORS
. By default, the DATABASES
parameters are used but can be overridden in this new constant.
- This dictionary stores configuration about how backups are made,
- the path of the backup command (
/bin/mysqldump
), add a suffix or a prefix
to the command line, environment variable, etc.
The system has been kept pretty simple and can detect alone how to backup your DB. If it can’t just submit to us your Django DB Engine and we’ll try to fix it.
SQLite¶
Previously a backup was made by copying the database file. Now you have the choice between making a raw snapshot or making a real SQL dump. It can be useful to exclude tables or to not overwrite data.
If you want to restore your old backups choose
dbbackup.db.sqlite.SqliteCPConnector
.
Storage engine¶
All storage engines has been removed from DBBackup except the basic filesystem storage. The project now uses Django storages as an intermediary driver.
settings.DBBACKUP_STORAGE
must now be a full path to a Django storage, for
example 'django.core.files.storage.FileSystemStorage'
.
settings.DBBACKUP_STORAGE_OPTIONS
maintains its function of containing the options of the storage.
If you were using a storage backend that has been removed, don’t worry, we will ensure you have a solution by testing and writing an equivalent using Django-Storages.
Contributing guide¶
Dbbackup is a free license software where all help are welcomed. This documentation aims to help users or developers to bring their contributions to this project.
Submit a bug, issue or enhancement¶
All communication are made with GitHub issues. Do not hesitate to open a issue if:
- You have an improvement idea
- You found a bug
- You’ve got a question
- More generally something seems wrong for you
Make a patch¶
We use GitHub pull requests to manage all patches. For a better handling of requests we advise you to:
- Fork the project and make a new branch
- Make your changes with tests if possible and documentation if needed
- Push changes to your fork repository and test it with Travis
- If it succeeds, open a pull request
- Bother us until we give you an answer
Note
We advise you to launch it with Python 2 & 3 before push and try it in Travis. DBBackup uses a lot of file operations, so breaks between Python versions are easy.
Test environment¶
We provides tools to help developers to quickly test and develop DBBackup. There are 2 majors scripts:
runtests.py
: Unit tests launcher and equivalent ofmanage.py
in the test project.functional.sh
: Shell script that useruntests.py
to create a database backup and restore it, the same with media, and test if they are restored.
runtests.py
¶
You can test code on your local machine with the runtests.py
script:
python runtests.py
But if argument are provided, it acts as manage.py
so you can simply
launch some other command to test deeply, example:
# Enter in Python shell
python runtests.py shell
# Launch a particular test module
python runtests.py test dbbackup.tests.test_utils
All tests are stored in dbbackup.tests
.
functional.sh
¶
It tests at a higher level if backup/restore mechanism is alright. It becomes powerful because of the configuration you can give to it. See the next chapter for explanation about it.
Configuration¶
DBBackup contains a test Django project at dbbackup.tests
and its
settings
module. This configuration takes care of the following
environment variables:
DB_ENGINE - Default: django.db.backends.sqlite3
Databank-Engine to use. See in django.db.backends for default backends.
DB_NAME - Default: :memory:
Database name. Should be set correctly if a db other than sqlite3 is used.
DB_USER - Default: None
DB Username
DB_PASSWORD - Default: None
DB Password
DB_HOST - Default: None
DB Host
MEDIA_ROOT - Default= tempfile.mkdtemp()
Django’s MEDIA_ROOT
, useful if you want test media backup from filesystem
STORAGE - Default: dbbackup.tests.utils.FakeStorage
Storage used for backups
STORAGE_OPTIONS
Options for instantiating the chosen storage. It must be formatted as
"key1=foo,key2=bar"
and will be converted into a dict
.
Online CI¶
We use Travis which tests Dbbackup with a matrix of components’ versions: Several versions of Django and several versions of Python including 2, 3 and PyPy.
Code coverage is ensured with Coveralls and the project has not yet reached a minimum coverage limit.
Code health is checked with Landscape
Changelog¶
4.0.0b0 (2021-12-19)¶
- Fix RemovedInDjango41Warning related to default_app_config #413
- Add authentication database support for MongoDB #379
- Remove six dependency #371
- Explicitly support Python 3.6+. #408
- Drop support for end of life Django versions. Currently support 2.2, 3.2, 4.0. #408
- Replace ugettext_lazy with gettext_lazy #342
- Changed logging settings from settings.py to late init #332
- Fix authentication error when postgres is password protected #361
- Use exclude-table-data instead of exclude-table #363
- Add support for exclude tables data in the command interface #375
- Move author and version information into setup.py to allow building package in isolated
environment (e.g. with the
build
package). #414 - Documentation fixes #341 #333 #349 #348 #337 #411
3.3.0 (2020-04-14)¶
- Documentation fixes #341 #333 #328 #320 #305 #303 #302 #298 #281 #266 #349 #348 #337
- “output-filename” in mediabackup command #324
- Fixes for test infrastructure and mongodb support #318
- sqlite3: don’t throw warnings if table already exists #317
- Fixes for django3 and updated travis (and File handling) #316
- Restoring from FTP #313
- Fixes to run dbbackup management command in Postgres for non-latin Windows. #273
- Apply changes from pull request 244; Update to include sftp storage #280
- Quick fix for proper selection of DB name to restore #260
Compatibility¶
As we want to ensure a lot of platforms will be able to save data before upgrading, Django-DBBackup supports PyPy, 3.2 to 3.5 and Django greater than 2.2