ANVIL Documentation¶
Everything about ANVIL, a set of python scripts and utilities to forge raw openstack into a productive tool!
Summary¶
Anvil is a forging tool to help build OpenStack components and their dependencies into a complete package-oriented system.
It automates the git checkouts of the OpenStack components, analyzes & builds their dependencies and the components themselves into packages.
It allows a developer to setup an environment using the automatically created
packages (and dependencies, ex. RPMs
) with the help of anvil configuring
the components to work correctly for the developer’s needs.
The distinguishing part from devstack (besides being written in Python and not
shell), is that after building those packages (currently RPMs
) the same
packages can be used later (or at the same time) to actually deploy at a
larger scale using tools such as chef, salt, or puppet (to name a few).
Features¶
Configurations¶
A set of configuration files (in yaml format) that is used for common, component, distribution, code origins configuration...
All the yaml configuration files could be found in:
conf/templates/keystone/
conf/components/
conf/distros/
conf/origins/
- subdirectories of
conf/personas/
Packaging¶
- Automatically downloading source from git and performing tag/branch checkouts.
- Automatically verifying and translating requirement files to known pypi/rpm packages.
- Automatically installing and building missing dependencies (pypi and rpm) for you.
- Automatically configuring the needed files, symlinks, adjustments, and any patches.
Code decoupling¶
- Components & actions are isolated as individual classes.
- Supports installation personas that define what is to be installed, thus decoupling the ‘what’ from the ‘how’.
Note
This encouraging re-use by others...
Extensive logging¶
- All commands executed are logged in standard output, all configuration files read/written (and so on).
Note
Debug mode can be activated with -v
option...
Package tracking and building¶
- Creation of a single
RPM
set for your installation.- This freezes what is needed for that release to a known set of packages and dependencies.
- Automatically building and/or including all needed dependencies.
- Includes your distributions existing native/pre-built packages (when and where applicable).
How anvil is architected¶
This little HOWTO
can be used by those who wish to understand how anvil
does things and why some of its architectural decisions were made.
History¶
Once upon a time there was a idea of replacing the then existing devstack with a more robust, more error-tolerant and more user/developer friendly OpenStack setup toolkit. Since the existing devstack did (and still does not support very well) complex intercomponent (and interpackage management system) dependencies and installing/packaging/starting/stopping/uninstalling of OpenStack components.
To solve this problem it was thought that there could be a toolkit that could handle this better. It would also be in Python the language of choice for the rest of the OpenStack components thus making it easier to understand for programmers who are already working in OpenStack code. Thus devstack2 was born which was later renamed devstack.py and after a little while once again got renamed to anvil.
Structure¶
Anvil is designed to have the following set of software components:
- Actions: an action is a sequence of function calls on a set of
implementing classes which follows a logically flow from one step to the
next. At the end of each step an action may choose to negate a step of
another action.
- Preparing
- Downloading source code
- Post-download patching of the source code
- Deep dependency & requirement analysis
- Downloading and packaging of missing python dependencies
- Packaging downloaded source code into SRPMs (aka source
RPMs
) that is placed into a SRPM repository.
- Building
- Creation of a binary RPM repository with all built packages and dependencies (converting the prepared source RPMs into binary RPMs).
- Preparing
- Phases: a phase is a step of an action which can be tracked as an
individual unit and can be marked as being completed. In the above install
action for each component that installed when each step occurs for that
component it will be recorded into a file so that if
ctrl-c
aborts anvil and later the install is restarted anvil can notice that the previous phases have already been completed and those phases can be skipped.- This is how anvil does action and step resuming.
- Components: a component is a class which implements the above
steps (which are literally methods on an instance) and is registered with
the persona and configuration to be activated. To aid in making it easier
to add in new components a set of generic base classes exist that provide
common functionality that should work in most simplistic installs. These can
be found in
anvil/components/
. All current components that exist either use these base classes directly or inherit from them and override functions to provide additional capabilities needed to perform the specified action. - Distributions: a distribution is a yaml file that is tied to a operating
system distribution and provides references for components to use in a
generic manner. Some of these references include how to map a
components
pip-requires
file dependencies to distribution specific dependencies (possibly usingyum
orapt
) or what non-specified dependencies are useful in getting the component up and running (such asguestfish
for image mounting and manipulation). Other helpful references include allowing for components to specify standard identifiers for configuration such aspip
. This allows the underlying yaml file to map thepip
command to a distribution-centric command (in RHEL it it’s really namedpip-python
), see the commands key in the yaml files for examples of these settings. Note that each distribution yaml file that exists inconf/distros
provides this set of references for each component and gets selected based on the yaml key in that file named platform_pattern. - Configuration: central to how anvil operates is the ability to be largely
configuration driven (code when you need it but avoid it if you can).
Distributions as seen by the
conf/distros
folder specify distribution-specific configuration that can be referenced by standard keys by a given component. Each component also receives additional configuration (accessible via a componentsget_option
function) via the yaml files specified inconf/components
which provides for a way to have configuration that is not distribution specific but instead is component specific (say for configuring nova to use kvm instead of qemu).- This configuration drive approach (as much as can be possible) was a key design goal that drives how anvil was and is developed. It has even seemed to be ahead of its time due to how anvil has a distribution yaml file that has specified component dependencies long before the OpenStack community even recognized such a dependency list was useful.
- Personas: a persona is a way for anvil to know what components (and
possibly subsystems of those components) you wish to have the given action
applied to. Since not everyone can agree on what is an install of OpenStack
this concept allows for those who wish to have a different set to do so. It
is as all other configuration another yaml file and can be examined by
looking into the
conf/personas
folders.- Each yaml file contains the list of components to be performed for the given action, a simple set of options for those components (for options that may not be applicable to be in the component configuration yaml) and which subsystems a given component will have enabled (if the component supports this concept) as well as which distribution the persona supports (if there is a desire to restrict a given persona to a given distribution this field can be used to accomplish that goal).
Getting started¶
Made to be as simple as possible, but not too simple...
Prerequisites¶
RTFM¶
Read the great documentation for developers/admins at
This will vastly help you understand what the configurations and options do when ANVIL configures them.
Linux¶
One of the tested distributions.
- RHEL 6.2+
- CentOS 6.2+
- Oracle Enteprise Linux 6.2+
You can get CentOS 6.2+ (64-bit is preferred) from https://www.centos.org/
Installation¶
Pre-setup¶
Since RHEL requires a tty to perform sudo
commands we need
to disable this so sudo
can run without a tty. This seems needed
since nova and other components attempt to do sudo
commands. This
isn’t possible in RHEL unless you disable this (since those
instances won’t have a tty).
$ sudo visudo
Then comment out line
Default requiretty
Also disable selinux:
$ sudo vi /etc/sysconfig/selinux
Change SELINUX=enforcing to SELINUX=disabled then reboot.
$ sudo reboot
Create specifc user to isolate all the Anvil processes from root user
$ sudo useradd <username>
$ sudo passwd <username>
Set user as sudoer
$ sudo visudo
Add <username> ALL=(ALL) ALL
Make all the rest of actions as <username> user
$ sudo su - <username>
Get git!¶
$ sudo yum install git -y
Download¶
We’ll grab the latest version of ANVIL via git:
$ git clone https://git.openstack.org/openstack/anvil.git
$ cd anvil
Configuration¶
Any configuration to be updated should now be done.
Please edit the corresponding yaml files in conf/components/
or
conf/components/personas
to fit your desired configuration of nova/glance
and the other OpenStack components.
Note
You can use -p <conf/components/required_file.yaml>
to specify a
different persona.
To specify which versions of OpenStack components you want to install select
or edit an origins configuration file from <conf/origins/>
.
Note
You can use -o <conf/origins/origins_file.yaml>
to specify this
different origins file.
Respository notes for those with RedHat subscriptions¶
To enable the needed repositories for various requirements please also run:
sudo subscription-manager repos --enable rhel-6-server-optional-rpms
You can also include the RDO repositories (which has even more of the needed requirements). This will ensure that anvil has to build less dependencies overall.
Pre-installing¶
In order to ensure that anvil will have its correct dependencies you need to first run the bootstrapping code that will setup said dependencies for your operating system.
sudo ./smithy --bootstrap
Preparing¶
Now prepare OpenStacks components by running the following:
./smithy -a prepare
You should see a corresponding OpenStack repositories getting downloaded using git, python setups occurring and configuration files being written as well as source rpm packages being built and a repository setup from those components [1].
Building¶
Now build OpenStacks components by running the following:
sudo ./smithy -a build
You should see a corresponding OpenStack components and dependencies at this stage being packaged into rpm files and two repositories being setup for you [1]. One repository will be the dependencies that the OpenStack components need to run and th other will be the OpenStack components themselves.
Issues¶
Please report issues/bugs to https://launchpad.net/anvil. Much appreciated!
[1] | (1, 2) If you desire more informational output add a -v or a -vv to the command. |
Documentation¶
For great documentation on all things OpenStack check out the following relevant links and webpages.
For admins/users¶
For developers¶
Adding your own distribution¶
This little HOWTO
can be used by those who wish to
add-on to anvil to be able to support their own distribution
or unsupported operating system (so that it can be
supported).
Diving in!¶
First you have to have a little background on anvil and how it operates. So let’s dive in and learn a little on how we can add in your own distribution support.
- smithy
The main shell script that bootstraps the needed dependencies for anvil to be able to start (including items such as termcolor, progressbar and netifaces). The code here is written in bash shell script so that it can execute in an environment without the needed prerequisites.
When to adjust: Adjust the bootstrapping functions in this file to install any needed prerequisites for your operating system to run anvil. Look at how we are bootstrapping rhel (and how we are detecting rhel) for an example.- conf/distros
This set of yaml files contains definitions for what packages, what pip to package mappings and what code entrypoints are used when setting up a given component. The critical key here is the
platform_pattern
key which is used as a regular expression to determine if the provided yaml file will work in the given running distribution. Other keys are used to identify which packaging class to use (iepackager_name
) and how to map a component name to its action classes (i.e.action_classes/install
will be constructed when an install action occurs). Thecommands
section can be used to house arbitrary commands which may vary between operating systems (such as thepip
executable name)When to adjust the distro: If a suitable distribution already exists (which may be the case for many rhel variants), just go ahead and add-on to the regular expression your pattern. Ensure that your regular expression matches the output of the following command:
python -c "import platform; print(platform.platform())"
which is what anvil uses internally to match a given yaml file to a given distribution.When to add a new file: If no suitable distribution exists (which may be the case for ubuntu), you will need to go ahead and create a new file for that distribution and include its dependencies and any variations in packaging and pip -> package mappings needed to setup that distribution with the openstack component software.
- anvil/distros
These are typically subclasses of components that may override generic functionality to correct for a given distribution doing or requiring something different to occur before/after or during an install or other action.
When to adjust: Feel free to add-on your own subclasses here as needed to handle any special actions that your new distribution may require and make sure you reference those classes/entrypoints in your conf/distros yaml file so that the correct subclass will be used. The rhel distro has a good set of examples that overload various key points so that rhel can work correctly.
- anvil/packaging
The modules in this folder will be referenced in your conf/distros yaml file and will control how to install packages (i.e. using yum and pip) and how to uninstall those same packages. This code will also get activated when a ‘package’ action occurs which currently will cause the necessary actions to occur to create a RPM
spec
file which can be used with therpmbuild
command.When to adjust: If needed it should be simple to look at the packaging interface and add your own. After adding make sure you reference them in your conf/distros yaml file so that the correct subclass will be used. If you are going to want to create package files from the installed code then you will need to hook-in to a file similar to the RPM module that exists there.
Questions and Answers¶
How do I cause the anvil dependencies to be reinstalled?¶
Anvil bootstraps itself via shell script (if you look at the code
in the file smithy
you will see that it is actually a bash
script).
This bootstrapping occurs to ensure that anvils pypi/rpm/deb
dependencies are installed before anvil can actually be used.
To remove the files that are left behind to let the shell script
know when this happens delete files located at $HOME/.anvil_bootstrapped
and at $PWD/.anvil_bootstrapped
to cause bootstrapping to occur again.
Another way to make this happen temporarily is to use the following:
sudo BOOT_FILES=/dev/null ./smithy
This will make anvil think those files are coming from /dev/null
which will always return nothing. Using the same variable
also allows you to retarget the locations where the smithy
shell script will look for the ‘marker’ files if
you so choose (say in a continuous integration environment).
How do I run a specific OpenStack milestone?¶
Anvil has the same tag names as OpenStack releases so to run against a specific milestone of OpenStack just checkout the same tag in anvil and run the same actions as you would have ran previously.
An example of this, lets adjust nova
to use the stable/essex
branch.
- Open
conf/origins/master.yaml
file in your favorite editor - Locate lines that describe the
nova
component - Change branch parameter to the desired one
nova:
repo: https://github.com/openstack/nova.git
branch: stable/essex
- Component origin parameters are:
repo: <repo_url>
- requiredbranch: <branch>
- optionaltag: <tag>
- optional
If no branch nor tag parameters were specified then
branch: master
is used by default.Note: tag overrides branch (so you can’t really include both)
Bugs & Hugs & Code¶
ANVIL is an open-source tool released under the apache version 2.0 license. It depends on its community to keep it alive.
Source code¶
The source code is on github located at:
http://git.openstack.org/cgit/openstack/anvil (mirrored @ http://github.com/openstack/anvil/).
Feel free to fork it and contribute to it.
Branches¶
Anvil tries to work across different OpenStack releases as of the havana release...
If it does not work across the majority of OpenStack releases please file a bug.
Hacking¶
Feel free to hack but please try to follow the hacking guidelines.
Links¶
Please visit as often as you want at the following urls:
- http://launchpad.net/anvil (blueprints for features, bugs, q/a...)
- http://launchpad.net/~anvil-dev (talk to the devs directly)
Help and developer work/time is always much appreciated!
Examples¶
Bootstrapping¶
This is needed to get ready for the rest of anvils stages by installing anvils python dependencies so that anvil can correctly run using said dependencies.
$ sudo ./smithy --bootstrap
Terminal recording: http://showterm.io/effa75ea631777a2e74a0/
Preparing¶
This stage does the download of the source repositories, analysis of dependencies, download of missing dependencies and building of source repositories and missing dependencies into source rpms.
$ ./smithy -a prepare
Terminal recording: http://showterm.io/12c29e87094f128d945fa/
Building¶
This is the stage responsible for translating the previously prepared source rpms into installable rpms (of the non-source type). The output of this phase is two repositories, one with the dependencies and one with the rpms for the openstack components themselves.
$ sudo ./smithy -a build
Terminal recording: http://showterm.io/2fee38794dcf536ccd437/
Packaging¶
To see the packages built (after prepare has finished).
$ ls /home/harlowja/openstack/deps/rpmbuild/SPECS/ | cat
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 | openstack-deps.spec
pylint.spec
pyparsing.spec
python-babel.spec
python-cheetah.spec
python-cinderclient.spec
python-cinder.spec
python-cliff.spec
python-cliff-tablib.spec
python-cmd2.spec
python-colorama.spec
python-coverage.spec
python-crypto.spec
python-decorator.spec
python-discover.spec
python-docutils.spec
python-extras.spec
python-fixtures.spec
python-glanceclient.spec
python-glance.spec
python-hp3parclient.spec
python-httplib2.spec
python-jinja2.spec
python-jsonpatch.spec
python-jsonpointer.spec
python-jsonschema.spec
python-keystoneclient.spec
python-keystone.spec
python-ldap.spec
python-logilab-astng.spec
python-logilab-common.spec
python-lxml.spec
python-markdown.spec
python-markupsafe.spec
python-mimeparse.spec
python-netaddr.spec
python-nose-exclude.spec
python-nosehtmloutput.spec
python-nose.spec
python-nosexcover.spec
python-novaclient.spec
python-nova.spec
python-openstack-nose-plugin.spec
python-oslo-config.spec
python-pam.spec
python-pastedeploy.spec
python-pep8.spec
python-prettytable.spec
python-pygments.spec
python-pysqlite.spec
python-neutronclient.spec
python-repoze-lru.spec
python-routes.spec
python-setuptools-git.spec
python-setuptools.spec
python-sphinx.spec
python-sqlalchemy-migrate.spec
python-sqlalchemy.spec
python-subunit.spec
python-tablib.spec
python-tempita.spec
python-termcolor.spec
python-testrepository.spec
python-testtools.spec
python-unittest2.spec
python-warlock.spec
python-webob.spec
python-wsgiref.spec
python-xattr.spec
|
$ cat openstack-deps.spec
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 | Name: openstack-deps
Version: 2013.6.3
Release: 0
License: Apache 2.0
Summary: OpenStack dependencies
BuildArch: noarch
Requires: MySQL-python
Requires: avahi
Requires: coreutils
Requires: curl
Requires: dnsmasq
Requires: dnsmasq-utils
Requires: ebtables
Requires: fuse
Requires: gawk
Requires: git
Requires: guestfish
Requires: iptables
Requires: iputils
Requires: iscsi-initiator-utils
Requires: kpartx
Requires: libguestfs
Requires: libguestfs-mount
Requires: libguestfs-tools
Requires: libvirt
Requires: libvirt-client
Requires: libvirt-python
Requires: libxml2-devel
Requires: libxslt-devel
Requires: lsof
Requires: mlocate
Requires: mysql
Requires: mysql-server
Requires: openssh-server
Requires: parted
Requires: postgresql-devel
Requires: psmisc
Requires: python
Requires: python-devel
Requires: python-distutils-extra
Requires: python-setuptools
Requires: qemu-img
Requires: qemu-kvm
Requires: rabbitmq-server
Requires: sqlite
Requires: sqlite-devel
Requires: sudo
Requires: tcpdump
Requires: unzip
Requires: vconfig
Requires: wget
%description
%pre
# rabbitmq-server
service qpidd stop 2>/dev/null || true
chkconfig qpidd off 2>/dev/null || true
%files
|
$ cat python-nova.spec
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 | %define pkg_name nova
%define version 2013.1
%define unmangled_version 2013.1
%define unmangled_version 2013.1
%define release 1
Summary: cloud computing fabric controller
Name: python-nova
Epoch: 2
Version: %{version}
Release: %{release}
Source0: %{pkg_name}-%{unmangled_version}.tar.gz
License: UNKNOWN
Group: Development/Libraries
BuildRoot: %{_tmppath}/%{pkg_name}-%{version}-%{release}-buildroot
Prefix: %{_prefix}
BuildArch: noarch
Vendor: OpenStack <nova@lists.launchpad.net>
Requires: python-sqlalchemy >= 0.7.8 python-sqlalchemy < 0.7.99 python-cheetah >= 2.4.4 python-amqplib >= 0.6.1 python-anyjson >= 0.2.4 python-argparse python-boto python-eventlet >= 0.9.17 python-kombu >= 1.0.4 python-lxml >= 2.3 python-routes >= 1.12.3 python-webob = 1.2.3 python-greenlet >= 0.3.1 python-pastedeploy >= 1.5 python-paste python-sqlalchemy-migrate >= 0.7.2 python-netaddr >= 0.7.6 python-suds >= 0.4 python-paramiko python-pyasn1 python-babel >= 0.9.6 python-iso8601 >= 0.1.4 python-httplib2 python-setuptools-git >= 0.4 python-cinderclient >= 2:1.0.1 python-neutronclient >= 2:2.2 python-neutronclient < 2:3 python-glanceclient >= 2:0.5 python-glanceclient < 2:2 python-keystoneclient >= 2:0.2 python-stevedore >= 0.7 python-websockify < 0.4 python-oslo-config >= 2:1.1
Url: http://www.openstack.org/
%description
UNKNOWN
%prep
%setup -n %{pkg_name}-%{unmangled_version} -n %{pkg_name}-%{unmangled_version}
%build
python setup.py build
%install
python setup.py install --single-version-externally-managed -O1 --root=$RPM_BUILD_ROOT --record=INSTALLED_FILES
abspath_installed_files=$(readlink -f INSTALLED_FILES)
(
cd $RPM_BUILD_ROOT
for i in usr/*/python*/site-packages/* usr/bin/*; do
if [ -e "$i" ]; then
sed -i "s@/$i/@DELETE_ME@" "$abspath_installed_files"
echo "/$i"
fi
done
if [ -d usr/man ]; then
rm -rf usr/share/man
mkdir -p usr/share
mv usr/man usr/share/
sed -i "s@/usr/man/@DELETE_ME@" "$abspath_installed_files"
for i in usr/share/man/*; do
echo "/$i/*"
done
fi
) >> GATHERED_FILES
{ sed '/^DELETE_ME/d' INSTALLED_FILES; cat GATHERED_FILES; } | sort -u > INSTALLED_FILES.tmp
mv -f INSTALLED_FILES{.tmp,}
%clean
rm -rf $RPM_BUILD_ROOT
%files -f INSTALLED_FILES
%defattr(-,root,root)
|