AutoWIG: Automatic Wrapper and Interface Generator¶
High-level programming languages, such as Python and R, are popular among scientists. They are concise, readable, lead to rapid development cycles, but suffer from performance drawback compared to compiled language. However, these languages allow to interface C, C++ and Fortran code. In this way, most of the scientific packages incorporate compiled scientific libraries to both speed up the code and reuse legacy libraries. While several semi-automatic solutions and tools exist to wrap these compiled libraries, the process of wrapping a large library is cumbersome and time consuming. AutoWIG is a Python library that wraps automatically compiled libraries into high-level languages. Our approach consists in parsing C++ code using the LLVM/Clang technologies and generating the wrappers using the Mako templating engine. Our approach is automatic, extensible, and applies to very complex C++ libraries, composed of thousands of classes or incorporating modern meta-programming constructs.
Installation¶
Test it with Docker¶
Note
Docker [Mer14] is an open-source project that automates the deployment of Linux applications inside software containers.
We provide Docker images to enable to run AutoWIG on various platforms (in particular Windows and MacOS).
For the installation of Docker, please refers to its documentation.
Then, you can use the statiskit/autowig
Docker image to run AutoWIG:
$ docker run -i -t -p 8888:8888 statiskit/autowig
A list of all available images can be found here.
The image tagged latest
is unstable, it could be preferable to use the one attached with the AutoWIG paper submitted in Journal of Computational Science (tagged v1.0.0
) as follows:
$ docker run -i -t -p 8888:8888 statiskit/autowig:v1.0.0
For convenience, examples are presented in Jupyter notebooks. You can therefore proceed – in the container’s terminal – as follows to run examples:
Launch the Jupyter notebook with the following command
$ jupyter notebook --ip='*' --port=8888 --no-browser
Copy the URL given in the container’s terminal and paste it in your browser. This URL should looks like
http://localhost:8888/?token=/[0-9a-fA-F]+/
.Click on the notebooks you want to run (denoted by
*.ipynb
) and then click on Run All item of the Cell top menu bar.
Warning
For some systems as Ubuntu, Docker requires root permissions (see this page for more information).
Installation from binaries¶
In order to ease the installation of the AutoWIG software on multiple operating systems, the Conda package and environment management system is used. To install Conda, please refers to its documentation or follow the installation instructions given on the StatisKit documentation. Once Conda installed, you can install AutoWIG binaries into a special environment that will be used for wrapper generation by typing the following command line in your terminal:
$ conda create -n autowig python-autowig python-clanglite python-scons python-dev libdev -c statiskit -c conda-forge
Warning
When compiling wrappers generated by AutoWIG in its environment some issues can be encountered at compile time or run time (from within the Python interpreter) due to compiler or dependency incompatibilies. This is why it is recommended to install AutoWIG in a separate environment that will only be used for the wrappers generation. If the problem persits, please refers to the StatisKit documentation concerning the configuration of the development environment.
Installation from source code¶
For installing AutoWIG from source code, please refers to the StatisKit documentation concerning the configuration of the development environment.
Documentation¶
User guide¶
Note
In this section, we introduce wrapping problems and how AutoWIG aims at minimize developers effort. Basic concepts and conventions are introduced.
Problem setting¶
Consider a scientist who has designed multiple C++ libraries for statistical analysis. He would like to distribute his libraries and decide to make them available in Python in order to reach a public of statisticians but also less expert scientists such as biologists. Yet, he is not interested in becoming an expert in C++/Python wrapping, even if it exists classical approaches consisting in writing wrappers with SWIG [Bea03] or Boost.Python [AGK03]. Moreover, he would have serious difficulties to maintain the wrappers, since this semi-automatic process is time consuming and error prone. Instead, he would like to automate the process of generating wrappers in sync with his evolving C++ libraries. That’s what the AutoWIG software aspires to achieve.
Automating the process¶
Building such a system entails achieving some minimal features:
- C++ parsing
- In order to automatically expose C++ components in Python, the system requires parsing full legacy code implementing the last C++ standard. It has also to represent C++ constructs in Python, like namespaces, enumerators, enumerations, variables, functions, classes or aliases.
- Documentation
- The documentation of C++ components has to be associated automatically to their corresponding Python components in order to reduce the redundancy and to keep it up-to-date in only one place.
- Pythonic interface
- To respect the Python philosophy, C++ language patterns need to be consistently translated into Python.
Some syntax or design patterns in C++ code are specific and need to be adapted in order to obtain a functional Python package.
Note that this is particularly sensible for C++ operators (e.g.
()
,<
,[]
) and corresponding Python special functions (e.g.__call__
,__lt__
,__getitem__
,__setitem__
) or for object serialization. - Memory management
- C++ libraries expose in their interfaces either raw pointers, shared pointers or references, while Python handles memory allocation and garbage collection automatically.
The concepts of pointer or references are thus not meaningful in Python.
These language differences entail several problems in the memory management of C++ components into Python.
A special attention is therefore required for dealing with references (
&
) and pointers (*
) that are highly used in C++. - Error management
- C++ exceptions need to be consistently managed in Python. Python doesn’t have the necessary equipment to properly unwind the C++ stack when exception are thrown. It is therefore important to make sure that exceptions thrown by C++ code do not pass into the Python interpreter core. All C++ exceptions thrown by wrappers must therefore be translated into Python errors. This translation must preserve exception names and contents in order to raise informative Python errors.
- Dependency management between components
- The management of multiple dependencies between C++ libraries with Python bindings is required at run-time from Python.
C++ libraries tends to have dependencies.
For instance the C++ Standard Template Library containers [PLMS00] are used in many C++ libraries (e.g
std::vector
,std::set
). For such cases, it doesn’t seem relevant that every wrapped C++ library contains wrappers for usual STL containers (e.g.std::vector< double >
,std::set< int >
). Moreover, loading in the Python interpreter multiple compiled libraries sharing different wrappers from same C++ components could lead to serious side effects. It is therefore required that dependencies across different library bindings can be handled automatically.
Examples¶
Note
In the following section, we present some examples using AutoWIG in order to emphasize particular aspects of the wrapping process.
This examples can be executed on-line on a Binder server.
They can be found in the doc/examples directory and are recognizable by their *.ipynb
extension.
Warning
Binder does not provide webhooks that could allow to rebuild the Docker image at each changes. In order to able to run these examples, it is possible that you need to rebuild the Docker image. For this, go to the Binder status page and click on on the rebuild button.
Here are the pre-executed examples:
Wrapping a basic library¶
We here aim at presenting the interactive wrapping workflow. For the sake of simplicity, we consider a basic example of C++ library.
First, import AutoWIG.
In [ ]:
import autowig
Then, to install and compile the C++ library we use available Conda recipes.
In [ ]:
!conda remove libbasic -y
!conda build -q basic/conda/libbasic -c statiskit
!conda install -y -q libbasic --use-local -c statiskit
Once the headers have been installed in the system, we parse them with relevant compilation flags.
In [ ]:
%%time
import sys
asg = autowig.AbstractSemanticGraph()
asg = autowig.parser(asg, [sys.prefix + '/include/basic/overload.h',
sys.prefix + '/include/basic/binomial.h'],
['-x', 'c++', '-std=c++11'],
silent = True)
Since most of AutoWIG guidelines are respected, the default
controller
implementation is suitable.
In [ ]:
%%time
autowig.controller.plugin = 'default'
asg = autowig.controller(asg)
In order to wrap the library we need to select the
boost_python_internal
generator
implementation.
In [ ]:
%%time
autowig.generator.plugin = 'boost_python_internal'
wrappers = autowig.generator(asg,
module = 'basic/src/py/_basic.cpp',
decorator = 'basic/src/py/basic/_basic.py',
prefix = 'wrapper_')
The wrappers are only generated in-memory. It is therefore needed to write them on the disk to complete the process.
In [ ]:
%%time
wrappers.write()
Here is an example of the generated wrappers. We here present the
wrappers for the BinomialDistribution
class.
In [ ]:
!pygmentize basic/src/py/wrapper_4046a8421fe9587c9dfbc97778162c7d.cpp
Once the wrappers are written on disk, we need to compile and install the Python bindings.
In [ ]:
!conda build -q basic/conda/python-basic -c statiskit
!conda install -y -q python-basic --use-local -c statiskit --force
Finally, we can hereafter use the C++ library in the Python interpreter.
In [ ]:
import basic
binomial = basic.BinomialDistribution(1, .5)
binomial
In [ ]:
binomial.pmf(0)
In [ ]:
binomial.pmf(1)
In [ ]:
binomial.n = 0
binomial
In [ ]:
binomial.pmf(0)
In [ ]:
try:
binomial.set_pi(1.1)
except basic.ProbabilityError as error:
print error.message
else:
raise Exception('A `basic.ProbabilityError` should have been raise')
Wrapping a subset of a very large library¶
Sometimes, for a very large library, only a subset of available C++
components is useful for end-users. Wrapping such libraries therefore
requires AutoWIG to be able to consider only a subset of the C++
components during the Generate
step. The Clang library is a
complete C/C++ compiler. Clang is a great tool, but its stable
Python interface (i.e. libclang) is lacking some useful features
that are needed by AutoWIG. In particular, class template
specializations are not available in the abstract syntax tree.
Fortunately, most of the classes that would be needed during the
traversal of the C++ abstract syntax tree are not template
specializations. We therefore proposed to bootstrap the Clang
Python bindings using the libclang
parser
of AutoWIG. This
new Clang Python interface is called PyClangLite and is able
to parse class template specializations. As for libclang, this
interface is proposed only for a subset of the Clang library
sufficient enough for proposing the new pyclanglite
parser
.
This repository already has wrappers, we therefore need to remove them.
In [ ]:
!git clone https://github.com/StatisKit/ClangLite ClangLite
!git -C ClangLite checkout a13322e37683012ca346595e88abc48ac591112c
In [ ]:
from path import Path
import shutil
srcdir = Path('ClangLite')/'src'/'py'
for wrapper in srcdir.walkfiles('*.cpp'):
wrapper.unlink()
for wrapper in srcdir.walkfiles('*.h'):
wrapper.unlink()
wrapper = srcdir/'clanglite'/'_clanglite.py'
if wrapper.exists():
wrapper.unlink()
blddir = srcdir.parent.parent/'build'
if blddir.exists():
shutil.rmtree(srcdir.parent.parent/'build')
In addition to the Clang libraries, the ClangLite library is
needed in order to have access to some functionalities. The tool.h
header of this ClangLite library includes all necessary Clang
headers. This library is installed using the SCons cpp
target.
In [ ]:
!conda remove libclanglite -y
!conda build ClangLite/conda/libclanglite -c statiskit -c conda-forge
!conda install -y libclanglite --use-local -c statiskit -c conda-forge
Once these preliminaries done, we can proceed to the actual generation of wrappers for the Clang library. For this, we import AutoWIG and create an empty Abstract Semantic Graph (ASG).
In [ ]:
import autowig
asg = autowig.AbstractSemanticGraph()
We then parse the tool.h
header of the ClangLite library with
relevant compilation flags.
In [ ]:
%%time
import sys
prefix = Path(sys.prefix).abspath()
autowig.parser.plugin = 'libclang'
asg = autowig.parser(asg, [prefix/'include'/'clanglite'/'tool.h'],
flags = ['-x', 'c++', '-std=c++11',
'-D__STDC_LIMIT_MACROS',
'-D__STDC_CONSTANT_MACROS',
'-I' + str((prefix/'include').abspath())],
libpath = prefix/'lib'/'libclang.so',
bootstrap = False,
silent = True)
Since most of AutoWIG guidelines are respected in the Clang
library, the default
controller
implementation could be
suitable. Nevertheless, we need to force some C++ components to be
wrapped or not. We therefore implements a new controller
.
In [ ]:
def clanglite_controller(asg):
for node in asg['::boost::python'].classes(nested = True):
node.is_copyable = True
for node in asg.classes():
node.boost_python_export = False
for node in asg.functions(free=True):
node.boost_python_export = False
for node in asg.variables(free = True):
node.boost_python_export = False
for node in asg.enumerations():
node.boost_python_export = False
for node in asg.enumerators():
if node.parent.boost_python_export:
node.boost_python_export = False
for node in asg.typedefs():
node.boost_python_export = False
from autowig.default_controller import refactoring
asg = refactoring(asg)
if autowig.parser.plugin == 'libclang':
for fct in asg.functions(free=False):
asg._nodes[fct._node]['_is_virtual'] = False
asg._nodes[fct._node]['_is_pure'] = False
asg['class ::clang::QualType'].is_abstract = False
asg['class ::clang::QualType'].is_copyable = True
asg['class ::llvm::StringRef'].is_abstract = False
asg['class ::llvm::StringRef'].is_copyable = True
asg['class ::clang::FileID'].is_abstract = False
asg['class ::clang::FileID'].is_copyable = True
asg['class ::clang::SourceLocation'].is_abstract = False
asg['class ::clang::SourceLocation'].is_copyable = True
asg['class ::clang::TemplateArgument'].is_abstract = False
asg['class ::clang::TemplateArgument'].is_copyable = True
for cls in ['::clang::FriendDecl', '::clang::CapturedDecl', '::clang::OMPThreadPrivateDecl',
'::clang::NonTypeTemplateParmDecl', '::clang::TemplateArgumentList', '::clang::ImportDecl',
'::clang::TemplateTemplateParmDecl', '::clang::CapturedDecl', '::clang::OMPThreadPrivateDecl',
'::clang::NonTypeTemplateParmDecl', '::clang::TemplateArgumentList', '::clang::ImportDecl',
'::clang::TemplateTemplateParmDecl']:
asg['class ' + cls].is_abstract = False
asg['class ::boost::python::api::object'].boost_python_export = True
asg['class ::boost::python::list'].boost_python_export = True
asg['class ::boost::python::str'].boost_python_export = True
subset = []
classes = [asg['class ::clang::QualType'],
asg['class ::clang::Type'],
asg['class ::clang::Decl']]
subset += classes
for cls in classes:
subset += cls.subclasses(recursive=True)
for cls in subset:
if not cls.globalname.strip('class ') in ['::clang::QualType',
'::llvm::StringRef',
'::clang::FileID',
'::clang::SourceLocation',
'::clang::TemplateArgument',
'::clang::FriendDecl',
'::clang::CapturedDecl',
'::clang::OMPThreadPrivateDecl',
'::clang::NonTypeTemplateParmDecl',
'::clang::TemplateArgumentList',
'::clang::ImportDecl',
'::clang::TemplateTemplateParmDecl']:
cls.is_copyable = False
else:
cls.is_copyable = True
subset.append(asg['class ::llvm::StringRef'])
subset.append(asg['class ::clang::ASTUnit'])
subset.append(asg['class ::clang::ASTContext'])
subset.append(asg['class ::clang::SourceManager'])
subset.append(asg['class ::clang::FileID'])
subset.append(asg['class ::clang::SourceLocation'])
subset.append(asg['class ::clang::CXXBaseSpecifier'])
subset.append(asg['class ::clang::DeclContext'])
subset.append(asg['class ::clang::TemplateArgument'])
subset.append(asg['class ::clang::TemplateArgumentList'])
subset.append(asg['enum ::clang::Type::TypeClass'])
subset.append(asg['enum ::clang::AccessSpecifier'])
subset.append(asg['enum ::clang::LinkageSpecDecl::LanguageIDs'])
subset.append(asg['enum ::clang::BuiltinType::Kind'])
subset.append(asg['enum ::clang::TemplateArgument::ArgKind'])
subset.append(asg['enum ::clang::Decl::Kind'])
# subset.extend(asg['::boost::python'].classes(nested = True))
# subset.extend(asg['::boost::python'].enumerations(nested = True))
subset.extend(asg.nodes('::clanglite::build_ast_from_code_with_args'))
for node in subset:
node.boost_python_export = True
for fct in asg['::clanglite'].functions():
if not fct.localname == 'build_ast_from_code_with_args':
fct.parent = fct.parameters[0].qualified_type.desugared_type.unqualified_type
fct.boost_python_export = True
for mtd in asg['class ::clang::ASTContext'].methods(pattern='.*getSourceManager.*'):
if mtd.return_type.globalname == 'class ::clang::SourceManager &':
mtd.boost_python_export = True
break
if autowig.parser.plugin == 'libclang':
for node in (asg.functions(pattern='.*(llvm|clang).*_(begin|end)')
+ asg.functions(pattern='::clang::CXXRecordDecl::getCaptureFields')
+ asg.functions(pattern='.*(llvm|clang).*getNameAsString')
+ asg.nodes('::clang::NamedDecl::getQualifiedNameAsString')
+ asg.functions(pattern='.*::clang::ObjCProtocolDecl')
+ asg.nodes('::clang::ObjCProtocolDecl::collectInheritedProtocolProperties')
+ asg.nodes('::clang::ASTUnit::LoadFromASTFile')
+ asg.nodes('::clang::ASTUnit::getCachedCompletionTypes')
+ asg.nodes('::clang::ASTUnit::getBufferForFile')
+ asg.nodes('::clang::CXXRecordDecl::getCaptureFields')
+ asg.nodes('::clang::ASTContext::SectionInfos')
+ asg.nodes('::clang::ASTContext::getAllocator')
+ asg.nodes('::clang::ASTContext::getObjCEncoding.*')
+ asg.nodes('::clang::ASTContext::getAllocator')
+ asg.nodes('::clang::QualType::getAsString')
+ asg.nodes('::clang::SourceLocation::printToString')
+ asg['class ::llvm::StringRef'].methods()):
node.boost_python_export = False
if autowig.parser.plugin == 'clanglite':
for mtd in asg['class ::clang::Decl'].methods():
if mtd.localname == 'hasAttr':
mtd.boost_python_export = False
import sys
from path import path
for header in (path(sys.prefix)/'include'/'clang').walkfiles('*.h'):
asg[header.abspath()].is_external_dependency = False
return asg
This controller
is then dynamically registered and used on the ASG.
In [ ]:
%%time
autowig.controller['clanglite'] = clanglite_controller
autowig.controller.plugin = 'clanglite'
asg = autowig.controller(asg)
In order to wrap a subset of the Clang library, we need to select
the boost_python_internal
generator
implementation.
In [ ]:
%%time
autowig.generator.plugin = 'boost_python_pattern'
wrappers = autowig.generator(asg,
module = srcdir/'_clanglite.cpp',
decorator = srcdir/'clanglite'/'_clanglite.py',
closure = False)
The wrappers are only generated in-memory. It is therefore needed to write them on the disk to complete the process.
In [ ]:
%%time
wrappers.write()
Here is an example of the generated wrappers. We here present the
wrappers for the clang::Decl
class.
In [ ]:
!pygmentize ClangLite/src/py/wrapper_a6aedb4654a55a40aeecf4b1dc5fcc98.cpp
Once the wrappers are written on the disk, the bingings must be compiled
and installed. This can be done using the SCons py
target.
In [ ]:
!conda build ClangLite/conda/python-clanglite -c statiskit -c conda-forge
!conda install -y python-clanglite --use-local -c statiskit -c conda-forge
In [ ]:
import autowig
from clanglite.autowig_parser import autowig_parser
autowig.parser['clanglite'] = autowig_parser
autowig.parser.plugin = 'clanglite'
from path import Path
import sys
for wrapper in srcdir.walkfiles('*.cpp'):
wrapper.unlink()
for wrapper in srcdir.walkfiles('*.h'):
wrapper.unlink()
wrapper = srcdir/'clanglite'/'_clanglite.py'
if wrapper.exists():
wrapper.unlink()
prefix = Path(sys.prefix).abspath()
asgbis = autowig.AbstractSemanticGraph()
asgbis = autowig.parser(asgbis, [prefix/'include'/'clanglite'/'tool.h'],
flags = ['-x', 'c++', '-std=c++11',
'-D__STDC_CONSTANT_MACROS',
'-D__STDC_FORMAT_MACROS',
'-D__STDC_LIMIT_MACROS',
'-I' + str((prefix/'include').abspath()),
'-I' + str((prefix/'include'/'python2.7').abspath())],
bootstrap = False,
silent = True)
autowig.controller['clanglite'] = clanglite_controller
autowig.controller.plugin = 'clanglite'
asgbis = autowig.controller(asgbis)
autowig.generator.plugin = 'boost_python_pattern'
wrappers = autowig.generator(asgbis,
module = srcdir/'_clanglite.cpp',
decorator = srcdir/'clanglite'/'_clanglite.py',
closure = False)
wrappers.write()
In [ ]:
!conda remove python-clanglite -y
!conda build ClangLite/conda/python-clanglite -c statiskit -c conda-forge
!conda install -y python-clanglite --use-local -c statiskit -c conda-forge
Wrapping a template library¶
A template library is a library where there are only template classes
that can be instantiated. Wrapping such libraries therefore requires
AutoWIG to be able to consider various C++ template classes
instantiations during the Parse
step. It is therefore required to
install the pyclanglite
parser
.
The Standard Template Library (STL) library is a C++ library that
provides a set of common C++ template classes such as containers and
associative arrays. These classes can be used with any built-in or
user-defined type that supports some elementary operations (e.g.
copying, assignment). It is divided in four components called
algorithms, containers, functional and iterators. STL containers
(e.g. std::vector
, std::set
) are used in many C++ libraries.
In such a case, it does not seem relevant that every wrapped C++
library contains wrappers for usual STL containers (e.g.
std::vector< double >
, std::set< int >
). We therefore proposed
Python bindings for sequence containers (i.e. pair
, array
,
vector
, deque
, forward_list
and list
of the std
namespace) and associative containers (set
, multiset
, map
,
multimap
, unordered_set
, unordered_multiset
,
unordered_map
and unordered_multimap
of the std
namespace).
These template instantiations are done for C++ fundamental types
(bool
, signed char
, unsigned char
, char
, wchar_t
,
int
(with sign modifiers signed
and signed
combined or not
with size modifiers short
, long
and long long
), float
,
double
, long double
) and strings (string
, wstring
of the
std
namespace). For ordered associative containers both
std::less
and std::greater
comparators are used. We here only
illustrate the procedure on the std::vector
template class. For the
complete procedure refers to the AutoWIG.py
file situed at the root
of the PySTL repository.
In [ ]:
!git clone https://github.com/StatisKit/STL STL
!git -C STL checkout b9569c67ebc59482dc99a8fa11aa685faebc981d
Then, to install and compile the C++ library we use available Conda recipes.
In [ ]:
!conda build -q STL/conda/libstatiskit_stl -c statiskit
!conda install -y -q libstatiskit_stl --use-local -c statiskit
As presented below, in order to wrap a template library, the user needs to write headers containing aliases for desired template class instantiations.
In [ ]:
!pygmentize STL/src/cpp/STL.h
Once these preliminaries done, we can proceed to the actual generation of wrappers for the PySTL library. For this, we import AutoWIG and create an empty Abstract Semantic Graph (ASG).
We need then to install the C++ headers. This is done using the
cpp
target in SCons.
In [ ]:
!scons cpp -C STL
Once the headers habe been installed in the system, we parse headers with relevant compilation flags.
In [ ]:
!scons autowig -c -C STL
!scons autowig -C STL
Here is an example of the generated wrappers. We here present the
wrappers for the std::vector< int >
class.
In [ ]:
!pygmentize STL/src/py/wrapper/wrapper_6b9ae5eac40858c9a0f5e6e21c15d1d3.cpp
Once the wrappers are written on disk, we need to compile and install the Python bindings.
In [ ]:
!conda build STL/conda/python-statiskit_stl -c statiskit
!conda install -y python-statiskit_stl --use-local -c statiskit --force
Finally, we can hereafter use the C++ library in the Python interpreter.
In [ ]:
from statiskit.stl import VectorInt
v = VectorInt()
v.push_back(-1)
v.push_back(0)
v.push_back(1)
v
In [ ]:
list(v)
In [ ]:
v[0]
In [ ]:
v[0] = -2
v[0]
In [ ]:
VectorInt([0, 1])
Frequently Asked Questions¶
Note
Frequently asked questions about the project and contributing.
How to use AutoWIG on Windows or MacOS ?¶
Currently, AutoWIG binaries for Windows or MacOs X are proposed and can be installed using Conda but are not guaranteed to be working perfectly. However, we provide a Docker image that can be used on these operating systems. Please follow the Test it with Docker procedure.