Welcome to oscar.flag’s documentation!¶
oscar.flag
provides extensible, namespaced flags which can be
parsed from environment variables, command-line arguments and config
files.
Flags are declared where they are used in an application or library, and they are accessed through a namespace matching their fully qualified module path.
Documentation lives at Read the Docs, the code on GitHub.
Example¶
Application entry-point:
import sys
from oscar import flag
import other_module
FLAGS = flag.namespace(__name__)
FLAGS.some_int = flag.Int('some integer value', default=1)
if __name__ == '__main__':
flag.parse_commandline(sys.argv[1:])
flag.die_on_missing_required()
print 'other_module.multiply_by(%d) = %d' % (
FLAGS.some_int,
other_module.multiply_by(FLAGS.some_int))
other_module.py:
from oscar import flag
FLAGS = flag.namespace(__name__)
FLAGS.multiplier = flag.Int('some integer', default=flag.REQUIRED)
def multiply_by(i):
return i * FLAGS.multiplier
shell:
$ python example.py
Missing required flags:
[other_module.]multiplier
Usage of example.py:
__main__:
[__main__.]some_int=None: some integer value
other_module:
[other_module.]multiplier=<required>: some integer
# Note the namespaced reference --other_module.multiplier.
$ python example.py --other_module.multiplier=2 --some_int=3
other_module.multiply_by(3) = 6
License¶
Copyright 2015 Mulberry Health Inc.
Licensed under the Apache License, Version 2.0.
Contents:
Introduction¶
At Oscar, we wanted to address settings and configuration with a
system that would improve operability, solve the need to change values
deep in the code-base, and consume a variety of sources
(i.e. command-line arguments, environment variables, configuration
files). oscar.flag
is not a module for creating user-friendly
command-line interfaces, but it is a module for faceting a large
code-base for configuration.
Goals¶
Allow any module-level constants and values to be configurable. Any constant value should be a flag, and there should be no penalty for doing so.
Eliminate collisions and reduce clutter by providing namespaces. Each flag should be namespaced to its containing module.
Move flags in code close to the places they are used. Avoid central “config” or “settings” modules. Flags should be consumed in code similar to module constants.
ZK_ENSEMBLE = 'zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181' # Becomes this. FLAGS.zk_ensemble = oscar.flag.String('zookeeper ensemble')
Provide clarity from the outside (i.e. command-line). Where a value/flag is used in code should be in the flag name.
# We know immediately that this value is set to a flag that resides in utils.aurora $ ps u | grep 'zookeeper-1' ian 72838 0.0 0.0 2423356 240 s003 R+ 6:31PM 0:00.00 python my_app.py --utils.aurora.zk_ensemble="zookeeper-1:2181"
Allow flags to be set and read from anywhere.
import utils.aurora # Run tests against local zookeeper. utils.aurora.FLAGS.zk_ensemble = 'localhost:2181'
Non-Goals¶
- Provide a set of tools for creating command-line user interfaces. Services and applications are managed through automation (e.g. configuration management).
- Validate flag values. Flag values must marshall, but no further validation is provided.
Installation and Requirements¶
Installation¶
Install using pip or easy_install. oscar.flags
has zero
required external dependencies.
$ pip install oscar.flag
Requirements¶
Python 2.6 and 2.7 are supported.
Usage¶
Declaring Flags¶
Many constant values should be declared as flags. Obtained a
NamespaceFlagSet
from the GLOBAL_FLAGS
singleton GlobalFlagSet
object via the
namespace()
function.
from oscar import flag
FLAGS = flag.namespace(__name__)
There should not be a need to create a GlobalFlagSet
object manually
in typical usage. Flag primitives predefined in the flag module:
-
class
oscar.flag.
String
(description, default=None, secure=False)[source] String-valued flag.
-
class
oscar.flag.
Int
(description, default=None, secure=False)[source] Integer-valued flag.
-
class
oscar.flag.
Float
(description, default=None, secure=False)[source] Float-valued flag.
-
class
oscar.flag.
Bool
(description, default=None, secure=False)[source] Boolean-valued flag.
-
class
oscar.flag.
List
(inner_type, separator, description, default=None, secure=False)[source] Flag that is a list of another flag type.
These are all constructed with at least a description. default
and
secure
are optional.
Flags must be declared at module level, and can only be declared once. Redefining a flag results in an error.
FLAGS.some_int = flag.Int('some int value')
FLAGS.some_string = flag.String('some string value', 'default string')
FLAGS.some_bool = flag.Bool('secure bool (why?)', secure=True)
Required values are indicated by setting default
to
REQUIRED
.
FLAGS.required_float = flag.Float('some required float', flag.REQUIRED)
Finally, List
is provided which can be used to define a
flag which is a list (cannot be polymorphic). It requires two
additional arguments: a Var
subclass defining the
primitive type, and a separator:
FLAGS.int_list = flag.List(flag.Int, ',', 'a list of integer values')
Using flag values in Python¶
Getting¶
Flag values can be read directly from the
NamespaceFlagSet
object (FLAGS
above).
FLAGS.some_int = flag.Int('some int value')
print FLAGS.some_int * 10
Setting¶
Setting is almost the same, but values should be parseable strings, not raw values (since the setter is how the various parsers actually set the values).
>>> FLAGS.some_int = '42'
>>> FLAGS.some_int
42
Positional Arguments¶
Finally, the flag module may expose positional arguments if a
command-line was parsed. These are available via args()
:
-
oscar.flag.
args
()[source] Return positional
args
fromGLOBAL_FLAGS
.Return type: list[str]
Parsing in __main__
¶
There are three parsers provided, a command-line parser, an environment
parser and a parser based on ConfigParser
.
-
oscar.flag.
parse_commandline
(args)[source] Parse commandline
args
withGLOBAL_FLAGS
.
-
oscar.flag.
parse_environment
(args)[source] Parse environment
args
withGLOBAL_FLAGS
.
-
oscar.flag.
parse_ini
(file_p)[source] Parse a
ConfigParser
compatible file withGLOBAL_FLAGS
.
Flag parsing must be done explicitly. Each parser can be used independently or with another parser. It is suggested to use the environment parser first followed by the command-line parser, since the environment parser can read SECURED_ settings, and flags can further be overridden by the command line.
import os
import sys
from oscar import flag
if __name__ == '__main__':
flag.parse_environment(os.environ.items())
flag.parse_commandline(sys.argv[1:])
The ConfigParser
parser requires an object providing
readline()
, which includes a standard opened file.
Note that required flags must be explicitly checked via
die_on_missing_required()
. If a config file is read, but the
path to that config file can be set on the command line, it is useful
to refrain from checking if required flags have been set until after
the config file is parsed.
import os
import sys
from oscar import flag
FLAGS = flag.namespace(__name__)
FLAGS.config_file = flag.String('path to config file')
if __name__ == '__main__':
flag.parse_commandline(sys.argv[1:])
if FLAGS.config_file:
with open(FLAGS.config_file) as config:
flag.parse_ini(config)
die_on_missing_required()
Setting Flags from The Outside¶
Short Names and Full Names¶
All flags are fully namespaced and are available by referencing them
through their module path. For instance, if a flag baz
is declared
in module foo.bar
, it can be referenced on the command line or in
the environment through foo.bar.baz
.
As a convenience, any uniquely named flag can be referenced on the
command line or in the environment through a short name. The short name
is for convenience and should not be used in scripts or configuration
files. Referencing an ambiguous short name will raise a
KeyError
.
Environment Variables¶
If parse_environment()
is called on
os.environ.items()
, environment variables will be mapped onto
flags. As noted, the environment parser supports short and full names.
The environment parser will recognize SECURED_SETTING_* environment
variables. These are base64 decoded and set on the appropriate flag if
present (while also setting the secure
attribute on the flag to
True
).
The environment parser will ignore extraneous environment variables that do not map to a flag.
Note
The last parse method called may overwrite flags set by previous
parse methods. It is probably preferable to call
parse_environment()
before calling
parse_commandline()
.
Command Line¶
Command-line flags are expected to precede any positional arguments. The presence of a single “–” argument can be used to denote the end of command-line flags. A single “-” or a double “–” are equivalent in denoting a flag. A flag and its value can be separated into separate arguments (i.e. via whitespace on the command line) or a single “=”.
# A single '-' is acceptable.
$ ./my_bin.pex -foo=bar
$ ./my_bin.pex -foo bar
# A double '--' is also acceptable.
$ ./my_bin.pex --foo=bar
$ ./my_bin.pex --foo bar
# Everything after '--' becomes a positional argument.
$ ./my_bin.pex -- --foo --bar --baz
Bool
flags are special-cased in the command-line parser. A
standalone Bool
flag with no value indicates
True
. A Bool
flag can only be set explicitly
using “=”, and a space between the flag and the value is
invalid. Valid boolean values are any case folded variation of
yes
, true
, on
, 1
for True
, and no
,
false
, off
, 0
for False
. Other values result in
an error.
./my_bin.pex --some_bool # some_bool is set to True
./my_bin.pex --some_bool=False # some_bool is set to False
./my_bin.pex --some_bool False # this is invalid and will throw a parse error
Command-line flags also may be referenced by their short name if it’s non-ambiguous, though this is provided as a shortcut for users, and the fully qualified flag name should be used in configuration and scripting.
Finally, if any flags are encountered that do not map to a flag in the application, an error will be raised. All command-line flags must map to a declared flag.
ConfigParser
ini files¶
There is also support for ini files. Sections map to namespaces, and key/value pairs within the sections map to flags within those namespaces.
[__main__]
output=output.txt
[utils.zookeeper]
ensemble=zookeeper-1,zookeeper-2,zookeeper-3
The ini parser will need to be ran against a file-like object.
with open('~/.config.ini') as config:
flag.parse_ini(config)
Finally, every section and key/value within an ini file must map to a namespace and flag. Unexpected sections and keys will raise an error.
Additional Public API for flag¶
-
oscar.flag.
GLOBAL_FLAGS
= <oscar.flag.GlobalFlagSet object> GlobalFlagSet is a collection of namespaces and flag logic.
-
GLOBAL_FLAGS.
usage
¶ This attribute can be replaced with a function that will print usage (invoked automatically by –help on the command line). The function accepts a single parameter: the
GlobalFlagSet
object calling it. The default implementation callsGlobalFlagSet.write_flags()
. Default value isdefault_usage()
.
-
GLOBAL_FLAGS.
usage_long
¶ This attribute can be replaced with a function that will print long usage (invoked automatically by –helplong on the command line). The function accepts a single parameter: the
GlobalFlagSet
object calling it. The default implementation callsGlobalFlagSet.write_flags_long()
. Default value isdefault_usage_long()
.
-
GlobalFlagSet.
visit
(func) Walk all set flags, calling
func
on each.Parameters: func (F(str, str, Var)) – visiting function
-
GlobalFlagSet.
visit_all
(func) Walk all flags, calling
func
on each.Parameters: func (F(str, str, Var)) – visiting function
-
oscar.flag¶
-
oscar.flag.
GLOBAL_FLAGS
= <oscar.flag.GlobalFlagSet object>¶ GlobalFlagSet is a collection of namespaces and flag logic.
-
oscar.flag.
REQUIRED
= <oscar.flag._Required object>¶ Setting a flag’s
default
to this constant marks it as required.
-
class
oscar.flag.
Bool
(description, default=None, secure=False)[source]¶ Boolean-valued flag.
-
set
(value)¶
-
type_str
= 'Bool'¶
-
-
class
oscar.flag.
Float
(description, default=None, secure=False)[source]¶ Float-valued flag.
-
set
(value)¶
-
type_str
= 'Float'¶
-
-
class
oscar.flag.
GlobalFlagSet
(usage=<function default_usage>, usage_long=<function default_usage_long>)[source]¶ GlobalFlagSet is a collection of namespaces and flag logic.
-
check_required
()¶ Returns a list of
(namespace, name, flag)
of all unset, required flags.Return type: list[tuple(str, str, Var)]
-
find_short
(flag)¶ Find the namespace for a non-qualified
flag
.If the
flag
is not found or multiple flags are found,KeyError
is raised.Return type: str Raises: KeyError – if the flag is not found or ambiguous
-
get
(namespace, flag)¶ get the flag object associated with
flag
innamespace
.Return type: Var Raises: KeyError – if the flag is not found
-
namespace
(namespace)¶ Returns a
NamespaceFlagSet
associated withnamespace
.Return type: NamespaceFlagSet
-
parse_commandline
(args)¶ Parse a commandline into flags and arguments.
Parse a commandline. a single ‘-‘ and a double ‘–’ are treated as equivalent for denoting a flag. Flag values may be separated by ‘=’ or be the next argument. Booleans are a special case that indicate a true value or must have their values separated by ‘=’.
Raises: - KeyError – on unknown flag
- ParseException – on invalid command-line syntax
-
parse_environment
(args)¶ Parse environment variable tuples.
- Call with os.environ:
>>> import os >>> flag.parse_environment(os.environ.items())
Recognizes SECURED_SETTING_ prefixed flags, translates from base64 and maps to the short name. secure is also set to True on the underlying flag object, which should be respected by users of
visit()
andvisit_all()
.
-
parse_ini
(file_p)¶ Parse a
ConfigParser
compatible file object.Namespaces are section headers, keys are flags:
[__main__] foo=bar [foo.bar] baz=42
file_p
only needs to implementreadline(size=0)
.
-
visit
(func)¶ Walk all set flags, calling
func
on each.Parameters: func (F(str, str, Var)) – visiting function
-
visit_all
(func)¶ Walk all flags, calling
func
on each.Parameters: func (F(str, str, Var)) – visiting function
-
write_flags
(out, namespace='__main__')¶ Prints the usage to
out
.
-
write_flags_long
(out)¶ Prints all flag usage to
out
.
-
-
class
oscar.flag.
Int
(description, default=None, secure=False)[source]¶ Integer-valued flag.
-
set
(value)¶
-
type_str
= 'Int'¶
-
-
class
oscar.flag.
List
(inner_type, separator, description, default=None, secure=False)[source]¶ Flag that is a list of another flag type.
-
set
(value)¶
-
type_str
= 'List[_]'¶
-
-
class
oscar.flag.
String
(description, default=None, secure=False)[source]¶ String-valued flag.
-
set
(value)¶
-
type_str
= 'String'¶
-
-
class
oscar.flag.
Var
(description, default=None, secure=False)[source]¶ Base of all flag accessors.
-
get
()¶ Return the flag value, or default if it is not set.
-
type_str
= 'Unknown'¶
-
value
= <object object>¶
-
-
oscar.flag.
args
()[source]¶ Return positional
args
fromGLOBAL_FLAGS
.Return type: list[str]
-
oscar.flag.
default_usage_long
(globalflags, return_code=0)[source]¶ Default for printing out long-form usage.
-
oscar.flag.
namespace
(name)[source]¶ Return a namespace from
GLOBAL_FLAGS
.Return type: NamespaceFlagSet
-
oscar.flag.
parse_commandline
(args)[source]¶ Parse commandline
args
withGLOBAL_FLAGS
.
-
oscar.flag.
parse_environment
(args)[source]¶ Parse environment
args
withGLOBAL_FLAGS
.
-
oscar.flag.
parse_ini
(file_p)[source]¶ Parse a
ConfigParser
compatible file withGLOBAL_FLAGS
.