mozharness.base package

Submodules

mozharness.base.config module

Generic config parsing and dumping, the way I remember it from scripts gone by.

The config should be built from script-level defaults, overlaid by config-file defaults, overlaid by command line options.

(For buildbot-analogues that would be factory-level defaults,
builder-level defaults, and build request/scheduler settings.)

The config should then be locked (set to read-only, to prevent runtime alterations). Afterwards we should dump the config to a file that is uploaded with the build, and can be used to debug or replicate the build at a later time.

TODO:

  • check_required_settings or something – run at init, assert that these settings are set.
class mozharness.base.config.BaseConfig(config=None, initial_config_file=None, config_options=None, all_actions=None, default_actions=None, volatile_config=None, option_args=None, require_config_file=False, append_env_variables_from_configs=False, usage='usage: %prog [options]')[source]

Bases: object

Basic config setting/getting.

get_actions()[source]
get_cfgs_from_files(all_config_files, options)[source]

Returns the configuration derived from the list of configuration files. The result is represented as a list of (filename, config_dict) tuples; they will be combined with keys in later dictionaries taking precedence over earlier.

all_config_files is all files specified with –config-file and –opt-config-file; options is the argparse options object giving access to any other command-line options.

This function is also responsible for downloading any configuration files specified by URL. It uses parse_config_file in this module to parse individual files.

This method can be overridden in a subclass to add extra logic to the way that self.config is made up. See mozharness.mozilla.building.buildbase.BuildingConfig for an example.

get_read_only_config()[source]
list_actions()[source]
parse_args(args=None)[source]

Parse command line arguments in a generic way. Return the parser object after adding the basic options, so child objects can manipulate it.

set_config(config, overwrite=False)[source]

This is probably doable some other way.

verify_actions(action_list, quiet=False)[source]
verify_actions_order(action_list)[source]
class mozharness.base.config.ExtendOption(*opts, **attrs)[source]

Bases: optparse.Option

from http://docs.python.org/library/optparse.html?highlight=optparse#adding-new-actions

ACTIONS = ('store', 'store_const', 'store_true', 'store_false', 'append', 'append_const', 'count', 'callback', 'help', 'version', 'extend')
ALWAYS_TYPED_ACTIONS = ('store', 'append', 'extend')
STORE_ACTIONS = ('store', 'store_const', 'store_true', 'store_false', 'append', 'append_const', 'count', 'extend')
TYPED_ACTIONS = ('store', 'append', 'callback', 'extend')
take_action(action, dest, opt, value, values, parser)[source]
class mozharness.base.config.ExtendedOptionParser(**kwargs)[source]

Bases: optparse.OptionParser

OptionParser, but with ExtendOption as the option_class.

class mozharness.base.config.LockedTuple[source]

Bases: tuple

class mozharness.base.config.ReadOnlyDict(dictionary)[source]

Bases: dict

clear(*args)[source]
lock()[source]
pop(*args)[source]
popitem(*args)[source]
setdefault(*args)[source]
update(*args)[source]
mozharness.base.config.download_config_file(url, file_name)[source]
mozharness.base.config.make_immutable(item)[source]
mozharness.base.config.parse_config_file(file_name, quiet=False, search_path=None, config_dict_name='config')[source]

Read a config file and return a dictionary.

mozharness.base.errors module

Generic error lists.

Error lists are used to parse output in mozharness.base.log.OutputParser.

Each line of output is matched against each substring or regular expression in the error list. On a match, we determine the ‘level’ of that line, whether IGNORE, DEBUG, INFO, WARNING, ERROR, CRITICAL, or FATAL.

TODO: Context lines (requires work on the OutputParser side)

TODO: We could also create classes that generate these, but with the appropriate level (please don’t die on any errors; please die on any warning; etc.) or platform or language or whatever.

exception mozharness.base.errors.VCSException[source]

Bases: exceptions.Exception

mozharness.base.gaia_test module

mozharness.base.log module

Generic logging classes and functionalities for single and multi file logging. Capturing console output and providing general logging functionalities.

Attributes:
FATAL_LEVEL (int): constant logging level value set based on the logging.CRITICAL
value

DEBUG (str): mozharness debug log name INFO (str): mozharness info log name WARNING (str): mozharness warning log name CRITICAL (str): mozharness critical log name FATAL (str): mozharness fatal log name IGNORE (str): mozharness ignore log name LOG_LEVELS (dict): mapping of the mozharness log level names to logging values ROOT_LOGGER (logging.Logger): instance of a logging.Logger class

TODO: - network logging support. - log rotation config

class mozharness.base.log.BaseLogger(log_level='info', log_format='%(message)s', log_date_format='%H:%M:%S', log_name='test', log_to_console=True, log_dir='.', log_to_raw=False, logger_name='', append_to_log=False)[source]

Bases: object

Base class in charge of logging handling logic such as creating logging files, dirs, attaching to the console output and managing its output.

Attributes:
LEVELS (dict): flat copy of the LOG_LEVELS attribute of the log module.

TODO: status? There may be a status object or status capability in either logging or config that allows you to count the number of error,critical,fatal messages for us to count up at the end (aiming for 0).

LEVELS = {'info': 20, 'warning': 30, 'critical': 50, 'error': 40, 'debug': 10, 'fatal': 60}
add_console_handler(log_level=None, log_format=None, date_format=None)[source]

create a logging.StreamHandler using sys.stderr for logging the console output and add it to the all_handlers member variable

Args:
log_level (str, optional): useless argument. Not used here.
Defaults to None.
log_format (str, optional): format used for the Formatter attached to the
StreamHandler. Defaults to None.
date_format (str, optional): format used for the Formatter attached to the
StreamHandler. Defaults to None.
add_file_handler(log_path, log_level=None, log_format=None, date_format=None)[source]

create a logging.FileHandler base on the path, log and date format and add it to the all_handlers member variable.

Args:

log_path (str): filepath to use for the FileHandler. log_level (str, optional): useless argument. Not used here.

Defaults to None.
log_format (str, optional): log format to use for the Formatter constructor.
Defaults to the current instance log format.
date_format (str, optional): date format to use for the Formatter constructor.
Defaults to the current instance date format.
create_log_dir()[source]

create a logging directory if it doesn’t exits. If there is a file with same name as the future logging directory it will be deleted.

get_log_formatter(log_format=None, date_format=None)[source]

create a logging.Formatter base on the log and date format.

Args:
log_format (str, optional): log format to use for the Formatter constructor.
Defaults to the current instance log format.
date_format (str, optional): date format to use for the Formatter constructor.
Defaults to the current instance date format.
Returns:
logging.Formatter: instance created base on the passed arguments
get_logger_level(level=None)[source]
translate the level name passed to it and return its numeric value
according to LEVELS values.
Args:
level (str, optional): level name to be translated. Defaults to the current
instance log_level.
Returns:
int: numeric value of the log level name passed to it or 0 (NOTSET) if the
name doesn’t exists
init_message(name=None)[source]

log an init message stating the name passed to it, the current date and time and, the current working directory.

Args:
name (str, optional): name to use for the init log message. Defaults to
the current instance class name.
log_message(message, level='info', exit_code=-1, post_fatal_callback=None)[source]
Generic log method.
There should be more options here – do or don’t split by line, use os.linesep instead of assuming
, be able to pass in log level

by name or number.

Adding the IGNORE special level for runCommand.

Args:

message (str): message to log using the current logger level (str, optional): log level of the message. Defaults to INFO. exit_code (int, optional): exit code to use in case of a FATAL level is used.

Defaults to -1.
post_fatal_callback (function, optional): function to callback in case of
of a fatal log level. Defaults None.
new_logger()[source]

Create a new logger based on the ROOT_LOGGER instance. By default there are no handlers. The new logger becomes a member variable of the current instance as self.logger.

class mozharness.base.log.LogMixin[source]

Bases: object

This is a mixin for any object to access similar logging functionality

The logging functionality described here is specially useful for those objects with self.config and self.log_obj member variables

critical(message)[source]

calls the log method with CRITICAL as logging level

Args:
message (str): message to log
debug(message)[source]

calls the log method with DEBUG as logging level

Args:
message (str): message to log
error(message)[source]

calls the log method with ERROR as logging level

Args:
message (str): message to log
exception(message=None, level='error')[source]

log an exception message base on the log level passed to it.

This function fetches the information of the current exception being handled and adds it to the message argument.

Args:
message (str, optional): message to be printed at the beginning of the log.
Default to an empty string.

level (str, optional): log level to use for the logging. Defaults to ERROR

Returns:
None
fatal(message, exit_code=-1)[source]

calls the log method with FATAL as logging level

Args:

message (str): message to log exit_code (int, optional): exit code to use for the SystemExit

exception to be raised. Default to -1.
info(message)[source]

calls the log method with INFO as logging level

Args:
message (str): message to log
log(message, level='info', exit_code=-1)[source]

log the message passed to it according to level, exit if level == FATAL

Args:

message (str): message to be logged level (str, optional): logging level of the message. Defaults to INFO exit_code (int, optional): exit code to log before the scripts calls

SystemExit.
Returns:
None
warning(message)[source]

calls the log method with WARNING as logging level

Args:
message (str): message to log
worst_level(target_level, existing_level, levels=None)[source]

Compare target_level with existing_level according to levels values and return the worst among them.

Args:
target_level (str): minimum logging level to which the current object
should be set

existing_level (str): current logging level levels (list(str), optional): list of logging levels names to compare

target_level and existing_level against. Defaults to mozharness log level list sorted from most to less critical.
Returns:
str: the logging lavel that is closest to the first levels value,
i.e. levels[0]
class mozharness.base.log.MultiFileLogger(logger_name='Multi', log_format='%(asctime)s %(levelname)8s - %(message)s', log_dir='logs', log_to_raw=True, **kwargs)[source]

Bases: mozharness.base.log.BaseLogger

Subclass of the BaseLogger class. Create a log per log level in log_dir. Possibly also output to the terminal and a raw log (no prepending of level or date)

new_logger()[source]

calls the BaseLogger.new_logger method and adds a file handler per logging level in the LEVELS class attribute.

class mozharness.base.log.OutputParser(config=None, log_obj=None, error_list=None, log_output=True)[source]

Bases: mozharness.base.log.LogMixin

Helper object to parse command output.

This will buffer output if needed, so we can go back and mark [(linenum - 10) : linenum+10] as errors if need be, without having to get all the output first.

linenum+10 will be easy; we can set self.num_post_context_lines to 10, and self.num_post_context_lines– as we mark each line to at least error level X.

linenum-10 will be trickier. We’ll not only need to save the line itself, but also the level that we’ve set for that line previously, whether by matching on that line, or by a previous line’s context. We should only log that line if all output has ended (self.finish() ?); otherwise store a list of dictionaries in self.context_buffer that is buffered up to self.num_pre_context_lines (set to the largest pre-context-line setting in error_list.)

add_lines(output)[source]

process a string or list of strings, decode them to utf-8,strip them of any trailing whitespaces and parse them using parse_single_line

strings consisting only of whitespaces are ignored.

Args:
output (str | list): string or list of string to parse
parse_single_line(line)[source]

parse a console output line and check if it matches one in error_list, if so then log it according to log_output.

Args:
line (str): command line output to parse.
class mozharness.base.log.SimpleFileLogger(log_format='%(asctime)s %(levelname)8s - %(message)s', logger_name='Simple', log_dir='logs', **kwargs)[source]

Bases: mozharness.base.log.BaseLogger

Subclass of the BaseLogger.

Create one logFile. Possibly also output to the terminal and a raw log (no prepending of level or date)

new_logger()[source]

calls the BaseLogger.new_logger method and adds a file handler to it.

mozharness.base.log.numeric_log_level(level)[source]

Converts a mozharness log level (string) to the corresponding logger level (number). This function makes possible to set the log level in functions that do not inherit from LogMixin

Args:
level (str): log level name to convert.
Returns:
int: numeric value of the log level name.

mozharness.base.mar module

mozharness.base.parallel module

Generic ways to parallelize jobs.

class mozharness.base.parallel.ChunkingMixin[source]

Bases: object

Generic signing helper methods.

query_chunked_list(possible_list, this_chunk, total_chunks, sort=False)[source]

Split a list of items into a certain number of chunks and return the subset of that will occur in this chunk.

Ported from build.l10n.getLocalesForChunk in build/tools.

mozharness.base.python module

Python usage, esp. virtualenv.

class mozharness.base.python.InfluxRecordingMixin[source]

Bases: object

Provides InfluxDB stat recording to scripts.

This class records stats to an InfluxDB server, if enabled. Stat recording is enabled in a script by inheriting from this class, and adding an influxdb_credentials line to the influx_credentials_file (usually oauth.txt in automation). This line should look something like:

Where DBNAME, DBUSERNAME, and DBPASSWORD correspond to the database name, and user/pw credentials for recording to the database. The stats from mozharness are recorded in the ‘mozharness’ table.

influxdb_recording_init()[source]
influxdb_recording_post_action(action, success=None)[source]
influxdb_recording_pre_action(action)[source]
record_influx_stat(json_data)[source]
record_mach_stats(action, success=None)[source]
class mozharness.base.python.ResourceMonitoringMixin(*args, **kwargs)[source]

Bases: object

Provides resource monitoring capabilities to scripts.

When this class is in the inheritance chain, resource usage stats of the executing script will be recorded.

This class requires the VirtualenvMixin in order to install a package used for recording resource usage.

While we would like to record resource usage for the entirety of a script, since we require an external package, we can only record resource usage after that package is installed (as part of creating the virtualenv). That’s just the way things have to be.

class mozharness.base.python.VirtualenvMixin(*args, **kwargs)[source]

Bases: object

BaseScript mixin, designed to create and use virtualenvs.

Config items:
  • virtualenv_path points to the virtualenv location on disk.
  • virtualenv_modules lists the module names.
  • MODULE_url list points to the module URLs (optional)

Requires virtualenv to be in PATH. Depends on ScriptMixin

activate_virtualenv()[source]

Import the virtualenv’s packages into this Python interpreter.

create_virtualenv(modules=(), requirements=())[source]

Create a python virtualenv.

The virtualenv exe can be defined in c[‘virtualenv’] or c[‘exes’][‘virtualenv’], as a string (path) or list (path + arguments).

c[‘virtualenv_python_dll’] is an optional config item that works around an old windows virtualenv bug.

virtualenv_modules can be a list of module names to install, e.g.

virtualenv_modules = [‘module1’, ‘module2’]

or it can be a heterogeneous list of modules names and dicts that define a module by its name, url-or-path, and a list of its global options.

virtualenv_modules = [
{
‘name’: ‘module1’, ‘url’: None, ‘global_options’: [‘–opt’, ‘–without-gcc’]

}, {

‘name’: ‘module2’, ‘url’: ‘http://url/to/package‘, ‘global_options’: [‘–use-clang’]

}, {

‘name’: ‘module3’, ‘url’: os.path.join(‘path’, ‘to’, ‘setup_py’, ‘dir’) ‘global_options’: []

}, ‘module4’

]

virtualenv_requirements is an optional list of pip requirements files to use when invoking pip, e.g.,

virtualenv_requirements = [
‘/path/to/requirements1.txt’, ‘/path/to/requirements2.txt’

]

install_module(module=None, module_url=None, install_method=None, requirements=(), optional=False, global_options=[], no_deps=False, editable=False)[source]

Install module via pip.

module_url can be a url to a python package tarball, a path to a directory containing a setup.py (absolute or relative to work_dir) or None, in which case it will default to the module name.

requirements is a list of pip requirements files. If specified, these will be combined with the module_url (if any), like so:

pip install -r requirements1.txt -r requirements2.txt module_url

is_python_package_installed(package_name, error_level='warning')[source]

Return whether the package is installed

package_versions(pip_freeze_output=None, error_level='warning', log_output=False)[source]

reads packages from pip freeze output and returns a dict of {package_name: ‘version’}

python_paths = {}
query_python_path(binary='python')[source]

Return the path of a binary inside the virtualenv, if c[‘virtualenv_path’] is set; otherwise return the binary name. Otherwise return None

query_python_site_packages_path()[source]
query_virtualenv_path()[source]
register_virtualenv_module(name=None, url=None, method=None, requirements=None, optional=False, two_pass=False, editable=False)[source]

Register a module to be installed with the virtualenv.

This method can be called up until create_virtualenv() to register modules that should be installed in the virtualenv.

See the documentation for install_module for how the arguments are applied.

site_packages_path = None

mozharness.base.script module

Generic script objects.

script.py, along with config.py and log.py, represents the core of mozharness.

class mozharness.base.script.BaseScript(config_options=None, ConfigClass=<class 'mozharness.base.config.BaseConfig'>, default_log_level='info', **kwargs)[source]

Bases: mozharness.base.script.ScriptMixin, mozharness.base.log.LogMixin, object

action_message(message)[source]
add_failure(key, message='%(key)s failed.', level='error', increment_return_code=True)[source]
add_summary(message, level='info')[source]
clobber()[source]

Delete the working directory

copy_logs_to_upload_dir()[source]

Copies logs to the upload directory

copy_to_upload_dir(target, dest=None, short_desc='unknown', long_desc='unknown', log_level='debug', error_level='error', max_backups=None, compress=False, upload_dir=None)[source]

Copy target file to upload_dir/dest.

Potentially update a manifest in the future if we go that route.

Currently only copies a single file; would be nice to allow for recursive copying; that would probably done by creating a helper _copy_file_to_upload_dir().

short_desc and long_desc are placeholders for if/when we add upload_dir manifests.

dump_config(file_path=None, config=None, console_output=True, exit_on_finish=False)[source]

Dump self.config to localconfig.json

file_sha512sum(file_path)[source]
new_log_obj(default_log_level='info')[source]
query_abs_dirs()[source]

We want to be able to determine where all the important things are. Absolute paths lend themselves well to this, though I wouldn’t be surprised if this causes some issues somewhere.

This should be overridden in any script that has additional dirs to query.

The query_* methods tend to set self.VAR variables as their runtime cache.

query_failure(key)[source]
return_code
run()[source]

Default run method. This is the “do everything” method, based on actions and all_actions.

First run self.dump_config() if it exists. Second, go through the list of all_actions. If they’re in the list of self.actions, try to run self.preflight_ACTION(), self.ACTION(), and self.postflight_ACTION().

Preflight is sanity checking before doing anything time consuming or destructive.

Postflight is quick testing for success after an action.

run_action(action)[source]
run_and_exit()[source]

Runs the script and exits the current interpreter.

summarize_success_count(success_count, total_count, message='%d of %d successful.', level=None)[source]
summary()[source]

Print out all the summary lines added via add_summary() throughout the script.

I’d like to revisit how to do this in a prettier fashion.

class mozharness.base.script.PlatformMixin[source]

Bases: object

mozharness.base.script.PostScriptAction(action=None)[source]

Decorator for methods that will be called at the end of each action.

This behaves similarly to PreScriptAction. It varies in that it is called after execution of the action.

The decorated method will receive the action name as a positional argument. It will then receive the following named arguments:

success - Bool indicating whether the action finished successfully.

The decorated method will always be called, even if the action threw an exception.

The return value is ignored.

mozharness.base.script.PostScriptRun(func)[source]

Decorator for methods that will be called after script execution.

This is similar to PreScriptRun except it is called at the end of execution. The method will always be fired, even if execution fails.

mozharness.base.script.PreScriptAction(action=None)[source]

Decorator for methods that will be called at the beginning of each action.

Each method on a BaseScript having this decorator will be called during BaseScript.run() before an individual action is executed. The method will receive the action’s name as an argument.

If no values are passed to the decorator, it will be applied to every action. If a string is passed, the decorated function will only be called for the action of that name.

The return value of the method is ignored. Exceptions will abort execution.

mozharness.base.script.PreScriptRun(func)[source]

Decorator for methods that will be called before script execution.

Each method on a BaseScript having this decorator will be called at the beginning of BaseScript.run().

The return value is ignored. Exceptions will abort execution.

class mozharness.base.script.ScriptMixin[source]

Bases: mozharness.base.script.PlatformMixin

This mixin contains simple filesystem commands and the like.

It also contains some very special but very complex methods that, together with logging and config, provide the base for all scripts in this harness.

WARNING !!! This class depends entirely on LogMixin methods in such a way that it will only works if a class inherits from both ScriptMixin and LogMixin simultaneously.

Depends on self.config of some sort.

Attributes:
env (dict): a mapping object representing the string environment. script_obj (ScriptMixin): reference to a ScriptMixin instance.
chdir(dir_name)[source]
chmod(path, mode)[source]

change path mode to mode.

Args:
path (str): path whose mode will be modified. mode (hex): one of the values defined at stat

https://docs.python.org/2/library/os.html#os.chmod

copyfile(src, dest, log_level='info', error_level='error', copystat=False, compress=False)[source]

copy or compress src into dest.

Args:

src (str): filepath to copy. dest (str): filepath where to move the content to. log_level (str, optional): log level to use for normal operation. Defaults to

INFO

error_level (str, optional): log level to use on error. Defaults to ERROR copystat (bool, optional): whether or not to copy the files metadata.

Defaults to False.
compress (bool, optional): whether or not to compress the destination file.
Defaults to False.
Returns:
int: -1 on error None: on success
copytree(src, dest, overwrite='no_overwrite', log_level='info', error_level='error')[source]

An implementation of shutil.copytree that allows for dest to exist and implements different overwrite levels: - ‘no_overwrite’ will keep all(any) existing files in destination tree - ‘overwrite_if_exists’ will only overwrite destination paths that have

the same path names relative to the root of the src and destination tree
  • ‘clobber’ will replace the whole destination tree(clobber) if it exists
Args:

src (str): directory path to move. dest (str): directory path where to move the content to. overwrite (str): string specifying the overwrite level. log_level (str, optional): log level to use for normal operation. Defaults to

INFO

error_level (str, optional): log level to use on error. Defaults to ERROR

Returns:
int: -1 on error None: on success
download_file(url, file_name=None, parent_dir=None, create_parent_dir=True, error_level='error', exit_code=3, retry_config=None)[source]

Python wget. Download the filename at url into file_name and put it on parent_dir. On error log with the specified error_level, on fatal exit with exit_code. Execute all the above based on retry_config parameter.

Args:

url (str): URL path where the file to be downloaded is located. file_name (str, optional): file_name where the file will be written to.

Defaults to urls’ filename.
parent_dir (str, optional): directory where the downloaded file will
be written to. Defaults to current working directory
create_parent_dir (bool, optional): create the parent directory if it
doesn’t exist. Defaults to True
error_level (str, optional): log level to use in case an error occurs.
Defaults to ERROR
retry_config (dict, optional): key-value pairs to be passed to
self.retry. Defaults to None
Returns:
str: filename where the downloaded file was written to. unknown: on failure, failure_status is returned.
env = None
get_filename_from_url(url)[source]

parse a filename base on an url.

Args:
url (str): url to parse for the filename
Returns:
str: filename parsed from the url, or netloc network location part
of the url.
get_output_from_command(command, cwd=None, halt_on_failure=False, env=None, silent=False, log_level='info', tmpfile_base_path='tmpfile', return_type='output', save_tmpfiles=False, throw_exception=False, fatal_exit_code=2, ignore_errors=False, success_codes=None)[source]

Similar to run_command, but where run_command is an os.system(command) analog, get_output_from_command is a command analog.

Less error checking by design, though if we figure out how to do it without borking the output, great.

TODO: binary mode? silent is kinda like that. TODO: since p.wait() can take a long time, optionally log something every N seconds? TODO: optionally only keep the first or last (N) line(s) of output? TODO: optionally only return the tmp_stdout_filename?

ignore_errors=True is for the case where a command might produce standard error output, but you don’t particularly care; setting to True will cause standard error to be logged at DEBUG rather than ERROR

Args:
command (str | list): command or list of commands to
execute and log.
cwd (str, optional): directory path from where to execute the
command. Defaults to None.
halt_on_failure (bool, optional): whether or not to redefine the
log level as FATAL on error. Defaults to False.
env (dict, optional): key-value of environment values to use to
run the command. Defaults to None.
silent (bool, optional): whether or not to output the stdout of
executing the command. Defaults to False.
log_level (str, optional): log level name to use on normal execution.
Defaults to INFO.
tmpfile_base_path (str, optional): base path of the file to which
the output will be writen to. Defaults to ‘tmpfile’.
return_type (str, optional): if equal to ‘output’ then the complete
output of the executed command is returned, otherwise the written filenames are returned. Defaults to ‘output’.
save_tmpfiles (bool, optional): whether or not to save the temporary
files created from the command output. Defaults to False.
throw_exception (bool, optional): whether or not to raise an
exception if the return value of the command is not zero. Defaults to False.
fatal_exit_code (int, optional): call self.fatal if the return value
of the command match this value.
ignore_errors (bool, optional): whether or not to change the log
level to ERROR for the output of stderr. Defaults to False.
success_codes (int, optional): numeric value to compare against
the command return value.
Returns:
None: if the cwd is not a directory. None: on IOError. tuple: stdout and stderr filenames. str: stdout output.
is_exe(fpath)[source]

Determine if fpath is a file and if it is executable.

mkdir_p(path, error_level='error')[source]

Create a directory if it doesn’t exists. This method also logs the creation, error or current existence of the directory to be created.

Args:
path (str): path of the directory to be created. error_level (str): log level name to be used in case of error.
Returns:
None: for sucess. int: -1 on error
move(src, dest, log_level='info', error_level='error', exit_code=-1)[source]

recursively move a file or directory (src) to another location (dest).

Args:

src (str): file or directory path to move. dest (str): file or directory path where to move the content to. log_level (str): log level to use for normal operation. Defaults to

INFO

error_level (str): log level to use on error. Defaults to ERROR

Returns:
int: 0 on success. -1 on error.
opened(*args, **kwds)[source]

Create a context manager to use on a with statement.

Args:

file_path (str): filepath of the file to open. verbose (bool, optional): useless parameter, not used here.

Defaults to True.
open_mode (str, optional): open mode to use for openning the file.
Defaults to r
error_level (str, optional): log level name to use on error.
Defaults to ERROR
Yields:
tuple: (file object, error) pair. In case of error None is yielded
as file object, together with the corresponding error. If there is no error, None is returned as the error.
query_env(partial_env=None, replace_dict=None, purge_env=(), set_self_env=None, log_level='debug', avoid_host_env=False)[source]

Environment query/generation method. The default, self.query_env(), will look for self.config[‘env’] and replace any special strings in there ( %(PATH)s ). It will then store it as self.env for speeding things up later.

If you specify partial_env, partial_env will be used instead of self.config[‘env’], and we don’t save self.env as it’s a one-off.

Args:
partial_env (dict, optional): key-value pairs of the name and value
of different environment variables. Defaults to an empty dictionary.
replace_dict (dict, optional): key-value pairs to replace the old
environment variables.
purge_env (list): environment names to delete from the final
environment dictionary.
set_self_env (boolean, optional): whether or not the environment
variables dictionary should be copied to self. Defaults to True.
log_level (str, optional): log level name to use on normal operation.
Defaults to DEBUG.
avoid_host_env (boolean, optional): if set to True, we will not use
any environment variables set on the host except PATH. Defaults to False.
Returns:
dict: environment variables names with their values.
query_exe(exe_name, exe_dict='exes', default=None, return_type=None, error_level='fatal')[source]

One way to work around PATH rewrites.

By default, return exe_name, and we’ll fall through to searching os.environ[“PATH”]. However, if self.config[exe_dict][exe_name] exists, return that. This lets us override exe paths via config file.

If we need runtime setting, we can build in self.exes support later.

Args:

exe_name (str): name of the executable to search for. exe_dict(str, optional): name of the dictionary of executables

present in self.config. Defaults to exes.
default (str, optional): default name of the executable to search
for. Defaults to exe_name.
return_type (str, optional): type to which the original return
value will be turn into. Only ‘list’, ‘string’ and None are supported. Defaults to None.

error_level (str, optional): log level name to use on error.

Returns:
list: in case return_type is ‘list’ str: in case return_type is ‘string’ None: in case return_type is None Any: if the found executable is not of type list, tuple nor str.
query_msys_path(path)[source]

replaces the Windows harddrive letter path style with a linux path style, e.g. C:// –> /C/ Note: method, not used in any script.

Args:
path (str?): path to convert to the linux path style.
Returns:
str: in case path is a string. The result is the path with the new notation. type(path): path itself is returned in case path is not str type.
read_from_file(file_path, verbose=True, open_mode='r', error_level='error')[source]

Use self.opened context manager to open a file and read its content.

Args:

file_path (str): filepath of the file to read. verbose (bool, optional): whether or not to log the file content.

Defaults to True.
open_mode (str, optional): open mode to use for openning the file.
Defaults to r
error_level (str, optional): log level name to use on error.
Defaults to ERROR
Returns:
None: on error. str: file content on success.
retry(action, attempts=None, sleeptime=60, max_sleeptime=300, retry_exceptions=(<type 'exceptions.Exception'>, ), good_statuses=None, cleanup=None, error_level='error', error_message='%(action)s failed after %(attempts)d tries!', failure_status=-1, log_level='info', args=(), kwargs={})[source]

generic retry command. Ported from `util.retry`_

Args:

action (func): callable object to retry. attempts (int, optinal): maximum number of times to call actions.

Defaults to self.config.get(‘global_retries’, 5)
sleeptime (int, optional): number of seconds to wait between
attempts. Defaults to 60 and doubles each retry attempt, to a maximum of `max_sleeptime’
max_sleeptime (int, optional): maximum value of sleeptime. Defaults
to 5 minutes
retry_exceptions (tuple, optional): Exceptions that should be caught.
If exceptions other than those listed in `retry_exceptions’ are raised from `action’, they will be raised immediately. Defaults to (Exception)
good_statuses (object, optional): return values which, if specified,
will result in retrying if the return value isn’t listed. Defaults to None.
cleanup (func, optional): If `cleanup’ is provided and callable
it will be called immediately after an Exception is caught. No arguments will be passed to it. If your cleanup function requires arguments it is recommended that you wrap it in an argumentless function. Defaults to None.
error_level (str, optional): log level name in case of error.
Defaults to ERROR.
error_message (str, optional): string format to use in case
none of the attempts success. Defaults to ‘%(action)s failed after %(attempts)d tries!’
failure_status (int, optional): flag to return in case the retries
were not successfull. Defaults to -1.
log_level (str, optional): log level name to use for normal activity.
Defaults to INFO.

args (tuple, optional): positional arguments to pass onto action. kwargs (dict, optional): key-value arguments to pass onto action.

Returns:
object: return value of action. int: failure status in case of failure retries.
rmtree(path, log_level='info', error_level='error', exit_code=-1)[source]

Delete an entire directory tree and log its result. This method also logs the platform rmtree function, its retries, errors, and current existence of the directory.

Args:

path (str): path to the directory tree root to remove. log_level (str, optional): log level name to for this operation. Defaults

to INFO.
error_level (str, optional): log level name to use in case of error.
Defaults to ERROR.
exit_code (int, optional): useless parameter, not use here.
Defaults to -1
Returns:
None: for success
run_command(command, cwd=None, error_list=None, halt_on_failure=False, success_codes=None, env=None, partial_env=None, return_type='status', throw_exception=False, output_parser=None, output_timeout=None, fatal_exit_code=2, error_level='error', **kwargs)[source]

Run a command, with logging and error parsing. TODO: context_lines

error_list example: [{‘regex’: re.compile(‘^Error: LOL J/K’), level=IGNORE},

{‘regex’: re.compile(‘^Error:’), level=ERROR, contextLines=‘5:5’}, {‘substr’: ‘THE WORLD IS ENDING’, level=FATAL, contextLines=‘20:’}

] (context_lines isn’t written yet)

Args:
command (str | list | tuple): command or sequence of commands to
execute and log.
cwd (str, optional): directory path from where to execute the
command. Defaults to None.
error_list (list, optional): list of errors to pass to
mozharness.base.log.OutputParser. Defaults to None.
halt_on_failure (bool, optional): whether or not to redefine the
log level as FATAL on errors. Defaults to False.
success_codes (int, optional): numeric value to compare against
the command return value.
env (dict, optional): key-value of environment values to use to
run the command. Defaults to None.
partial_env (dict, optional): key-value of environment values to
replace from the current environment values. Defaults to None.
return_type (str, optional): if equal to ‘num_errors’ then the
amount of errors matched by error_list is returned. Defaults to ‘status’.
throw_exception (bool, optional): whether or not to raise an
exception if the return value of the command doesn’t match any of the success_codes. Defaults to False.
output_parser (OutputParser, optional): lets you provide an
instance of your own OutputParser subclass. Defaults to OutputParser.
output_timeout (int): amount of seconds to wait for output before
the process is killed.
fatal_exit_code (int, optional): call self.fatal if the return value
of the command is not on in success_codes. Defaults to 2.
error_level (str, optional): log level name to use on error. Defaults
to ERROR.

**kwargs: Arbitrary keyword arguments.

Returns:
int: -1 on error. Any: command return value is returned otherwise.
script_obj = None
unpack(filename, extract_to)[source]

This method allows us to extract a file regardless of its extension

Args:
filename (str): filename of the compressed file. extract_to (str): where to extract the compressed file.
which(program)[source]

OS independent implementation of Unix’s which command

Args:
program (str): name or path to the program whose executable is
being searched.
Returns:
None: if the executable was not found. str: filepath of the executable file.
write_to_file(file_path, contents, verbose=True, open_mode='w', create_parent_dir=False, error_level='error')[source]

Write contents to file_path, according to open_mode.

Args:

file_path (str): filepath where the content will be written to. contents (str): content to write to the filepath. verbose (bool, optional): whether or not to log contents value.

Defaults to True
open_mode (str, optional): open mode to use for openning the file.
Defaults to w
create_parent_dir (bool, optional): whether or not to create the
parent directory of file_path

error_level (str, optional): log level to use on error. Defaults to ERROR

Returns:
str: file_path on success None: on error.
mozharness.base.script.platform_name()[source]

mozharness.base.signing module

Generic signing methods.

class mozharness.base.signing.AndroidSigningMixin[source]

Bases: object

Generic Android apk signing methods.

Dependent on BaseScript.

align_apk(unaligned_apk, aligned_apk, error_level='error')[source]

Zipalign apk. Returns None on success, not None on failure.

key_passphrase = None
passphrase()[source]
postflight_passphrase()[source]
sign_apk(apk, keystore, storepass, keypass, key_alias, remove_signature=True, error_list=None, log_level='info', error_level='error')[source]

Signs an apk with jarsigner.

store_passphrase = None
unsign_apk(apk, **kwargs)[source]
verify_passphrases()[source]
class mozharness.base.signing.BaseSigningMixin[source]

Bases: object

Generic signing helper methods.

query_filesize(file_path)[source]
query_sha512sum(file_path)[source]

mozharness.base.transfer module

Generic ways to upload + download files.

class mozharness.base.transfer.TransferMixin[source]

Bases: object

Generic transfer methods.

Dependent on BaseScript.

load_json_from_url(url, timeout=30, log_level='debug')[source]
rsync_download_directory(ssh_key, ssh_user, remote_host, remote_path, local_path, rsync_options=None, error_level='error')[source]

rsync+ssh the content of a remote directory to local_path

Returns:
None: on success
-1: if local_path is not a directory -3: rsync fails to download from the remote directory
rsync_upload_directory(local_path, ssh_key, ssh_user, remote_host, remote_path, rsync_options=None, error_level='error', create_remote_directory=True)[source]

Create a remote directory and upload the contents of a local directory to it via rsync+ssh.

Returns:
None: on success

-1: if local_path is not a directory -2: if the remote_directory cannot be created

(it only makes sense if create_remote_directory is True)

-3: rsync fails to copy to the remote directory

Module contents