mozharness.mozilla.testing package

Submodules

mozharness.mozilla.testing.device module

Interact with a device via ADB or SUT.

This code is largely from https://hg.mozilla.org/build/tools/file/default/sut_tools

class mozharness.mozilla.testing.device.ADBDeviceHandler(**kwargs)[source]

Bases: mozharness.mozilla.testing.device.BaseDeviceHandler

check_device()[source]
cleanup_device(reboot=False)[source]
connect_device()[source]
disconnect_device()[source]
install_app(file_path)[source]
ping_device(auto_connect=False, silent=False)[source]
query_device_exe(exe_name)[source]
query_device_file_exists(file_name)[source]
query_device_id(auto_connect=True)[source]
query_device_root(silent=False)[source]
query_device_time()[source]
reboot_device()[source]
remove_device_root(error_level='error')[source]
remove_etc_hosts(hosts_file='/system/etc/hosts')[source]
set_device_time(device_time=None, error_level='error')[source]
uninstall_app(package_name, package_root='/data/data', error_level='error')[source]
wait_for_device(interval=60, max_attempts=20)[source]
class mozharness.mozilla.testing.device.BaseDeviceHandler(log_obj=None, config=None, script_obj=None)[source]

Bases: mozharness.base.script.ScriptMixin, mozharness.base.log.LogMixin

add_device_flag(flag)[source]
check_device()[source]
cleanup_device(reboot=False)[source]
default_port = None
device_flags = []
device_id = None
device_root = None
install_app(file_path)[source]
ping_device()[source]
query_device_id()[source]
query_device_root()[source]
query_download_filename(file_id=None)[source]
reboot_device()[source]
wait_for_device(interval=60, max_attempts=20)[source]
exception mozharness.mozilla.testing.device.DeviceException[source]

Bases: exceptions.Exception

class mozharness.mozilla.testing.device.DeviceMixin[source]

Bases: object

BaseScript mixin, designed to interface with the device.

check_device()[source]
cleanup_device(**kwargs)[source]
device_handler = None
device_root = None
install_app()[source]
query_device_handler()[source]
reboot_device()[source]
class mozharness.mozilla.testing.device.SUTDeviceHandler(**kwargs)[source]

Bases: mozharness.mozilla.testing.device.BaseDeviceHandler

check_device()[source]
cleanup_device(reboot=False)[source]
install_app(file_path)[source]
ping_device()[source]
query_device_root(strict=False)[source]
query_device_time()[source]
query_devicemanager()[source]
reboot_device()[source]
remove_etc_hosts(hosts_file='/system/etc/hosts')[source]
set_device_time()[source]
wait_for_device(interval=60, max_attempts=20)[source]
class mozharness.mozilla.testing.device.SUTDeviceMozdeviceMixin(**kwargs)[source]

Bases: mozharness.mozilla.testing.device.SUTDeviceHandler

This SUT device manager class makes calls through mozdevice (from mozbase) [1] directly rather than calling SUT tools.

[1] https://github.com/mozilla/mozbase/blob/master/mozdevice/mozdevice/devicemanagerSUT.py

dm = None
get_logcat()[source]
query_devicemanager()[source]
query_file(filename)[source]
set_device_epoch_time(timestamp=1440001627)[source]

mozharness.mozilla.testing.errors module

Mozilla error lists for running tests.

Error lists are used to parse output in mozharness.base.log.OutputParser.

Each line of output is matched against each substring or regular expression in the error list. On a match, we determine the ‘level’ of that line, whether IGNORE, DEBUG, INFO, WARNING, ERROR, CRITICAL, or FATAL.

mozharness.mozilla.testing.mozpool module

Interact with mozpool/lifeguard/bmm.

class mozharness.mozilla.testing.mozpool.MozpoolMixin[source]

Bases: object

determine_mozpool_host(device)[source]
mobile_imaging_format = 'http://mobile-imaging'
mozpool_handler = None
query_mozpool_handler(device=None, mozpool_api_url=None)[source]
retrieve_android_device(b2gbase)[source]
retrieve_b2g_device(b2gbase)[source]

mozharness.mozilla.testing.talos module

run talos tests in a virtualenv

class mozharness.mozilla.testing.talos.Talos(**kwargs)[source]

Bases: mozharness.mozilla.testing.testbase.TestingMixin, mozharness.base.vcs.vcsbase.MercurialScript, mozharness.mozilla.blob_upload.BlobUploadMixin

install and run Talos tests: https://wiki.mozilla.org/Buildbot/Talos

clone_talos()[source]
config_options = [[['--talos-url'], {'action': 'store', 'dest': 'talos_url', 'default': 'https://hg.mozilla.org/build/talos/archive/tip.tar.gz', 'help': 'Specify the talos package url'}], [['--use-talos-json'], {'action': 'store_true', 'dest': 'use_talos_json', 'default': False, 'help': 'Use talos config from talos.json'}], [['--suite'], {'action': 'store', 'dest': 'suite', 'help': 'Talos suite to run (from talos json)'}], [['--branch-name'], {'action': 'store', 'dest': 'branch', 'help': 'Graphserver branch to report to'}], [['--system-bits'], {'help': 'Testing 32 or 64 (for talos json plugins)', 'dest': 'system_bits', 'choices': ['32', '64'], 'default': '32', 'action': 'store', 'type': 'choice'}], [['--add-option'], {'action': 'extend', 'dest': 'talos_extra_options', 'default': None, 'help': 'extra options to talos'}], [['--spsProfile'], {'dest': 'sps_profile', 'action': 'store_true', 'default': False, 'help': 'Whether or not to profile the test run and save the profile results'}], [['--spsProfileInterval'], {'dest': 'sps_profile_interval', 'default': 0, 'type': 'int', 'help': 'The interval between samples taken by the profiler (milliseconds)'}], [['-a', '--tests'], {'action': 'extend', 'dest': 'tests', 'default': [], 'help': 'Specify the tests to run'}], [['--results-url'], {'action': 'store', 'dest': 'results_url', 'default': None, 'help': 'URL to send results to'}], [['--installer-url'], {'action': 'store', 'dest': 'installer_url', 'default': None, 'help': 'URL to the installer to install'}], [['--installer-path'], {'action': 'store', 'dest': 'installer_path', 'default': None, 'help': 'Path to the installer to install. This is set automatically if run with --download-and-extract.'}], [['--binary-path'], {'action': 'store', 'dest': 'binary_path', 'default': None, 'help': 'Path to installed binary. This is set automatically if run with --install.'}], [['--exe-suffix'], {'action': 'store', 'dest': 'exe_suffix', 'default': None, 'help': 'Executable suffix for binaries on this platform'}], [['--test-url'], {'action': 'store', 'dest': 'test_url', 'default': None, 'help': 'URL to the zip file containing the actual tests'}], [['--test-packages-url'], {'action': 'store', 'dest': 'test_packages_url', 'default': None, 'help': 'URL to a json file describing which tests archives to download'}], [['--jsshell-url'], {'action': 'store', 'dest': 'jsshell_url', 'default': None, 'help': 'URL to the jsshell to install'}], [['--download-symbols'], {'action': 'store', 'dest': 'download_symbols', 'type': 'choice', 'help': 'Download and extract crash reporter symbols.', 'choices': ['ondemand', 'true']}], [['--venv-path', '--virtualenv-path'], {'action': 'store', 'dest': 'virtualenv_path', 'default': 'venv', 'help': 'Specify the path to the virtualenv top level directory'}], [['--virtualenv'], {'action': 'store', 'dest': 'virtualenv', 'help': 'Specify the virtualenv executable to use'}], [['--find-links'], {'action': 'extend', 'dest': 'find_links', 'help': 'URL to look for packages at'}], [['--pip-index'], {'action': 'store_true', 'default': True, 'dest': 'pip_index', 'help': 'Use pip indexes (default)'}], [['--no-pip-index'], {'action': 'store_false', 'dest': 'pip_index', 'help': "Don't use pip indexes"}], [['--blob-upload-branch'], {'dest': 'blob_upload_branch', 'help': "Branch for blob server's metadata"}], [['--blob-upload-server'], {'dest': 'blob_upload_servers', 'action': 'extend', 'help': "Blob servers's location"}]]
create_virtualenv(**kwargs)[source]

VirtualenvMixin.create_virtualenv() assuemes we’re using self.config[‘virtualenv_modules’]. Since we are installing talos from its source, we have to wrap that method here.

download_talos_json()[source]
postflight_create_virtualenv()[source]

This belongs in download_and_install() but requires the virtualenv to be set up :(

The real fix here may be a –tpmanifest option for PerfConfigurator.

preflight_run_tests()[source]
query_abs_dirs()[source]
query_abs_pagesets_paths()[source]

Returns a bunch of absolute pagesets directory paths. We need this to make the dir and copy the manifest to the local dir.

query_pagesets_manifest_filename()[source]
query_pagesets_manifest_parent_path()[source]
query_pagesets_manifest_path()[source]

We have to copy the tp manifest from webroot to talos root when those two directories aren’t the same, until bug 795172 is fixed.

Helper method to avoid hardcodes.

query_pagesets_parent_dir_path()[source]

We have to copy the pageset into the webroot separately.

Helper method to avoid hardcodes.

query_pagesets_url()[source]

Certain suites require external pagesets to be downloaded and extracted.

query_sps_profile_options()[source]
query_talos_json_config()[source]

Return the talos json config; download and read from the talos_json_url if need be.

query_talos_json_url()[source]

Hacky, but I haven’t figured out a better way to get the talos json url before we install the build.

We can’t get this information after we install the build, because we have to create the virtualenv to use mozinstall, and talos_url is specified in the talos json.

query_talos_options()[source]
query_talos_repo()[source]

Where do we install the talos python package from? This needs to be overrideable by the talos json.

query_talos_revision()[source]

Which talos revision do we want to use? This needs to be overrideable by the talos json.

query_tests()[source]

Determine if we have tests to run.

Currently talos json will take precedence over config and command line options; if that’s not a good default we can switch the order.

run_tests(args=None, **kw)[source]

run Talos tests

talos_conf_path(conf)[source]

return the full path for a talos .yml configuration file

talos_options(args=None, **kw)[source]

return options to talos

class mozharness.mozilla.testing.talos.TalosOutputParser(config=None, log_obj=None, error_list=None, log_output=True)[source]

Bases: mozharness.base.log.OutputParser

minidump_output = None
minidump_regex = <_sre.SRE_Pattern object at 0x35b9d50>
parse_single_line(line)[source]

In Talos land, every line that starts with RETURN: needs to be printed with a TinderboxPrint:

worst_tbpl_status = 'SUCCESS'

mozharness.mozilla.testing.testbase module

class mozharness.mozilla.testing.testbase.TestingMixin(*args, **kwargs)[source]

Bases: mozharness.base.python.VirtualenvMixin, mozharness.mozilla.buildbot.BuildbotMixin, mozharness.base.python.ResourceMonitoringMixin, mozharness.mozilla.tooltool.TooltoolMixin, mozharness.mozilla.testing.try_tools.TryToolsMixin

The steps to identify + download the proper bits for [browser] unit tests and Talos.

binary_path = None
default_tools_repo = 'https://hg.mozilla.org/build/tools'
download_and_extract(target_unzip_dirs=None, suite_categories=None)[source]

download and extract test zip / download installer

download_file(*args, **kwargs)[source]

This function helps not to use download of proxied files since it does not support authenticated downloads. This could be re-factored and fixed in bug 1087664.

download_proxied_file(url, file_name=None, parent_dir=None, create_parent_dir=True, error_level='fatal', exit_code=3)[source]
get_test_output_parser(suite_category, strict=False, fallback_parser_class=<class 'mozharness.mozilla.testing.unittest.DesktopUnittestOutputParser'>, **kwargs)[source]

Derive and return an appropriate output parser, either the structured output parser or a fallback based on the type of logging in use as determined by configuration.

install()[source]
install_app(app=None, target_dir=None, installer_path=None)[source]

Dependent on mozinstall

installer_path = None
installer_url = None
jsshell_url = None
minidump_stackwalk_path = None
postflight_read_buildbot_config()[source]

Determine which files to download from the buildprops.json file created via the buildbot ScriptFactory.

postflight_run_tests()[source]

preflight commands for all tests

preflight_download_and_extract()[source]
preflight_install()[source]
preflight_run_tests()[source]

preflight commands for all tests

proxxy = None
query_build_dir_url(file_name)[source]

Resolve a file name to a potential url in the build upload directory where that file can be found.

query_minidump_filename()[source]
query_minidump_stackwalk()[source]
query_minidump_tooltool_manifest()[source]
query_symbols_url()[source]
query_value(key)[source]

This function allows us to check for a value in the self.tree_config first and then on self.config

structured_output(suite_category)[source]

Defines whether structured logging is in use in this configuration. This may need to be replaced with data from a different config at the resolution of bug 1070041 and related bugs.

symbols_path = None
symbols_url = None
test_packages_url = None
test_url = None
test_zip_path = None
tree_config = {}

mozharness.mozilla.testing.unittest module

class mozharness.mozilla.testing.unittest.DesktopUnittestOutputParser(suite_category, **kwargs)[source]

Bases: mozharness.base.log.OutputParser

A class that extends OutputParser such that it can parse the number of passed/failed/todo tests from the output.

append_tinderboxprint_line(suite_name)[source]
evaluate_parser(return_code, success_codes=None)[source]
parse_single_line(line)[source]
class mozharness.mozilla.testing.unittest.EmulatorMixin[source]

Bases: object

Currently dependent on both TooltoolMixin and TestingMixin)

install_emulator()[source]
install_emulator_from_tooltool(manifest_path, do_unzip=True)[source]
class mozharness.mozilla.testing.unittest.TestSummaryOutputParserHelper(regex=<_sre.SRE_Pattern object>, **kwargs)[source]

Bases: mozharness.base.log.OutputParser

evaluate_parser()[source]
parse_single_line(line)[source]
print_summary(suite_name)[source]
mozharness.mozilla.testing.unittest.tbox_print_summary(pass_count, fail_count, known_fail_count=None, crashed=False, leaked=False)[source]

Module contents