gecko_taskgraph package

Subpackages

Submodules

gecko_taskgraph.config module

class gecko_taskgraph.config.GraphConfig(config, root_dir)

Bases: object

register()

Add the project’s taskgraph directory to the python path, and register any extensions present.

property taskcluster_yml
gecko_taskgraph.config.load_graph_config(root_dir)
gecko_taskgraph.config.validate_graph_config(config)

gecko_taskgraph.create module

gecko_taskgraph.create.create_task(session, task_id, label, task_def)
gecko_taskgraph.create.create_tasks(graph_config, taskgraph, label_to_taskid, params, decision_task_id)

gecko_taskgraph.decision module

gecko_taskgraph.decision.full_task_graph_to_manifests_by_task(full_task_json)
gecko_taskgraph.decision.full_task_graph_to_runnable_jobs(full_task_json)
gecko_taskgraph.decision.get_decision_parameters(graph_config, options)

Load parameters from the command-line options for ‘taskgraph decision’. This also applies per-project parameters, based on the given project.

gecko_taskgraph.decision.get_existing_tasks(rebuild_kinds, parameters, graph_config)

Find the decision task corresponding to the on-push graph, and return a mapping of labels to task-ids from it. This will skip the kinds specificed by rebuild_kinds.

gecko_taskgraph.decision.read_artifact(filename)
gecko_taskgraph.decision.rename_artifact(src, dest)
gecko_taskgraph.decision.set_decision_indexes(decision_task_id, params, graph_config)
gecko_taskgraph.decision.set_try_config(parameters, task_config_file)
gecko_taskgraph.decision.taskgraph_decision(options, parameters=None)

Run the decision task. This function implements mach taskgraph decision, and is responsible for

  • processing decision task command-line options into parameters

  • running task-graph generation exactly the same way the other mach taskgraph commands do

  • generating a set of artifacts to memorialize the graph

  • calling TaskCluster APIs to create the graph

gecko_taskgraph.decision.try_syntax_from_message(message)

Parse the try syntax out of a commit message, returning ‘’ if none is found.

gecko_taskgraph.decision.try_task_config_schema = <Schema({'tasks': [<class 'str'>], 'browsertime': <class 'bool'>, 'chemspill-prio': <class 'bool'>, 'disable-pgo': <class 'bool'>, 'env': {<class 'str'>: <class 'str'>}, 'gecko-profile': <class 'bool'>, 'gecko-profile-interval': <class 'float'>, 'gecko-profile-entries': <class 'int'>, 'gecko-profile-features': <class 'str'>, 'gecko-profile-threads': <class 'str'>, 'perftest-options': <class 'object'>, 'optimize-strategies': <class 'str'>, 'rebuild': <class 'int'>, 'tasks-regex': {'include': Any(None, [<class 'str'>], msg=None), 'exclude': Any(None, [<class 'str'>], msg=None)}, 'use-artifact-builds': <class 'bool'>, 'worker-overrides': {<class 'str'>: <class 'str'>}, 'routes': [<class 'str'>]}, extra=PREVENT_EXTRA, required=False) object>

Schema for try_task_config.json files.

gecko_taskgraph.decision.write_artifact(filename, data)

gecko_taskgraph.docker module

gecko_taskgraph.docker.build_context(name, outputFile, args=None)

Build a context.tar for image with specified name.

gecko_taskgraph.docker.build_image(name, tag, args=None)

Build a Docker image of specified name.

Output from image building process will be printed to stdout.

gecko_taskgraph.docker.get_image_digest(image_name)
gecko_taskgraph.docker.load_image(url, imageName=None, imageTag=None)

Load docker image from URL as imageName:tag, if no imageName or tag is given it will use whatever is inside the zstd compressed tarball.

Returns an object with properties ‘image’, ‘tag’ and ‘layer’.

gecko_taskgraph.docker.load_image_by_name(image_name, tag=None)
gecko_taskgraph.docker.load_image_by_task_id(task_id, tag=None)

gecko_taskgraph.files_changed module

Support for optimizing tasks based on the set of files that have changed.

gecko_taskgraph.files_changed.check(params, file_patterns)

Determine whether any of the files changed in the indicated push to https://hg.mozilla.org match any of the given file patterns.

gecko_taskgraph.files_changed.get_changed_files(repository, revision)

Get the set of files changed in the push headed by the given revision. Responses are cached, so multiple calls with the same arguments are OK.

gecko_taskgraph.files_changed.get_locally_changed_files(repo)

gecko_taskgraph.filter_tasks module

gecko_taskgraph.filter_tasks.filter_target_tasks(graph, parameters, graph_config)

Proxy filter to use legacy target tasks code.

This should go away once target_tasks are converted to filters.

gecko_taskgraph.filter_tasks.filter_task(name)

Generator to declare a task filter function.

gecko_taskgraph.generator module

class gecko_taskgraph.generator.Kind(name: str, path: str, config: dict, graph_config: gecko_taskgraph.config.GraphConfig)

Bases: object

classmethod load(root_dir, graph_config, kind_name)
load_tasks(parameters, loaded_tasks, write_artifacts)
exception gecko_taskgraph.generator.KindNotFound

Bases: Exception

Raised when trying to load kind from a directory without a kind.yml.

class gecko_taskgraph.generator.TaskGraphGenerator(root_dir, parameters, decision_task_id='DECISION-TASK', write_artifacts=False)

Bases: object

The central controller for taskgraph. This handles all phases of graph generation. The task is generated from all of the kinds defined in subdirectories of the generator’s root directory.

Access to the results of this generation, as well as intermediate values at various phases of generation, is available via properties. This encourages the provision of all generation inputs at instance construction time.

property full_task_graph

the full task set, with edges representing dependencies.

@type: TaskGraph

Type

The full task graph

property full_task_set

all tasks defined by any kind (a graph without edges)

@type: TaskGraph

Type

The full task set

property graph_config

The configuration for this graph.

@type: TaskGraph

property label_to_taskid

A dictionary mapping task label to assigned taskId. This property helps in interpreting optimized_task_graph.

@type: dictionary

property morphed_task_graph

The optimized task graph, with any subsequent morphs applied. This graph will have the same meaning as the optimized task graph, but be in a form more palatable to TaskCluster.

@type: TaskGraph

property optimized_task_graph

The set of targetted tasks and all of their dependencies; tasks that have been optimized out are either omitted or replaced with a Task instance containing only a task_id.

@type: TaskGraph

property parameters

The properties used for this graph.

@type: Properties

property target_task_graph

The set of targetted tasks and all of their dependencies

@type: TaskGraph

property target_task_set

The set of targetted tasks (a graph without edges)

@type: TaskGraph

verify_attributes(all_tasks)
verify_kinds(kinds)
verify_parameters(parameters)
verify_run_using()
gecko_taskgraph.generator.load_tasks_for_kind(parameters, kind, root_dir=None)

Get all the tasks of a given kind.

This function is designed to be called from outside of taskgraph.

gecko_taskgraph.graph module

class gecko_taskgraph.graph.Graph(nodes, edges)

Bases: object

Generic representation of a directed acyclic graph with labeled edges connecting the nodes. Graph operations are implemented in a functional manner, so the data structure is immutable.

It permits at most one edge of a given name between any set of nodes. The graph is not checked for cycles, and methods may hang or otherwise fail if given a cyclic graph.

The nodes and edges attributes may be accessed in a read-only fashion. The nodes attribute is a set of node names, while edges is a set of (left, right, name) tuples representing an edge named name going from node left to node `right..

Return a dictionary mapping each node to a set of the nodes it links to (omitting edge names)

Return a two-level dictionary mapping each node to a dictionary mapping edge names to labels.

Return a dictionary mapping each node to a set of the nodes linking to it (omitting edge names)

transitive_closure(nodes, reverse=False)

Return the transitive closure of <nodes>: the graph containing all specified nodes as well as any nodes reachable from them, and any intervening edges.

If reverse is true, the “reachability” will be reversed and this will return the set of nodes that can reach the specified nodes.

Example

a ——> b ——> c

`——-> d

transitive_closure([b]).nodes == set([a, b]) transitive_closure([c]).nodes == set([c, b, a]) transitive_closure([c], reverse=True).nodes == set([c]) transitive_closure([b], reverse=True).nodes == set([b, c, d])

visit_postorder()

Generate a sequence of nodes in postorder, such that every node is visited after any nodes it links to.

Behavior is undefined (read: it will hang) if the graph contains a cycle.

visit_preorder()

Like visit_postorder, but in reverse: evrey node is visited before any nodes it links to.

gecko_taskgraph.main module

class gecko_taskgraph.main.Command(func, args, kwargs, defaults)

Bases: tuple

args

Alias for field number 1

defaults

Alias for field number 3

func

Alias for field number 0

kwargs

Alias for field number 2

gecko_taskgraph.main.action_callback(options)
gecko_taskgraph.main.argument(*args, **kwargs)
gecko_taskgraph.main.build_image(args)
gecko_taskgraph.main.command(*args, **kwargs)
gecko_taskgraph.main.create_parser()
gecko_taskgraph.main.decision(options)
gecko_taskgraph.main.dump_output(out, path=None, params_spec=None)
gecko_taskgraph.main.format_taskgraph(options, parameters, logfile=None)
gecko_taskgraph.main.format_taskgraph_json(taskgraph)
gecko_taskgraph.main.format_taskgraph_labels(taskgraph)
gecko_taskgraph.main.format_taskgraph_yaml(taskgraph)
gecko_taskgraph.main.generate_taskgraph(options, parameters, logdir)
gecko_taskgraph.main.get_filtered_taskgraph(taskgraph, tasksregex)

Filter all the tasks on basis of a regular expression and returns a new TaskGraph object

gecko_taskgraph.main.get_taskgraph_generator(root, parameters)

Helper function to make testing a little easier.

gecko_taskgraph.main.image_digest(args)
gecko_taskgraph.main.load_image(args)
gecko_taskgraph.main.main(args=['doc'])
gecko_taskgraph.main.setup_logging()
gecko_taskgraph.main.show_taskgraph(options)
gecko_taskgraph.main.test_action_callback(options)

gecko_taskgraph.morph module

Graph morphs are modifications to task-graphs that take place after the optimization phase.

These graph morphs are largely invisible to developers running ./mach locally, so they should be limited to changes that do not modify the meaning of the graph.

gecko_taskgraph.morph.add_eager_cache_index_tasks(taskgraph, label_to_taskid, parameters, graph_config, decision_task_id)

Some tasks (e.g. cached tasks) we want to exist in the index before they even run/complete. Our current use is to allow us to depend on an unfinished cached task in future pushes. This graph morph adds “eager-index tasks” that depend on the decision task and do the index insertions directly, which does not need to wait on the pointed at task to complete.

gecko_taskgraph.morph.add_index_tasks(taskgraph, label_to_taskid, parameters, graph_config, decision_task_id)

The TaskCluster queue only allows 10 routes on a task, but we have tasks with many more routes, for purposes of indexing. This graph morph adds “index tasks” that depend on such tasks and do the index insertions directly, avoiding the limits on task.routes.

gecko_taskgraph.morph.add_try_task_duplicates(taskgraph, label_to_taskid, parameters, graph_config, decision_task_id)
gecko_taskgraph.morph.amend_taskgraph(taskgraph, label_to_taskid, to_add)

Add the given tasks to the taskgraph, returning a new taskgraph

gecko_taskgraph.morph.derive_misc_task(target_task, purpose, image, taskgraph, label_to_taskid, parameters, graph_config, dependencies)

Create the shell of a task that depends on dependencies and on the given docker image.

gecko_taskgraph.morph.make_index_task(parent_task, taskgraph, label_to_taskid, parameters, graph_config, index_paths, index_rank, purpose, dependencies)
gecko_taskgraph.morph.morph(taskgraph, label_to_taskid, parameters, graph_config, decision_task_id)

Apply all morphs

gecko_taskgraph.parameters module

exception gecko_taskgraph.parameters.ParameterMismatch

Bases: Exception

Raised when a parameters.yml has extra or missing parameters.

class gecko_taskgraph.parameters.Parameters(strict=True, **kwargs)

Bases: mozbuild.util.ReadOnlyDict

An immutable dictionary with nicer KeyError messages on failure

check()
file_url(path, pretty=False)

Determine the VCS URL for viewing a file in the tree, suitable for viewing by a human.

Parameters

  • path (text_type) – The path, relative to the root of the repository.

  • pretty (bool) – Whether to return a link to a formatted version of the file, or the raw file version.

Return text_type

The URL displaying the given path.

static format_spec(spec)

Get a friendly identifier from a parameters specifier.

Parameters

spec (str) – Parameters specifier.

Returns

Name to identify parameters by.

Return type

str

property id
is_try()

Determine whether this graph is being built on a try project or for mach try fuzzy.

release_level()

Whether this is a staging release or not.

Return str

One of “production” or “staging”.

gecko_taskgraph.parameters.extend_parameters_schema(schema)

Extend the schema for parameters to include per-project configuration.

This should be called by the gecko_taskgraph.register function in the graph-configuration.

gecko_taskgraph.parameters.get_app_version(product_dir='browser')
gecko_taskgraph.parameters.get_contents(path)
gecko_taskgraph.parameters.get_head_ref()
gecko_taskgraph.parameters.get_version(product_dir='browser')
gecko_taskgraph.parameters.load_parameters_file(spec, strict=True, overrides=None, trust_domain=None)

Load parameters from a path, url, decision task-id or project.

Examples

task-id=fdtgsD5DQUmAQZEaGMvQ4Q project=mozilla-central

gecko_taskgraph.parameters.parameters_loader(spec, strict=True, overrides=None)

gecko_taskgraph.target_tasks module

gecko_taskgraph.target_tasks.accept_raptor_android_build(platform)

Helper function for selecting the correct android raptor builds.

gecko_taskgraph.target_tasks.filter_by_regex(task_label, regexes, mode='include')

Filters tasks according to a list of pre-compiled reguar expressions.

If mode is “include”, a task label must match any regex to pass. If it is “exclude”, a task label must _not_ match any regex to pass.

gecko_taskgraph.target_tasks.filter_by_uncommon_try_tasks(task, optional_filters=None)

Filters tasks that should not be commonly run on try.

Parameters

  • task (str) – String representing the task name.

  • optional_filters (list, optional) – Additional filters to apply to task filtering.

Returns

True if task does not match any known filters.

False otherwise.

Return type

(Boolean)

gecko_taskgraph.target_tasks.filter_for_hg_branch(task, parameters)

Filter tasks by hg branch. If run_on_hg_branch is not defined, then task runs on all branches

gecko_taskgraph.target_tasks.filter_for_project(task, parameters)

Filter tasks by project. Optionally enable nightlies.

gecko_taskgraph.target_tasks.filter_on_platforms(task, platforms)

Filter tasks on the given platform

gecko_taskgraph.target_tasks.filter_out_cron(task, parameters)

Filter out tasks that run via cron.

gecko_taskgraph.target_tasks.filter_out_devedition(task, parameters)
gecko_taskgraph.target_tasks.filter_out_missing_signoffs(task, parameters)
gecko_taskgraph.target_tasks.filter_out_shippable(task)
gecko_taskgraph.target_tasks.filter_out_shipping_phase(task, parameters)
gecko_taskgraph.target_tasks.filter_release_tasks(task, parameters)
gecko_taskgraph.target_tasks.filter_tests_without_manifests(task, parameters)

Remove test tasks that have an empty ‘test_manifests’ attribute.

This situation can arise when the test loader (e.g bugbug) decided there weren’t any important manifests to run for the given push. We filter tasks out here rather than in the transforms so that the full task graph is still aware that the task exists (which is needed by the backfill action).

gecko_taskgraph.target_tasks.filter_unsupported_artifact_builds(task, parameters)
gecko_taskgraph.target_tasks.get_method(method)

Get a target_task_method to pass to a TaskGraphGenerator.

gecko_taskgraph.target_tasks.index_exists(index_path, reason='')
gecko_taskgraph.target_tasks.is_geckoview(task, parameters)
gecko_taskgraph.target_tasks.make_desktop_nightly_filter(platforms)

Returns a filter that gets all nightly tasks on the given platform.

gecko_taskgraph.target_tasks.standard_filter(task, parameters)
gecko_taskgraph.target_tasks.target_tasks_autoland(full_task_graph, parameters, graph_config)

In addition to doing the filtering by project that the ‘default’ filter does, also remove any tests running against shippable builds for non-backstop pushes.

gecko_taskgraph.target_tasks.target_tasks_bouncer_check(full_task_graph, parameters, graph_config)

Select the set of tasks required to perform bouncer version verification.

gecko_taskgraph.target_tasks.target_tasks_build_linux64_clang_trunk_perf(full_task_graph, parameters, graph_config)

Select tasks required to run perf test on linux64 build with clang trunk

gecko_taskgraph.target_tasks.target_tasks_chromium_update(full_task_graph, parameters, graph_config)

Select tasks required for building latest chromium versions.

gecko_taskgraph.target_tasks.target_tasks_codereview(full_task_graph, parameters, graph_config)

Select all code review tasks needed to produce a report

gecko_taskgraph.target_tasks.target_tasks_condprof(full_task_graph, parameters, graph_config)

Select tasks required for building conditioned profiles.

gecko_taskgraph.target_tasks.target_tasks_customv8_update(full_task_graph, parameters, graph_config)

Select tasks required for building latest d8/v8 version.

gecko_taskgraph.target_tasks.target_tasks_daily_releases(full_task_graph, parameters, graph_config)

Select the set of tasks required to identify if we should release. If we determine that we should the task will communicate to ship-it to schedule the release itself.

gecko_taskgraph.target_tasks.target_tasks_default(full_task_graph, parameters, graph_config)

Target the tasks which have indicated they should be run on this project via the run_on_projects attributes.

gecko_taskgraph.target_tasks.target_tasks_file_update(full_task_graph, parameters, graph_config)

Select the set of tasks required to perform nightly in-tree file updates

gecko_taskgraph.target_tasks.target_tasks_general_perf_testing(full_task_graph, parameters, graph_config)

Select tasks required for running performance tests 3 times a week.

gecko_taskgraph.target_tasks.target_tasks_graphics(full_task_graph, parameters, graph_config)

In addition to doing the filtering by project that the ‘default’ filter does, also remove artifact builds because we have csets on the graphics branch that aren’t on the candidate branches of artifact builds

gecko_taskgraph.target_tasks.target_tasks_kaios(full_task_graph, parameters, graph_config)

The set of tasks to run for kaios integration

gecko_taskgraph.target_tasks.target_tasks_l10n_bump(full_task_graph, parameters, graph_config)

Select the set of tasks required to perform l10n bumping.

gecko_taskgraph.target_tasks.target_tasks_l10n_cross_channel(full_task_graph, parameters, graph_config)

Select the set of tasks required to run l10n cross-channel.

gecko_taskgraph.target_tasks.target_tasks_merge_automation(full_task_graph, parameters, graph_config)

Select the set of tasks required to perform repository merges.

gecko_taskgraph.target_tasks.target_tasks_mozilla_beta(full_task_graph, parameters, graph_config)

Select the set of tasks required for a promotable beta or release build of desktop, plus android CI. The candidates build process involves a pipeline of builds and signing, but does not include beetmover or balrog jobs.

gecko_taskgraph.target_tasks.target_tasks_mozilla_central(full_task_graph, parameters, graph_config)

In addition to doing the filtering by project that the ‘default’ filter does, also remove any tests running against regular (aka not shippable, asan, etc.) opt builds.

gecko_taskgraph.target_tasks.target_tasks_mozilla_esr91(full_task_graph, parameters, graph_config)

Select the set of tasks required for a promotable beta or release build of desktop, without android CI. The candidates build process involves a pipeline of builds and signing, but does not include beetmover or balrog jobs.

gecko_taskgraph.target_tasks.target_tasks_mozilla_release(full_task_graph, parameters, graph_config)

Select the set of tasks required for a promotable beta or release build of desktop, plus android CI. The candidates build process involves a pipeline of builds and signing, but does not include beetmover or balrog jobs.

gecko_taskgraph.target_tasks.target_tasks_nightly_asan(full_task_graph, parameters, graph_config)

Select the set of tasks required for a nightly build of asan. The nightly build process involves a pipeline of builds, signing, and, eventually, uploading the tasks to balrog.

gecko_taskgraph.target_tasks.target_tasks_nightly_desktop(full_task_graph, parameters, graph_config)

Select the set of tasks required for a nightly build of linux, mac, windows.

gecko_taskgraph.target_tasks.target_tasks_nightly_linux(full_task_graph, parameters, graph_config)

Select the set of tasks required for a nightly build of linux. The nightly build process involves a pipeline of builds, signing, and, eventually, uploading the tasks to balrog.

gecko_taskgraph.target_tasks.target_tasks_nightly_macosx(full_task_graph, parameters, graph_config)

Select the set of tasks required for a nightly build of macosx. The nightly build process involves a pipeline of builds, signing, and, eventually, uploading the tasks to balrog.

gecko_taskgraph.target_tasks.target_tasks_nightly_win32(full_task_graph, parameters, graph_config)

Select the set of tasks required for a nightly build of win32 and win64. The nightly build process involves a pipeline of builds, signing, and, eventually, uploading the tasks to balrog.

gecko_taskgraph.target_tasks.target_tasks_nightly_win64(full_task_graph, parameters, graph_config)

Select the set of tasks required for a nightly build of win32 and win64. The nightly build process involves a pipeline of builds, signing, and, eventually, uploading the tasks to balrog.

gecko_taskgraph.target_tasks.target_tasks_nightly_win64_aarch64(full_task_graph, parameters, graph_config)

Select the set of tasks required for a nightly build of win32 and win64. The nightly build process involves a pipeline of builds, signing, and, eventually, uploading the tasks to balrog.

gecko_taskgraph.target_tasks.target_tasks_nothing(full_task_graph, parameters, graph_config)

Select nothing, for DONTBUILD pushes

gecko_taskgraph.target_tasks.target_tasks_perftest(full_task_graph, parameters, graph_config)

Select perftest tasks we want to run daily

gecko_taskgraph.target_tasks.target_tasks_perftest_autoland(full_task_graph, parameters, graph_config)

Select perftest tasks we want to run daily

gecko_taskgraph.target_tasks.target_tasks_perftest_s7(full_task_graph, parameters, graph_config)

Select tasks required for running raptor page-load tests on geckoview against S7

gecko_taskgraph.target_tasks.target_tasks_pine(full_task_graph, parameters, graph_config)

Bug 1339179 - no mobile automation needed on pine

gecko_taskgraph.target_tasks.target_tasks_promote_desktop(full_task_graph, parameters, graph_config)

Select the superset of tasks required to promote a beta or release build of a desktop product. This should include all non-android mozilla_{beta,release} tasks, plus l10n, beetmover, balrog, etc.

gecko_taskgraph.target_tasks.target_tasks_push_desktop(full_task_graph, parameters, graph_config)

Select the set of tasks required to push a build of desktop to cdns. Previous build deps will be optimized out via action task.

gecko_taskgraph.target_tasks.target_tasks_raptor_tp6m(full_task_graph, parameters, graph_config)

Select tasks required for running raptor cold page-load tests on fenix and refbrow

gecko_taskgraph.target_tasks.target_tasks_release_simulation(full_task_graph, parameters, graph_config)

Select builds that would run on push on a release branch.

gecko_taskgraph.target_tasks.target_tasks_scriptworker_canary(full_task_graph, parameters, graph_config)

Select the set of tasks required to run scriptworker canaries.

gecko_taskgraph.target_tasks.target_tasks_searchfox(full_task_graph, parameters, graph_config)

Select tasks required for indexing Firefox for Searchfox web site each day

gecko_taskgraph.target_tasks.target_tasks_ship_desktop(full_task_graph, parameters, graph_config)

Select the set of tasks required to ship desktop. Previous build deps will be optimized out via action task.

gecko_taskgraph.target_tasks.target_tasks_ship_geckoview(full_task_graph, parameters, graph_config)

Select the set of tasks required to ship geckoview nightly. The nightly build process involves a pipeline of builds and an upload to maven.mozilla.org.

gecko_taskgraph.target_tasks.target_tasks_staging_release(full_task_graph, parameters, graph_config)

Select all builds that are part of releases.

gecko_taskgraph.target_tasks.target_tasks_system_symbols(full_task_graph, parameters, graph_config)

Select tasks for scraping and uploading system symbols.

gecko_taskgraph.target_tasks.target_tasks_try(full_task_graph, parameters, graph_config)
gecko_taskgraph.target_tasks.target_tasks_try_auto(full_task_graph, parameters, graph_config)

Target the tasks which have indicated they should be run on autoland (rather than try) via the run_on_projects attributes.

Should do the same thing as the default target tasks method.

gecko_taskgraph.target_tasks.target_tasks_try_select(full_task_graph, parameters, graph_config)
gecko_taskgraph.target_tasks.target_tasks_try_select_uncommon(full_task_graph, parameters, graph_config)
gecko_taskgraph.target_tasks.target_tasks_updatebot_cron(full_task_graph, parameters, graph_config)

Select tasks required to run Updatebot’s cron job

gecko_taskgraph.task module

class gecko_taskgraph.task.Task(kind, label, attributes, task, description='', optimization=None, dependencies=NOTHING, soft_dependencies=NOTHING, if_dependencies=NOTHING, release_artifacts=None)

Bases: object

Representation of a task in a TaskGraph. Each Task has, at creation:

  • kind: the name of the task kind

  • label; the label for this task

  • attributes: a dictionary of attributes for this task (used for filtering)

  • task: the task definition (JSON-able dictionary)

  • optimization: optimization to apply to the task (see gecko_taskgraph.optimize)

  • dependencies: tasks this one depends on, in the form {name: label}, for example {‘build’: ‘build-linux64/opt’, ‘docker-image’: ‘docker-image-desktop-test’}

  • soft_dependencies: tasks this one may depend on if they are available post optimisation. They are set as a list of tasks label.

  • if_dependencies: only run this task if at least one of these dependencies are present.

And later, as the task-graph processing proceeds:

  • task_id – TaskCluster taskId under which this task will be created

This class is just a convenience wrapper for the data type and managing display, comparison, serialization, etc. It has no functionality of its own.

classmethod from_json(task_dict)

Given a data structure as produced by taskgraph.to_json, re-construct the original Task object. This is used to “resume” the task-graph generation process, for example in Action tasks.

property name
to_json()

gecko_taskgraph.taskgraph module

class gecko_taskgraph.taskgraph.TaskGraph(tasks, graph)

Bases: object

Representation of a task graph.

A task graph is a combination of a Graph and a dictionary of tasks indexed by label. TaskGraph instances should be treated as immutable.

In the graph, tasks are said to “link to” their dependencies. Whereas tasks are “linked from” their dependents.

for_each_task(f, *args, **kwargs)
classmethod from_json(tasks_dict)

This code is used to generate the a TaskGraph using a dictionary which is representative of the TaskGraph.

to_json()

Return a JSON-able object representing the task graph, as documented

gecko_taskgraph.try_option_syntax module

class gecko_taskgraph.try_option_syntax.TryOptionSyntax(parameters, full_task_graph, graph_config)

Bases: object

find_all_attribute_suffixes(graph, prefix)
generate_test_tiers(full_task_graph)
handle_alias(test, all_tests)

Expand a test if its name refers to an alias, returning a list of test dictionaries cloned from the first (to maintain any metadata).

parse_build_types(build_types_arg, full_task_graph)
parse_jobs(jobs_arg)
parse_platforms(options, full_task_graph)
parse_test_chunks(all_tests, tests)

Test flags may include parameters to narrow down the number of chunks in a given push. We don’t model 1 chunk = 1 job in taskcluster so we must check each test flag to see if it is actually specifying a chunk.

parse_test_option(attr_name, test_arg, full_task_graph)

Parse a unittest (-u) or talos (-t) option, in the context of a full task graph containing available unittest_try_name or talos_try_name attributes. There are three cases:

  • test_arg is == ‘none’ (meaning an empty list)

  • test_arg is == ‘all’ (meaning use the list of jobs for that job type)

  • test_arg is comma string which needs to be parsed

parse_test_opts(input_str, all_platforms)

Parse testspec,testspec,.., where each testspec is a test name optionally followed by a list of test platforms or negated platforms in [].

No brackets indicates that tests should run on all platforms for which builds are available. If testspecs are provided, then each is treated, from left to right, as an instruction to include or (if negated) exclude a set of test platforms. A single spec may expand to multiple test platforms via UNITTEST_PLATFORM_PRETTY_NAMES. If the first test spec is negated, processing begins with the full set of available test platforms; otherwise, processing begins with an empty set of test platforms.

task_matches(task)
gecko_taskgraph.try_option_syntax.alias_contains(infix)
gecko_taskgraph.try_option_syntax.alias_matches(pattern)
gecko_taskgraph.try_option_syntax.alias_prefix(prefix)
gecko_taskgraph.try_option_syntax.escape_whitespace_in_brackets(input_str)

In tests you may restrict them by platform [] inside of the brackets whitespace may occur this is typically invalid shell syntax so we escape it with backslash sequences .

gecko_taskgraph.try_option_syntax.parse_message(message)
gecko_taskgraph.try_option_syntax.split_try_msg(message)

Module contents