gecko_taskgraph.optimize package¶

Submodules¶

gecko_taskgraph.optimize.backstop module¶

class gecko_taskgraph.optimize.backstop.SkipUnlessBackstop¶

Bases: gecko_taskgraph.optimize.OptimizationStrategy

Always removes tasks except on backstop pushes.

should_remove_task(task, params, _)¶: Determine whether to optimize this task by removing it. Returns True to remove.

class gecko_taskgraph.optimize.backstop.SkipUnlessPushInterval(push_interval, remove_on_projects=None)¶

Bases: gecko_taskgraph.optimize.OptimizationStrategy

Always removes tasks except every N pushes.

Parameters: push_interval (int) – Number of pushes

property description¶

should_remove_task(task, params, _)¶: Determine whether to optimize this task by removing it. Returns True to remove.

gecko_taskgraph.optimize.bugbug module¶

class gecko_taskgraph.optimize.bugbug.BugBugPushSchedules(confidence_threshold, tasks_only=False, use_reduced_tasks=False, fallback=None, num_pushes=1, select_configs=False)¶

Bases: gecko_taskgraph.optimize.OptimizationStrategy

Query the ‘bugbug’ service to retrieve relevant tasks and manifests.

Parameters

confidence_threshold (float) – The minimum confidence threshold (in range [0, 1]) needed for a task to be scheduled.
tasks_only (bool) – Whether or not to only use tasks and no groups (default: False)
use_reduced_tasks (bool) – Whether or not to use the reduced set of tasks provided by the bugbug service (default: False).
fallback (str) – The fallback strategy to use if there was a failure in bugbug (default: None)
num_pushes (int) – The number of pushes to consider for the selection (default: 1).
select_configs (bool) – Whether to select configurations for manifests too (default: False).

should_remove_task(task, params, importance)¶: Determine whether to optimize this task by removing it. Returns True to remove.

class gecko_taskgraph.optimize.bugbug.DisperseGroups(target_counts=None, unseen_modifier=1)¶

Bases: gecko_taskgraph.optimize.OptimizationStrategy

Disperse groups across test configs.

Each task has an associated ‘importance’ dict passed in via the arg. This is of the form {<group>: <importance>}.

Where ‘group’ is a test group id (usually a path to a manifest), and ‘importance’ is one of {‘lowest’, ‘low’, ‘medium’, ‘high’}.

Each importance value has an associated ‘count’ as defined in self.target_counts. It guarantees that ‘manifest’ will run in at least ‘count’ different configurations (assuming there are enough tasks containing ‘manifest’).

On configurations that haven’t been seen before, we’ll increase the target count by self.unseen_modifier to increase the likelihood of scheduling a task on that configuration.

Parameters

target_counts (dict) – Override DEFAULT_TARGET_COUNTS with custom counts. This is a dict mapping the importance value (‘lowest’, ‘low’, etc) to the minimum number of configurations manifests with this value should run on.
unseen_modifier (int) – Override DEFAULT_UNSEEN_MODIFIER to a custom value. This is the amount we’ll increase ‘target_count’ by for unseen configurations.

DEFAULT_TARGET_COUNTS = {'high': 3, 'low': 1, 'lowest': 0, 'medium': 2}¶

DEFAULT_UNSEEN_MODIFIER = 1¶

should_remove_task(task, params, importance)¶: Determine whether to optimize this task by removing it. Returns True to remove.

class gecko_taskgraph.optimize.bugbug.SkipUnlessDebug¶

Bases: gecko_taskgraph.optimize.OptimizationStrategy

Only run debug platforms.

should_remove_task(task, params, arg)¶: Determine whether to optimize this task by removing it. Returns True to remove.

gecko_taskgraph.optimize.bugbug.merge_bugbug_replies(data, new_data)¶: Merge a bugbug reply (stored in the new_data argument) into another (stored in the data argument).

gecko_taskgraph.optimize.schema module¶

gecko_taskgraph.optimize.schema.set_optimization_schema(schema_tuple)¶

Sets OptimizationSchema so it can be imported by the task transform. This function is called by projects that extend Firefox’s taskgraph. It should be called by the project’s taskgraph:register function before any transport or job runner code is imported.

Parameters: schema_tuple (tuple) – Tuple of possible optimization strategies

gecko_taskgraph.optimize.strategies module¶

class gecko_taskgraph.optimize.strategies.IndexSearch¶

Bases: gecko_taskgraph.optimize.OptimizationStrategy

fmt = '%Y-%m-%dT%H:%M:%S.%fZ'¶

should_replace_task(task, params, deadline, index_paths)¶: Look for a task with one of the given index paths

class gecko_taskgraph.optimize.strategies.SkipUnlessChanged¶

Bases: gecko_taskgraph.optimize.OptimizationStrategy

should_remove_task(task, params, file_patterns)¶: Determine whether to optimize this task by removing it. Returns True to remove.

class gecko_taskgraph.optimize.strategies.SkipUnlessHasRelevantTests¶

Bases: gecko_taskgraph.optimize.OptimizationStrategy

Optimizes tasks that don’t run any tests that were in child directories of a modified file.

get_changed_dirs(repo, rev)¶

should_remove_task(task, params, _)¶: Determine whether to optimize this task by removing it. Returns True to remove.

class gecko_taskgraph.optimize.strategies.SkipUnlessSchedules¶

Bases: gecko_taskgraph.optimize.OptimizationStrategy

scheduled_by_push(repository, revision)¶

should_remove_task(task, params, conditions)¶: Determine whether to optimize this task by removing it. Returns True to remove.

Module contents¶

The objective of optimization is to remove as many tasks from the graph as possible, as efficiently as possible, thereby delivering useful results as quickly as possible. For example, ideally if only a test script is modified in a push, then the resulting graph contains only the corresponding test suite task.

See taskcluster/docs/optimization.rst for more information.

class gecko_taskgraph.optimize.Alias(strategy)¶

Bases: gecko_taskgraph.optimize.CompositeStrategy

Provides an alias to an existing strategy.

This can be useful to swap strategies in and out without needing to modify the task transforms.

property description¶: A textual description of the combined substrategies.

reduce(results)¶: Given all substrategy results as a generator, return the overall result.

class gecko_taskgraph.optimize.All(*substrategies, **kwargs)¶

Bases: gecko_taskgraph.optimize.CompositeStrategy

Given one or more optimization strategies, remove or replace a task if all of them says to.

Replacement will use the value returned by the first strategy passed in. Note the values used for replacement need not be the same, as long as they all say to replace.

property description¶: A textual description of the combined substrategies.

classmethod reduce(results)¶: Given all substrategy results as a generator, return the overall result.

class gecko_taskgraph.optimize.Always¶

Bases: gecko_taskgraph.optimize.OptimizationStrategy

should_remove_task(task, params, arg)¶: Determine whether to optimize this task by removing it. Returns True to remove.

class gecko_taskgraph.optimize.Any(*substrategies, **kwargs)¶

Bases: gecko_taskgraph.optimize.CompositeStrategy

Given one or more optimization strategies, remove or replace a task if any of them says to.

Replacement will use the value returned by the first strategy that says to replace.

property description¶: A textual description of the combined substrategies.

classmethod reduce(results)¶: Given all substrategy results as a generator, return the overall result.

class gecko_taskgraph.optimize.CompositeStrategy(*substrategies, **kwargs)¶

Bases: gecko_taskgraph.optimize.OptimizationStrategy

abstract property description¶: A textual description of the combined substrategies.

abstract reduce(results)¶: Given all substrategy results as a generator, return the overall result.

should_remove_task(*args)¶: Determine whether to optimize this task by removing it. Returns True to remove.

should_replace_task(*args)¶: Determine whether to optimize this task by replacing it. Returns a taskId to replace this task, True to replace with nothing, or False to keep the task.

class gecko_taskgraph.optimize.ExperimentalOverride(base, overrides)¶

Bases: object

Overrides dictionaries that are stored in a container with new values.

This can be used to modify all strategies in a collection the same way, presumably with strategies affecting kinds of tasks tangential to the current context.

Parameters

base (object) – A container class supporting attribute access.
overrides (dict) – Values to update any accessed dictionaries with.

class gecko_taskgraph.optimize.Not(strategy)¶

Bases: gecko_taskgraph.optimize.CompositeStrategy

Given a strategy, returns the opposite.

property description¶: A textual description of the combined substrategies.

reduce(results)¶: Given all substrategy results as a generator, return the overall result.

class gecko_taskgraph.optimize.OptimizationStrategy¶

Bases: object

should_remove_task(task, params, arg)¶: Determine whether to optimize this task by removing it. Returns True to remove.

should_replace_task(task, params, deadline, arg)¶: Determine whether to optimize this task by replacing it. Returns a taskId to replace this task, True to replace with nothing, or False to keep the task.

class gecko_taskgraph.optimize.experimental¶

Bases: object

Experimental strategies either under development or used as benchmarks.

These run as “shadow-schedulers” on each autoland push (tier 3) and/or can be used with ./mach try auto. E.g:

./mach try auto –strategy relevant_tests

bugbug_debug_disperse = {'test': <gecko_taskgraph.optimize.Any object>}¶: Restricts tests to debug platforms.

bugbug_disperse_high = {'test': <gecko_taskgraph.optimize.Any object>}¶: Disperse tests across platforms, high confidence threshold.

bugbug_disperse_low = {'test': <gecko_taskgraph.optimize.Any object>}¶: Disperse tests across platforms, low confidence threshold.

bugbug_disperse_medium = {'test': <gecko_taskgraph.optimize.Any object>}¶: Disperse tests across platforms, medium confidence threshold.

bugbug_disperse_medium_no_unseen = {'test': <gecko_taskgraph.optimize.Any object>}¶: Disperse tests across platforms (no modified for unseen configurations), medium confidence threshold.

bugbug_disperse_medium_only_one = {'test': <gecko_taskgraph.optimize.Any object>}¶: Disperse tests across platforms (one platform per group), medium confidence threshold.

bugbug_disperse_reduced_medium = {'test': <gecko_taskgraph.optimize.Any object>}¶: Disperse tests across platforms, medium confidence threshold with reduced tasks.

bugbug_reduced = {'test': <gecko_taskgraph.optimize.Any object>}¶: Use the reduced set of tasks (and no groups) chosen by bugbug.

bugbug_reduced_high = {'test': <gecko_taskgraph.optimize.Any object>}¶: Use the reduced set of tasks (and no groups) chosen by bugbug, high confidence threshold.

bugbug_reduced_manifests_config_selection_low = {'test': <gecko_taskgraph.optimize.Any object>}¶: Choose configs selected by bugbug, low confidence threshold with reduced tasks.

bugbug_reduced_manifests_config_selection_medium = {'test': <gecko_taskgraph.optimize.Any object>}¶: Choose configs selected by bugbug, medium confidence threshold with reduced tasks.

bugbug_tasks_high = {'test': <gecko_taskgraph.optimize.Any object>}¶: Doesn’t limit platforms, high confidence threshold.

bugbug_tasks_medium = {'test': <gecko_taskgraph.optimize.Any object>}¶: Doesn’t limit platforms, medium confidence threshold.

relevant_tests = {'test': <gecko_taskgraph.optimize.Any object>}¶: Runs task containing tests in the same directories as modified files.

gecko_taskgraph.optimize.get_subgraph(target_task_graph, removed_tasks, replaced_tasks, label_to_taskid, decision_task_id)¶

Return the subgraph of target_task_graph consisting only of non-optimized tasks and edges between them.

To avoid losing track of taskIds for tasks optimized away, this method simultaneously substitutes real taskIds for task labels in the graph, and populates each task definition’s dependencies key with the appropriate taskIds. Task references are resolved in the process.

gecko_taskgraph.optimize.optimize_task_graph(target_task_graph, requested_tasks, params, do_not_optimize, decision_task_id, existing_tasks=None, strategy_override=None)¶: Perform task optimization, returning a taskgraph and a map from label to assigned taskId, including replacement tasks.

class gecko_taskgraph.optimize.project¶

Bases: object

Strategies that should be applied per-project.

autoland = {'build': <gecko_taskgraph.optimize.All object>, 'test': <gecko_taskgraph.optimize.Any object>}¶: Strategy overrides that apply to autoland.

gecko_taskgraph.optimize.register_strategy(name, args=())¶

gecko_taskgraph.optimize.remove_tasks(target_task_graph, requested_tasks, params, optimizations, do_not_optimize)¶: Implement the “Removing Tasks” phase, returning a set of task labels of all removed tasks.

gecko_taskgraph.optimize.replace_tasks(target_task_graph, params, optimizations, do_not_optimize, label_to_taskid, removed_tasks, existing_tasks)¶: Implement the “Replacing Tasks” phase, returning a set of task labels of all replaced tasks. The replacement taskIds are added to label_to_taskid as a side-effect.

gecko_taskgraph.optimize.split_bugbug_arg(arg, substrategies)¶

Split args for bugbug based strategies.

Many bugbug based optimizations require passing an empty dict by reference to communicate to downstream strategies. This function passes the provided arg to the first (non bugbug) strategies and a shared empty dict to the bugbug strategy and all substrategies after it.