mozbuild.vendor package

Submodules

mozbuild.vendor.host_base module

class mozbuild.vendor.host_base.BaseHost(manifest)

Bases: object

upstream_snapshot(revision)
upstream_tag(revision)

Temporarily clone the repo to get the latest tag and timestamp

mozbuild.vendor.host_github module

class mozbuild.vendor.host_github.GitHubHost(manifest)

Bases: mozbuild.vendor.host_base.BaseHost

upstream_commit(revision)

Query the github api for a git commit id and timestamp.

upstream_snapshot(revision)

mozbuild.vendor.host_gitlab module

class mozbuild.vendor.host_gitlab.GitLabHost(manifest)

Bases: mozbuild.vendor.host_base.BaseHost

upstream_commit(revision)

Query the gitlab api for a git commit id and timestamp.

upstream_snapshot(revision)

mozbuild.vendor.host_googlesource module

class mozbuild.vendor.host_googlesource.GoogleSourceHost(manifest)

Bases: mozbuild.vendor.host_base.BaseHost

upstream_commit(revision)

Query for a git commit and timestamp.

upstream_snapshot(revision)
upstream_tag()

Temporarily clone the repo to get the latest tag and timestamp

mozbuild.vendor.mach_commands module

mozbuild.vendor.mach_commands.check_modified_files(command_context)

Ensure that there aren’t any uncommitted changes to files in the working copy, since we’re going to change some state on the user.

mozbuild.vendor.mach_commands.vendor(command_context, library, revision, ignore_modified=False, check_for_update=False, add_to_exports=False, verify=False)

Vendor third-party dependencies into the source repository.

Vendoring rust and python can be done with ./mach vendor [rust/python]. Vendoring other libraries can be done with ./mach vendor [arguments] path/to/file.yaml

mozbuild.vendor.mach_commands.vendor_python(command_context, **kwargs)
mozbuild.vendor.mach_commands.vendor_rust(command_context, **kwargs)

mozbuild.vendor.moz_yaml module

class mozbuild.vendor.moz_yaml.License

Bases: object

Voluptuous validator which verifies the license(s) are valid as per our allow list.

exception mozbuild.vendor.moz_yaml.MozYamlVerifyError(filename, error)

Bases: Exception

mozbuild.vendor.moz_yaml.RE_FIELD(string, pos=0, endpos=9223372036854775807)

Scan through string looking for a match, and return a corresponding match object instance.

Return None if no position in the string matches.

mozbuild.vendor.moz_yaml.RE_SECTION(string, pos=0, endpos=9223372036854775807)

Scan through string looking for a match, and return a corresponding match object instance.

Return None if no position in the string matches.

class mozbuild.vendor.moz_yaml.UpdateActions

Bases: object

Voluptuous validator which verifies the update actions(s) are valid.

class mozbuild.vendor.moz_yaml.UpdatebotTasks

Bases: object

Voluptuous validator which verifies the updatebot task(s) are valid.

mozbuild.vendor.moz_yaml.VALID_SOURCE_HOSTS = ['gitlab', 'googlesource', 'github']

— # Third-Party Library Template # All fields are mandatory unless otherwise noted

# Version of this schema schema: 1

bugzilla:

# Bugzilla product and component for this directory and subdirectories product: product name component: component name

# Document the source of externally hosted code origin:

# Short name of the package/library name: name of the package

description: short (one line) description

# Full URL for the package’s homepage/etc # Usually different from repository url url: package’s homepage url

# Human-readable identifier for this version/release # Generally “version NNN”, “tag SSS”, “bookmark SSS” release: identifier

# Revision to pull in # Must be a long or short commit SHA (long preferred) revision: sha

# The package’s license, where possible using the mnemonic from # https://spdx.org/licenses/ # Multiple licenses can be specified (as a YAML list) # A “LICENSE” file must exist containing the full license text license: MPL-2.0

# If the package’s license is specified in a particular file, # this is the name of the file. # optional license-file: COPYING

# Configuration for automatic updating system. # optional updatebot:

# TODO: allow multiple users to be specified # Phabricator username for a maintainer of the library, used for assigning # reviewers maintainer-phab: tjr

# Bugzilla email address for a maintainer of the library, used for needinfos maintainer-bz: tom@mozilla.com

# Type of git reference (commit, tag) to track updates from. # If omitted, will default to tracking commits. tracking: commit

# The tasks that Updatebot can run. Only one of each task is currently permitted # optional tasks:

# Configuration for the automated vendoring system. # optional vendoring:

# Repository URL to vendor from # eg. https://github.com/kinetiknz/nestegg # Any repository host can be specified here, however initially we’ll only # support automated vendoring from selected sources. url: source url (generally repository clone url)

# Type of hosting for the upstream repository # Valid values are ‘gitlab’, ‘github’ source-hosting: gitlab

# Base directory of the location where the source files will live in-tree. # If omitted, will default to the location the moz.yaml file is in. vendor-directory: third_party/directory

# List of patch files to apply after vendoring. Applied in the order # specified, and alphabetically if globbing is used. Patches must apply # cleanly before changes are pushed # All patch files are implicitly added to the keep file list. # optional patches:

  • file

  • path/to/file

  • path/*.patch

# List of files that are not deleted while vendoring # Implicitly contains “moz.yaml”, any files referenced as patches # optional keep:

  • file

  • path/to/file

  • another/path

  • *.mozilla

# Files/paths that will not be vendored from source repository # Implicitly contains “.git”, and “.gitignore” # optional exclude:

  • file

  • path/to/file

  • another/path

  • docs

  • src/*.test

# Files/paths that will always be vendored, even if they would # otherwise be excluded by “exclude”. # optional include:

  • file

  • path/to/file

  • another/path

  • docs/LICENSE.*

# If neither “exclude” or “include” are set, all files will be vendored # Files/paths in “include” will always be vendored, even if excluded # eg. excluding “docs/” then including “docs/LICENSE” will vendor just the # LICENSE file from the docs directory

# All three file/path parameters (“keep”, “exclude”, and “include”) support # filenames, directory names, and globs/wildcards.

# Actions to take after updating. Applied in order. # The action subfield is required. It must be one of: # - copy-file # - move-dir # - replace-in-file # - delete-path # - run-script # Unless otherwise noted, all subfields of action are required. # # If the action is copy-file or move-dir: # from is the source file # to is the destination # # If the action is replace-in-file: # pattern is what in the file to search for. It is an exact strng match. # with is the string to replace it with. Accepts the special keyword # ‘{revision}’ for the commit we are updating to. # File is the file to replace it in. # # If the action is delete-path # path is the file or directory to recursively delete # # If the action is run-script: # script is the script to run # cwd is the directory the script should run with as its cwd # # Unless specified otherwise, all files/directories are relative to the # vendor-directory. If the vendor-directory is different from the # directory of the yaml file, the keyword ‘{yaml_dir}’ may be used # to make the path relative to that directory. # ‘run-script’ supports the addictional keyword {cwd} which, if used, # must only be used at the beginning of the path. # # optional update-actions:

  • action: copy-file from: include/vcs_version.h.in to: ‘{yaml_dir}/vcs_version.h’

  • action: replace-in-file pattern: ‘@VCS_TAG@’ with: ‘{revision}’ file: ‘{yaml_dir}/vcs_version.h’

  • action: delete-path path: ‘{yaml_dir}/config’

  • action: run-script script: ‘{cwd}/generate_sources.sh’ cwd: ‘{yaml_dir}’

mozbuild.vendor.moz_yaml.load_moz_yaml(filename, verify=True, require_license_file=True)

Loads and verifies the specified manifest.

mozbuild.vendor.moz_yaml.update_moz_yaml(filename, release, revision, verify=True, write=True)

Update origin:release and vendoring:revision without stripping comments or reordering fields.

mozbuild.vendor.rewrite_mozbuild module

exception mozbuild.vendor.rewrite_mozbuild.MozBuildRewriteException

Bases: Exception

mozbuild.vendor.rewrite_mozbuild.add_file_to_moz_build_file(normalized_filename_to_add, moz_yaml_dir=None, vendoring_dir=None)

This is the overall function. Given a filename, relative to the gecko root (aka normalized), we look for a moz.build file to add it to, look for the place in the moz.build file to add it, and then edit that moz.build file in-place.

It accepted two optional parameters. If one is specified they both must be. If a library is vendored in a separate place from the moz.yaml file, these parameters specify those two directories.

mozbuild.vendor.rewrite_mozbuild.assignment_node_to_source_filename_list(code, node)

If the list of filenames is not a list of constants (e.g. it’s a generated list) it’s (probably) infeasible to try and figure it out. At least we’re not going to try right now. Maybe in the future?

If this happens, we’ll return an empty list. The consequence of this is that we won’t be able to match a file against this list, so we may not be able to add it.

(But if the file matches a generated list, perhaps it will be included in the Sources list automatically?)

mozbuild.vendor.rewrite_mozbuild.ast_get_source_segment(code, node)
mozbuild.vendor.rewrite_mozbuild.edit_moz_build_file_to_add_file(normalized_mozbuild_filename, unnormalized_filename_to_add, unnormalized_list_of_files)

This function edits the moz.build file in-place

I had _really_ hoped to replace this whole damn thing with something that adds a node to the AST, dumps the AST out, and then runs black on the file but there are some issues:

  • third party moz.build files (or maybe all moz.build files) aren’t always run through black

  • dumping the ast out losing comments

mozbuild.vendor.rewrite_mozbuild.edit_moz_build_file_to_remove_file(normalized_mozbuild_filename, unnormalized_filename_to_remove)

This function edits the moz.build file in-place

mozbuild.vendor.rewrite_mozbuild.filenames_directory_is_in_filename_list(filename_normalized, list_of_normalized_filenames)

Given a normalized filename and a list of normalized filenames, first turn them into a containing directory, and a list of containing directories. Then test if the containing directory of the filename is in the list.

ex:

f = filenames_directory_is_in_filename_list f(“foo/bar/a.c”, [“foo/b.c”]) -> false f(“foo/bar/a.c”, [“foo/b.c”, “foo/bar/c.c”]) -> true f(“foo/bar/a.c”, [“foo/b.c”, “foo/bar/baz/d.c”]) -> false

mozbuild.vendor.rewrite_mozbuild.find_all_posible_assignments_from_filename(source_assignments, filename_normalized)

Given a list of source assignments and a normalized filename, narrow the list to assignments that contain a file whose directory matches the filename’s directory.

mozbuild.vendor.rewrite_mozbuild.get_all_mozbuild_filenames(gecko_root)

Find all the third party moz.build files in the gecko repo

mozbuild.vendor.rewrite_mozbuild.get_all_target_filenames_normalized(all_mozbuild_filenames_normalized)

Given a list of moz.build files, returns all the files listed in all the souce assignments in the file.

This function is only used for debug/testing purposes - there is no reason to call this as part of ‘the algorithm’

mozbuild.vendor.rewrite_mozbuild.get_attribute_label(node)
mozbuild.vendor.rewrite_mozbuild.get_closest_mozbuild_file(normalized_filename, moz_yaml_dir=None, vendoring_dir=None, all_mozbuild_filenames_normalized=None)

Returns the closest moz.build file in the directory tree to a normalized filename

mozbuild.vendor.rewrite_mozbuild.get_file_reference_modes(source_assignments)

Given a set of source assignments, this function traverses through the files references in those assignments to see if the files are referenced using absolute paths (relative to gecko root) or relative paths.

It will return all the modes that are seen.

mozbuild.vendor.rewrite_mozbuild.get_gecko_root()

Using __file__ as a base, find the gecko root

mozbuild.vendor.rewrite_mozbuild.get_mozbuild_file_search_order(normalized_filename, moz_yaml_dir=None, vendoring_dir=None, all_mozbuild_filenames_normalized=None)

Returns an ordered list of normalized moz.build filenames to consider for a given filename

normalized_filename: a source filename normalized to the gecko root moz_yaml_dir: the path from gecko_root to the moz.yaml file (which is the root of the

moz.build files)

moz_yaml_dir: the path to where the library’s source files are all_mozbuild_filenames_normalized: (optional) the list of all third-party moz.build files

If all_mozbuild_filenames_normalized is not specified, we look in the filesystem.

The list is built out of two distinct steps.

In Step 1 we will walk up a directory tree, looking for moz.build files. We append moz.build files in this order, preferring the lowest moz.build we find, then moving on to one in a higher directory. The directory we start in is a little complicated. We take the series of subdirectories between vendoring_dir and the file in question, and then append them to the moz.yaml directory.

Example

When moz_yaml directory != vendoring_directory:

moz_yaml_dir = foo/bar/ vendoring_dir = third_party/baz/ normalized_filename = third_party/baz/asm/arm/a.S starting_directory: foo/bar/asm/arm/

When moz_yaml directory == vendoring_directory

(In this case, these variables will actually be ‘None’ but the algorthm is the same) moz_yaml_dir = foo/bar/ vendoring_dir = foo/bar/ normalized_filename = foo/bar/asm/arm/a.S starting_directory: foo/bar/asm/arm/

In Step 2 we get a bit desparate. When the vendoring directory and the moz_yaml directory are not the same, there is no guarentee that the moz_yaml directory will adhere to the same directory structure as the vendoring directory. And indeed it doesn’t in some cases (e.g. libdav1d.) So in this situation we start at the root of the moz_yaml directory and walk downwards, adding _any_ moz.build file we encounter to the list. Later on (in all cases, not just moz_yaml_dir != vendoring_dir) we only consider a moz.build file if it has source files whose directory matches the normalized_filename, so this step, though desparate, is safe-ish and believe it or not has worked for some file additions.

mozbuild.vendor.rewrite_mozbuild.guess_best_assignment(source_assignments, filename_normalized)

Given several assignments, all of which contain the same directory as the filename, pick one we think is best and return its source-assignment-location.

We do this by looking at the filename itself (not just its directory) and picking the assignment which contains a filename with the longest matching prefix.

e.g: “foo/asm_neon.c” compared to [“foo/main.c”, “foo/all_utility.c”], [“foo/asm_arm.c”]

-> [“foo/asm_arm.c”] (match of foo/asm_)

mozbuild.vendor.rewrite_mozbuild.log(*args, **kwargs)
mozbuild.vendor.rewrite_mozbuild.mozbuild_file_to_source_assignments(normalized_mozbuild_filename, assignment_type)

Returns a dictionary of ‘source-assignment-location’ -> ‘normalized source filename list’ contained in the moz.build file specified

normalized_mozbuild_filename: the moz.build file to read

mozbuild.vendor.rewrite_mozbuild.node_to_name(code, node)
mozbuild.vendor.rewrite_mozbuild.node_to_readable_file_location(code, node, child_node=None)
mozbuild.vendor.rewrite_mozbuild.normalize_filename(normalized_mozbuild_filename, filename)
mozbuild.vendor.rewrite_mozbuild.remove_file_from_moz_build_file(normalized_filename_to_remove, moz_yaml_dir=None, vendoring_dir=None)

Given a filename, relative to the gecko root (aka normalized), we look for the nearest moz.build file, look in that file for the file, and then edit that moz.build file in-place.

mozbuild.vendor.rewrite_mozbuild.renormalize_filename(mode, moz_yaml_dir, vendoring_dir, normalized_mozbuild_filename, normalized_filename_to_act_on)
Edit the normalized_filename_to_act_on to either

  • Make it an absolute path from gecko root (if we’re in that mode)

  • Get a relative path from the vendoring directory to the yaml directory where the moz.build file is (If they are in separate directories)

mozbuild.vendor.rewrite_mozbuild.test_all_third_party_files(gecko_root, all_mozbuild_filenames_normalized)

Run the algorithm on every source file in a third party moz.build file and output the results

mozbuild.vendor.rewrite_mozbuild.try_to_match_target_file(all_mozbuild_filenames_normalized, target_filename_normalized)

Runs ‘the algorithm’ on a target file, and returns if the algorithm was successful

all_mozbuild_filenames_normalized: the list of all third-party moz.build files target_filename_normalized - the target filename, normalized to the gecko root

mozbuild.vendor.rewrite_mozbuild.unnormalize_filename(normalized_mozbuild_filename, normalized_filename)
mozbuild.vendor.rewrite_mozbuild.validate_directory_parameters(moz_yaml_dir, vendoring_dir)

mozbuild.vendor.vendor_manifest module

class mozbuild.vendor.vendor_manifest.VendorManifest(topsrcdir, settings, log_manager, topobjdir=None, mozconfig=<object object>, virtualenv_name=None)

Bases: mozbuild.base.MozbuildObject

clean_upstream()

Remove files we don’t want to import.

fetch_and_unpack(revision)

Fetch and unpack upstream source

get_source_host()
update_files(revision, yaml_file)
update_moz_build(vendoring_dir, moz_yaml_dir, add_to_exports)
update_yaml(yaml_file, revision, timestamp)
vendor(yaml_file, manifest, revision, check_for_update, add_to_exports)

mozbuild.vendor.vendor_python module

class mozbuild.vendor.vendor_python.VendorPython(topsrcdir, settings, log_manager, topobjdir=None, mozconfig=<object object>, virtualenv_name=None)

Bases: mozbuild.base.MozbuildObject

vendor(keep_extra_files=False)

mozbuild.vendor.vendor_rust module

class mozbuild.vendor.vendor_rust.VendorRust(topsrcdir, settings, log_manager, topobjdir=None, mozconfig=<object object>, virtualenv_name=None)

Bases: mozbuild.base.MozbuildObject

BUILDTIME_LICENSE_WHITELIST = {'BSD-3-Clause': ['bindgen', 'fuchsia-zircon', 'fuchsia-zircon-sys', 'fuchsia-cprng', 'glsl', 'instant']}
ICU4X_LICENSE_SHA256 = '02420cc1b4c26d9a3318d60fd57048d015831249a5b776a1ada75cd227e78630'
RUNTIME_LICENSE_FILE_PACKAGE_WHITELIST = {'deque': '6485b8ed310d3f0340bf1ad1f47645069ce4069dcc6bb46c7d5c6faf41de1fdb', 'fixed_decimal': '02420cc1b4c26d9a3318d60fd57048d015831249a5b776a1ada75cd227e78630', 'fuchsia-cprng': '03b114f53e6587a398931762ee11e2395bfdba252a329940e2c8c9e81813845b', 'icu': '02420cc1b4c26d9a3318d60fd57048d015831249a5b776a1ada75cd227e78630', 'icu_datetime': '02420cc1b4c26d9a3318d60fd57048d015831249a5b776a1ada75cd227e78630', 'icu_decimal': '02420cc1b4c26d9a3318d60fd57048d015831249a5b776a1ada75cd227e78630', 'icu_locale_canonicalizer': '02420cc1b4c26d9a3318d60fd57048d015831249a5b776a1ada75cd227e78630', 'icu_locid': '02420cc1b4c26d9a3318d60fd57048d015831249a5b776a1ada75cd227e78630', 'icu_locid_macros': '02420cc1b4c26d9a3318d60fd57048d015831249a5b776a1ada75cd227e78630', 'icu_plurals': '02420cc1b4c26d9a3318d60fd57048d015831249a5b776a1ada75cd227e78630', 'icu_provider': '02420cc1b4c26d9a3318d60fd57048d015831249a5b776a1ada75cd227e78630', 'icu_provider_macros': '02420cc1b4c26d9a3318d60fd57048d015831249a5b776a1ada75cd227e78630', 'icu_uniset': '02420cc1b4c26d9a3318d60fd57048d015831249a5b776a1ada75cd227e78630', 'litemap': '02420cc1b4c26d9a3318d60fd57048d015831249a5b776a1ada75cd227e78630', 'writeable': '02420cc1b4c26d9a3318d60fd57048d015831249a5b776a1ada75cd227e78630', 'yoke': '02420cc1b4c26d9a3318d60fd57048d015831249a5b776a1ada75cd227e78630', 'yoke-derive': '02420cc1b4c26d9a3318d60fd57048d015831249a5b776a1ada75cd227e78630'}
RUNTIME_LICENSE_PACKAGE_WHITELIST = {'BSD-2-Clause': ['arrayref', 'cloudabi', 'Inflector', 'mach', 'qlog'], 'BSD-3-Clause': []}
RUNTIME_LICENSE_WHITELIST = ['Apache-2.0', 'Apache-2.0 WITH LLVM-exception', 'CC0-1.0', 'ISC', 'MIT', 'MPL-2.0', 'Unlicense', 'Zlib']
check_cargo_version(cargo)

Ensure that cargo is new enough. cargo 1.42 fixed some issue with the vendor command. cargo 1.47 similarly did so for windows, but as of this writing is the current nightly, so we restrict this check only to the platform it’s actually required on

check_modified_files()

Ensure that there aren’t any uncommitted changes to files in the working copy, since we’re going to change some state on the user. Allow changes to Cargo.{toml,lock} since that’s likely to be a common use case.

check_openssl()

Set environment flags for building with openssl.

MacOS doesn’t include openssl, but the openssl-sys crate used by mach-vendor expects one of the system. It’s common to have one installed in /usr/local/opt/openssl by homebrew, but custom link flags are necessary to build against it.

get_cargo_path()
static runtime_license(package, license_string)

Cargo docs say: — https://doc.rust-lang.org/cargo/reference/manifest.html

This is an SPDX 2.1 license expression for this package. Currently crates.io will validate the license provided against a whitelist of known license and exception identifiers from the SPDX license list 2.4. Parentheses are not currently supported.

Multiple licenses can be separated with a /, although that usage is deprecated. Instead, use a license expression with AND and OR operators to get more explicit semantics. — But I have no idea how you can meaningfully AND licenses, so we will abort if that is detected. We’ll handle / and OR as equivalent and approve is any is in our approved list.

vendor(ignore_modified=False, build_peers_said_large_imports_were_ok=False)

Module contents