Writing new DAMP performance tests¶
See DAMP Performance tests for an overall description of our performance tests. Here, we will describe how to write a new test and register it to run in DAMP.
Note
This page contains the general documentation for writing DAMP tests. See also:
Performance test writing example for a practical example of creating a new test
Performance test writing tips for detailed tips on how to write a good and efficient test
Test location¶
Tests are located in testing/talos/talos/tests/devtools/addon/content/tests. You will find subfolders for panels already tested in DAMP (debugger, inspector, …) as well as other subfolders for tests not specific to a given panel (server, toolbox).
Tests are isolated in dedicated files. Some examples of tests:
Basic test¶
The basic skeleton of a test is:
const {
testSetup,
testTeardown,
SIMPLE_URL,
} = require("devtools/docs/head");
module.exports = async function() {
await testSetup(SIMPLE_URL);
// Run some measures here
await testTeardown();
};
always start the test by calling
testSetup(url), with theurlof the document to usealways end the test with
testTeardown()
Test documents¶
DevTools performance heavily depends on the document against which DevTools are opened. There are two “historical” documents you can use for tests for any panel:
“Simple”, an empty webpage. This one helps highlighting the load time of panels,
“Complicated”, a copy of bild.be, a German newspaper website. This allows us to examine the performance of the tools when inspecting complicated, big websites.
The URL of those documents are exposed by tests/head.js. The Simple page can be found at testing/talos/talos/tests/devtools/addon/content/pages/simple.html. The Complicated page is downloaded via tooltool automatically the first time you run the DAMP tests.
You can create also new test documents under testing/talos/talos/tests/devtools/addon/content/pages. See the pages in the custom subfolder for instance. If you create a document in pages/custom/mypanel/index.html, the URL of the document in your tests should be PAGES_BASE_URL + "custom/mypanel/index.html". The constant PAGES_BASE_URL is exposed by head.js.
Note that modifying any existing test document will most likely impact the baseline for existing tests.
Finally you can also create very simple test documents using data urls. Test documents don’t have to contain any specific markup or script to be valid DAMP test documents, so something as simple as testSetup("data:text/html,my test document"); is valid.
Test helpers¶
Helper methods have been extracted in shared modules:
tests/head.js for the most common ones
tests/{subfolder}/{subfolder}-helpers.js for folder-specific helpers (example)
To measure something which is not covered by an existing helper, you should use runTest, exposed by head.js.
module.exports = async function() {
await testSetup(SIMPLE_URL);
// Calling `runTest` will immediately start recording your action duration.
// You can execute any necessary setup action you don't want to record before calling it.
const test = runTest(`mypanel.mytest.mymeasure`);
await doSomeThings(); // <== Do an action you want to record here
// Once your action is completed, call `runTest` returned object's `done` method.
// It will automatically record the action duration and appear in PerfHerder as a new subtest.
// It also creates markers in the profiler so that you can better inspect this action in
// profiler.firefox.com.
test.done();
await testTeardown();
};
If your measure is not simply the time spent by an asynchronous call (for instance computing an average, counting things…) there is a lower level helper called logTestResult which will directly log a value. See this example.
Test runner¶
If you need to dive into the internals of the DAMP runner, most of the logic is in testing/talos/talos/tests/devtools/addon/content/damp.js.
How to name your test and register it?¶
If a new test file was created, it needs to be registered in the test suite. To register the new test, add it in damp-tests.js. This file acts as the manifest for the DAMP test suite.
If your are writing a test executing against Simple and Complicated documents, your test name will look like: (simple|complicated).${tool-name}.${test-name}.
So for our example, it would be simple.inspector.click and complicated.inspector.click.
For independent tests that don’t use the Simple or Complicated documents, the test name only needs to start with the tool name, if the test is specific to that tool
For the example, it would be inspector.click.
In general, the test name should try to match the path of the test file. As you can see in damp-tests.js this naming convention is not consistently followed. We have discrepancies for simple/complicated/custom tests, as well as for webconsole tests. This is largely for historical reasons.
How to run your new test?¶
You can run any performance test with this command:
./mach talos-test --activeTests damp --subtest ${your-test-name}
By default, it will run the test 25 times. In order to run it just once, do:
./mach talos-test --activeTests damp --subtest ${your-test-name} --cycles 1 --tppagecycles 1
--cycles controls the number of times Firefox is restarted
--tppagecycles defines the number of times we repeat the test after each Firefox start
Also, you can record a profile while running the test. To do that, execute:
./mach talos-test --activeTests damp --subtest ${your-test-name} --cycles 1 --tppagecycles 1 --gecko-profile --gecko-profile-entries 100000000
--gecko-profile enables the profiler
--gecko-profile-entries defines the profiler buffer size, which needs to be large while recording performance tests
Once it is done executing, the profile lives in a zip file you have to uncompress like this:
unzip testing/mozharness/build/blobber_upload_dir/profile_damp.zip
Then you have to open https://profiler.firefox.com/ and manually load the profile file that lives here: profile_damp/page_0_pagecycle_1/cycle_0.profile