Sheriffing/How To/Beta simulations

Beta Simulations

Open the doc with the past beta simulations and known issues - Request the link and edit access in the sheriffs IRC channel if you are new to this.
Add the new day for which you’re making the simulation by copying the entry for the last day and modifying the date.

Go to mozilla-central, take the HG link from the latest central push and add it as a base revision in the document.

SILENCING KNOWN PERMAFAILS

If a permanent failure is known from previous simulations but not fixed according to the bug, it's helpful to prevent this failure to reduce the need for failure classifications and to provide a better picture when one looks at the whole beta simulation. These are the possibilities:

• Apply fix: If there is a patch in the bug about the perma failure - which hasn't landed on mozilla-central yet - , apply that similar to how patches are landed on inbound (DON'T PUSH)
• Fix on phabricator: If there is a fix hosted on phabricator, import it to migitate the issue. By default, the arc patch command will import all revisions a patch dependens on (has as anchestors). If it's a patch series, import the last patch to also auto-import the previous ones:
  

arc patch D<number> --nobranch
The patch number is mentioned at the top of the phabricator page for the patch.
If the dependencies should not be imported, e.g. because they are not needed, they can be skipped like this:
arc patch D<number> --nobranch --skip-dependencies

• Backout change causing failure: If the bug about the perma failure mentions the bug whose check-ins cause the failures, those can be backed out locally for the beta simulation:
  • hg oops -esr/er <revisions> -n <bug_number>//The revisions of the bug whose backout fixes the issue. The backout revisions will have to be deleted with hg histedit once the beta simulations have been created.
• Disable failing test: Similar to how frequently failing tests are disabled. Disadvantage: Additional failures or changes in the failure message in the test get missed.

TRUNK AS EARLY BETA

• Open the console and run the following commands:
  • hg pull central
  • hg update central
  • ./mach try release -v 68.0b1 --tasks release-sim --migration central-to-beta
    • -v 68.0b1: Sets the version number to use in the beta simulation. Replace "68" with the version number mention in the beta simulation document.
    • --tasks release-sim: Activates the tasks which shall run for beta simulations to get scheduled.
    • --migration central-to-beta: Activates modification of the configuration to switch from central to beta.

This changes the configuration to the beta simulation, pushes to the Try server and reverts the changes. Open the first Treeherder link in the console. It might take a few minutes for Treeherder to find that job, though. Add the link after you deselect the running and the green jobs and check that the classified jobs are visible, in the current date section at Run Links: Trunk as Early Beta in the Gdoc.

TRUNK AS LATE BETA

Go to the console and use the following commands:

• ./mach try release -v 67.0b12 --tasks release-sim --migration central-to-beta --migration early-to-late-beta

Note the different version number 67.0b12 compared the early beta's 67.0b1.
Add the link in the document after you deselect running and green jobs, in the current date section at RUN LINKS Trunk as Late Beta.
Also, in the New Bugs Section - you can write TBD (“To Be Done”) until the jobs on your try pushes have ran and you can see if there are failures or not.
NOTE: When creating a bug, add tracking only for permafails. Low frequent intermittents or frequent intermittents also affecting mozilla-central don’t need that flag but should be created and not be added to the Google Doc about the simulations.

When done, clean up the repository:

• hg purge: Removes the backup files generate when the command to generate the beta simulations modified configuration files.

After the beta simulations are done:

• Remove the “TBD” from the bug list for today’s simulations if there is no new failure and replace it with “None”
• If there were known issues before the beta simulations which you didn’t fix with a backout or patch import and which are gone, open those bugs (should be “Resolved Fixed”) and set the status_firefox<version number> to “verified” and change “Resolved Fixed” to “Verified Fixed”. Also add a comment that it has been verified fixed and add the link to today’s simulation.

VERSION INCREASE SIMULATION

Helpful tip: To have a more accessible command when pushing to try, open the .hgrc file and below the [alias] line add trymc = push -f ssh://hg.mozilla.org/try
To check if the next increase of the application version will cause failures, you need to open the gdoc and search for a previous Version Increase link.
In your console import the link from the Version increase link from the doc.

• hg import <link>
• hg trymc
• hg histedit to remove the commit for the version increase.