This is a technical blog and describes the technical details behind the GSoC project. In case you wish to see a developer’s guide to use this feature only, kindly refer to this. Lastly, there is a post with a list of all the PRs in the project.

Treeherder

Since the “Add New Jobs” feature had been done earlier for BuildBot jobs, my major goal has been to create a parallel approach for TaskCluster jobs.

Unlike BuildBot jobs, TaskCluster jobs do not have a buildername property which can be used to uniquely identify a certain job. However, Dustin’s big-graph changes introduced full-task-graph.json, which has the magical TaskLabel. A typical task label looks like this - TaskLabel==A1gDl4RUQriAf28c6gkuCw.

Now task labels uniquely identify tasks in a push. They differ from task IDs. So if a single push has two tasks which do the same work, they would have identical task labels but different task IDs. Task IDs are produced when tasks are actually created. Another difference is that task labels are consistent across pushes. Task IDs are not, and can uniquely identify a particular task in a particular push.

So now on pressing “Add New Jobs”, I pass the decision task id (only if it’s complete) to the Django API. The API downloads the full-task-graph.json file and sends back a list having both BuildBot jobs and TaskCluster jobs. TaskLabel==<...> is used as a ref_name to uniquely identify jobs.

Hence, the code for BuildBot is reused without further additions. However, two major changes have been made in the resulting pulse messages. We are passing down the Decision Task ID and a timestamp along with the list of requested jobs’ ref_names, which would be buildername for BB and TaskLabel==<...> for TaskCluster.

TaskCluster jobs only show on try pushes due to a UI restriction imposed. Bug 1286894 has been filed to decide how to implement this feature on other repositories.

You can go ahead and try out the “Add New Jobs” feature. You can see the Pulse messages on this exchange.

Pulse Actions / MozCI

Like before, Pulse Action continues to listen to these pulse messages generated by Treeherder. It now separates TaskCluster job requests by checking whether the ref_name starts with TaskLabel== or not.

TaskCluster jobs are then separated. MozCI reads the list of task labels and the Decision Task ID. It downloads action.yml a file which is YAML representation of Action Tasks (explained in next section). and are appropriately replaced with the Decision Task ID received in the Pulse Message and a comma separated list of requested task labels.

This is converted to a JSON task and scheduled using python TaskCluster client. The resulting task shows up on Treeherder as an Action Task.

TaskCluster

This describes the technical process in-tree, which enable the scheduling of Action Tasks.

The major change here is the addition of a mach command, mach taskgraph action-task. This command accepts two parameters, decision-id and task-labels.

This command is parsed and processed in taskcluster/takgraph/action.py. The decision task ID is used to download the full-task-graph file along with the list of exisiting tasks.

Based on the set of task labels requested, the full-task-graph is simplified to leave just the nodes requested along with its dependencies. After this, the graph goes through an optimization procedure where all nodes present in the original push (procured from the task-to-label file in the decision task) are removed unless specially requested by the user.

Once the graph is optimized, the dependencies are replaced with appropriate task IDs and TaskCluster schedules this graph in the same push. The requested jobs then show up on Treeherder.