workflows/check-by-name: Pin nixpkgs-check-by-name tool

Before this, the tool for CI would update when nixos-unstable updated,
which is kind of terrible because you don't know when it happens, and it
might break master.

In fact, the tooling _right now_ has a serious bug and shouldn't be used!

This PR addresses this by _pinning_ the tooling in Nixpkgs itself.

Updating the tooling now requires two PRs:
- The first PR to update the tooling source
- (wait for Hydra to build and publish it in nixos-unstable)
- The second PR to update the pinned tooling

In turn you know exactly when the changes are going to take effect.

This change however has additional benefits:
- It makes CI more reproducible, because it doesn't depend on the state
  of nixos-unstable anymore
- Updates to the tooling can be tested with the workflow itself,
  because PRs that update the pinned tool will be tested on the updated
  version
- CI gets a sizable speed boost, because there's no need to download and
  evaluate a channel anymore
- It makes it more realistic to move the source of the tool into a
  separate repository
- It removes the brittle branch-specific logic that was previously
  needed to ensure that release branches use their own version of the
  tooling.

.github/workflows/check-by-name.yml

@@ -92,7 +92,7 @@ jobs:
           echo "base=$base" >> "$GITHUB_ENV"
       - uses: cachix/install-nix-action@7ac1ec25491415c381d9b62f0657c7a028df52a7 # v24
       - name: Fetching the tool
-        run: pkgs/test/nixpkgs-check-by-name/scripts/fetch-tool.sh "$GITHUB_BASE_REF" result
+        run: pkgs/test/nixpkgs-check-by-name/scripts/fetch-pinned-tool.sh result
       - name: Running nixpkgs-check-by-name
         run: |
           if result/bin/nixpkgs-check-by-name --base "$base" .; then

pkgs/test/nixpkgs-check-by-name/scripts/README.md

@@ -1,6 +1,7 @@
 # CI-related Scripts
 
-This directory contains scripts used and related to the CI running the `pkgs/by-name` checks in Nixpkgs. See also the [CI GitHub Action](../../../../.github/workflows/check-by-name.yml).
+This directory contains scripts and files used and related to the CI running the `pkgs/by-name` checks in Nixpkgs.
+See also the [CI GitHub Action](../../../../.github/workflows/check-by-name.yml).
 
 ## `./run-local.sh BASE_BRANCH [REPOSITORY]`
 
@@ -15,12 +16,18 @@ Arguments:
 - `BASE_BRANCH`: The base branch to use, e.g. master or release-23.11
 - `REPOSITORY`: The repository to fetch the base branch from, defaults to https://github.com/NixOS/nixpkgs.git
 
-## `./fetch-tool.sh BASE_BRANCH OUTPUT_PATH`
+## `./update-pinned-tool.sh`
 
-Fetches the Hydra-prebuilt nixpkgs-check-by-name to use from the NixOS channel corresponding to the given base branch.
+Updates the pinned CI tool in [`./pinned-tool.json`](./pinned-tool.json) to the
+[latest version from the `nixos-unstable` channel](https://hydra.nixos.org/job/nixos/trunk-combined/nixpkgs.tests.nixpkgs-check-by-name.x86_64-linux)
+
+This script is called manually once the CI tooling needs to be updated.
+
+## `./fetch-pinned-tool.sh OUTPUT_PATH`
+
+Fetches the pinned tooling specified in [`./pinned-tool.json`](./pinned-tool.json).
 
 This script is used both by [`./run-local.sh`](#run-local-sh-base-branch-repository) and CI.
 
 Arguments:
-- `BASE_BRANCH`: The base branch to use, e.g. master or release-23.11
 - `OUTPUT_PATH`: The output symlink path for the tool

pkgs/test/nixpkgs-check-by-name/scripts/fetch-pinned-tool.sh

@@ -0,0 +1,30 @@
+#!/usr/bin/env bash
+# Try to not use nix-shell here to avoid fetching Nixpkgs,
+# especially since this is used in CI
+# The only dependency is `jq`, which in CI is implicitly available
+# And when run from ./run-local.sh is provided by that parent script
+
+set -o pipefail -o errexit -o nounset
+
+trace() { echo >&2 "$@"; }
+
+SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
+
+pin_file=$SCRIPT_DIR/pinned-tool.json
+
+if (( $# < 1 )); then
+    trace "Usage: $0 fetch OUTPUT_PATH"
+    trace "OUTPUT_PATH: The output symlink path for the tool"
+    exit 1
+fi
+output=$1
+
+trace "Reading $pin_file.. "
+rev=$(jq -r .rev "$SCRIPT_DIR"/pinned-tool.json)
+trace -e "Git revision is \e[34m$rev\e[0m"
+path=$(jq -r .path "$SCRIPT_DIR"/pinned-tool.json)
+trace "Tooling path is $path"
+
+trace -n "Fetching the prebuilt version of nixpkgs-check-by-name.. "
+nix-store --add-root "$output" -r "$path" >/dev/null
+realpath "$output"

pkgs/test/nixpkgs-check-by-name/scripts/fetch-tool.sh

@@ -1,47 +1,19 @@
 #!/usr/bin/env bash
-# Fetches the prebuilt nixpkgs-check-by-name to use from
-# the NixOS channel corresponding to the given base branch
-
-set -o pipefail -o errexit -o nounset
+# Legacy script to make CI work for the PR that replaces this
+# Needed due to `.github/workflows/check-by-name.yml` using `pull_request_target`,
+# which uses the workflow from the base branch, which still uses this script.
+# This file can be removed after the PR replacing it is merged.
 
 trace() { echo >&2 "$@"; }
 
 if (( $# < 2 )); then
     trace "Usage: $0 BASE_BRANCH OUTPUT_PATH"
-    trace "BASE_BRANCH: The base branch to use, e.g. master or release-23.11"
+    trace "BASE_BRANCH (unused): The base branch to use, e.g. master or release-23.11"
     trace "OUTPUT_PATH: The output symlink path for the tool"
     exit 1
 fi
-baseBranch=$1
 output=$2
 
-trace -n "Determining the channel to use for PR base branch $baseBranch.. "
-if [[ "$baseBranch" =~ ^(release|staging|staging-next)-([0-9][0-9]\.[0-9][0-9])$ ]]; then
-  # Use the release channel for all PRs to release-XX.YY, staging-XX.YY and staging-next-XX.YY
-  preferredChannel=nixos-${BASH_REMATCH[2]}
-else
-  # Use the nixos-unstable channel for all other PRs
-  preferredChannel=nixos-unstable
-fi
+SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
 
-# Check that the channel exists. It doesn't exist for fresh release branches
-if curl -fSs "https://channels.nixos.org/$preferredChannel"; then
-    channel=$preferredChannel
-    trace "$channel"
-else
-    # Fall back to nixos-unstable, makes sense for fresh release branches
-    channel=nixos-unstable
-    trace -e "\e[33mWarning: Preferred channel $preferredChannel could not be fetched, using fallback: $channel\e[0m"
-fi
-
-trace -n "Fetching latest version of channel $channel.. "
-# This is probably the easiest way to get Nix to output the path to a downloaded channel!
-nixpkgs=$(nix-instantiate --find-file nixpkgs -I nixpkgs=channel:"$channel")
-trace "$nixpkgs"
-
-# This file only exists in channels
-trace -e "Git revision of channel $channel is \e[34m$(<"$nixpkgs/.git-revision")\e[0m"
-
-trace -n "Fetching the prebuilt version of nixpkgs-check-by-name.. "
-nix-build -o "$output" "$nixpkgs" -A tests.nixpkgs-check-by-name -j 0 >/dev/null
-realpath "$output" >&2
+"$SCRIPT_DIR"/fetch-pinned-tool.sh "$output"

pkgs/test/nixpkgs-check-by-name/scripts/pinned-tool.json

@@ -0,0 +1,4 @@
+{
+  "rev": "9b19f5e77dd906cb52dade0b7bd280339d2a1f3d",
+  "path": "/nix/store/qlls5ca8q88qpyygg9ddi60gl1nmvpij-nixpkgs-check-by-name"
+}

pkgs/test/nixpkgs-check-by-name/scripts/run-local.sh

@@ -1,4 +1,5 @@
-#!/usr/bin/env bash
+#!/usr/bin/env nix-shell
+#!nix-shell -i bash -p jq
 
 set -o pipefail -o errexit -o nounset
 
@@ -61,7 +62,7 @@ trace -n "Merging base branch into the HEAD commit in $tmp/merged.. "
 git -C "$tmp/merged" merge -q --no-edit "$baseSha"
 trace -e "\e[34m$(git -C "$tmp/merged" rev-parse HEAD)\e[0m"
 
-"$tmp/merged/pkgs/test/nixpkgs-check-by-name/scripts/fetch-tool.sh" "$baseBranch" "$tmp/tool"
+"$tmp/merged/pkgs/test/nixpkgs-check-by-name/scripts/fetch-pinned-tool.sh" "$tmp/tool"
 
 trace "Running nixpkgs-check-by-name.."
 "$tmp/tool/bin/nixpkgs-check-by-name" --base "$tmp/base" "$tmp/merged"

pkgs/test/nixpkgs-check-by-name/scripts/update-pinned-tool.sh

@@ -0,0 +1,31 @@
+#!/usr/bin/env nix-shell
+#!nix-shell -i bash -p jq
+
+set -o pipefail -o errexit -o nounset
+
+trace() { echo >&2 "$@"; }
+
+SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
+
+channel=nixos-unstable
+pin_file=$SCRIPT_DIR/pinned-tool.json
+
+trace -n "Fetching latest version of channel $channel.. "
+# This is probably the easiest way to get Nix to output the path to a downloaded channel!
+nixpkgs=$(nix-instantiate --find-file nixpkgs -I nixpkgs=channel:"$channel")
+trace "$nixpkgs"
+
+# This file only exists in channels
+rev=$(<"$nixpkgs/.git-revision")
+trace -e "Git revision of channel $channel is \e[34m$rev\e[0m"
+
+
+trace -n "Fetching the prebuilt version of nixpkgs-check-by-name.. "
+path=$(nix-build --no-out-link "$nixpkgs" -A tests.nixpkgs-check-by-name -j 0 | tee /dev/stderr)
+
+trace "Updating $pin_file"
+jq -n \
+    --arg rev "$rev" \
+    --arg path "$path" \
+    '$ARGS.named' \
+    > "$pin_file"