Files
pytorch/scripts/release/apply-release-changes.sh

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

63 lines
2.8 KiB
Bash
Raw Permalink Normal View History

#!/usr/bin/env bash
#
# Step 2 after branch cut is complete.
#
# Creates PR with release only changes.
#
# Prerequisite: Must be successfully authenticated in aws fbossci account.
#
# Usage (run from root of project):
# DRY_RUN=disabled ./scripts/release/apply-release-changes.sh
#
# RELEASE_VERSION: Version of this current release
set -eou pipefail
GIT_TOP_DIR=$(git rev-parse --show-toplevel)
RELEASE_VERSION=${RELEASE_VERSION:-$(cut -d'.' -f1-2 "${GIT_TOP_DIR}/version.txt")}
[RelEng] Tag docker images for release, pin unstable and disabled jobs, apply release only changes (#114355) 1. This tags docker images using docker pull/tag/push for current release 2. Sets RELEASE_VERSION_TAG var and regenerates the workflows using the new docker tag 3. Remove conda token setting and Binary tests release changes these are already automated 4. Pin unstable and disabled jobs, autumate: https://github.com/pytorch/pytorch/pull/111675 Test: ``` RELEASE_VERSION=2.2 ./scripts/release/apply-release-changes.sh Tagging pytorch/manylinux-builder:cuda11.8-main to pytorch/manylinux-builder:cuda11.8-2.2 , dry_run: enabled Tagging pytorch/manylinux-builder:cuda12.1-main to pytorch/manylinux-builder:cuda12.1-2.2 , dry_run: enabled Tagging pytorch/libtorch-cxx11-builder:cuda11.8-main to pytorch/libtorch-cxx11-builder:cuda11.8-2.2 , dry_run: enabled Tagging pytorch/libtorch-cxx11-builder:cuda12.1-main to pytorch/libtorch-cxx11-builder:cuda12.1-2.2 , dry_run: enabled Tagging pytorch/manylinux-builder:rocm5.6-main to pytorch/manylinux-builder:rocm5.6-2.2 , dry_run: enabled Tagging pytorch/manylinux-builder:rocm5.7-main to pytorch/manylinux-builder:rocm5.7-2.2 , dry_run: enabled Tagging pytorch/libtorch-cxx11-builder:rocm5.6-main to pytorch/libtorch-cxx11-builder:rocm5.6-2.2 , dry_run: enabled Tagging pytorch/libtorch-cxx11-builder:rocm5.7-main to pytorch/libtorch-cxx11-builder:rocm5.7-2.2 , dry_run: enabled Tagging pytorch/manylinux-builder:cpu-main to pytorch/manylinux-builder:cpu-2.2 , dry_run: enabled Tagging pytorch/libtorch-cxx11-builder:cpu-main to pytorch/libtorch-cxx11-builder:cpu-2.2 , dry_run: enabled Tagging pytorch/manylinuxcxx11-abi-builder:cpu-cxx11-abi-main to pytorch/manylinuxcxx11-abi-builder:cpu-cxx11-abi-2.2 , dry_run: enabled Tagging pytorch/manylinuxaarch64-builder:cpu-aarch64-main to pytorch/manylinuxaarch64-builder:cpu-aarch64-2.2 , dry_run: enabled Tagging pytorch/conda-builder:cuda11.8-main to pytorch/conda-builder:cuda11.8-2.2 , dry_run: enabled Tagging pytorch/conda-builder:cuda12.1-main to pytorch/conda-builder:cuda12.1-2.2 , dry_run: enabled Tagging pytorch/conda-builder:cpu-main to pytorch/conda-builder:cpu-2.2 , dry_run: enabled /data/users/atalman/pytorch/.github/workflows/generated-linux-binary-manywheel-nightly.yml /data/users/atalman/pytorch/.github/workflows/generated-linux-binary-conda-nightly.yml /data/users/atalman/pytorch/.github/workflows/generated-linux-binary-libtorch-cxx11-abi-nightly.yml /data/users/atalman/pytorch/.github/workflows/generated-linux-binary-libtorch-pre-cxx11-nightly.yml /data/users/atalman/pytorch/.github/workflows/generated-linux-aarch64-binary-manywheel-nightly.yml /data/users/atalman/pytorch/.github/workflows/generated-linux-binary-manywheel-main.yml /data/users/atalman/pytorch/.github/workflows/generated-linux-binary-libtorch-cxx11-abi-main.yml /data/users/atalman/pytorch/.github/workflows/generated-linux-binary-libtorch-pre-cxx11-main.yml /data/users/atalman/pytorch/.github/workflows/generated-windows-binary-wheel-nightly.yml /data/users/atalman/pytorch/.github/workflows/generated-windows-binary-conda-nightly.yml /data/users/atalman/pytorch/.github/workflows/generated-windows-binary-libtorch-release-nightly.yml /data/users/atalman/pytorch/.github/workflows/generated-windows-binary-libtorch-debug-nightly.yml /data/users/atalman/pytorch/.github/workflows/generated-windows-binary-libtorch-release-main.yml /data/users/atalman/pytorch/.github/workflows/generated-windows-binary-libtorch-debug-main.yml /data/users/atalman/pytorch/.github/workflows/generated-macos-binary-wheel-nightly.yml /data/users/atalman/pytorch/.github/workflows/generated-macos-binary-conda-nightly.yml /data/users/atalman/pytorch/.github/workflows/generated-macos-binary-libtorch-cxx11-abi-nightly.yml /data/users/atalman/pytorch/.github/workflows/generated-macos-arm64-binary-libtorch-cxx11-abi-nightly.yml /data/users/atalman/pytorch/.github/workflows/generated-macos-arm64-binary-wheel-nightly.yml /data/users/atalman/pytorch/.github/workflows/generated-macos-arm64-binary-conda-nightly.yml ```` Result of pinning unstable and disabled jobs: ``` # The link to the published list of disabled jobs DISABLED_JOBS_URL = "https://ossci-metrics.s3.amazonaws.com/disabled-jobs.json?versionid=kKJlAXdrUbk3CilXbKu.6OwNTGQB8a.B" # and unstable jobs UNSTABLE_JOBS_URL = "https://ossci-metrics.s3.amazonaws.com/unstable-jobs.json?versionid=vzaicOxSsh55iXBXwgGrW6dFeVtPfrhr" ``` Pull Request resolved: https://github.com/pytorch/pytorch/pull/114355 Approved by: https://github.com/malfet
2023-11-23 02:14:22 +00:00
DRY_RUN=${DRY_RUN:-enabled}
echo "Applying to workflows"
for i in .github/workflows/*.yml; do
sed -i -e s#@main#@"release/${RELEASE_VERSION}"# $i;
done
echo "Applying to templates"
for i in .github/templates/*.yml.j2; do
sed -i 's#common.checkout(\(.*\))#common.checkout(\1, checkout_pr_head=False)#' $i;
sed -i -e s#main#"release/${RELEASE_VERSION}"# $i;
done
echo "Applying to changes to linux binary builds"
for i in ".github/workflows/_binary-build-linux.yml" ".github/workflows/_binary-test-linux.yml"; do
sed -i "/github.event_name == 'pull_request'/d" $i;
sed -i -e s#main#"release/${RELEASE_VERSION}"# $i;
done
sed -i -e "/generate_ci_workflows.py/i \\\t\t\t\texport RELEASE_VERSION_TAG=${RELEASE_VERSION}" .github/workflows/lint.yml
# Triton wheel
echo "Triton Changes"
sed -i -e s#-\ main#"-\ release\/${RELEASE_VERSION}"# .github/workflows/build-triton-wheel.yml
# XLA related changes
echo "XLA Changes"
sed -i -e s#--quiet#-b\ r"${RELEASE_VERSION}"# .ci/pytorch/common_utils.sh
sed -i -e s#.*#r"${RELEASE_VERSION}"# .github/ci_commit_pins/xla.txt
[RelEng] Tag docker images for release, pin unstable and disabled jobs, apply release only changes (#114355) 1. This tags docker images using docker pull/tag/push for current release 2. Sets RELEASE_VERSION_TAG var and regenerates the workflows using the new docker tag 3. Remove conda token setting and Binary tests release changes these are already automated 4. Pin unstable and disabled jobs, autumate: https://github.com/pytorch/pytorch/pull/111675 Test: ``` RELEASE_VERSION=2.2 ./scripts/release/apply-release-changes.sh Tagging pytorch/manylinux-builder:cuda11.8-main to pytorch/manylinux-builder:cuda11.8-2.2 , dry_run: enabled Tagging pytorch/manylinux-builder:cuda12.1-main to pytorch/manylinux-builder:cuda12.1-2.2 , dry_run: enabled Tagging pytorch/libtorch-cxx11-builder:cuda11.8-main to pytorch/libtorch-cxx11-builder:cuda11.8-2.2 , dry_run: enabled Tagging pytorch/libtorch-cxx11-builder:cuda12.1-main to pytorch/libtorch-cxx11-builder:cuda12.1-2.2 , dry_run: enabled Tagging pytorch/manylinux-builder:rocm5.6-main to pytorch/manylinux-builder:rocm5.6-2.2 , dry_run: enabled Tagging pytorch/manylinux-builder:rocm5.7-main to pytorch/manylinux-builder:rocm5.7-2.2 , dry_run: enabled Tagging pytorch/libtorch-cxx11-builder:rocm5.6-main to pytorch/libtorch-cxx11-builder:rocm5.6-2.2 , dry_run: enabled Tagging pytorch/libtorch-cxx11-builder:rocm5.7-main to pytorch/libtorch-cxx11-builder:rocm5.7-2.2 , dry_run: enabled Tagging pytorch/manylinux-builder:cpu-main to pytorch/manylinux-builder:cpu-2.2 , dry_run: enabled Tagging pytorch/libtorch-cxx11-builder:cpu-main to pytorch/libtorch-cxx11-builder:cpu-2.2 , dry_run: enabled Tagging pytorch/manylinuxcxx11-abi-builder:cpu-cxx11-abi-main to pytorch/manylinuxcxx11-abi-builder:cpu-cxx11-abi-2.2 , dry_run: enabled Tagging pytorch/manylinuxaarch64-builder:cpu-aarch64-main to pytorch/manylinuxaarch64-builder:cpu-aarch64-2.2 , dry_run: enabled Tagging pytorch/conda-builder:cuda11.8-main to pytorch/conda-builder:cuda11.8-2.2 , dry_run: enabled Tagging pytorch/conda-builder:cuda12.1-main to pytorch/conda-builder:cuda12.1-2.2 , dry_run: enabled Tagging pytorch/conda-builder:cpu-main to pytorch/conda-builder:cpu-2.2 , dry_run: enabled /data/users/atalman/pytorch/.github/workflows/generated-linux-binary-manywheel-nightly.yml /data/users/atalman/pytorch/.github/workflows/generated-linux-binary-conda-nightly.yml /data/users/atalman/pytorch/.github/workflows/generated-linux-binary-libtorch-cxx11-abi-nightly.yml /data/users/atalman/pytorch/.github/workflows/generated-linux-binary-libtorch-pre-cxx11-nightly.yml /data/users/atalman/pytorch/.github/workflows/generated-linux-aarch64-binary-manywheel-nightly.yml /data/users/atalman/pytorch/.github/workflows/generated-linux-binary-manywheel-main.yml /data/users/atalman/pytorch/.github/workflows/generated-linux-binary-libtorch-cxx11-abi-main.yml /data/users/atalman/pytorch/.github/workflows/generated-linux-binary-libtorch-pre-cxx11-main.yml /data/users/atalman/pytorch/.github/workflows/generated-windows-binary-wheel-nightly.yml /data/users/atalman/pytorch/.github/workflows/generated-windows-binary-conda-nightly.yml /data/users/atalman/pytorch/.github/workflows/generated-windows-binary-libtorch-release-nightly.yml /data/users/atalman/pytorch/.github/workflows/generated-windows-binary-libtorch-debug-nightly.yml /data/users/atalman/pytorch/.github/workflows/generated-windows-binary-libtorch-release-main.yml /data/users/atalman/pytorch/.github/workflows/generated-windows-binary-libtorch-debug-main.yml /data/users/atalman/pytorch/.github/workflows/generated-macos-binary-wheel-nightly.yml /data/users/atalman/pytorch/.github/workflows/generated-macos-binary-conda-nightly.yml /data/users/atalman/pytorch/.github/workflows/generated-macos-binary-libtorch-cxx11-abi-nightly.yml /data/users/atalman/pytorch/.github/workflows/generated-macos-arm64-binary-libtorch-cxx11-abi-nightly.yml /data/users/atalman/pytorch/.github/workflows/generated-macos-arm64-binary-wheel-nightly.yml /data/users/atalman/pytorch/.github/workflows/generated-macos-arm64-binary-conda-nightly.yml ```` Result of pinning unstable and disabled jobs: ``` # The link to the published list of disabled jobs DISABLED_JOBS_URL = "https://ossci-metrics.s3.amazonaws.com/disabled-jobs.json?versionid=kKJlAXdrUbk3CilXbKu.6OwNTGQB8a.B" # and unstable jobs UNSTABLE_JOBS_URL = "https://ossci-metrics.s3.amazonaws.com/unstable-jobs.json?versionid=vzaicOxSsh55iXBXwgGrW6dFeVtPfrhr" ``` Pull Request resolved: https://github.com/pytorch/pytorch/pull/114355 Approved by: https://github.com/malfet
2023-11-23 02:14:22 +00:00
# Regenerate templates
export RELEASE_VERSION_TAG=${RELEASE_VERSION}
./.github/regenerate.sh
# Pin Unstable and disabled jobs and tests
[RelEng] Tag docker images for release, pin unstable and disabled jobs, apply release only changes (#114355) 1. This tags docker images using docker pull/tag/push for current release 2. Sets RELEASE_VERSION_TAG var and regenerates the workflows using the new docker tag 3. Remove conda token setting and Binary tests release changes these are already automated 4. Pin unstable and disabled jobs, autumate: https://github.com/pytorch/pytorch/pull/111675 Test: ``` RELEASE_VERSION=2.2 ./scripts/release/apply-release-changes.sh Tagging pytorch/manylinux-builder:cuda11.8-main to pytorch/manylinux-builder:cuda11.8-2.2 , dry_run: enabled Tagging pytorch/manylinux-builder:cuda12.1-main to pytorch/manylinux-builder:cuda12.1-2.2 , dry_run: enabled Tagging pytorch/libtorch-cxx11-builder:cuda11.8-main to pytorch/libtorch-cxx11-builder:cuda11.8-2.2 , dry_run: enabled Tagging pytorch/libtorch-cxx11-builder:cuda12.1-main to pytorch/libtorch-cxx11-builder:cuda12.1-2.2 , dry_run: enabled Tagging pytorch/manylinux-builder:rocm5.6-main to pytorch/manylinux-builder:rocm5.6-2.2 , dry_run: enabled Tagging pytorch/manylinux-builder:rocm5.7-main to pytorch/manylinux-builder:rocm5.7-2.2 , dry_run: enabled Tagging pytorch/libtorch-cxx11-builder:rocm5.6-main to pytorch/libtorch-cxx11-builder:rocm5.6-2.2 , dry_run: enabled Tagging pytorch/libtorch-cxx11-builder:rocm5.7-main to pytorch/libtorch-cxx11-builder:rocm5.7-2.2 , dry_run: enabled Tagging pytorch/manylinux-builder:cpu-main to pytorch/manylinux-builder:cpu-2.2 , dry_run: enabled Tagging pytorch/libtorch-cxx11-builder:cpu-main to pytorch/libtorch-cxx11-builder:cpu-2.2 , dry_run: enabled Tagging pytorch/manylinuxcxx11-abi-builder:cpu-cxx11-abi-main to pytorch/manylinuxcxx11-abi-builder:cpu-cxx11-abi-2.2 , dry_run: enabled Tagging pytorch/manylinuxaarch64-builder:cpu-aarch64-main to pytorch/manylinuxaarch64-builder:cpu-aarch64-2.2 , dry_run: enabled Tagging pytorch/conda-builder:cuda11.8-main to pytorch/conda-builder:cuda11.8-2.2 , dry_run: enabled Tagging pytorch/conda-builder:cuda12.1-main to pytorch/conda-builder:cuda12.1-2.2 , dry_run: enabled Tagging pytorch/conda-builder:cpu-main to pytorch/conda-builder:cpu-2.2 , dry_run: enabled /data/users/atalman/pytorch/.github/workflows/generated-linux-binary-manywheel-nightly.yml /data/users/atalman/pytorch/.github/workflows/generated-linux-binary-conda-nightly.yml /data/users/atalman/pytorch/.github/workflows/generated-linux-binary-libtorch-cxx11-abi-nightly.yml /data/users/atalman/pytorch/.github/workflows/generated-linux-binary-libtorch-pre-cxx11-nightly.yml /data/users/atalman/pytorch/.github/workflows/generated-linux-aarch64-binary-manywheel-nightly.yml /data/users/atalman/pytorch/.github/workflows/generated-linux-binary-manywheel-main.yml /data/users/atalman/pytorch/.github/workflows/generated-linux-binary-libtorch-cxx11-abi-main.yml /data/users/atalman/pytorch/.github/workflows/generated-linux-binary-libtorch-pre-cxx11-main.yml /data/users/atalman/pytorch/.github/workflows/generated-windows-binary-wheel-nightly.yml /data/users/atalman/pytorch/.github/workflows/generated-windows-binary-conda-nightly.yml /data/users/atalman/pytorch/.github/workflows/generated-windows-binary-libtorch-release-nightly.yml /data/users/atalman/pytorch/.github/workflows/generated-windows-binary-libtorch-debug-nightly.yml /data/users/atalman/pytorch/.github/workflows/generated-windows-binary-libtorch-release-main.yml /data/users/atalman/pytorch/.github/workflows/generated-windows-binary-libtorch-debug-main.yml /data/users/atalman/pytorch/.github/workflows/generated-macos-binary-wheel-nightly.yml /data/users/atalman/pytorch/.github/workflows/generated-macos-binary-conda-nightly.yml /data/users/atalman/pytorch/.github/workflows/generated-macos-binary-libtorch-cxx11-abi-nightly.yml /data/users/atalman/pytorch/.github/workflows/generated-macos-arm64-binary-libtorch-cxx11-abi-nightly.yml /data/users/atalman/pytorch/.github/workflows/generated-macos-arm64-binary-wheel-nightly.yml /data/users/atalman/pytorch/.github/workflows/generated-macos-arm64-binary-conda-nightly.yml ```` Result of pinning unstable and disabled jobs: ``` # The link to the published list of disabled jobs DISABLED_JOBS_URL = "https://ossci-metrics.s3.amazonaws.com/disabled-jobs.json?versionid=kKJlAXdrUbk3CilXbKu.6OwNTGQB8a.B" # and unstable jobs UNSTABLE_JOBS_URL = "https://ossci-metrics.s3.amazonaws.com/unstable-jobs.json?versionid=vzaicOxSsh55iXBXwgGrW6dFeVtPfrhr" ``` Pull Request resolved: https://github.com/pytorch/pytorch/pull/114355 Approved by: https://github.com/malfet
2023-11-23 02:14:22 +00:00
UNSTABLE_VER=$(aws s3api list-object-versions --bucket ossci-metrics --prefix unstable-jobs.json --query 'Versions[?IsLatest].[VersionId]' --output text)
DISABLED_VER=$(aws s3api list-object-versions --bucket ossci-metrics --prefix disabled-jobs.json --query 'Versions[?IsLatest].[VersionId]' --output text)
SLOW_VER=$(aws s3api list-object-versions --bucket ossci-metrics --prefix slow-tests.json --query 'Versions[?IsLatest].[VersionId]' --output text)
DISABLED_TESTS_VER=$(aws s3api list-object-versions --bucket ossci-metrics --prefix disabled-tests-condensed.json --query 'Versions[?IsLatest].[VersionId]' --output text)
sed -i -e s#unstable-jobs.json#"unstable-jobs.json?versionId=${UNSTABLE_VER}"# .github/scripts/filter_test_configs.py
sed -i -e s#disabled-jobs.json#"disabled-jobs.json?versionId=${DISABLED_VER}"# .github/scripts/filter_test_configs.py
sed -i -e s#disabled-tests-condensed.json#"disabled-tests-condensed.json?versionId=${DISABLED_TESTS_VER}"# tools/stats/import_test_stats.py
# Optional
git commit -m "[RELEASE-ONLY CHANGES] Branch Cut for Release {RELEASE_VERSION}"
git push origin "${RELEASE_BRANCH}"