From: David Galloway Date: Mon, 22 Aug 2022 20:12:36 +0000 (-0400) Subject: doc: Update release process doc to accurately reflect current process X-Git-Tag: v17.2.4~75^2 X-Git-Url: http://git-server-git.apps.pok.os.sepia.ceph.com/?a=commitdiff_plain;h=d6bf5118ce1c259202891fcdcde28db38bd91cc7;p=ceph.git doc: Update release process doc to accurately reflect current process Signed-off-by: David Galloway (cherry picked from commit 4829af891ce9fbe41f4b48b10d342e774366dd52) --- diff --git a/doc/dev/release-process.rst b/doc/dev/release-process.rst index 5a1a316af881..3750759b846b 100644 --- a/doc/dev/release-process.rst +++ b/doc/dev/release-process.rst @@ -2,172 +2,224 @@ Ceph Release Process ====================== -1. Build environment -==================== +Prerequisites +============= -There are multiple build environments, debian based packages are built via pbuilder for multiple distributions. The build hosts are listed in the ``deb_hosts`` file, and the list of distributions are in ``deb_dist``. All distributions are build on each of the build hosts. Currently there is 1 64 bit and 1 32 bit build host. +Signing Machine +--------------- +The signing machine is a virtual machine in the `Sepia lab +`_. SSH access to the signing +machine is limited to the usual Infrastructure Admins along with a few other +component leads (e.g., nfs-ganesha, ceph-iscsi). -The RPM based packages are built natively, so one distribution per build host. The list of hosts is found in ``rpm_hosts``. +The ``ubuntu`` user on the machine has some `build scripts `_ that help with pulling, pushing, and signing packages. -Prior to building, it's necessary to update the pbuilder seed tarballs:: +The GPG signing key permanently lives on a `Nitrokey Pro `_ and is passed through to the VM via RHV. This helps to ensure that the key cannot be exported or leave the datacenter in any way. - ./update_all_pbuilders.sh +New Major Releases +------------------ +For each new major (alphabetical) release, you must create one ``ceph-release`` RPM for each RPM repo (e.g., one for el8 and one for el9). `chacra `_ is a python service we use to store DEB and RPM repos. The chacra repos are configured to include this ceph-release RPM, but it must be built separately. You must make sure that chacra is properly configured to include this RPM for each particular release. -2. Setup keyring for signing packages -===================================== +1. Update chacra so it is aware of the new Ceph release. See `this PR `_ for an example. +2. Redeploy chacra (e.g., ``ansible-playbook chacra.ceph.com.yml``) +3. Run https://jenkins.ceph.com/view/all/job/ceph-release-rpm/ -:: +Summarized build process +======================== - export GNUPGHOME= +1. QE finishes testing and finds a stopping point. That commit is pushed to the ``$release-release`` branch in ceph.git (e.g., ``quincy-release``). This allows work to continue in the working ``$release`` branch without having to freeze it during the release process. +2. The Ceph Council approves and notifies the "Build Lead". +3. The "Build Lead" starts the `Jenkins multijob `_, which triggers all builds. +4. Packages are pushed to chacra.ceph.com. +5. Packages are pulled from chacra.ceph.com to the Signer VM. +6. Packages are signed. +7. Packages are pushed to download.ceph.com. +8. Release containers are built and pushed to quay.io. - # verify it's accessible - gpg --list-keys +Hotfix Release Process Deviation +-------------------------------- -The release key should be present:: +A hotfix release has a couple differences. - pub 4096R/17ED316D 2012-05-20 - uid Ceph Release Key +1. Check out the most recent tag. For example, if we're releasing a hotfix on top of 17.2.3, ``git checkout -f -B quincy-release origin/v17.2.3`` +2. ``git cherry-pick -x`` the necessary hotfix commits +3. ``git push -f origin quincy-release`` +4. Notify the "Build Lead" to start the build. +5. The "Build Lead" should set ``RELEASE_TYPE=HOTFIX`` instead of ``STABLE``. +Security Release Process Deviation +---------------------------------- -3. Set up build area -==================== +A security/CVE release is similar to a hotfix release with two differences: -Clone the ceph and ceph-build source trees:: + 1. The fix should be pushed to the `ceph-private `_ repo instead of ceph.git (requires GitHub Admin Role). + 2. The tags (e.g., v17.2.4) must be manually pushed to ceph.git by the "Build Lead." - git clone http://github.com/ceph/ceph.git - git clone http://github.com/ceph/ceph-build.git +1. Check out the most recent tag. For example, if we're releasing a security fix on top of 17.2.3, ``git checkout -f -B quincy-release origin/v17.2.3`` +2. ``git cherry-pick -x`` the necessary security fix commits +3. ``git remote add security git@github.com:ceph/ceph-private.git`` +4. ``git push -f security quincy-release`` +5. Notify the "Build Lead" to start the build. +6. The "Build Lead" should set ``RELEASE_TYPE=SECURITY`` instead of ``STABLE``. +7. Finally, the `ceph-tag `_ steps need to be manually run by the "Build Lead" as close to the Announcement time as possible:: -In the ceph source directory, checkout next branch (for point releases use the {codename} branch):: + # Example using quincy pretending 17.2.4 is the security release version + # Add the ceph-releases repo (also requires GitHub Admin Role). The `ceph-setup `_ job will have already created and pushed the tag to ceph-releases.git. + git remote add releases git@github.com:ceph/ceph-releases.git + git fetch --all + # Check out the version commit + git checkout -f -B quincy-release releases/quincy-release + git push -f origin quincy-release + git push origin v17.2.4 + # Now create a Pull Request of quincy-release targeting quincy to merge the version commit and security fixes back into the quincy branch - git checkout next +1. Preparing the release branch +=============================== -Checkout the submodules:: +Once QE has determined a stopping point in the working (e.g., ``quincy``) branch, that commit should be pushed to the corresponding ``quincy-release`` branch. - git submodule update --force --init --recursive +Notify the "Build Lead" that the release branch is ready. -4. Update Build version numbers -================================ - -Substitute the ceph release number where indicated below by the string ``0.xx``. - -Edit configure.ac and update the version number. Example diff:: - - -AC_INIT([ceph], [0.54], [ceph-devel@vger.kernel.org]) - +AC_INIT([ceph], [0.55], [ceph-devel@vger.kernel.org]) - -Update the version number in the debian change log:: - - DEBEMAIL user@host dch -v 0.xx-1 - -Commit the changes:: - - git commit -a - -Tag the release:: - - ../ceph-build/tag-release v0.xx - - -5. Create Makefiles -=================== - -The actual configure options used to build packages are in the -``ceph.spec.in`` and ``debian/rules`` files. At this point we just -need to create a Makefile.:: - - ./do_autogen.sh - - -6. Run the release scripts -========================== - -This creates tarballs and copies them, with other needed files to -the build hosts listed in deb_hosts and rpm_hosts, runs a local build -script, then rsyncs the results back to the specified release directory.:: - - ../ceph-build/do_release.sh /tmp/release - - -7. Create RPM Repo -================== - -Copy the rpms to the destination repo:: - - mkdir /tmp/rpm-repo - ../ceph-build/push_to_rpm_repo.sh /tmp/release /tmp/rpm-repo 0.xx - -Next add any additional rpms to the repo that are needed such as leveldb. -See RPM Backports section +2. Starting the build +===================== -Finally, sign the rpms and build the repo indexes:: +We'll use a stable/regular 15.2.17 release of Octopus as an example throughout this document. - ../ceph-build/sign_and_index_rpm_repo.sh /tmp/release /tmp/rpm-repo 0.xx +1. Browse to https://jenkins.ceph.com/view/all/job/ceph/build?delay=0sec +2. Log in with GitHub OAuth +3. Set the parameters as necessary:: + BRANCH=octopus + TAG=checked + VERSION=15.2.17 + RELEASE_TYPE=STABLE + ARCHS=x86_64 arm64 -8. Create Debian repo -===================== +4. Use https://docs.ceph.com/en/latest/start/os-recommendations/?highlight=debian#platforms to determine the ``DISTROS`` parameter. For example, -The key-id used below is the id of the ceph release key from step 2:: + +-------------------+-------------------------------------------+ + | Release | Distro Codemap | + +===================+===========================================+ + | octopus (15.X.X) | ``focal bionic centos7 centos8 buster`` | + +-------------------+-------------------------------------------+ + | pacific (16.X.X) | ``focal bionic centos8 buster bullseye`` | + +-------------------+-------------------------------------------+ + | quincy (17.X.X) | ``focal centos8 centos9 bullseye`` | + +-------------------+-------------------------------------------+ - mkdir /tmp/debian-repo - ../ceph-build/gen_reprepro_conf.sh /tmp/debian-repo key-id - ../ceph-build/push_to_deb_repo.sh /tmp/release /tmp/debian-repo 0.xx main +5. Click ``Build``. +3. Release Notes +================ -Next add any addition debian packages that are needed such as leveldb. -See the Debian Backports section below. +Packages take hours to build. Use those hours to create the Release Notes and Announcements: -Debian packages are signed when added to the repo, so no further action is -needed. +1. ceph.git Release Notes (e.g., `v15.2.17's ceph.git (docs.ceph.com) PR `_) +2. ceph.io Release Notes (e.g., `v15.2.17's ceph.io.git (www.ceph.io) PR `_) +3. E-mail announcement +See `the Ceph Tracker wiki page that explains how to write the release notes `_. -9. Push repos to ceph.org -========================== +4. Signing and Publishing the Build +=================================== -For a development release:: +#. Obtain the sha1 of the version commit from the `build job `_ or the ``sha1`` file created by the `ceph-setup `_ job. - rcp ceph-0.xx.tar.bz2 ceph-0.xx.tar.gz \ - ceph_site@ceph.com:ceph.com/downloads/. - rsync -av /tmp/rpm-repo/0.xx/ ceph_site@ceph.com:ceph.com/rpm-testing - rsync -av /tmp/debian-repo/ ceph_site@ceph.com:ceph.com/debian-testing +#. Download the packages from chacra.ceph.com to the signing virtual machine. These packages get downloaded to ``/opt/repos`` where the `Sepia Lab Long Running (Ceph) Cluster `_ is mounted. -For a stable release, replace {CODENAME} with the release codename (e.g., ``argonaut`` or ``bobtail``):: + .. prompt:: bash $ - rcp ceph-0.xx.tar.bz2 \ - ceph_site@ceph.com:ceph.com/downloads/ceph-0.xx.tar.bz2 - rcp ceph-0.xx.tar.gz \ - ceph_site@ceph.com:ceph.com/downloads/ceph-0.xx.tar.gz - rsync -av /tmp/rpm-repo/0.xx/ ceph_site@ceph.com:ceph.com/rpm-{CODENAME} - rsync -auv /tmp/debian-repo/ ceph_site@ceph.com:ceph.com/debian-{CODENAME} + ssh ubuntu@signer.front.sepia.ceph.com + sync-pull ceph [pacific|quincy|etc] + + Example:: + + $ sync-pull ceph octopus 8a82819d84cf884bd39c17e3236e0632ac146dc4 + sync for: ceph octopus + ******************************************** + Found the most packages (332) in ubuntu/bionic. + No JSON object could be decoded + No JSON object could be decoded + ubuntu@chacra.ceph.com:/opt/repos/ceph/octopus/8a82819d84cf884bd39c17e3236e0632ac146dc4/ubuntu/bionic/flavors/default/* /opt/repos/ceph/octopus-15.2.17/debian/jessie/ + -------------------------------------------- + receiving incremental file list + db/ + db/checksums.db + 180.22K 100% 2.23MB/s 0:00:00 (xfr#1, to-chk=463/467) + db/contents.cache.db + 507.90K 100% 1.95MB/s 0:00:00 (xfr#2, to-chk=462/467) + db/packages.db + + etc... + +#. Sign the DEBs: + + .. prompt:: bash + + merfi gpg /opt/repos/ceph/octopus-15.2.17/debian + + Example:: + + $ merfi gpg /opt/repos/ceph/octopus-15.2.17/debian + --> Starting path collection, looking for files to sign + --> 18 matching paths found + --> will sign with the following commands: + --> gpg --batch --yes --armor --detach-sig --output Release.gpg Release + --> gpg --batch --yes --clearsign --output InRelease Release + --> signing: /opt/repos/ceph/octopus-15.2.17/debian/jessie/dists/bionic/Release + --> Running command: gpg --batch --yes --armor --detach-sig --output Release.gpg Release + --> Running command: gpg --batch --yes --clearsign --output InRelease Release + --> signing: /opt/repos/ceph/octopus-15.2.17/debian/jessie/dists/focal/Release + --> Running command: gpg --batch --yes --armor --detach-sig --output Release.gpg Release + --> Running command: gpg --batch --yes --clearsign --output InRelease Release + + etc... + +#. Sign the RPMs: + + .. prompt:: bash + + sign-rpms octopus + + Example:: + + $ sign-rpms octopus + Checking packages in: /opt/repos/ceph/octopus-15.2.17/centos/7 + signing: /opt/repos/ceph/octopus-15.2.17/centos/7/SRPMS/ceph-release-1-1.el7.src.rpm + /opt/repos/ceph/octopus-15.2.17/centos/7/SRPMS/ceph-release-1-1.el7.src.rpm: + signing: /opt/repos/ceph/octopus-15.2.17/centos/7/SRPMS/ceph-15.2.17-0.el7.src.rpm + /opt/repos/ceph/octopus-15.2.17/centos/7/SRPMS/ceph-15.2.17-0.el7.src.rpm: + signing: /opt/repos/ceph/octopus-15.2.17/centos/7/noarch/ceph-mgr-modules-core-15.2.17-0.el7.noarch.rpm + + etc... + +5. Publish the packages to download.ceph.com: + + .. prompt:: bash $ + + sync-push octopus + +5. Build Containers +=================== -10. Update Git -============== +Start the following two jobs: -Point release -------------- +#. https://2.jenkins.ceph.com/job/ceph-container-build-ceph-base-push-imgs/ +#. https://2.jenkins.ceph.com/job/ceph-container-build-ceph-base-push-imgs-arm64/ -For point releases just push the version number update to the -branch and the new tag:: +6. Announce the Release +======================= - git push origin {codename} - git push origin v0.xx +Version Commit PR +----------------- -Development and Stable releases -------------------------------- +The `ceph-tag Jenkins job `_ creates a Pull Request in ceph.git that targets the release branch. -For a development release, update tags for ``ceph.git``:: +If this was a regular release (not a hotfix release or a security release), the only commit in that Pull Request should be the version commit. For example, see `v15.2.17's version commit PR `_. - git push origin v0.xx - git push origin HEAD:last - git checkout master - git merge next - git push origin master - git push origin HEAD:next +Request a review and then merge the Pull Request. -Similarly, for a development release, for both ``teuthology.git`` and ``ceph-qa-suite.git``:: +Announcing +---------- - git checkout master - git reset --hard origin/master - git branch -f last origin/next - git push -f origin last - git push -f origin master:next +Publish the Release Notes on ceph.io before announcing the release by email, because the e-mail announcement references the ceph.io blog post.