From: Nathan Cutler Date: Fri, 6 Sep 2019 13:46:37 +0000 (+0200) Subject: script/ceph-backport.sh: help the user set up the script X-Git-Tag: v15.1.0~1586^2 X-Git-Url: http://git-server-git.apps.pok.os.sepia.ceph.com/?a=commitdiff_plain;h=d1aa31890e558b6bac7ee7da74e344da12ff9b71;p=ceph.git script/ceph-backport.sh: help the user set up the script This commit adds several options: --setup --setup-advice --usage-advice --troubleshooting-advice It replaces the "comment block at the beginning of the script" which was lamely masquerading as documentation. It also implements heuristic deduction of GitHub remotes. Signed-off-by: Nathan Cutler --- diff --git a/src/script/ceph-backport.sh b/src/script/ceph-backport.sh index 17ec110c7258..35d2015a4791 100755 --- a/src/script/ceph-backport.sh +++ b/src/script/ceph-backport.sh @@ -4,235 +4,212 @@ # # Credits: This script is based on work done by Loic Dachary # -# With proper setup, this script automates the entire process of creating a -# backport -- all it needs is the number of the Backport tracker issue. -# Working in a local clone, the script creates a properly named wip branch for -# the backport, fetches the commits from GitHub, and cherry-picks them. If all -# the commits cherry-pick cleanly, the script goes on to open the backport PR, -# update the Backport tracker issue, and cross-linking the backport PR with the -# tracker issue. # -# Some setup is required for the script to work -- this is described -# in the "Setup instructions" and "Instructions for use" sections, below. -# If issues persist even after reading and following those instructions -# carefully, the "Troubleshooting notes" and "Reporting bugs" sections might -# come in handy, as well. +# This script automates the process of staging a backport starting from a +# Backport tracker issue. # +# Setup, usage and troubleshooting: # -# Setup instructions -# ------------------ +# ceph-backport.sh --help +# ceph-backport.sh --setup-advice +# ceph-backport.sh --usage-advice +# ceph-backport.sh --troubleshooting-advice # -# First, copy the latest version of the script (from the "master" branch) into -# the PATH. In particular, do not use any version of the script from a stable -# (named) branch such as "nautilus", as these versions are not maintained. Only -# the version in master is maintained. -# -# Second, obtain the correct values for the following: -# -# redmine_key # "My account" -> "API access key" -> "Show" -# redmine_user_id # "Logged in as foobar", click on foobar link, Redmine User ID - # is in the URL, i.e. https://tracker.ceph.com/users/[redmine_user_id] -# github_token # https://github.com/settings/tokens -> Generate new token -> - # ensure it has "Full control of private repositories" scope -# github_user # Your github username -# -# The script supports two ways of setting these variables - via the environment -# or by sourcing a file. If one or more of these variables are not found in the -# environment, the script will attempt to source a file -# $HOME/bin/backport_common.sh. -# -# Care should be taken to keep the values of redmine_key and github_token secret. -# -# The above variables must be set explicitly, as the script has no way of -# determining reasonable defaults. In addition, the following two variables -# should at least be checked against the output of "git remote -v" in your clone: -# -# fork_remote # The "git remote" name of your Ceph fork on GitHub; -# # defaults to "origin" if not set -# upstream_remote # The "git remote" name of the upstream Ceph repo on GitHub; -# # defaults to "upstream" if not set -# -# Without correct values for all of the above variables, this script will not -# work! -# -# -# Instructions for use -# -------------------- -# -# After having completed the setup steps described in the previous section, -# the script can be used to automate backporting. -# -# First, change the working directory ("cd") to the top-level directory of the -# local clone. -# -# Then, choose a Backport tracker issue you would like to stage a backport for. -# Let's assume the Backport tracker issue you chose has the number 31459 and -# the backport targets luminous. -# -# Run: -# -# ceph-backport.sh 31459 -# -# Assuming 31459 is a properly-linked Backport issue and the "Pull request ID" -# field of the parent issue has been correctly populated, the script will -# automatically: -# -# 1. determine which stable branch the backport is targeting (luminous) -# 2. create a local backport branch, wip-31459-luminous, based on the tip of -# luminous -# 3. determine parent issue and master PR and cherry-pick the commits from the -# master PR -# -# If there were no cherry-pick conflicts, the script will continue (see steps -# 4-6, below). -# -# If any commit fails to cherry-pick cleanly, the script will abort, giving -# the user an opportunity to resolve the conflicts manually and re-run the -# script (using the same exact command as the first time). When the script sees -# that the local backport branch (wip-31459-luminous) already exists, it -# will assume that cherry-picking has been completed, and start from step 4: -# -# 4. push the wip branch to the user's fork -# 5. open the backport PR on GitHub -# 6. cross-link the PR with the Backport tracker issue -# -# Finally, if a process called "firefox" is running, the script will open the -# PR and the updated tracker issue in that web browser to facilitate visual -# confirmation. -# -# It is also possible to prepare the wip branch and do the cherry-picking -# manually before running the script. For a a luminous backport whose Backport -# tracker ID is 31459, the process would be: -# -# git remote add ceph http://github.com/ceph/ceph.git -# git fetch ceph -# git checkout -b wip-31459-luminous ceph/luminous -# git cherry-pick -x ... -# (resolve conflicts if necessary) -# ceph-backport.sh 31459 -# -# CAVEAT: the local branch name must be "wip-31459-luminous", where 31459 is -# the number of a Redmine issue in the Backport tracker, with Release set to -# luminous. The script will not work, otherwise. -# -# Optionally, if you have sufficient GitHub privileges on the upstream Ceph -# repo, the script can automate setting of the label and milestone in the -# backport PR. This behavior is triggered by passing --set-milestone on the -# command line, or via the environment variable CEPH_BACKPORT_SET_MILESTONE -# (set it to any value other than the empty string to activate the behavior). -# -# The component label that will be added to the PR defaults to "core", but a -# different value can be set with an environment variable: -# -# COMPONENT=dashboard ceph-backport.sh 31459 -# -# or by passing the component as an argument: -# -# ceph-backport.sh --component dashboard 31459 -# -# -# Troubleshooting notes -# --------------------- -# -# If the script inexplicably fails with: -# -# error: a cherry-pick or revert is already in progress -# hint: try "git cherry-pick (--continue | --quit | --abort)" -# fatal: cherry-pick failed -# -# This is because HEAD is not where git expects it to be: -# -# $ git cherry-pick --abort -# warning: You seem to have moved HEAD. Not rewinding, check your HEAD! -# -# This can be fixed by issuing the command: -# -# $ git cherry-pick --quit -# -# -# Reporting bugs -# -------------- -# -# Please report any bugs in this script to https://tracker.ceph.com/projects/ceph/issues/new -# Be sure to mention the exact version of git you are using. -# -# (Ideally, the bug report would include a typescript obtained while -# reproducing the bug with the --debug option. To understand what is meant by -# "typescript", see "man script".) -# -# -# Other backporting resources -# --------------------------- -# -# See https://tracker.ceph.com/projects/ceph-releases/wiki/HOWTO_backport_commits -# for more info on cherry-picking. -# -# Happy backporting! -# Nathan # this_script=$(basename "$0") +how_to_get_setup_advice="For setup advice, run: ${this_script} --setup-advice" + +if [[ $* == *--debug* ]]; then + set -x +fi + +function cherry_pick_phase { + local offset=0 + populate_original_issue + if [ -z "$original_issue" ] ; then + error "Could not find original issue" + info "Does ${redmine_url} have a \"Copied from\" relation?" + false + fi + info "Parent issue: ${redmine_endpoint}/issues/${original_issue}" + + populate_original_pr + if [ -z "$original_pr" ]; then + error "Could not find original PR" + info "Is the \"Pull request ID\" field of ${redmine_endpoint}/issues/${original_issue} populated?" + false + fi + info "Parent issue ostensibly fixed by: ${original_pr_url}" + + debug "Counting commits in ${original_pr_url}" + number=$(curl --silent https://api.github.com/repos/ceph/ceph/pulls/${original_pr}?access_token=${github_token} | jq .commits) + if [ -z "$number" ] ; then + error "Could not determine the number of commits in ${original_pr_url}" + return 1 + fi + info "Found $number commits in ${original_pr_url}" + + debug "Fetching latest commits from $upstream_remote" + git fetch $upstream_remote + + debug "Initializing local branch $local_branch to $milestone" + if git show-ref --verify --quiet refs/heads/$local_branch ; then + error "Cannot initialize $local_branch - local branch already exists" + false + else + git checkout $upstream_remote/$milestone -b $local_branch + fi + + debug "Fetching latest commits from ${original_pr_url}" + git fetch $upstream_remote pull/$original_pr/head:pr-$original_pr + + info "Attempting to cherry pick $number commits from ${original_pr_url} into local branch $local_branch" + let offset=$number-1 || true # don't fail on set -e when result is 0 + for ((i=$offset; i>=0; i--)) ; do + debug "Cherry-picking commit $(git log --oneline --max-count=1 --no-decorate pr-$original_pr~$i)" + if git cherry-pick -x "pr-$original_pr~$i" ; then + true + else + if [ "$VERBOSE" ] ; then + git status + fi + error "Cherry pick failed" + info "Next, manually fix conflicts and complete the current cherry-pick" + if [ "$i" -gt "0" ] >/dev/null 2>&1 ; then + info "Then, cherry-pick the remaining commits from ${original_pr_url}, i.e.:" + for ((j=$i-1; j>=0; j--)) ; do + info "-> missing commit: $(git log --oneline --max-count=1 --no-decorate pr-$original_pr~$j)" + done + info "Finally, re-run the script" + else + info "Then re-run the script" + fi + false + fi + done + info "Cherry picking completed without conflicts" +} function debug { - log debug $@ + log debug "$@" +} + +function deduce_remote { + local remote_type="$1" + local remote="" + local url_component="" + if [ "$remote_type" = "upstream" ] ; then + url_component="ceph" + elif [ "$remote_type" = "fork" ] ; then + url_component="$github_user" + else + error "Internal error in deduce_remote" + exit 1 + fi + remote=$(git remote -v | egrep --ignore-case '(://|@)github.com(/|:)'$url_component'/ceph(\s|\.|\/)' | head -n1 | cut -f 1) + if [ "$remote" ] ; then + true + else + error "Cannot auto-determine ${remote_type}_remote" + info "Please set this variable explicitly and also file a bug report at ${redmine_endpoint}" + exit 1 + fi + echo "$remote" } function eol { log mtt=$1 error "$mtt is EOL" - exit 1 + false } function error { - log error $@ + log error "$@" } -function failed_required_variable_check { +function failed_mandatory_var_check { local varname="$1" - error "$varname not defined. Did you create $HOME/bin/backport_common.sh?" - info "(For detailed instructions, see comment block at the beginning of the script)" - exit 1 + error "$varname not defined" + setup_ok="" } function info { - log info $@ + log info "$@" } -function init_mandatory_vars { - debug Initializing mandatory variables - test "$redmine_key" || failed_required_variable_check redmine_key - test "$redmine_user_id" || failed_required_variable_check redmine_user_id - test "$github_token" || failed_required_variable_check github_token - test "$github_user" || failed_required_variable_check github_user - if [ -z "$fork_remote" -a -n "$github_repo" ] ; then - fork_remote="$github_repo" - fi - true "${fork_remote:=origin}" - if [ -z "$upstream_remote" -a -n "$ceph_repo" ] ; then - upstream_remote="$ceph_repo" - fi - true "${upstream_remote:=upstream}" - true "${redmine_endpoint:="https://tracker.ceph.com"}" - true "${github_endpoint:="https://github.com/ceph/ceph"}" - debug "Redmine user: ${redmine_user_id}" - debug "GitHub user: ${github_user}" - debug "Checking fork remote ->${fork_remote}<-" - local fork_remote_exists=$(git remote -v | egrep ^${fork_remote}\\s+) - if [ "$fork_remote_exists" ] ; then - true # remote exists; good +function init_github_user { + debug "Initializing mandatory variables - GitHub user" + if [ "$github_user" ] ; then + true else - error "git remote ->$fork_remote<- not found" - info "(Hint: are you setting fork_remote as described in the documentation?)" - exit 1 + warning "github_user variable not set, falling back to \$USER" + github_user="$USER" + if [ "$github_user" ] ; then + true + else + failed_mandatory_var_check github_user + info "$how_to_get_setup_advice" + exit 1 + fi fi - debug "Checking upstream remote ->${upstream_remote}<-" - local upstream_remote_exists=$(git remote -v | egrep ^${upstream_remote}\\s+) - if [ "$upstream_remote_exists" ]; then - true # remote exists; good - else - error "git remote ->$upstream_remote<- not found" - info "(Hint: are you setting upstream_remote as described in the documentation?)" - exit 1 +} + +function init_mandatory_vars { + debug "Initializing mandatory variables - endpoints" + redmine_endpoint="${redmine_endpoint:-"https://tracker.ceph.com"}" + github_endpoint="${github_endpoint:-"https://github.com/ceph/ceph"}" + debug "Initializing mandatory variables - GitHub remotes" + upstream_remote="${upstream_remote:-$(deduce_remote upstream)}" + fork_remote="${fork_remote:-$(deduce_remote fork)}" +} + +function setup_summary { + local not_set="!!! NOT SET !!!" + local redmine_user_id_display="$not_set" + local redmine_key_display="$not_set" + local github_user_display="$not_set" + local github_token_display="$not_set" + local upstream_remote_display="$not_set" + local fork_remote_display="$not_set" + [ "$redmine_user_id" ] && redmine_user_id_display="$redmine_user_id" + [ "$redmine_key" ] && redmine_key_display="(OK, not shown)" + [ "$github_user" ] && github_user_display="$github_user" + [ "$github_token" ] && github_token_display="(OK, not shown)" + [ "$upstream_remote" ] && upstream_remote_display="$upstream_remote" + [ "$fork_remote" ] && fork_remote_display="$fork_remote" + debug Re-checking mandatory variables + test "$redmine_key" || failed_mandatory_var_check redmine_key + test "$redmine_user_id" || failed_mandatory_var_check redmine_user_id + test "$github_user" || failed_mandatory_var_check github_user + test "$github_token" || failed_mandatory_var_check github_token + test "$upstream_remote" || failed_mandatory_var_check upstream_remote + test "$fork_remote" || failed_mandatory_var_check fork_remote + test "$redmine_endpoint" || failed_mandatory_var_check redmine_endpoint + test "$github_endpoint" || failed_mandatory_var_check github_endpoint + if [ "$SETUP_ONLY" ] ; then + read -r -d '' setup_summary < /dev/null 2>&1 +redmine_user_id $redmine_user_id_display +redmine_key $redmine_key_display +github_user $github_user_display +github_token $github_token_display +upstream_remote $upstream_remote_display +fork_remote $fork_remote_display +EOM + log bare "================================" + log bare "Setup summary" + log bare "--------------------------------" + log bare "variable name value" + log bare "--------------------------------" + log bare "$setup_summary" + log bare "================================" + elif [ "$VERBOSE" ] ; then + debug "redmine_user_id $redmine_user_id_display" + debug "redmine_key $redmine_key_display" + debug "github_user $github_user_display" + debug "github_token $github_token_display" + debug "upstream_remote $upstream_remote_display" + debug "fork_remote $fork_remote_display" fi } @@ -260,7 +237,7 @@ function log { verbose_only="1" ;; esac - if [ "$verbose_only" -a ! "$verbose" ] ; then + if [ "$verbose_only" -a -z "$VERBOSE" ] ; then true else msg="${prefix}${msg}" @@ -285,7 +262,7 @@ function milestone_number_from_remote_api { info "Valid values are $(curl -s -X GET 'https://api.github.com/repos/ceph/ceph/milestones?access_token='$github_token | jq '.[].title')" info "(This probably means the Release field of ${redmine_url} is populated with" info "an unexpected value - i.e. it does not match any of the GitHub milestones.)" - exit 1 + false fi } @@ -306,71 +283,72 @@ function populate_original_pr { fi } -function prepare { - local offset=0 - populate_original_issue - if [ -z "$original_issue" ] ; then - error "Could not find original issue" - info "Does ${redmine_url} have a \"Copied from\" relation?" - exit 1 - fi - info "Parent issue: ${redmine_endpoint}/issues/${original_issue}" +function setup_advice { + cat < "API access key" -> "Show" +redmine_user_id # "Logged in as foobar", click on foobar link, Redmine User ID + # is in the URL, i.e. https://tracker.ceph.com/users/[redmine_user_id] +github_token # https://github.com/settings/tokens -> Generate new token -> + # ensure it has "Full control of private repositories" scope +github_user # Your github username - debug "Initializing local branch $local_branch to $milestone" - if git show-ref --verify --quiet refs/heads/$local_branch ; then - error "Cannot initialize $local_branch - local branch already exists" - exit 1 - else - git checkout $upstream_remote/$milestone -b $local_branch - fi +The above variables must be set explicitly, as the script has no way of +determining reasonable defaults. If you prefer, you can ensure the variables +are set in the environment before running the script. Alternatively, you can +create a file, \$HOME/bin/backport_common.sh (this exact path), with the +variable assignments in it. The script will detect that the file exists and +"source" it. - debug "Fetching latest commits from ${original_pr_url}" - git fetch $upstream_remote pull/$original_pr/head:pr-$original_pr +In any case, care should be taken to keep the values of redmine_key and +github_token secret. - info "Attempting to cherry pick $number commits from ${original_pr_url} into local branch $local_branch" - let offset=$number-1 || true # don't fail on set -e when result is 0 - for ((i=$offset; i>=0; i--)) ; do - debug "Cherry-picking commit $(git log --oneline --max-count=1 --no-decorate pr-$original_pr~$i)" - if git cherry-pick -x "pr-$original_pr~$i" ; then - true - else - if [ "$VERBOSE" ] ; then - git status - fi - error "Cherry pick failed" - info "Next, manually fix conflicts and complete the current cherry-pick" - if [ "$i" -gt "0" ] >/dev/null 2>&1 ; then - info "Then, cherry-pick the remaining commits from ${original_pr_url}, i.e.:" - for ((j=$i-1; j>=0; j--)) ; do - info "-> missing commit: $(git log --oneline --max-count=1 --no-decorate pr-$original_pr~$j)" - done - info "Finally, re-run the script" - else - info "Then re-run the script" - fi - exit 1 - fi - done - info "Cherry picking completed without conflicts" +The script expects to run in a local clone of a Ceph repo with +at least two remotes defined - pointing to: + + https://github.com/ceph/ceph.git + https://github.com/\$github_user/ceph.git + +In other words, the upstream GitHub repo and the user's fork thereof. It makes +no difference what these remotes are called - the script will determine the +right remote names automatically. + +To find out whether you have any obvious problems with your setup before +actually using the script to stage a backport, run: + + ${this_script} --setup + +EOM +} + +function troubleshooting_advice { + cat <&2 +Documentation: + + ${this_script} --setup-advice | less + ${this_script} --usage-advice | less + ${this_script} --troubleshooting-advice | less + +Usage: + ${this_script} [OPTIONS] BACKPORT_TRACKER_ISSUE_NUMBER + +Options: + -c/--component COMPONENT (for use with --set-milestone) + --debug (turns on "set -x") + -m/--set-milestone (requires elevated GitHub privs) + -s/--setup (just check the setup) + -v/--verbose + +Example: + ${this_script} -c dashboard -m 31459 + (if cherry-pick conflicts are present, finish cherry-picking phase manually + and run the script again with the same arguments) + +CAVEAT: The script must be run from inside a local git clone. +EOM +} + +function usage_advice { + cat </dev/null 2>&1 ; then debug "In a local git clone. Good." else error "This script must be run from inside a local git clone" - exit 1 + false fi @@ -432,34 +477,53 @@ fi # process command-line arguments # -munged_options=$(getopt -o c:dhmpv --long "component:,debug,help,prepare,set-milestone,verbose" -n "$this_script" -- "$@") +munged_options=$(getopt -o c:dhmpsv --long "component:,debug,help,prepare,set-milestone,setup,setup-advice,troubleshooting-advice,usage-advice,verbose" -n "$this_script" -- "$@") eval set -- "$munged_options" +ADVICE="" DEBUG="" SET_MILESTONE="" EXPLICIT_COMPONENT="" EXPLICIT_PREPARE="" HELP="" ISSUE="" +SETUP_ADVICE="" +SETUP_ONLY="" +TROUBLESHOOTING_ADVICE="" +USAGE_ADVICE="" VERBOSE="" while true ; do case "$1" in --component|-c) shift ; EXPLICIT_COMPONENT="$1" ; shift ;; --debug|-d) DEBUG="$1" ; shift ;; - --help|-h) HELP="$1" ; shift ;; + --help|-h) ADVICE="1" ; HELP="$1" ; shift ;; --prepare|-p) EXPLICIT_PREPARE="$1" ; shift ;; --set-milestone|-m) SET_MILESTONE="$1" ; shift ;; + --setup|-s) SETUP_ONLY="$1" ; shift ;; + --setup-advice) ADVICE="1" ; SETUP_ADVICE="$1" ; shift ;; + --troubleshooting-advice) ADVICE="$1" ; TROUBLESHOOTING_ADVICE="$1" ; shift ;; + --usage-advice) ADVICE="$1" ; USAGE_ADVICE="$1" ; shift ;; --verbose|-v) VERBOSE="$1" ; shift ;; --) shift ; ISSUE="$1" ; break ;; *) echo "Internal error" ; false ;; esac done +if [ "$ADVICE" ] ; then + [ "$HELP" ] && usage + [ "$SETUP_ADVICE" ] && setup_advice + [ "$USAGE_ADVICE" ] && usage_advice + [ "$TROUBLESHOOTING_ADVICE" ] && troubleshooting_advice + exit 0 +fi + +[ "$SETUP_ONLY" ] && ISSUE="0" if [[ $ISSUE =~ ^[0-9]+$ ]] ; then issue=$ISSUE else error "Invalid or missing argument" - usage # does not return + usage + false fi if [ "$DEBUG" ]; then @@ -467,11 +531,8 @@ if [ "$DEBUG" ]; then VERBOSE="--verbose" fi -if [ "$HELP" ]; then - usage # does not return -fi - if [ "$VERBOSE" ]; then + info "Verbose mode ON" VERBOSE="--verbose" fi @@ -482,7 +543,29 @@ fi BACKPORT_COMMON=$HOME/bin/backport_common.sh [ -f "$BACKPORT_COMMON" ] && source $HOME/bin/backport_common.sh +setup_ok="1" +init_github_user init_mandatory_vars +setup_summary +if [ "$setup_ok" ] ; then + if [ "$SETUP_ONLY" ] ; then + log bare "Overall setup is OK" + exit 0 + elif [ "$VERBOSE" ] ; then + debug "Overall setup is OK" + fi +else + if [ "$SETUP_ONLY" ] ; then + log bare "Setup is NOT OK" + log bare "$how_to_get_setup_advice" + false + else + error "Problem detected in your setup" + info "Run the script with --setup for a full report" + info "$how_to_get_setup_advice" + false + fi +fi # @@ -499,7 +582,7 @@ if [ "$tracker" = "Backport" ]; then else error "Issue $redmine_url is not a Backport" info "(This script only works with Backport tracker issues.)" - exit 1 + false fi debug "Looking up release/milestone of $redmine_url" @@ -508,7 +591,7 @@ if [ "$milestone" ] ; then debug "Release/milestone: $milestone" else error "could not obtain release/milestone from ${redmine_url}" - exit 1 + false fi tracker_title=$(echo $remote_api_output | jq -r '.issue.subject') @@ -544,19 +627,17 @@ local_branch=wip-${issue}-${target_branch} if git show-ref --verify --quiet refs/heads/$local_branch ; then if [ "$EXPLICIT_PREPARE" ] ; then error "local branch $local_branch already exists -- cannot --prepare" - exit 1 + false fi - warning "local branch $local_branch already exists: skipping cherry-pick phase!" - PREPARE="" + info "local branch $local_branch already exists: skipping cherry-pick phase" else info "$local_branch does not exist: will create it and attempt automated cherry-pick" - PREPARE="--prepare" + cherry_pick_phase fi -[ "$PREPARE" ] && prepare # -# at this point, local branch exists and is assumed to contain cherry-pick(s) +# PR phase # current_branch=$(git rev-parse --abbrev-ref HEAD) @@ -600,7 +681,7 @@ number=$(echo "$remote_api_output" | jq -r .number) if [ "$number" = "null" ] ; then error "Remote API call failed" log bare "$remote_api_output" - exit 1 + false fi if [ "$SET_MILESTONE" -o "$CEPH_BACKPORT_SET_MILESTONE" ] ; then