backport-resolve-issue: prune duplicate URLs

I have seen PR descriptions with a single backport tracker URL mentioned twice.
When backport-resolve-issue hit one of these, it would say:

Found backport tracker: https://tracker.ceph.com/issues/39272
Found backport tracker: https://tracker.ceph.com/issues/39272
-----------------------------------------------------------------
INFO:root:Tracker https://tracker.ceph.com/issues/39272 links to PR https://github.com/ceph/ceph/pull/29153
INFO:root:Backport Tracker 39272 target version already populated with correct value v12.2.13
INFO:root:Backport Tracker 39272 status is already set to Resolved
-----------------------------------------------------------------
INFO:root:Tracker https://tracker.ceph.com/issues/39272 links to PR https://github.com/ceph/ceph/pull/29153
INFO:root:Backport Tracker 39272 target version already populated with correct value v12.2.13
INFO:root:Backport Tracker 39272 status is already set to Resolved
=================================================================

This commit fixes the issue.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

backport-resolve-issue: recognize that Target version is populated

Due to a regression, the script stopped recognizing that a Backport tracker
issue's "Target version" field is populated.

Fixes: af43b3cc3b05e70ec248ad5c23da6fbd4c2e78a1
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #30988 from smithfarm/wip-bri-multi

backport-resolve-issue: resolve multiple backport issues

Merge pull request #31163 from callithea/wip-issue-42489

doc: Small update of SubmittingPatches-backports

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #30879 from smithfarm/wip-cbs-check-missing-milestone

ceph-backport.sh: implement --milestones feature and more-careful vetting

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #29930 from IlsooByun/rocksdb_ver

cmake: require RocksDB 5.14 or higher

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #30369 into master

* refs/pull/30369/head:
doc/cephfs: add doc for cephfs io path

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31201 into master

* refs/pull/31201/head:
doc: document rank option for journal reset

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #31181 into master

* refs/pull/31181/head:
ceph-daemon: only pass podman -it if need an interactive shell

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #31159 from alfredodeza/wip-rm42484

ceph-volume tests remove xenial from functional testing

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #31222 from sebastian-philipp/python-common-avoid-setup_requires

python-common: avoid using setup_requires in setup.py

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-daemon: only pass podman -it if need an interactive shell

Otherwise, we get errors like

2019-10-26T17:29:42.004 INFO:tasks.workunit.client.0.mira109.stderr:+ sudo /usr/sbin/ceph-daemon shell -- ceph -v
2019-10-26T17:29:42.004 INFO:tasks.workunit.client.0.mira109.stderr:+ grep 'ceph version'
2019-10-26T17:29:42.149 INFO:tasks.workunit.client.0.mira109.stderr:the input device is not a TTY

Fixes: https://tracker.ceph.com/issues/42499
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31059 into master

* refs/pull/31059/head:
mgr/rook: implement update_mds
mgr/ssh: implement 'mds update'
mgr/orchestrator_cli: add 'mds update' command
mgr/ssh: unique names for mds instances
mgr/ssh: name mgrs with random string id; add removal support
mgr/ssh: provide get_unique_name helper
mgr/orchestrator_cli: rename 'mds rm' arg
mgr/orchestrator_cli: rename 'mds add' arg
mgr/ssh: add 'mds rm'
mgr/ssh: (broken) mds rm
mgr/ssh: make mds add work
mgr/orchestrator: add count, hosts args to 'mds add'
mgr/ssh: be consistently verbose
mgr/ActivePyModules: expose 'mds_metadata' to modules

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #31197 from jan--f/c-v-fix-lvm-mokeypatch

ceph-volume: mokeypatch calls to lvm related binaries

python-common: avoid using setup_requires in setup.py

Fixes: https://tracker.ceph.com/issues/42528
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #31023 from idryomov/wip-krbd-udev-enumerate-retry

krbd: retry on transient errors from udev_enumerate_scan_devices()

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

cmake: require RocksDB 5.14 or higher

error: ‘class rocksdb::Env’ has no member named ‘SetAllowNonOwnerAccess’
       opt.env->SetAllowNonOwnerAccess(false);
                    ^~~~~~~~~~~~~~~~~~~~~~

Signed-off-by: Ilsoo Byun <ilsoobyun@linecorp.com>

Merge pull request #31072 from sebastian-philipp/ceph-volume-inventory-device-id

ceph-volume: add Ceph's device id to inventory

Merge PR #31063 into master

* refs/pull/31063/head:
qa: disable too few PG warning during Mimic deploy

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #31178 from tchaikov/wip/rpm/pyyaml

ceph.spec.in: add missing python-yaml dependency for mgr-k8sevents

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge PR #31168 into master

* refs/pull/31168/head:
ceph-daemon: try py2 import before py3
qa/suites/rados/singleton-nomsgr/ceph-daemon: make sure python3 is installed
qa/standalone/test_ceph_damon.sh: test with python2 and python3
mgr/ssh: python, not python3
ceph-daemon: python, not python3
ceph-daemon: os.makedirs
ceph-daemon: configparser is ConfigParser on py2
ceph-daemon: avoid py3-isms

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Alfredo Deza <adeza@redhat.com>

mgr/rook: implement update_mds

Note that this still doesn't provide control over active vs standby MDS
counts.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: implement 'mds update'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator_cli: add 'mds update' command

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: unique names for mds instances

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: name mgrs with random string id; add removal support

- stop naming mgrs after the host, since that doesn't scale well
- add mgr removal support.  prefer mgrs that aren't recognized by the
  cluster.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: provide get_unique_name helper

Signed-off-by: Sage Weil <sage@redhat.com>

doc: document rank option for journal reset

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-daemon: try py2 import before py3

See https://github.com/ceph/ceph-medic/commit/f499a89b15b72a5bb842e1ebbc7cafc4daf58c1c

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/singleton-nomsgr/ceph-daemon: make sure python3 is installed

Centos7 doesn't have it by default.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/standalone/test_ceph_damon.sh: test with python2 and python3

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: python, not python3

python 2.x is okay too.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: python, not python3

python2 is okay too.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: os.makedirs

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: configparser is ConfigParser on py2

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: avoid py3-isms

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator_cli: rename 'mds rm' arg

'name' can be either a group prefix or an instance name, so use a general
variable name.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator_cli: rename 'mds add' arg

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: add 'mds rm'

This can take *either* an mds instance name *or* a group prefix.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: (broken) mds rm

This wrong because it takes a daemon name, not a group name, and because
it relies on mds_metadata--connected daemons, not deployed daemons.  Must
fix once we have better inventory.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: make mds add work

This is clearly primitive and not where we want to end up, but an initial
step.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator: add count, hosts args to 'mds add'

Rook already uses the count; the hosts hint is not relevant.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: be consistently verbose

Print args and stdin in _run_ceph_daemon; drop various debug lines for
printing keyrings and configs along the way.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ActivePyModules: expose 'mds_metadata' to modules

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #31180 from liewegas/revert-bluestore-pinned-list

os/bluestore: revert cache pinned list

Reviewed-by: Mark Nelson <mnelson@redhat.com>

ceph-volume: add Ceph's device id to inventory

This will benefit the orchestrator and dashboard to show a unified view of devices with SMART data

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

ceph-volume: mokeypatch calls to lvm related binaries

Otherwise these tests fail occasionally when run on a system that does
not have lvm installed.

Fixes: https://tracker.ceph.com/issues/42045
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #31071 from sebastian-philipp/python-common-mypy

python-common: Add mypy testing

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #30995 from wjwithagen/wjw-wip-volume-zfs-inventory

ceph-volume-zfs: add the inventory command

Merge pull request #31014 from tchaikov/wip-38378

mgr/dashboard: accept expected exception when SSL handshaking

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #31092 from liewegas/wip-revert-42026-test

Revert "test: librados startup/shutdown racer test"

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #31153 from ceph/wip-restful-node-items

restful: Query nodes_by_id for items

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31176 from tchaikov/wip/mgr/diskprediction_cloud/cleanup

mgr/diskprediction_cloud: refactor timeout() decorator

Reviewed-by: Rick Chen <rick.chen@prophetstor.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #31175 from tchaikov/wip/mon/OSDMonitor/cleanup

mon/OSDMonitor: use initializer_list<> for {si,iec}_options

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #30279 from runsisi/wip-fix-strerror_r

common: fix compat of strerror_r

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29836 from majianpeng/fix-async-msgr-perfcounter-bug

msg/async: Don't dec(msgr_active_connections) if conn still in accept…

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #31156 from cyx1231st/wip-seastar-msgr-testpeer

test/crimson: implement a remote async TestPeer for crimson msgr tests

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/crimson: add nonstop option in async TestPeer

If set, TestPeer won't shutdown if all tests are done, and will wait for
another round of tests.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: add delay to markdown_peer()

Let test sleep awhile for peer markdown propagated.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: async msgr TestPeer for v2 failover tests

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: allow v2 failover tests to connect to remote TestPeer

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: extract msgr test commands to header

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #30610 from ivancich/wip-no-schedule-reshard-during-reshard

rgw: prevent bucket reshard scheduling if bucket is resharding

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #30795 from ivancich/wip-dynamic-reshard-adjustments

rgw: adjust allowable bucket index shard counts for dynamic resharding

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30845 from croit/rgw-bucket-stats-add-num-shards

rgw: add num_shards to radosgw-admin bucket stats

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #30956 from joke-lee/rgw-storage-class-post-obj

rgw: set correct storage class for post object upload

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #30970 from theanalyst/rgw/list-common-prefix

rgw: url encode common prefixes for List Objects response

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #31179 from liewegas/cleanup-exec-bit

os/bluestore,comon,erasure-code: chmod -x source files

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31171 from liewegas/bug-42496

qa/tasks/cbt: run stop-all.sh while shutting down

Reviewed-by: Kefu Chai <kchai@redhat.com>

Revert "Merge pull request #30964 from markhpc/wip-bs-cache-trim-pinned"

This reverts commit 304c37f52165363cad4aaf31386d62ea5a573425, reversing
changes made to 7b4f9a083f0c8cc015f7b0cc12a3bc88db632610.

This causes some bluestore test failures due to an ENOENT right after a
new object is created.  The simplest reproducer is the
ObjectStore/StoreTest.FiemapEmpty/2 test.

Fixes: https://tracker.ceph.com/issues/42495
Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore,comon,erasure-code: chmod -x source files

Fixes

*** WARNING: ./usr/src/debug/ceph-15.0.0-6512.g62bd825.el8.x86_64/src/common/obj_bencher.cc is executable but has empty or no shebang, removing executable bit
*** WARNING: ./usr/src/debug/ceph-15.0.0-6512.g62bd825.el8.x86_64/src/erasure-code/shec/determinant.c is executable but has empty or no shebang, removing executable bit
*** WARNING: ./usr/src/debug/ceph-15.0.0-6512.g62bd825.el8.x86_64/src/os/bluestore/AvlAllocator.cc is executable but has empty or no shebang, removing executable bit
*** WARNING: ./usr/src/debug/ceph-15.0.0-6512.g62bd825.el8.x86_64/src/os/bluestore/AvlAllocator.h is executable but has empty or no shebang, removing executable bit

in rpm build.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31170 into master

* refs/pull/31170/head:
src/telemetry: remove, now lives in ceph-telemetry.git

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #31131 into master

* refs/pull/31131/head:
crush: remove invalid upmap items

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #31154 into master

* refs/pull/31154/head:
auth/cephx: always initialize local variables

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #31167 into master

* refs/pull/31167/head:
os/bluestore: do not mark per_pool_omap updated unless we fixed it

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

mgr/dashboard: accept socket error 0

see
https://github.com/cherrypy/cheroot/commit/9d73226ab28a07a38cbb6816edf9ff46be4f9de3

Fixes: https://tracker.ceph.com/issues/38378
Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: accept exceptions from builtin SSL

see also https://github.com/cherrypy/cheroot/pull/4

so we don't panic when client is trying to talk with us with an
unsupported protocol, the exception should be accepted, and the
client can fallback to supported protocol.

Fixes: https://tracker.ceph.com/issues/38378
Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: extract monkey patches for cherrypy out

so they are less distracting to the dashboard developers. and probably
less scaring.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph.spec.in: add missing python-yaml dependency for mgr-k8sevents

otherwise we might have:

```
ceph/src/pybind/mgr/k8sevents/__init__.py", line 1, in <module>
    from .module import Module
  File "/home/kchai/ceph/src/pybind/mgr/k8sevents/module.py", line 28, in <module>
    import yaml
ImportError: No module named yaml
```

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/diskprediction_cloud: refactor timeout() decorator

* timeout() is never passed any parameter when being called, so let's
  remove the parameters list of "seconds" and "error_message"
* use `getattr()` instead of `hasattr()` for retrieving the
  member variable of `self`
* pass `self` to wrapper function explicitly.
* return `func()` right away.
* hardwire the error message of `TimeoutError` to "Timer expired",
  because
  - as neither errno.ETIME nor errno.ETIMEOUT is portable
  - the only caller of `TimeoutError` is `timeout()`, so there is
    no need to have the flexibility to pass a different error message
* use `wraps()` as a decorator, simpler this way.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/OSDMonitor: use initializer_list<> for {si,iec}_options

* use initializer_list<> for {si,iec}_options, no need to uset set<>
* remove the comments, the variable names are self-documented.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31010 from pdvian/wip-fix-target-max

mon/OSDMonitor.cc: Allow pool set target_max_(objects/bytes) with SI/IEC units

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30434 from smithfarm/wip-41820

qa: enable dashboard tests to be run with "--suite rados/dashboard"

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31005 from tchaikov/wip/qa/tasks/ceph/cleanup

qa/tasks/ceph.py: remove unused variables

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

qa/tasks/cbt: run stop-all.sh when finishing up

stop-all.sh will work if the right deps are there (currently we lack 'nc')

also killall -9 java to be sure.

Fixes: https://tracker.ceph.com/issues/42496
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31086 into master

* refs/pull/31086/head:
doc/bootstrap.rst: fix github's url

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #31169 into master

* refs/pull/31169/head:
mgr/ssh: implement (synchronous) describe_service
ceph-daemon: ls: replace 'active' bool with 'state' enum
ceph-daemon: include container_id and version in 'ls' output
ceph-daemon: tolerate systemctl is-* exit codes
mgr/ssh: modernize timeout config option

Reviewed-by: Sebastian Wagner <swagner@suse.com>

doc/bootstrap.rst: fix github's url

Signed-off-by: Alexandre Bruyelles <jack@jack.fr.eu.org>

Merge pull request #31126 from wjwithagen/wjw-fix-std-bitset

rgw: Select the std::bitset to resolv ambiguity

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

src/telemetry: remove, now lives in ceph-telemetry.git

Signed-off-by: Dan Mick <dan.mick@redhat.com>

mgr/ssh: implement (synchronous) describe_service

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: ls: replace 'active' bool with 'state' enum

('running', 'inactive', 'error', 'unknown')

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: include container_id and version in 'ls' output

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: tolerate systemctl is-* exit codes

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: modernize timeout config option

(and rename it)

Signed-off-by: Sage Weil <sage@redhat.com>

qa: add script to stress udev_enumerate_scan_devices()

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

krbd: retry on an empty list from udev_enumerate_scan_devices()

systemd 219 doesn't have the issue that is worked around in the
previous commit, but has a different one: udev_enumerate_scan_devices()
always succeeds, but sometimes returns an empty list when the device is
actually there.  This happens rarely and at random so I haven't been
able to get to the bottom of it yet, but it looks like another similar
race condition in libudev.

Since an empty list is expected if the device isn't there, retry just
twice with a small sleep in-between.  This appears to be enough: I got
7 occurrences per 600000 "rbd unmap" invocations, all of which needed
a single retry:

  rbd: udev enumerate missed a device, tries = 1

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge PR #31141 into master

* refs/pull/31141/head:
ceph-daemon: fix warning message
ceph-daemon: fix enable with conflicting ceph.target files
test_ceph_daemon.sh: use same fsid as qa/standalone/test_ceph_daemon.sh
test_ceph_daemon.sh: use other latest-master image
ceph-daemon: fix pathify (fix shell --config/--keyring)
ceph-daemon: /var/run/ceph -> /var/run/ceph/$fsid

Reviewed-by: Sebastian Wagner <swagner@suse.com>

ceph-daemon: fix warning message

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #31164 from tchaikov/wip/doc/rados/fix-typo

doc/rados/deployment/ceph-deploy-mon: fix typo

Reviewed-by: Josh Durgin <jdurgin@redhat.com>