ceph-backport.sh: do remote API call to vet the github token

Before, the script was satisfied if the variable github_token was set to *any*
value. With this commit, it is only satisfied if the GitHub API recognizes it as
a valid token.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph-backport.sh: introduce BACKPORT_COMMON env var

Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph-backport.sh: better --verbose handling

Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph-backport.sh: drop unused --set-milestone option

Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph-backport.sh: implement --milestones feature

When --milestones is provided, the script will:

1. determine the set of active milestones by querying GitHub API
2. for each active milestone, get all PRs targeting the milestone, regardless of
   actual milestone setting
3. for each PR targeting an active milestone, check that actual milestone
   setting matches the base branch
4. if a problem is detected, fix it and "flag" the PR
5. at the end, print a report summarizing any problems that were detected

Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph-backport.sh: implement stricter vetting of backport issue

Refuse to work on a backport that is

* not in "New" or "Need More Info" status
* targeting a release that is not among the GitHub active milestones

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #30974 from tchaikov/wip/do_cmake/enable-amqp

do_cmake.sh: enable amqp and rdma for EL8

Reviewed-by: Yuval Lifshitz <yuvalif@yahoo.com>

Merge pull request #31044 from tchaikov/wip/pybind/rados-py3

pybind/rados: do not slice zip()

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #30710 into master

* refs/pull/30710/head:
mds: add configurable snapshot limit

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #31065 from changchengx/mon_doc

doc: fix typo to auto scale pg number

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #30887 into master

* refs/pull/30887/head:
mgr/volumes: refactor dir handle cleanup

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc: fix typo to auto scale pg number

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

Merge PR #30772 into master

* refs/pull/30772/head:
cmake/modules/GetGitRevisionDescription: update to work with git-worktree

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #31003 into master

* refs/pull/31003/head:
 test/pybind/test_rados.py: test test_aio_remove

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #31052 into master

* refs/pull/31052/head:
auth/cephx/CephxClientHandler: handle decode errors
auth/cephx/CephxProtocol: handle decode errors in CephXTicketHandler::verify_service_ticket_reply
auth/cephx/CephxServiceHandler: handle decode errors

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #30837 from Amrita42/patch-2

doc/start/hardware-recommendations.rst: minor tweaks

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31060 into master

* refs/pull/31060/head:
ceph-daemon: make rm-cluster clear out previously failed units
ceph-daemon: fix get_config_and_both_keyrings() return value

Merge PR #31007 into master

* refs/pull/31007/head:
mgr/progress: remove "since" from duration string

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge PR #30975 into master

* refs/pull/30975/head:
rpm: make librados2, libcephfs2 own (create) /etc/ceph

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-daemon: make rm-cluster clear out previously failed units

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: fix get_config_and_both_keyrings() return value

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #31022 from drunkard/master

doc: add missed word 'than' in doc/man/8/rbd.rst

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #30826 into master

* refs/pull/30826/head:
ceph-daemon: deploy: behave when crash keyring isn't provided
ceph-daemon: simplify calls from rm_cluster
ceph-daemon: explicitly wait for mon to start
ceph-daemon: bootstrap finish message
ceph-daemon: shell: LANG=C
ceph-daemon: verify mon IP is valid before continuing
ceph-daemon: cleanly capture stdout, err; log
ceph-daemon: --debug for more log detail
test_ceph_daemon: default to test on 127.0.0.1; accept ip as arg
ceph-daemon: bootstrap: assimilate and minimize input config file

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

auth/cephx/CephxClientHandler: handle decode errors

Signed-off-by: Sage Weil <sage@redhat.com>

auth/cephx/CephxProtocol: handle decode errors in CephXTicketHandler::verify_service_ticket_reply

Signed-off-by: Sage Weil <sage@redhat.com>

auth/cephx/CephxServiceHandler: handle decode errors

Decode errors should trigger EPERM, not throw exceptions that trigger
timeouts/retries in the messenger.

Fixes: https://tracker.ceph.com/issues/42369
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #30867 from rishabh-d-dave/cv-api-lvm

ceph-volume: rearrange api/lvm.py

 test/pybind/test_rados.py: test test_aio_remove

Signed-off-by: Zhang Jiao <zhangjiao@cmss.chinamobile.com>

doc/start/hardware-recommendations.rst: minor tweaks

* Hard Disk Drive: Replaced word 'accelerate' with 'optimize'
* Networks: Replaced word 'have' with 'has'

Fixes: https://tracker.ceph.com/issues/42379
Signed-off-by: Amrita Sakthivel <asakthiv@redhat.com>

rpm: make librados2, libcephfs2 own (create) /etc/ceph

Fixes: https://tracker.ceph.com/issues/42352
Signed-off-by: Nathan Cutler <ncutler@suse.com>

test/pybind/test_rados: always pass bytes to rados.Object.set_xattr()

> an empty '' is of type str, not bytes.

python3 says. so let's be more explicit.

this change address failures like:
```
======================================================================
ERROR: test_rados.TestIoctx.test_obj_xattrs
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/nose/case.py", line 197, in
runTest
    self.test(*self.arg)
  File "/var/ssd/ceph/src/test/pybind/test_rados.py", line 380, in
test_obj_xattrs
    obj.set_xattr(key, value)
  File "rados.pyx", line 4119, in rados.set_object_locator.retfunc
  File "rados.pyx", line 4129, in rados.set_object_namespace.retfunc
  File "rados.pyx", line 4204, in rados.Object.set_xattr
  File "rados.pyx", line 572, in rados.requires.wrapper.validate_func
  File "rados.pyx", line 560, in rados.requires.check_type
TypeError: xattr_value must be bytes
```

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/rados: do not slice zip()

in python2, zip() returns a list. while in python3, zip() returns an
iterator. so we cannot slice the return value of zip(...) anymore. let's
just materialized the iterator before slicing it.

this change address the failure of
```
ERROR: test_rados.TestIoctx.test_applications
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/nose/case.py", line 197, in
runTest
    self.test(*self.arg)
  File "/var/ssd/ceph/src/test/pybind/test_rados.py", line 879, in
test_applications
    eq([], self.ioctx.application_metadata_list("app1"))
  File "rados.pyx", line 4074, in rados.Ioctx.application_metadata_list
TypeError: 'zip' object is unsliceable
```

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/volumes: refactor dir handle cleanup

introduce with statement in rmtree. This change
simplifies the code's handling of directory cleanup.

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge PR #30505 into master

* refs/pull/30505/head:
test/libcephfs: introduce (uint64_t)ceph_pthread_self()
test/libcephfs: Only use sys/xattr.h on Linux
cephfs: Create a separate dirent{} for FreeBSD
test/fs: Only use features.h on Linux

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #30813 into master

* refs/pull/30813/head:
qa: get rid of iteritems for python3 compatibility

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #30886 into master

* refs/pull/30886/head:
mgr/volumes: cleanup leftovers from earlier purge job implementation

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #30971 into master

* refs/pull/30971/head:
qa: whitelist "Error recovering journal" for cephfs-data-scan

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #30986 into master

* refs/pull/30986/head:
qa: allow client mount to reset fully

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #30897 from tchaikov/wip-bluestore/avl-allocator

os/bluestore: AVL-tree & extent - based space allocator

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

ceph-daemon: deploy: behave when crash keyring isn't provided

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: simplify calls from rm_cluster

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: explicitly wait for mon to start

All this does is make things easier to understand if there is a failure
that prevents the mon from responding.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: bootstrap finish message

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: shell: LANG=C

This gets rid of all these warnings:

bash: warning: setlocale: LC_CTYPE: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_COLLATE: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_MESSAGES: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_NUMERIC: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_TIME: cannot change locale (en_US.UTF-8): No such file or directory

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: verify mon IP is valid before continuing

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: cleanly capture stdout, err; log

- New Popen wrapper that grabs stdout, stderr, and return code.  This is
adapted from ceph-volume's process.py helper(s).

- Optionally log output to logger

- By default, be more verbose if the command fails.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: --debug for more log detail

Signed-off-by: Sage Weil <sage@redhat.com>

test_ceph_daemon: default to test on 127.0.0.1; accept ip as arg

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: bootstrap: assimilate and minimize input config file

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #30957 from tchaikov/wip/crimson/mon/client

crimson/{osd,mon}: lower log level when sending a replicated op

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/mon: skip CEPHX_V2 challenge if client doesn't support it

port 321548010578d6ff7bbf2e5ce8a550008b131423 from src/mon/MonClient

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: lower log level when sending a replicated op

it's but normal i/o, no need to print it using "info"

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: add missed word 'than' in doc/man/8/rbd.rst

Signed-off-by: Drunkard Zhang <gongfan193@gmail.com>

Merge pull request #30925 from cyx1231st/wip-seastar-msgr-policies

test/crimson: verify msgr v2 behavior with different policies

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa: allow client mount to reset fully

... without this there is a sutle race where it takes a
bit of time after a hard reset of a client mount causing
further checks to fail as the (still up) client is still
connected to the MDS.

Fixes: http://tracker.ceph.com/issues/42213
Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #31008 from SUSE/wip-doc-telemetry-default-interval

doc/mgr/telemetry: update default interval

Merge pull request #30965 from idryomov/wip-krbd-udev-socket-overrun

krbd: avoid udev netlink socket overrun

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc/mgr/telemetry: update default interval

Commit 712987d533 changed the default interval to 24h;
updating the docs to match (this also should go to
the Nautilus branch as that commit landed there too
in https://github.com/ceph/ceph/pull/27709)

Signed-off-by: Tim Serong <tserong@suse.com>

mgr/progress: remove "since" from duration string

quote from the description of the tracker ticket:

> This is a duration and therefore "since" doesn't make sense since it's
> not the start of the event.

Fixes: https://tracker.ceph.com/issues/40976
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30973 from tchaikov/wip/qa/tasks/ceph_manager

qa/tasks/ceph_manager: do not panic if "pg_num_target" is missing

Reviewed-by: Neha Ojha <nojha@redhat.com>

qa/tasks/ceph: tolerate 'T' or ' ' as date and time separator

str.replace() does not change the string in-place, so we need to assign
its return value to `t`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/ceph_manager: do not panic of "pg_num_target" is missing

we don't have "pg_num_target" in "osd dump" back in mimic, so we don't
need to check it if it is missing when performing upgrade test.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30947 from dillaman/wip-42332

common/ceph_context: avoid unnecessary wait during service thread shutdown

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30840 from majianpeng/async-messenger-narrow-lock

msg/async: narrow scope of AsyncMessenger::lock in fun connect_to.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #30883 from majianpeng/msg-remove-submit-message

msg/async: move submit_message() into send_to()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30695 from dsavineau/ceph_cli_cephconf_rc

ceph.in: check ceph-conf returncode

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30758 from kshtsk/wip-python3-print

tests: use python3 compatible print

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30987 from trociny/wip-42370

mgr/balancer: python3 compatibility issue

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30705 from smithfarm/wip-sp-backporting-2

doc: split up SubmittingPatches.rst

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #30989 from tchaikov/wip/ceph-crash/logger-name

ceph-crash: use "ceph-crash" as logger name

qa: add script to test udev event reaping

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

krbd: increase udev netlink socket receive buffer to 2M

Even though with the previous commit we no longer block between binding
the socket and starting handling events, we still want a larger receive
buffer to accommodate for scheduling delays.  Since the filtering is
done in the listener, an estimate focused on just rbd is not accurate,
but anyway: a pair of "rbd" and "block" events for "rbd map" take 2048
bytes in the receive buffer.  This allows for roughly a thousand of
them ("rbd map" and "rbd unmap" require root and libudev makes use of
SO_RCVBUFFORCE so rmem_max limit is ignored).

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

ceph-crash: use "ceph-crash" as logger name

"ceph-crash" is a better name than "__main__", we don't use ceph-crash
as a module, so its `__name__` is always `__main__`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/balancer: python3 compatibility issue

Fixes: https://tracker.ceph.com/issues/42370
Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #30701 from pdvian/wip-fix-target-size-bytes

mon/OSDMonitor: Fix pool set target_size_bytes (etc) with unit suffix

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #30870 from rosinL/wip-addlog-setnumaaffinity

osd/osd: add an err log to set_numa_affinty

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30549 from aclamk/fix-garbled-wal

os/bluestore/BlueFS: clear newly allocated space for WAL logs

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #30978 from idryomov/wip-krbd-modprobe

krbd: modprobe before calling build_map_buf()

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

ceph-volume: make get_pv() accepts list of PVs

...so that the code calling it can pass its copy which would prevent
creating a new one within get_pv().

Signed-off-by: Rishabh Dave <ridave@redhat.com>

ceph-volume: make get_vg() accepts list of VGs

...so that the code calling it can pass its copy which would prevent
creating a new one within get_vg().

Signed-off-by: Rishabh Dave <ridave@redhat.com>

ceph-volume: rearrange api/lvm.py

Move code for PVs, VGs and LVs, individually, next to each other.

Fixes: https://tracker.ceph.com/issues/42368
Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #30790 from kshtsk/wip-python3-print-ceph-volume

ceph-volume: use python3 compatible print

Merge pull request #30829 from kshtsk/wip-misc-drop-legacy-code-in-skeleton_config

tasks/ceph: drop testdir replacement in skeleton_config

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30852 from dzafman/wip-42115

osd: Turn off repair pg state when leaving recovery

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30646 from shyukri/wip-qa-mgr-balancer

qa/mgr/balancer: Add cram based test for altering target_max_misplaced_ratio setting

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

doc: split up SubmittingPatches.rst

Split the existing SubmittingPatches.rst into three files:

* SubmittingPatches.rst (general guidelines, for GitHub/master branch)
* SubmittingPatches-kernel.rst (for kernel patches)
* SubmittingPatches-backports.rst (for backports)

Fixes: http://tracker.ceph.com/issues/20953
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #30293 from rosinL/wip-fix-kvstorebench

kv_store_bench: fix teuthology_tests() return value

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #30833 from majianpeng/async-batch-counter

msg/async: remove unsued code.

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #30288 from ricardoasmarques/display-uuid-lunid

mgr/dashboard: Display WWN and LUN number in iSCSI target details

Reviewed-by: Tiago Melo <tmelo@suse.com>

krbd: modprobe before calling build_map_buf()

Otherwise add_key() in set_kernel_secret() fails as if running against
an ancient kernel and we fall back to secret= in options for the first
image being mapped on the machine.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

do_cmake.sh: enable amqp and rdma for EL8

the PowerTools repo now has the related dependencies, so let's enable
them.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30940 from cbodley/wip-qa-rgw-crypt

qa/rgw: refactor the kms backend configuration

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

qa: whitelist "Error recovering journal" for cephfs-data-scan

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
Fixes: https://tracker.ceph.com/issues/41836

objectstore/test/aging: Added avl, added more test cases, added reporting free score.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

qa: add AvlAllocator test script

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

src/test/objectstore/Allocator_{test,bench}.cc: test AvlAllocator too

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: implement avl-extent-based allocator

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30937 from rjfd/wip-dashboard-fix-home-py2

mgr/dashboard: fix python2 failure in home controller

Reviewed-by: Tiago Melo <tmelo@suse.com>

mds: add configurable snapshot limit

Added config option "mds_max_snaps_per_dir" defaulting to 100

Fixes: https://tracker.ceph.com/issues/41209
Signed-off-by: Milind Changire <mchangir@redhat.com>

Merge pull request #30930 from majianpeng/osd-pg-remove-unused-code

osd/PG: remove unused code

Reviewed-by: Kefu Chai <kchai@redhat.com>