test/pybind/test_rados: always pass bytes to rados.Object.set_xattr()

> an empty '' is of type str, not bytes.

python3 says. so let's be more explicit.

this change address failures like:
```
======================================================================
ERROR: test_rados.TestIoctx.test_obj_xattrs
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/nose/case.py", line 197, in
runTest
    self.test(*self.arg)
  File "/var/ssd/ceph/src/test/pybind/test_rados.py", line 380, in
test_obj_xattrs
    obj.set_xattr(key, value)
  File "rados.pyx", line 4119, in rados.set_object_locator.retfunc
  File "rados.pyx", line 4129, in rados.set_object_namespace.retfunc
  File "rados.pyx", line 4204, in rados.Object.set_xattr
  File "rados.pyx", line 572, in rados.requires.wrapper.validate_func
  File "rados.pyx", line 560, in rados.requires.check_type
TypeError: xattr_value must be bytes
```

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/rados: do not slice zip()

in python2, zip() returns a list. while in python3, zip() returns an
iterator. so we cannot slice the return value of zip(...) anymore. let's
just materialized the iterator before slicing it.

this change address the failure of
```
ERROR: test_rados.TestIoctx.test_applications
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/nose/case.py", line 197, in
runTest
    self.test(*self.arg)
  File "/var/ssd/ceph/src/test/pybind/test_rados.py", line 879, in
test_applications
    eq([], self.ioctx.application_metadata_list("app1"))
  File "rados.pyx", line 4074, in rados.Ioctx.application_metadata_list
TypeError: 'zip' object is unsliceable
```

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30897 from tchaikov/wip-bluestore/avl-allocator

os/bluestore: AVL-tree & extent - based space allocator

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #30957 from tchaikov/wip/crimson/mon/client

crimson/{osd,mon}: lower log level when sending a replicated op

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/mon: skip CEPHX_V2 challenge if client doesn't support it

port 321548010578d6ff7bbf2e5ce8a550008b131423 from src/mon/MonClient

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: lower log level when sending a replicated op

it's but normal i/o, no need to print it using "info"

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30925 from cyx1231st/wip-seastar-msgr-policies

test/crimson: verify msgr v2 behavior with different policies

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31008 from SUSE/wip-doc-telemetry-default-interval

doc/mgr/telemetry: update default interval

Merge pull request #30965 from idryomov/wip-krbd-udev-socket-overrun

krbd: avoid udev netlink socket overrun

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc/mgr/telemetry: update default interval

Commit 712987d533 changed the default interval to 24h;
updating the docs to match (this also should go to
the Nautilus branch as that commit landed there too
in https://github.com/ceph/ceph/pull/27709)

Signed-off-by: Tim Serong <tserong@suse.com>

Merge pull request #30973 from tchaikov/wip/qa/tasks/ceph_manager

qa/tasks/ceph_manager: do not panic if "pg_num_target" is missing

Reviewed-by: Neha Ojha <nojha@redhat.com>

qa/tasks/ceph: tolerate 'T' or ' ' as date and time separator

str.replace() does not change the string in-place, so we need to assign
its return value to `t`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/ceph_manager: do not panic of "pg_num_target" is missing

we don't have "pg_num_target" in "osd dump" back in mimic, so we don't
need to check it if it is missing when performing upgrade test.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30947 from dillaman/wip-42332

common/ceph_context: avoid unnecessary wait during service thread shutdown

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30840 from majianpeng/async-messenger-narrow-lock

msg/async: narrow scope of AsyncMessenger::lock in fun connect_to.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #30883 from majianpeng/msg-remove-submit-message

msg/async: move submit_message() into send_to()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30695 from dsavineau/ceph_cli_cephconf_rc

ceph.in: check ceph-conf returncode

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30758 from kshtsk/wip-python3-print

tests: use python3 compatible print

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30987 from trociny/wip-42370

mgr/balancer: python3 compatibility issue

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30705 from smithfarm/wip-sp-backporting-2

doc: split up SubmittingPatches.rst

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #30989 from tchaikov/wip/ceph-crash/logger-name

ceph-crash: use "ceph-crash" as logger name

qa: add script to test udev event reaping

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

krbd: increase udev netlink socket receive buffer to 2M

Even though with the previous commit we no longer block between binding
the socket and starting handling events, we still want a larger receive
buffer to accommodate for scheduling delays.  Since the filtering is
done in the listener, an estimate focused on just rbd is not accurate,
but anyway: a pair of "rbd" and "block" events for "rbd map" take 2048
bytes in the receive buffer.  This allows for roughly a thousand of
them ("rbd map" and "rbd unmap" require root and libudev makes use of
SO_RCVBUFFORCE so rmem_max limit is ignored).

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

ceph-crash: use "ceph-crash" as logger name

"ceph-crash" is a better name than "__main__", we don't use ceph-crash
as a module, so its `__name__` is always `__main__`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/balancer: python3 compatibility issue

Fixes: https://tracker.ceph.com/issues/42370
Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #30701 from pdvian/wip-fix-target-size-bytes

mon/OSDMonitor: Fix pool set target_size_bytes (etc) with unit suffix

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #30870 from rosinL/wip-addlog-setnumaaffinity

osd/osd: add an err log to set_numa_affinty

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30549 from aclamk/fix-garbled-wal

os/bluestore/BlueFS: clear newly allocated space for WAL logs

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #30978 from idryomov/wip-krbd-modprobe

krbd: modprobe before calling build_map_buf()

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #30790 from kshtsk/wip-python3-print-ceph-volume

ceph-volume: use python3 compatible print

Merge pull request #30829 from kshtsk/wip-misc-drop-legacy-code-in-skeleton_config

tasks/ceph: drop testdir replacement in skeleton_config

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30852 from dzafman/wip-42115

osd: Turn off repair pg state when leaving recovery

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30646 from shyukri/wip-qa-mgr-balancer

qa/mgr/balancer: Add cram based test for altering target_max_misplaced_ratio setting

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

doc: split up SubmittingPatches.rst

Split the existing SubmittingPatches.rst into three files:

* SubmittingPatches.rst (general guidelines, for GitHub/master branch)
* SubmittingPatches-kernel.rst (for kernel patches)
* SubmittingPatches-backports.rst (for backports)

Fixes: http://tracker.ceph.com/issues/20953
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #30293 from rosinL/wip-fix-kvstorebench

kv_store_bench: fix teuthology_tests() return value

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #30833 from majianpeng/async-batch-counter

msg/async: remove unsued code.

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #30288 from ricardoasmarques/display-uuid-lunid

mgr/dashboard: Display WWN and LUN number in iSCSI target details

Reviewed-by: Tiago Melo <tmelo@suse.com>

krbd: modprobe before calling build_map_buf()

Otherwise add_key() in set_kernel_secret() fails as if running against
an ancient kernel and we fall back to secret= in options for the first
image being mapped on the machine.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #30940 from cbodley/wip-qa-rgw-crypt

qa/rgw: refactor the kms backend configuration

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

objectstore/test/aging: Added avl, added more test cases, added reporting free score.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

qa: add AvlAllocator test script

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

src/test/objectstore/Allocator_{test,bench}.cc: test AvlAllocator too

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: implement avl-extent-based allocator

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30937 from rjfd/wip-dashboard-fix-home-py2

mgr/dashboard: fix python2 failure in home controller

Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #30930 from majianpeng/osd-pg-remove-unused-code

osd/PG: remove unused code

Reviewed-by: Kefu Chai <kchai@redhat.com>

krbd: avoid udev netlink socket overrun

Because the event(s) we are interested in can be deliveled while we are
still in the kernel finishing map or unmap, we start listening for udev
events before going into the kernel.  However, if (un)mapping takes its
time, udev netlink socket can be fairly easily overrun -- the filtering
is done on the listener side, so we get to process everything, not just
rbd events.  If any of the events of interest get dropped (ENOBUFS), we
hang in poll().

Go into the kernel in a separate thread and leave the main thread to
run the event loop.  The return value is communicated to the reactor
though a pipe.

Fixes: https://tracker.ceph.com/issues/41404
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

krbd: reap all available events before polling again

This also exposes errors from udev_monitor_receive_device() which were
previously ignored.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

krbd: separate event reaping from event processing

Move event processing into UdevMapHandler and UdevUnmapHandler
functors and replace wait_for_udev_{add,remove}() with a single
wait_for_mapping() template.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

krbd: get rid of poll() timeout

This timeout was added as a (very poor) workaround for an issue
addressed in commit 42dd1eae630f ("krbd: fix rbd map hang due to udev
return subsystem unordered").

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #30958 from tchaikov/wip-cmake/dashboard-frontend

cmake: dashboard: enable frontend on arm64

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Lenz Grimmer <LGrimmer@suse.com>

Merge pull request #30944 from cbodley/wip-qa-rgw-whitelist-slow-ops

qa/rgw: whitelist SLOW_OPS failures against ec pools

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #30952 from mxdInspur/trash_restore_info_warn

rbd: delete redundant words when trash restore fails because of same name

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #28952 from MahatiC/wip-ordering-ios

librbd: implement ordering for overlapping IOs

Merge pull request #30953 from tchaikov/wip-doc/rbd

doc/rbd: s/guess/xml/ for codeblock lexer

Reviewed-by: Jason DIllaman <dillaman@redhat.com>

Merge pull request #30942 from dillaman/wip-rbd-qa-suites

qa/suites/rbd: fix errant tab in yaml which is causing parsing failures

Reviewed-by: Mykola Golub <mgolub@suse.com>

cmake: dashboard: move "if(WITH_MGR_DASHBOARD_FRONTEND)" up

two reasons:

* we don't need to run the tox test if WITH_MGR_DASHBOARD_FRONTEND is
  OFF
* better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: dashboard: enable frontend on arm64

it was disable when we added dashboard(v2) to ceph in
24e216b519795ba6a5d427790399a595fbb19b27 because some npm package did
not install or build on arm64. but "mgr-dashboard-frontend-build" now
builds fine at the time of writing. so let's enable it!

Signed-off-by: Kefu Chai <kchai@redhat.com>

rbd:delete redundant words when trash restore failed because of same name

Signed-off-by: Xiangdong Mu <muxiangdong@inspur.com>

doc/rbd: s/guess/xml/ for codeblock lexer

this change silences the warning of

```
doc/rbd/qemu-rbd.rst:174: WARNING: Pygments lexer name 'guess' is not
known
```

see http://pygments.org/docs/lexers/, we should use "xml" for XML .

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/crimson: v2 policy test reconnect after server/acceptor down

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: v2 policy test server/acceptor markdown

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: v2 peer policy test markdown

lossless_peer_reuse policy won't get remote reset compared with
lossless_peer/stateful_server/lossless_client policy when peer is
markdown.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: v2 lossless policy test markdown

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: send message only after getting the connection

It is possible for stateless/stateful server to lose the pending message
due to the remote markdown, causing unexpected test failure.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: don't fail when nothing queued for sweep

Fix a corner case when write is dispatching but the queued events are
cleared by a remote reset.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: v2 stateless server policy test markdown

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: v2 lossy client policy test markdown

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

librbd: implement ordering for overlapping IOs

..and block flushes until previous writes are completed

Signed-off-by: Mahati Chamarthy <mahati.chamarthy@intel.com>

Merge pull request #30927 from zjcmszh/test_remove

test/pybind/rbd.pyx: add test_remove_snap_by_id case in test_rbd.py

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #30865 from zy751713126/config_pool

pybind/rbd: add pool config_set/get/remove api in rbd.pyx

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

common/ceph_context: avoid unnecessary wait during service thread shutdown

Fixes: https://tracker.ceph.com/issues/42332
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/rgw: whitelist SLOW_OPS failures against ec pools

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/suites/rbd: fix errant tab in yaml which is causing parsing failures

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/rgw: use config overrides for vault kms backend

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: use config overrides for barbican kms backend

except for the barbican endpoint, which isn't known until the barbican
task runs

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: use config overrides for 'testing' kms backend

and configure the 'testing' backend for the rgw/verify suite

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30595 from p-se/wip-pna-extend-smart-types

mgr/dashboard: extend types of `smart` response

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: home controller unit test

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: home: fix fallback to default language

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: home: fix python2 failure in regex processing

Signed-off-by: Ricardo Dias <rdias@suse.com>

Merge pull request #30558 from dillaman/wip-41555

rbd-mirror: image status should report remote status

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #30671 from tspmelo/wip-i18n-vars

mgr/dashboard: Fix translation of variables

Reviewed-by: Patrick Seidensal <pnawracay@suse.com>

Merge pull request #30803 from cyx1231st/wip-seastar-msgr-failover

test/crimson: v2 failover tests with ack/keepalive

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30779 from rhcs-dashboard/tox-ini-fixes

mgr/dashboard: tox.ini fixes

Reviewed-by: Ernesto Puertat <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

test/pybind/rbd.pyx: add test_remove_snap_by_id case in test_rbd.py

Signed-off-by: Zhang Jiao <zhangjiao@cmss.chinamobile.com>

osd/PG: remove unused code.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #30600 from yuvalif/boto3_extensions_examples

rgw/examples: adding examples for boto3 extensions to AWS S3

msg/async: clean up code.

Currently, function submit_message only used by send_to, so conbine
them to one.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #30874 from uweigand/add-cycles-ibmz

Cycles: Add support for IBM Z

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #30816 into master

* refs/pull/30816/head:
qa: use small default pg count for CephFS pools
*: remove config of "mon pg warn min per osd" in testing

Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge PR #30641 into master

* refs/pull/30641/head:
pybind/cephfs: add cephfs python API removexattr()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Varsha Rao <varao@redhat.com>

Merge PR #30873 into master

* refs/pull/30873/head:
qa: get rid of iterkeys for py3 compatibility

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge PR #30744 into master

* refs/pull/30744/head:
mds: Reorganize class members in MDLog header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #30206 into master

* refs/pull/30206/head:
test: add test_statx for new statx()
client: add statx in cephfs.pyx

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #29626 into master

* refs/pull/29626/head:
client: fix return code handling from _lseek

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #30585 into master

* refs/pull/30585/head:
docs/cephfs-shell: update doc about pyscript
cephfs-shell: make compatible with cmd2 versions after 0.9.13

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Varsha Rao <varao@redhat.com>

qa: use small default pg count for CephFS pools

The pg count needs to be a power-of-two since
dff5697464edb9931d5dfa08cd4a30f85c1f237e.

Also, mon_pg_warn_min_per_osd is disabled by default now (or set to a
low value in vstart/testing) so there's no need to base the pg count on
this value.

Ideally someday we can remove this so that the default cluster value is
used but we need to keep this for deployments of older versions of Ceph.

Fixes: https://tracker.ceph.com/issues/42228
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

rgw/examples: adding info on boto3 extensions to docs

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>

Merge pull request #30830 from tchaikov/wip/rpm-install-deps/python_provide

install-deps, rpm: use python_provide macro and cleanups

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #30151 from ofriedma/wip-ofriedma-lc-twice

fix rgw lc does not delete objects that do not have exactly the same tags as the rule