ceph_test_watch_notify: try notify 10x if ALLOW_TIMEOUTS is set

See https://tracker.ceph.com/issues/44062

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 2183edcc65c76bff075f3650ea4ebc73edc55b16)

ceph_test_rados_api_misc: ShutdownRace timeout if ALLOW_TIMEOUTS is set

See https://tracker.ceph.com/issues/44582

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 3a57e70028e6226a41ef295217160c038441cd70)

qa/suites/rados/verify: set ALLOW_TIMEOUTS for workunits

When running under valgrind (and thrashing) things can be slow.  Tell
tests in case they need to tolerate timeouts.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 40a7bcea706b72388056e254db752a44d1a59fcc)

qa/suites/rados/verify/validater/valgrind: tolerate SLOW_OPS

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 7c19c1534badf00adf756e3a4e342e3d009a63ca)

qa/suites/rados/verify: whitelist MON_DOWN when using valgrind

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 52d706c75f2c70c80bfc4d8182217d5831bfbf4e)

qa/suites/rados/verify/validater/valgrind: less bluestore logging

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit baeb051910744bfd7a279776afc44c8f9aa6a6e6)

qa/suites/rados/verify/validater: increase heartbeat grace

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 4fda9d50f09d527262fd65eab9b9cff3fd700aad)

Revert "qa/suites/rados/verify/validator/valgrind: debug refs = 5"

This reverts commit 65e81e6eb4f136bf21b67e5de10ab49f028f9e95.

Signed-off-by: Nathan Cutler <ncutler@suse.com>
Conflicts:
    reverted the nautilus commit instead of cherry-picking the revert of
    the master commit

Merge pull request #34640 from rhcs-dashboard/wip-44327-nautilus

nautilus: mgr/dashboard: fix backport #33764

Merge pull request #34450 from rhcs-dashboard/wip-44980-nautilus

nautilus: monitoring: Fix pool capacity incorrect

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Patrick Seidensal <pseidensal@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #34847 from bk201/wip-45273-nautilus

nautilus: mgr/dashboard: fixing RBD purge error in backend

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puertat <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #35016 from rhcs-dashboard/45515-fix-py2-strptime-error

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #31586 from bk201/fix-42651

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

mgr/dashboard: fix py2 strptime ImportError (not thread safe)

Fixes: https://tracker.ceph.com/issues/45515
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #34884 from tchaikov/nautilus-upgrade-mimic-retry

nautilus: qa/tasks/ceph.py: do not use option mimic does not understand

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #34873 from tchaikov/nautilus-qa-upgrade-from-rados-wo-object-size

nautilus: qa/suites/rados: use default objectsize for upgrade tests

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #34870 from badone/wip-ms_dispatch-kcore-selinux-denial

nautilus: selinux: Allow getattr access to /proc/kcore

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #34767 from smithfarm/wip-45212-nautilus

nautilus: client: reset requested_max_size if file write is not wanted

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #34679 from kotreshhr/nautilus-subvolume-info

nautilus: mgr/volumes: Add interface to get subvolume metadata

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #34470 from Vicente-Cheng/wip-44655-nautilus

nautilus: qa: fix various py3 cephfs qa bugs x2

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #34107 from yuvalif/notification-backports-nautilus

nautilus: rgw/notifications: backporting features and bug fix

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #34915 from tchaikov/nautilus-mimic-x-disable-min-pg-per-osd-warning

nautilus: qa/suites/upgrade: disable min pg per osd warning

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #34771 from smithfarm/wip-45210-nautilus

nautilus: ceph-fuse: link to libfuse3 and pass "-o big_writes" to libfuse if libfuse < 3.0.0

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #34817 from tchaikov/nautilus-44668

nautilus: qa: use small default pg count for CephFS pools

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #34563 from Vicente-Cheng/wip-45050-nautilus

nautilus: mgr: force purge normal ceph entities from service map

Reviewed-by: Ramana Raja <rraja@redhat.com>

qa/suites/upgrade: disable min pg per osd warning

disable the TOO_FEW_PGS warning, as
1ac34a5ea3d1aca299b02e574b295dd4bf6167f4 is not backported to mimic, we
will have TOO_FEW_PGS warnings when a healthy cluster is expected when
upgrading from mimic.

this change disables this warning by setting "mon_pg_warn_min_per_osd" to
"0".

this change is not cherry-picked from master. as
1ac34a5ea3d1aca299b02e574b295dd4bf6167f4 is already included by master,
and we don't perform upgrade from mimic on master branch.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa: fix various py3 cephfs qa bugs x2

Fixes: https://tracker.ceph.com/issues/43515
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit ae28ed11d7b9b1dac7dab1e8f5dc0c9643b5de76)

qa: fix various py3 cephfs qa bugs

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 6ea2730ea669b6cf7ac668f606f63628ef8918b7)

Conflicts:
qa/tasks/cephfs/test_pool_perm.py
  - manually modify the py3 compatibility related fix

Merge pull request #34833 from smithfarm/wip-44653-nautilus

nautilus: rgw: increase log level for same or older period pull msg

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #34413 from votdev/wip-42151-nautilus

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #34862 from tchaikov/nautilus-el-7.8

nautilus: qa/distros: point {centos,rhel}_latest.yaml to 7.8

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>
Reviewed-by: David Galloway <dgallowa@redhat.com>

Merge pull request #34832 from smithfarm/wip-45330-nautilus

nautilus: bluestore/bdev: initialize size when creating object

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #34796 from smithfarm/wip-45045-nautilus

nautilus: os/bluestore: Don't pollute old journal when add new device

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #34677 from Vicente-Cheng/wip-44468-nautilus

nautilus: mon: Get session_map_lock before remove_session

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #34611 from ifed01/wip-ifed-beter-bluefs-expand-nautilus

nautilus: os/bluestore: open DB in read-only when expanding DB/WAL

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #34774 from ajarr/nautilus-fix-45229-45225

nautilus: mon/FSCommands: Fix 'add_data_pool' command and 'fs new' command

Reviewed-by: Nathan Cutler <ncutler@suse.com>

qa/tasks/ceph.py: do not use option mimic does not understand

mimic does not have `mon-client-directed-command-retry` option, so we
should not pass this option to a mimic ceph client.

in this change, we fall back to plain retry, if command fails. this
change is not cherry-picked from master. as we don't run upgrade test
from mimic to master.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34579 from tchaikov/nautilus-mimic-x

nautilus: qa/tasks/ceph_manager: do not cancel pending pg num changes on mimic

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

qa/suites/rados: use default objectsize for upgrade tests

in pre-nautilus release, rados cli does not accept `-O` option, so we
should not pass `-O` to this tool. otherwise we will have following
failure:
```
2020-05-01T05:47:04.863 INFO:tasks.radosbench.radosbench.2.smithi183.stderr:unrecognized command -O; -h or --help for usage
2020-05-01T05:47:04.865 DEBUG:teuthology.orchestra.run:got remote process result: 1
```

this change is not cherry-picked from master. as we don't perfor
upgrade tests from pre-nautilus releases

Signed-off-by: Kefu Chai <kchai@redhat.com>

selinux: Allow getattr access to /proc/kcore

Required for an fstat call in BlkDev::get_devid

Fixes: https://tracker.ceph.com/issues/40743
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

qa/distros: point {centos,rhel}_latest.yaml to 7.8

this change is not cherry-picked from master, as master is using el8
already

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/mgr/dashboard/test_rbd: wait longer when purging

Fixes: https://tracker.ceph.com/issues/44743
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>
Co-authored-by: Kefu Chai <kefu@redhat.com>
Co-authored-by: Volker Theile <vtheile@suse.com>
(cherry picked from commit 1c3d6539d93961f9f7bd5308771725410bde32ec)

Conflicts:
qa/tasks/mgr/dashboard/test_rbd.py

mgr/dashboard: fixing RBD purge error in backend

Current UTC datetime should be used as `now`, otherwise the expiration
comparison might fail on machines with non-UTC localtime.

Fixes: https://tracker.ceph.com/issues/45149
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>
(cherry picked from commit 00881adf55cac6116887bfffc506444473f0ee82)

Conflicts:
src/pybind/mgr/dashboard/controllers/rbd.py
- Namespace is not supported nautilus.
- Use logger instance from mgr.

qa/tasks: do not cancel pending pg num changes on mimic

mimic does not support auto split/merge, but we do test mimic-x on
nautilus, which ends up with failures like:

ceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/teuthology/contextutil.py", line 34, in nested
    yield vars
  File "/home/teuthworker/src/git.ceph.com_ceph_nautilus/qa/tasks/ceph.py", line 1928, in task
    ctx.managers[config['cluster']].stop_pg_num_changes()
  File "/home/teuthworker/src/git.ceph.com_ceph_nautilus/qa/tasks/ceph_manager.py", line 1806, in stop_pg_num_changes
    if pool['pg_num'] != pool['pg_num_target']:
KeyError: 'pg_num_target'

so we need to skip this if 'pg_num_target' is not in pg_pool_t::dump().

this change is not cherry-picked from master, as we don't test
mimic-x on master.

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: increase log level for same or older period pull msg

Signed-off-by: Ali Maredia <amaredia@redhat.com>
(cherry picked from commit 90cce10fc1364df16ab12632b2dca403894cbe44)

bluestore/bdev: initialize size when creating object.

The other partss of the struct are initialized by their
ctors. Only for unint_64 there is no ctor.

Otherwise ceph-dencoder tests will fail in comparing
the exported output.
[~/master36] wjw@cephdev.digiware.nl> build/bin/ceph-dencoder type bluestore_bdev_label_t select_test 1 encode export /tmp/typ-yFISvjvgj
[~/master36] wjw@cephdev.digiware.nl> hexdump -C !$
hexdump -C /tmp/typ-yFISvjvgj
00000000  62 6c 75 65 73 74 6f 72  65 20 62 6c 6f 63 6b 20 |bluestore block |
00000010  64 65 76 69 63 65 0a 30  30 30 30 30 30 30 30 2d |device.00000000-|
00000020  30 30 30 30 2d 30 30 30  30 2d 30 30 30 30 2d 30 |0000-0000-0000-0|
00000030  30 30 30 30 30 30 30 30  30 30 30 0a 02 01 28 00 |00000000000...(.|
00000040  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |................|
00000050  00 00 70 74 03 04 00 00  00 00 00 00 00 00 00 00 |..pt............|
00000060  00 00 00 00 00 00 00 00  00 00 |..........|
0000006a

[~/master36] wjw@cephdev.digiware.nl> build/bin/ceph-dencoder type bluestore_bdev_label_t select_test 1 encode decode encode export /tmp/typ-MjWXdCpzJ
[~/master36] wjw@cephdev.digiware.nl> hexdump -C !$
hexdump -C /tmp/typ-MjWXdCpzJ
00000000  62 6c 75 65 73 74 6f 72  65 20 62 6c 6f 63 6b 20 |bluestore block |
00000010  64 65 76 69 63 65 0a 30  30 30 30 30 30 30 30 2d |device.00000000-|
00000020  30 30 30 30 2d 30 30 30  30 2d 30 30 30 30 2d 30 |0000-0000-0000-0|
00000030  30 30 30 30 30 30 30 30  30 30 30 0a 02 01 28 00 |00000000000...(.|
00000040  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |................|
00000050  00 00 73 64 00 00 00 00  00 00 00 00 00 00 00 00 |..sd............|
00000060  00 00 00 00 00 00 00 00  00 00 |..........|
0000006a

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>
(cherry picked from commit d411ee26fb2f6cbe610f8bbc81b777cf28d839c2)

qa: use small default pg count for CephFS pools

The pg count needs to be a power-of-two since
dff5697464edb9931d5dfa08cd4a30f85c1f237e.

Also, mon_pg_warn_min_per_osd is disabled by default now (or set to a
low value in vstart/testing) so there's no need to base the pg count on
this value.

Ideally someday we can remove this so that the default cluster value is
used but we need to keep this for deployments of older versions of Ceph.

Fixes: https://tracker.ceph.com/issues/42228
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit fc88e6c6c55402120a432ea47f05f321ba4c9bb1)

Conflicts:
qa/tasks/cephfs/filesystem.py: this commit was orignally
backported by #34055, but it failed to cherry-pick all necessary
bits. in this change, the missing bit is picked up.

Merge pull request #34518 from tchaikov/nautilus-27422

nautilus: doc: fix parameter to set pg autoscale mode

Reviewed-by: Neha Ojha <nojha@redhat.com>

doc: fix parameter to set pg autoscale mode

osd_pool_default_pg_autoscale_mode is the right parameter to
set placement-group autoscale mode.

Signed-off-by: Changcheng Liu <changcheng.liu@intel.com>
(cherry picked from commit c0df98fc7e78bbd366333d810f78ddbeed0e6729)

Merge pull request #34736 from smithfarm/wip-44648-nautilus

nautilus: qa/workunits/rbd: wait for nbd map to close after unmap

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

os/bluestore: Don't pollute old journal when add new device

Fixes: https://tracker.ceph.com/issues/44774
Signed-off-by: Yang Honggang <yanghonggang@kuaishou.com>
(cherry picked from commit ade4d46981e660c8d57cec64180b5afa4561b945)

Merge pull request #34735 from smithfarm/wip-45260-nautilus

nautilus: rgw: reshard: skip stale bucket id entries from reshard queue

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #34628 from shyukri/wip-45070-nautilus

nautilus: mgr/telegraf: catch FileNotFoundError exception

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34512 from Vicente-Cheng/wip-44324-nautilus

nautilus: osd/PeeringState.h: ignore RemoteBackfillReserved in WaitLocalBackfillReserved

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #34500 from Vicente-Cheng/wip-44289-nautilus

nautilus: mon: stash newer map on bootstrap when addr doesn't match

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #34419 from votdev/wip-44954-nautilus

nautilus: monitoring: root volume full alert fires false positives

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

mon/FSCommands: Fix 'fs new' command

After creating a filesystem using the 'fs new' command, the value
of the 'data' and 'metadata' key of the datapool and metadatapool's
application tag 'cephfs' should be the filesystem's name. This
didn't happen when the data or metadata pool's application metadata
'cephfs' was enabled before the pool was used in the 'fs new' command.
Fix this during the handling of the 'fs new' command by setting the
value of the key of the pool's application metadata 'cephfs' to the
filesystem's name even when the application metadata 'cephfs' is
already enabled or set.

Fixes: https://tracker.ceph.com/issues/43761
Signed-off-by: Ramana Raja <rraja@redhat.com>
(cherry picked from commit 2f45558eb84782e334d997e2c545fd99ab455783)

mon/FSCommands: Fix 'add_data_pool' command

After making a RADOS pool a filesystem's data pool using the
'add_data_pool' command, the value of the 'data' key of the pool's
application metadata 'cephfs' should be the filesystem's name. This
didn't happen when the pool's application metadata 'cephfs' was
enabled before the pool was made the data pool. Fix this during the
handling of the 'add_data_pool' command by setting the value of
the 'data' key of the pool's application metadata 'cephfs' to the
filesystem's name even when the application metadata 'cephfs' is
already enabled or set.

Fixes: https://tracker.ceph.com/issues/43061
Signed-off-by: Ramana Raja <rraja@redhat.com>
(cherry picked from commit 3c49092322720b7f1d612f45e73fb26820ff7885)

cmake: detect fuse3 as well

Fixes: https://tracker.ceph.com/issues/44891
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit c9b38a5210954f9fd3d46bc9560f455444f77f50)

cmake: check for FUSE version

and refactor FindFUSE.cmake a little bit to conditionalize OSX support

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 220fa91d80175daa045022d9df50a1e44e5aebe9)

cmake: link against FUSE::FUSE

instead of specifying the include directory and linkage separately

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit b823952d245db4967828033003729a643d8e276f)

Conflicts:
src/CMakeLists.txt
- ceph-fuse target_link_libraries call lacks ${EXTRALIBS} parameter in
  nautilus

cmake: rename Findfuse.cmake to FindFUSE.cmake

so its name is consistent with upstream name, and with the variables
exposed by FindFUSE.cmake.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 2b3adb5c5ee518cf4f03d1f96256f411aec1def2)

ceph-fuse: link to libfuse3 if fuse3 is installed

Fixes: https://tracker.ceph.com/issues/44891
Signed-off-by: Xiubo Li <xiubli@redhat.com>
(cherry picked from commit cb0a600acfca76c5b4653e4c6f34c1712a2da9de)

Conflicts:
src/client/fuse_ll.cc
- nautilus CephFuse::Handle::Handle initializer list has stag_lock

rbd-fuse: remove FUSE_CAP_BIG_WRITES when linking to libfuse3

The big-writes is enabled as default from 3.0.

Fixes: https://tracker.ceph.com/issues/44891
Signed-off-by: Xiubo Li <xiubli@redhat.com>
(cherry picked from commit 32a45b624b5142582f1ef58f87b7442d273b0ce7)

rbd-fuse: switch to use utimens instead

The utime is deprecated.

Fixes: https://tracker.ceph.com/issues/44891
Signed-off-by: Xiubo Li <xiubli@redhat.com>
(cherry picked from commit d5999a9ac44cba09f8b9733260262bce49e659bd)

ceph-fuse: enable the big writes as default

When the libfuse version < 3, without the big_writes options enabled
the write perf is very poor.

Fixes: https://tracker.ceph.com/issues/44885
Signed-off-by: Xiubo Li <xiubli@redhat.com>
(cherry picked from commit 1c3dd60f217129f6ed718ca40098994a12259262)

ceph-fuse: don't get mount options from /etc/fstab when doing remount

If there happen to be an kcephfs entry in /etc/fstab for ceph-fuse's
mount point. 'mount -o remount' may get options from that entry. fuse
may not understand some options (E.g name option).

Fixes: https://tracker.ceph.com/issues/44771
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 7aa22fac602b6e1c31943e2de69a2f8e3a3002d0)

client: reset requested_max_size if file write is not wanted

write can stuck at waiting for larger max_size in following sequence of
events:

- client opens a file and writes to position 'A' (larger than unit of
  max size increment)
- client closes the file handle and updates wanted caps (not wanting
  file write caps)
- client opens and truncates the file, writes to position 'A' again.

At the 1st event, client set inode's requested_max_size to 'A'. At the
2nd event, mds removes client's writable range, but client does not reset
requested_max_size. At the 3rd event, client does not request max size
because requested_max_size is already larger than 'A'.

Fixes: https://tracker.ceph.com/issues/44801
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 1d676851e3491c26d3bf543bf357731d11d82a85)

monitoring: Fix pool capacity incorrect

Signed-off-by: James Cheng <james59988@gmail.com>
(cherry picked from commit 1b980ef88c5f8f45b5b39b79c395a66233908266)

Merge pull request #34662 from votdev/wip-45157-nautilus

nautilus: mgr/dashboard: Refactor Python unittests and controller

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

rgw/pubsub: send notifications from multi-delete op

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit 754f7edbb8156727de45c17c03826c9a0df6dbb8)
Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>

rgw/notification: add opaque data

opaque data may be set in topic configuration and later on sent inside
nottifications triggered by that topic.

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit 07630a8759b0836f56d4ee6938e2b711ffb4e169)
Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
Conflicts:
src/test/rgw/rgw_multi/tests_ps.py

rgw/pubsub: notification filtering by object tags

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit d19474f3d7b4e0a16cc2ef55f2bb4e3fd027545c)
Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
Conflicts:
doc/radosgw/notifications.rst
examples/boto3/README.md
examples/boto3/notification_filters.py
examples/boto3/service-2.sdk-extras.json
src/common/ceph_json.h
src/rgw/rgw_common.cc
src/rgw/rgw_common.h
src/rgw/rgw_op.cc
src/rgw/rgw_pubsub.h
src/test/rgw/rgw_multi/tests_ps.py

Merge pull request #34627 from shyukri/wip-45043-nautilus

nautilus: mgr/PyModule: fix missing tracebacks in handle_pyerror()

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #34542 from smithfarm/wip-44486-nautilus

nautilus: common/ceph_time: tolerate mono time going backwards

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34539 from smithfarm/wip-45073-nautilus

nautilus: build/ops: address SElinux denials observed in rgw/multisite test run

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34527 from smithfarm/wip-45035-nautilus

nautilus: spec: address some warnings raised by RPM 4.15.1

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34256 from smithfarm/wip-44793-nautilus

nautilus: tests: qa/tasks/ceph.py: fail if any osd role doesn't get a device

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/workunits/rbd: wait for nbd map to close after unmap

The unmap action only sends a signal to the kernel to notify the
rbd-nbd daemon to disconnect. Therefore, it's possible that an
unmap followed by an immediate re-map to the same device might
fail since the unmap is still in-progress.

Fixes: https://tracker.ceph.com/issues/44567
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 80a3f18cafb4add1624cc690bba436a1284dc634)

rgw: reshard: skip stale bucket id entries from reshard queue

If we encounter a reshard queue entry that has an older ID compared to the
bucket's current ID, it'd mean that some other process or a manual reshard has
already processed this entry, skip processing the entry this time. An
alternative is to verify the num_shards that we have in queue >= the current
shards, but this would mean that we may reshard a recently manual resharded
bucket again which might not be intended

Fixes: https://tracker.ceph.com/issues/45134
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit 02664fc091674e28233559cd1c42f954d5776d86)

Conflicts:
src/rgw/rgw_reshard.cc
- immediately preceding "get_bucket_info" call looks different in
  nautilus

nautilus: mgr/volumes: Add interface to get subvolume metadata

The following interface is added

"ceph fs subvolume info <vol_name> <sub_name> [<group_name>]"

The output is in json format with following fields

1. atime: access time of subvolume path in the format "YYYY-MM-DD HH:MM:SS"
2. mtime: modification time of subvolume path in the format "YYYY-MM-DD HH:MM:SS"
3. ctime: change time of subvolume path in the format "YYYY-MM-DD HH:MM:SS"
4. uid: uid of subvolume path
5. gid: gid of subvolume path
6. mode: mode of subvolume path
7. mon_addrs: list of monitor addresses
8. bytes_pcent: quota used in percentage if quota is set, else displays "undefined"
9. bytes_quota: quota size in bytes if quota is set, else displays "infinite"
10. bytes_used: current used size of the subvolume in bytes
11. created_at: time of creation of subvolume in the format "YYYY-MM-DD HH:MM:SS"
12. data_pool: data pool the subvolume belongs to
13. path: absolute path of a subvolume
14. type: subvolume type indicating whether it's clone or subvolume

Fixes: https://tracker.ceph.com/issues/44277
Signed-off-by: Kotresh HR <khiremat@redhat.com>
(cherry picked from commit 70659ffee19c1ef23314994aa3ecb5fe8027d224)

Conflicts:
   src/pybind/mgr/volumes/fs/volume.py [ context: commit a44de38
      ("mgr_util: add CephfsClient implementation") not in nautilus ]

Merge pull request #34551 from tspmelo/wip-45085-nautilus

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #34550 from tspmelo/wip-45082-nautilus

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #34497 from votdev/wip-45019-nautilus

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #34399 from votdev/wip-44920-nautilus

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Sebastian Krah <skrah@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

mon: Get session_map_lock before remove_session

Get session_map_lock to prevent race.

Fixes: https://tracker.ceph.com/issues/44407
Signed-off-by: Xiaofei Cui <cuixiaofei@sangfor.com.cn>
(cherry picked from commit c916b8a8156f9855fb74c62508b54308ec5e7b32)

Merge pull request #34597 from smithfarm/wip-43134-nautilus

nautilus: rgw: find oldest period and update RGWMetadataLogHistory()

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #34585 from smithfarm/wip-45040-nautilus

nautilus: mon: calculate min_size on osd pool set size

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #34489 from rhcs-dashboard/nautilus-fix-coverage-path

nautilus: mgr/dashboard: fix COVERAGE_PATH in run-backend-api-tests.sh

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #34394 from votdev/wip-44917-nautilus

nautilus: monitoring: alert for prediction of disk and pool fill up broken

Reviewed-by: Patrick Seidensal <pnawracay@suse.com>

Merge pull request #34392 from tchaikov/nautilus-cbt

nautilus: qa/suites: use "nautilus" branch for cbt based testing

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #34609 from smithfarm/wip-44267-nautilus

nautilus: rgw: fix bug with (un)ordered bucket listing and marker w/ namespace

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #34599 from smithfarm/wip-44232-nautilus

nautilus: rgw: ReplaceKeyPrefixWith and ReplaceKeyWith can not set at the same …

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #34598 from smithfarm/wip-44136-nautilus

nautilus: rgw: set bucket attr twice when delete lifecycle config.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa: whitelist cluster warning message for removed MDS

Fixes: http://tracker.ceph.com/issues/44677
Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 3c531197a6070917206a4fbe31d7291019ca57b1)

mgr: force purge normal ceph entities from service map

Normal ceph services can send task status updates to manager.
Task status is tracked in service map implying that normal
ceph services have entries in service map and daemon tracking
index (daemon state). But the manager prunes entries from daemon
state when it receives an updated map (fs, mon, etc...). This
causes periodic pruning of service map entries to fail for normal
ceph services (those which send task status updates) since it
expects a corresponding entry in daemon state.

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit bccbf1fa03ed2fb02ad2e50e6aaf963b36d8bd30)

Merge pull request #34278 from jan--f/wip-44688-nautilus

nautilus: ceph-volume: prepare: use *-slots arguments for implicit sizing

Merge pull request #34478 from jan--f/wip-44998-nautilus

nautilus: ceph-volume/batch: return success when all devices are filtered

Merge pull request #34481 from jan--f/wip-45002-nautilus

nautilus: ceph-volume/batch: check lvs list before access

Merge pull request #34445 from jan--f/wip-44974-nautilus

nautilus: devices/simple/scan: Fix string in log statement