test/store_test: introduce test case for per-pool omap repair.

This also reproduces
https://tracker.ceph.com/issues/43824

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #32853 into master

* refs/pull/32853/head:
qa/suites/fs/upgrade: finish at octopus
qa/suites/fs/upgrade: set min-compat-client to octopus
qa/suites/fs/upgrade: set pg_autoscale_mode=off after upgrade

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32883 into master

* refs/pull/32883/head:
qa/workunits/rados/test_crash.sh: do not fail if coredump not found
rgw: set bucket quota only when necessary

Reviewed-by: Sage Weil <sage@redhat.com>

monitoring: add details to Prometheus' alerts (#32803)

monitoring: add details to Prometheus' alerts

Reviewed-by: Andreas Jaeger <aj@suse.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #32434 from trociny/wip-41559

mgr/rbd_support: implement mirror snapshot scheduler

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: optimize mirror_image_info_list (don't open image context)

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: make mirror get info request not require image context

Signed-off-by: Mykola Golub <mgolub@suse.com>

qa/workunits/rados/test_crash.sh: do not fail if coredump not found

Fixes: https://tracker.ceph.com/issues/43653
Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: set bucket quota only when necessary

this change fixes the regression introduced by 9a7f76a7

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/rbd_support: optimize scanning images in mirror snapshot mode

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: add API to list mirror image info

Signed-off-by: Mykola Golub <mgolub@suse.com>

mgr/rbd_support: implement mirror snapshot scheduler

Signed-off-by: Mykola Golub <mgolub@suse.com>

pybind/mgr/rbd_support: split module.py into several files

(it is becoming too large)

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #32734 from trociny/wip-43589

librbd: don't call refresh from mirror::GetInfoRequest state machine

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #32766 from rzarzynski/wip-common-ceph_atomic

crimson, common: introduce ceph::atomic and apply it on bufferlist.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32760 from trociny/wip-43666

librbd: fix potential snapshot remove failure due to duplicate RPC messages

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #32730 from tchaikov/wip-mgr-maybe-unused

mgr: better error handling when reading option

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #32758 from ifed01/wip-ifed-fix-legacy-omap

os/bluestore: upgrade legacy omap to per-pool format automatically.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #32764 from liewegas/fix-43552

mon/Session: only index osd ids >= 0

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #32753 from mgfritch/cephadm-tox-escape-seq

cephadm: fix tox DeprecationWarning

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #32352 from rzarzynski/wip-bug-43070

msg/async: reset the V1's session_security in proper EventCenter.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32785 from rzarzynski/wip-encoding-no-bl-copy_in-2

common: encode for std::list<T> doesn't use bl::copy_in() anymore.

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/suites/fs/upgrade: finish at octopus

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/fs/upgrade: set min-compat-client to octopus

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #30272 from ivancich/wip-move-some-bucket-listing-logic-to-cls

rgw: move delimiter-based bucket listing/filtering logic to cls

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/suites/fs/upgrade: set pg_autoscale_mode=off after upgrade

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32750 into master

* refs/pull/32750/head:
qa/suites/upgrade/*-x/stress-split: run latest python tests at end
qa/tasks/ceph_manager: kludge around /var/log/ceph permissions
mgr/pg_autoscaler: don't check anything until cluster is nautilus
qa/suites/upgrade: install python3-* as part of final upgrade step
qa/tasks/radosbench: only set object size if != block size
qa/tasks/ceph: simplify mon_health_to_clog suppression during restart
cls_hello: alias write_return_data -> writes_dont_return_data
ceph_test_cls_hello: only do returndata test on octopus+
qa: remove unnecessary package excludes in upgrades/nautilus*
qa: exclude cephadm from upgrade/nautilus-x
qa/suites/upgrade/mimic-x/parallel: fix msgr2 vs octopus ordering
qa/suites/upgrade/nautilus-x/stress-split: fix msgr2 vs octopus ordering
qa/suites/upgrade/mimic-x: fix msgr2 vs octopus ordering
qa/suites/upgrade/nautilus-x: end on octopus
qa/suites/upgrade/mimic-x: finish at octopus
qa/suites/upgrade/nautilus-x: disable TOO_FEW_PGS warning
qa/tasks/ceph: set mon_health_to_clog=false via mon config
qa/suites/upgrade/mimic-x: disable TOO_FEW_PGS warning

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #32804 into master

* refs/pull/32804/head:
pybind: Add test for listxattr
pybind: Add listxattr

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32786 into master

* refs/pull/32786/head:
mon/ConfigMonitor: put global/ keys in global section
mon/ConfigMonitor: transition old keys to have global/ prefix
mon/ConfigMonitor: always prefix global config keys with global/
mon/ConfigMonitor: switch to use pending_cleanup boost::optional

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #32665 into master

* refs/pull/32665/head:
os/bluestore: avoid race between split_cache and get/put pin/unpin
os/bluestore: remove no-op line from split_cache

Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge PR #32755 into master

* refs/pull/32755/head:
pybind/mgr/*: fix config_notify handling of default values

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #32809 into master

* refs/pull/32809/head:
common/options: bluestore 64k min_alloc_size for HDD

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

rgw: let newer RGWs work with older OSDs re: cls-side filtering

Previous commits moved bucket list filtering when a delimiter was
specified to the osd/cls layer. However, since rgw's are often
upgraded before osd's are, until we reach verison ceph version 16, an
rgw cannot assume that the osd/cls did the filtering. This is
addressed in the following ways....

First rgw_cls_list_ret now indicates whether filtering was done on the
osd/cls side.

And second, the old filtering code in the rgw is maintained in
RGWRados::Bucket::List::list_objects_ordered, so it can still be
triggered when all osd's are not doing the filtering.

Once we reach ceph version 16, and there is no chance that the rgw is
working with a osd running "young" version 14 code, we can remove the
backward compatibility code in
RGWRados::Bucket::List::list_objects_ordered.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

rgw: add unit testing for delimiter filtering in cls

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

rgw: move some bucket index delimiter filtering logic to cls

Currently the cls code that returns bucket index entries does no
filtering when a delimiter is specified. This means that the cls code
will return many entries that the rgw code does not need. This means
unnecessary bucket index entries will be encoded, decoded, and consume
network bandwidth.

A change is made to send a delimiter, if it's specified, to the cls
code, so the filtering can be done there.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #32735 from trociny/wip-doc-mirror-snapshot

doc: document "mode" param for "rbd mirror image enable" command

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rgw: reformat some code to make it easier to read

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

rgw: rename search variable to better indicate semantics

cls_cxx_map_get_vals takes a parameter of where in the sequence of
keys to start returning values. In code it's often referred to as the
"start_key", but in fact it only returns values who have keys *after*
it. To clarify the semantics, rename the variable to "start_after_key"
in the bucket listing functions in both rgw and cls.

Also, declare a type RGWRados::check_filter_t, so the type does not
have to be fully literalized multiple times.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

rgw: change scope of bucket index entry flags

Redefine global constants that define flags in bucket index entries to
struct-scoped.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

rgw: add documentation to bucket index code

Some of the logic surrounding bucket indexes and the listing thereof
can be subtle. Add some comments to help future developers working on
this code.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

rgw: move after-delimiter logic so available in both rgw & cls/osd

Since we need to be able to calculate a key a path subdirectory both
in rgw and cls, move the code that calculates that to cls_rgw_types.h,
which is shared by both rgw and cls/osd.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

rgw: use a constexpr to specify standard # of bucket entries to list

Rather than hard-coding literals (of 1000) throughout the code, use a
constexpr given it's the standard chunk we're reading from a bucket
index.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

rgw: add missing editor directive to file

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge PR #32686 into master

* refs/pull/32686/head:
mgr/cephadm: speed up when not refresh is needed
mgr/cephadm: track age of service metadata
mgr/orchestrator: include age of 'service ls' metadata
mgr/mgr_util: add to_pretty_timedelta

Reviewed-by: Sebastian Wagner <swagner@suse.com>

mgr/cephadm: speed up when not refresh is needed

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: track age of service metadata

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator: include age of 'service ls' metadata

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/mgr_util: add to_pretty_timedelta

This mirrors the Ceph C++ implementation.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #28603 into master

* refs/pull/28603/head:
mgr: Add get_rates_from_data to mgr_util.py

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #32788 into master

* refs/pull/32788/head:
qa/tasks/mgr/dashboard: set pg_num to 32
mgr/pg_autoscaler: default to pg_num[_min] = 32

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/dashboard: check if user has config-opt permissions (#32638)

mgr/dashboard: check if user has config-opt permissions

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

monitoring: add details to Prometheus' alerts

Fixes: https://tracker.ceph.com/issues/43764
Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

qa/suites/upgrade/*-x/stress-split: run latest python tests at end

Use the master/test/x branch, so that we run the python3 variant of the
test.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Improve the size of forms (#31863)

mgr/dashboard: Improve the size of forms

Reviewed-by: Sebastian Krah <skrah@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #32802 from uweigand/cmake-atomic-check

cmake: Improve test for 16-byte atomic support on IBM Z

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

mgr/dashboard: Unit test does not do the mentioned thing (#32799)

mgr/dashboard: Unit test does not do the mentioned thing

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

qa/tasks/ceph_manager: kludge around /var/log/ceph permissions

The ceph.py task normally makes these permissive.  But a package upgrade
can reset the permissions so that we can't read and write the temp
export files.  (We put them in these dirs now because it's alreadly
mapped out of cephadm containers to the host.)

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/pg_autoscaler: don't check anything until cluster is nautilus

This prevents any health alerts about too many/few PGs due to old pools
that are defaulted to 'warn'.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/upgrade: install python3-* as part of final upgrade step

These are new packages, so they won't install just by upgrading the old
packages, and they are needed for some of the tests.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/radosbench: only set object size if != block size

This is mostly pointless, except that the -O option for objects size
used to be -o for pre-octopus, so passing -O breaks the upgrade tests.
Fortunately, the upgrades use the defaults, so we can just skate by here.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph: simplify mon_health_to_clog suppression during restart

This only does one thing--do that.  More simply.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32798 from rhcs-dashboard/fix-make-check-deps-centos8

ceph.spec.in: fix 'make check' deps for centos8

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #32478 into master

* refs/pull/32478/head:
qa: update xfstests version
qa: install some dependencies for xfstests

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge PR #32754 into master

* refs/pull/32754/head:
mgr/cephadm: health alert for stray hosts and/or services

Reviewed-by: Michael Fritch <mfritch@suse.com>

qa: update xfstests version

Building xfstests on CentOS 8 has build errors:

    2020-01-23T21:15:14.939 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/xfstests/src/fssum.c:616: undefined reference to `major'
    2020-01-23T21:15:14.939 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/xfstests/src/fssum.c:617: undefined reference to `minor'
    2020-01-23T21:15:14.940 INFO:tasks.workunit.client.0.smithi155.stderr:collect2: error: ld returned 1 exit status

From: /ceph/teuthology-archive/pdonnell-2020-01-23_20:47:47-fs-wip-pdonnell-testing-20200122.210533-distro-basic-smithi/4697257/teuthology.log

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: install some dependencies for xfstests

These are needed for CentOS 8 / RHEL8.

Also organize the YAML to use the cleaner array syntax.

Fixes: https://tracker.ceph.com/issues/43459
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32350 into master

* refs/pull/32350/head:
mds: Reorganize class members in SnapServer header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32722 into master

* refs/pull/32722/head:
mds: fix null pointer dereference in Server::handle_client_link()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

cls_hello: alias write_return_data -> writes_dont_return_data

This allows pre-octopus tests to reach this (renamed) method via the old
method name.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_cls_hello: only do returndata test on octopus+

Signed-off-by: Sage Weil <sage@redhat.com>

qa: remove unnecessary package excludes in upgrades/nautilus*

These packages are present in nautilus.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

qa: exclude cephadm from upgrade/nautilus-x

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

qa/suites/upgrade/mimic-x/parallel: fix msgr2 vs octopus ordering

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/upgrade/nautilus-x/stress-split: fix msgr2 vs octopus ordering

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/upgrade/mimic-x: fix msgr2 vs octopus ordering

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/upgrade/nautilus-x: end on octopus

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/upgrade/mimic-x: finish at octopus

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/upgrade/nautilus-x: disable TOO_FEW_PGS warning

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph: set mon_health_to_clog=false via mon config

This actually works better on new versions.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/upgrade/mimic-x: disable TOO_FEW_PGS warning

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32616 into master

* refs/pull/32616/head:
cephadm: annotate call()
cephadm: podman inspect: image field was called `ImageID`:

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32795 into master

* refs/pull/32795/head:
cephadm: require fsid if name contains a daemon id
cephadm: infer the fsid by name

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #31155 from zhangsw/fix-rgw-reshard-process

rgw: one log shard fails shouldn't block other shards process when reshard buckets

Reviewed-by: Mark Kogan <mkogan@redhat.com>

Merge pull request #31103 from zhangsw/fix-rgw-reshard-cleanindex

rgw: clean index and remove bucket instance info when setting resharding status fails

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30920 from GaryHyg/wip-quota-bug

RGW: Set appropriate bucket quota value (when quota value is less than 0)

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30880 from jiahuizeng/radosgw

rgw: support radosgw-admin zone/zonegroup placement get command

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30861 from IlsooByun/rgw_lc_missing

rgw: do not miss the 1000th element of every iteration during lifecycle processing

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

mon/ConfigMonitor: put global/ keys in global section

Signed-off-by: Sage Weil <sage@redhat.com>

librbd: don't call refresh from mirror::GetInfoRequest state machine

Fixes: https://tracker.ceph.com/issues/43589
Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #32797 from tchaikov/wip-seastar-and-errorator

seastar,crimson: pickup change to pin socket to fixed core

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #29703 from baixueyu/wip-quota

rgw:  when you abort a multipart upload request, the quota may be not updated

Merge pull request #28622 from IlsooByun/master

rgw: incorrect return value when processing CORS headers

Merge pull request #29733 from baixueyu/wip-reload-coredump

rgw: maybe coredump when reload operator happened

doc: document "mode" param for "rbd mirror image enable" command

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #29795 from xxcs/xxcs03

rgw: fix one part of the bulk delete(RGWDeleteMultiObj_ObjStore_S3) fails but no error messages

Merge pull request #30913 from ofriedma/wip-ofriedma-lc-logs

rgw: Add more details to the LC delete and transit log

Merge PR #32787 into master

* refs/pull/32787/head:
qa/tasks/cephadm: work around .git suffix on ceph_repo
qa/tasks/cephadm: learn to pull cephadm from github

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

common/options: bluestore 64k min_alloc_size for HDD

Until we understand the performance regression and allocator behavior,
go back to 64k.  This will continue to have a high space overhead for
small objects and EC, but will preserve the current performance levels
for all workloads.

This partially reverts 0ec75c99ddb103c664f66c04450003ed4c407708

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: health alert for stray hosts and/or services

Fixes: https://tracker.ceph.com/issues/43714
Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Add column filtering feature to cd-table (#32568)

mgr/dashboard: Add column filtering feature to cd-table

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>