Merge PR #27069 into master

* refs/pull/27069/head:
mgr: Configure Py root logger for Mgr modules

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #27084 into master

* refs/pull/27084/head:
osd: process_copy_chunk remove obc ref before pg unlock

Reviewed-by: Myoungwon Oh <omwmw@sk.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #27142 into master

* refs/pull/27142/head:
osd/OSDMap: add log for better debugging
osd/OSDMap: calc_pg_upmaps - restrict optimization to origin pools only

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #27155 into master

* refs/pull/27155/head:
mon/ConfigMonitor: make 'config reset' idempotent

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #27172 into master

* refs/pull/27172/head:
mgr/devicehealth: Fix python 3 incompatiblity

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #27183 from tchaikov/wip-ceph.spec.in-devtoolset-8

rpm: s/devtoolset-7/devtoolset-8/

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #27177 from tchaikov/wip-crimson-stop-before-exit

crimson/osd: call at_exit() before stopping the engine

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #27192 from idryomov/wip-krbd-discard-fixups

qa/workunits/rbd: fixups for the new krbd discard behavior

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #26925 from votdev/issue_38265

mgr/dashboard/qa: Improve tasks.mgr.test_dashboard.TestDashboard.test_standby

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #26396 from familyuu/interval

mgr/dashboard: Add refresh  interval to the dashboard landing page

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kanika Murarka <kmurarka@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #27136 from trociny/wip-38895

librbd: trash move return EBUSY instead of EINVAL for migrating image

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #27137 from trociny/wip-qa-rbd-qos

qa: add RBD QOS functional test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #27151 from trociny/wip-38928

librbd: look for pool metadata in default namespace

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #27118 from sebastian-philipp/mypy-gitignore

scrips/run_mypy: add .gitignore

Reviewed-by: Alfonso Martínez <almartin@redhat.com>

scrips/run_mypy: add .gitignore

Co-Authored-by: Patrick Nawracay <pnawracay@suse.com>
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #27181 from tchaikov/wip-test/test_weighted_shuffle

test/test_weighted_shuffle: enlarge epsilon

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

rpm: s/devtoolset-7/devtoolset-8/

should activate devtoolset-8 not devtoolset-7 for enabling GCC-8.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/test_weighted_shuffle: enlarge epsilon

be more permissive to address the following failure:

test_weighted_shuffle.cc:36: Failure
The difference between float(w) / total_weight and float(f.front()) /
samples is 0.010334432125091553, which exceeds epsilon, where
float(w) / total_weight evaluates to 0.53763443231582642,
float(f.front()) / samples evaluates to 0.52730000019073486, and
epsilon evaluates to 0.0099999997764825821

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: call at_exit() before stopping the engine

if we mkfs and stop the engine, any calls using the engine should/will
fail after `seastar::engine().exit(0)`. in the case of reactor::at_exit(),
it has:

assert(!_stopping);

in this change, we register the at_exit() calls before scheduling the
`engine().exit(0)` call.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27149 from wjwithagen/wjw-fix-stdFileSystem

cmake: use StdFilesystem::filesystem instead of stdc++fs

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27134 from tchaikov/wip-devtools-8

install-dep,rpm: use devtools-8 on amd64

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #26697 from tchaikov/wip-crimson-pg-fsm

crimon/osd: serve read requests

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd/pg: specify file variable values for editors

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: add pg backends

* add exceptions.h for EIO, ENOENT error handling
* add PGBackend for serving i/o requests

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/devicehealth: Fix python 3 incompatiblity
Fixes: http://tracker.ceph.com/issues/38939
Signed-off-by: Marius Schiffer <marius@mschiffer.de>

qa/workunits/rbd: fixups for the new krbd discard behavior

With discard_granularity set to alloc_size, we no longer get object
size alignment from blk_bio_discard_split().

This assumption is pretty deeply ingrained in krbd_data_pool.sh, so
make it explicit.  For krbd_fallocate.sh, just fix the expectation.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa: krbd_fallocate.sh: zero can be munged to truncate

The test case is issuing discards that span two objects: the tail of
the first object is truncated, the head of the second object is zeroed.
These discards aren't serial, so there is a race:

  discard i ~ i + 1: truncate i, zero i + 1
  discard i + 1 ~ i + 2: truncate i + 1, zero i + 2

can be executed as

  truncate i + 1, zero i + 2, truncate i, zero i + 1

For object i + 1, the sequence ends up being truncate tail, then zero
head.  This zero op is munged to truncate on the OSD, resulting in size
0 instead of OBJECT_SIZE / 2.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge PR #27124 into master

* refs/pull/27124/head:
osd/pg: use emplace() to construct new element in-place

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #27079 into master

* refs/pull/27079/head:
mgr/pg_autoscaler: Fix python3 incompatibility

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #27113 into master

* refs/pull/27113/head:
mgr/diskprediction_cloud: Correct base64 encode translate table

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #27156 into master

* refs/pull/27156/head:
common/blkdev: get_device_id: behave if model is lvm and id_model_enc isn't there

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #26100 from ceph/wip-s3a-hadoop-ver

qa/tests: update s3a hadoop versions used for test

Reviewed-by: Casey Bodley <cbodley@redhat.com>

osd: process_copy_chunk remove obc ref before pg unlock

drop the obc ref when pg locked is more safer, if not, on_change
may failed to clear it, and on_flushed may find leak.

Fixes: http://tracker.ceph.com/issues/38842
Signed-off-by: Kefu Chai kchai@redhat.com
Signed-off-by: Zengran Zhang <zhangzengran@sangfor.com.cn>

Merge pull request #27150 from tchaikov/wip-ceph.spec.in/immutable-object-cache

rpm: immutable-object-cache related changes

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

crimson/osd/pg: wait until pg is active

hold i/o request until pg is active

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: handle MOSDOp in OSD

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27126 from tchaikov/wip-monclient-srv-priority

mon/MonClient: add proper SRV priority support

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #25808 from joke-lee/fix_dump_usage_categories_info

rgw: usage dump_unsigned instead dump_int in dump_usage_categories_info

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #26070 from iliul/fix-bug-38002

rgw: Replace COMPLETE_MULTIPART_MAX_LEN with rgw_max_put_param_size

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #26947 from IrekFasikhov/fixRGWMultiDeleteObject

rgw: fix RGWDeleteMultiObj::verify_permission()

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

common/blkdev: get_device_id: behave if model is lvm and id_model_enc isn't there

If that happens, leave ID_MODEL blank and we'll fall back to something
else, like ID_SERIAL (which is present in each case I've seen this).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #26634 from theanalyst/scheduler-beast-ctx

rgw: build async scheduler only when beast is built

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

mon/ConfigMonitor: make 'config reset' idempotent

This partially revert 1bc9c86d08cfb408768c080ed5ea1ea82325ed57.
It's generally not a good idea if a command is not idempotent.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #25451 from nathan-weinberg/gateway-tests

mgr/dashboard: Added breadcrumb tests to Object Gateway menu items

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

osd/OSDMap: add log for better debugging

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/OSDMap: calc_pg_upmaps - restrict optimization to origin pools only

The current implementation will try to cancel any pg_upmaps that
would otherwise re-map a PG out from an underfull osd, which is wrong,
e.g., because it could reliably fire the following assert:

src/osd/OSDMap.cc: 4405: FAILED assert(osd_weight.count(i.first))

Also it would not match the expectation if automatic balancing
has been strictly restricted to some specific pools by admin.

Fix by excluding any wild PG that does not belong to the origin pools
passed in when trying to do upmap/unmap.

Fixes: http://tracker.ceph.com/issues/38897
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #26850 from nathan-weinberg/nfs-breadcrumb

mgr/dashboard: Added breadcrumb tests to NFS menu

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

librbd: look for pool metadata in default namespace

when applying pool level config overrides

Fixes: https://tracker.ceph.com/issues/38928
Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: trash move return EBUSY instead of EINVAL for migrating image

Fixes: https://tracker.ceph.com/issues/38895
Signed-off-by: Mykola Golub <mgolub@suse.com>

rpm,deb: let immutable-object-cache depend on librados2

ceph-common depends on librados2, but ceph-immutable-object-cache only
uses librados2's C++ API, so librados2 would suffice.

rpmbuild is able to figure out the .so dependencies, and let
ceph-immutable-object-cache depend on the package which offers
librados2, but we'd better make sure immutable-object-cache is linked
against the librados2 of the same version, to ensure that librados2
offers the correct C++ symbols. as we only version the C++ API in the
librados2 after nautilus.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27090 from xiexingguo/wip-config-mon-core

mon/ConfigMonitor: make 'num' of 'config reset' optional; allow target version 0

Reviewed-by: Sage Weil <sage@redhat.com>

rpm: remove libradospp1 from immutable-object-cache's Requires

* remove "libradospp1" from immutable-object-cache's runtime
  dependencies. this package was removed in 65c8733b

Signed-off-by: Kefu Chai <kchai@redhat.com>

rpm: do not use "-n" for immutable-object-cache

as immutable-object-cache has the prefix of "ceph", no need to use "-n"
option. so this change is but a cleanup.

* use "%package immutable-object-cache" not "%package
  ceph-immutable-object-cache" for naming subpackage of
  "ceph-immutable-object-cache". this would result in package name of
  "ceph-immutable-object-cache". see
  http://ftp.rpm.org/max-rpm/s1-rpm-subpack-spec-file-changes.html
* the same applies to other directives with "-n" options, see
  http://ftp.rpm.org/max-rpm/s1-rpm-inside-package-directive.html

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/pg: use emplace() to construct new element in-place

replace emplace() and emplace_back() with insert() and push_back()
calls to construct the new element in-place and avoid the overhead
of move constructor calls

Signed-off-by: Zengran Zhang <zhangzengran@sangfor.com.cn>

mon/ConfigMonitor: make 'num' of 'config reset' command optional

This way it can be used to fast cancel/undo the last command.
Also make the tip message a litter bit nicer..

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #27147 into master

* refs/pull/27147/head:
qa/workunits/mon/config.sh: s|bin/ceph|ceph|

Reviewed-by: Sage Weil <sage@redhat.com>

cmake: use StdFilesystem::filesystem instead of stdc++fs

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #26531 from majianpeng/msg-async-optimization

msg/async: optimizations

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #27107 from neha-ojha/wip-38845

mon/Monitor.cc: print min_mon_release correctly

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27097 from tchaikov/wip-option-sv-as-keys

common/config: use string_view for keys

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #26220 from ashitakasam/master

common/Formatter: escape printed buffer in XMLFormatter::dump_format_va()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #27138 into master

* refs/pull/27138/head:
qa/suites/upgrade/nautilus-x: misc updates

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

qa/workunits/mon/config.sh: s|bin/ceph|ceph|

so we can verify that ceph is able to reject wrong debug levels.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27143 from dzafman/wip-38896

doc: Fix the pg states and auto repair config options

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: Fix the pg states and auto repair config options

Fixes: http://tracker.ceph.com/issues/38896
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #27140 from neha-ojha/wip-fix-cron

qa/crontab/teuthology-cronjobs: fix suite-branch

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #27129 from yuriw/wip-yuriw-subsets-master

qa/tests: added the subset clause for nautilus branch

qa/crontab/teuthology-cronjobs: fix suite-branch

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/suites/upgrade/nautilus-x: misc updates

- start with msgr2 enabled (defaults)
- run nautilus branch for workunits
- drop msgr2 enable step at teh end
- add octopus placeholder (although it is empty for now)

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #26939 into master

* refs/pull/26939/head:
os/bluestore: unconditionally cap chunks returned by allocator to 2^31
os/bluestore: start using 64-bit intervals for bitmap allocator
os/bluestore: make bluestore interval base template.
tests/fastbmap_alloc: UT to reproduce 4G allocation bug
os/bluestore: os/bluestore: implement dump for bitmap allocator
os/bluestore be more tolerant to lack of space for bluefs.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #27086 into master

* refs/pull/27086/head:
msg: remove unused header file in Messenger.h

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #27111 into master

* refs/pull/27111/head:
pybind/rados: should pass "name" to cstr()

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #27112 into master

* refs/pull/27112/head:
qa/suites: do not test luminous-x upgrade path

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #27047 from tianshan/fix_38811

rgw: fix read not exists null version return wrong

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27102 from adamemerson/wip-one-rados-handle-to-rule-them-all

One Rados Handle to Rule Them All

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #25999 from cbodley/wip-37945

rgw: Fix S3 compatibility bug when CORS is not found

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #26998 from cbodley/wip-38328

rgw: don't crash on missing /etc/mime.types

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #26952 from cbodley/wip-rgw-bucket-tagging-notsupp

rgw: reject bucket tagging requests and document unsupported

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #26722 from cbodley/wip-rgw-sync-counters

rgw multisite: add perf counters to data sync

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #27008 from ofriedma/wip-34308-beast-tcp-nodelay

rgw: Adding tcp_nodelay option to Beast

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27105 from neha-ojha/wip-21174

osd/PGLog.h: print olog_can_rollback_to before deciding to rollback

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Xie Xingguo <xie.xingguo@zte.com.cn>

Merge pull request #26819 from tchaikov/wip-rpm-more-recommends

rpm: use Recommends on fedora also

Reviewed-by: Nathan Cutler <ncutler@suse.com>

install-dep,rpm: use devtools-8 on amd64

seems devtools-8 is not packaged on aarch64. and we are suffering from
ICE on centos7 when building with GCC-7.3. so let's try GCC-8.2!

Fixes: http://tracker.ceph.com/issues/38892
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27108 from tchaikov/wip-crimson-vstart.sh

 vstart.sh: add an option to use crimson-osd

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

osd/PGLog.h: print olog_can_rollback_to before deciding to rollback

This is particularly important as the crt might have changed before
being passed to merge_object_divergent_entries().

Fixes: http://tracker.ceph.com/issues/38894
Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #27083 from neha-ojha/wip-38635

qa/objectstore: test with reduced value of osd_memory_target

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>

vstart.sh: pass osd specific options to OSD

--osd-args : for passing osd specific args

this allows us to pass seastar related options to crimson-osd

Signed-off-by: chunmei Liu <chunmei.liu@intel.com>

qa/tests: added the subset clause for nuatilis branch
--subset x/2999 should generate ~250 jobs as result

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

qa/suites/rbd: enable QOS functional test

Signed-off-by: Mykola Golub <mgolub@suse.com>

qa/workunits/rbd: add QOS functional test

Signed-off-by: Mykola Golub <mgolub@suse.com>

qa/suites: do not test luminous-x upgrade path

in this change,

* suites/rados/upgrade: luminous-x-singleton => mimic-x-singleton
* suites/upgrade: luminous-x => nautilus-x

we support upgrade from n to n+2 release. otherwise monitor refuses to
do so:

mon.a@-1(probing) e1 current monmap has min_mon_release 15 (luminous)
which is >2 releases older than me 15 (octopus), stopping.

Fixes: https://tracker.ceph.com/issues/38845
Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/MonClient: take all-0 weights as a special case

if all weights of monitors are 0, it means the administrator want to use
uniform distribution for the load balance. see page.2 of
https://www.ietf.org/rfc/rfc2782.txt .

so remove the default weight of 10.

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: reject bucket tagging requests and document unsupported

Signed-off-by: Casey Bodley <cbodley@redhat.com>

common/weighted_shufft: rewrite it to avoid copyright issue

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/MonClient: respect priority in SRV

before this change, we always choose the monitors with the lowest
priority and ignore the ones with higher priorty. but per
https://www.ietf.org/rfc/rfc2782.txt

> A client MUST attempt to contact the target host with the
> lowest-numbered priority it can reach

so, if a monitor is not reachable, we should try the ones with the
lowest ones *reachable*.

after this change, MonClient will memorize the monitors have been
contacted. and will continue hunting the ones not tried if the last
batch fail.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr: Configure Py root logger for Mgr modules

Add the CPlusPlusHandler to the Python root logger, thus the Mgr module itself and all 3rd party libraries will log to the Ceph log. This may help to identify problems in used 3rd party libraries.

Fixes: https://tracker.ceph.com/issues/38888
Signed-off-by: Volker Theile <vtheile@suse.com>

mgr/diskprediction_cloud: Correct base64 encode translate table

Correct base64 encode/decode string translate table

Signed-off-by: Rick Chen <rick.chen@prophetstor.com>

Merge pull request #25545 from zhouyuan/wip-ceph-immutable-obj-cache-daemon

tools: adding ceph level immutable obj cache daemon

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #26675 from trociny/wip-rbd-io-simple-scheduler

librbd: simple scheduler plugin for object dispatcher layer

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/pg_autoscaler: Fix python3 incompatibility
Fixes: http://tracker.ceph.com/issues/38626
Signed-off-by: Marius Schiffer <marius@mschiffer.de>