crimson/osd: partition args the right way

before this change, all "unknown" args are considered `ceph_args`, but
there is chance we have command line like

crimson-osd -i 1 -c ceph.conf --smp 1 --cpuset 00000001

in this case, "1" is listed in `unknown_args`, but it is also in
`parsed.options`. so we will put the second "1" into `ceph_args` even it
should be appended to `app_args`.

so, in this change, `std::partition_copy()` is replaced with a
hand-written loop which advances through the `argv` and `unknown_args`,
try to match as many consecutive options are possible. it assumes that
the order of args in `unknown_args` is the same as their order in
`argv[]`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27190 from tchaikov/wip-ceph.in-asan-only-for-debug

 ceph.in: only preload asan library for Debug build

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #26813 from DengkeDu/master

cmake/FindRocksDB: fix IMPORTED_LOCATION for ROCKSDB_LIBRARIES

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27204 from xiexingguo/wip-clean-weird-brace

osd/PG: move '}' to the proper place

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

cmake/FindRocksDB: fix IMPORTED_LOCATION for ROCKSDB_LIBRARIES

Signed-off-by: dudengke <pinganddu90@gmail.com>

ceph.in: only preload asan library if it is enabled

my ceph cli hung when i build ceph using `CMAKE_BUILD_TYPE=RelWithDebInfo` and
`WITH_SEASTAR=ON`. because in that case, seastar detects for Sanitize flags
support of the compiler, but it does not pull in the build flags to enable
sanitizers. while `ceph` cli still preloaded the `asan_lib_path` because
`ASAN_LIBRARY` was set. this is not right, as we should enable ASAN at
runtime only if we enable it at compile-time.

so, in this change, we preload the ASAN library when `CMAKE_BUILD_TYPE`
is `Debug` and `WITH_SEASTAR=ON`, or `WITH_ASAN=ON`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph.in: normalize BOOL values found by get_cmake_variables()

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/PG: move '}' to the proper place

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #26942 from dzafman/wip-38616

Feature: Improvements to auto repair

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #27162 into master

* refs/pull/27162/head:
os/bluestore: nice used space report for bluefs-bdev-sizes command.
tools/ceph-kvstore-tool: command to show RocksDB compaction stats.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #26852 from theanalyst/wip-stale-instance-multisite

rgw admin: disable stale instance delete in a multiste env

Reviewed-by: Casey Bodley <cbodley@redhat.com>

os/bluestore: nice used space report for bluefs-bdev-sizes command.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

tools/ceph-kvstore-tool: command to show RocksDB compaction stats.

In fact this command will provide zeros for most of compaction stuff as
completed compactions are required to provide most of numbers.
But one can use it to learn per-level DB statistics: (amount of files,
occupied space, etc).

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #26936 into master

* refs/pull/26936/head:
msg: default to debug_ms=0

Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge PR #27069 into master

* refs/pull/27069/head:
mgr: Configure Py root logger for Mgr modules

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #27084 into master

* refs/pull/27084/head:
osd: process_copy_chunk remove obc ref before pg unlock

Reviewed-by: Myoungwon Oh <omwmw@sk.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #27142 into master

* refs/pull/27142/head:
osd/OSDMap: add log for better debugging
osd/OSDMap: calc_pg_upmaps - restrict optimization to origin pools only

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #27155 into master

* refs/pull/27155/head:
mon/ConfigMonitor: make 'config reset' idempotent

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #27172 into master

* refs/pull/27172/head:
mgr/devicehealth: Fix python 3 incompatiblity

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

ceph.in: use get_cmake_variables(*args)

and return values in a list for better readability and less repeating

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27183 from tchaikov/wip-ceph.spec.in-devtoolset-8

rpm: s/devtoolset-7/devtoolset-8/

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #27177 from tchaikov/wip-crimson-stop-before-exit

crimson/osd: call at_exit() before stopping the engine

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #27192 from idryomov/wip-krbd-discard-fixups

qa/workunits/rbd: fixups for the new krbd discard behavior

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #26925 from votdev/issue_38265

mgr/dashboard/qa: Improve tasks.mgr.test_dashboard.TestDashboard.test_standby

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #26396 from familyuu/interval

mgr/dashboard: Add refresh  interval to the dashboard landing page

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kanika Murarka <kmurarka@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #27136 from trociny/wip-38895

librbd: trash move return EBUSY instead of EINVAL for migrating image

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #27137 from trociny/wip-qa-rbd-qos

qa: add RBD QOS functional test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #27151 from trociny/wip-38928

librbd: look for pool metadata in default namespace

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #27118 from sebastian-philipp/mypy-gitignore

scrips/run_mypy: add .gitignore

Reviewed-by: Alfonso Martínez <almartin@redhat.com>

scrips/run_mypy: add .gitignore

Co-Authored-by: Patrick Nawracay <pnawracay@suse.com>
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #27181 from tchaikov/wip-test/test_weighted_shuffle

test/test_weighted_shuffle: enlarge epsilon

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

rpm: s/devtoolset-7/devtoolset-8/

should activate devtoolset-8 not devtoolset-7 for enabling GCC-8.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/test_weighted_shuffle: enlarge epsilon

be more permissive to address the following failure:

test_weighted_shuffle.cc:36: Failure
The difference between float(w) / total_weight and float(f.front()) /
samples is 0.010334432125091553, which exceeds epsilon, where
float(w) / total_weight evaluates to 0.53763443231582642,
float(f.front()) / samples evaluates to 0.52730000019073486, and
epsilon evaluates to 0.0099999997764825821

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: call at_exit() before stopping the engine

if we mkfs and stop the engine, any calls using the engine should/will
fail after `seastar::engine().exit(0)`. in the case of reactor::at_exit(),
it has:

assert(!_stopping);

in this change, we register the at_exit() calls before scheduling the
`engine().exit(0)` call.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27149 from wjwithagen/wjw-fix-stdFileSystem

cmake: use StdFilesystem::filesystem instead of stdc++fs

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27134 from tchaikov/wip-devtools-8

install-dep,rpm: use devtools-8 on amd64

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #26697 from tchaikov/wip-crimson-pg-fsm

crimon/osd: serve read requests

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd/pg: specify file variable values for editors

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: add pg backends

* add exceptions.h for EIO, ENOENT error handling
* add PGBackend for serving i/o requests

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: Document new pg state and changes to auto repair behavior

Fixes: http://tracker.ceph.com/issues/38616
Signed-off-by: David Zafman <dzafman@redhat.com>

osd, test: Add num_shards_repaired to osd_stat_t for pushes with repair set 3(3)

Fixes: http://tracker.ceph.com/issues/38616
Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Track num_objects_repaired in pg stats 2(3)

Leave repair pg state on until recovery finishes or a new scrub starts

Fixes: http://tracker.ceph.com/issues/38616
Signed-off-by: David Zafman <dzafman@redhat.com>

mgr/devicehealth: Fix python 3 incompatiblity
Fixes: http://tracker.ceph.com/issues/38939
Signed-off-by: Marius Schiffer <marius@mschiffer.de>

qa/workunits/rbd: fixups for the new krbd discard behavior

With discard_granularity set to alloc_size, we no longer get object
size alignment from blk_bio_discard_split().

This assumption is pretty deeply ingrained in krbd_data_pool.sh, so
make it explicit.  For krbd_fallocate.sh, just fix the expectation.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa: krbd_fallocate.sh: zero can be munged to truncate

The test case is issuing discards that span two objects: the tail of
the first object is truncated, the head of the second object is zeroed.
These discards aren't serial, so there is a race:

  discard i ~ i + 1: truncate i, zero i + 1
  discard i + 1 ~ i + 2: truncate i + 1, zero i + 2

can be executed as

  truncate i + 1, zero i + 2, truncate i, zero i + 1

For object i + 1, the sequence ends up being truncate tail, then zero
head.  This zero op is munged to truncate on the OSD, resulting in size
0 instead of OBJECT_SIZE / 2.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge PR #27124 into master

* refs/pull/27124/head:
osd/pg: use emplace() to construct new element in-place

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #27079 into master

* refs/pull/27079/head:
mgr/pg_autoscaler: Fix python3 incompatibility

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #27113 into master

* refs/pull/27113/head:
mgr/diskprediction_cloud: Correct base64 encode translate table

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #27156 into master

* refs/pull/27156/head:
common/blkdev: get_device_id: behave if model is lvm and id_model_enc isn't there

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #26100 from ceph/wip-s3a-hadoop-ver

qa/tests: update s3a hadoop versions used for test

Reviewed-by: Casey Bodley <cbodley@redhat.com>

osd: process_copy_chunk remove obc ref before pg unlock

drop the obc ref when pg locked is more safer, if not, on_change
may failed to clear it, and on_flushed may find leak.

Fixes: http://tracker.ceph.com/issues/38842
Signed-off-by: Kefu Chai kchai@redhat.com
Signed-off-by: Zengran Zhang <zhangzengran@sangfor.com.cn>

Merge pull request #27150 from tchaikov/wip-ceph.spec.in/immutable-object-cache

rpm: immutable-object-cache related changes

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

crimson/osd/pg: wait until pg is active

hold i/o request until pg is active

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: handle MOSDOp in OSD

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27126 from tchaikov/wip-monclient-srv-priority

mon/MonClient: add proper SRV priority support

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #25808 from joke-lee/fix_dump_usage_categories_info

rgw: usage dump_unsigned instead dump_int in dump_usage_categories_info

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #26070 from iliul/fix-bug-38002

rgw: Replace COMPLETE_MULTIPART_MAX_LEN with rgw_max_put_param_size

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #26947 from IrekFasikhov/fixRGWMultiDeleteObject

rgw: fix RGWDeleteMultiObj::verify_permission()

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

common/blkdev: get_device_id: behave if model is lvm and id_model_enc isn't there

If that happens, leave ID_MODEL blank and we'll fall back to something
else, like ID_SERIAL (which is present in each case I've seen this).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #26634 from theanalyst/scheduler-beast-ctx

rgw: build async scheduler only when beast is built

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

mon/ConfigMonitor: make 'config reset' idempotent

This partially revert 1bc9c86d08cfb408768c080ed5ea1ea82325ed57.
It's generally not a good idea if a command is not idempotent.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #25451 from nathan-weinberg/gateway-tests

mgr/dashboard: Added breadcrumb tests to Object Gateway menu items

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

osd/OSDMap: add log for better debugging

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/OSDMap: calc_pg_upmaps - restrict optimization to origin pools only

The current implementation will try to cancel any pg_upmaps that
would otherwise re-map a PG out from an underfull osd, which is wrong,
e.g., because it could reliably fire the following assert:

src/osd/OSDMap.cc: 4405: FAILED assert(osd_weight.count(i.first))

Also it would not match the expectation if automatic balancing
has been strictly restricted to some specific pools by admin.

Fix by excluding any wild PG that does not belong to the origin pools
passed in when trying to do upmap/unmap.

Fixes: http://tracker.ceph.com/issues/38897
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #26850 from nathan-weinberg/nfs-breadcrumb

mgr/dashboard: Added breadcrumb tests to NFS menu

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

librbd: look for pool metadata in default namespace

when applying pool level config overrides

Fixes: https://tracker.ceph.com/issues/38928
Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: trash move return EBUSY instead of EINVAL for migrating image

Fixes: https://tracker.ceph.com/issues/38895
Signed-off-by: Mykola Golub <mgolub@suse.com>

rpm,deb: let immutable-object-cache depend on librados2

ceph-common depends on librados2, but ceph-immutable-object-cache only
uses librados2's C++ API, so librados2 would suffice.

rpmbuild is able to figure out the .so dependencies, and let
ceph-immutable-object-cache depend on the package which offers
librados2, but we'd better make sure immutable-object-cache is linked
against the librados2 of the same version, to ensure that librados2
offers the correct C++ symbols. as we only version the C++ API in the
librados2 after nautilus.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27090 from xiexingguo/wip-config-mon-core

mon/ConfigMonitor: make 'num' of 'config reset' optional; allow target version 0

Reviewed-by: Sage Weil <sage@redhat.com>

rpm: remove libradospp1 from immutable-object-cache's Requires

* remove "libradospp1" from immutable-object-cache's runtime
  dependencies. this package was removed in 65c8733b

Signed-off-by: Kefu Chai <kchai@redhat.com>

rpm: do not use "-n" for immutable-object-cache

as immutable-object-cache has the prefix of "ceph", no need to use "-n"
option. so this change is but a cleanup.

* use "%package immutable-object-cache" not "%package
  ceph-immutable-object-cache" for naming subpackage of
  "ceph-immutable-object-cache". this would result in package name of
  "ceph-immutable-object-cache". see
  http://ftp.rpm.org/max-rpm/s1-rpm-subpack-spec-file-changes.html
* the same applies to other directives with "-n" options, see
  http://ftp.rpm.org/max-rpm/s1-rpm-inside-package-directive.html

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/pg: use emplace() to construct new element in-place

replace emplace() and emplace_back() with insert() and push_back()
calls to construct the new element in-place and avoid the overhead
of move constructor calls

Signed-off-by: Zengran Zhang <zhangzengran@sangfor.com.cn>

mon/ConfigMonitor: make 'num' of 'config reset' command optional

This way it can be used to fast cancel/undo the last command.
Also make the tip message a litter bit nicer..

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #27147 into master

* refs/pull/27147/head:
qa/workunits/mon/config.sh: s|bin/ceph|ceph|

Reviewed-by: Sage Weil <sage@redhat.com>

cmake: use StdFilesystem::filesystem instead of stdc++fs

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #26531 from majianpeng/msg-async-optimization

msg/async: optimizations

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #27107 from neha-ojha/wip-38845

mon/Monitor.cc: print min_mon_release correctly

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27097 from tchaikov/wip-option-sv-as-keys

common/config: use string_view for keys

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #26220 from ashitakasam/master

common/Formatter: escape printed buffer in XMLFormatter::dump_format_va()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #27138 into master

* refs/pull/27138/head:
qa/suites/upgrade/nautilus-x: misc updates

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

test, osd: Improvements to auto_repair 1(3)

Allow auto_repair for replicated bluestore pools
Regular scrub within auto repair parameters will trigger deep scrub
New state failed_repair if PG repair attempt could not fix everything
Set failed_repair if not possible to repair anything

Fixes: http://tracker.ceph.com/issues/38616
Signed-off-by: David Zafman <dzafman@redhat.com>

test: osd-scrub-repair.sh: use corrupt_and_repair_lrc for lrc tests

Fix for argument handling of create_ec_pool()
Always pass a value for allow_overwrites for consistency

Caused by: 3ca750d41dfe33c6efea4abc96d2bd426a9742b9

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Publish stats after all changes made

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Fixes for 64-bit PG state

Signed-off-by: David Zafman <dzafman@redhat.com>

qa/workunits/mon/config.sh: s|bin/ceph|ceph|

so we can verify that ceph is able to reject wrong debug levels.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27143 from dzafman/wip-38896

doc: Fix the pg states and auto repair config options

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: Fix the pg states and auto repair config options

Fixes: http://tracker.ceph.com/issues/38896
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #27140 from neha-ojha/wip-fix-cron

qa/crontab/teuthology-cronjobs: fix suite-branch

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #27129 from yuriw/wip-yuriw-subsets-master

qa/tests: added the subset clause for nautilus branch

qa/crontab/teuthology-cronjobs: fix suite-branch

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/suites/upgrade/nautilus-x: misc updates

- start with msgr2 enabled (defaults)
- run nautilus branch for workunits
- drop msgr2 enable step at teh end
- add octopus placeholder (although it is empty for now)

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #26939 into master

* refs/pull/26939/head:
os/bluestore: unconditionally cap chunks returned by allocator to 2^31
os/bluestore: start using 64-bit intervals for bitmap allocator
os/bluestore: make bluestore interval base template.
tests/fastbmap_alloc: UT to reproduce 4G allocation bug
os/bluestore: os/bluestore: implement dump for bitmap allocator
os/bluestore be more tolerant to lack of space for bluefs.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #27086 into master

* refs/pull/27086/head:
msg: remove unused header file in Messenger.h

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #27111 into master

* refs/pull/27111/head:
pybind/rados: should pass "name" to cstr()

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #27112 into master

* refs/pull/27112/head:
qa/suites: do not test luminous-x upgrade path

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #27047 from tianshan/fix_38811

rgw: fix read not exists null version return wrong

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27102 from adamemerson/wip-one-rados-handle-to-rule-them-all

One Rados Handle to Rule Them All

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #25999 from cbodley/wip-37945

rgw: Fix S3 compatibility bug when CORS is not found

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #26998 from cbodley/wip-38328

rgw: don't crash on missing /etc/mime.types

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #26952 from cbodley/wip-rgw-bucket-tagging-notsupp

rgw: reject bucket tagging requests and document unsupported

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>