Merge PR #27249 into master

* refs/pull/27249/head:
qa/suites/rados/thrash-old-clients: fix up v1/v2 options

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #27133 into master

* refs/pull/27133/head:
log,global: do not start flusher thread until after we have our mon config
log: buffer log entries until flusher thread starts
log: open log file from flusher thread
common/ceph_context: fix log_to_file observer

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #26976 into master

* refs/pull/26976/head:
doc: add doc for cephfs lazyio

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #27199 into master

* refs/pull/27199/head:
mds: fix dump routine for session_info_t.used_inos
mds: fix some misleading log messages
client: comment fix in _lookup
mds: fix comment over MDRequestImpl
mds: use stdbool.h instead of hand-rolling our own bool type

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #27242 from theanalyst/beast-fe-ports

rgw: beast port parsing

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: beast: set a default port for endpoints

if no port is specified we don't set one(or set to 0), and don't exactly error
out, so rgw keeps running without being bound to a port. Set the port to default
in these cases

Fixes: http://tracker.ceph.com/issues/39000
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #27213 from ajarr/fix-38946

pybind: fix use of WriteOpCtx and ReadOpCtx

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27157 from Devp00l/issue-38933

mgr/dashboard: FixtureHelper

Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #27247 from Exotelis/polish-translation

mgr/dashboard: Add polish translation

Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: FixtureHelper

Fixes: https://tracker.ceph.com/issues/38933
Signed-off-by: Stephan Müller <smueller@suse.com>

Merge pull request #27148 from tchaikov/ceph-volume-terminal-unicode

ceph-volume: terminal: encode unicode when writing to stdout

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #27252 from yuriw/wip-yuriw-crontab

qa/tests: added nautilus-x run

Merge pull request #27250 from ivancich/wip-update-resharding-docs

rgw: updates to resharding documentation

Reviewed-by: Adam Emerson <aemerson@redhat.com>

rgw: updates to resharding documentation

Some minor updates to the resharding documentation with the aim of
clarifying the language and using terms consistently.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

qa/tests: added nautilus-x run

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

qa/suites/rados/thrash-old-clients: fix up v1/v2 options

- don't specify ceph.py options in the ceph.py
- instead, specify them in the per-version facet

Note that we don't currently have a way to do v2 only for the mon IPs, so
in the v2only cases, we are still binding the mons to v1.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #27243 from theanalyst/doc-scheduler

config-ref: add a note on current scheduler settings.

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge PR #27202 into master

* refs/pull/27202/head:
doc: Miscellaneous spelling fixes

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #27154 into master

* refs/pull/27154/head:
mgr/pg_autoscaler: apply bias to pg_num selection
mgr/pg_autoscaler: include pg_autoscale_bias in autoscale-status table
osd/osd_types,mon: add pg_autoscale_bias pool property

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #27012 into master

* refs/pull/27012/head:
common/options: allow (but to not prefer or require) secure mode
common/options: make clients prefer to connect to mons via secure mode
common/options: allow connections to mons in secure mode
common/options: prefer secure mode between monitors
auth/AuthRegistry: remove experimental flag for 'secure' mode

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

docs: rgw: add a x-ref to rados dmclock docs

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

mgr/dashboard: Add polish translation

Add polish translation.
Removed 5 unused trans-units.

Fixes: https://tracker.ceph.com/issues/38236
Signed-off-by: Sebastian Krah <skrah@suse.com>

config-ref: add a note on current scheduler settings.

Adding a note on configuirables for max concurrent requests and the rest of
experimental options for tuning dmclock scheduler

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

mds: fix dump routine for session_info_t.used_inos

It says it's dumping used_inos, but it's really dumping prealloc_inos.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #26887 from theanalyst/reshard-tenant

rgw admin: add tenant argument to reshard cancel

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27222 from ceph/wip-doc-dahsboard

mgr/dashboard: fix hacking.rst

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #27233 from tchaikov/script/ceph-backport.sh

script/ceph-backport.sh: allow user to specify remote repo

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #27135 from ricardoasmarques/iscsi-filter-images-based-on-features

mgr/dashboard: Validate iSCSI images features

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

script/ceph-backport.sh: allow user to specify remote repo

before this change, the script always push to "origin", in my case,
"origin" points to "ceph/ceph", while we need to create the branch in
private repo. and the name of private repo might vary. so would be great
if we can customize this setting.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26914 from votdev/issue_38331

mgr/dashboard: Add separate option to config SSL port

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #27166 from rhcs-dashboard/add-protractor-screenshoter-plugin

mgr/dashboard: Add protractor-screenshoter-plugin

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #24678 from rouming/master

test/fio: introduce fio ioengine: fio_ceph_messenger

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>

script/ceph-backport.sh: fix sanity test of issue number

* no need for checking negative numbers
* use regex for matching the number instead of using bash's pattern
  matching, otherwise i have
./ceph-backport.sh: line 79: syntax error in conditional expression:
unexpected token `('
  the pattern matching works fine if i use it in command line, but
  it fails to work in shell script.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27200 from neha-ojha/wip-21174-2

osd/PGLog: preserve original_crt to check rollbackability

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #27218 from yuriw/wip-yuriw-crontab

qa/tests: added nautilus-p2p to cron

mgr/dashboard: fix hacking.rst

Add missing dash to `-prod` and specify full command.

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

qa/tests: added nautilus-p2p to cron

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #27165 from yuriw/wip-yuriw-crontab-master_2

qa/tests: reduced number of jobs for `kcephfs`

Merge PR #27169 into master

* refs/pull/27169/head:
common/config: parse --default-$option as a default value

Reviewed-by: Sébastien Han <seb@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #27022 into master

* refs/pull/27022/head:
msg/async v2: make v2 work on rdma.

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge PR #27184 into master

* refs/pull/27184/head:
mgr/DaemonServer: handle_conf_change - fix broken locking

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #27187 into master

* refs/pull/27187/head:
mgr/dashboard/qa: Increase timeout for test_disable (tasks.mgr.dashboard.test_mgr_module.MgrModuleTelemetryTest)

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

ceph-volume: terminal: encode unicode when writing to stdout

python determins the encoding of stdout and stderr based on the LC_CTYPE
and PYTHONIOENCODING env variable, by default, python3's sys.stdout uses
'utf-8' as its encoding, so it will be able to write unicode string even
the stdout is not attached to a tty device. but when it comes to
python2, it will default to ascii if neither of these variabls is set.
so, if we are writing unicode using `_Write` in an environment where
LC_CTYPE and/or PYTHONIOENCODING are using non UTF-8 encoding, it chokes
by raising `UnicodeEncodeError` exception.

in this change, we add a wrapper around `_Write._writer` so it is able
to write unicode string in such a non-unicode-friendly environment.

for more info related the encoding of stdout and stderr, see
https://docs.python.org/3/using/cmdline.html#envvar-PYTHONIOENCODING .

Signed-off-by: Alfredo Deza <adeza@redhat.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #27196 into master

* refs/pull/27196/head:
mon/MonClient: do not dereference auth_supported.end()

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Add protractor-screenshoter-plugin

A plugin to save screenshots on E2E tests failures as well as
console logs, raw HTML, ...
All info visualized in an HTML report (screenshots are clickable to see full image).
Useful for debugging/fixing E2E tests.

Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #27211 from tchaikov/wip-crimson-partition-args

crimson/osd: partition args the right way

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #27153 from Devp00l/issue-38932

mgr/dashboard: Fixes tooltip behavior

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

test/pybind: fix use of rados.ReadOpCtx and rados.WriteOpCtx

classes WriteOpCtx and ReadOpCtx don't take in any arguments
during instantiation

Signed-off-by: Ramana Raja <rraja@redhat.com>

ceph_volume_client: fix use of rados.WriteOpCtx

Class WriteOpCtx doesn't take in any arguments during instantiation.

Fixes: http://tracker.ceph.com/issues/38946
Signed-off-by: Ramana Raja <rraja@redhat.com>

Merge pull request #27159 from smithfarm/wip-38934

doc: move Developer Guide to its own subdirectory

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #26561 from Devp00l/issue-38418

mgr/dashboard: Make preventDefault work with 400 errors

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

crimson/osd: partition args the right way

before this change, all "unknown" args are considered `ceph_args`, but
there is chance we have command line like

crimson-osd -i 1 -c ceph.conf --smp 1 --cpuset 00000001

in this case, "1" is listed in `unknown_args`, but it is also in
`parsed.options`. so we will put the second "1" into `ceph_args` even it
should be appended to `app_args`.

so, in this change, `std::partition_copy()` is replaced with a
hand-written loop which advances through the `argv` and `unknown_args`,
try to match as many consecutive options are possible. it assumes that
the order of args in `unknown_args` is the same as their order in
`argv[]`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27190 from tchaikov/wip-ceph.in-asan-only-for-debug

 ceph.in: only preload asan library for Debug build

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #26813 from DengkeDu/master

cmake/FindRocksDB: fix IMPORTED_LOCATION for ROCKSDB_LIBRARIES

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27204 from xiexingguo/wip-clean-weird-brace

osd/PG: move '}' to the proper place

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

doc: add doc for cephfs lazyio

Fixes: http://tracker.ceph.com/issues/38729
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

cmake/FindRocksDB: fix IMPORTED_LOCATION for ROCKSDB_LIBRARIES

Signed-off-by: dudengke <pinganddu90@gmail.com>

ceph.in: only preload asan library if it is enabled

my ceph cli hung when i build ceph using `CMAKE_BUILD_TYPE=RelWithDebInfo` and
`WITH_SEASTAR=ON`. because in that case, seastar detects for Sanitize flags
support of the compiler, but it does not pull in the build flags to enable
sanitizers. while `ceph` cli still preloaded the `asan_lib_path` because
`ASAN_LIBRARY` was set. this is not right, as we should enable ASAN at
runtime only if we enable it at compile-time.

so, in this change, we preload the ASAN library when `CMAKE_BUILD_TYPE`
is `Debug` and `WITH_SEASTAR=ON`, or `WITH_ASAN=ON`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph.in: normalize BOOL values found by get_cmake_variables()

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/PG: move '}' to the proper place

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #26942 from dzafman/wip-38616

Feature: Improvements to auto repair

Reviewed-by: Neha Ojha <nojha@redhat.com>

doc: Miscellaneous spelling fixes

While looking over the documentation I found a few words being misspelled.

Signed-off-by: Bryan Stillwell <bstillwell@godaddy.com>

Merge PR #27162 into master

* refs/pull/27162/head:
os/bluestore: nice used space report for bluefs-bdev-sizes command.
tools/ceph-kvstore-tool: command to show RocksDB compaction stats.

Reviewed-by: Sage Weil <sage@redhat.com>

osd/PGLog: preserve original_crt in merge_log()

Fixes: https://tracker.ceph.com/issues/21174
Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #26852 from theanalyst/wip-stale-instance-multisite

rgw admin: disable stale instance delete in a multiste env

Reviewed-by: Casey Bodley <cbodley@redhat.com>

os/bluestore: nice used space report for bluefs-bdev-sizes command.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

tools/ceph-kvstore-tool: command to show RocksDB compaction stats.

In fact this command will provide zeros for most of compaction stuff as
completed compactions are required to provide most of numbers.
But one can use it to learn per-level DB statistics: (amount of files,
occupied space, etc).

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #26936 into master

* refs/pull/26936/head:
msg: default to debug_ms=0

Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge PR #27069 into master

* refs/pull/27069/head:
mgr: Configure Py root logger for Mgr modules

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #27084 into master

* refs/pull/27084/head:
osd: process_copy_chunk remove obc ref before pg unlock

Reviewed-by: Myoungwon Oh <omwmw@sk.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #27142 into master

* refs/pull/27142/head:
osd/OSDMap: add log for better debugging
osd/OSDMap: calc_pg_upmaps - restrict optimization to origin pools only

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #27155 into master

* refs/pull/27155/head:
mon/ConfigMonitor: make 'config reset' idempotent

Reviewed-by: Sage Weil <sage@redhat.com>

mon/MonClient: do not dereference auth_supported.end()

if we are unable to find a supported method in allowed_methods, we will
dereference `auth_supported.end()` for searching it in desperation.

Reported-by: xie xingguo <xie.xingguo@zte.com.cn>
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #27172 into master

* refs/pull/27172/head:
mgr/devicehealth: Fix python 3 incompatiblity

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

common/config: parse --default-$option as a default value

Sometimes it is useful to specify an alternative default value for an
option via the command line such that it has a lower priority than the
mon config database, config file, the rest of the command line, or the
environment.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph.in: use get_cmake_variables(*args)

and return values in a list for better readability and less repeating

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27183 from tchaikov/wip-ceph.spec.in-devtoolset-8

rpm: s/devtoolset-7/devtoolset-8/

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #27177 from tchaikov/wip-crimson-stop-before-exit

crimson/osd: call at_exit() before stopping the engine

Reviewed-by: Samuel Just <sjust@redhat.com>

mds: fix some misleading log messages

There is no process_cap_release function. Change it to print __func__
instead.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

client: comment fix in _lookup

We're checking for Fs caps on the directory, _not_ a dentry lease.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

mds: fix comment over MDRequestImpl

Signed-off-by: Jeff Layton <jlayton@redhat.com>

mds: use stdbool.h instead of hand-rolling our own bool type

Signed-off-by: Jeff Layton <jlayton@redhat.com>

mgr/dashboard/qa: Increase timeout for test_disable (tasks.mgr.dashboard.test_mgr_module.MgrModuleTelemetryTest)

Fixes: https://tracker.ceph.com/issues/38820
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #27192 from idryomov/wip-krbd-discard-fixups

qa/workunits/rbd: fixups for the new krbd discard behavior

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #26925 from votdev/issue_38265

mgr/dashboard/qa: Improve tasks.mgr.test_dashboard.TestDashboard.test_standby

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #26396 from familyuu/interval

mgr/dashboard: Add refresh  interval to the dashboard landing page

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kanika Murarka <kmurarka@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #27136 from trociny/wip-38895

librbd: trash move return EBUSY instead of EINVAL for migrating image

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #27137 from trociny/wip-qa-rbd-qos

qa: add RBD QOS functional test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #27151 from trociny/wip-38928

librbd: look for pool metadata in default namespace

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #27118 from sebastian-philipp/mypy-gitignore

scrips/run_mypy: add .gitignore

Reviewed-by: Alfonso Martínez <almartin@redhat.com>

scrips/run_mypy: add .gitignore

Co-Authored-by: Patrick Nawracay <pnawracay@suse.com>
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #27181 from tchaikov/wip-test/test_weighted_shuffle

test/test_weighted_shuffle: enlarge epsilon

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

mgr/DaemonServer: handle_conf_change - fix broken locking

The is_locked_by_me() is only meant to be safe if you know
you hold the lock. It's meant to be used in an assertion.

Fixes: http://tracker.ceph.com/issues/38899
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

rpm: s/devtoolset-7/devtoolset-8/

should activate devtoolset-8 not devtoolset-7 for enabling GCC-8.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/test_weighted_shuffle: enlarge epsilon

be more permissive to address the following failure:

test_weighted_shuffle.cc:36: Failure
The difference between float(w) / total_weight and float(f.front()) /
samples is 0.010334432125091553, which exceeds epsilon, where
float(w) / total_weight evaluates to 0.53763443231582642,
float(f.front()) / samples evaluates to 0.52730000019073486, and
epsilon evaluates to 0.0099999997764825821

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: call at_exit() before stopping the engine

if we mkfs and stop the engine, any calls using the engine should/will
fail after `seastar::engine().exit(0)`. in the case of reactor::at_exit(),
it has:

assert(!_stopping);

in this change, we register the at_exit() calls before scheduling the
`engine().exit(0)` call.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27149 from wjwithagen/wjw-fix-stdFileSystem

cmake: use StdFilesystem::filesystem instead of stdc++fs

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27134 from tchaikov/wip-devtools-8

install-dep,rpm: use devtools-8 on amd64

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #26697 from tchaikov/wip-crimson-pg-fsm

crimon/osd: serve read requests

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd/pg: specify file variable values for editors

Signed-off-by: Kefu Chai <kchai@redhat.com>