test/fio/fio_ceph_messenger: catch up v2 proto changes by using dummy auth

This change makes v2 protocol work.

Signed-off-by: Roman Penyaev <rpenyaev@suse.de>

Merge pull request #27148 from tchaikov/ceph-volume-terminal-unicode

ceph-volume: terminal: encode unicode when writing to stdout

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #27252 from yuriw/wip-yuriw-crontab

qa/tests: added nautilus-x run

Merge pull request #27250 from ivancich/wip-update-resharding-docs

rgw: updates to resharding documentation

Reviewed-by: Adam Emerson <aemerson@redhat.com>

rgw: updates to resharding documentation

Some minor updates to the resharding documentation with the aim of
clarifying the language and using terms consistently.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

qa/tests: added nautilus-x run

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #27243 from theanalyst/doc-scheduler

config-ref: add a note on current scheduler settings.

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge PR #27202 into master

* refs/pull/27202/head:
doc: Miscellaneous spelling fixes

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #27154 into master

* refs/pull/27154/head:
mgr/pg_autoscaler: apply bias to pg_num selection
mgr/pg_autoscaler: include pg_autoscale_bias in autoscale-status table
osd/osd_types,mon: add pg_autoscale_bias pool property

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #27012 into master

* refs/pull/27012/head:
common/options: allow (but to not prefer or require) secure mode
common/options: make clients prefer to connect to mons via secure mode
common/options: allow connections to mons in secure mode
common/options: prefer secure mode between monitors
auth/AuthRegistry: remove experimental flag for 'secure' mode

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

docs: rgw: add a x-ref to rados dmclock docs

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

config-ref: add a note on current scheduler settings.

Adding a note on configuirables for max concurrent requests and the rest of
experimental options for tuning dmclock scheduler

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #26887 from theanalyst/reshard-tenant

rgw admin: add tenant argument to reshard cancel

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27222 from ceph/wip-doc-dahsboard

mgr/dashboard: fix hacking.rst

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #27233 from tchaikov/script/ceph-backport.sh

script/ceph-backport.sh: allow user to specify remote repo

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #27135 from ricardoasmarques/iscsi-filter-images-based-on-features

mgr/dashboard: Validate iSCSI images features

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

script/ceph-backport.sh: allow user to specify remote repo

before this change, the script always push to "origin", in my case,
"origin" points to "ceph/ceph", while we need to create the branch in
private repo. and the name of private repo might vary. so would be great
if we can customize this setting.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26914 from votdev/issue_38331

mgr/dashboard: Add separate option to config SSL port

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #27166 from rhcs-dashboard/add-protractor-screenshoter-plugin

mgr/dashboard: Add protractor-screenshoter-plugin

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #24678 from rouming/master

test/fio: introduce fio ioengine: fio_ceph_messenger

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>

script/ceph-backport.sh: fix sanity test of issue number

* no need for checking negative numbers
* use regex for matching the number instead of using bash's pattern
  matching, otherwise i have
./ceph-backport.sh: line 79: syntax error in conditional expression:
unexpected token `('
  the pattern matching works fine if i use it in command line, but
  it fails to work in shell script.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27200 from neha-ojha/wip-21174-2

osd/PGLog: preserve original_crt to check rollbackability

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #27218 from yuriw/wip-yuriw-crontab

qa/tests: added nautilus-p2p to cron

mgr/dashboard: fix hacking.rst

Add missing dash to `-prod` and specify full command.

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

qa/tests: added nautilus-p2p to cron

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #27165 from yuriw/wip-yuriw-crontab-master_2

qa/tests: reduced number of jobs for `kcephfs`

Merge PR #27169 into master

* refs/pull/27169/head:
common/config: parse --default-$option as a default value

Reviewed-by: Sébastien Han <seb@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #27022 into master

* refs/pull/27022/head:
msg/async v2: make v2 work on rdma.

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge PR #27184 into master

* refs/pull/27184/head:
mgr/DaemonServer: handle_conf_change - fix broken locking

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #27187 into master

* refs/pull/27187/head:
mgr/dashboard/qa: Increase timeout for test_disable (tasks.mgr.dashboard.test_mgr_module.MgrModuleTelemetryTest)

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

ceph-volume: terminal: encode unicode when writing to stdout

python determins the encoding of stdout and stderr based on the LC_CTYPE
and PYTHONIOENCODING env variable, by default, python3's sys.stdout uses
'utf-8' as its encoding, so it will be able to write unicode string even
the stdout is not attached to a tty device. but when it comes to
python2, it will default to ascii if neither of these variabls is set.
so, if we are writing unicode using `_Write` in an environment where
LC_CTYPE and/or PYTHONIOENCODING are using non UTF-8 encoding, it chokes
by raising `UnicodeEncodeError` exception.

in this change, we add a wrapper around `_Write._writer` so it is able
to write unicode string in such a non-unicode-friendly environment.

for more info related the encoding of stdout and stderr, see
https://docs.python.org/3/using/cmdline.html#envvar-PYTHONIOENCODING .

Signed-off-by: Alfredo Deza <adeza@redhat.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #27196 into master

* refs/pull/27196/head:
mon/MonClient: do not dereference auth_supported.end()

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Add protractor-screenshoter-plugin

A plugin to save screenshots on E2E tests failures as well as
console logs, raw HTML, ...
All info visualized in an HTML report (screenshots are clickable to see full image).
Useful for debugging/fixing E2E tests.

Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #27211 from tchaikov/wip-crimson-partition-args

crimson/osd: partition args the right way

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #27153 from Devp00l/issue-38932

mgr/dashboard: Fixes tooltip behavior

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #27159 from smithfarm/wip-38934

doc: move Developer Guide to its own subdirectory

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #26561 from Devp00l/issue-38418

mgr/dashboard: Make preventDefault work with 400 errors

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

crimson/osd: partition args the right way

before this change, all "unknown" args are considered `ceph_args`, but
there is chance we have command line like

crimson-osd -i 1 -c ceph.conf --smp 1 --cpuset 00000001

in this case, "1" is listed in `unknown_args`, but it is also in
`parsed.options`. so we will put the second "1" into `ceph_args` even it
should be appended to `app_args`.

so, in this change, `std::partition_copy()` is replaced with a
hand-written loop which advances through the `argv` and `unknown_args`,
try to match as many consecutive options are possible. it assumes that
the order of args in `unknown_args` is the same as their order in
`argv[]`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27190 from tchaikov/wip-ceph.in-asan-only-for-debug

 ceph.in: only preload asan library for Debug build

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #26813 from DengkeDu/master

cmake/FindRocksDB: fix IMPORTED_LOCATION for ROCKSDB_LIBRARIES

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27204 from xiexingguo/wip-clean-weird-brace

osd/PG: move '}' to the proper place

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

cmake/FindRocksDB: fix IMPORTED_LOCATION for ROCKSDB_LIBRARIES

Signed-off-by: dudengke <pinganddu90@gmail.com>

ceph.in: only preload asan library if it is enabled

my ceph cli hung when i build ceph using `CMAKE_BUILD_TYPE=RelWithDebInfo` and
`WITH_SEASTAR=ON`. because in that case, seastar detects for Sanitize flags
support of the compiler, but it does not pull in the build flags to enable
sanitizers. while `ceph` cli still preloaded the `asan_lib_path` because
`ASAN_LIBRARY` was set. this is not right, as we should enable ASAN at
runtime only if we enable it at compile-time.

so, in this change, we preload the ASAN library when `CMAKE_BUILD_TYPE`
is `Debug` and `WITH_SEASTAR=ON`, or `WITH_ASAN=ON`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph.in: normalize BOOL values found by get_cmake_variables()

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/PG: move '}' to the proper place

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #26942 from dzafman/wip-38616

Feature: Improvements to auto repair

Reviewed-by: Neha Ojha <nojha@redhat.com>

doc: Miscellaneous spelling fixes

While looking over the documentation I found a few words being misspelled.

Signed-off-by: Bryan Stillwell <bstillwell@godaddy.com>

Merge PR #27162 into master

* refs/pull/27162/head:
os/bluestore: nice used space report for bluefs-bdev-sizes command.
tools/ceph-kvstore-tool: command to show RocksDB compaction stats.

Reviewed-by: Sage Weil <sage@redhat.com>

osd/PGLog: preserve original_crt in merge_log()

Fixes: https://tracker.ceph.com/issues/21174
Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #26852 from theanalyst/wip-stale-instance-multisite

rgw admin: disable stale instance delete in a multiste env

Reviewed-by: Casey Bodley <cbodley@redhat.com>

os/bluestore: nice used space report for bluefs-bdev-sizes command.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

tools/ceph-kvstore-tool: command to show RocksDB compaction stats.

In fact this command will provide zeros for most of compaction stuff as
completed compactions are required to provide most of numbers.
But one can use it to learn per-level DB statistics: (amount of files,
occupied space, etc).

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #26936 into master

* refs/pull/26936/head:
msg: default to debug_ms=0

Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge PR #27069 into master

* refs/pull/27069/head:
mgr: Configure Py root logger for Mgr modules

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #27084 into master

* refs/pull/27084/head:
osd: process_copy_chunk remove obc ref before pg unlock

Reviewed-by: Myoungwon Oh <omwmw@sk.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #27142 into master

* refs/pull/27142/head:
osd/OSDMap: add log for better debugging
osd/OSDMap: calc_pg_upmaps - restrict optimization to origin pools only

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #27155 into master

* refs/pull/27155/head:
mon/ConfigMonitor: make 'config reset' idempotent

Reviewed-by: Sage Weil <sage@redhat.com>

mon/MonClient: do not dereference auth_supported.end()

if we are unable to find a supported method in allowed_methods, we will
dereference `auth_supported.end()` for searching it in desperation.

Reported-by: xie xingguo <xie.xingguo@zte.com.cn>
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #27172 into master

* refs/pull/27172/head:
mgr/devicehealth: Fix python 3 incompatiblity

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

common/config: parse --default-$option as a default value

Sometimes it is useful to specify an alternative default value for an
option via the command line such that it has a lower priority than the
mon config database, config file, the rest of the command line, or the
environment.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph.in: use get_cmake_variables(*args)

and return values in a list for better readability and less repeating

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27183 from tchaikov/wip-ceph.spec.in-devtoolset-8

rpm: s/devtoolset-7/devtoolset-8/

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #27177 from tchaikov/wip-crimson-stop-before-exit

crimson/osd: call at_exit() before stopping the engine

Reviewed-by: Samuel Just <sjust@redhat.com>

mgr/dashboard/qa: Increase timeout for test_disable (tasks.mgr.dashboard.test_mgr_module.MgrModuleTelemetryTest)

Fixes: https://tracker.ceph.com/issues/38820
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #27192 from idryomov/wip-krbd-discard-fixups

qa/workunits/rbd: fixups for the new krbd discard behavior

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #26925 from votdev/issue_38265

mgr/dashboard/qa: Improve tasks.mgr.test_dashboard.TestDashboard.test_standby

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #26396 from familyuu/interval

mgr/dashboard: Add refresh  interval to the dashboard landing page

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kanika Murarka <kmurarka@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #27136 from trociny/wip-38895

librbd: trash move return EBUSY instead of EINVAL for migrating image

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #27137 from trociny/wip-qa-rbd-qos

qa: add RBD QOS functional test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #27151 from trociny/wip-38928

librbd: look for pool metadata in default namespace

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #27118 from sebastian-philipp/mypy-gitignore

scrips/run_mypy: add .gitignore

Reviewed-by: Alfonso Martínez <almartin@redhat.com>

scrips/run_mypy: add .gitignore

Co-Authored-by: Patrick Nawracay <pnawracay@suse.com>
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #27181 from tchaikov/wip-test/test_weighted_shuffle

test/test_weighted_shuffle: enlarge epsilon

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

mgr/DaemonServer: handle_conf_change - fix broken locking

The is_locked_by_me() is only meant to be safe if you know
you hold the lock. It's meant to be used in an assertion.

Fixes: http://tracker.ceph.com/issues/38899
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

rpm: s/devtoolset-7/devtoolset-8/

should activate devtoolset-8 not devtoolset-7 for enabling GCC-8.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/test_weighted_shuffle: enlarge epsilon

be more permissive to address the following failure:

test_weighted_shuffle.cc:36: Failure
The difference between float(w) / total_weight and float(f.front()) /
samples is 0.010334432125091553, which exceeds epsilon, where
float(w) / total_weight evaluates to 0.53763443231582642,
float(f.front()) / samples evaluates to 0.52730000019073486, and
epsilon evaluates to 0.0099999997764825821

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: call at_exit() before stopping the engine

if we mkfs and stop the engine, any calls using the engine should/will
fail after `seastar::engine().exit(0)`. in the case of reactor::at_exit(),
it has:

assert(!_stopping);

in this change, we register the at_exit() calls before scheduling the
`engine().exit(0)` call.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27149 from wjwithagen/wjw-fix-stdFileSystem

cmake: use StdFilesystem::filesystem instead of stdc++fs

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27134 from tchaikov/wip-devtools-8

install-dep,rpm: use devtools-8 on amd64

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #26697 from tchaikov/wip-crimson-pg-fsm

crimon/osd: serve read requests

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd/pg: specify file variable values for editors

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: add pg backends

* add exceptions.h for EIO, ENOENT error handling
* add PGBackend for serving i/o requests

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: Document new pg state and changes to auto repair behavior

Fixes: http://tracker.ceph.com/issues/38616
Signed-off-by: David Zafman <dzafman@redhat.com>

osd, test: Add num_shards_repaired to osd_stat_t for pushes with repair set 3(3)

Fixes: http://tracker.ceph.com/issues/38616
Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Track num_objects_repaired in pg stats 2(3)

Leave repair pg state on until recovery finishes or a new scrub starts

Fixes: http://tracker.ceph.com/issues/38616
Signed-off-by: David Zafman <dzafman@redhat.com>

mgr/dashboard: iscsi: feature names in error messages

Required/supported RBD features should be included in
the error messages.

Signed-off-by: Ricardo Marques <rimarques@suse.com>

mgr/devicehealth: Fix python 3 incompatiblity
Fixes: http://tracker.ceph.com/issues/38939
Signed-off-by: Marius Schiffer <marius@mschiffer.de>

mgr/dashboard: Validate iSCSI images features

Will also filter the list of available images.

Fixes: https://tracker.ceph.com/issues/38074
Signed-off-by: Ricardo Marques <rimarques@suse.com>

qa/workunits/rbd: fixups for the new krbd discard behavior

With discard_granularity set to alloc_size, we no longer get object
size alignment from blk_bio_discard_split().

This assumption is pretty deeply ingrained in krbd_data_pool.sh, so
make it explicit.  For krbd_fallocate.sh, just fix the expectation.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa: krbd_fallocate.sh: zero can be munged to truncate

The test case is issuing discards that span two objects: the tail of
the first object is truncated, the head of the second object is zeroed.
These discards aren't serial, so there is a race:

  discard i ~ i + 1: truncate i, zero i + 1
  discard i + 1 ~ i + 2: truncate i + 1, zero i + 2

can be executed as

  truncate i + 1, zero i + 2, truncate i, zero i + 1

For object i + 1, the sequence ends up being truncate tail, then zero
head.  This zero op is munged to truncate on the OSD, resulting in size
0 instead of OBJECT_SIZE / 2.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

osd/PGLog: preserve original_crt in proc_replica_log()

This is to prevent rolling back further than deleted object version.

Signed-off-by: Neha Ojha <nojha@redhat.com>

osd/PGLog: should not rollback further than deleted object version

When a deleted object becomes a divergent entry in the pg log,
we should not be able to rollback to a version of the deleted
object that doesn't exist.

To avoid this, we need to preserve the original crt of the pg log,
before we update it in rewind_from_head() and use that to decide whether
we can rollback or not in _merge_object_divergent_entries().

Fixes: http://tracker.ceph.com/issues/36739
Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge PR #27124 into master

* refs/pull/27124/head:
osd/pg: use emplace() to construct new element in-place

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #27079 into master

* refs/pull/27079/head:
mgr/pg_autoscaler: Fix python3 incompatibility

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #27113 into master

* refs/pull/27113/head:
mgr/diskprediction_cloud: Correct base64 encode translate table

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #27156 into master

* refs/pull/27156/head:
common/blkdev: get_device_id: behave if model is lvm and id_model_enc isn't there

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #26100 from ceph/wip-s3a-hadoop-ver

qa/tests: update s3a hadoop versions used for test

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/tests: reduced number of jobs for `kcephfs` suite
now we get 502 jobs and run 3 times a week
this change will generate/run 44 jobs every day

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

osd: process_copy_chunk remove obc ref before pg unlock

drop the obc ref when pg locked is more safer, if not, on_change
may failed to clear it, and on_flushed may find leak.

Fixes: http://tracker.ceph.com/issues/38842
Signed-off-by: Kefu Chai kchai@redhat.com
Signed-off-by: Zengran Zhang <zhangzengran@sangfor.com.cn>

Merge pull request #27150 from tchaikov/wip-ceph.spec.in/immutable-object-cache

rpm: immutable-object-cache related changes

Reviewed-by: Jason Dillaman <dillaman@redhat.com>