Merge PR #30308 into master

* refs/pull/30308/head:
doc/dev: add teuthology priority recommendations

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

doc/dev: add teuthology priority recommendations

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #30269 from dabukalam/patch-1

doc: Update ceph-deploy docs from dumpling to nautilus

Reviewed-by: Alfredo Deza <adeza@redhat.com>

doc: Update ceph-deploy docs from dumpling to nautilus

Signed-off-by: Danny Abukalam <danny@softiron.co.uk>

Merge pull request #30274 from alfredodeza/wip-rm41660

ceph-volume: fix stderr failure to decode/encode when redirected

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #30287 from tchaikov/wip-run-cbt

script/run-cbt.sh: check option correctly

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #30290 from tchaikov/wip-doc-cbt

doc/dev/crimson.rst: document CBT testing

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #30289 from wjwithagen/wjw-fix-pylint-dashboard-progress.py

dashboard/services: fix lint error

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

dashboard/services: fix lint error

```
lint run-test: commands[0] | flake8
 many blank lines (3)
1     E303 too many blank lines (3)
```

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

ceph-volume tests create a test file for checking unicode output

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests create a shell test for functional unicode

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests verify new logging fallback and encodings in terminal

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume terminal remove unicode stream handler

This caused problems in environments where stderr was redirected, since
stderr sets the encoding to None. Getting it back again allows
everything to work correctly, and keeps all the current unit tests
passing

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: instantiate the new terminal logger in main()

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume create a logger for the terminal

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge PR #30155 into master

* refs/pull/30155/head:
mgr/MgrClient: send MMgrCommand for octopus+ mgrs
mon/MgrMap: include mgr_features in mgrmap
mgr: accept MMgrCommand[Reply] messages (in addition to MCommand[Reply])

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #30162 from cyx1231st/wip-seastar-msgr-test-failover-crimson

crimson/test: v2 failover tests with crimson FailoverTestPeer

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/dev/crimson.rst: document CBT testing

Signed-off-by: Kefu Chai <kchai@redhat.com>

script/run-cbt.sh: check option correctly

* s/source_dir/source-dir/: as the option is named "--source-dir", not
  "--source_dir"
* cd $build_dir, before launching/stopping vstart. as vstart.sh is
  supposed to run from the build directory

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30111 from tspmelo/wip-nfs-read

mgr/dashboard: Set RO as the default access_type for RGW NFS exports

Reviewed-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: progress: support rbd_support module async tasks (#29424)

mgr/dashboard: progress: support rbd_support module async tasks

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #30095 from rjfd/wip-fix-progress-qa

qa/mgr/progress: fix timeout error when waiting for osd in event

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #30078 from rhcs-dashboard/fix-41602-master

mgr/dashboard: disable pylint's --py3k flag

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>

Merge pull request #30126 from dvanders/dvanders_spec

ceph.spec.in: reserve more memory per build job

Reviewed-By: Nathan Cutler <nculter@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30258 from tchaikov/wip-41721

test/cls_rbd/test_cls_rbd: update TestClsRbd.sparsify

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #30273 from neha-ojha/wip-documenting-ceph

doc/start/documenting-ceph.rst: make better doc recommendations

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

doc/start/documenting-ceph.rst: make better doc recommendations

I do not think "Editors" is relevant anymore.

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge PR #30187 into master

* refs/pull/30187/head:
script/backport-resolve-issue: better error message

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-volume create a new tox.ini for shell-based tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #30226 from neha-ojha/wip-41385-2

osd/MissingLoc.cc: do not rely on missing_loc_sources only

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed By: Xie Xingguo <xie.xingguo@zte.com.cn>

Merge PR #30196 into master

* refs/pull/30196/head:
qa/tasks/ceph: restart: stop osd, mark down, then start

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #30268 from cbodley/wip-41719

cls/rgw: fix bilog trim tests in ceph_test_cls_rgw

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge PR #30220 into master

* refs/pull/30220/head:
test: Allow fractional milliseconds to make test possible

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

cls/rgw: fix bilog trim tests in ceph_test_cls_rgw

the bucket index prepare step no longer writes a bilog entry, so the
tests were expecting more entries

Fixes: https://tracker.ceph.com/issues/41719
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29127 from bk201/wip-40337

mgr/dashboard: orchestrator integration initial works

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

test/cls_rbd/test_cls_rbd: update TestClsRbd.sparsify

after #30061, extents returned by sparse read will always be
an empty map as long as the extents to be read is empty or the extent(s)
in it are empty, even of the objectstorage does not support sparse read.

Fixes: https://tracker.ceph.com/issues/41721
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30115 from alfredodeza/wip-rm41378-2

ceph-volume: pre-install python-apt and its variants before test runs

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #30142 from votdev/issue_41648

mgr/dashboard: Handle always-on Ceph Manager modules correctly

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30079 from uweigand/issue-41605-fixes

Fix big-endian handling

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #30087 from egggHang/ec_benchmark

test/erasure-code: add exception handling to k & m

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29892 from Yan-waller/wip-yanj-fixreed_sol_r6_op

osd: fix the missing default value m=2 of reed_sol_r6_op in profile

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30125 from majianpeng/msg-protocolv1-reduece-swap

msg/async/ProtocolV1: avoid unnecessary bufferlist::swap.

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

PendingReleaseNotes: note about changes in `ceph osd erasure-code-profile set`

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

osd: add sanity check for invalid m/w of liber8tion

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

ec/jerasure: save the default m=2 of reed_sol_r6_op in profile

save the default value of m to profile so that it could be dumped
in command 'ceph osd erasure-code-profile get xxx', which is more
useful and friendly to user.

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

Merge pull request #30109 from rzarzynski/wip-bug-25098-bluefs_layout_t

os/bluestore: verify disk layout of BlueFS

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #30197 from bk201/wip-41675

mgr/orchestrator: fix refs property of progresses

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: Handle always-on Ceph Manager modules correctly

Currently always-on modules are not marked as enabled in the WebUI and can be disabled. This PR will fix that.
Note, this PR will NOT implement code that will prevent a developer from trying to disable an always-on module through the REST API. The Mgr Python extension will throw an adequate exception.

This PR will also do:
* Remove old code fragments from a previous Mgr Module management UI that is obsolete now.
* Cleanup code in BaseMgrModule code.

Fixes: https://tracker.ceph.com/issues/41648
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #30199 from IlsooByun/cmake_ptyhon_var

build/ops: fix build fail related to PYTHON_EXECUTABLE variable

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #29986 into master

* refs/pull/29986/head:
mgr/pg_autoscaler: use 'stored' for pool_logical_used

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #30139 into master

* refs/pull/30139/head:
mgr/DaemonServer: skip adjusting pgp_num when merging is in-progress

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #30061 from xiexingguo/wip-sparse-read

os, osd: readv

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #30059 from xiexingguo/wip-clean-vs-repair

osd/PG: fix _finish_recovery vs repair race

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #30190 from tchaikov/wip-python-rpm-macro

install-deps.sh: install `python*-devel` for python*rpm-macros

Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

osd/MissingLoc.cc: do not rely on missing_loc_sources only

In 624ade487ea4aeaf988cc1767e0b293f76addd5b, we relied on missing_loc_sources
to check for strays and remove an OSD from missing_loc. However, it is
possible that missing_loc_sources is empty while there are still OSDs
present in missing_loc. Since the aim is to just remove a stray OSD from
missing_loc, we do not need to rely on missing_loc_sources. We still
clean missing_loc_sources if any stray is present in it.

Signed-off-by: Neha Ojha <nojha@redhat.com>

test: Allow fractional milliseconds to make test possible

Fixes: https://tracker.ceph.com/issues/41689
Signed-off-by: David Zafman <dzafman@redhat.com>

install-deps.sh: install `python*-devel` for python*rpm-macros

in 087ea813, we installed '*rpm-macros' for the macros, so we can have
access to the latest python packaging related macros for preparing the
build dependencies.

but we could run into https://bugs.centos.org/view.php?id=16379, if
we already have an old version of python-devel installed. as the newer
version of python-rpm-macros conflicts with it.

it was a chicken-and-egg problem, as we don't know the exact name of
*rpm-macros packages. that's why we chose to install all of them. but
we have to upgrade the existing python-devel package to resolve the
conflict. but the since there is no python3-devel in RHEL7/CentOS7,
what they have is python36-devel. so we have to hardwire the
`%{python3_pkgversion}` to "36" even before we have access to this
macro, and upgrade the python36-devel package beforehand. but this
renders installing the rpm-macro package less useful -- we intend to
use the macro offered by the package to figure out "36".

as a workaround, we pretend that we know the "main" version of python3
in current RHEL/CentOS. and always install python36-devel for
python-rpm-macros. as the former requires the latter.

once all python3*-devel on all builders are upgraded, we will be safe
to install '*rpm-macros' again without installing python36-devel first.
by then, we could revert this change, or continue installing
python36-devel until the distro bumps up the "main" python version to 3.7

Fixes: https://tracker.ceph.com/issues/41603
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22574 from aclamk/bs-allocator-ageing-test

bluestore/allocator: Ageing test for bluestore allocators.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #30028 from tchaikov/wip-rpath

cmake: selectively rewrite install rpath

Reviewed-by: Sridhar Seshasayee <sseshasa@redhat.com>

Merge pull request #30015 from neha-ojha/wip-41429

mon/Monitor.cc: fix condition that checks for unrecognized auth mode

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Joao Eduardo Luis <joao@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30023 from tchaikov/wip-get-req

mon,osd: use get_req<> instead of static_cast<>(get_req())

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #30210 from ctrlaltdel/doc-typo

doc: Fix a typo in balancer documentation

Reviewed-by: Lenz Grimmer <LGrimmer@suse.com>

Merge pull request #29520 from pcuzner/add-events-mgr-module

mgr/k8sevents: Add mgr module for kubernetes event integration

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Travis Nielsen <tnielsen@redhat.com>

doc: Fix a typo in balancer documentation

Signed-off-by: Francois Deppierraz <francois@ctrlaltdel.ch>

Merge PR #29786 into master

* refs/pull/29786/head:
allow for multiple "build" dirs to exist

Reviewed-by: Sage Weil <sage@redhat.com>

build/ops: fix build fail related to PYTHON_EXECUTABLE variable

Fixes: https://tracker.ceph.com/issues/41676
Signed-off-by: Ilsoo Byun <ilsoobyun@linecorp.com>

ceph.spec.in: reserve 2500MB per build job

On our 14GB build machines, 8 parallel jobs results in the compiler
going OOM. Reserving 2500MB per job allows it to succeed with 5 jobs.

Fixes: https://tracker.ceph.com/issues/41659
Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>

allow for multiple "build" dirs to exist

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>

Merge pull request #29985 from xiexingguo/wip-41514-plus

osd/PrimaryLogPG: cancel in-flight manifest ops on interval changing; fix race with scrub

Reviewed-by: Myoungwon Oh <ommw@sk.com>

msg/async/ProtocolV1: avoid unnecessary bufferlist::swap.

Currently, for MOSDOpRely it don't prepare in func send_message.
So no need do swap for empty bufferlist.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

mgr/orchestrator: fix refs property of progresses

refs for events should be a list of tuples.

Fixes: https://tracker.ceph.com/issues/41675
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

Merge pull request #30184 from tchaikov/wip-make-check

run-make-check.sh: extract run-make.sh

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #30085 from xiexingguo/wip-41601

osd/PrimaryLogPG: update oi.size on write op implicitly truncating object up

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mgr/MgrClient: send MMgrCommand for octopus+ mgrs

This allows us to (eventually) leave MCommand for tell-style commands
only.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MgrMap: include mgr_features in mgrmap

Signed-off-by: Sage Weil <sage@redhat.com>

mgr: accept MMgrCommand[Reply] messages (in addition to MCommand[Reply])

MCommand is used both for tell-style commands and for CLI commands
implemented by the mgr.  Allow the latter to be send via MMgrCommand
instead.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #29842 into master

* refs/pull/29842/head:
kv/KeyValueDB: fix estimate_prefix_size()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #29855 into master

* refs/pull/29855/head:
pybind/mgr: remove type conversions for ceph native options
mgr: return get_ceph_option result as typed Py object (not string)

Reviewed-by: Tim Serong <tserong@suse.com>

qa/tasks/ceph: restart: stop osd, mark down, then start

If we stop, start, and then mark down, we may (likely) end up marking
the *new* instance down, which is noisy (generates a cluster warning
message) and inefficient.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #30013 from ivancich/wip-beanjoy-update-attrib-tags

rgw: fix refcount tags to match and update object's idtag

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #30062 from david-z/wip-fix-test-assert

test: remove useless ASSERT_XXX macros for rgw test

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29673 from rafaelquint/imagetrash

mgr/dashboard: RBD Image Purge Trash, Move to Trash and Restore

Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #28755 from dzafman/wip-network

feature: Health warnings on long network ping times, add "dump_osd_network" to get a report

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

script/backport-resolve-issue: better error message

When a point release is published, sometimes the new target version is not yet
in Redmine. Issue a better error message in this case.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

run-make-check.sh: extract run-make.sh

so we can reuse run-make.sh for building the artifact used by other
tests than "make check", for instance, dashboard's E2E test and
crimson's performance test.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29402 from Devp00l/wip-40828

mgr/dashboard: Use onCancel on any modal event

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #30064 from bk201/wip-41368

mgr/dashboard: fix cdEncode decorator is not working on class

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #30145 from theanalyst/doc/releases/14.2.3

doc: releases: 14.2.3 dashboard note

https://github.com/ceph/ceph/pull/24179

mgr/dashboard: Empty datatable rendered before data has been f… (#30103)

mgr/dashboard: Empty datatable rendered before data has been fetched

Reviewed-by: Patrick Seidensal <pnawracay@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

os/bluestore: verify disk layout of BlueFS.

Fixes: http://tracker.ceph.com/issues/25098
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

os/bluestore: bluefs_layout_t tracks WAL presence.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mgr/dashboard: Set RO as the default access_type for RGW NFS exports

It will also show an warning if the user selects RW access_type.

Fixes: https://tracker.ceph.com/issues/40186
Signed-off-by: Tiago Melo <tmelo@suse.com>

immutable_object_cache: Endian fix for HeaderHelper

struct HeaderHelper is used to overlay buffer list data, which
was generated via encode, and therefore holds integer data in
fixed little-endian format.

Thus, HeaderHelper needs to use ceph_le32 instead of uint32_t
when accessing the len field.

Fixes (partially): https://tracker.ceph.com/issues/41605

Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>

osd: Endian fix for PGTempMap handling

struct PGTempMap in osd/OSDMap.h tracks a number of int32_t pointers
pointing into a buffer list. But that list was generated via encode,
which means int members are bytes-swapped. Fixed by using ceph_le32
pointers instead.

Fixes (partially): https://tracker.ceph.com/issues/41605

Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>

librbd: Endian fix for handling old image format resize requests

ResizeRequest<I>::send_update_header contains a comment
    // NOTE: format 1 image headers are not stored in fixed endian
format

This statement appears incorrect; Linux kernel code handling the old
format headers does assume they are in fixed little-endian format.

Fixed by performing a byte-swap as everywhere else; this actually
fixes test suite failures on big-endian systems.

Fixes (partially): https://tracker.ceph.com/issues/41605

Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>

rgw,test: Add missing init_le calls

Add calls to init_le16/32/64 to properly byte-swap values on
big-endian systems when assigning to ceph_le16/32/64 variables.

Fixes (partially): https://tracker.ceph.com/issues/41605

Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>

librados,test: Fix incorrect use of __le16/32/64

Use ceph_le16/32/64 instead of __le16/32/64 (which are no-op outside
of kernel code).

Fixes (partially): https://tracker.ceph.com/issues/41605

Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>

transaction: Fix incorrect use of __le16/32/64

Use ceph_le16/32/64 instead of __le16/32/64 (which are no-op outside
of kernel code).

Note that this updates only those uses of __le16/32/64 which are
part of data structures that are serialized to disk/network
(i.e. Transaction::Op and Transaction::TransactionData).

Also note that code in this file performs combined operations on
little-endian values (in particular ++, +=, and |=) which are not
supported on the ceph_le16/32/64 class, and are therefore replaced
by more primitive operations.

Fixes (partially): https://tracker.ceph.com/issues/41605

Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>

bluestore: Fix incorrect use of __le16/32/64

Use ceph_le16/32/64 instead of __le16/32/64 (which are no-op outside
of kernel code).

Fixes (partially): https://tracker.ceph.com/issues/41605

Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>

async,crimson: Add missing init_le calls

Add calls to init_le16/32/64 to properly byte-swap values on
big-endian systems when assigning to ceph_le16/32/64 variables.

Fixes (partially): https://tracker.ceph.com/issues/41605

Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>