Merge pull request #46068 from neesingh-rh/change_perf_stats_o/p_structure

mgr/stats: change in structure of perf_stats o/p

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #47778 from xxhdx1985126/wip-fix-output-error-cleaner

crimson/os/seastore/async_cleaner: fix several log related errors

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #47751 from zdover23/wip-doc-2022-08-23-mgr-orchestrator-editorial-pass

doc/mgr: edit orchestrator.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #47079 from neesingh-rh/fix_56483

mgr/stats: missing clients in perf stats command output.

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #47729 from tchaikov/wip-crimson-os-cleanup

crimson/os: use IsFullKey in *::insert_prefix{,_at}()

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

doc/mgr: edit orchestrator.rst

This PR improves the English language in the "Orchestrator CLI"
section of the MGR documentation. It adds a couple of section
headers in order to signpost the information in the document
a bit more than had already been done, but it makes no major
structural changes to the presentation of the information here.

This PR was motivated by feedback from the 2022 Ceph User Survey
in which one of the respondents wrote "better ceph orch documen-
tation".

The final section on this page, "Current Implementation Status",
must be verified by someone who is familiar with the current state
of "ceph orch" and a date stamp should be applied to the top of
the section so that the word "current" has a meaningful referent.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

crimson/os/seastore/async_cleaner: fix several log related errors

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #47646 from cyx1231st/wip-seastore-move-cleaner-to-epm

crimson/os/seastore: move AsyncCleaner to EPM

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>
Reviewed-by: Zhang Song <zhangsong325@gmail.com>

Merge pull request #47561 from dang/wip-dang-zipper-cast

RGW - Zipper - Remove a bunch of casts

Reviewed-by: Soumya Koduri <skoduri@redhat.com>

Merge pull request #45949 from neesingh-rh/feature-51434

mgr/volumes: Add volume info command

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Nikhilkumar Shelke <nshelke@redhat.com>

Merge pull request #44501 from SMIL-Infra/fix-mds-evicted

mon/MDSMonitor: fix standby-replay mds being removed from MDSMap unexpectedly

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #47678 from dang/wip-dang-admin-user

RGW - Admin - Pass user to created bucket

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #47642 from ljflores/wip-librados-fix

qa/workunits/rados: pull librados files from "main" instead of "master"

Merge pull request #47624 from Mer1997/wip-57138

mgr/snap-schedule: use the right way to check the result returned by…

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #47301 from rzarzynski/wip-bug-55851

msg: reset ProtocolV2's frame assembler in appropriate thread

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #46943 from kalaspuffar/python39

mgr/diskprediction_local: Support Python 3.9 for disk prediction module.

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #47672 from Matan-B/wip-matanb-buffer-next

include/buffer: clear_and_dispose() fix implicitly-declared operator= usage

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #47710 from tchaikov/denc-concept

include/denc: define is_const_iterator as a concept

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #47718 from tchaikov/wip-no-isp2

*: replace functions in intarith.h with the ones provided by standard library

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #46419 from Matan-B/wip-matanb-c-snaps

crimson/osd: Snaps IO path

Reviewed-by: Samuel Just <sjust@redhat.com>

test/crimson/seastore: fix teardown with cleanups

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: move AsyncCleaner to EPM

There are two purposes related to generic device tiering:

* Make AsyncCleaner (need to introduce a generic interface class later)
  to hide differences between segment-based tier and RB-based tier
  cleaner implementations.
* Make EPM to coordinate cleaning transactions across tiers with general
  AsyncCleaners.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson/seastore: drop unnecessary run_until_halt()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #47182 from lxbsz/wip-56638

libcephfs: define AT_NO_ATTR_SYNC back for backward compatibility

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>
Reviewed-by: Milind Changire <mchangir@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Nikhilkumar Shelke <nshelke@redhat.com>

crimson/os/seastore/async_cleaner: cleanup and sort interfaces

To prepare for a general interface class for device tiering.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/async_cleaner: cleanup, replace init_complete by explicit states

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #47737 from ceph/release-docs2

admin: Fix check if PR or release branch docs build

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #47542 from batrick/i56140

tools/cephfs: add basic detection/cleanup tool for dentry first damage

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #43771 from rhcs-dashboard/wip-52701-master

mgr/dashboard: add backend-driven UI tables

Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
Reviewed-by: sherrif10 <NOT@FOUND>
Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #46599 from kotreshhr/rename_config_on_flush

mgr/volumes: Better handle config file on osd full scenario

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #47063 from lxbsz/wip-56529

mds/client: fail the request if the peer MDS doesn't support getvxattr op

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Milind Changire <mchangir@redhat.com>

Merge pull request #47132 from lxbsz/_rmsnap_finish_clean

mds: remove useless code in _rmsnap_finish()

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

Merge pull request #47384 from lxbsz/wip-56633

mds: add more debug logs before crashing the MDS

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Nikhilkumar Shelke <nshelke@redhat.com>

Merge pull request #47613 from cbodley/wip-56920

rgw/coroutine: check for null stack on wakeup

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge PR #47732 into main

* refs/pull/47732/head:
Revert "Merge pull request #47092 from dparmar18/wip-dparmar-cephadm-simple-1"

Reviewed-by: Adam King <adking@redhat.com>

rgw/coroutine: check for null stack on wakeup

Fixes: https://tracker.ceph.com/issues/56920
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #47690 from mty22/update-inactive-ceph-mirrors

doc: Remove inactive Ceph mirrors.

admin: Fix check if PR or release branch docs build

Uses built-in RTD vars.  https://docs.readthedocs.io/en/stable/environment-variables.html.

Follow up to https://github.com/ceph/ceph/pull/46917#discussion_r942359130.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #45552 from wxypro/man-cephfs-mirror

doc/man/cephfs-mirror.rst: Update missing options for cephfs-mirror

Merge pull request #47727 from snosratiershad/docs/correct-radosgw-admin-doc-bucket-name

doc/man/8: Remove slash behind bucket name in radosgw-admin.rst

Revert "Merge pull request #47092 from dparmar18/wip-dparmar-cephadm-simple-1"

This reverts commit 1c4da3dbd20d9683fe681cd7083478809cbb19b9, reversing
changes made to ee1e163b1e69c4db558cdf6b857c7c82cd4820d8.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc/man/8: Remove slash behind bucket name in radosgw-admin.rst

Signed-off-by: Salar Nosrati-Ershad <snosratiershad@gmail.com>

Merge pull request #47371 from rhcs-dashboard/mstart-configurable

mstart: make some mstart env variables configurable

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

crimson/os: use key_hobj_t instead of full_key_t<>

less indirection this way. for better readability

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #47207 from oOHenry/rgw-dashboard-ssl-error

mgr/dashboard: fix rgw connect when using ssl

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: David Galloway <dgallowa@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #47720 from tchaikov/wip-mgr-dashboard-test-with-headers

qa/tasks/mgr/dashboard: accept headers as a param

Reviewed-by: Nizamudeen A <nia@redhat.com>

crimson/os: use IsFullKey in *::insert_prefix{,_at}()

to fade out KeyT, so we can have more straightforward definitions.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #47716 from MalteJ/patch-1

doc: virtio discard is supported since kernel 5.0

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

include/intarith: drop popcount()

as it is not used anymore.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

common, test: use std::popcount() when appropriate

prefer the facilities provided by standard library over the homebrew
ones for better readability and maintainability.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

include/intarith: drop clz()

it is not used anymore

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

os/bluestore: use std::countl_zero() when appropriate

prefer the facilities provided by standard library over the homebrew
ones for better readability and maintainability.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

include/intarith: rewrite cbits() in C++20

rewrite cbits() using concept and std::countl_zero(), for better
readability and maintanability.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

include/intarith: drop isp2()

as it is not used anymore.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

*: s/isp2/has_single_bit/

prefer the facilities provided by standard library over the homebrew
ones for better readability and maintainability.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

doc: virtio discard is supported since kernel 5.0

Signed-off-by: Malte Janduda <mail@janduda.net>

Merge pull request #46702 from rhcs-dashboard/expand-cluster-hide-maintenance

mgr/dashboard: Hide maintenance option on expand cluster

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
Reviewed-by: sunilangadi2 <NOT@FOUND>

libcephfs: define AT_NO_ATTR_SYNC back for backward compatibility

This was introduce by commit e2a67f2a65553ad45721bb391081bc61aa97e0e9,
for the third part applications they may still use the old macro.

Add it back and marked it as deprecated.

Fixes: https://tracker.ceph.com/issues/56638
Signed-off-by: Xiubo Li <xiubli@redhat.com>

qa: add test_non_existing_mds_rank and modify test_query_mds_filter

Fixes: https://tracker.ceph.com/issues/56169
Signed-off-by: Neeraj Pratap Singh <neesingh@redhat.com>

mgr/stats: Fixing DeprecationWarning for RegEx.

Declaring the RegEx pattern as a raw string by prepending `r`.

Signed-off-by: Neeraj Pratap Singh <neesingh@redhat.com>

qa: Update the qa tests to be compatible with the new structure of 'perf stats' o/p.

test_client_metrics_and_metadataand other tests has been
updated as earlier it was checking according to the old structure
of perf stats o/p, which has been changed in this PR.

Fixes: https://tracker.ceph.com/issues/56162
Signed-off-by: Neeraj Pratap Singh <neesingh@redhat.com>

qa/tasks/mgr/dashboard: accept headers as an option param

* add optional "headers" parameter to _request() and _get()
* correct the test of test_force_no_gzip, as the header in response
  should be "application/vnd.ceph.api.v1.0+json", not "application/json"

with this change, we are able to pass headers to _get() in dashboard api
tests.

without this change, we'd have following error when testing
`test_force_no_gzip` defined by `tasks.mgr.dashboard.test_requests.RequestsTest`:

raceback (most recent call last):
  File "/home/jenkins-build/build/workspace/ceph-api/src/pybind/mgr/dashboard/services/exception.py", line 47, in dashboard_exception_handler
    return handler(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/cherrypy/_cpdispatch.py", line 54, in __call__
    return self.callable(*self.args, **self.kwargs)
  File "/home/jenkins-build/build/workspace/ceph-api/src/pybind/mgr/dashboard/controllers/_base_controller.py", line 263, in inner
    ret = func(*args, **kwargs)
TypeError: Summary.__call__() got an unexpected keyword argument 'headers'

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

mgr/dashboard: add out_client dir to api and e2e checks

Signed-off-by: Nizamudeen A <nia@redhat.com>

vstart: isolate the client logs

Signed-off-by: Nizamudeen A <nia@redhat.com>

mstart: make some mstart env variables configurable

Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #47711 from tchaikov/wip-crimson-os-cleanups

crimson/os: cleanups

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #47715 from maryfrances01/wip-doc-punctuation-error

doc/radosgw: Fix punctuation and grammar in encryption.rst

doc: Fixes a punctuation and grammar issue

include/buffer: avoid it = it->next() usage

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #47639 from ronen-fr/wip-rf-scrubmap-swap

osd: fix ScrubMap::swap() to include all ScrubMap data members
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

include/denc: use concept to define denc_traits

to be more specific, define denc_traits for types with ext types

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

include/denc: rewrite ExtType with is_any_of<>

just for better readability

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

include/denc: rewrite denc_traits for enums

for better readability

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

include/denc: rewrite is_any_of using concept

just for better readability

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

include/denc: redefine is_const_iterator

to just check for the return type of `it.get_pos_add(n)`, this
is exactly what we expect from a buffer::ptr::const_iterator.
it is more specific and less confusing.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

include/denc: define is_const_iterator as a concept

just for better readability

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

doc: Fixes a punctuation and grammar issue

Signed-off-by: Mary Frances Hull mhull@redhat.com

Merge pull request #44356 from SMIL-Infra/clog

common,mon: use fmt::print(cerr, ...) for stderr logging

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

crimson/os: use IsFullKey in *::estimate_insert()

to fade out KeyT, so we can have more straightforward definitions.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

crimson/os: use use IsFullKey in node_fields_2_t::insert_at()

to fade out KeyT, so we can have more straightforward definitions.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #47325 from CongMinYin/fix-owner-lock-ordering

librbd/cache/pwl: narrow the scope of m_lock in write_image_cache_state()

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

crimson/os: use IsFullKey in _node_fields_013_t::append_key()

to fade out KeyT, so we can have more straightforward definitions.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

crimson/os: avoid using KeyT in is_valid_key()

and we don't need to specify KeyT when instantiating this template
function.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

crimson/os: avoid using KeyT in ns_oid_view_t

so we don't need to use it when instantiating the template member
functions of this class.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

include/denc: initialize local variable

to silence warning like:

../src/include/denc.h:834:27: warning: variable 'len' is uninitialized when used here [-Wuninitialized]
    v.push_back(p.get_ptr(len));
                          ^~~

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #47628 from mkogan1/wip-fix-tcmalloc-link

rgw: fix linking with tcmalloc when specified by -DALLOCATOR

Reviewed-by: Casey Bodley <cbodley@redhat.com>

osd: modify two ScrubMap data members from fields to full variables

as there is no size advantage in this specific case.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd: fix ScrubMap::swap() to include all ScrubMap data members

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #47632 from linuxbox2/wip-rgw-53455

rgw/s3: add conditional json output for s3:ListBucket

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #46443 from ybwang0211/head_obj

rgw: The error reported in the "rgw_s3_prepare_decrypt" function is unreasonable

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #46547 from Huber-ming/check_appendable

rgw: check object appendable when check_disk_state

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #47228 from Huber-ming/multi_meta

rgw: 'bucket check' deletes index of multipart meta when its pending_map is nonempty

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #47547 from mdw-at-linuxbox/wip-master-xamzdate-1

rgw: x-amz-date change breaks certain cases of aws sig v4.

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #45036 from rishabh-d-dave/qa-fix-is_mounted

qa/cephfs: upgrade is_mounted() in mount.py

Reviewed-by: Kotresh HR <khiremat@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Nikhilkumar Shelke <nshelke@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #45869 from neesingh-rh/fix_55234

mgr/snap_schedule: replace .snap with the client configured snap dir name

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Nikhilkumar Shelke <nshelke@redhat.com>

Merge pull request #46749 from ibm-s390-cloud/fsmap_switch_to_iterator

fsmap: switch to using iterator based loop

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

msg: reset ProtocolV2's frame assembler in appropriate thread

`set_is_rev1()` actually resets the internal state of `FrameAssembler`,
so -- to avoid racing -- it should be performed by the same thread that
takes care about executing `ProtocolV2::write_message()`.

```cppp
void set_is_rev1(bool is_rev1) {
  m_descs.clear();
  m_flags = 0;
  m_is_rev1 = is_rev1;
}
```

See comments in the tracker for analysis.

Fixes: https://tracker.ceph.com/issues/55851
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/os/seastore/transaction_manager: change read_extent_iertr to use get_pin_iertr

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #47691 from myoungwon/wip-cbjournal-integration

crimson/os/seastore: CBJournal integration works

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #47597 from NitzanMordhai/wip-nitzan-crimson-watch-list-empty-return-code

crimson/osd: watch list empty return incorrect

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Sam Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>