Merge pull request #45531 from ifed01/wip-ifed-alloc-replay-with-bin

os/bluestore: proper locking for Allocators' dump

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>

Merge pull request #45684 from CongMinYin/pwl-add-stats

librbd/cache/pwl: add pwl metrics in "rbd status" display

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #45357 from cbodley/wip-54531

rgw: disable RGWDataChangesLog::add_entry() when log_data is off

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

os/bluestore: proper locking for Allocators' dump methods

Plus renaming parametrized dump to foreach()
Fixes: https://tracker.ceph.com/issues/54973
Signed-off-by: Igor Fedotov <igor.fedotov@croit.io>

test/allocator_replay_test: introduce check for duplicates

This performs check for duplicates using free dump.

Signed-off-by: Igor Fedotov <igor.fedotov@croit.io>

test/allocator_replay_test: introduce binary format for free list dump

Adding new command to export free dump to binary format plus capability
to use new format for replay.
This dramatically increases large dump loading.

Signed-off-by: Igor Fedotov <igor.fedotov@croit.io>

Merge pull request #45677 from ideepika/wip-ninja-default

ceph.spec: make ninja-build package install always

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Tim Serong <tserong@suse.com>

Merge pull request #45820 from liu-chunmei/crimson-do_osd_ops_params_t

crimson: keep do_osd_ops_params_t alive when call do_osd_ops

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #45137 from myoungwon/wip-51627-2

osd: return appropriate error if the object is not manifest

Reviewed-by: Samuel Just <sjust@redhat.com>

crimson: keep do_osd_ops_params_t alive when call do_osd_ops

otherwise stack-under-overflow

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

Merge pull request #45744 from idryomov/wip-stretch-last-force-resend

mon/OSDMonitor: properly set last_force_op_resend in stretch mode

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #45670 from pdvian/wip-fix-mgr-daemon-state

mgr, mon: Keep upto date metadata with mgr for MONs

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #45599 from amathuria/amathuri-54994-fix

osd: add scrub duration for scrubs after recovery

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #45547 from nkshirsagar/master

Catch exception if thrown by __generate_command_map()

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #45505 from pdvian/wip-fix-daemon-version

mgr, mgr/prometheus: Fix regression with prometheus metrics

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #45822 from liu-chunmei/crimson-enametoolong

crimson: check -ENAMETOOLONG for Name, Locator, NameSpace

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #45584 from votdev/issue_54983_default_rgw_daemon

mgr/dashboard: RGW users and buckets tables are empty if the selected gateway is down

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #45790 from rhcs-dashboard/host-toggle-column-fix

mgr/dashboard: datatable in Cluster Host page hides wrong column on selection

Reviewed-by: Sarthak0702 <NOT@FOUND>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #45007 from VallariAg/dashboard-complexity-cleanup

mgr/dashboard: reduce method (cyclomatic) complexity

Reviewed-by: VallariAg <NOT@FOUND>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #45070 from rkachach/fix_issue_52042

mgr/cephadm: Making default cephadm shell cmd easier

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #45056 from ktdreyer/explain-cephadm-tox

cephadm: add comment explaining docker.io grep test

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #45815 from cbodley/wip-55232

test/rgw: use mock OpsLogSink instead of OpsLogSocket

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #45770 from yuvalif/wip-yuval-fix-54416

test/multisite: dont use path when mrun outside of src tree

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #45347 from mgfritch/cephadm-config-noreplace

cephadm: preserve `authorized_keys` file during upgrade

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #45589 from melissa-kun-li/bootstrap_registry_warning

cephadm: show error during bootstrap if private registry cred not provided

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #45685 from rkachach/fix_issue_47905

mgr/cephadm: improving logging to send errors to stderr

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #45798 from adk3798/iscsi-only-pid-limit

cephadm: only apply unlimited pids-limit to iscsi and rgw

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #45689 from vshankar/wip-55110

mount.ceph: remove `ms_mode' mount option when switching to old-syntax

Reviewed-by: Venky Shankar <vshankar@redhat.com>

qa: test_iscsi_pids_limit.sh: increase sleep time

It could take longer than 30 seconds to fork off 40000 processes on
a busy system.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

rbd: include persistent cache metrics in "rbd status" report

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

mount.ceph: remove `ms_mode' mount option when switching to old-syntax

... and switch to using v1 addresses (if users haven't specified those
explicitly). kernel versions <5.11 do not understand `ms_mode' mount
option which would result in mount failure.

Fixes: http://tracker.ceph.com/issues/55110
Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #45791 from rhcs-dashboard/rm-true

build: install-deps failing in docker build

Reviewed-by: David Galloway <dgallowa@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

rbd: factor out get_percentage() helper

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd/cache/pwl: no need to set clean and empty in remove_pool_file()

It is redundant -- the only caller sets both since commit 6593e31fff18
("librbd/cache/pwl: correct cache state").

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd/cache/pwl: avoid inconsistencies in ImageCacheState

When empty and/or clean bools are updated in I/O handling code paths,
ImageCacheState becomes inconistent for a short while: e.g. with clean
transitioned to true, dirty_bytes counter could still be positive
because the counters are updated only in periodic_stats().  Move to
updating the counters in update_image_cache_state(Context*) to avoid
this.

update_image_cache_state(Context*) now requires m_lock -- most call
sites already hold it anyway.  The only problematic call site was
AbstractWriteLog::shut_down() callback chain: perf_stop() needed to
be moved to the very end since perf counters must be alive now for
update_image_cache_state() to work.

Don't override expect_op_work_queue() in unit tests: completing
context in the same thread now results in a deadlock on m_lock in
all test cases that call AbstractWriteLog::init().

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd/cache/pwl: handle invalid ImageCacheState json

get_json_format() and create_image_cache_state() attempt to get
particular keys which could result in an unhandled std::runtime_error
exception.  Conversely, ImageCacheState constructor just swallows that
exception which could leave the newly constructed object incorrectly
initialized.  Avoid doing parsing in the constructor and introduce
init_from_config() and init_from_metadata() methods instead.

While at it, move everything out from under "persistent_cache" key.
Also fix init_state_json_write test case which stopped working now
that types are enforced by json_spirit.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd/cache/pwl: add basic metrics to ImageCacheState

Add basic metrics to ImageCacheState and persist them, including
allocated_bytes, cached_bytes, dirty_bytes, free_bytes and hit/miss
info.

Leverage periodic_stats() timer to call update_image_cache_state.
In order to avoid outputting too much debug information, the original
statistics output log level is changed to 5.

Switch to json_spirit for encoding because encode_json encodes bool as
"true"/"false" string.

Remove rbd_persistent_cache_log_periodic_stats option because we need
to always update cache state.

[ idryomov: add cached_bytes and hits_partial; report misses and
  miss_bytes instead of respective totals; naming ]

Fixes: https://tracker.ceph.com/issues/50614
Signed-off-by: Yin Congmin <congmin.yin@intel.com>
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #45832 from anthonyeleven/anthonyeleven/min-alloc-size-53752

doc/rados/configuration: document min_alloc_size values and space amplification

Reviewed-by: Zac Dover <zac.dover@gmail.com>

doc/rados/configuration: document min_alloc_size values and space amplification

Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>

crimson: check -ENAMETOOLONG for Name, Locator, NameSpace

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

Merge pull request #45749 from josephsawaya/fix-rook-tests

Remove orchestrator from rook task and suite

cmake: move rgw_lua_request.cc from rgw_common target to rgw_a

resolves a clang linker error where `rgw::lua::request::RequestLog()`
from rgw_lua_request.cc (in rgw_common) looks for `rgw_log_op()` from
rgw_log.cc (in rgw_a)

rgw_a depends on rgw_common, not the other way around. so this moves
rgw_lua_request.cc into the same target as rgw_log.cc

lua is now a public dependency of rgw_common so it's not hidden from
rgw_a or unit tests

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #45734 from kalebskeithley/vtable-dtor

rgw: add missing virtual decls for dtors

Reviewed-by: Casey Bodley <cbodley@redhat.com>

cephadm: show error message if private registry credentials not provided

Raise UnauthorizedRegistryError in `_pull_image` if user tries to pull from a private registry without authentication, handle error in `command_bootstrap`, `commond_adopt`, `command_pull`

Fixes: https://tracker.ceph.com/issues/55015
Signed-off-by: Melissa Li <melissali@redhat.com>

Merge pull request #45758 from linuxbox2/wip-rgwlc-dmfix

rgwlc:  don't incorrectly expire delete markers when !next_key_name

mgr/cephadm: Making default cephadm shell cmd easier
Fixes: https://tracker.ceph.com/issues/52042
Signed-off-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #45597 from adk3798/node-exporter-paths

cephadm: Specify proc/sys path for node-exporter to use

Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #45304 from dvanders/cleanup_46456

rgw_reshard: drop olh entries with empty name

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

cephadm: only apply pids-limit to iscsi and rgw

Signed-off-by: Adam King <adking@redhat.com>

mgr/cephadm: improving logging to send errors to stderr
Fixes: https://tracker.ceph.com/issues/47905
Signed-off-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #45639 from dparmar18/dhairya_mds_doc

doc/cephfs/add-remove-mds: added cephadm note, refined "Adding an MDS"

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #44666 from s0nea/correct_metric_name

doc/mgr/prometheus: correct metric name

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>
Reviewed-by: Paul Cuzner <pcuzner@redhat.com>

mgr/dashboard: datatable in Cluster Host page hides wrong column on selection

Signed-off-by: Sarthak0702 <sarthak.dev.0702@gmail.com>

Merge pull request #45787 from kalebskeithley/zipper-api

rgw: zipper: move functions so they're on correct side of API

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

build: install-deps failing in docker build

install-deps.sh was failing in our docker build due to the recent change in
the script. Failure can be seen here: https://github.com/rhcs-dashboard/ceph-dev/runs/5844502455?check_suite_focus=true#step:3:2586

This seems to fix the issue.

Signed-off-by: Nizamudeen A <nia@redhat.com>

test/rgw: use mock OpsLogSink instead of OpsLogSocket

Fixes: https://tracker.ceph.com/issues/55232
Signed-off-by: Casey Bodley <cbodley@redhat.com>

workunits/rgw: reshard cleans empty name entries

Check that reshard prunes olh entries with empty name.

Fixes: https://tracker.ceph.com/issues/54500
Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>

Merge pull request #45751 from ceph/rhel85

qa: Latest RHEL8 -> RHEL8.5

Merge pull request #45760 from pritha-srivastava/wip-rgw-gc-54742

rgw: adding BUCKET_REWRITE and OBJECT_REWRITE OPS to

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #44676 from linuxbox2/wip-lc-name-cleanup

rgwlc: improve naming of lifecycle entries

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #45781 from sseshasa/wip-update-mclock-rel-notes

PendingReleaseNotes: Update mclock release note regarding an existing issue.

Reviewed-by: Neha Ojha <nojha@redhat.com>

test/multisite: dont use path when mrun outside of src tree

when running in teuthology we should take the binaries
from the machine without specifying a path

Fixes: https://tracker.ceph.com/issues/54416
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #45782 from zdover23/wip-doc-2022-04-05-os-recommendations

doc/start: add testing support information

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #45187 from rhcs-dashboard/update-monitoring-stack-versions

mgr/cephadm: update monitoring stack versions

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: p-se <NOT@FOUND>

doc/start: add testing support information

This PR adds information about support for testing,
and information about which distros the Ceph project
builds packages for.

This is one in a series of PRs including the following:

https://github.com/ceph/ceph/pull/45385
https://github.com/ceph/ceph/pull/45764

This PR specifically includes the information that Ernesto
Puerta collected here:
https://github.com/ceph/ceph/pull/45385#pullrequestreview-911766656

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #45793 from rhcs-dashboard/update-piechart-vonage-versions

mgr/dashboard: upgrade grafana pie-chart and vonage-status-panel versions

Reviewed-by: MrFreezeex <NOT@FOUND>
Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

mgr/dashboard: upgrade grafana pie-chart and vonage-status-panel versions

Fixes:https://tracker.ceph.com/issues/55195
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

PendingReleaseNotes: Update mclock release note regarding an existing issue

Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>

rgw: zipper: move functions so they're on correct side of API

When we eventually split out compilation units for the Rados SAL
plugin, a few functions will be out of place wrt the SAL API and
will result in undefined references either in the library or the
the Rados plugin.

Preemptively moving these now, vs. later

Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>

Merge pull request #45774 from liu-chunmei/crimson-cmp-xattr

crimson: cmp xattr canceled operation

Reviewed-by: Samuel Just <sjust@redhat.com>

mgr/cephadm: update monitoring stack versions

Fixes: https://tracker.ceph.com/issues/54311
Signed-off-by: Aashish Sharma <aasharma@redhat.com>
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #45740 from cbodley/wip-doc-cmake-bundled-deps

README: document cmake options for system packages

Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #45743 from m-ildefons/docs-python3-librados

docs: update install steps for python librados

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

crimson: store->get_attr return nodata, cancel following operations

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

crimson: if do_cmp_xattr failed, cancel the following operations

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

Merge pull request #45764 from zdover23/wip-doc-2022-04-04-os-recommendations

doc/start: remove old OS tables

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #45720 from xxhdx1985126/wip-55143

crimson/os/seastore: use different segment seq allocator from journal and ool segments

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #45735 from chrigl/fix-54620

mgr/cephadm: support non-root ssh-user w permissions

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #45710 from rkachach/fix_issue_55132

mgr/cephadm: fixing public network conf parsing

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #45733 from benhanokh/simple_bitmap_fix

os/BlueStore: NCB fix for SimpleBitmap boundary check

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>

Merge pull request #45349 from guits/check-check-device-is-available

ceph-volume: various fixes in arg_validators

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #45687 from yuvalif/wip-yuval-multisite-test-rgws

multisite/test: running the multisite test script with multiple RGWs per zone

Merge pull request #45696 from cbodley/wip-cmake-system-utf8proc

cmake: WITH_SYSTEM_UTF8PROC defaults to OFF

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #45763 from rhcs-dashboard/fix-55172-master

monitoring/grafana: fix version

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #45514 from tchaikov/wip-cxx20-fixes-cephfs

cephfs: C++20 related fixes

Reviewed-by: Venky Shankar <vshankar@redhat.com>

doc/start: remove old OS tables

This PR removes tables that contained outdated
information about Ceph's supported operating
systems. Ernesto Puerta had the idea to consolidate
the information that was in those tables into a single
table, and we're doing that.

Part of that process involves removing the old tables.
That's what this PR does.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

mgr/cephadm: fixing public network conf parsing
Fixes: https://tracker.ceph.com/issues/55132
Signed-off-by: Redouane Kachach <rkachach@redhat.com>

monitoring/grafana: fix version

Fixes: https://tracker.ceph.com/issues/55172
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #45254 from travisn/prometheus-rules-typos

prometheus: Spell check the alert descriptions

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: sunilangadi2 <NOT@FOUND>
Reviewed-by: Travis Nielsen <tnielsen@redhat.com>

Merge pull request #45746 from idryomov/wip-crush-rule-sizes-cleanup

doc,test: clean up crush rule min/max_size leftovers

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

doc/cephfs/add-remove-mds: added cephadm note, refined "Adding an MDS"

Description: 1) Add a note about using cephadm for setting up the
                cluster and mds(s), also mention the use of ceph
                orchestrator if one needs to setup mds(s) manually.
     2) Changed the term `data point` to `directory` in
                point 1 under "Adding an MDS" section for better
                clarity.

Fixes: https://tracker.ceph.com/issues/54551
Signed-off-by: dparmar18 <dparmar@redhat.com>

build: make ninja-build package install always

we use ninja build as default build now, having it installed only with
make check enabled may make builds fail, if ran without make check.

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

rgw: adding BUCKET_REWRITE and OBJECT_REWRITE OPS to
gc_ops_list to initialize gc and prevent crashes when
send_chain() is invoked.

Fixes: https://tracker.ceph.com/issues/54742
Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

Merge pull request #45385 from zdover23/wip-doc-2022-03-14-os-recommendations

doc/start: update os-recommendations.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #45705 from SUSE/wip-really-fix-55079

ceph.spec.in: remove build directory in %clean, not %install

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

rgwlc:  don't incorrectly expire delete markers when !next_key_name

This is a fencepost case when operating on the last element of a
listing, apparently masked before Mark Kogan's segfault fix).

Fixes: https://tracker.ceph.com/issues/55168
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #44612 from rzarzynski/wip-bs-lazy4freebsd

bdev: fix FTBFS on FreeBSD, keep the huge paged read buffers.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #45719 from liu-chunmei/crimson-setopflag

Crimson handle CEPH_OSD_OP_FLAG_FAILOK

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/os/seastore: use different segment seq allocator from journal and ool segments

Fixes: https://tracker.ceph.com/issues/55143
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: rename segment_header_t::journal_segment_seq to segment_seq

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>