test/allocator_replay_test: introduce binary format for free list dump

Adding new command to export free dump to binary format plus capability
to use new format for replay.
This dramatically increases large dump loading.

Signed-off-by: Igor Fedotov <igor.fedotov@croit.io>

Merge pull request #45584 from votdev/issue_54983_default_rgw_daemon

mgr/dashboard: RGW users and buckets tables are empty if the selected gateway is down

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #45790 from rhcs-dashboard/host-toggle-column-fix

mgr/dashboard: datatable in Cluster Host page hides wrong column on selection

Reviewed-by: Sarthak0702 <NOT@FOUND>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #45007 from VallariAg/dashboard-complexity-cleanup

mgr/dashboard: reduce method (cyclomatic) complexity

Reviewed-by: VallariAg <NOT@FOUND>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #45070 from rkachach/fix_issue_52042

mgr/cephadm: Making default cephadm shell cmd easier

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #45056 from ktdreyer/explain-cephadm-tox

cephadm: add comment explaining docker.io grep test

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #45815 from cbodley/wip-55232

test/rgw: use mock OpsLogSink instead of OpsLogSocket

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #45770 from yuvalif/wip-yuval-fix-54416

test/multisite: dont use path when mrun outside of src tree

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #45347 from mgfritch/cephadm-config-noreplace

cephadm: preserve `authorized_keys` file during upgrade

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #45589 from melissa-kun-li/bootstrap_registry_warning

cephadm: show error during bootstrap if private registry cred not provided

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #45685 from rkachach/fix_issue_47905

mgr/cephadm: improving logging to send errors to stderr

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #45798 from adk3798/iscsi-only-pid-limit

cephadm: only apply unlimited pids-limit to iscsi and rgw

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #45689 from vshankar/wip-55110

mount.ceph: remove `ms_mode' mount option when switching to old-syntax

Reviewed-by: Venky Shankar <vshankar@redhat.com>

qa: test_iscsi_pids_limit.sh: increase sleep time

It could take longer than 30 seconds to fork off 40000 processes on
a busy system.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

mount.ceph: remove `ms_mode' mount option when switching to old-syntax

... and switch to using v1 addresses (if users haven't specified those
explicitly). kernel versions <5.11 do not understand `ms_mode' mount
option which would result in mount failure.

Fixes: http://tracker.ceph.com/issues/55110
Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #45791 from rhcs-dashboard/rm-true

build: install-deps failing in docker build

Reviewed-by: David Galloway <dgallowa@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #45832 from anthonyeleven/anthonyeleven/min-alloc-size-53752

doc/rados/configuration: document min_alloc_size values and space amplification

Reviewed-by: Zac Dover <zac.dover@gmail.com>

doc/rados/configuration: document min_alloc_size values and space amplification

Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #45749 from josephsawaya/fix-rook-tests

Remove orchestrator from rook task and suite

cmake: move rgw_lua_request.cc from rgw_common target to rgw_a

resolves a clang linker error where `rgw::lua::request::RequestLog()`
from rgw_lua_request.cc (in rgw_common) looks for `rgw_log_op()` from
rgw_log.cc (in rgw_a)

rgw_a depends on rgw_common, not the other way around. so this moves
rgw_lua_request.cc into the same target as rgw_log.cc

lua is now a public dependency of rgw_common so it's not hidden from
rgw_a or unit tests

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #45734 from kalebskeithley/vtable-dtor

rgw: add missing virtual decls for dtors

Reviewed-by: Casey Bodley <cbodley@redhat.com>

cephadm: show error message if private registry credentials not provided

Raise UnauthorizedRegistryError in `_pull_image` if user tries to pull from a private registry without authentication, handle error in `command_bootstrap`, `commond_adopt`, `command_pull`

Fixes: https://tracker.ceph.com/issues/55015
Signed-off-by: Melissa Li <melissali@redhat.com>

Merge pull request #45758 from linuxbox2/wip-rgwlc-dmfix

rgwlc:  don't incorrectly expire delete markers when !next_key_name

mgr/cephadm: Making default cephadm shell cmd easier
Fixes: https://tracker.ceph.com/issues/52042
Signed-off-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #45597 from adk3798/node-exporter-paths

cephadm: Specify proc/sys path for node-exporter to use

Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #45304 from dvanders/cleanup_46456

rgw_reshard: drop olh entries with empty name

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

cephadm: only apply pids-limit to iscsi and rgw

Signed-off-by: Adam King <adking@redhat.com>

mgr/cephadm: improving logging to send errors to stderr
Fixes: https://tracker.ceph.com/issues/47905
Signed-off-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #45639 from dparmar18/dhairya_mds_doc

doc/cephfs/add-remove-mds: added cephadm note, refined "Adding an MDS"

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #44666 from s0nea/correct_metric_name

doc/mgr/prometheus: correct metric name

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>
Reviewed-by: Paul Cuzner <pcuzner@redhat.com>

mgr/dashboard: datatable in Cluster Host page hides wrong column on selection

Signed-off-by: Sarthak0702 <sarthak.dev.0702@gmail.com>

Merge pull request #45787 from kalebskeithley/zipper-api

rgw: zipper: move functions so they're on correct side of API

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

build: install-deps failing in docker build

install-deps.sh was failing in our docker build due to the recent change in
the script. Failure can be seen here: https://github.com/rhcs-dashboard/ceph-dev/runs/5844502455?check_suite_focus=true#step:3:2586

This seems to fix the issue.

Signed-off-by: Nizamudeen A <nia@redhat.com>

test/rgw: use mock OpsLogSink instead of OpsLogSocket

Fixes: https://tracker.ceph.com/issues/55232
Signed-off-by: Casey Bodley <cbodley@redhat.com>

workunits/rgw: reshard cleans empty name entries

Check that reshard prunes olh entries with empty name.

Fixes: https://tracker.ceph.com/issues/54500
Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>

Merge pull request #45751 from ceph/rhel85

qa: Latest RHEL8 -> RHEL8.5

Merge pull request #45760 from pritha-srivastava/wip-rgw-gc-54742

rgw: adding BUCKET_REWRITE and OBJECT_REWRITE OPS to

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #44676 from linuxbox2/wip-lc-name-cleanup

rgwlc: improve naming of lifecycle entries

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #45781 from sseshasa/wip-update-mclock-rel-notes

PendingReleaseNotes: Update mclock release note regarding an existing issue.

Reviewed-by: Neha Ojha <nojha@redhat.com>

test/multisite: dont use path when mrun outside of src tree

when running in teuthology we should take the binaries
from the machine without specifying a path

Fixes: https://tracker.ceph.com/issues/54416
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #45782 from zdover23/wip-doc-2022-04-05-os-recommendations

doc/start: add testing support information

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #45187 from rhcs-dashboard/update-monitoring-stack-versions

mgr/cephadm: update monitoring stack versions

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: p-se <NOT@FOUND>

doc/start: add testing support information

This PR adds information about support for testing,
and information about which distros the Ceph project
builds packages for.

This is one in a series of PRs including the following:

https://github.com/ceph/ceph/pull/45385
https://github.com/ceph/ceph/pull/45764

This PR specifically includes the information that Ernesto
Puerta collected here:
https://github.com/ceph/ceph/pull/45385#pullrequestreview-911766656

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #45793 from rhcs-dashboard/update-piechart-vonage-versions

mgr/dashboard: upgrade grafana pie-chart and vonage-status-panel versions

Reviewed-by: MrFreezeex <NOT@FOUND>
Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

mgr/dashboard: upgrade grafana pie-chart and vonage-status-panel versions

Fixes:https://tracker.ceph.com/issues/55195
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

PendingReleaseNotes: Update mclock release note regarding an existing issue

Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>

rgw: zipper: move functions so they're on correct side of API

When we eventually split out compilation units for the Rados SAL
plugin, a few functions will be out of place wrt the SAL API and
will result in undefined references either in the library or the
the Rados plugin.

Preemptively moving these now, vs. later

Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>

Merge pull request #45774 from liu-chunmei/crimson-cmp-xattr

crimson: cmp xattr canceled operation

Reviewed-by: Samuel Just <sjust@redhat.com>

mgr/cephadm: update monitoring stack versions

Fixes: https://tracker.ceph.com/issues/54311
Signed-off-by: Aashish Sharma <aasharma@redhat.com>
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #45740 from cbodley/wip-doc-cmake-bundled-deps

README: document cmake options for system packages

Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #45743 from m-ildefons/docs-python3-librados

docs: update install steps for python librados

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

crimson: store->get_attr return nodata, cancel following operations

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

crimson: if do_cmp_xattr failed, cancel the following operations

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

Merge pull request #45764 from zdover23/wip-doc-2022-04-04-os-recommendations

doc/start: remove old OS tables

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #45720 from xxhdx1985126/wip-55143

crimson/os/seastore: use different segment seq allocator from journal and ool segments

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #45735 from chrigl/fix-54620

mgr/cephadm: support non-root ssh-user w permissions

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #45710 from rkachach/fix_issue_55132

mgr/cephadm: fixing public network conf parsing

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #45733 from benhanokh/simple_bitmap_fix

os/BlueStore: NCB fix for SimpleBitmap boundary check

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>

Merge pull request #45349 from guits/check-check-device-is-available

ceph-volume: various fixes in arg_validators

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #45687 from yuvalif/wip-yuval-multisite-test-rgws

multisite/test: running the multisite test script with multiple RGWs per zone

Merge pull request #45696 from cbodley/wip-cmake-system-utf8proc

cmake: WITH_SYSTEM_UTF8PROC defaults to OFF

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #45763 from rhcs-dashboard/fix-55172-master

monitoring/grafana: fix version

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #45514 from tchaikov/wip-cxx20-fixes-cephfs

cephfs: C++20 related fixes

Reviewed-by: Venky Shankar <vshankar@redhat.com>

doc/start: remove old OS tables

This PR removes tables that contained outdated
information about Ceph's supported operating
systems. Ernesto Puerta had the idea to consolidate
the information that was in those tables into a single
table, and we're doing that.

Part of that process involves removing the old tables.
That's what this PR does.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

mgr/cephadm: fixing public network conf parsing
Fixes: https://tracker.ceph.com/issues/55132
Signed-off-by: Redouane Kachach <rkachach@redhat.com>

monitoring/grafana: fix version

Fixes: https://tracker.ceph.com/issues/55172
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #45254 from travisn/prometheus-rules-typos

prometheus: Spell check the alert descriptions

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: sunilangadi2 <NOT@FOUND>
Reviewed-by: Travis Nielsen <tnielsen@redhat.com>

Merge pull request #45746 from idryomov/wip-crush-rule-sizes-cleanup

doc,test: clean up crush rule min/max_size leftovers

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

doc/cephfs/add-remove-mds: added cephadm note, refined "Adding an MDS"

Description: 1) Add a note about using cephadm for setting up the
                cluster and mds(s), also mention the use of ceph
                orchestrator if one needs to setup mds(s) manually.
     2) Changed the term `data point` to `directory` in
                point 1 under "Adding an MDS" section for better
                clarity.

Fixes: https://tracker.ceph.com/issues/54551
Signed-off-by: dparmar18 <dparmar@redhat.com>

rgw: adding BUCKET_REWRITE and OBJECT_REWRITE OPS to
gc_ops_list to initialize gc and prevent crashes when
send_chain() is invoked.

Fixes: https://tracker.ceph.com/issues/54742
Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

Merge pull request #45385 from zdover23/wip-doc-2022-03-14-os-recommendations

doc/start: update os-recommendations.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #45705 from SUSE/wip-really-fix-55079

ceph.spec.in: remove build directory in %clean, not %install

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

rgwlc:  don't incorrectly expire delete markers when !next_key_name

This is a fencepost case when operating on the last element of a
listing, apparently masked before Mark Kogan's segfault fix).

Fixes: https://tracker.ceph.com/issues/55168
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #44612 from rzarzynski/wip-bs-lazy4freebsd

bdev: fix FTBFS on FreeBSD, keep the huge paged read buffers.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #45719 from liu-chunmei/crimson-setopflag

Crimson handle CEPH_OSD_OP_FLAG_FAILOK

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/os/seastore: use different segment seq allocator from journal and ool segments

Fixes: https://tracker.ceph.com/issues/55143
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: rename segment_header_t::journal_segment_seq to segment_seq

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore/cache: clean up in Cache::get_next_dirty_extents()

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #45658 from ceph/wip-debug-install

build: Add some debugging messages

qa: Latest RHEL8 -> RHEL8.5

Signed-off-by: David Galloway <dgallowa@redhat.com>

build: Add some debugging messages

Having a unique string like "CI_DEBUG" will help me know where we are in the build process in Jenkins logs.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #45739 from rhcs-dashboard/fix-55155-master

grafana/Makefile: don't push to docker

doc/start: update os-recommendations.rst

This PR updates the extremely out-of-date
os-recommendations.rst file.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Remove orchestrator from rook task and suite

This commit removes orchestrator commands from the
Rook task and the Rook test suite because the Rook
orchestrator is not being maintained, and the Rook
orchestrator CLI is obsolete. This should also
clarify the issue:

https://tracker.ceph.com/issues/53680

Signed-off-by: Joseph Sawaya <jsawaya@redhat.com>

src/crush: remove sample.txt

It is hopelessly outdated -- the weights were moved from crushmap to
osdmap in 2008, etc.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

doc,test: clean up crush rule min/max_size leftovers

These fields were removed in quincy in commit f95eb04411cd ("crush:
eliminate min_size and max_size").

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

grafana/Makefile: don't push to docker

Fixes: https://tracker.ceph.com/issues/55155
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #45583 from p-se/monitoring-alert-mtu-group-by-devices

mgr/dashboard: Compare values of MTU alert by device

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: p-se <NOT@FOUND>

docs: update install steps for python librados

- Put the correct package names for python librados in the install
  instructions.
- librados-devel does not actually pull in python3-librados as
  dependency anymore, remove mention of this.
- Add instructions for openSUSE/SLE

Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

Merge pull request #45742 from tchaikov/wip-python-debian

debian: install from site-packages to dist-packages

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

debian: install from site-packages to dist-packages

this change addresses the regression introduced by
637dd7b40404e644519b1fc3b5ef03f2d18def00

dist-packages is a debian specific directory for holding 3rd party
python modules. so install the pure python packages into it.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

crimson: set result = 0 when op.flags include CEPH_OSD_OP_FLAG_FAILOK

Signed-off-by: Chunmei Liu <chunmei.liu@intel.com>

crimson: handle osd_op.op.flags include CEPH_OSD_OP_FLAG_FAILOK

If the osd_op failed, still continue to next op when it include
CEPH_OSD_OP_FLAG_FAILOK.

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

Merge pull request #45707 from rzarzynski/wip-crimson-errorator-2nd-arg-handle

crimson/common: let errorator's handlers to conditionally forward errors

Reviewed-by: Sam Just <sjust@redhat.com>
                       liu-chunmei <chunmei.liu@intel.com>

README: document cmake options for system packages

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #45578 from rhcs-dashboard/fix-grafana-build

mgr/dashboard: remove transition-through-oci image workaround in grafana  build

Reviewed-by: Dan Mick <dmick@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

os/BlueStore: NCB fix for SimpleBitmap boundary check
The boundary check in SimpleBitmap is off by one causing an assert to trigger
Also fixed a bug when asking for the next clear_extent on a unaligned map when the last bits in the map were set.
Adding unit-tests
Fixes: https://tracker.ceph.com/issues/55145
Signed-off-by: Gabriel BenHanokh <gbenhano@redhat.com>

mgr/cephadm: support non-root ssh-user w permissions

Restructured code, so that in case of non-root, the resulting file will
be created with permissions set to the ssh-user. This allows the
subsequent scp to be able to write the file. The remaining code kept the
same, so that file permissions are restored to the expected ones, but
just runs after the scp.

Fixes: https://tracker.ceph.com/issues/54620
Signed-off-by: Christoph Glaubitz <c.glaubitz@syseleven.de>

bdev: fix FTBFS on FreeBSD, keep the huge paged read buffers.

Special thanks to Willem Jan Withagen!

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #45660 from CongMinYin/fix-pwl-status

librbd/cache/pwl: correct cache state

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>