crimson/osd/ops_executor: apply delta stats for pure read ops

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #48295 from NitzanMordhai/wip-nitzan-osd-start-osdmap-hang-adding-debug-msg

OSD: during test start, not all osds started due to consum map hang

Merge pull request #45247 from andriytk/update-ceph-mgw

rgw: update motr sal

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge PR #49842 into main

* refs/pull/49842/head:
Revert "cmake: do not link ceph-dencoder against global"

Reviewed-by: Tim Serong <tserong@suse.com>

Merge pull request #49946 from cityofships/pkg-docs

doc/cephadm: grammar / syntax in install.rst

Signed-off-by: Piotr Parczewski <piotr@stackhpc.com>

Revert "cmake: do not link ceph-dencoder against global"

Reverting since this is causing the following ceph-dencoder exception:

        [src/tcmalloc.cc:332] Attempt to free invalid pointer 0x558f14e3a000

NOTE: I didn't debug this any more further since there is sizable backlog
of PRs that need testing/merging, but reverting the change makes it work.

Fixes: http://tracker.ceph.com/issues/58219
Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #49913 from neesingh-rh/wip-51459

doc: add the damage types that scrub can repair

Reviewed-by: Zac Dover <zac.dover@proton.me>

Merge PR #49048 into main

* refs/pull/49048/head:
mds: fix and skip submitting invalid osd request

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Milind Changire <mchangir@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

doc: add the damage types that scrub can repair

Fixes: https://tracker.ceph.com/issues/51459
Signed-off-by: Neeraj Pratap Singh <neesingh@redhat.com>

Rollback submodules changes made by accident

Signed-off-by: Andriy Tkachuk <andriy.tkachuk@seagate.com>

Merge pull request #48541 from lxbsz/wip-qa-upgrade-fix

qa/cephfs: fix confusing config when upgrading mds nodes

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

Merge pull request #48049 from dparmar18/wip-dparmar-qa-mixed-clients-57280

qa/fs/mixed-clients: specify distros for tests

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>

Merge pull request #48251 from dparmar18/wip-dparmar-MDS-57359

mds/Server: do not allow -ve reclaim flags to cause client eviction

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Milind Changire <mchangir@redhat.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>

Merge pull request #47753 from mchangir/mon-disable-snap-id-allocation-for-fsmap-pools

mon: disable snap id allocation for fsmap pools

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #47114 from lxbsz/wip-56532

ceph_fuse: retry the test_dentry_handling if fails

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Milind Changire <mchangir@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

client: switch to old method to invalidate dcache if euid != 0

Force to use the old and slow method to invalidate the dcache
if the euid is non-root, or the remount may fail with return
code 1 or 32.

https://tracker.ceph.com/issues/56532
Signed-off-by: Xiubo Li <xiubli@redhat.com>

ceph_fuse: make it to force invalidating dentries when kernel >=3.18

The remount will fail randomly for unknown reasons. And in certain
circumstance we can reprodce this very easy, which will block our
testing. Make it posible to force to old method to invalidate the
dcache when the "client_try_dentry_invalidate" option is enabled
even kernel version >= 3.18.0

Fixes: https://tracker.ceph.com/issues/56532
Signed-off-by: Xiubo Li <xiubli@redhat.com>

ceph_fuse: retry the test_dentry_handling if fails

For some unknown reason the remount will fail, we need to retry it
for the test_dentry_handling.

Fixes: https://tracker.ceph.com/issues/56532
Signed-off-by: Xiubo Li <xiubli@redhat.com>

client: move a client's option to mds-client.yaml

mds_max_retries_on_remount_failure option is used by Client.cc only.

Fixes: https://tracker.ceph.com/issues/56532
Signed-off-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #49906 from evershalik/main

Update README.md

Merge pull request #49905 from zdover23/wip-doc-2023-01-29-rados-config-osd-config-ref-osd-op-thread-suicide-timeout

doc/rados: document osd_op_thread_suicide_timeout

Merge pull request #49908 from Matan-B/wip-matanb-upgrade-doc

doc/dev/developer_guide: Add Upgrade Testing Documentation

Reviewed-by: Zac Dover <zac.dover@proton.me>

doc/dev/developer_guide/testing_integration_tests: Add Upgrade Testing Documentation

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

doc/rados: document osd_op_thread_suicide_timeout

Add default value for osd_op_thread_suicide_timeout configuration value
to the OSD Configuraton Reference.

https://tracker.ceph.com/issues/58590

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Update README.md

Signed-off-by: SHANKAR <98207888+evershalik@users.noreply.github.com>

Merge branch 'master' into update-ceph-mgw

Merge pull request #49891 from bluikko/bluikko-doc-host-management-prompts

doc/cephadm: Fix prompts in host-management.rst

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #49897 from bluikko/bluikko-doc-rados-operations-prompt

doc/rados/operations: Fix double prompt

Reviewed-by: Zac Dover <zac.dover@proton.me>

doc/cephadm: Fix example commands

Separate command output block from the command to avoid printing prompt on it.
Also remove double prompt.

Fix indentation of two example commands after prompt.

Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>

doc/rados/operations: Fix double prompt

In monitoring.rst a double prompt was rendered, one non-selectable and one selectable. Remove the selectable prompt.

Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>

Merge pull request #49894 from bluikko/bluikko-doc-rados-operations-control-indentation

doc/rados/operations: Fix indentation

Reviewed-by: Zac Dover <zac.dover@gmail.com>

doc/rados/operations: Fix indentation

Fix invalid indentation that caused indentation to be rendered wrong in control.rst.

Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>

Merge pull request #49890 from bluikko/bluikko-doc-ceph-krb-auth

doc/dev: use underscores in config vars

Reviewed-by: Zac Dover <zac.dover@proton.me>

doc/dev: use underscores in config vars

Use underscores instead of spaces in config vars in ceph_krb_auth.rst.

Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>

Merge pull request #49109 from NitzanMordhai/wip-nitzan-fixing-few-rados/test.sh

Wip nitzan fixing few rados/test.sh

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #49887 from cbodley/wip-qa-rgw-run-s3tests-tox

qa/workunits/rgw: run-s3tests.sh uses tox

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #49799 from phlogistonjohn/jjm-cephadm-utils-tests

cephadm/tests: add test coverage for copy_tree, copy_files, move_files, recursive_chown

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #48597 from ronen-fr/wip-rf-osd-bitfields

osd: change two ObjectContext flags from bitfields to full variables

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

qa/workunits/rgw: run-s3tests.sh uses tox

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #49624 from ronen-fr/wip-rf-unique-formatter

common: extend Formatter API with a unique_ptr-returning static function

Merge pull request #49537 from rzarzynski/wip-bug-57546

Revert "osd/PeeringState: proc_lease_ack break once found from OSD"

Merge pull request #48854 from ifed01/wip-ifed-small-chunk-bluefs

os/bluestore: enable 4K allocation unit for BlueFS

Merge pull request #49314 from SUSE/wip-tserong-qa-crash-chown

qa/workunits/rados/test_crash: chown crash files to ceph user

Merge pull request #48799 from DanWritesCode/add-exception-handling

mon: add exception handling to ceph health mute

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #48836 from ljflores/wip-safe-timer-crash

osd: shut down the MgrClient before osd_fast_shutdown

Merge pull request #49741 from adamemerson/wip-datalog-nonnull-yield-mostly

rgw: Remove (most) uses of null_yield in datalog

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #49826 from cbodley/wip-qa-rgw-s3tests-tox

qa/s3tests: run with tox instead of bootstrap+nose

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #49879 from ljflores/wip-16.2.11-doc-fix

doc/releases: fix PR list

doc/releases: fix PR list

Signed-off-by: Laura Flores <lflores@redhat.com>

Merge pull request #46535 from cbodley/wip-use-epel-arrow

rpm: use system arrow/utf8proc packages by default

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #49769 from alimaredia/wip-rgw-optional-yield-create-bucket

rgw: Thread optional_yield through create bucket op

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #48731 from alimaredia/wip-rgw-admin-api-secondary-zone-user-create-fix

rgw: set keys from from master zone on admin api user create

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rpm: use system arrow packages for fedora/centos9

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rpm: combine system_utf8proc option into system_arrow

libarrow-devel depends on utf8proc-devel, so we can't enable
system_arrow without system_utf8proc

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rpm: fix package names for arrow/parquet

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #49839 from ceph/wip-yuriw-release-16.2.11-main

doc: 16.2.11 Release Notes

doc: 16.2.11 Release Notes

See https://tracker.ceph.com/issues/58257#note-2 for details

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #49432 from jmolmo/fix_hardcoded_ns

mgr/rook: Replace hardcoded namespace

Replace hardcoded namespace

Reviewed-by:
Travis Nielsen <tnielsen@redhat.com>
Nick Gentoli <gentoli@live.com>

Merge pull request #49819 from ivancich/wip-typing-policies-maxnum

rgw: minor clean-ups to user policy code

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #49873 from zdover23/wip-doc-2023-01-26-dev-developer-guide-slack

doc/dev: add Slack to Dev Guide essentials

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/dev: add Slack to Dev Guide essentials

Add Ceph's Slack to doc/developer_guide/essentials.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #49797 from cbodley/wip-58513

qa/openssl_keys: 'rm' ignores file not found

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #49313 from ifed01/wip-ifed-fix-cls-queue-list

cls/queue: use larger read chunks in queue_list_entries

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Mark Kogan <mkogan@redhat.com>

Merge pull request #49171 from guits/tracker_58137

ceph-volume: fix a bug in lsblk_all()

rgw: set keys from from master zone on admin api user modify

When users were being modified via the admin_api
to generate keys on a secondary zone, the keys
returned in the json output are not correct.

This commit ensures the proper keys are set to be
returned to the user via the json output.

fixes: https://tracker.ceph.com/issues/57724

Signed-off-by: Ali Maredia <amaredia@redhat.com>

rgw: set keys from from master zone on admin api user create

When users were being created via the admin_api
on a secondary zone the keys returned in the json
output are not correct.

This commit ensures the proper keys are set to be
returned to the user via the json output.

fixes: https://tracker.ceph.com/issues/57724

Signed-off-by: Ali Maredia <amaredia@redhat.com>

rgw: stop blocking for asio threads for create_bucket()

Asio thread for create_bucket operation are no
longer blocked because optional yield is passed
down in previous instances where asio threads for
op were blocked.

Signed-off-by: Ali Maredia <amaredia@redhat.com>

rgw: add conditional backtrace logging after "WARNING: blocking librados call"

Signed-off-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #49818 from adk3798/iscsi-gw-rm-async

mgr/cephadm: call iscsi post_remove from serve loop

Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #49802 from rkachach/fix_issue_58517

doc/cephadm: Adding doc for daemon_cache_timeout config option

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

Merge pull request #49392 from ifed01/wip-ifed-fix-spillover-test

test/store_test: use original rocksdb settings to enable bluefs

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #48972 from Matan-B/wip-matanb-mon-dump-slow-ops

mon/MonCommands: Support dump_historic_slow_ops

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #48876 from Matan-B/wip-matanb-address-log

osd: Properly log OpRequest, ObjectContext and SnapSetContext

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #48835 from ifed01/wip-ifed-noerr-alloc-restore

os/bluestore: do not spoil log with errors from __restore_allocator

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
Reviewed-by: Gabriel BenHanokh <benhanokh@gmail.com>

Merge pull request #47332 from NitzanMordhai/wip-nitzan-pglog-ec-getattr-error

Wip nitzan pglog ec getattr error

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #48428 from zmc/cephadm-smoke-small

qa/suites: Add orch/cephadm/smoke-small suite

Reviewed-by: Adam King <adking@redhat.com>

doc/cephadm: Adding doc for daemon_cache_timeout config option
Fixes: https://tracker.ceph.com/issues/58517
Signed-off-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #49256 from yuvalif/wip-yuval-remove-pubsub-struct

rgw/pubsub: remove subscription data structures and cleanup

reviewed-by: cbodley

Merge pull request #49796 from batrick/first-damage-big-dir

tools/cephfs/first-damage: fix iteration of big dirfrags

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #49814 from ivancich/wip-clean-up-trailing-spaces

rgw: clean up header files

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Ali Maredia <amaredia@redhat.com>

rgw: minor clean-ups to user policy and config options

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

rgw: clean up header files

Remove trailing spaces and use "#pragma once" rather than "#ifndef" to
assure a header files is only loaded once during compilation.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #49176 from cbodley/wip-58127

rgw: preserve RGWZoneGroup::master_zone on last zone deletion

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-By: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #49815 from adk3798/no-overwrite-logrotate

cephadm: don't overwrite cluster logrotate file

Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #48552 from rkachach/fix_issue_57878

mgr/rgw: Enabling typing checks for rgw module

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #48880 from rkachach/fix_issue_58021

mgr/cephadm: fix haproxy monitoring endpoint

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #49813 from rkachach/fix_issue_58526

mgr/cephadm: Build service discovery end-point by using mgr-ip only

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #49827 from adk3798/remove-offline-host-ingress

mgr/cephadm: fix removing offline hosts with ingress daemons

Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Redouane Kachach <rkachach@redhat.com>

rgw/notifications: add const to APIs when possible

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

rgw: Remove (most) uses of null_yield in datalog

The only ones remaining are in calls from top-level threads.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw/notifications: avoid unnecessary heap allocations

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

rgw/pubsub: remove subscription data structures

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #49760 from kalebskeithley/ceph-rgw-zipper

rgw: refactor selected files for better above- vs below-the-line

Merge pull request #48828 from yuvalif/wip-yuval-cython-fails-clang16

rgw/file: change function signature to match the one generated by cython

reviewed-by: cbodley, mattbenjamin

Merge pull request #49530 from yuvalif/wip-yuval-signed-comparisson

rgw: fix signed to unsigned comparisson warning

reviwed by: mattbenjamin

Merge pull request #49642 from cbodley/wip-58286

rgw/beast: fix interaction between keepalive and 100-continue

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

rgw: refactor selected files for better above- vs below-the-line

Move more files into driver/rados for better above- vs below-
the-line. Use #pragma once everywhere (versus fixing all the

Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>

Merge pull request #49820 from zdover23/wip-doc-2023-01-21-rados-config-pool-pg-config-ref

doc/rados: refine pool-pg-config-ref.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

Merge pull request #49778 from zdover23/wip-doc-2023-01-18-rados-config-ceph-conf-runtime-changes

doc/rados: refine ceph-conf.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>