doc/rbd: refine rbd-snapshot.rst

Refine the text in rbd-snapshot.rst

https://tracker.ceph.com/issues/57001

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #49173 from rhcs-dashboard/fix-encryption-config

mgr/dashboard: fix server side encryption config error

Merge pull request #49452 from tchaikov/wip-crimson-test-fmt-v9

test/crimson: fixes related to fmtlib v9

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

mgr/dashboard: fix server side encryption config error

Fixes: https://tracker.ceph.com/issues/58296
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

test/crimson: include fmt/os.h

so we can format std::error_code

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

test/crimson: print conn instead of its address

now that crimson::net::Connection can be formatted using fmt::format(),
instead of printing the pointer, let's print its value.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

test/crimson: trade operator<<() for fmt::format<>()

so we are able to format Payload using fmtlib v9. before fmtlib v9,
fmtlib is able to fall back to the operator<<() to format the the
types with the operator<<() defined. but after fmtlib v9, we need to
explicitly define them for accessing the specialized formatter.

now that we have specialized fmt::format<> for Payload, and the only
consumer of operator<<() is the fmtlib, we can safely drop the latter.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

test/crimson: specialize fmt::formater<> for types to be formatted

since fmt v9, fmt::formatter<> is not specialized for the types with
operator<<(ostream&, ...) anymore. so we need to specialize it manually.
in this change, fmt::formatter<> is specialized for some types to be
formatted with fmtlib, so the tree can compile with fmt v9.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

msg/msg_types.h: include fmt/format.h

for accessing the macro definition of FMT_VERSION, so we can have
definition of fmt::format<> if required.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

test/crimson: print pointer using fmt::ptr()

in fmtlib v9, it does not format typed pointers anymore, instead,
we need to print them using fmt::ptr().

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

test/crimson: include system and 3rd-party headers first

see also https://google.github.io/styleguide/cppguide.html#Names_and_Order_of_Includes

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #49467 from zdover23/wip-doc-2022-12-16-rados-ec-default-simplest

doc/rados: clarify default EC pool from simplest

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: clarify default EC pool from simplest

Distinguish between the default erasure-coded pool (k=2 m=2) and the
simplest erasure-coded pool (k=2 m=1).

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48991 from kamoltat/wip-ksirivad-fix-bz-2121452

mon/Elector: Change how we handle removed_ranks and notify_rank_removed()
Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

Merge pull request #49436 from Thingee/add-alex-to-board

doc/foundation: Adding Alex Marangone to the board

Merge pull request #49416 from ljflores/wip-telemetry-label

.github: automatically add telemetry label to relevant files

Merge pull request #48641 from NitzanMordhai/wip-nitzan-valgrind-supp-adding-rocksdb-dl-init

src/valgrind.supp: Adding know leaks unrelated to ceph

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #48416 from Rethan/wip-bluestore-discard-thread

blk/KernelDevice: don't start discard thread if device not support_di…

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #40066 from ideepika/wip-mempool-buffers

os/bluestore: improve cache_onode and cache_buffer mempool accounting

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #49435 from dang/wip-dang-motr-build

RGW - Fix DAOS and MOTR stores to not use rgw_obj_key::to_str()

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #49021 into main

* refs/pull/49021/head:
qa: switch to https protocol for repos' server

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

Merge pull request #48707 from guits/guits-tracker-57918

ceph-volume: fix a bug in get_all_devices_vgs()

Merge pull request #49445 from zdover23/wip-doc-2022-12-15-cephfs-posix-eg-fix

doc/cephfs: fix "e.g." in posix.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/cephfs: fix "e.g." in posix.rst

Remove "e.g." and replace it with "for example".

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #49441 from zdover23/wip-doc-2022-12-15-cephfs-homophone-error

doc/cephfs - s/yet to here/yet to hear/ posix.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #49442 from zdover23/wip-doc-2022-12-15-cephfs-homophone-error-2

doc/cephfs: s/all of there are/all of these are/

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #48089 from aisakaki/wip-omap-rmkeyrange

crimson/os/seastore: introduce omap range remove

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #49427 from tchaikov/wip-crimson-fmt

crimson: fmtlib v9 related changes

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #49408 from tchaikov/wip-crimson-device_spec_t

crimson/os: define device_spec_t as packed

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

doc/cephfs: s/all of there are/all of these are/

s/all of there are/all of these are/ in posix.rst.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

doc/cephfs - s/yet to here/yet to hear/ posix.rst

Correct "yet to here" to "yet to hear" in posix.rst.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48710 from cbodley/wip-rgw-process-env

rgw: pass RGWProcessEnv into process_request()

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #49395 from adamemerson/wip-policy-useful-error-messages

rgw: Give useful errors when policies fail to parse

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

mon/Monitor.cc: notify_new_monmap() skips removal of non-exist rank

Problem:
In RHCS the user can choose to manually remove a monitor rank
before shutting the monitor down. Causing inconsistency in monmap.
for example we remove mon.a from the monmap, there is a short period
where mon.a is still operational and will try to remove itself from
monmap but we will run into an assertion in
ConnectionTracker::notify_ranks_removed().

Solution:
In Monitor::notify_new_monmap() we prevent the func
from going into removing our own rank, or
ranks that doesn't exists in monmap.

FYI: this is an RHCS problem only, in ODF,
we never remove a monitor from monmap
before shutting it down.

Fixes: https://tracker.ceph.com/issues/58049
Signed-off-by: Kamoltat <ksirivad@redhat.com>

Merge pull request #49141 from cbodley/wip-58111

rgw: rgw_parse_url_bucket() rejects empty bucket names after 'tenant:'

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@cern.ch>

Merge pull request #48226 from sseshasa/wip-fix-recovery-backfill-limits-for-mclock

osd: Reduce backfill/recovery default limits for mClock and other optimizations

Reviewed-by: Vikhyat Umrao <vikhyat@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

doc/foundation: Adding Alex Marangone to the board

Signed-off-by: Mike Perez <thingee@gmail.com>

RGW - Fix DAOS and MOTR stores to not use rgw_obj_key::to_str()

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

ceph-volume: fix a bug in get_all_devices_vgs()

When a physical volume isn't a member of any volume group,
the function `get_all_devices_vgs()` in `ceph_volume.api.lvm`
shouldn't try to return an `VolumeGroup()` object against this
device.

Fixes: https://tracker.ceph.com/issues/57918
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

.github: automatically add telemetry label to relevant files

Signed-off-by: Laura Flores <ljflores@redhat.com>

Merge pull request #49178 from JoshSalomon/chg-primary

osd: Add cli upmap command to change the primary OSD of a pg

Merge pull request #48742 from rhcs-dashboard/add-tooltip

mgr/dashboard: add tooltip mirroring pools table

Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #49418 from tchaikov/wip-cls-fifo-fmt

cls/fifo: specialize fmt::formater<> for rados::cls::fifo::info

Reviewed-by: Casey Bodley <cbodley@redhat.com>

osd, mon: Add 'osd pg-upmap-primary' and 'osd rm-pg-upmap-primary' commands.

Changed OSDMap to keep upmap-primary records
The upmap-primary records are visible in json/xml and in dump commands.

Signed-off-by: Josh Salomon <jsalomon@redhat.com>

crimson/osd: include fmt/ranges.h

so we are able to format ss.clone_snaps, which is a
std::map<snapid_t, std::vector<snapid_t>>. before fmtlib v9,
fmtlib is able to fall back to the operator<<() to format the the types
with the operator<<() defined. but after fmtlib v9, we need to
explicitly define them for accessing the specialized formatter.

in this change, fmt/ranges.h is included so we can access the formatter
for std::map and std::vector.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

crimson/os: specialize fmt::formater<> for lba_pin_list_t

since fmt v9, fmt::formatter<> is not specialized for the types with
operator<<(ostream&, ...) anymore. so we need to specialize it manually.
in this change, fmt::formatter<lba_pin_list_t> is defined so
the tree can compile with fmt v9.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

crimson/osd: print oid of missing clone

instead of printing the empty optional<>, print the oid of missing
clone. the oid is more interesting in this context.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #49419 from tchaikov/wip-seastar

seastar: bump up seastar submodule to recent master

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os: specialize fmt::formater<> for types

since fmt v9, fmt::formatter<> is not specialized for the types with
operator<<(ostream&, ...) anymore. so we need to specialize it manually.
in this change, fmt::formatter<> is specialized for tree_stats_t and
test_item_t so the tree can compile with fmt v9.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

crimson/os: print pointer using fmt::ptr()

in fmtlib v9, it does not format typed pointers anymore, instead,
we need to print them using fmt::ptr().

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

seastar: bump up seastar submodule to recent master

to address the fmtlib v9 compatibility issues.

please note, the submodule is bump up to a commit which includes

* a revert commit on top of upstream master HEAD. the revert is to
  work around the a recent change which adds a constraint on the parameter
  type of handle_exception(). we are using errorator along with
  handle_exception(), but errorator cannot fulfill the requirement of
  seastar::Future(), so we need to drop the change before we have a
  better solution.
* a commit which defines __NR_pidfd_open if it is not defined.
  this should address the FTBFS on CentOS8 where glibc does not have
  this macro defined.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #49421 from zdover23/wip-doc-2022-12-14-glossary-fqdn

doc/glossary: add "FQDN" entry

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/glossary: add "FQDN" entry

Add "FQDN" definition to doc/glossary.rst.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

crimson/os/seastore: Seastore::omap_list cleanup

Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>

test/cls_fio: specialize fmt::formater<> for bpo::options_description

since fmt v9, fmt::formatter<> is not specialized for the types with
operator<<(ostream&, ...) anymore. so we need to specialize it manually.
in this change, fmt::formatter<bpo::options_description> is defined so
the tree can compile with fmt v9.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

cls/fifo: specialize fmt::formater<> for rados::cls::fifo::info

since fmt v9, fmt::formatter<> is not specialized for the types with
operator<<(ostream&, ...) anymore. so we need to specialize it manually.
in this change, fmt::formatter<rados::cls::fifo::info> is defined so
the tree can compile with fmt v9.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #49242 from myoungwon/wip-enable-rbm-test-seastore

test/crimson/seastore: enable RBM-based unittest-seastore

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #49407 from tchaikov/cmake-find-boost

cmake: add 1.79 and 1.80 to known versions

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: added rgw-policy-check

rgw-policy-check - a program to do syntax checking on bucket policy.
This program just reads the policy into memory, so it is not
checking anything except syntax.

Signed-off-by: Marcus Watts <mwatts@redhat.com>
rgw: Fix return value of `rgw-policy-check`

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
rgw: Use ceph initialization in `rgw-policy-check`

Specifically so we can pull in the options from `ceph.conf` and similar.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Put useful policy error messages in error document

This affects the various create/put operations that take a policy document.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Add `rgw_policy_reject_invalid_principals` and messages

Reject policies with invalid principals by default and provide more
useful error messages while doing so.

(Log them but do *not* reject the policy if it's set to false.)

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Provide useful error messages from policy parser

It would be much nicer to give people an idea why their policies are
failing rather than just telling them where they're failing.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #48214 from ljflores/wip-telemetry-bluestore-compression-mode

mgr/telemetry: add `basic_pool_options_bluestore` collection

Reviewed-by: Yaarit Hatuka <yaarit@redhat.com>

rgw/lua: pass luarocks_path directly, remove from sal

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/lua: split out a struct RGWLuaProcessEnv

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #49190 from tchaikov/wip-fmt

fmt: pickup v9.1.0

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #49410 from ronen-fr/wip-rf-cinode-tm

mds: remove unused declaration

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

mds: remove unused declaraion

as some Jenkins builds are set to treat warnings as errors, and
complain.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

crimson/os: define device_spec_t as packed

per C++ standard 20, alignas cannot be applied to scoped enum. despite
that there is a resolution to address this, see
https://cplusplus.github.io/CWG/issues/2354.html, it's not included in
C++20. so we have to use a different way to address

> runtime error: reference binding to misaligned address 0x610000008395
> for type 'device_type_t', which requires 4 byte alignment

otherwise the code would fail to compile with Clang-15, like:

> <source>:3:13: error: 'alignas' attribute cannot be applied to an enumeration

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

cmake: adapt FindBoost.cmake to our needs

the vanilla FindBoost.cmake pulled from cmake has couple assumptions
which do not hold in our environment. so address them case by case.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

cmake: add 1.79 and 1.80 to known versions

sync with
https://github.com/Kitware/CMake/blob/468a995346271c02f2248adf3539fbb34edaef2e/Modules/FindBoost.cmake

for v1.79 and v1.80 support

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

crimson/os/seastore/omap: adjust omap_list_config_t

Add inclusive behavior for both first and last in omap_list_config_t,
and optimize the usage.
Set first closed last open for range for omap range remove.

Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>

crimson/os/seastore: introduce omap range remove

Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>

crimson/os/seastore/omap: add tests for range remove

Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>

crimson/os/seastore/omap: introduce omap range remove

Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>

crimson/os/seastore/omap: add tests to range list

Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>

crimson/os/seastore/omap: support two borders list method

Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>

Merge pull request #49250 from Matan-B/wip-matanb-debug-subsys-doc

doc/rados/troubleshooting/log-and-debug: Update according to subsys.h

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #48449 from batrick/i57851

pybind/mgr: use memory temp_store

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Milind Changire <mchangir@redhat.com>

Merge pull request #49329 from rhcs-dashboard/fix-cors

mgr/dashboard: allow Origin url for CORS if present in config

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #49396 from zdover23/wip-doc-2022-12-13-glossary-secrets

doc/glossary - add "secrets"

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/glossary - add "secrets"

Add "secrets" to doc/glossary.rst.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

test/crimson/seastore: enable RBM-based unittest-seastore

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

rgw: Minor cleanups in `rgw_iam_policy.cc`

Unused using, confusing indentation, bracing.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #49391 from ceph/wip-yuri-crontab-main

qa/tests: added quincy-p2p tp the mix

Reviewed-by: Laura Flores <lflores@redhat.com>

qa/tests: added quincy-p2p tp the mix

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #49251 from shreyanshjain7174/cppool_space

rados: fix extra tabs on warning for pool copy

Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #47596 from YiteGu/dump-slow-ops-count

osd: add dump slow ops count

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #48711 from cfsnyder/wip-cfsnyder-del-perf-iops

rgw: add 'inline_data' zone placement info option

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #49278 from Matan-B/wip-matanb-EIO-flag-fix

test/librados/aio_cxx: Fix Pool EIO flag tests

Merge pull request #49180 from anthonyeleven/anthonyeleven/rework-pool-create-pg-error-message

src/mon: clarify message for pool creation failure due to max_pgs_per_osd

Merge PR #45192 into main

* refs/pull/45192/head:
mds: use the whole string as the snapshot long name

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Nikhilkumar Shelke <nshelke@redhat.com>

Merge PR #45312 into main

* refs/pull/45312/head:
qa: new snapshot test for snapshot naming limits
mds: limit the snapshot names to 240 characters

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>

Merge PR #48450 into main

* refs/pull/48450/head:
qa: add test_mdsdir_scrub_backtrace
mds: avoid ~mdsdir's scrubbing and reporting damage health status

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Milind Changire <mchangir@redhat.com>
Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

Merge PR #48642 into main

* refs/pull/48642/head:
qa: Fix test_subvolume_snapshot_info_if_orphan_clone

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

Merge PR #48786 into main

* refs/pull/48786/head:
mds/PurgeQueue: don't consider filer_max_purge_ops when _calculate_ops

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #48873 into main

* refs/pull/48873/head:
mount: fix mount failure with old kernels

Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #49117 into main

* refs/pull/49117/head:
mds: remove redundant frag check in MDBalancer::maybe_fragment()

Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #48258 from adk3798/upgrade-mgr-loop

mgr/cephadm: fix handling of mgr upgrades with 3 or more mgrs

Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #48550 from rkachach/fix_issue_57894

mgr/cephadm: moving Prometheus spec check to service_spec module

Reviewed-by: Adam King <adking@redhat.com>