Merge pull request #52300 from cbodley/wip-rgw-zonegroup-feature-compress-encrypted

rgw: require zonegroup feature to compress encrypted objects

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #52222 from rhcs-dashboard/cluster-upgrade-apis

mgr/dashboard: expose cluster upgrade  API endpoints

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #52306 from cyx1231st/wip-seastar-multi-core-osd-cleanup

crimson/osd: fixes and cleanups around multi-core OSD

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: chunmei-liu <chunmei.liu@intel.com>

Merge pull request #52355 from rhcs-dashboard/cleanup-script-remove-dash-cephadm-e2e

mgr/dashboard: move cephadm e2e cleanup to jenkins job config

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #52366 from zdover23/wip-doc-2023-07-09-rados-operations-add-or-rm-osds-1-of-x

doc/rados: edit add-or-rm-osds (1 of x)

Reviewed-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: edit add-or-rm-osds (1 of x)

Edit docs/rados/operations/add-or-rm-osds.

Co-authored-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>
Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #52371 from zdover23/wip-doc-2023-07-10-rados-troubleshooting-mon-emdash

doc/rados: introduce emdash

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #52338 from vedanshbhartia/coverity_1512094

rgw: Add check for whether RGWObjState was created in dbstore.cc

Reviewed-by: Soumya Koduri <skoduri@redhat.com>

Merge pull request #52372 from soumyakoduri/wip-skoduri-lc-acl

rgw/dbstore: Do not update mtime for attr change

reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #52369 from zhscn/wip-transaction-conflict

crimson/os/seastore: add metrics about total invalidated transactions

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

rgw/dbstore: Do not update mtime for attr change

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

Merge pull request #52091 from cyx1231st/wip-seastar-msgr-multi-core-perf

crimson/tools/perf_crimson_msgr: integrate multi-core msgr with various improvements

Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: add metrics about total invalidated transactions

Signed-off-by: Zhang Song <zhangsong325@gmail.com>

doc/rados: introduce emdash

Introduce the emdash character in
docs/rados/troubleshooting/troubleshooting-mon.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #52327 from batrick/i61865

doc: add information on expediting MDS recovery

Reviewed-by: Zac Dover <zac.dover@proton.me>

Merge pull request #51781 from rzarzynski/wip-bs-no-inc-in-middle

os/bluestore: get rid of inclusion-in-the-middle from BlueStore.cc

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #51275 from mchangir/mon-block-osd-pool-mksnap-for-fs-pools

mon: block osd pool mksnap for fs pools

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #50326 from ifed01/wip-ifed-better-osd-robust

osd: improve OSD robustness.

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #52360 from zdover23/wip-doc-2023-07-08-radosgw-multisite

doc/radosgw: add Zonegroup policy explanation

Reviewed-by: Casey Bodley <cbodley@redhat.com>

PendingReleaseNotes: note rgw's compress-encrypted zonegroup feature flag

Signed-off-by: Casey Bodley <cbodley@redhat.com>

doc/rgw: document compress-encrypted zonegroup feature flag

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #52301 from adk3798/cephadm-fix-rgw-migration

mgr/cephadm: fix rgw spec migration with simple specs

Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Redouane Kachach <rkachach@redhat.com>

rgw: require zonegroup feature to compress encrypted objects

Reef adds support for combining server-side encryption and compression,
but previous releases would not replicate such objects correctly. gate
the feature behind a zonegroup feature flag to require that all zones
upgrade to reef before enabling. this feature is not enabled by default
in new zonegroups

Fixes: https://tracker.ceph.com/issues/57905
Signed-off-by: Casey Bodley <cbodley@redhat.com>

doc/rgw: don't capitalize zonegroup feature flags

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: don't enable all zone features by default

on zonegroup creation, all `supported` features get enabled by default.
however, some features should remain opt-in. add a separate list of
`enabled` features for the subset of features we want enabled by default
on zonegroup creation

Signed-off-by: Casey Bodley <cbodley@redhat.com>

doc/radosgw: add Zonegroup policy explanation

Add revised Zonegroup policy for "multi-zonegroups". This commit
includes changes that Casey Bodley made in
https://github.com/ceph/ceph/pull/52324#discussion_r1253482258 and that
I have integrated into the docs only now.

Co-authored-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #51863 from phlogistonjohn/jjm-cephadm-argument-spec

cephadm: add a new ArgumentSpec type for advanced argument handling

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #52353 from bluikko/patch-12

doc/man/radosgw: use underscores in configuration var names

mgr/dashboard: move cephadm e2e cleanup to jenkins job config

Follows: https://github.com/ceph/ceph-build/pull/2145

Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: expose cluster upgrade orch API endpoints

Add Cluster Upgrade Management support from REST API endpoints.

Fixes: https://tracker.ceph.com/issues/61847
Signed-off-by: avanthakkar <avanjohn@gmail.com>

doc/man/radosgw: more underscores in config var names

Missed one section of `ceph.conf` example where spaces were used instead of underscores.

Change spaces in config var names to underscores.

doc/man/radosgw: use underscores in configuration

Most of the documentation already uses underscores instead of spaces in configuration variable names.

Update radosgw(8) man page to follow, changing spaces into underscores.

Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>

Merge pull request #52337 from ronen-fr/wip-rf-osdtypes-const

osd/osd_types_fmt: mark fmt::formatter<>::format() const

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #51179 from aisakaki/wip-lazy-read-r2

crimson/os/seastore: realize lazy read in split overwrite with overwrite refactor

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Myoungwon Oh <myoungwon.oh@samsung.com>
Reviewed-by: Zhang Song <zhangsong325@gmail.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #51770 from myoungwon/wip-multicore-rbm

crimson/os/seastore/rbm: make rbm support multiple shards

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #52324 from zdover23/wip-doc-2023-07-06-radosgw-multisite

doc/radosgw: add Zonegroup purpose

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #52319 from Thingee/foundation-mem-update-202307

doc/foundation: Updating foundation members for July 2023

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/radosgw: add Zonegroup purpose

Add a note about the telos of zonegroups. The information added in this
commit was originally sent to ceph-users by Casey Bodley on 30 Jun 2023
in response to a question from Yixin Jin. Alexander Patrakov then
suggested that Casey's response should be added to
doc/radsogw/multisite.rst.

Co-authored-by: Casey Bodley <cbodley@redhat.com>
Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

doc: add information on expediting MDS recovery

Fixes: https://tracker.ceph.com/issues/61865
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #52087 from AliMasarweh/wip-alimasa-persistant-q-observability

RGW: Add observability over the persistent topics queue

Merge pull request #52216 from cbodley/wip-61838

qa/s3tests: filter on 'sts_tests' and 'webidentity_tests'

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #50458 from Matan-B/wip-matanb-c-perf-basic

qa/suites/crimson-rados: Introduce performance testing

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Mark Nelson <mark.nelson@clyso.com>

rgw: Add check for whether RGWObjState was created in dbstore.cc

Signed-off-by: Vedansh Bhartia <vedanshbhartia@gmail.com>

osd/osd_types_fmt: mark fmt::formatter<>::format() const

For some formatters missed or added after PR#47830 (that fixed
most of the missing 'const's in this file)

Required to allow the use of the formatters in const member functions.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

RGW: Add observability over the persistent topics queue

Signed-off-by: Ali Masarwa <ali.saed.masarwa@gmail.com>

Merge pull request #52293 from ronen-fr/wip-rf-consttime

common: make some common ctors and accessors constexpr

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

src/test/crimson/seastore/nvmedevice: update a test case based on rbm multi shard change

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore/rbm: make rbm support multiple shards

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

Merge PR #52290 into main

* refs/pull/52290/head:
pybind/rbd: drop GIL when calling into librbd
pybind/cephfs: drop gil during cephfs callouts

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

doc/foundation: Updating foundation members for July 2023

Signed-off-by: Mike Perez <thingee@gmail.com>

mgr/cephadm: fix rgw spec migration with simple specs

The rgw spec migration code, intended to formalize
the rgw_frontend_type spec option, doesn't work with
simple specs i.e.

service_type: rgw
service_id: rgw.1
service_name: rgw.rgw.1
placement:
  label: rgw

because the migration code assumes there will always be
a "spec" section inside the spec. This is the case for
more involved rgw specs such as

service_type: rgw
service_id: foo
placement:
  label: rgw
  count_per_host: 2
spec:
  rgw_realm: myrealm
  rgw_zone: myzone
  rgw_frontend_type: "beast"
  rgw_frontend_port: 5000

which is what the migration is actually concerned about
(verification of the rgw_frontend_type in these specs).

In the case where the spec is more simple, we should
just leave the spec alone and move on. Unfortunately
the current code assumes the field will always be
there and hits an unhandled KeyError when trying to
migrate the more simple specs. This causes the
cephadm module to crash shortly after starting an
upgrade to a version that includes this migration
and it's very difficult to find the root cause. This
can be worked around by adding fields to the rgw
spec before upgrade so the "spec" field exists in
the spec and the migration works as intended.

This commit fixes the migration in the simple
case as well as adding testing for that case to
both the unit tests and orch/cephadm teuthology
upgrade tests

Fixes: https://tracker.ceph.com/issues/61889
Signed-off-by: Adam King <adking@redhat.com>

Merge pull request #50558 from rhcs-dashboard/fix-dataUnits-graph-plotting

mgr/dashboard: fix a bug where data would plot wrongly

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

common: make some common ctors and accessors constexpr

Add the constexpr qualifier to:
- some utime_t ctors and accessors;
- a shard_id_t ctor;
- ceph_le operator

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #51852 from Matan-B/wip-matanb-crimson-watch-notify-logs

crimson/osd/watch: Add logs around Watch/Notify

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #52086 from pkalever/fix-mirror-status-desc

rbd-mirror: fix image replayer shut down description on force promote

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #52207 from zdover23/wip-doc-2023-06-27-rados-operations-crush-map-edits-2-of-x-2nd-attempt

doc/rados: edit crush-map-edits (2 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: edit crush-map-edits (2 of x)

Edit doc/rados/operations/crush-map-edits.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Co-authored-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #52308 from zdover23/wip-doc-2023-07-05-seqdiag-test

doc/dev: remove seqdiag assets

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc/dev: remove seqdiag assets

Remove seqdiag assets to determine whether the docs can be built if they
are absent. (Currently they cannot be built when they are present.) If
this works, then these diagrams will be replaced, probably with .png
files.

Signed-off-by: Zac Dover <zac.dover@proton.me>

qa/tasks/cbt.py: Install collectl manually

See: d4a04809fd7fd8aaf447005f76a1090db99d75c5

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

qa/suites/crimson-rados: Introduce performance testing

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

test/crimson/seastore: test if lazy read work correctly

Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>

test/crimson/seastore: test overwrite pin

Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>

test/crimson/seastore: add test for TransactionManager::remap_pin

Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>

crimson/os/seastore: check if length exist in mark_space_used and mark_space_free

to avoid misuse of buffer length and data length when there is a no actual data extent

Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>

crimson/os/seastore: deepcopy the buffer of exist clean extent when duplicate_for_write

Beacuse exist clean extent shares buffer with original clean extent.

Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>

crimson/os/seastore: realize lazy read in split overwrite with overwrite refactor

Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>

crimson/os/seastore/transaction_manager: implement remap_pin

Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>

crimson/os/seastore/transaction_manager: implement alloc_remapped_extent

Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>

crimson/os/seastore: support no data extent in read path

Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>

crimson/osd: cleanup ShardDispatcher

ShardDispatcher logically equals to seastar::sharded<PGShardManager>,
drop unnecessary members.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/osd: avoid unnecessary foreign copy in ms_dispatch()

Foreign-copy also implies that requests may be out-of-order, which is
incorrect in the data path.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/osd: make sure OSD is constructing in the primary core

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #51422 from yuvalif/wip-yuval-cephadm-lua-doc

doc/rgw/lua: add info uploading a script in cephadm deployment

Reviewed-by: Zac Dover <zac.dover@proton.me>

pybind/rbd: drop GIL when calling into librbd

This was missing for rbd_mirror_peer_site_add() and
rbd_get_data_pool_id().

While at it, add a test for data_pool_id().

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #52247 from cbodley/wip-57905

rgw: rgwx-skip-decrypt also skips decompression of encrypted objects

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #52249 from mkogan1/wip-t61661-d3n-valg

rgw: d3n: fix valgrind reported leak related to libaio ops

crimson/tools/perf_crimson_msgr: client don't skip core 0 if specified explicitly

Usually, we skip core 0 to avoid unwanted performance impacts, but that
may not be always the case. Add the configuration not to skip core 0.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/tools/perf_crimson_msgr: don't create client/server if not necessary

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/tools/perf_crimson_msgr: report server-side conns/iops/throughput

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: let CachedExtent::ptr optional

separate buffer length and data length, since no buffer extent is allowed

Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>

crimson/tools/perf_crimson_msgr: use lowres clock in the hot path

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/tools/perf_crimson_msgr: integrate multi-core messenger as server

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/tools/perf_crimson_msgr: configure clients and client connections

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/tools/perf_crimson_msgr: change depth to depth-per-job

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/tools/perf_crimson_msgr: add server multi core option

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/tools/perf_crimson_msgr: decrease sampling rate

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/tools/perf_crimson_msgr: support stop signal in server mode

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/tools/perf_crimson_msgr: collect server reactor utilizations without client

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/tools/perf_crimson_msgr: collect server reactor utilizations with client

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/tools/perf_crimson_msgr: report client reactor utilizations

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/tools/perf_crimson_msgr: report aggregated client summary

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/tools/perf_crimson_msgr: rearrange client cores

Move client cores to the largest smp shards, and reserve core-0 to
reduce performance impacts.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/tools/perf_crimson_msgr: misc cleanups

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/tools/perf_async_msgr: print configurations

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/tools/perf_async_msgr: support configure ms_async_op_threads

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #52160 from soumyakoduri/wip-skoduri-lc-acl

rgw/set_attrs: Do not modify mtime

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #51983 from clwluvw/aws4-date-header

rgw: pick http_date in case of http_x_amz_date absence

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

pybind/cephfs: drop gil during cephfs callouts

This has disastorous consequences including the possibility of deadlock.
In the best case, you have the rmdir holding the GIL until the MDS
responds!

Fixes: https://tracker.ceph.com/issues/61869
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

crimson/osd/watch: don't print bufferlists

The bufferlists clutter the logs and don't infrom of anything useful:
```
DEBUG 2023-05-26 23:36:29,780 [shard 2] osd - send_completion sending notify replies: {notify_reply_t{watcher_gid=4220, watcher_cookie=139786421930752, bl=buffer::list(len=0,

)}, notify_reply_t{watcher_gid=4220, watcher_cookie=139786422032096, bl=buffer::list(len=0,

)}}
```

Signed-off-by: Matan Breizman <mbreizma@redhat.com>