doc/radosgw: improve grammar - notifications.rst

This commit makes the changes suggested by C. Mitchell in
https://github.com/ceph/ceph/pull/48486.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48422 from aravind-wdc/refactor

crimson/zns: refactor zone operations

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #48486 from zdover23/wip-doc-2022-10-14-radosgw-push_endpoint

doc/radosgw: add push_endpoint for rabbitmq

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

crimson/zns: refactor zone operation

Moved all zone operations into one routine.

Signed-off-by: Aravind Ramesh <Aravind.Ramesh@wdc.com>

doc/radosgw: add push_endpoint for rabbitmq

This commit directs users of rabbitmq to use "push_endpoint"
(with an underscore) instead of "push-endpoint" (with a hy-
phen). This commit adds a note that contains such a direct-
ive. It does not alter the examples already present in the
text.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48475 from rzarzynski/wip-crimson-resolve_oid-clone_snaps-fix-search

crimson/osd: resolve_oid() looks in clone_snaps by requested snap ID

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #48405 from athanatos/sjust/wip-57759

crimson: tmap support, list_snaps support, fixes for zero and cmpext

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

crimson/osd: resolve_oid() looks in clone_snaps by requested snap ID

Before the patch the method was looking for `lower_bound(oid.snap)`
from `SnapSet::clones` which leads to reading from head instead of
returning the `ENOENT` expected in `TestLibRBD.TestIOToSnapshot`.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #48437 from rosinL/fix-vstart

src/vstart.sh: correct top_cpu's value

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #48457 from cyx1231st/wip-fix-msgr-test

test/crimson/test_messenger: add pause before marking down a connection

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #48463 from zdover23/wip-doc-2022-10-14-dev-dev-guide-basic-workflow-submodule-update-2

doc/dev: add submodule-update link to dev guide

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/dev: add submodule-update link to dev guide

This commit links to the procedure in install/clone-source.rst that
explains how to update submodules.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48460 from zdover23/wip-doc-2022-10-14-install-clone-source-submodules

doc/install: improve updating submodules procedure

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

crimson/osd: CEPH_OSD_OP_ZERO should become truncate if past end of object

librbd image diff seems to rely on this behavior to modify oi.size for
LIST_SNAPS.

Fixes: https://tracker.ceph.com/issues/57791
Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd: add CEPH_OSD_OP_LIST_SNAPS

Fixes: https://tracker.ceph.com/issues/57789
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #47853 from WOnder93/selinux-io_uring

selinux: prepare for anon inode controls enablement

doc/install: improve updating submodules procedure

This commit beefs up the "Updating Submodules" procedure,
which (prior to this commit) was quite brief.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

crimson/net: print connection memory address as the identity

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson/test_messenger: add pause before marking down a connection

To make sure the ACK messages are consumed, otherwise there can be
unexpected connect attempts to break the test.

Fixes: https://tracker.ceph.com/issues/57693
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #48377 from ktdreyer/win-deps-clone

win32: speed up and simplify deps cloning

Merge pull request #48406 from athanatos/sjust/wip-57798

qa/suites/crimson-rados: add CRIMSON_COMPAT to workunit env

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #48446 from zdover23/wip-doc-2022-10-12-glossary-ceph-monitor

doc/glossary: add "ceph monitor" entry

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/glossary: add "ceph monitor" entry

This commit adds information to the "ceph monitor" entry in
glossary.rst.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48196 from neesingh-rh/wip-human-readable-volume-info

mgr/volumes: Add human-readable flag to volume info command

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #48367 from stefan-chivu/namespace-optarg-fix

rbd: device map/unmap --namespace handling fixes

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #45395 from tobias-urdin/rgw-keystone-service-token

rgw/auth: Add service token support for Keystone auth

src/vstart.sh: correct top_cpu's value

Signed-off-by: luo rixin <luorixin@huawei.com>

Merge pull request #48415 from rhcs-dashboard/repaired-metric-name

mgr/prometheus: change pg_repaired_objects name to pool_repaired_objects

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #47495 from rhcs-dashboard/rgw-s3-encryption

mgr/dashboard: add server side encryption to rgw/s3

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #46331 from lxbsz/wip-55332

mds: wait unlink to finish to avoid conflict when creating same dentries

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Milind Changire <mchangir@redhat.com>

mgr/dashboard: add server side encryption to rgw/s3

Fixes:https://tracker.ceph.com/issues/57826
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #48350 from athanatos/sjust/wip-57739

crimson/.../replicated_request,logmissing_request: retain reference to req

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Radoslaw Zarzynski <rzarzynski@redhat.com>

Merge pull request #48308 from cyx1231st/wip-seastore-offset

crimson/os/seastore: introduce (64-8)-bit signed device_off_t and 32-bit signed segment_off_t

Reviewed-by: Myoungwon Oh <myoungwon.oh@samsung.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Liu-Chunmei <chunmei.liu@intel.com>

qa/suites/crimson-rados: add CRIMSON_COMPAT to workunit env

Fixes: https://tracker.ceph.com/issues/57798
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #48349 from rzarzynski/wip-crimson-cltreq-errreply-debug

crimson/osd: introduce debug to ClientRequest::reply_op_error()

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Liu-Chunmei <chunmei.liu@intel.com>

Merge pull request #47754 from batrick/i57248

qa: cleanup volumes on unwind

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

PendingReleaseNotes: add "rbd device unmap --namespace" note

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/workunits/rbd-nbd: also test map/unmap with --namespace option

The entire script is using specs, let's cover non-spec syntax as well.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

win32: speed up and simplify deps cloning

Use --depth 1 for all the dependencies we clone to speed up the process.

Use the --branch argument for cloning all dependencies. This simplifies
the process and makes it easier to use other copies in an offline
environment where github.com is inaccessible.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #48423 from zdover23/wip-doc-2022-10-11-releases-pacific-grammar

doc/releases: improve grammar in pacific.rst

Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #48396 from rhcs-dashboard/fix-57792-main

mgr/dashboard: fix broken login links

Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #48337 from batrick/first-python

tools/cephfs: convert first-damage.sh to python

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>

Merge pull request #46772 from cbodley/wip-rgw-doc-bucket-index

doc/dev/radosgw: start describing the bucket index

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/releases: improve grammar in pacific.rst

This commit accepts the grammar suggestions that were made by Cole
Mitchell in https://github.com/ceph/ceph/pull/48404.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

mgr/prometheus: change pg_repaired_objects name to pool_repaired_objects

Fixes: https://tracker.ceph.com/issues/57806
Signed-off-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #48331 from tchaikov/crimson-stop

crimson/osd: shutdown on osdmaps' demand

Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/osd/pg_backend: remove incorrect length limitation on CMPEXT

We specifically send back errors < -MAX_ERRNO to indicate offset.

Fixes: https://tracker.ceph.com/issues/57773
Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd: add tmap support

rbdv1 headers use tmap, it's somewhat easier to add support to crimson
than to modify those tests to avoid rbdv1 on crimson.

Fixes: https://tracker.ceph.com/issues/57759
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #48238 from sisco0/fix-rgw-amqp-switch-compiler-warning-2

rgw: fix rabbitmq version checking for AMQP_STATUS_SSL_SET_ENGINE_FAILED

mgr/dashboard: fix broken login links

Fixes: https://tracker.ceph.com/issues/57792
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

rbd: Removed device_name argument from wnbd unmap

Right now, rbd-wnbd doesn't actually use disk path
identifiers such as "/dev/*" or "\\.\PhysicalDrive*".

So instead of accepting two arguments that are basically
handled more or less the same, we're dropping the device_name
argument and sticking to the image spec.

Signed-off-by: Stefan Chivu <schivu@cloudbasesolutions.com>

rbd: Moved get_image_or_snap_spec and parse_options to utils

At the moment, there are multiple copies of "get_image_or_snap_spec".

For this reason, we're moving it to Utils.cc so that all backends may
re-use it, also benefiting from the recent pool related fix.

We are also renaming parse_options to append_options_as_args and
changing its type to void

Signed-off-by: Stefan Chivu <schivu@cloudbasesolutions.com>

rbd: Added namespace optarg for rbd device unmap

When executing rbd device unmap, an explicit namespace optarg wasn't available

Fixes: https://tracker.ceph.com/issues/57765
Signed-off-by: Stefan Chivu <schivu@cloudbasesolutions.com>

Merge pull request #48275 from liu-chunmei/rados-api-test

crimson/qa: add rados_api_tests.yaml for seastore

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #48162 from Matan-B/wip-matanb-c-rados-test

qa/suites/crimson-rados/thrash: Adding workloads / Simplifying thrashers

Reviewed-by: Samuel Just <sjust@redhat.com>

qa/workunits/fs/damage: update first-damage test for python

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

tools/cephfs: convert first-damage.sh to python

To have only one RADOS instance and improve performance/reliability.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #46925 from rkachach/fix_issue_44461

mgr/cephadm: add a simple mechanism to check grafana cert/key

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #48218 from mchangir/qa-test_scrub_pause_and_resume_with_abort-wait-for-scrub-to-finish

qa: wait for scrub to finish

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Neeraj Pratap Singh <neesingh@redhat.com>

crimson/osd: introduce debug to ClientRequest::reply_op_error()

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #48041 from cbodley/wip-rgw-notify-shutdown

rgw: rados service shuts down rados client on shutdown()

Reviewed-by: Soumya Koduri <skoduri@redhat.com>
Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #48152 from Vfrost98/GHC22OSD_Ceph

mgr/telemetry: fixed log exceptions as "exception" instead of "error"

Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Yaarit Hatuka <yaarithatuka@gmail.com>

RGWServices_Def::shutdown() shuts down the rest of the services too

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: shut down RGWDataChangesLog before other services

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: rados service shuts down rados client on shutdown()

if we leave the rados client running, it will keep delivering
AioCompletions while we're shutting other things down

this resolves a valgrind use-after-free where rgw::notify::Manager
gets completions after its destruction

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #48155 from liangmingyuanneo/wip-rgw-aync-refcount

rgw: async refcount operate in copy_obj

Reviewed-by: Anthony D Atri <anthony.datri@gmail.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #47186 from cfsnyder/wip-cfsnyder-56645

rgw: log deletion status of individual objects in multi object delete request

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

rgw: async refcount operate in copy_obj

Signed-off-by: Mingyuan Liang <liangmingyuan@baidu.com>

crimson/osd/stop_signal: s/signaled/on_signal/

to better reflect that it is a callback gets called when receiving a
signal.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

crimson/.../replicated_request,logmissing_request: retain reference to req

Otherwise, operator<< and dump_detail would need to accessing req after
the call to handle_rep_op.

Fixes: https://tracker.ceph.com/issues/57739
Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd: signal should_stop in OSD::shutdown()

so osd is able to shut itself down per monitor's request.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

crimson/osd: drop redundant code

this change is a cleanup.

we already update `superblock` with the latest `boost_epoch` and
osdmap's epoch in `OSD::handle_osd_map()`, and `committed_osd_maps()` is
called at end of this function. `shutdown()` is called when we marked
down and stop by the monitor in `committed_osd_maps()`. so these
assignment statements are noops. instead, we should stop the whole osd
service. let's leave it for another commit.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

crimson/osd: use abort_source in stop_signal

before this change, `stop_signal::wait()` waits until it receives
`SIGTERM` or `SIGINT`, but we also need to stop the service per the
request of monitor or when a serious health condition is detected.

so, an `abort_source` should allow the server to request abort by
itself. also, as the single truth of stop, `stop_signal` will be
able to send the message to its subscribers to abort any "blocking"
calls which might prevent or delay the stop process.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

crimson/osd: vendor stop_signal.h

stop_signal.h is copied from seastar/apps/lib/stop_signal.hh. this
change copies it int our project so we can customize it in a following
commit. we will need to add an `abort_source` member to it.

the class `stop_signal` works great for us. but under some
circumstances, we also need to stop the crimson server programmatically
from the server itsef, for instance, per the request of monitor. in that
case, `stop_signal` cannot fulfill our needs, as the only thing which
can stop it is, SIGTERM or SIGTERM. we could send SIGINT to ourselves to
unblock `stop_signal::wait()`. but it would be better if we could leverage
`abort_source` for this purpose. for two reasons:

* `abort_source` allows use to cancel a "blocking" op.
  we have a couple background tasks in crimson, like `AdminSocket`'s
  accept and handle task, which could have been stopped by an
  `abort_source`. but now it checks for the `stop_gate` before accepting
  an incoming connection. this pattern cannot be always be repeated.
  because we cannot *abort* a "blocking" task. for instance, we use
  a homebrew `tri_mutex` to protect the read/write consistency of
  an obc, what if the server would need to stop when a request is
  still trying to acquire the lock? with the help of the `abort_source`,
  we should be able to either subscribe from the `abort_source`,
  and set an abort exception for each of the waiters. the same applies
  to other "blocking" calls like `shared_promise::get_shared_future()`,
  this allows us to abort the waiter before the promise is resolved.
  this could be handy if we need to abort potentially op which
  might take long and should be aborted when monitor wants to
  kill the osd instance.
* `abort_source` allows us to abort the services programmatically.
  this is the use case mentioned in the beginning of this commit
  message.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #48322 from zdover23/wip-doc-2022-10-01-rados-ops-ec-front-matter-rewrite

doc/rados: rewrite EC intro

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/rados: rewrite EC intro

This PR rewrites the front matter in the "Erasure Code"
section of the RADOS documentation. Previously, the information
in this section was syntactically confused. I have also fleshed
out the distinction between erasure coding and replication.

Co-Author: Cole Mitchell <cole.mitchell@gmail.com>
Signed-off-by: Zac Dover <zac.dover@gmail.com>
Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #43567 from jdurgin/wip-gov

doc/governance: add proposed structure

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>

doc: merge new and old governance description

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

doc/governance: updates based on review comments

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #48072 from ceph/wip-yuriw-release-17.2.4-main

doc: 17.2.4 Release Notes

doc: 17.2.4 Release Notes

Fixes: https://tracker.ceph.com/issues/57472
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
Signed-off-by: Laura Flores <lflores@redhat.com>

Merge pull request #48259 from ktdreyer/build-win-fixes

win32: support building windows client on RHEL

Reviewed-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
Reviewed-by: Justin Caratzas <jcaratza@redhat.com>

doc/governance: update based on review and CLT discussions

Clarified some parts of council that were discussed previously,
specifying the number of members and a staggered term.

Added a bit more about the steering committee - thinking the meetings
could be split between tactical (3/4 weeks) and strategic (monthly),
and still open to anyone to join the discussion, but restricted to
only members voting.

Removed the meeting section since that belongs more in a separate
place, like the ceph.io website.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

doc/governance: add proposed structure

This is Sage's summary of the Ceph leadership team discussions around
this topic. Still many details to be worked out, this is just one
concrete proposal as a basis for further discussion.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

crimson/os/seastore: drop the replaced seastore/block_off_t

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: use device_off_t for offset at seastore level

* Replace the reset of seastore_off_t by the extended device_off_t.
* Extend offset from 32-bit to 56-bit signed integer at seastore level.
* res_paddr_t to embed device_off_t.
* blk_paddr_t to use signed device_off_t for consistency.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #47249 from nSedrickm/aria-labels-accessibility-update

mgr/dashboard: Improve level A accessibility for missing aria labels in tabs and listitems

Reviewed-by: Pegonzal <NOT@FOUND>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

crimson/os/seastore: use segment_off_t for segment offsets

Segment offset has range from 32-bit signed integer.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: use extent_len_t where appropriate

Replacing inappropriate usages of seastore_off_t.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: validate segment_size, size and num_segments limits

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: relax journal size limit beyound segment-size

Introduce a signed device_off_t type with the 56-bit limit, larger than
the 32-bit seastore_off_t.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/qa: add rados_api_tests.yaml for seastore

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

doc/dev/radosgw: describe the bucket index

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #48194 from zmc/python36-prettytable

qa: add tests for 'fs volume command --human_readable'

Fixes: https://tracker.ceph.com/issues/57620
Signed-off-by: Neeraj Pratap Singh <neesingh@redhat.com>

doc: documenting the --human_readable flag for volume info cmd

Fixes: https://tracker.ceph.com/issues/57620
Signed-off-by: Neeraj Pratap Singh <neesingh@redhat.com>

qa: wait for scrub to finish

Wait for scrub to finish during test_scrub_pause_and_resume_with_abort
which otherwise races and fails with an incorrect assertion.

Fixes: https://tracker.ceph.com/issues/48812
Signed-off-by: Milind Changire <mchangir@redhat.com>