crimson/zns: refactor zone operation

Moved all zone operations into one routine.

Signed-off-by: Aravind Ramesh <Aravind.Ramesh@wdc.com>

Merge pull request #48475 from rzarzynski/wip-crimson-resolve_oid-clone_snaps-fix-search

crimson/osd: resolve_oid() looks in clone_snaps by requested snap ID

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #48405 from athanatos/sjust/wip-57759

crimson: tmap support, list_snaps support, fixes for zero and cmpext

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

crimson/osd: resolve_oid() looks in clone_snaps by requested snap ID

Before the patch the method was looking for `lower_bound(oid.snap)`
from `SnapSet::clones` which leads to reading from head instead of
returning the `ENOENT` expected in `TestLibRBD.TestIOToSnapshot`.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #48437 from rosinL/fix-vstart

src/vstart.sh: correct top_cpu's value

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #48457 from cyx1231st/wip-fix-msgr-test

test/crimson/test_messenger: add pause before marking down a connection

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #48463 from zdover23/wip-doc-2022-10-14-dev-dev-guide-basic-workflow-submodule-update-2

doc/dev: add submodule-update link to dev guide

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/dev: add submodule-update link to dev guide

This commit links to the procedure in install/clone-source.rst that
explains how to update submodules.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48460 from zdover23/wip-doc-2022-10-14-install-clone-source-submodules

doc/install: improve updating submodules procedure

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

crimson/osd: CEPH_OSD_OP_ZERO should become truncate if past end of object

librbd image diff seems to rely on this behavior to modify oi.size for
LIST_SNAPS.

Fixes: https://tracker.ceph.com/issues/57791
Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd: add CEPH_OSD_OP_LIST_SNAPS

Fixes: https://tracker.ceph.com/issues/57789
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #47853 from WOnder93/selinux-io_uring

selinux: prepare for anon inode controls enablement

Merge pull request #48427 from batrick/obsolete-comments

mds: remove obsolete comments

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

doc/install: improve updating submodules procedure

This commit beefs up the "Updating Submodules" procedure,
which (prior to this commit) was quite brief.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

crimson/net: print connection memory address as the identity

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson/test_messenger: add pause before marking down a connection

To make sure the ACK messages are consumed, otherwise there can be
unexpected connect attempts to break the test.

Fixes: https://tracker.ceph.com/issues/57693
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #48377 from ktdreyer/win-deps-clone

win32: speed up and simplify deps cloning

Merge pull request #48406 from athanatos/sjust/wip-57798

qa/suites/crimson-rados: add CRIMSON_COMPAT to workunit env

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #48446 from zdover23/wip-doc-2022-10-12-glossary-ceph-monitor

doc/glossary: add "ceph monitor" entry

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/glossary: add "ceph monitor" entry

This commit adds information to the "ceph monitor" entry in
glossary.rst.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48196 from neesingh-rh/wip-human-readable-volume-info

mgr/volumes: Add human-readable flag to volume info command

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #48367 from stefan-chivu/namespace-optarg-fix

rbd: device map/unmap --namespace handling fixes

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #45395 from tobias-urdin/rgw-keystone-service-token

rgw/auth: Add service token support for Keystone auth

Merge pull request #48412 from rhcs-dashboard/fix-rgw-subuser-permissions

mgr/dashboard: Unable to change rgw subuser permission

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

src/vstart.sh: correct top_cpu's value

Signed-off-by: luo rixin <luorixin@huawei.com>

Merge pull request #48415 from rhcs-dashboard/repaired-metric-name

mgr/prometheus: change pg_repaired_objects name to pool_repaired_objects

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #47495 from rhcs-dashboard/rgw-s3-encryption

mgr/dashboard: add server side encryption to rgw/s3

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #46331 from lxbsz/wip-55332

mds: wait unlink to finish to avoid conflict when creating same dentries

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Milind Changire <mchangir@redhat.com>

mgr/dashboard: Unable to change rgw subuser permission

Tried to edit the permission of a subuser but once changed the permission, edited permission not seen in user info of UI(ceph dashboard) and from CLI as well

Fixes:https://tracker.ceph.com/issues/57805
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #48407 from rhcs-dashboard/fix-weird-data-osd

mgr/dashboard: fix weird data in osd details

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #48429 from zdover23/wip-doc-2022-10-11-security-cve-2022-0670-grammar

doc/security: improve grammar in CVE-2022-0670.rst

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

Merge pull request #48418 from tchaikov/wip-ceph-dencoder

cmake: do not link ceph-dencoder against global

Reviewed-by: Tim Serong <tserong@suse.com>

mgr/dashboard: add server side encryption to rgw/s3

Fixes:https://tracker.ceph.com/issues/57826
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

doc/security: improve grammar in CVE-2022-0670.rst

This commit accepts the suggestions made by Cole Mitchell in
https://github.com/ceph/ceph/pull/48404.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48163 from rhcs-dashboard/refactor-cephadm-e2e

mgr/dashboard: refactor dashboard cephadm e2e tests

Reviewed-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #47812 from rishabh-d-dave/json-decode-error

qa/cephfs: handle non-numeric values for json.loads()

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #47982 from SUSE/wip-fix-57390-2

ceph-dencoder: Add global to denc-mod-osd's target_link_libraries

Reviewed-by: Kefu Chai <tchaikov@gmail.com

Merge pull request #48350 from athanatos/sjust/wip-57739

crimson/.../replicated_request,logmissing_request: retain reference to req

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Radoslaw Zarzynski <rzarzynski@redhat.com>

Merge pull request #48308 from cyx1231st/wip-seastore-offset

crimson/os/seastore: introduce (64-8)-bit signed device_off_t and 32-bit signed segment_off_t

Reviewed-by: Myoungwon Oh <myoungwon.oh@samsung.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Liu-Chunmei <chunmei.liu@intel.com>

qa/suites/crimson-rados: add CRIMSON_COMPAT to workunit env

Fixes: https://tracker.ceph.com/issues/57798
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #48349 from rzarzynski/wip-crimson-cltreq-errreply-debug

crimson/osd: introduce debug to ClientRequest::reply_op_error()

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Liu-Chunmei <chunmei.liu@intel.com>

mds: remove obsolete comments

It's no longer necessary to put messages.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #47754 from batrick/i57248

qa: cleanup volumes on unwind

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #48064 from lxbsz/asok_fs

qa: the default CEPH_ASOK_DIR is asok/ now

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Milind Changire <mchangir@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

Merge pull request #48161 from trociny/wip-57589

cephfs-data-scan: make scan_links more verbose

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #48182 from lxbsz/wip-57580

qa: make mount to be compatible with nautilus for blocklist

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #48421 from tchaikov/wip-cephadm-pyfakefs

cephadm: run tests as root

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

PendingReleaseNotes: add "rbd device unmap --namespace" note

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/workunits/rbd-nbd: also test map/unmap with --namespace option

The entire script is using specs, let's cover non-spec syntax as well.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

win32: speed up and simplify deps cloning

Use --depth 1 for all the dependencies we clone to speed up the process.

Use the --branch argument for cloning all dependencies. This simplifies
the process and makes it easier to use other copies in an offline
environment where github.com is inaccessible.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #48423 from zdover23/wip-doc-2022-10-11-releases-pacific-grammar

doc/releases: improve grammar in pacific.rst

Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #48396 from rhcs-dashboard/fix-57792-main

mgr/dashboard: fix broken login links

Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #48337 from batrick/first-python

tools/cephfs: convert first-damage.sh to python

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>

Merge pull request #46772 from cbodley/wip-rgw-doc-bucket-index

doc/dev/radosgw: start describing the bucket index

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

cephadm: pin pyfakefs to 5.0

so the test is more repeatable.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

cephadm: fix the indent

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

cephadm: run tests as root

as we will be create/access directories which are only accessible by
root with pyfakefs, and pyfake respects the uid of current effective
user, so if we run the test using non-root user, these tests would fail
after the fix for https://github.com/pytest-dev/pyfakefs/issues/489
is addressed in the pyfakefs we are using.

in this change, we will run the test on behalf of root, so we can
create and access the directories as we did before.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

doc/releases: improve grammar in pacific.rst

This commit accepts the grammar suggestions that were made by Cole
Mitchell in https://github.com/ceph/ceph/pull/48404.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

cmake: do not link ceph-dencoder against global

it does not use any symbol from global, so no need to link it
against global.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

mgr/prometheus: change pg_repaired_objects name to pool_repaired_objects

Fixes: https://tracker.ceph.com/issues/57806
Signed-off-by: Pere Diaz Bou <pdiazbou@redhat.com>

Revert "mgr/dashboard: unselect rows in datatables"

This reverts commit b79e2a6c6a9368a4fc167b05970db463cd60edab.

Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: refactor dashboard cephadm e2e tests

Fixes: https://tracker.ceph.com/issues/57511
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: fix weird data in osd details

The devices section in the OSD Details and Host Details shows more than
one daemon and device path in the column when you view the details of a
single osd details/host details. This is because more than one osd
is created on a device with same `deviceid`. I am not sure if this will
happen in the real environment but its mostly reproducible in
environments with QEMU emulated devices.

Fixes: https://tracker.ceph.com/issues/57803
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #48331 from tchaikov/crimson-stop

crimson/osd: shutdown on osdmaps' demand

Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/osd/pg_backend: remove incorrect length limitation on CMPEXT

We specifically send back errors < -MAX_ERRNO to indicate offset.

Fixes: https://tracker.ceph.com/issues/57773
Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd: add tmap support

rbdv1 headers use tmap, it's somewhat easier to add support to crimson
than to modify those tests to avoid rbdv1 on crimson.

Fixes: https://tracker.ceph.com/issues/57759
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #48238 from sisco0/fix-rgw-amqp-switch-compiler-warning-2

rgw: fix rabbitmq version checking for AMQP_STATUS_SSL_SET_ENGINE_FAILED

Merge pull request #48401 from zdover23/wip-doc-2022-10-09-crush-pdf-link-update

doc/various: update link to CRUSH pdf

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/various: update link to CRUSH pdf

This commit updates link to the research paper that announces and
explains the CRUSH algorithm. This link was broken in the migration from
the old Ceph website to ceph.io.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48399 from underacloud/wip-doc-spellin-errors

doc/radosgw: Fixes spelling errors in vault.rst

doc: Fixes spelling errors in vault.rst

Signed-off-by: Mary Frances underacloud.tech@gmail.com

Merge pull request #48393 from rzarzynski/wip-crimson-drop-omapiter

crimson: drop OmapIterator, use omap_get_values() instead

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #48073 from rzarzynski/wip-crimson-asok-dump_recovery_reservations

crimson/osd: bring the dump_recovery_reservations asok cmd

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Liu-Chunmei <chunmei.liu@intel.com>

crimson/os: drop the OmapIterator concept

`get_omap_values()` is powerful enough to be used instead.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

test/crimson: unittest-seastar uses omap_get_values() instead of OmapIterator

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: convert seastarized PGLog from OmapIterator to omap_get_values()

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: read_omap_for_push_op() takes max_len as ref, not ptr

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: ReplicatedRecoveryBackend uses omap_get_values() instead of OmapIterator

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #48374 from rkachach/fix_issue_57769

doc/cephadm: fixing grafana cert/key generation documentation

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

Merge pull request #48128 from alimaredia/wip-intrusive-lru-destructor-add

common:  intrusive_lru destructor add

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc/cephadm: fixing grafana cet/key generation documentation
Fixes: https://tracker.ceph.com/issues/57769
Signed-off-by: Redouane Kachach <rkachach@redhat.com>

mgr/dashboard: fix broken login links

Fixes: https://tracker.ceph.com/issues/57792
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #48378 from ceph/wip-yuriw-crontab-master

qa/testing: added pacific-x and quincy-x for reef release

Reviewed-by: Laura Flores <lflores@redhat.com>

crimson/osd: bring the dump_recovery_reservations asok cmd

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

rbd: Removed device_name argument from wnbd unmap

Right now, rbd-wnbd doesn't actually use disk path
identifiers such as "/dev/*" or "\\.\PhysicalDrive*".

So instead of accepting two arguments that are basically
handled more or less the same, we're dropping the device_name
argument and sticking to the image spec.

Signed-off-by: Stefan Chivu <schivu@cloudbasesolutions.com>

rbd: Moved get_image_or_snap_spec and parse_options to utils

At the moment, there are multiple copies of "get_image_or_snap_spec".

For this reason, we're moving it to Utils.cc so that all backends may
re-use it, also benefiting from the recent pool related fix.

We are also renaming parse_options to append_options_as_args and
changing its type to void

Signed-off-by: Stefan Chivu <schivu@cloudbasesolutions.com>

rbd: Added namespace optarg for rbd device unmap

When executing rbd device unmap, an explicit namespace optarg wasn't available

Fixes: https://tracker.ceph.com/issues/57765
Signed-off-by: Stefan Chivu <schivu@cloudbasesolutions.com>

Merge pull request #48275 from liu-chunmei/rados-api-test

crimson/qa: add rados_api_tests.yaml for seastore

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #48162 from Matan-B/wip-matanb-c-rados-test

qa/suites/crimson-rados/thrash: Adding workloads / Simplifying thrashers

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #48277 from cyx1231st/wip-crimson-tools

crimson/net: fix msgr tools and test helpers

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #48351 from athanatos/sjust/wip-57738

crimson/osd/replicated_backend: block _submit_transaction on sending the messages

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #47934 from phlogistonjohn/jjm-nfs-sectype

mgr/nfs: add sectype option

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #48301 from PepperJo/cmp_write_C_vec

librbd: add compare-and-write vector C API

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #48285 from adk3798/upgrade-start-version

qa/cephadm: do not start upgrades from Octopus

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #47903 from rkachach/fix_issue_57366

mgr/cephadm: reconfig prometheus when service-discovery port changes

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #48372 from idryomov/wip-57766

qa/tasks/rbd_fio: bump default to fio 3.32

Reviewed-by: Sunny Kumar <sunkumar@redhat.com>

PendingReleaseNotes: add rbd_aio_compare_and_writev note

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #47731 from mjenglish/multiple-actuator-osd-specification

python-common/ceph/deployment: filter drives by actuators when creating osds

Merge pull request #48319 from gregsfortytwo/wip-57737-cephfs-doc

doc: discuss the standard multi-tenant CephFS security model

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #48379 from zdover23/wip-doc-2022-10-06-install-clone-source-master-to-main

doc/install: clone-source.rst s/master/main

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>