mgr/prometheus: avoid duplicates and deleted entries for rbd_stats_pool

Fixes: https://tracker.ceph.com/issues/57797
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
Avoid duplicate entries (pool[/namespace]) for rbd_stats_pools config and also avoid deleted pools.

Merge pull request #48374 from rkachach/fix_issue_57769

doc/cephadm: fixing grafana cert/key generation documentation

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

Merge pull request #48128 from alimaredia/wip-intrusive-lru-destructor-add

common:  intrusive_lru destructor add

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc/cephadm: fixing grafana cet/key generation documentation
Fixes: https://tracker.ceph.com/issues/57769
Signed-off-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #48378 from ceph/wip-yuriw-crontab-master

qa/testing: added pacific-x and quincy-x for reef release

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #48275 from liu-chunmei/rados-api-test

crimson/qa: add rados_api_tests.yaml for seastore

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #48162 from Matan-B/wip-matanb-c-rados-test

qa/suites/crimson-rados/thrash: Adding workloads / Simplifying thrashers

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #48277 from cyx1231st/wip-crimson-tools

crimson/net: fix msgr tools and test helpers

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #48351 from athanatos/sjust/wip-57738

crimson/osd/replicated_backend: block _submit_transaction on sending the messages

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #47934 from phlogistonjohn/jjm-nfs-sectype

mgr/nfs: add sectype option

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #48301 from PepperJo/cmp_write_C_vec

librbd: add compare-and-write vector C API

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #48285 from adk3798/upgrade-start-version

qa/cephadm: do not start upgrades from Octopus

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #47903 from rkachach/fix_issue_57366

mgr/cephadm: reconfig prometheus when service-discovery port changes

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #48372 from idryomov/wip-57766

qa/tasks/rbd_fio: bump default to fio 3.32

Reviewed-by: Sunny Kumar <sunkumar@redhat.com>

PendingReleaseNotes: add rbd_aio_compare_and_writev note

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #47731 from mjenglish/multiple-actuator-osd-specification

python-common/ceph/deployment: filter drives by actuators when creating osds

Merge pull request #48319 from gregsfortytwo/wip-57737-cephfs-doc

doc: discuss the standard multi-tenant CephFS security model

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #48379 from zdover23/wip-doc-2022-10-06-install-clone-source-master-to-main

doc/install: clone-source.rst s/master/main

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/install: clone-source.rst s/master/main

s/master/main in doc/install/clone-source.rst.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #46925 from rkachach/fix_issue_44461

mgr/cephadm: add a simple mechanism to check grafana cert/key

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #48353 from athanatos/sjust/wip-dump-live-ops

crimson: improve dump_ops_in_flight

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #48352 from athanatos/sjust/wip-57740

crimson: allow pg creation to be canceled if pool is removed

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #48195 from athanatos/sjust/wip-57617

crimson/osd: populate version/user_version in reply for duplicate ops

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

qa/testing: added pacific-x and quincy-x for reef release

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #48289 from adamemerson/wip-warn-warn-warn

src/: silence Clang and GCC Warnings

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

doc: discuss the standard multi-tenant CephFS security model

Fixes: https://tracker.ceph.com/issues/57737
Signed-off-by: Greg Farnum <gfarnum@redhat.com>

doc/mgr/nfs: document --sectype option for export create commands

Add documentation for the option to specify the sectype (for enabling kerberos)
when creating a new export.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

mgr/nfs: add test cases to verify sectype can be set

Signed-off-by: John Mulligan <jmulligan@redhat.com>

mgr/nfs: add --sectype flag to nfs export create functions

Fixes: https://tracker.ceph.com/issues/57404
Add the sectype argument, also for use as a CLI option, to the functions
that create nfs exports. NFS Ganesha can use the sectype configuration
option to decide what kind of security will be applied/required for
connection to an export.

NOTE: This option is only useful when Ganesha is configured along with
LDAP/Kerberos integration. Configuration of that integration is outside
the scope of these patches.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

mgr/nfs: reformat long function calls to one line per arg

Using a hand rolled version of the long function call style similar to
that produced by the "Black" tool, put every argument to the
create_export function on a line of its own. This makes the code easier
to read and, importantly for an upcoming change, easier to add new
arguments.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

mgr/nfs: add sectype argument to nfs export management funcs

If the caller has supplied a sectype argument, pass it along to the
ganesha configuration functions.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

mgr/nfs: add awareness of the SecType option to ganesha_conf.py

This NFS-Ganesha option may be used to select kerberos integration in the
nfs client connection. If the option is not present in the configuration
dict, the option will not be present in the ganesha config blocks, just
as it was before.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

qa/cephadm: do not start upgrades from Octopus

now that we officially have v18 in main the upgrade
will fail if upgrading from Octopus (v15) to main
because they are now considered 3 major releases apart

Additionally adding more debugging to the end
of the upgrade tests as something like `ceph health detail`
can be very useful to see at the end of failed
upgrade tests

Fixes: https://tracker.ceph.com/issues/57695
Signed-off-by: Adam King <adking@redhat.com>

qa/tasks/rbd_fio: bump default to fio 3.32

fio 3.16 doesn't build against recent kernel headers due to removal of
linux/raw.h.

Fixes: https://tracker.ceph.com/issues/57766
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #44564 from zdover23/wip-doc-2022-01-13-44150-cleanup-grafana-data-source-name

doc/mgr: name data source in "Man Install & Config"

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #48362 from zdover23/wip-doc-2022-10-05-rados-ops-crush-map-edits-grammar

doc/rados: refine English in crush-map-edits.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/rados: refine English in crush-map-edits.rst

This commit makes several refinements to the English in
rados/operations/crush-map-edits.rst, which refinements were suggested
by Cole Mitchell and Anthony D'Atri in the discussion of PR#48085.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

crimson/osd/osd_operation: drop unused OSDOperationRegistry::dump_client_requests

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd: notify waiters if pool for creating pg does not exist

Fixes: https://tracker.ceph.com/issues/57740
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #48085 from zdover23/wip-doc-2022-09-14-rados-ops-crush-map-prompt-update

doc/rados: update prompts in crush-map-edits.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

Merge pull request #47458 from rzarzynski/wip-all-kickoff-r

kickoff v18 reef

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>
Reviewed-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>

librbd: add compare-and-write vector C API

Add compare and write vector C API.
Require cmp_iov and write_iov to be the same length.

Signed-off-by: Jonas Pfefferle <pepperjo@japf.ch>

Merge pull request #46037 from mohmacht/ebdplugin

blk/kernel: add plugin system for devices with compression and move VDO support into plugin

Merge pull request #48346 from zdover23/wip-doc-2022-10-03-glossary-remove-redundancies

doc/glossary.rst: remove duplicates

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/dev/release-checklists: mention LIBRBD_VER_MINOR and VERSION

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd: bump minor version to match R release

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #48218 from mchangir/qa-test_scrub_pause_and_resume_with_abort-wait-for-scrub-to-finish

qa: wait for scrub to finish

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Neeraj Pratap Singh <neesingh@redhat.com>

doc, qa/standalone/mon/misc: verify that len(monmap.features.persistent) == 10

Also updates the release checklist.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc/dev/release-checklists.rst: document vX.0.0 tagging and mark it complete.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #48041 from cbodley/wip-rgw-notify-shutdown

rgw: rados service shuts down rados client on shutdown()

Reviewed-by: Soumya Koduri <skoduri@redhat.com>
Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #48152 from Vfrost98/GHC22OSD_Ceph

mgr/telemetry: fixed log exceptions as "exception" instead of "error"

Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Yaarit Hatuka <yaarithatuka@gmail.com>

Merge pull request #48150 from li3317/ghc-osd-li3317

osd: break multi condition asserts into multiple asserts

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #47340 from kamoltat/wip-ksirivad-recreate-zilla-2104207

mon/OSDMonitor: Added extra check before mon.go_recovery_stretch_mode()

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #46284 from votdev/issue_55673_cephadm_vagrantfile

mgr/cephadm: Deploying a cluster with the Vagrantfile fails

RGWServices_Def::shutdown() shuts down the rest of the services too

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: shut down RGWDataChangesLog before other services

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: rados service shuts down rados client on shutdown()

if we leave the rados client running, it will keep delivering
AioCompletions while we're shutting other things down

this resolves a valgrind use-after-free where rgw::notify::Manager
gets completions after its destruction

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #48155 from liangmingyuanneo/wip-rgw-aync-refcount

rgw: async refcount operate in copy_obj

Reviewed-by: Anthony D Atri <anthony.datri@gmail.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #47186 from cfsnyder/wip-cfsnyder-56645

rgw: log deletion status of individual objects in multi object delete request

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

doc/glossary.rst: remove duplicates

This commit removes similar but distinct entries for the following:
   * CephFS
   * Ceph Client

Removal of a glossary term that is referred to in the body of the
documentation suite requires the alteration of the text string
that refers to the glossary term. Alterations of this kind have
been made to doc/architecture.rst and doc/rados/api/index.rst.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48292 from kotreshhr/doc-symlink-fix

doc: Fix disaster recovery doc

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

Merge pull request #46821 from batrick/i55470

qa: postgresql test suite workunit

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #48179 from batrick/i57598

qa: use correct recovery procedure for alternate meta pools

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #47795 from batrick/i57091

mds: check for some dentry damage in scrub

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #48333 from zdover23/wip-doc-2022-10-02-glossary-alphabetize

doc/glossary.rst: alphabetize glossary terms

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/glossary.rst: alphabetize glossary terms

This commit (finally) alphabetizes the terms in the glossary.
This is not a grammar-correcting or usage-correcting commit.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

rgw: async refcount operate in copy_obj

Signed-off-by: Mingyuan Liang <liangmingyuan@baidu.com>

Merge pull request #48332 from zdover23/wip-doc-2022-10-01-rados-ops-ec-prompts

doc/rados: fix prompts in erasure-code.rst

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

Merge pull request #48303 from PepperJo/cmp_write_test_bl_greater_fix

librbd: remove unnecessary write in compare-and-write test

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

doc/rados: fix prompts in erasure-code.rst

This commit adds unselectable prompts to doc/rados/
erasure-code.rst.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

crimson/common/operation: demangle names in dump

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd_admin: adapt DumpOpsInFlight to dump from all cores

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/osd_operations/peering_event: dump events as well

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/pg_shard_manager: introduce invoke_on_each_shard_seq

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/admin: capture PGShardManager for DumpPGStateHistory rather than OSD

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../peering_event: remove ifdef'd out tracking_events bits

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common/operation: record op holding OrderedExclusivePhase, add dump_detail

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common/operation: don't share op ids between types

This way, a particular id will be unique regardless of op type and we
can use it as an unambiguous identifier.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common/operation: add OpRegistryT::for_each_op

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/pg_map: fix pg_creating removal order

Simpler to fulfill the promise only after restoring the
pg_creating/pgs invariant.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common/errorator: add unthrowable_wrapper::exception_ptr()

We need a way to get a representation that can be fed to
promise::set_exception.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/replicated_backend: block _submit_transaction on sending the messages

Otherwise, the send operations can reorder.

Fixes: https://tracker.ceph.com/issues/57738
Signed-off-by: Samuel Just <sjust@redhat.com>

common/operation: make OperationT::get_registry() protected

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/shard_services: add FORWARD_TO_LOCAL_CONST helper

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #48322 from zdover23/wip-doc-2022-10-01-rados-ops-ec-front-matter-rewrite

doc/rados: rewrite EC intro

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/rados: rewrite EC intro

This PR rewrites the front matter in the "Erasure Code"
section of the RADOS documentation. Previously, the information
in this section was syntactically confused. I have also fleshed
out the distinction between erasure coding and replication.

Co-Author: Cole Mitchell <cole.mitchell@gmail.com>
Signed-off-by: Zac Dover <zac.dover@gmail.com>
Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #43567 from jdurgin/wip-gov

doc/governance: add proposed structure

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>

doc: merge new and old governance description

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

doc/governance: updates based on review comments

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

test/erasure-code: Fix signed compare warnings

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #48072 from ceph/wip-yuriw-release-17.2.4-main

doc: 17.2.4 Release Notes

doc: 17.2.4 Release Notes

Fixes: https://tracker.ceph.com/issues/57472
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
Signed-off-by: Laura Flores <lflores@redhat.com>

Merge pull request #48259 from ktdreyer/build-win-fixes

win32: support building windows client on RHEL

Reviewed-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
Reviewed-by: Justin Caratzas <jcaratza@redhat.com>

doc/governance: update based on review and CLT discussions

Clarified some parts of council that were discussed previously,
specifying the number of members and a staggered term.

Added a bit more about the steering committee - thinking the meetings
could be split between tactical (3/4 weeks) and strategic (monthly),
and still open to anyone to join the discussion, but restricted to
only members voting.

Removed the meeting section since that belongs more in a separate
place, like the ceph.io website.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

doc/governance: add proposed structure

This is Sage's summary of the Ceph leadership team discussions around
this topic. Still many details to be worked out, this is just one
concrete proposal as a basis for further discussion.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

mon: Fix snprintf truncation warning by switching to fmtlib

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

mon: Use `ssize` to get rid of signed compare warning

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

test/perf_local: Mark unused but useful variables as `maybe_unused`

Since they're used for a print that's commented out, keep them around
but silence the warning about them.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Prefer ssize or safe comparison to casts

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

test/librados: Use a call that can sleep for fractions of seconds

sleep(2) accepts an integral argument and `.01` rounds down to
zero. Use a sleep function that sleep for fractions of a second.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>