doc/install: link to "cephadm installing ceph"

Link to "Installing Ceph" in the cephadm documentation instead of (as
was the case before this commit) to the cephadm overview page. Anyone
who clicks on the "cephadm" link in the context of the
doc/install/index.rst page is more likely to expect installation
instructions than to expect an explanation of what cephadm is.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #49734 from linuxkidd/wip-rgw-gap-orphan-tool-error-checks

rgw/rgw-orphan-list: refactor and add more checks to the tool

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

rgw/rgw-orphan-list: refactor / add error checking

standardize log output
add check that supplied pool[s] exist on the cluster
add return code checks for all commands that can affect output
add check for empty intermediate files
add usage output, including a -h command line option
add command line option to specify temporary directory to use
  ( instead of /tmp )
require command line supplied pools to be encased in double quotes "
  when multiple pools are provided
add warning for indexless buckets
move major segments to separate functions

Signed-off-by: Michael J. Kidd <linuxkidd@gmail.com>

Merge pull request #48632 from adamemerson/wip-57562

rgw: Fix race condition on FIFO client on new head creation

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Krunal Chheda <kchheda3@bloomberg.net>
Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #47794 from rhcs-dashboard/auth-add

mgr/dashboard: create authx users

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #49765 from rhcs-dashboard/fix-bucket-encryption-checkbox

mgr/dashboard: fix bucket encryption checkbox

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #48729 from ivancich/wip-flight-select-add-flight

rgw: initial commit adding Arrow Flight functionality

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #48509 from jliebl-git/jliebl-ldap-fix

rgw: LDAP fix resource leak with wrong credentials

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #49636 from liangmingyuanneo/wip-read-olh-log

rgw: remove guard_reshard in bucket_index_read_olh_log

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #49463 from ivancich/wip-log-follow-bi-list

rgw: add logging to radosgw-admin bi list

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #49648 from ivancich/wip-fix-fp-error

rgw: fix FP error when calculating enteries per bi shard

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #49181 from ljflores/wip-envlibrados-rocksdb-fix

qa/workunits/rados: skip running envlibrados rocksdb tests on ubuntu

qa/workunits/rados: skip running envlibrados rocksdb tests on ubuntu

This test passes on centos and rhel, but fails on ubuntu from an
invalid pointer. Since the envlibrados rocksdb tests are experimental
and don't have any actual users, we can just run them on rhel and
centos.

At the moment, the actual bug is not fully understood, but it was
decided that fixing it is low priority, and removing the test from
problematic distros is okay for the time being. This commit
is considered a workaround to the actual issue.

Related tracker: https://tracker.ceph.com/issues/57632
Signed-off-by: Laura Flores <lflores@redhat.com>

mgr/dashboard: fix bucket encryption checkbox

Fixes: https://tracker.ceph.com/issues/58474
The encryption checkbox in the bucket creation form remains disabled after setting the vault authentication method as agent.

Signed-off-by: Aashish Sharma <aasharma@redhat.com>

mgr/dashboard: create authx users

Signed-off-by: Pere Diaz Bou <pdiazbou@redhat.com>
Signed-off-by: Nizamudeen A <nia@redhat.com>
Co-authored-by: Nizamudeen A <nia@redhat.com>

Merge pull request #49533 from mkogan1/wip-fips-openssl3

rgw: under fips & openssl 3.x allow md5 iusage in select rgw ops

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #49710 from cyx1231st/wip-seastore-cleanup-overwrite

crimson/os/seastore/object_data_handler: cleanup the overwrite path

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #49743 from xxhdx1985126/wip-omap-fixes

crimson/os/seastore/omap_manager: minor bugs fixes

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson/seastore/test_omap_manager: keep test_omap_mappings consistent with "rm_key_range" config

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore/omap_manager: fix a capture-by-reference related issue

Variables that are not guaranteed to be available as long as the continuations
live shouldn't be captured by reference

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #49725 from anthonyeleven/56521

doc/releases: Clarify Quincy CephFS MDS upgrade process

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #49747 from zdover23/wip-doc-2023-01-16-luks-versions

doc/ceph-volume: update LUKS docs

crimson/os/seastore: add FIXME to consider pending extent with absolute paddr

During object data overwrite.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/object_data_handler: cleanup the overwrite path

* implement factory and type for extent_to_write_t.
* refactor overwrite_plan_t with explicit size calculations and
  defined operations.
* cleanup the overwrite path accordingly.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #49716 from zdover23/wip-doc-2023-01-12-rbd-iscsi-initiator-linux-prompts

doc/rbd: refine iscsi-initiator-linux.rbd

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/ceph-volume: update LUKS docs

Remove references that claim that Ceph uses only LUKS version 1.

https://tracker.ceph.com/issues/58354

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #49745 from zdover23/wip-doc-2023-01-15-start-documenting-ceph-escaping-bold-characters-within-words

doc/start: add RST escape character rules for bold

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/rbd: format iscsi-initiator-linux.rbd better

Add prompts and clean up the lists in doc/rbd/iscsi-initiator-linux.rbd.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

doc/start: add RST escape character rules for bold

Explain how to escape the bold notation (**) within words in RST.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #49736 from zdover23/wip-doc-2023-01-14-dev-basic-workflow-git-commands

doc/dev: add git branch management commands

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/dev: add git branch management commands

Add git branch deleting and search commands to the "Basic Workflow" page
of the Developer Guide.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48496 from adk3798/apply-spec-parsing

cephadm: only pull host info from applied spec, don't try to parse yaml

Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #49024 from rkachach/fix_issue_58067

mgr/cephadm: fixing mgr port shown in ceph orch ps

Reviewed-by: Adam King <adking@redhat.com>

Merge PR #49731 into main

* refs/pull/49731/head:
tools/cephfs: fix traceback from missing envvar

Reviewed-by: Milind Changire <mchangir@redhat.com>

Merge pull request #48617 from adk3798/upgrade-status-help

mgr/orchestrator: fix upgrade status help message

Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #48384 from jmolmo/Rook_PDFetcher

mgr/rook: Device inventory

Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Travis Nielsen <tnielsen@redhat.com>

Merge pull request #48592 from adk3798/offline-upgrade

mgr/cephadm: improve offline host handling, mostly around upgrade

Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #48574 from adk3798/debug-log-spam

mgr/cephadm: make logging refresh metadata to debug logs configurable

Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #47383 from rhcs-dashboard/ceph-exporter-cephadm-changes

exporter: cephadm changes

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #49389 from adk3798/git-archive-fix

qa/tasks/cephadm.py: fix pulling cephadm from git.ceph.com

Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #49726 from rhcs-dashboard/cephadm-e2e-fixes

mgr/dashboard: cephadm dashboard e2e fixes

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #49635 from rkachach/fix_issue_58378

mgr/mgr_util: fixing some tox issues

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

mgr/dashboard: cephadm dashboard e2e fixes

Fixes: https://tracker.ceph.com/issues/58406
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #49417 from athanatos/sjust/backburner/wip-smp-option

crimson/osd/main: get full config before starting reactor

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

tools/cephfs: fix traceback from missing envvar

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

rgw: implement initial flight server functionality

Implements the ability for a flight to be created when the object is
retrieved by an S3 get.

Adds FlightServer abilities ListFlights, GetFlightInfo, GetSchema, and
DoGet. Adds an interface for a store for flight information and adds
an in-memory implemtation of it.

This code is functionality is early-stage and lacks some planned
efficiencies.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

doc/releases: Clarify Quincy CephFS MDS upgrade process

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

Merge pull request #49713 from rhcs-dashboard/fix-sse-daemon-name

mgr/dashboard: rgw server side encryption daemon name fix

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #49540 from xxhdx1985126/wip-rebuild-missing-set

crimson/osd/pg: rebuild missing set when a new interval is created

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #48575 from liu-chunmei/multicore-cyanstore

crimson/os: cyanstore support multicore

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #49712 from tchaikov/wip-packaging-in-setuptools

mgr/prometheus: use vendored "packaging" instead

Merge pull request #49719 from zdover23/wip-doc-2023-01-12-dev-developer-guide-basic-workflow-miscellaneous-reorganization

doc/dev: add push.autoSetupRemote instructions

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/dev: add push.autoSetupRemote instructions

Explain how to set up push.autoSetupRemote so that the origin branch is
properly set up when running "get push" without the need for the
"--set-upstream" option.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48756 from Matan-B/wip-matanb-rollback-snap

crimson/osd: Support snapshot rollbacks

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #49568 from Matan-B/wip-matanb-crimson-modified-ranges

crimson/osd/*recovery_backend: Rename Push/Pull Info/Op

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: chunmei-liu <chunmei.liu@intel.com>

Merge pull request #49715 from zdover23/wip-doc-2023-01-12-cephadm-host-management-osd-capitalization

doc/cephadm: s/osd/OSD/ where appropriate

Merge PR #49711 into main

* refs/pull/49711/head:
ceph.spec.in: remove the bcond_without cephfs_java option

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

doc/cephadm: s/osd/OSD/ where appropriate

Capitalize the initialization "OSD" where it occurs in natural language
in cephadm/host-management.rst. This PR answers a request made by
Anthony D'Atri and seconded by Cole Mitchell in https://github.com/ceph/ceph/pull/49699#discussion_r1066171002.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

mgr/prometheus: use vendored "packaging" instead

instead of using the top-level "packaging" module, use the one
vendored by setuptools.

packaging python module provides versioning defined by PEP-440.
but python3-packaging is provided by CentOS8 powertools repo,
which is not enabled by default. and in CentOS9, this package
is provided by AppStream instead of BaseOS.

as prometheus mgr module is included by ceph-mgr-module-core,
it would be desirable if our user can install ceph-mgr-module-core
without enabling powertools or AppStream repo on a CentOS or
its derivative distros.

fortunately, setuptools vendors packaging module. and both
CentOS8 and CentOS9 provide python3-setuptools in their BaseOS
repos.

in this change, instead of using "packging" module, we use the
venderored one, which is in turn embedded in pkg_resources.
this python module is provided by python3-setuptools on CentOS
distros, and python3-pkg-resources on Debian and its derivatives.

the packaging recipes are updated accordingly to reflect the
new runtime dependency.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

mgr/dashboard: rgw server side encryption daemon name fix

The config values for the server side encryption currently are wrongly set to the mon daemon by default. This PR intends to fix this issue

Fixes: https://tracker.ceph.com/issues/58419
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #49686 from xxhdx1985126/wip-crimson-dump-metrics

crimson/admin: correct metrics dumping methods

Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/os: make cyanstore not use proxy

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

crimson/os: make load_pgs() load pg on proper core

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

crimson/os: support multicores for cyanstore

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

ceph.spec.in: remove the bcond_without cephfs_java option

This effectively disable cephfs_java for rpm builds for now.
The only user appears to be cephfs jni bindings.

Fixes: https://tracker.ceph.com/issues/58382
Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/admin: correct metrics dumping methods

First level object has no key, so we have to inclose metric objects with
an outer one

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #49699 from zdover23/wip-doc-2023-01-11-cephadm-host-management-removing-hosts

doc/cephadm: refine "Removing Hosts"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #49703 from zdover23/wip-doc-2023-01-11-rados-operations-move-colon

doc/rados: move colon

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

Merge pull request #49700 from zdover23/wip-doc-2023-01-11-css-h3-padding

doc/css: add top-bar padding for h3 html element

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/rados: move colon

Move colon in add-or-rm-osds.rst so that the sentence reads properly.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

doc/css: add top-bar padding for h3 html element

Add "scroll-margin-top: 4em;" property to h3 html element.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

doc/cephadm: refine "Removing Hosts"

An intended edit to remove a redundant indefinite article became a
longer (but still brief) full editorial pass.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #49692 from zdover23/wip-doc-2023-01-11-header-fix

doc/css: add "span" padding to custom.css

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #49690 from idryomov/wip-58398

doc/man/ceph-rbdnamer: remove obsolete udev rule

Reviewed-by: Christopher Hoffman <choffman@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

doc/css: add "span" padding to custom.css

Add "scroll-top-bar: 2em;" for the "span" html element in custom.css so
that the top bar doesn't get in the way of headings bounded by the "span
element".

See also https://github.com/ceph/ceph/pull/49644.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

doc/man/ceph-rbdnamer: remove obsolete udev rule

Fixes: https://tracker.ceph.com/issues/58398
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge PR #49630 into main

* refs/pull/49630/head:
doc: document the relevance of mds_namespace mount option

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

doc: document the relevance of mds_namespace mount option

Fixes: https://tracker.ceph.com/issues/57673
Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #49664 from batrick/first-damage-repair-nosnap

tools/cephfs/first-damage: repair specific first=CEPH_NOSNAP damage

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #49677 from zdover23/wip-doc-2023-01-10-rados-replacing-an-osd

doc/rados: link to cephadm replacing osd section

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

cls/fifo: Retry on push to nonexistent part

A racing client may delete the part we're trying to push to. Use
`assert_exists()` to check, and re-read metadata if we receive
-ENOENT.

Fixes: https://tracker.ceph.com/issues/57562
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

cls/fifo: Fix race condition on `_prepare_new_head`

First, make `_prepare_new_head` take the new head part number, so two
calls racing from the same push will attempt to create the same head.

Also remove the neorados FIFO since it doesn't have all the bug fixes
in the legacy version and will be rewritten in terms of `async_compose`
anyway.

Fixes: https://tracker.ceph.com/issues/57562
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

cls/fifo: Journal is flat_set, not multimap

We don't really need the overhead and complexity of a multimap.

Fixes: https://tracker.ceph.com/issues/57562
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

doc/rados: link to cephadm replacing osd section

Direct readers to the "Replacing an OSD" section in the cephadm
documentation, for cases in which the instructions in "Replacing an OSD"
in the RADOS documentation don't work.

https://tracker.ceph.com/issues/58401

Signed-off-by: Zac Dover <zac.dover@gmail.com>

cls/fifo: Move version inc into `apply_update()`

Only increment the version if we make an actual change.

Return whether we have changed something or not so the OSD side can
skip writing if there's no change.

Fixes: https://tracker.ceph.com/issues/57562
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

cls/fifo: Merge duplicate journal entries

Since we no longer use tags, journal entries are just an operation and
a part number. If an entry being added is already in the journal, skip
it.

Fixes: https://tracker.ceph.com/issues/57562
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

cls/fifo: Use spaceship operator to implement comparisons

Except for the version type, where only versions with the same tag are
comparable.

Fixes: https://tracker.ceph.com/issues/57562
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

cls/fifo: Remove all use of part tags

Part tags make part creation and setting the head non-idempotent,
leading to issues where racing RGWs may get confused about the correct
tag for a part. (Or worse, potentially have the metadata header hold
different value for a part than the part's header.)

Consistently only requires that all nodes agree on the number.

Fixes: https://tracker.ceph.com/issues/57562
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

cls/fifo: Fix `same_or_later()` comparison

Fix logic error where disjunction was used instead of conjunction.

Fixes: https://tracker.ceph.com/issues/57562
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #48696 from adk3798/agent-testing-binary

cephadm: unit tests for CephadmAgent, MgrListener, AgentGatherer

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #49495 from BryceCao/wip-fix-result-when-users-topics-is-null

rgw: fix result when user's topics is null

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #49629 from guits/cv-fix-tox-allowlist_externals

ceph-volume/tests: add allowlist_externals to tox.ini

Merge pull request #49665 from zdover23/wip-doc-2023-01-08-glossary-ceph-object-storage

doc/glossary: Clean up "Ceph Object Storage"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/glossary: Clean up "Ceph Object Storage"

Remove redundant material under the "Ceph Object Storage" headword and
add a "See 'Ceph Object Store'" link. A future PR will provide a couple
of sentences that explain how object storage is what's really supporting
both CephFS and RBD.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

tools/cephfs/first-damage: repair specific first=CEPH_NOSNAP damage

first==CEPH_NOSNAP is never valid. Apparently somehow the MDS may set
this in some code path yet to be discovered, even without snapshots.

Note, we probably should not necessarily automate this fix via the MDS
because we do not know if this dentry (visible in HEAD) should also be
visible before NEXT_SNAP.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

tools/cephfs/first-damage.py: load CEPH_CONF

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #49663 from bradfitz/patch-1

doc: fix a typo

doc: fix a typo

Signed-off-by: Brad Fitzpatrick <brad@danga.com>

Merge pull request #49658 from zdover23/wip-doc-2023-01-15-h2-scroll-margin-top

doc/css: Add scroll-margin-top to h2 html element

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #49653 from zdover23/wip-doc-2023-01-06-crushtool-options

doc/man: define --num-rep, --min-rep and --max-rep

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Laura Flores <lflores@redhat.com>