cls/fifo: Retry on push to nonexistent part

A racing client may delete the part we're trying to push to. Use
`assert_exists()` to check, and re-read metadata if we receive
-ENOENT.

Fixes: https://tracker.ceph.com/issues/57562
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

cls/fifo: Fix race condition on `_prepare_new_head`

First, make `_prepare_new_head` take the new head part number, so two
calls racing from the same push will attempt to create the same head.

Also remove the neorados FIFO since it doesn't have all the bug fixes
in the legacy version and will be rewritten in terms of `async_compose`
anyway.

Fixes: https://tracker.ceph.com/issues/57562
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

cls/fifo: Journal is flat_set, not multimap

We don't really need the overhead and complexity of a multimap.

Fixes: https://tracker.ceph.com/issues/57562
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

cls/fifo: Move version inc into `apply_update()`

Only increment the version if we make an actual change.

Return whether we have changed something or not so the OSD side can
skip writing if there's no change.

Fixes: https://tracker.ceph.com/issues/57562
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

cls/fifo: Merge duplicate journal entries

Since we no longer use tags, journal entries are just an operation and
a part number. If an entry being added is already in the journal, skip
it.

Fixes: https://tracker.ceph.com/issues/57562
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

cls/fifo: Use spaceship operator to implement comparisons

Except for the version type, where only versions with the same tag are
comparable.

Fixes: https://tracker.ceph.com/issues/57562
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

cls/fifo: Remove all use of part tags

Part tags make part creation and setting the head non-idempotent,
leading to issues where racing RGWs may get confused about the correct
tag for a part. (Or worse, potentially have the metadata header hold
different value for a part than the part's header.)

Consistently only requires that all nodes agree on the number.

Fixes: https://tracker.ceph.com/issues/57562
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

cls/fifo: Fix `same_or_later()` comparison

Fix logic error where disjunction was used instead of conjunction.

Fixes: https://tracker.ceph.com/issues/57562
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #48696 from adk3798/agent-testing-binary

cephadm: unit tests for CephadmAgent, MgrListener, AgentGatherer

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #49495 from BryceCao/wip-fix-result-when-users-topics-is-null

rgw: fix result when user's topics is null

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #49629 from guits/cv-fix-tox-allowlist_externals

ceph-volume/tests: add allowlist_externals to tox.ini

Merge pull request #49665 from zdover23/wip-doc-2023-01-08-glossary-ceph-object-storage

doc/glossary: Clean up "Ceph Object Storage"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/glossary: Clean up "Ceph Object Storage"

Remove redundant material under the "Ceph Object Storage" headword and
add a "See 'Ceph Object Store'" link. A future PR will provide a couple
of sentences that explain how object storage is what's really supporting
both CephFS and RBD.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #49663 from bradfitz/patch-1

doc: fix a typo

doc: fix a typo

Signed-off-by: Brad Fitzpatrick <brad@danga.com>

Merge pull request #49658 from zdover23/wip-doc-2023-01-15-h2-scroll-margin-top

doc/css: Add scroll-margin-top to h2 html element

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #49653 from zdover23/wip-doc-2023-01-06-crushtool-options

doc/man: define --num-rep, --min-rep and --max-rep

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #49657 from dang/wip-dang-motr-build

Fix MOTRStore build

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc/css: Add scroll-margin-top to h2 html element

Add "scroll-margin-top: 4em;" to the h2 html element's definition in
custom.css. This moves the text under all h2 html elements out of the
way of the sticky-header-style top bar, which previously obscured the
text.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #49643 from zdover23/wip-doc-2023-01-05-add-padding-to-sections

doc/_static: add scroll-margin-top to custom.css

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/man: define --num-rep, --min-rep and --max-rep

Explain the "--num-rep", "--min-rep", and "--max-rep" options, which are
required when running "crushtool" commands with the "--show-mappings"
flag. Originally reported by Brad Fitzpatrick.

https://tracker.ceph.com/issues/58374

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #49373 from inspur-wyq/wip-doc-rgwtar

src/rgw/rgw_tar.h: fix typos

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #49631 from petrutlucian94/rbd_py_test2

qa: test_rbd_wnbd.py - disable PS progress bar

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #49651 from Thingee/update-foundation-20230105

doc: Updating Foundation members for January 2023

doc: Updating Foundation members for January 2023

Signed-off-by: Mike Perez <thingee@gmail.com>

Fix MOTRStore build

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

doc/_static: add scroll-margin-top to custom.css

Add 4em of padding to the class "section", so that linked-to
destinations are not obscured by the top bar.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #49500 from BryceCao/wip-fix-result-when-users-mfas-is-null

rgw: fix result when users mfas is null

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #49637 from zdover23/wip-doc-2023-01-05-top-bar-scroll-margin-top-repair

doc/css: add scroll-margin-top to dt elements

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/css: add scroll-margin-top to dt elements

add "scroll-margin-top: em3;" to custom.css so that the header bar
doesn't obscure the text of headwords in glossary.rst. Note that this
applies only to elements in the documentation that are rendered into
HTML with the dt (which stands for "description term" or "description
list") tag. Other modifications will be necessary in order to ensure
that the anchor points of non-dt elements are not obscured by the header
bar.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

qa: test_rbd_wnbd.py - disable PS progress bar

This test uses certain PS commands that attempt to display
a progress bar. However, this can cause issues when invoked
remotely (e.g. by the jenkins job).

For this reason, we're defining a helper (ps_execute) that runs
PS commands, disabling the progress bars and enabling the non
interactive mode.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Merge pull request #48730 from dang/wip-dang-zipper-chown

Fix chown to unlink

Reviewed-by: Casey Bodley <cbodley@redhat.com>

ceph-volume/tests: add allowlist_externals to tox.ini

typical failure seen in the CI:

```
py3-flake8: install_deps> ./tox_install_command.sh flake8
py3-flake8: failed with ./tox_install_command.sh (resolves to ./tox_install_command.sh) is not allowed, use allowlist_externals to allow it
```

Fixes: https://tracker.ceph.com/issues/58377
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #49618 from petrutlucian94/rbd_py_test2

qa: retry test_rbd_wnbd.py fs operations

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

qa: retry test_rbd_wnbd.py fs operations

Certain FS related operations can fail, especially under load
(e.g. initializing partitions, volume formatting, etc).

For this reason, we're going to introduce some retries.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Merge pull request #49620 from bradfitz/bradfitz/doc

doc: fix a couple grammatical things

Reviewed-by: Zac Dover <zac.dover@gmail.com>

doc: fix a couple grammatical things

Signed-off-by: Brad Fitzpatrick <brad@danga.com>

Merge pull request #49613 from zdover23/wip-doc-2023-01-02-start-documenting-ceph-49609-followup

doc/start: add Anthony D'Atri's suggestions

doc/start: add Anthony D'Atri's suggestions

Add the suggestions made by Anthony D'Atri in
https://github.com/ceph/ceph/pull/49609.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #49609 from zdover23/wip-doc-2023-01-02-start-documenting-ceph-cleanup

doc/start: refine "Quirks of RST"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/start: refine "Quirks of RST"

Refine the language that was added yesterday, language that explains how
certain aspects of RST work.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #49606 from zdover23/wip-doc-2023-01-01-start-documenting-ceph-quirks-of-rst

doc/start: add link-related metadocumentation

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/start: add link-related metadocumentation

Add two kinds of link-related metadocumentation (documentation about how
to write documentation) to the "Documenting Ceph" section of the "Intro
to Ceph" document: 1. metadocumentation about external links, and 2.
metadocumentation about internal links.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #49602 from zdover23/wip-doc-2022-12-31-glossary-DAS-capitalization

doc/glossary: capitalize "DAS" correctly

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/glossary: capitalize "DAS" correctly

Correctly capitalize "Direct-Attached Storage" in the glossary. (And
test the "Quincy" branch, which seems lately not to have picked up any
docs backports.)

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #49599 from zdover23/wip-doc-2022-12-30-glossary-releases

doc/glossary: collate "releases" entries

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/glossary: collate "releases" entries

Collect the "Releases"-related entries together under the "Releases"
headword, in order to give readers a sense at a glance of how the
different kinds of releases relate to one another.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #49488 from zdover23/wip-doc-2022-12-18-rbd-rbd-exclusive-locks-refine

doc/rbd: refine rbd-exclusive-locks.rst

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

doc/rbd: refine rbd-exclusive-locks.rst

Refine grammar (mostly semantics) in rbd-exclusive-locks.rst.

Co-authored-by: Ilya Dryomov <idryomov@redhat.com>
Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #49593 from zdover23/wip-doc-2022-12-29-glossary-osd-id-each

doc/glossary: s/an/each/ where it's needed

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/glossary: s/an/each/ where it's needed

s/an/each/ in accordance with the suggestion made by Anthony D'Atri
here: https://github.com/ceph/ceph/pull/49590/files#r1058390357

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #49584 from zdover23/wip-doc-2022-12-27-glossary-cleanup

doc/glossary: clean OSD id-related entries

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #49587 from zdover23/wip-doc-2022-12-27-rbd-wuold-typo

doc/rbd: s/wuold/would/ in rados-rbd-cmds.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #49583 from RanjiniMandyam/RMN-2151232-syntax

doc/dev/cephadm: fix host maintenance enter/exit syntax -

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rbd: s/wuold/would/ in rados-rbd-cmds.rst

s/wuold/would/ in rados-rbd-cmds.rst.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

doc/glossary: clean OSD id-related entries

Tidy up the sentences under the headwords "OSD fsid", "OSD id", and "OSD
uuid".

Signed-off-by: Zac Dover <zac.dover@gmail.com>

doc/dev/cephadm: fix host maintenance enter/exit syntax -

Signed-off-by: Ranjini Mandyam Narasiodeyar <rmandyam@rmandyam.remote.csb>

Merge pull request #48893 from weixinwei/master

librbd: avoid EUCLEAN error after "rbd rm" is interrupted

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #49261 from ajarr/fix-57726

mgr/rbd_support: remove localized schedule option during module startup

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #49573 from zdover23/wip-doc-2022-12-26-glossary-cleanup

doc/glossary: disambiguate clauses

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/glossary: disambiguate clauses

Disambiguate various clauses, most of which contain forms of the verb
"to require".

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #49569 from zdover23/wip-doc-2022-12-26-start-documenting-ceph-backport-fix

doc/start: update documenting-ceph.rst

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

Merge pull request #49567 from zdover23/wip-doc-2022-12-25-glossary-cleanup

doc/glossary: improve glossary entries

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/start: update documenting-ceph.rst

Add an article to documenting-ceph.rst in order to troubleshoot a
backporting problem.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

doc/glossary: improve glossary entries

Improve multiple glossary entries.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #49564 from zdover23/wip-doc-2022-12-25-start-documenting-ceph

doc/start: improve documenting-ceph.rst

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/start: improve documenting-ceph.rst

Improve the grammar and the pragmatics of the "TOC and Hyperlinks"
section of start/documenting-ceph.rst.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #49558 from zdover23/wip-doc-2022-12-24-rados-api-librados-intro-prompt-removal

doc/rados: remove prompt from php.ini line

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

Merge pull request #49557 from zdover23/wip-doc-2022-12-24-rados-api-python-typo

doc/rados: correct typo in python.rst

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/rados: remove prompt from php.ini line

Remove an unselectable prompt from the beginning of a line meant to be
included in the "php.ini" file.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

doc/rados: correct typo in python.rst

s/you Ceph configuration file/your Ceph configuration file/

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #49532 from petrutlucian94/rbd_py_test2

qa: fix test_rbd_wnbd.py, properly retrieving the drive letter

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

qa: add some retries to test_rbd_wnbd.py

The following operations may fail right after a block device
is attached:

* retrieving the disk number (can return -1)
* opening the disk
* setting the disk online or writable

For this reason, we'll need to add some retries. For convenience,
we're moving the existing retry logic to a separate decorator.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

qa: fix test_rbd_wnbd.py, properly retrieving the drive letter

Instead of trying to use the first partiton which may be reserved
by Windows, we'll fetch the first non-empty drive letter from
the disk that we've just mounted.

While at it, we're ensuring that the drive letter is actually a
letter and not a null character, which the Powershell command
returns in case of empty drive letters.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Merge pull request #49547 from idryomov/wip-rbd-qemu-xfstests-curl

qa: switch to curl for qemu-xfstests

Reviewed-by: Christopher Hoffman <choffman@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #49550 from zdover23/wip-doc-2022-12-23-rados-api-librados-intro-prompts

doc/rados: add prompts to librados-intro.rst

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: add prompts to librados-intro.rst

Add unselectable prompts to doc/rados/api/librados-intro.rst.

https://tracker.ceph.com/issues/57108

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge PR #49406 into main

* refs/pull/49406/head:
mds: fix scan_stray_dir not reset next.frag on each run of stray inode

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #49172 into main

* refs/pull/49172/head:
qa/test_nfs: added testcase test_non_existent_cluster
mgr/nfs: handle bad cluster name during info command

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #48907 into main

* refs/pull/48907/head:
test: verify that snapshot items are accounted when deciding to split/merge a directory
mds: account for snapshot items when deciding to split or merge a directory

Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>

Merge pull request #49259 from kamoltat/wip-ksirivad-fix-58155

mon/Elector.cc Added additional prank >= ranks_size sanity check
Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

qa: switch to curl for qemu-xfstests

This is a follow-up for commit 631899ffeb84 ("qa: switch back to git
protocol for qemu-xfstests"), needed for the same "ancient execution
environment" reason.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge PR #48281 into main

* refs/pull/48281/head:
mds: md_log_replay thread blocks waiting to be woken up

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #48822 into main

* refs/pull/48822/head:
mds: switch submit_mutex to fair mutex for MDLog

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

test: verify that snapshot items are accounted when deciding to split/merge a directory

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mds: account for snapshot items when deciding to split or merge a directory

Its easy to "overload" a directory object with large number of omap entries by
doing the following (one shot or over and over again):

      - touch dir/file{0..11000} ; create 11000 files (> mds_bal_split_size)
      - mkdir dir/.snap/snap_a
      - rm -f dir/file{0..11000}

End result - the directory object would have 11000 omap entries since the MDS
does not fragment directory snapshots. If the number of such entries exceed
`osd_deep_scrub_large_omap_object_key_threshold` (default: 200000), a cluster
health warning is generated:

       Large Omap objects found in pool...

CDir::should_merge() does not take into account COW'd inodes and the frags
get merged.

Fixes: http://tracker.ceph.com/issues/55215
Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge PR #49098 into main

* refs/pull/49098/head:
client/fuse: add support for 3.12 API of libfuse
client/fuse: set max_idle_threads to the correct value

Reviewed-by: Venky Shankar <vshankar@redhat.com>

qa/test_nfs: added testcase test_non_existent_cluster

Fixes: https://tracker.ceph.com/issues/58138
Signed-off-by: Dhairya Parmar <dparmar@redhat.com>

mgr/nfs: handle bad cluster name during info command

It should say the cluster does not exist

Fixes: https://tracker.ceph.com/issues/58138
Signed-off-by: Dhairya Parmar <dparmar@redhat.com>

Merge PR #49423 into main

* refs/pull/49423/head:
qa: remove qa/suites/marginal and restart task

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #49534 from idryomov/wip-rbd-qemu-xfstests-patchup

qa: switch back to git protocol for qemu-xfstests

Reviewed-by: Christopher Hoffman <choffman@redhat.com>

Merge pull request #48794 from dang/wip-dang-bucket-policy

RGW - Make sure PostObj set bucket on s->object

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Ali Maredia <amaredia@redhat.com>

qa: switch back to git protocol for qemu-xfstests

As noted in commit 89177d65988c ("qa: switch to https protocol for
repos' server"), git.ceph.com mirror doesn't make git:// available
anymore.  However, run_xfstests-obsolete.sh has "obsolete" in its
name for a reason -- due to an ancient execution environment, git://
is the only viable option:

  $ git clone https://git.ceph.com/xfstests-dev.git
  Cloning into 'xfstests-dev'...
  error: gnutls_handshake() failed: A TLS fatal alert has been received. while accessing https://git.ceph.com/xfstests-dev.git/info/refs
  fatal: HTTP request failed

Ditch git.ceph.com mirror for now.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #49529 from zdover23/wip-doc-2022-12-21-rados-bluestore-config-ref-prompts

doc/rados: add prompts to bluestore-config-ref.rst

Merge pull request #49403 from tchaikov/cmake-boost

cmake: pass cflags using compileflags in BuildBoost.cmake

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #49305 from petrutlucian94/dokan_subsys

dokan: use the right logging subsystem

Merge pull request #48929 from petrutlucian94/rbd_py_test

qa: fix test_rbd_wnbd.py

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

doc/rados: add prompts to bluestore-config-ref.rst

Add unselectable prompts to
doc/rados/configuration/bluestore-config-ref.rst.

https://tracker.ceph.com/issues/57108

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #49497 from BryceCao/wip-fix-mfa.rst-spelling-mistake

doc/radosgw: fix spelling mistake in mfa.rst

rgw: fix result when user's mfas is null

Fixes: https://tracker.ceph.com/issues/58308
Signed-off-by: caolei <cao.leilc@inspur.com>