os/bluestore: fix _extend_log seq advance

when extending the log, the sequence was left on a bad state because it would first create a transaction to update with the current seq number but leave the "real" transaction with the same sequence number which should be `extend_log_transaction.seq + 1`.

Signed-off-by: Pere Diaz Bou <pdiabou@redhat.com>

os/bluestore: test _extend_log sequence advance

Signed-off-by: Pere Diaz Bou <pdiabou@redhat.com>

Merge pull request #53389 from alvinowyong/wip-doc-remove-ec-profile-warning

doc/rados/operations: Add manual CRUSH rule remove warning to erasure-code-profile.rst

Reviewed-by: Zac Dover <zac.dover@proton.me>

doc: Add warning on manual CRUSH rule removal

Add warning for "osd erasure-code-profile rm" section under rados/operations.

Signed-off-by: Alvin Owyong <70066269+alvinowyong@users.noreply.github.com>

Merge pull request #53071 from rhcs-dashboard/disable-features-rbd-edit

mgr/dashboard: images -> edit -> disable checkboxes for layering and deef-flatten

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #53236 from rhcs-dashboard/subvolume-size-validator-fix

mgr/dashboard: add validator for size field in the forms

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>

Merge pull request #53371 from zdover23/wip-doc-2023-09-11-architecture-2-of-x

doc/architecture.rst - edit a sentence

doc/architecture.rst - edit a sentence

Change the sentence structure of a sentence because the verb
"experience" looked like the abstract noun "experience" when I read it
with fresh eyes. I chose the perhaps TESOL-unfriendly verb "incur", but
I believe it is right.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #53353 from zdover23/wip-doc-2023-09-10-architecture-1-of-x

doc/architecture.rst - edit up to "Cluster Map"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/architecture.rst - edit up to "Cluster Map"

Edit doc/architecture.rst up to "Cluster Map", but not including
"Cluster Map".

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #53341 from alvinowyong/wip-doc-create-osd-warning

doc/cephadm/services: Add package conflict warning to osd.rst

Merge PR #53331 into main

* refs/pull/53331/head:
Revert "Revert "Merge PR #53077 into main""

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #53334 from zdover23/wip-doc-2023-09-08-README-md-test-cluster-commands

doc: update test cluster commands in README.md

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #53335 from zdover23/wip-doc-2023-09-08-rados-config-mon-config-ref-background

doc/configuration: edit "bg" in mon-config-ref.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #52539 from Matan-B/wip-matanb-61820

ceph_mon: Fix MonitorDBStore usage

Merge pull request #52485 from ifed01/wip-ifed-fix-btree-allocator

os/bluestore: fix btree allocator

Merge pull request #52211 from rzarzynski/wip-crimson-ec-classical-refactor

osd: dissect and abstract RMWPipeline from ECBackend for sharing it with crimson

Merge pull request #52121 from cbodley/wip-blkdev-nodiscard

blkdev: fix nodiscard warning in get_device_metadata()

docs: update warning message

truncated warning message according to comments

Signed-off-by: Alvin Owyong <70066269+alvinowyong@users.noreply.github.com>

docs: add warning about potential package conflict

Add warning for "creating new osds" section under cephadm services.

Signed-off-by: Alvin Owyong <70066269+alvinowyong@users.noreply.github.com>

Merge pull request #51889 from idryomov/wip-61505

qa/suites/upgrade: enable default RBD image features

Reviewed-by: Mykola Golub <mgolub@suse.com>

doc/configuration: edit "bg" in mon-config-ref.rst

Edit the English in the section "Background" in
doc/rados/configuration/mon-config-ref.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

doc: update test cluster commands in README.md

The "rbd" pool has not been created by default since Luminous. This
commit removes a command that assumed the existence of the "rbd" pool.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Revert "Revert "Merge PR #53077 into main""

This reverts commit af4558f0ac3b0dc23a8b3cbba63b9c8acc498ea4.

Fixes: http://tracker.ceph.com/issues/62482
Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #53244 from rhcs-dashboard/fix-62665-main

mgr/dashboard: Object gateway sync status cards keeps loading when multisite is not configured

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #53239 from rhcs-dashboard/fix-62565-main

mgr/dashboard: Object gateway inventory card incorrect Buckets and user count

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #53272 from rhcs-dashboard/fix-62684-main

mgr/dashboard: replace sync progress bar with last synced timestamp in rgw multisite sync status card

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
Reviewed-by: Ali Maredia <amaredia@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #53248 from rhcs-dashboard/fix-62677-main

mgr/dashboard: rgw multisite topology view shows blank table for multisite entities

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #52866 from ivancich/wip-bad-multiparts-listing

rgw: when there are a large number of multiparts, the unorder list result may miss objects

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #51071 from ivancich/wip-versioned-reindex

rgw: reindex of objects in versioned buckets

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #52197 from imtzw/rm_op_ret

rgw: remove useless op ret assign

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #52198 from ceph/wip-nitzan-ceph-dencoder-rgw-extend-types-available

ceph-dencoder: RGW - Add missing types

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #52684 from RickyMaRui/fix-2-null-versionID

rgw: fix 2 null versionID after convert_plain_entry_to_versioned

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #53027 from ivancich/wip-add-versioning-to-stats

rgw: add versioning status during `radosgw-admin bucket stats`

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #53265 from yuvalif/wip-yuval-fix-62449

test/cls_2pc_queue: fix race condition with producers

Reviewed-by: ljflores mattbenjamin

Merge pull request #53079 from rishabh-d-dave/ior-make-failures

qa/cephfs: fix ior project build failure

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #41779 from rishabh-d-dave/fs-auth-subcmd

mon,osd,mds: allow fs authorize to update caps

Reviewed-by: Radosław Zarzyński <rzarzyns@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

submodule: update ceph-object-corpus submodule

Signed-off-by: Nitzan Mordechai <nmordech@redhat.com>

Merge pull request #53321 from rishabh-d-dave/fix-mon-fscmds

mon/FSCommands: fix variable names and function names in calls

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

ceph-dencoder: Add missing rgw types to ceph-dencoder for accurate encode-decode comparison

Currently, ceph-dencoder lacks certain rgw types, preventing us from accurately checking the ceph corpus for encode-decode mismatches.
This pull request aims to address this issue by adding the missing types to ceph-dencoder.

To successfully incorporate these types into ceph-dencoder, we need to introduce the necessary `dump` and `generate_test_instances`
functions that was missing in some types. These functions are essential for proper encode and decode of the added types.

This PR will enhance the functionality of ceph-dencoder by including the missing types, enabling a comprehensive analysis of encode-decode consistency.
With the addition of these types, we can ensure the robustness and correctness of the ceph corpus.

This update will significantly contribute to improving the overall reliability and accuracy of ceph-dencoder.
It allows for a more comprehensive assessment of the encode-decode behavior, leading to enhanced data integrity and stability within the ceph ecosystem.

Fixes: https://tracker.ceph.com/issues/61788
Signed-off-by: Nitzan Mordechai <nmordech@redhat.com>

Merge pull request #43804 from orozery/remove-crypto-image-dispatch

librbd: remove crypto image dispatch

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

src/test: fix test_bluefs.cc

BlueFS.add_block_device() doesn't accept an integer in parameters.

Introduced-by: 3cc75e8670d4fb43062ce41744262f858116f764
Fixes: https://tracker.ceph.com/issues/62734
Signed-off-by: Rishabh Dave <ridave@redhat.com>

mon/FSCommands: fix variable names and function names in calls

PR #51942 today and PR #52409 was merged last week. Latter PR changed
"fs" to "fsp" and made mds_map and fscid private and introduced
get_mds_map() and get_fsicd() methods instead.

Combination of former and latter PR introduced compilation errors & made
it impossible to build the binaries on the latest main branch. This
commit attemps to fix this issue.

Fixes: https://tracker.ceph.com/issues/62729
Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa/cephfs: CephFSTestCase.create_client() must keyring

Replace call to run_ceph_cmd() by call to get_ceph_cmd_stdout() in
method qa.tasks.cephfs.cephfs_test_case.CephFSTestCase.create_client().
run_ceph_cmd() will not return keyring which is wrong.
get_ceph_cmd_stdout() will return the stdout of "ceph auth add"
command, which is the keyring that is expected to be returned by
CephFSTestCase.create_client().

Fixes: https://tracker.ceph.com/issues/62246
Signed-off-by: Rishabh Dave <ridave@redhat.com>

doc/cephfs: add doc for fs authorize since it can now also update caps

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #53310 from ronen-fr/wip-rf-hour

osd/scrub: fixing the length of a day

Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #52519 from batrick/mds-stray-space

mds: remove stray space

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #52518 from batrick/i62057

mds: add event for batching getattr/lookup

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #52823 from athanatos/sjust/wip-62338

osd: ensure async recovery does not drop a pg below min_size

Merge pull request #50056 from rzarzynski/wip-osd-unused-msg-includes

osd: drop includes of messages that are not longer used in OSD.cc

Merge pull request #52628 from rzarzynski/wip-ec-static_encode_and_write

osd: make encode_and_write() in ECTransaction.cc static

Merge pull request #50948 from ideepika/wip-tracing-debug-quincy

src/common: add debug logs to opentelementry tracing

Merge pull request #50325 from ifed01/wip-ifed-reserved-by-bluefs

os/bluestore: make BlueFS an exclusive selector for volume reserved

Merge pull request #49872 from rzarzynski/wip-peering-state-waitdeletereserved-typo

osd: fix typo in WaitDeleteReserved's name

Merge pull request #53300 from zdover23/wip-doc-2023-09-06-rados-operations-pools-bulk-flag

doc/rados: add bulk flag to pools.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #53314 from zdover23/wip-doc-2023-09-07-man-radosgw-bucket-radoslist

doc/man: radosgw-admin.rst typo

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #51942 from trociny/wip-61599

mds: optionally forbid to use standby for another fs as last resort

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #53273 from petrutlucian94/enable_shared

win32_build.sh: use ENABLE_SHARED=ON by default

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

doc/man: radosgw-admin.rst typo

Edit the text in the "bucket radoslist" entry.

Signed-off-by: Zac Dover <zac.dover@proton.me>

doc/rados: add bulk flag to pools.rst

Add a link in pools.rst to the section in placement-groups.rst
describing the commands used to manage pools that are flagged "--bulk".

Signed-off-by: Zac Dover <zac.dover@proton.me>

osd/scrub: fixing the length of a day

It seems that there are 60 minutes in each hour.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #53302 from zdover23/wip-doc-2023-09-06-man-radosgw-unix-domain-sockets

doc/man: remove docs about support for unix domain sockets

mgr/dashboard: images: disable checkboxes for layering & deef-flatten

Fixes: https://tracker.ceph.com/issues/62502
Signed-off-by: avanthakkar <avanjohn@gmail.com>

win32_build.sh: use ENABLE_SHARED=ON by default

The Windows build script uses static linking by default, the
reason being that some tests were failing to build otherwise,
mostly due to unspecified dependencies.

Now that the issue was addressed, we can enable dynamic linking
by default.

Worth mentioning that the Ceph MSI build script already uses
dynamic linking.

While at it, we'll drop some duplicate defaults from
"win32_deps_build.sh". For better clarity, we'll avoid exporting
some "win32_build.sh" variables, instead passing them explicitly
to "win32_deps_build.sh".

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

doc/man: remove docs about support for unix domain sockets

doc/man: support for unix domain sockets is not implemented, hence we
removed documentation about it.

(Note: the changes in this commit were the work of Rok Jaklič in
https://github.com/ceph/ceph/pull/48537. This pull request has been
raised because that pull request was for some mysterious reason causing
merge conflicts that were never resolved.)

Co-authored-by: Rok Jaklič rjaklic@gmail.com
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #53221 from rhcs-dashboard/password-green-tick-fix

mgr/dashboard: remove green tick on old password field

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>

mgr/dashboard: add validator for size field in the forms

Fixes: https://tracker.ceph.com/issues/62655
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: Object gateway sync status cards keeps loading when multisite is not configured

Fixes: https://tracker.ceph.com/issues/62665
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

mgr/dashboard: replace sync status bar with last synced timestamp in rgw multisite sync status card

Fixes: https://tracker.ceph.com/issues/62684
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #53277 from rhcs-dashboard/image-progress-bar

mgr/dashboard: remove used and total used columns in favor of usage bar

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #53052 from phlogistonjohn/cephadm-split-refactor

cephadm: refactor cephadm into multiple files

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #52924 from rishabh-d-dave/test-nfs-pr-52556

qa: inherit RunCephCmd in CephTestCase instead of CephFSTestCase

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #52956 from aclamk/wip-aclamk-testing-variable-workload

tools: Add generator of variable workload

Merge pull request #52877 from guits/bz2064498

ceph-volume/cephadm: support lv devices in inventory

librbd: remove remap_to_* and image crypto layer

This commit removes the crypto image dispatch layer.
Instead, data offset calculation is taken from ImageCtx->encryption_format.
This change makes the remap_to_* api unnecessary, so it is removed.

Signed-off-by: Or Ozeri <oro@il.ibm.com>

qa/cephfs: fix ior project build failure

1. Re-install mpich packages.
2. Use a more recent version of ior project.
3. Log contents of binary directory for easy debugging.
4. Set "-x" for bash session so commands are also logged.

Fixes: https://tracker.ceph.com/issues/61399
Signed-off-by: Rishabh Dave <ridave@redhat.com>

test/cls_2pc_queue: fix race condition with producers

this is happening when all producers are finished before the consumer
starts.

Fixes: https://tracker.ceph.com/issues/62449
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

mgr/dashboard: remove used and total used columns in favor of usage bar

Fixes: https://tracker.ceph.com/issues/62697
Signed-off-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

mgr/dashboard: remove green tick on old password field

a green tick is showing to the field where we enter the old password in
login password change form. It starts showing green tick as soon as we
start typing on it. Removing that because its misleads the user.

Fixes: https://tracker.ceph.com/issues/62644
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #53037 from batrick/i62482

qa: do not create rbd pool for CephFS QA

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Milind Changire <mchangir@redhat.com>

Merge pull request #52964 from AliMasarweh/wip-alimasa-persistant-q-enhance

RGW: added a per topic configuration to control the notification persistency
Adding the configuration of persistency per topic that will override the global settings

Reviewed-by: yuvalif <ylifshit@redhat.com>

Merge pull request #53251 from idryomov/wip-61707

librbd: make CreatePrimaryRequest remove any unlinked mirror snapshots

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #52990 from ajarr/fix-61607

librbd: kick ExclusiveLock state machine on client being blocklisted when waiting for lock

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

RGW: added a per topic configuration to control the notification persistency

Signed-off-by: Ali Masarwa <ali.saed.masarwa@gmail.com>

Merge pull request #53264 from anthonyeleven/anthonyeleven/improve-radosgw-admin-usage

doc/man/8: improve radosgw-admin.rst

Reviewed-by: Zac Dover <zac.dover@proton.me>

doc/man/8: improve radosgw-admin.rst

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

Merge pull request #50360 from tgfree7/wip-doc-2023-03-03-multisite-migrating-a-single-site

doc/radosgw: multisite - edit "migrating a single-site"

Merge pull request #53166 from Matan-B/wip-matanb-crimson-ignore-pool-app

qa/config/crimson_qa_overrides: ignore POOL_APP_NOT_ENABLED

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: chunmei-liu <chunmei.liu@intel.com>

Merge pull request #53247 from zdover23/wip-doc-2023-09-01-readme-md-running-test-cluster

doc/README.md - improve "Running a test cluster"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #53249 from krafZLorG/fix_abc_table

doc/start: correct ABC test chart

Reviewed-by: Zac Dover <zac.dover@proton.me>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

qa: add "failover / failback loop" test for rbd-mirror

For snapshot-based mirroring, check that demote (or other mirror
snapshots) don't pile up.  Nothing in particular to assert on for
journal-based mirroring but the test is still useful.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd: make CreatePrimaryRequest remove any unlinked mirror snapshots

After commit ac552c9b4d65 ("librbd: localize snap_remove op for mirror
snapshots"), rbd-mirror daemon no longer removes mirror snapshots when
it's done syncing them -- instead it only unlinks from them.  However,
CreatePrimaryRequest state machine was not adjusted to compensate and
hence two cases were missed:

- primary demotion snapshot (rbd-mirror daemon unlinks from primary
  demotion snapshots just like it does from regular primary snapshots);
  this comes up when an image is demoted but then promoted on the same
  cluster

- non-primary demotion snapshot (unlike regular non-primary snapshots,
  non-primary demotion snapshots store peer uuids and rbd-mirror daemon
  does unlinking just like in the case of primary snapshots); this
  comes up when an image is demoted and promoted on the other cluster

Related is the case of orphan snapshots.  Since they are dummy to begin
with, CreatePrimaryRequest would now clean up the orphan snapshot after
the creation of the force promote snapshot.

Fixes: https://tracker.ceph.com/issues/61707
Co-authored-by: Christopher Hoffman <choffman@redhat.com>
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd: don't attempt to remove image state on orphan snapshots

Despite being mirror snapshots, orphan snapshots don't have image
state: see CreateNonPrimaryRequest::write_image_state() for a similar
is_orphan() check.  Attempting to remove image state generates bogus
"failed to read image state object" and "failed to remove image state"
errors.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #43217 from k0ste/fix_52656

mgr: prometheus: use nsec unit in desc for RBD _read_latency and _write_latency metrics

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #51197 from petrutlucian94/mingw-llvm

windows: use mingw-llvm

Reviewed-by: Adam Emerson <aemerson@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

doc/start: correct ABC test chart

Signed-off-by: Dmitry Kvashnin <dm.kvashnin@gmail.com>

mgr/dashboard: rgw multisite topology view shows blank table for multisite entities

Toplology view in multi-site dashboard displaying blank table regarding realm, zonegroup and zones in chrome browser

Fixes: https://tracker.ceph.com/issues/62677
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

mgr/dashboard: Object gateway inventory card incorrect Buckets and user count

Inventory(Buckets and users) data is wrong when you have buckets across multiple daemons.

Fixes: https://tracker.ceph.com/issues/62565
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

doc/README.md - improve "Running a test cluster"

Improve the English in the section "Running a test cluster in the
ceph/README.md file.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #53237 from idryomov/wip-16.2.14-notes-followup

doc: 16.2.14 release notes follow-up

Reviewed-by: Laura Flores <lflores@redhat.com>