RGW | bucket notification: adding more sleeping time to prevent test from failing

Signed-off-by: Ali Masarwa <ali.saed.masarwa@gmail.com>

Merge pull request #53072 from adk3798/tcmu-custom-configs

cephadm: make custom_configs work for tcmu-runner container

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #53252 from adk3798/no-tags-upgrade-ls

mgr/cephadm: don't use image tag in orch upgrade ls

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #53297 from phlogistonjohn/jjm-logging

cephadm: move logging configuration into cephadmlib

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #53081 from kchheda3/wip-add-log

rgw/notification: enrich the expired notification log line for persistent notificatio

reviewed-by: thotz, yuvalif

Merge pull request #53009 from kchheda3/wip-bucket-stats-notification

rgw/notification: Publish notification information in bucket stats

reviewed-by: yuvalif

Merge pull request #53073 from kchheda3/wip-filter-noti-uid

rgw/notification: Filter topic list based on uid

reviewed-by: yuvalif

Merge pull request #53383 from zdover23/wip-doc-2023-09-11-glossary-ceph-client-link

doc/glossary: link to "ceph clients" from entry

Merge pull request #53246 from rhcs-dashboard/subvolumes-in-subvolumegroups

mgr/dashboard: display the groups in cephfs subvolume tab

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #53323 from rhcs-dashboard/rgw-port-error

mgr/dashboard: fix rgw port manipulation error in dashboard

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>

Merge pull request #53389 from alvinowyong/wip-doc-remove-ec-profile-warning

doc/rados/operations: Add manual CRUSH rule remove warning to erasure-code-profile.rst

Reviewed-by: Zac Dover <zac.dover@proton.me>

doc: Add warning on manual CRUSH rule removal

Add warning for "osd erasure-code-profile rm" section under rados/operations.

Signed-off-by: Alvin Owyong <70066269+alvinowyong@users.noreply.github.com>

Merge pull request #53071 from rhcs-dashboard/disable-features-rbd-edit

mgr/dashboard: images -> edit -> disable checkboxes for layering and deef-flatten

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: CephFS add groups in subvolume tab

Adds subvolume groups into the subvolume tabs in order to select the subvolumes from the appropiate group.
Also adds the capabilities to manage the subvolume groups of the subvolume in the different actions, create, edit, remove.

Fixes: https://tracker.ceph.com/issues/62675
Signed-off-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

doc/glossary: link to "ceph clients" from entry

Link to the "Ceph Clients" section of doc/architecture.rst from the
"Ceph Clients" entry in the glossary. A glossary entry should be a short
summary of the topic with which it deals, and it should direct the
reader to further and more detailed reading if the reader is interested.
This does that.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #53236 from rhcs-dashboard/subvolume-size-validator-fix

mgr/dashboard: add validator for size field in the forms

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>

Merge pull request #53371 from zdover23/wip-doc-2023-09-11-architecture-2-of-x

doc/architecture.rst - edit a sentence

doc/architecture.rst - edit a sentence

Change the sentence structure of a sentence because the verb
"experience" looked like the abstract noun "experience" when I read it
with fresh eyes. I chose the perhaps TESOL-unfriendly verb "incur", but
I believe it is right.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #53353 from zdover23/wip-doc-2023-09-10-architecture-1-of-x

doc/architecture.rst - edit up to "Cluster Map"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/architecture.rst - edit up to "Cluster Map"

Edit doc/architecture.rst up to "Cluster Map", but not including
"Cluster Map".

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #53341 from alvinowyong/wip-doc-create-osd-warning

doc/cephadm/services: Add package conflict warning to osd.rst

Merge PR #53331 into main

* refs/pull/53331/head:
Revert "Revert "Merge PR #53077 into main""

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #53334 from zdover23/wip-doc-2023-09-08-README-md-test-cluster-commands

doc: update test cluster commands in README.md

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #53335 from zdover23/wip-doc-2023-09-08-rados-config-mon-config-ref-background

doc/configuration: edit "bg" in mon-config-ref.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #52539 from Matan-B/wip-matanb-61820

ceph_mon: Fix MonitorDBStore usage

Merge pull request #52485 from ifed01/wip-ifed-fix-btree-allocator

os/bluestore: fix btree allocator

Merge pull request #52211 from rzarzynski/wip-crimson-ec-classical-refactor

osd: dissect and abstract RMWPipeline from ECBackend for sharing it with crimson

Merge pull request #52121 from cbodley/wip-blkdev-nodiscard

blkdev: fix nodiscard warning in get_device_metadata()

docs: update warning message

truncated warning message according to comments

Signed-off-by: Alvin Owyong <70066269+alvinowyong@users.noreply.github.com>

docs: add warning about potential package conflict

Add warning for "creating new osds" section under cephadm services.

Signed-off-by: Alvin Owyong <70066269+alvinowyong@users.noreply.github.com>

Merge pull request #51889 from idryomov/wip-61505

qa/suites/upgrade: enable default RBD image features

Reviewed-by: Mykola Golub <mgolub@suse.com>

doc/configuration: edit "bg" in mon-config-ref.rst

Edit the English in the section "Background" in
doc/rados/configuration/mon-config-ref.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

doc: update test cluster commands in README.md

The "rbd" pool has not been created by default since Luminous. This
commit removes a command that assumed the existence of the "rbd" pool.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Revert "Revert "Merge PR #53077 into main""

This reverts commit af4558f0ac3b0dc23a8b3cbba63b9c8acc498ea4.

Fixes: http://tracker.ceph.com/issues/62482
Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #53244 from rhcs-dashboard/fix-62665-main

mgr/dashboard: Object gateway sync status cards keeps loading when multisite is not configured

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #53239 from rhcs-dashboard/fix-62565-main

mgr/dashboard: Object gateway inventory card incorrect Buckets and user count

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #53272 from rhcs-dashboard/fix-62684-main

mgr/dashboard: replace sync progress bar with last synced timestamp in rgw multisite sync status card

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
Reviewed-by: Ali Maredia <amaredia@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #53248 from rhcs-dashboard/fix-62677-main

mgr/dashboard: rgw multisite topology view shows blank table for multisite entities

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #52866 from ivancich/wip-bad-multiparts-listing

rgw: when there are a large number of multiparts, the unorder list result may miss objects

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #51071 from ivancich/wip-versioned-reindex

rgw: reindex of objects in versioned buckets

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #52197 from imtzw/rm_op_ret

rgw: remove useless op ret assign

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #52198 from ceph/wip-nitzan-ceph-dencoder-rgw-extend-types-available

ceph-dencoder: RGW - Add missing types

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #52684 from RickyMaRui/fix-2-null-versionID

rgw: fix 2 null versionID after convert_plain_entry_to_versioned

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #53027 from ivancich/wip-add-versioning-to-stats

rgw: add versioning status during `radosgw-admin bucket stats`

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #53265 from yuvalif/wip-yuval-fix-62449

test/cls_2pc_queue: fix race condition with producers

Reviewed-by: ljflores mattbenjamin

Merge pull request #53079 from rishabh-d-dave/ior-make-failures

qa/cephfs: fix ior project build failure

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #41779 from rishabh-d-dave/fs-auth-subcmd

mon,osd,mds: allow fs authorize to update caps

Reviewed-by: Radosław Zarzyński <rzarzyns@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

submodule: update ceph-object-corpus submodule

Signed-off-by: Nitzan Mordechai <nmordech@redhat.com>

Merge pull request #53321 from rishabh-d-dave/fix-mon-fscmds

mon/FSCommands: fix variable names and function names in calls

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

mgr/dashboard: fix rgw port manipulation error in dashboard

Fixes: https://tracker.ceph.com/issues/62735
Signed-off-by: Nizamudeen A <nia@redhat.com>

ceph-dencoder: Add missing rgw types to ceph-dencoder for accurate encode-decode comparison

Currently, ceph-dencoder lacks certain rgw types, preventing us from accurately checking the ceph corpus for encode-decode mismatches.
This pull request aims to address this issue by adding the missing types to ceph-dencoder.

To successfully incorporate these types into ceph-dencoder, we need to introduce the necessary `dump` and `generate_test_instances`
functions that was missing in some types. These functions are essential for proper encode and decode of the added types.

This PR will enhance the functionality of ceph-dencoder by including the missing types, enabling a comprehensive analysis of encode-decode consistency.
With the addition of these types, we can ensure the robustness and correctness of the ceph corpus.

This update will significantly contribute to improving the overall reliability and accuracy of ceph-dencoder.
It allows for a more comprehensive assessment of the encode-decode behavior, leading to enhanced data integrity and stability within the ceph ecosystem.

Fixes: https://tracker.ceph.com/issues/61788
Signed-off-by: Nitzan Mordechai <nmordech@redhat.com>

Merge pull request #43804 from orozery/remove-crypto-image-dispatch

librbd: remove crypto image dispatch

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

src/test: fix test_bluefs.cc

BlueFS.add_block_device() doesn't accept an integer in parameters.

Introduced-by: 3cc75e8670d4fb43062ce41744262f858116f764
Fixes: https://tracker.ceph.com/issues/62734
Signed-off-by: Rishabh Dave <ridave@redhat.com>

mon/FSCommands: fix variable names and function names in calls

PR #51942 today and PR #52409 was merged last week. Latter PR changed
"fs" to "fsp" and made mds_map and fscid private and introduced
get_mds_map() and get_fsicd() methods instead.

Combination of former and latter PR introduced compilation errors & made
it impossible to build the binaries on the latest main branch. This
commit attemps to fix this issue.

Fixes: https://tracker.ceph.com/issues/62729
Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa/cephfs: CephFSTestCase.create_client() must keyring

Replace call to run_ceph_cmd() by call to get_ceph_cmd_stdout() in
method qa.tasks.cephfs.cephfs_test_case.CephFSTestCase.create_client().
run_ceph_cmd() will not return keyring which is wrong.
get_ceph_cmd_stdout() will return the stdout of "ceph auth add"
command, which is the keyring that is expected to be returned by
CephFSTestCase.create_client().

Fixes: https://tracker.ceph.com/issues/62246
Signed-off-by: Rishabh Dave <ridave@redhat.com>

doc/cephfs: add doc for fs authorize since it can now also update caps

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #53310 from ronen-fr/wip-rf-hour

osd/scrub: fixing the length of a day

Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #52519 from batrick/mds-stray-space

mds: remove stray space

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #52518 from batrick/i62057

mds: add event for batching getattr/lookup

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #52823 from athanatos/sjust/wip-62338

osd: ensure async recovery does not drop a pg below min_size

Merge pull request #50056 from rzarzynski/wip-osd-unused-msg-includes

osd: drop includes of messages that are not longer used in OSD.cc

Merge pull request #52628 from rzarzynski/wip-ec-static_encode_and_write

osd: make encode_and_write() in ECTransaction.cc static

Merge pull request #50948 from ideepika/wip-tracing-debug-quincy

src/common: add debug logs to opentelementry tracing

Merge pull request #50325 from ifed01/wip-ifed-reserved-by-bluefs

os/bluestore: make BlueFS an exclusive selector for volume reserved

Merge pull request #49872 from rzarzynski/wip-peering-state-waitdeletereserved-typo

osd: fix typo in WaitDeleteReserved's name

Merge pull request #53300 from zdover23/wip-doc-2023-09-06-rados-operations-pools-bulk-flag

doc/rados: add bulk flag to pools.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #53314 from zdover23/wip-doc-2023-09-07-man-radosgw-bucket-radoslist

doc/man: radosgw-admin.rst typo

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #51942 from trociny/wip-61599

mds: optionally forbid to use standby for another fs as last resort

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #53273 from petrutlucian94/enable_shared

win32_build.sh: use ENABLE_SHARED=ON by default

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

doc/man: radosgw-admin.rst typo

Edit the text in the "bucket radoslist" entry.

Signed-off-by: Zac Dover <zac.dover@proton.me>

doc/rados: add bulk flag to pools.rst

Add a link in pools.rst to the section in placement-groups.rst
describing the commands used to manage pools that are flagged "--bulk".

Signed-off-by: Zac Dover <zac.dover@proton.me>

rgw/notification: Publish notification information in bucket stats

Signed-off-by: kchheda3 <kchheda3@bloomberg.net>

osd/scrub: fixing the length of a day

It seems that there are 60 minutes in each hour.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #53302 from zdover23/wip-doc-2023-09-06-man-radosgw-unix-domain-sockets

doc/man: remove docs about support for unix domain sockets

mgr/dashboard: images: disable checkboxes for layering & deef-flatten

Fixes: https://tracker.ceph.com/issues/62502
Signed-off-by: avanthakkar <avanjohn@gmail.com>

win32_build.sh: use ENABLE_SHARED=ON by default

The Windows build script uses static linking by default, the
reason being that some tests were failing to build otherwise,
mostly due to unspecified dependencies.

Now that the issue was addressed, we can enable dynamic linking
by default.

Worth mentioning that the Ceph MSI build script already uses
dynamic linking.

While at it, we'll drop some duplicate defaults from
"win32_deps_build.sh". For better clarity, we'll avoid exporting
some "win32_build.sh" variables, instead passing them explicitly
to "win32_deps_build.sh".

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

doc/man: remove docs about support for unix domain sockets

doc/man: support for unix domain sockets is not implemented, hence we
removed documentation about it.

(Note: the changes in this commit were the work of Rok Jaklič in
https://github.com/ceph/ceph/pull/48537. This pull request has been
raised because that pull request was for some mysterious reason causing
merge conflicts that were never resolved.)

Co-authored-by: Rok Jaklič rjaklic@gmail.com
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #53221 from rhcs-dashboard/password-green-tick-fix

mgr/dashboard: remove green tick on old password field

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>

mgr/dashboard: add validator for size field in the forms

Fixes: https://tracker.ceph.com/issues/62655
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: Object gateway sync status cards keeps loading when multisite is not configured

Fixes: https://tracker.ceph.com/issues/62665
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

mgr/dashboard: replace sync status bar with last synced timestamp in rgw multisite sync status card

Fixes: https://tracker.ceph.com/issues/62684
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #53277 from rhcs-dashboard/image-progress-bar

mgr/dashboard: remove used and total used columns in favor of usage bar

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

cephadm: black format logging.py

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: combine common logging config elements

Move and combine some of the common logging configuration elements.
This helps highlight what is the same and what is different between
the standard and interactive logging configs.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: move logging config in cephadm.py to use new module

Move to use the logging.py module recently added to cephadmlib.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: mark types private to the new logging.py as private

The config dicts and filter type are not used outside of the logging.py
file. Add the starting underscore convention to make this clear to
readers.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: add logging.py containing a new cephadm_init_logging func

In order to make this function behave correctly in a different module
than cephadm.py, this function must be altered to accept a logger
defined in another module. All other items are mostly copy and paste
and will eventually replace those in cephadm.py.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #53052 from phlogistonjohn/cephadm-split-refactor

cephadm: refactor cephadm into multiple files

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #52924 from rishabh-d-dave/test-nfs-pr-52556

qa: inherit RunCephCmd in CephTestCase instead of CephFSTestCase

Reviewed-by: Adam King <adking@redhat.com>

rgw/notification: Filter topic list based on uid on radosgw-admin

Signed-off-by: kchheda3 <kchheda3@bloomberg.net>

Merge pull request #52956 from aclamk/wip-aclamk-testing-variable-workload

tools: Add generator of variable workload

Merge pull request #52877 from guits/bz2064498

ceph-volume/cephadm: support lv devices in inventory

librbd: remove remap_to_* and image crypto layer

This commit removes the crypto image dispatch layer.
Instead, data offset calculation is taken from ImageCtx->encryption_format.
This change makes the remap_to_* api unnecessary, so it is removed.

Signed-off-by: Or Ozeri <oro@il.ibm.com>

qa/cephfs: fix ior project build failure

1. Re-install mpich packages.
2. Use a more recent version of ior project.
3. Log contents of binary directory for easy debugging.
4. Set "-x" for bash session so commands are also logged.

Fixes: https://tracker.ceph.com/issues/61399
Signed-off-by: Rishabh Dave <ridave@redhat.com>

test/cls_2pc_queue: fix race condition with producers

this is happening when all producers are finished before the consumer
starts.

Fixes: https://tracker.ceph.com/issues/62449
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

mgr/dashboard: remove used and total used columns in favor of usage bar

Fixes: https://tracker.ceph.com/issues/62697
Signed-off-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

mgr/dashboard: remove green tick on old password field

a green tick is showing to the field where we enter the old password in
login password change form. It starts showing green tick as soon as we
start typing on it. Removing that because its misleads the user.

Fixes: https://tracker.ceph.com/issues/62644
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #53037 from batrick/i62482

qa: do not create rbd pool for CephFS QA

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Milind Changire <mchangir@redhat.com>

Merge pull request #52964 from AliMasarweh/wip-alimasa-persistant-q-enhance

RGW: added a per topic configuration to control the notification persistency
Adding the configuration of persistency per topic that will override the global settings

Reviewed-by: yuvalif <ylifshit@redhat.com>

Merge pull request #53251 from idryomov/wip-61707

librbd: make CreatePrimaryRequest remove any unlinked mirror snapshots

Reviewed-by: Ramana Raja <rraja@redhat.com>