Merge pull request #50684 from yuvalif/wip-yuval-fix-59136

rgw/notifications: support bucket notification with bucket policy

Merge pull request #45693 from cbodley/wip-qa-rgw-scrub

qa/rgw: reenable wait-for-scrub

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #47649 into main

* refs/pull/47649/head:
mds: adjust MDSRank::command_tag_path invocation of enqueue_scrub()
doc/scrub: documented stray evaluation using recursive scrub
qa: added testcases
mds: make `scrub status` print flag `scrub_mdsdir`
mds: add scrub_mdsdir to ScrubHeader
mds: do not dump multiple JSON obj
mds: evaluate strays while performing scrub on root path
mds: remove inode from scrub_stack if being purged
mds: do not scrub inode if it is purging

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #50053 into main

* refs/pull/50053/head:
libcephfs: move ClearSetuid to suidsgid.cc
libcephfs: add test cases for dropping the suid/sgid in write/truncate
libcephfs: add test cases for dropping the suid/sgid in fallocate
libcephfs: fix ClearSetuid incorrectly using SETATTR_MODE mask
client: switch to clear_suid_sgid for ftruncate
client: switch to clear_suid_sgid for _write()
mds/client: clear the suid/sgid in fallocate path
client: allow unprivileged users to clear suid/sgid

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Milind Changire <mchangir@redhat.com>

Merge pull request #50687 from rishabh-d-dave/fs-qa-kernel-untar-tar

qa/workunit/fs: print commands for making debugging easier

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #50764 from zdover23/wip-doc-2023-03-230-start-documenting-ceph-squash-formatting

doc/start: format procedure in documenting-ceph

Merge PR #50692 into main

* refs/pull/50692/head:
tools/cephfs: include lost+found in scan_links

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #50742 from zdover23/wip-doc-2023-03-29-start-documenting-ceph-notify-us

doc/start: update "notify us" section

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/start: format procedure in documenting-ceph

Organize three steps in a procedure into one step with three sub-steps.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #50744 from zdover23/wip-doc-2023-03-29-rados-ops-health-checks

doc/rados: add link to ops/health-checks.rst

Merge PR #49773 into main

* refs/pull/49773/head:
mds: add config to decide whether to mark dentry bad
qa: add missing scan_links step for data scan recovery
qa/tasks/cephfs: test damage to dentry's first is caught
qa/tasks/cephfs: use rank_asok and allow specifying rank
qa/tasks: allow specifying timeout command prefix to ceph
mds: provide test configs for creating first corruption
mds: catch damage to dentry's first field
mds: add debugging for pre_cow_old_inode
mds: cleanup code

Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>

Merge pull request #50750 from Sainithin-bit/main

 doc/rados/operations: Fix typo in erasure-code.rst

Reviewed-by: Zac Dover <zac.dover@proton.me>

doc/rados/operations: Fix typo in erasure-code.rst

Signed-off-by: Sainithin Artham <sai.artham.19cse@bmu.edu.in>

Merge pull request #50633 from cbodley/wip-54102

qa/rgw: unpin centos for verify suite

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

mds: add config to decide whether to mark dentry bad

So admin can restore access to files if necessary.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: add missing scan_links step for data scan recovery

Without, the first field remains corrupt (HEAD).

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa/tasks/cephfs: test damage to dentry's first is caught

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa/tasks/cephfs: use rank_asok and allow specifying rank

This will use the more efficient:

    ceph tell mds.<fsname>:<rank> ...

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa/tasks: allow specifying timeout command prefix to ceph

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: provide test configs for creating first corruption

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: catch damage to dentry's first field

When possible. Abort the MDS before it can be written to the
journal/directory.

This is part of a series to address corruption first observed in [1].
How the corruption is introduced is yet unknown.

[1] https://tracker.ceph.com/issues/38452#note-10

Fixes: http://tracker.ceph.com/issues/58482
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: add debugging for pre_cow_old_inode

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: cleanup code

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #50553 from ljflores/wip-reef-crontab

qa/crontab: add reef upgrade tests and teuthology/nop

Merge pull request #50686 from rishabh-d-dave/fs-qa-client-upgrade

qa/suites/fs: check if cluster is healthy after upgrade

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #50307 from dparmar18/fix_pendingreleasenotes

PendingReleaseNotes: remove duplicate note and unwanted tracker link

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #50735 from yangdongsheng/rbd_quiesce

librbd: fix wrong attribute for rbd_quiesce_complete api

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

qa/suites/fs: check if cluster is healthy after upgrade

Signed-off-by: Rishabh Dave <ridave@redhat.com>

doc/rados: add link to ops/health-checks.rst

Add an internal navigation link to
doc/rados/operations/health-checks.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit f2f968f65bd1f1a6975fd340c66703cea335dcf6)

doc/start: update "notify us" section

Rewrite the "Notify Us" section in doc/start/documenting-ceph.rst so
that it is written in English sentences.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #50723 from zdover23/wip-doc-2023-03-28-documenting-ceph-squashing

doc/start: documenting-ceph - add squash procedure

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

qa/workunit/fs: print commands for making debugging easier

Print the commands and their arguments as they are being executed for
kernel_untar_tar.sh so that it's easier to debug when a teuthology
failure occurs due to it.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

librbd: fix wrong attribute for rbd_quiesce_complete api

When we use rbd_quiesce_complete api, we got an error:
/usr/bin/ld: undefined reference to `rbd_quiesce_complete'

Then we found the problem is the symbol of rbd_quiesce_complete
in librbd.so is LOCAL. After some investigation, we found
the attribute of rbd_quiesce_complete api is CEPH_RADOS_API
rather than expected CEPH_RBD_API.

Fixes: https://tracker.ceph.com/issues/59208
Signed-off-by: Dongsheng Yang <dongsheng.yang.linux@gmail.com>

Merge pull request #50718 from rhcs-dashboard/use-only-counter-dump-schema

exporter: user only counter dump/schema commands for extacting counters

Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #48183 from lxbsz/wip-57591

qa: enable kclient test for newop test

qa: enable kclient test for newop test

The kclient have already fix this. This will only enable the upstream
kclient with the testing branch, the downstream ones may not include
the fixing yet, so skip them for now.

The nautilus will only support the syntax v1. And for kclient there
is not need to do the upgrade.

Fixes: https://tracker.ceph.com/issues/57591
Signed-off-by: Xiubo Li <xiubli@redhat.com>

qa: add one func to distiguish the mount syntax version

Fixes: https://tracker.ceph.com/issues/57591
Signed-off-by: Xiubo Li <xiubli@redhat.com>

qa: remove the FuseMount/KernelMount check in test_newops.py

Will use the postmerge fragment to check this.

Fixes: https://tracker.ceph.com/issues/57591
Signed-off-by: Xiubo Li <xiubli@redhat.com>

qa: introduce postmerge for fuse/kclient mounts

Suggested by Patrick and this will check mounter's type.

Fixes: https://tracker.ceph.com/issues/57591
Signed-off-by: Xiubo Li <xiubli@redhat.com>

qa: remove the '0-' prefix

Both the nautilus and pacific will be run in parallel.

Fixes: https://tracker.ceph.com/issues/57591
Signed-off-by: Xiubo Li <xiubli@redhat.com>

doc/start: documenting-ceph - add squash procedure

Add a procedure to doc/start/documenting-ceph.rst that explains how to
perform an interactive rebase to squash commits.

Signed-off-by: Zac Dover <zac.dover@proton.me>

exporter: user only counter dump/schema commands for extacting counters

Fixes: https://tracker.ceph.com/issues/59191
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
Ceph exporter no more required the output of perf dump/schema, as the ``counter dump`` command
returns both labeled and unlabeled perf counters which exporter can fetch and export.
Removed the ``exporter_get_labeled_counters`` confiug option as exporter will now export
all the counters, labeled or unlabeled.
Also the fix includes the support for renaming the metrics name of rgw multi-site and
adding labels to it, similar to what is there in prometheus module.

Merge pull request #50713 from zdover23/wip-doc-2023-03-28-glossary-cephx

doc/glossary: improve "CephX" entry

Merge pull request #50529 from rhcs-dashboard/dashboard-edit-rgw-multisite

mgr/dashboard: edit realm in rgw-multisite

Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

doc/glossary: improve "CephX" entry

Improve the glossary entry for "CephX".

Signed-off-by: Zac Dover <zac.dover@proton.me>

mgr/dashboard: Add unit test for realm

Fixes: https://tracker.ceph.com/issues/59171
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #50653 from xxhdx1985126/wip-exist-clean

crimson/os/seastore/cache: consider EXIST_CLEAN extents as pending ones

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: use CachedExtent::is_mutable() where appropriate

CachedExtent::is_mutable() should only be used to check whether need to
call duplicate_for_write(extent).

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #50697 from zdover23/wip-doc-2023-03-28-glossary-scrubbing

doc/glossary: add "Scrubbing"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/glossary: add "Scrubbing"

Add "Scrubbing" to the glossary.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

tools/cephfs: include lost+found in scan_links

Otherwise, any injected dentries have incorrect first snapids.

Fixes: https://tracker.ceph.com/issues/59183
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #48937 from adk3798/device-ls-size

mgr/orchestrator: fix device size in `orch device ls` output

Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #50681 from lxbsz/qa-fscrypt-020

qa: fscrypt enable xfstests-dev generic/020 test case

disable default check if already set to true for selected realm

Fixes: https://tracker.ceph.com/issues/59171
Signed-off-by: avanthakkar <avanjohn@gmail.com>

disable create zonegroup if no master zone exist for existing master zonegroup

Fixes: https://tracker.ceph.com/issues/59171
Signed-off-by: avanthakkar <avanjohn@gmail.com>

mgr/dashboard: edit rgw-multisite

Fixes: https://tracker.ceph.com/issues/59171
Signed-off-by: Avan Thakkar <athakkar@redhat.com>

rgw/notifications: support bucket notification with bucket policy

following policy should be used to allow any user to get, put and delete
bucket notification on a bucket called "my-bucket":
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Statement",
      "Effect": "Allow",
      "Principal": "*",
      "Action": ["s3:GetBucketNotification", "s3:PutBucketNotification"],
      "Resource": "arn:aws:s3:::my-bucket"
    }
  ]
}

note that notification deletion uses the "PUT" permission.

Fixes: https://tracker.ceph.com/issues/59136
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

crimson/os/seastore/cache: consider EXIST_CLEAN extents as pending ones

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

qa: fscrypt enable xfstests-dev generic/020 test case

Since the https://git.ceph.com/xfstests-dev.git has pulled the
corresponding fix about the long attribute for generic/020 test case,
we can enable it now.

Signed-off-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #50675 from zdover23/wip-doc-2023-03-27-rados-operations-bluestore-migration-cleanup

doc/rados: clean up ops/bluestore-migration.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: clean up ops/bluestore-migration.rst

Clean up internal links, fix the numbering of a procedure, and implement
Anthony D'Atri's suggestions in
https://github.com/ceph/ceph/pull/50487 and
https://github.com/ceph/ceph/pull/50488.

https://tracker.ceph.com/issues/58485

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #50654 from zdover23/wip-doc-2023-03-24-glossary-user

doc/glossary: add "User"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/glossary: add "User"

Add "User" to glossary.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #50660 from zdover23/wip-doc-2023-03-25-rados-operations-bluestore-migration-prompt-fix

doc/operations: fix prompt in bluestore-migration

doc/operations: fix prompt in bluestore-migration

Fix a single prompt in bluestore-migration.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

qa/crontab: add reef upgrade tests and teuthology/nop

Signed-off-by: Laura Flores <lflores@redhat.com>

Merge pull request #50302 from weirdwiz/rbd-perf-counters

rbd-mirror: switch to labeled perf counters

Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #50497 from rishabh-d-dave/fs-qa-caps-helper

qa/cephfs: add more helper methods to caps_helper.py

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #49302 from petrutlucian94/adapter_resets

rbd-wnbd: optionally handle wnbd adapter restart events

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #50369 from rhcs-dashboard/exporter-labeled-counters

exporter: add support for exposing labeled perf counters

rbd-wnbd: consistently use negative error codes in rbd-wnbd

The rbd-wnbd iterators return positive errors, which is why
in certain cases we may end up with both positive and negative
error codes.

This change ensures that we'll consistently use negative
error codes.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

common, rbd-wnbd: bump Windows log level

We're increasing the log level for certain Windows operational log
messages.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

rbd-wnbd: optionally handle wnbd adapter restart events

The WNBD adapter may be reset in certain situations (e.g. driver
upgrade, MS WHQL tests, etc).

We're going to monitor the WNBD adapter using WMI[1] events, restarting
the rbd-wnbd disk mappings whenever necessary. Adapter monitoring can be
enabled by passing the --adapter-monitoring-enabled flag to the service.

This feature is optional for the following reasons:

* it's mainly used during development / driver certification
* we had to use a relatively small polling interval, which might imply
  additional resource usage. WMI quotas also have to be considered.

While at it, we're updating two lambdas that are submitted to thread pools,
avoiding default reference capturing and explicitly specifying the variables
that get copied.

[1] https://learn.microsoft.com/en-us/windows/win32/wmisdk/wmi-start-page

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

qa/cephfs: add more helper methods to caps_helper.py

Add methods that will accept read/write permissions, CephFS names and
CephFS mount point and in return will generate string form of MON, OSD
and MDS caps exactly as it is reported in Ceph keyrings.

Replace similar code in test_multifs_auth.py with calls to these helper
methods.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #50649 from Thingee/foundation-mem-update-20230323

doc/foundation: Update Foundation members

doc/foundation: Update Foundation members

Removing EasyStack, Vexxhost and adding 42on

Signed-off-by: Mike Perez <thingee@gmail.com>

Merge pull request #50545 from ivancich/wip-fix-bi-restore-script-installation

rgw: install rgw scripts with common files rather than radosgw files

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #50617 from ivancich/wip-add-unordered-list-restore-index

rgw: add unordered listing to reindex to force stats update

Reviewed-by: Casey Bodley <cbodley@redhat.com>

exporter: export labeled perf-counters as prometheus metrics

Fixes: https://tracker.ceph.com/issues/59063
Signed-off-by: Avan Thakkar <athakkar@redhat.com>

qa/rgw: reenable wait-for-scrub

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: run 'gc process' when rgw task unwinds

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #50634 from zdover23/wip-doc-2023-03-23-rados-operations-user-management-1-of-x

doc/rados: edit user-management.rst (1 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: edit user-management.rst (1 of x)

Edit doc/rados/operations/user-management.rst. (1 of x)

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #50555 from myoungwon/wip-rbm-get-stat

crimson/os/seastore: implement get_stat() for RBM

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/rbm: implement get_stat()

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

Merge pull request #50334 from kamoltat/wip-ksirivad-fix-autoscale-warn

pybind/mgr/pg_autoscaler: Reorderd if statement for the func: _maybe_adjust

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #49748 from ifed01/wip-ifed-bound-rm-range-keys

kv/RocksDBStore: use bounded iterators in rm_range_keys

Reviewed-by: Cory Snyder <csnyder@iland.com>

Merge pull request #49380 from Matan-B/wip-matanb-lrod-logs

osd/PeeringState: Add logs around can_serve_replica_read() / last_complete_ondisk()

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

qa/rgw: unpin centos for verify suite

use a random supported distro instead of centos

Fixes: https://tracker.ceph.com/issues/54102
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #50510 from cbodley/wip-rgw-server-name

rgw: add default Server response header

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #50504 from cbodley/wip-59048

rgw/s3: DeleteObjects response uses correct delete_marker flag

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

common: add win32/wstring.h

Windows APIs heavily use wchar. ceph-dokan and rbd-wnbd
have some duplicated helpers that convert wstrings to/from
utf8 strings.

To avoid duplication and allow reusing those helpers, we're moving
them to common/win32/wstring.h.

We're using the "win32" subfolder because it's unlikely that this
will ever be used on other platforms.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Merge pull request #50492 from Matan-B/wip-matanb-c-logging-levels

crimson/common/log: Fix debug logging macros

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #50491 from Matan-B/wip-matanb-c-snaps-tests

test/librados/snapshots*: Enable Crimson supported tests

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #50546 from Matan-B/wip-matanb-crimson-only-order-snap

crimson/osd: Support librados::OPERATION_ORDERSNAP

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #49328 from BryceCao/wip-fix-python-script-using-s3cmd-with-error-code-403

rgw : fix python script using s3cmd with error code 403

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #50606 from zdover23/wip-doc-2023-03-21-rados-operations-erasure-code

doc/rados: line-edit erasure-code.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

rgw: add unordered listing to reindex to force stats update

By including an unordered listing in the script, we will complete
placing objects in the bucket index and allow stats to be updated
rather than waiting for this to happen organically at a user's
request. Unordered is preferred as it can run more efficiently.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #50486 from linuxbox2/wip-dedup-opsqs

Do not duplicate query-string in ops-log

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #50468 from aclamk/wip-aclamk-harmonize-bluefs-log

os/bluestore: BlueFS: harmonize log read and writes modes