qa/workunit/fs: print commands for making debugging easier

Print the commands and their arguments as they are being executed for
kernel_untar_tar.sh so that it's easier to debug when a teuthology
failure occurs due to it.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #50718 from rhcs-dashboard/use-only-counter-dump-schema

exporter: user only counter dump/schema commands for extacting counters

Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #48183 from lxbsz/wip-57591

qa: enable kclient test for newop test

qa: enable kclient test for newop test

The kclient have already fix this. This will only enable the upstream
kclient with the testing branch, the downstream ones may not include
the fixing yet, so skip them for now.

The nautilus will only support the syntax v1. And for kclient there
is not need to do the upgrade.

Fixes: https://tracker.ceph.com/issues/57591
Signed-off-by: Xiubo Li <xiubli@redhat.com>

qa: add one func to distiguish the mount syntax version

Fixes: https://tracker.ceph.com/issues/57591
Signed-off-by: Xiubo Li <xiubli@redhat.com>

qa: remove the FuseMount/KernelMount check in test_newops.py

Will use the postmerge fragment to check this.

Fixes: https://tracker.ceph.com/issues/57591
Signed-off-by: Xiubo Li <xiubli@redhat.com>

qa: introduce postmerge for fuse/kclient mounts

Suggested by Patrick and this will check mounter's type.

Fixes: https://tracker.ceph.com/issues/57591
Signed-off-by: Xiubo Li <xiubli@redhat.com>

qa: remove the '0-' prefix

Both the nautilus and pacific will be run in parallel.

Fixes: https://tracker.ceph.com/issues/57591
Signed-off-by: Xiubo Li <xiubli@redhat.com>

exporter: user only counter dump/schema commands for extacting counters

Fixes: https://tracker.ceph.com/issues/59191
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
Ceph exporter no more required the output of perf dump/schema, as the ``counter dump`` command
returns both labeled and unlabeled perf counters which exporter can fetch and export.
Removed the ``exporter_get_labeled_counters`` confiug option as exporter will now export
all the counters, labeled or unlabeled.
Also the fix includes the support for renaming the metrics name of rgw multi-site and
adding labels to it, similar to what is there in prometheus module.

Merge pull request #50713 from zdover23/wip-doc-2023-03-28-glossary-cephx

doc/glossary: improve "CephX" entry

Merge pull request #50529 from rhcs-dashboard/dashboard-edit-rgw-multisite

mgr/dashboard: edit realm in rgw-multisite

Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

doc/glossary: improve "CephX" entry

Improve the glossary entry for "CephX".

Signed-off-by: Zac Dover <zac.dover@proton.me>

mgr/dashboard: Add unit test for realm

Fixes: https://tracker.ceph.com/issues/59171
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #50653 from xxhdx1985126/wip-exist-clean

crimson/os/seastore/cache: consider EXIST_CLEAN extents as pending ones

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: use CachedExtent::is_mutable() where appropriate

CachedExtent::is_mutable() should only be used to check whether need to
call duplicate_for_write(extent).

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #50697 from zdover23/wip-doc-2023-03-28-glossary-scrubbing

doc/glossary: add "Scrubbing"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/glossary: add "Scrubbing"

Add "Scrubbing" to the glossary.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #48937 from adk3798/device-ls-size

mgr/orchestrator: fix device size in `orch device ls` output

Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #50681 from lxbsz/qa-fscrypt-020

qa: fscrypt enable xfstests-dev generic/020 test case

disable default check if already set to true for selected realm

Fixes: https://tracker.ceph.com/issues/59171
Signed-off-by: avanthakkar <avanjohn@gmail.com>

disable create zonegroup if no master zone exist for existing master zonegroup

Fixes: https://tracker.ceph.com/issues/59171
Signed-off-by: avanthakkar <avanjohn@gmail.com>

mgr/dashboard: edit rgw-multisite

Fixes: https://tracker.ceph.com/issues/59171
Signed-off-by: Avan Thakkar <athakkar@redhat.com>

crimson/os/seastore/cache: consider EXIST_CLEAN extents as pending ones

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

qa: fscrypt enable xfstests-dev generic/020 test case

Since the https://git.ceph.com/xfstests-dev.git has pulled the
corresponding fix about the long attribute for generic/020 test case,
we can enable it now.

Signed-off-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #50675 from zdover23/wip-doc-2023-03-27-rados-operations-bluestore-migration-cleanup

doc/rados: clean up ops/bluestore-migration.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: clean up ops/bluestore-migration.rst

Clean up internal links, fix the numbering of a procedure, and implement
Anthony D'Atri's suggestions in
https://github.com/ceph/ceph/pull/50487 and
https://github.com/ceph/ceph/pull/50488.

https://tracker.ceph.com/issues/58485

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #50654 from zdover23/wip-doc-2023-03-24-glossary-user

doc/glossary: add "User"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/glossary: add "User"

Add "User" to glossary.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #50660 from zdover23/wip-doc-2023-03-25-rados-operations-bluestore-migration-prompt-fix

doc/operations: fix prompt in bluestore-migration

doc/operations: fix prompt in bluestore-migration

Fix a single prompt in bluestore-migration.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #50302 from weirdwiz/rbd-perf-counters

rbd-mirror: switch to labeled perf counters

Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #50497 from rishabh-d-dave/fs-qa-caps-helper

qa/cephfs: add more helper methods to caps_helper.py

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #49302 from petrutlucian94/adapter_resets

rbd-wnbd: optionally handle wnbd adapter restart events

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #50369 from rhcs-dashboard/exporter-labeled-counters

exporter: add support for exposing labeled perf counters

rbd-wnbd: consistently use negative error codes in rbd-wnbd

The rbd-wnbd iterators return positive errors, which is why
in certain cases we may end up with both positive and negative
error codes.

This change ensures that we'll consistently use negative
error codes.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

common, rbd-wnbd: bump Windows log level

We're increasing the log level for certain Windows operational log
messages.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

rbd-wnbd: optionally handle wnbd adapter restart events

The WNBD adapter may be reset in certain situations (e.g. driver
upgrade, MS WHQL tests, etc).

We're going to monitor the WNBD adapter using WMI[1] events, restarting
the rbd-wnbd disk mappings whenever necessary. Adapter monitoring can be
enabled by passing the --adapter-monitoring-enabled flag to the service.

This feature is optional for the following reasons:

* it's mainly used during development / driver certification
* we had to use a relatively small polling interval, which might imply
  additional resource usage. WMI quotas also have to be considered.

While at it, we're updating two lambdas that are submitted to thread pools,
avoiding default reference capturing and explicitly specifying the variables
that get copied.

[1] https://learn.microsoft.com/en-us/windows/win32/wmisdk/wmi-start-page

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

qa/cephfs: add more helper methods to caps_helper.py

Add methods that will accept read/write permissions, CephFS names and
CephFS mount point and in return will generate string form of MON, OSD
and MDS caps exactly as it is reported in Ceph keyrings.

Replace similar code in test_multifs_auth.py with calls to these helper
methods.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #50649 from Thingee/foundation-mem-update-20230323

doc/foundation: Update Foundation members

doc/foundation: Update Foundation members

Removing EasyStack, Vexxhost and adding 42on

Signed-off-by: Mike Perez <thingee@gmail.com>

Merge pull request #50545 from ivancich/wip-fix-bi-restore-script-installation

rgw: install rgw scripts with common files rather than radosgw files

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #50617 from ivancich/wip-add-unordered-list-restore-index

rgw: add unordered listing to reindex to force stats update

Reviewed-by: Casey Bodley <cbodley@redhat.com>

exporter: export labeled perf-counters as prometheus metrics

Fixes: https://tracker.ceph.com/issues/59063
Signed-off-by: Avan Thakkar <athakkar@redhat.com>

Merge pull request #50634 from zdover23/wip-doc-2023-03-23-rados-operations-user-management-1-of-x

doc/rados: edit user-management.rst (1 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: edit user-management.rst (1 of x)

Edit doc/rados/operations/user-management.rst. (1 of x)

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #50555 from myoungwon/wip-rbm-get-stat

crimson/os/seastore: implement get_stat() for RBM

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/rbm: implement get_stat()

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

Merge pull request #50334 from kamoltat/wip-ksirivad-fix-autoscale-warn

pybind/mgr/pg_autoscaler: Reorderd if statement for the func: _maybe_adjust

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #49748 from ifed01/wip-ifed-bound-rm-range-keys

kv/RocksDBStore: use bounded iterators in rm_range_keys

Reviewed-by: Cory Snyder <csnyder@iland.com>

Merge pull request #49380 from Matan-B/wip-matanb-lrod-logs

osd/PeeringState: Add logs around can_serve_replica_read() / last_complete_ondisk()

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #50510 from cbodley/wip-rgw-server-name

rgw: add default Server response header

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #50504 from cbodley/wip-59048

rgw/s3: DeleteObjects response uses correct delete_marker flag

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

common: add win32/wstring.h

Windows APIs heavily use wchar. ceph-dokan and rbd-wnbd
have some duplicated helpers that convert wstrings to/from
utf8 strings.

To avoid duplication and allow reusing those helpers, we're moving
them to common/win32/wstring.h.

We're using the "win32" subfolder because it's unlikely that this
will ever be used on other platforms.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Merge pull request #50492 from Matan-B/wip-matanb-c-logging-levels

crimson/common/log: Fix debug logging macros

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #50491 from Matan-B/wip-matanb-c-snaps-tests

test/librados/snapshots*: Enable Crimson supported tests

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #50546 from Matan-B/wip-matanb-crimson-only-order-snap

crimson/osd: Support librados::OPERATION_ORDERSNAP

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #49328 from BryceCao/wip-fix-python-script-using-s3cmd-with-error-code-403

rgw : fix python script using s3cmd with error code 403

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #50606 from zdover23/wip-doc-2023-03-21-rados-operations-erasure-code

doc/rados: line-edit erasure-code.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

rgw: add unordered listing to reindex to force stats update

By including an unordered listing in the script, we will complete
placing objects in the bucket index and allow stats to be updated
rather than waiting for this to happen organically at a user's
request. Unordered is preferred as it can run more efficiently.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #50486 from linuxbox2/wip-dedup-opsqs

Do not duplicate query-string in ops-log

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #50468 from aclamk/wip-aclamk-harmonize-bluefs-log

os/bluestore: BlueFS: harmonize log read and writes modes

rgw : fix python script using s3cmd with error code 403 w/https

Fixes: https://tracker.ceph.com/issues/54104
Signed-off-by: Mark Kogan <mkogan@redhat.com>

doc/rados: line-edit erasure-code.rst

Line-edit doc/rados/operations/erasure-code.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

rbd-mirror: add and rename perf counters for journal and snapshot mirroring

This commit renames the existing snapshot and journal based performance
counters in the rbd-mirror daemon to better reflect their purpose.
Additionally, new perf counters have been added to provide more detailed
information about the synchronization of snapshots between the source
and destination clusters.

This commit also switches to use labels instead of having the image spec
in the name of the counter.

Signed-off-by: Divyansh Kamboj <dkamboj@redhat.com>

Merge pull request #50602 from xxhdx1985126/wip-invalid-clean-pending-extents

crimson/os/seastore: don't set INVALID extents to CLEAN when reading extents

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: don't set INVALID extents to CLEAN when reading
extents

CLEAN_PENDING extents may be invalidated before read completes:

1. transaction A retired an laddr, which lead to a RetirePlaceHolder in
   Cache
2. transaction B try to read that extent, and replace A's
   RetirePlaceHolder with it;
3. transaction A commits and invalidate that extent;
4. transaction B complete reading that extent;

In this case, we shouldn't set the extent's state to CLEAN

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #50488 from zdover23/wip-doc-2023-03-13-rados-bluestore-migration-2-of-x

doc/rados: edit operations/bs-migration (2 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #50487 from zdover23/wip-doc-2023-03-12-rados-bluestore-migration

doc/rados: edit operations/bs-migration (1 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #50579 from zdover23/wip-doc-2023-03-18-rados-configuration-mon-lookup-dns

doc/rados: line edit mon-lookup-dns top matter

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: line edit mon-lookup-dns top matter

Improve the syntax of the top matter of
doc/rados/operations/mon-lookup-dns.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #49443 from cbodley/wip-qa-supported-distros

qa/distros: add ubuntu 22 as supported distro

Reviewed-by: Yuri Weinstein <yuriw@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #50498 from rishabh-d-dave/fs-qa-caps-helper-minor

qa/cephfs: minor corrections in caps_helper.py

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

Merge pull request #50556 from aravind-wdc/fix-header

crimson: fix build error

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #50559 from petrutlucian94/improve_win32_clock_check

common: avoid redefining clock type on Windows

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge PR #50544 into main

* refs/pull/50544/head:
test/libcephfs: skip flaky timestamp assertion on Windows

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #50106 from rhcs-dashboard/new-landing-page-and-toggle-button

mgr/dashboard: add button toggle to switch between landing pages

common: avoid redefining clock type on Windows

mingw >= 8.0.1 defines CLOCK_REALTIME_COARSE, so we'll avoid
overriding it if already set.

Clock precision [1]:

mingw < 8.0.1:
  * CLOCK_REALTIME: ~10-55ms (GetSystemTimeAsFileTime)
mingw >= 8.0.1:
  * CLOCK_REALTIME: <1us (GetSystemTimePreciseAsFileTime)
  * CLOCK_REALTIME_COARSE: ~10-55ms (GetSystemTimeAsFileTime)

* CLOCK_MONOTONIC: <1us if TSC is usable, ~10-55ms otherwise
                   (QueryPerformanceCounter)

[1] https://github.com/mirror/mingw-w64/commit/dcd990ed423381cf35702df9495d44f1979ebe50

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Merge pull request #50563 from zdover23/wip-doc-2023-03-17-rados-operations-upmap

doc/rados: line-edit upmap.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: line-edit upmap.rst

Edit all of doc/rados/upmap.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #50564 from Thingee/update-foundation-202303

docs: Update Foundation members for March 2023

docs: Update Foundation members for March 2023

Signed-off-by: Mike Perez <thingee@gmail.com>

crimson: fix build error

With -DWITH_SEASTAR=ON, build is failing.

src/crimson/osd/osd_operations/snaptrim_event.cc:174:29: error: ‘sleep’ is not a member of ‘seastar’

Signed-off-by: Aravind Ramesh <Aravind.Ramesh@wdc.com>

test/libcephfs: skip flaky timestamp assertion on Windows

There's a new libcephfs test that creates a snapshot and
compares ctime/mtime. The issue is that one of the assertion
fails on Windows, potentially due to reduced timestamp
precision.

For now, we'll skip those assertions on Windows.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

mgr/dashboard: add button toggle to switch between new and old landing page

Signed-off-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

Merge pull request #50534 from neha-ojha/wip-release-cl

doc/dev/release-checklists.rst: add after dev freeze section

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

test/librados/snapshots_cxx: Enable OrderSnap for Crimson

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/osd_operations/client_request: Support librados::OPERATION_ORDERSNAP

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/client_request: Move writer SnapContext initialization

Move the initialization from PG::do_osd_ops() to ClientRequest::do_process()
in order to allow for ORDERSNAP error checking before do_osd_ops. (See next commit)

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

qa/workunits/rbd: use bionic version of qemu-iotests for jammy

Same as in commit 2de2146c300c ("qa/workunits/rbd: use bionic version
of qemu-iotests for focal").

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

rgw: install rgw scripts with common files rather than radosgw files

Update ceph.spec.in and debian install files so
rgw-restore-bucket-index, rgw-orphan-list, rgw-gap-list,
rgw-gap-list-comparator are installed with common files.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #50214 from rhcs-dashboard/landing-page-prom-error-fix

mgr/dashboard: fix prometheus api issues on landing page v3

Reviewed-by: Pegonzal <NOT@FOUND>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #47974 from zhscn/wip-evict

crimson/os/seastore: evict cold data to slower devices

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

test/crimson/seasatore: add evction test

Signed-off-by: Zhang Song <zhangsong325@gmail.com>

crimson/os/seastore/EPM: cleanup alloc_new_extent

Signed-off-by: Zhang Song <zhangsong325@gmail.com>

crimson/os/seasotre/segment_manager_group: ensure the types of devices are same

Signed-off-by: Zhang Song <zhangsong325@gmail.com>

crimson/os/seastore: add EPHEMERAL_COLD device type

rename SEGMENTED_EPHEMERAL to EPHEMERAL_MAIN

Signed-off-by: Zhang Song <zhangsong325@gmail.com>

crimson/os/seastore/EPM/BackgroundProcess: introduce more eviction policy

Signed-off-by: Zhang Song <zhangsong325@gmail.com>

crimson/os/seastore/EPM/BackgroundProcess: reserve space on cold cleaner

Signed-off-by: Zhang Song <zhangsong325@gmail.com>

crimson/os/seastore: get segment info from cold segment provider

Signed-off-by: Zhang Song <zhangsong325@gmail.com>

crimson/os/seastore: support mark_space_{free, used} on cold_cleaner

Signed-off-by: Zhang Song <zhangsong325@gmail.com>