crimson/os/seastore/cache: consider EXIST_CLEAN extents as pending ones

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #50649 from Thingee/foundation-mem-update-20230323

doc/foundation: Update Foundation members

doc/foundation: Update Foundation members

Removing EasyStack, Vexxhost and adding 42on

Signed-off-by: Mike Perez <thingee@gmail.com>

Merge pull request #50545 from ivancich/wip-fix-bi-restore-script-installation

rgw: install rgw scripts with common files rather than radosgw files

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #50617 from ivancich/wip-add-unordered-list-restore-index

rgw: add unordered listing to reindex to force stats update

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #50634 from zdover23/wip-doc-2023-03-23-rados-operations-user-management-1-of-x

doc/rados: edit user-management.rst (1 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: edit user-management.rst (1 of x)

Edit doc/rados/operations/user-management.rst. (1 of x)

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #50555 from myoungwon/wip-rbm-get-stat

crimson/os/seastore: implement get_stat() for RBM

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/rbm: implement get_stat()

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

Merge pull request #50334 from kamoltat/wip-ksirivad-fix-autoscale-warn

pybind/mgr/pg_autoscaler: Reorderd if statement for the func: _maybe_adjust

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #49748 from ifed01/wip-ifed-bound-rm-range-keys

kv/RocksDBStore: use bounded iterators in rm_range_keys

Reviewed-by: Cory Snyder <csnyder@iland.com>

Merge pull request #49380 from Matan-B/wip-matanb-lrod-logs

osd/PeeringState: Add logs around can_serve_replica_read() / last_complete_ondisk()

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #50510 from cbodley/wip-rgw-server-name

rgw: add default Server response header

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #50504 from cbodley/wip-59048

rgw/s3: DeleteObjects response uses correct delete_marker flag

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #50492 from Matan-B/wip-matanb-c-logging-levels

crimson/common/log: Fix debug logging macros

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #50491 from Matan-B/wip-matanb-c-snaps-tests

test/librados/snapshots*: Enable Crimson supported tests

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #50546 from Matan-B/wip-matanb-crimson-only-order-snap

crimson/osd: Support librados::OPERATION_ORDERSNAP

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #49328 from BryceCao/wip-fix-python-script-using-s3cmd-with-error-code-403

rgw : fix python script using s3cmd with error code 403

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #50606 from zdover23/wip-doc-2023-03-21-rados-operations-erasure-code

doc/rados: line-edit erasure-code.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

rgw: add unordered listing to reindex to force stats update

By including an unordered listing in the script, we will complete
placing objects in the bucket index and allow stats to be updated
rather than waiting for this to happen organically at a user's
request. Unordered is preferred as it can run more efficiently.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #50486 from linuxbox2/wip-dedup-opsqs

Do not duplicate query-string in ops-log

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #50468 from aclamk/wip-aclamk-harmonize-bluefs-log

os/bluestore: BlueFS: harmonize log read and writes modes

rgw : fix python script using s3cmd with error code 403 w/https

Fixes: https://tracker.ceph.com/issues/54104
Signed-off-by: Mark Kogan <mkogan@redhat.com>

doc/rados: line-edit erasure-code.rst

Line-edit doc/rados/operations/erasure-code.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #50602 from xxhdx1985126/wip-invalid-clean-pending-extents

crimson/os/seastore: don't set INVALID extents to CLEAN when reading extents

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: don't set INVALID extents to CLEAN when reading
extents

CLEAN_PENDING extents may be invalidated before read completes:

1. transaction A retired an laddr, which lead to a RetirePlaceHolder in
   Cache
2. transaction B try to read that extent, and replace A's
   RetirePlaceHolder with it;
3. transaction A commits and invalidate that extent;
4. transaction B complete reading that extent;

In this case, we shouldn't set the extent's state to CLEAN

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #50488 from zdover23/wip-doc-2023-03-13-rados-bluestore-migration-2-of-x

doc/rados: edit operations/bs-migration (2 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #50487 from zdover23/wip-doc-2023-03-12-rados-bluestore-migration

doc/rados: edit operations/bs-migration (1 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #50579 from zdover23/wip-doc-2023-03-18-rados-configuration-mon-lookup-dns

doc/rados: line edit mon-lookup-dns top matter

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: line edit mon-lookup-dns top matter

Improve the syntax of the top matter of
doc/rados/operations/mon-lookup-dns.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #49443 from cbodley/wip-qa-supported-distros

qa/distros: add ubuntu 22 as supported distro

Reviewed-by: Yuri Weinstein <yuriw@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #50498 from rishabh-d-dave/fs-qa-caps-helper-minor

qa/cephfs: minor corrections in caps_helper.py

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

Merge pull request #50556 from aravind-wdc/fix-header

crimson: fix build error

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #50559 from petrutlucian94/improve_win32_clock_check

common: avoid redefining clock type on Windows

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge PR #50544 into main

* refs/pull/50544/head:
test/libcephfs: skip flaky timestamp assertion on Windows

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #50106 from rhcs-dashboard/new-landing-page-and-toggle-button

mgr/dashboard: add button toggle to switch between landing pages

common: avoid redefining clock type on Windows

mingw >= 8.0.1 defines CLOCK_REALTIME_COARSE, so we'll avoid
overriding it if already set.

Clock precision [1]:

mingw < 8.0.1:
  * CLOCK_REALTIME: ~10-55ms (GetSystemTimeAsFileTime)
mingw >= 8.0.1:
  * CLOCK_REALTIME: <1us (GetSystemTimePreciseAsFileTime)
  * CLOCK_REALTIME_COARSE: ~10-55ms (GetSystemTimeAsFileTime)

* CLOCK_MONOTONIC: <1us if TSC is usable, ~10-55ms otherwise
                   (QueryPerformanceCounter)

[1] https://github.com/mirror/mingw-w64/commit/dcd990ed423381cf35702df9495d44f1979ebe50

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Merge pull request #50563 from zdover23/wip-doc-2023-03-17-rados-operations-upmap

doc/rados: line-edit upmap.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: line-edit upmap.rst

Edit all of doc/rados/upmap.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #50564 from Thingee/update-foundation-202303

docs: Update Foundation members for March 2023

docs: Update Foundation members for March 2023

Signed-off-by: Mike Perez <thingee@gmail.com>

crimson: fix build error

With -DWITH_SEASTAR=ON, build is failing.

src/crimson/osd/osd_operations/snaptrim_event.cc:174:29: error: ‘sleep’ is not a member of ‘seastar’

Signed-off-by: Aravind Ramesh <Aravind.Ramesh@wdc.com>

test/libcephfs: skip flaky timestamp assertion on Windows

There's a new libcephfs test that creates a snapshot and
compares ctime/mtime. The issue is that one of the assertion
fails on Windows, potentially due to reduced timestamp
precision.

For now, we'll skip those assertions on Windows.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

mgr/dashboard: add button toggle to switch between new and old landing page

Signed-off-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

Merge pull request #50534 from neha-ojha/wip-release-cl

doc/dev/release-checklists.rst: add after dev freeze section

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

test/librados/snapshots_cxx: Enable OrderSnap for Crimson

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/osd_operations/client_request: Support librados::OPERATION_ORDERSNAP

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/client_request: Move writer SnapContext initialization

Move the initialization from PG::do_osd_ops() to ClientRequest::do_process()
in order to allow for ORDERSNAP error checking before do_osd_ops. (See next commit)

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

qa/workunits/rbd: use bionic version of qemu-iotests for jammy

Same as in commit 2de2146c300c ("qa/workunits/rbd: use bionic version
of qemu-iotests for focal").

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

rgw: install rgw scripts with common files rather than radosgw files

Update ceph.spec.in and debian install files so
rgw-restore-bucket-index, rgw-orphan-list, rgw-gap-list,
rgw-gap-list-comparator are installed with common files.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #50214 from rhcs-dashboard/landing-page-prom-error-fix

mgr/dashboard: fix prometheus api issues on landing page v3

Reviewed-by: Pegonzal <NOT@FOUND>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #47974 from zhscn/wip-evict

crimson/os/seastore: evict cold data to slower devices

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

test/crimson/seasatore: add evction test

Signed-off-by: Zhang Song <zhangsong325@gmail.com>

crimson/os/seastore/EPM: cleanup alloc_new_extent

Signed-off-by: Zhang Song <zhangsong325@gmail.com>

crimson/os/seasotre/segment_manager_group: ensure the types of devices are same

Signed-off-by: Zhang Song <zhangsong325@gmail.com>

crimson/os/seastore: add EPHEMERAL_COLD device type

rename SEGMENTED_EPHEMERAL to EPHEMERAL_MAIN

Signed-off-by: Zhang Song <zhangsong325@gmail.com>

crimson/os/seastore/EPM/BackgroundProcess: introduce more eviction policy

Signed-off-by: Zhang Song <zhangsong325@gmail.com>

crimson/os/seastore/EPM/BackgroundProcess: reserve space on cold cleaner

Signed-off-by: Zhang Song <zhangsong325@gmail.com>

crimson/os/seastore: get segment info from cold segment provider

Signed-off-by: Zhang Song <zhangsong325@gmail.com>

crimson/os/seastore: support mark_space_{free, used} on cold_cleaner

Signed-off-by: Zhang Song <zhangsong325@gmail.com>

src/vstart.sh: support specify the type of secondary device

doc/crimson: add options and example about SeaStore

Signed-off-by: Zhang Song <zhangsong325@gmail.com>

crimson/os/seastore: change the category of COLL_BLOCK

Signed-off-by: Zhang Song <zhangsong325@gmail.com>

crimson/os/seastore: specify the device type explicitly when constructing BlockSegmentManager

Signed-off-by: Zhang Song <zhangsong325@gmail.com>

crimson/os/seastore: make segment_manager work with HDD

Signed-off-by: Zhang Song <zhangsong325@gmail.com>

crimson/os/seasotre: add CLEANER_COLD transaction type

rename CLEANER to CLEANER_MAIN

Signed-off-by: Zhang Song <zhangsong325@gmail.com>

crimson/os/seastore: init cold segment cleaner

Signed-off-by: Zhang Song <zhangsong325@gmail.com>

crimson/os/seastore/SegmentCleaner: distinguish the cold cleaner from the normal

Signed-off-by: Zhang Song <zhangsong325@gmail.com>

crimson/os/seastore/EPM/BackgroundProcess: add cold_cleaner

Signed-off-by: Zhang Song <zhangsong325@gmail.com>

crimson/os/seastore: move ool segment seq allocator to EPM

Signed-off-by: Zhang Song <zhangsong325@gmail.com>

crimson/os/seastore/EPM: refactor reserve process

Signed-off-by: Zhang Song <zhangsong325@gmail.com>

crimson/os/seastore/EPM/BackgroundProcess: rename cleaner to main_cleaner

Signed-off-by: Zhang Song <zhangsong325@gmail.com>

Do not duplicate query-string in ops-log

Fixes: https://tracker.ceph.com/issues/59059
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

doc/dev/release-checklists.rst: add after dev freeze section

- add release to redmine
- add milestone

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #50339 from cbodley/wip-qa-rgw-reef-upgrade

qa/rgw: add rgw/upgrade suite

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #50409 from rhcs-dashboard/formly

mgr/dashboard: replace ajsf with formly

Reviewed-by: Pegonzal <NOT@FOUND>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #45960 from rishabh-d-dave/xfstests-dev-run-more-tests

qa/cephfs: upgrade xfstests_dev.py to run more tests

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #50390 from Svelar/vstart-doc

doc/crimson: add SeaStore when using vstart.

Reviewed-by: Zac Dover <zac.dover@proton.me>

Merge pull request #48411 from liangmingyuanneo/wip-rgw-reshard-collision

rgw/reshard: fix hold reshard lock collisions

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #48482 from cbodley/wip-rgw-site-config

rgw: add SiteConfig to load/create zonegroup configuration

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #50347 from cbodley/wip-rgw-sal-aio

rgw/aio: remove RGWSI_RADOS from generic Aio::get()

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #50343 from pkalever/fix-errormsg

librbd: report better errors when failing to enable mirroring on an image

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #50500 from jdurgin/wip-release-checklist

doc/dev/release-checklist: add another step for ceph-build

Reviewed-by: Neha Ojha <nojha@redhat.com>

rgw: add default Server response header

if a specific rgw_service_provider_name is not configured, add a default
Server response header that includes the ceph release name:

Server: Ceph Object Gateway (reef)

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Revert "rgw/beast: add extra_response_headers for Server name"

This reverts commit ffeacc095963436588da749f34c5890ca8c74aa4.

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #50494 from rhcs-dashboard/fix-58942-main

mgr/dashboard: force TLS 1.3

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

mgr/dashboard: replace ajsf with formly

ajsf json schema library for angular doesn't seem to be actively
maintained. Instead, fromly is a well maintained replacement with extra
stuff like validators builtin, support for json schemas, custom
components, etc...

Textareas weren't supported on ajsf, therefore, it made sense to move to
this dep instead.

Signed-off-by: Pere Diaz Bou <pdiazbou@redhat.com>
Signed-off-by: Nizamudeen A <nia@redhat.com>

rgw/s3: DeleteObjects response uses correct delete_marker flag

use delete_marker from the DeleteOp result, not from the sal::Object

Fixes: https://tracker.ceph.com/issues/59048
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #50147 from cbodley/wip-debian-mgr-pytest

debian/mgr: remove pytest from debian/ceph-mgr-dashboard.requires

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #50282 from mhmdawnallah/fix-access-a-destructed-temporary-variable-in-rgw_sync_checkpoint

src/rgw/rgw_sync_checkpoint.cc: fix a destructed temporary variable access

reviewed-by: adamemerson, yuvalif

Merge pull request #50482 from neha-ojha/wip-milestore

.github/milestone.yml: add reef

doc/dev/release-checklist: add another step for ceph-build

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #50493 into main

* refs/pull/50493/head:
qa: fixup quota values exceeding 64 bit signed int value
qa, cleanup: no need to specify human readable quota values in comment

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

Merge PR #48382 into main

* refs/pull/48382/head:
test: add test to validate attrs for snapdir after snap operations
client: update {m,c}time and change attr for snapdir from last modification time of a snap realm
mds: update last modification time and change attr for a snap realm node
mds: include last modification time and change attr of a snap realm node in snap trace reply
mds: use change_attr from closest ancestor when initializing a new snaprealm
mds: initialize last_modified to current timestamp for root inode snaprealm
mds: track last modification time and change attr for a snap realm node (srnode)

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Milind Changire <mchangir@redhat.com>

Merge PR #49970 into main

* refs/pull/49970/head:
mds: force replay sessionmap version

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #50049 into main

* refs/pull/50049/head:
qa: add cephfs_mirror test case to check root dir modes
cephfs_mirror: sync snap dir root mode with remote dir

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

rbd: improve log msgs when failing to enable mirroring on image

Attempting to enable mirroring on an an image within a namespace bails
out without any useful msg.

$ rbd mirror image enable mi_pool/ns_0/big snapshot
2021-11-17T08:33:21.321+0000 7f4b36f9c2c0 -1 librbd::api::Mirror:
  image_enable: cannot enable mirroring in the current pool mirroring mode

Fixes: https://tracker.ceph.com/issues/58895
Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>

qa/cephfs: minor corrections in caps_helper.py

Signed-off-by: Rishabh Dave <ridave@redhat.com>

doc/rados: edit operations/bs-migration (2 of x)

Disambiguate and improve the English language in
doc/rados/operations/bluestore-migration.rst up to but not including the
section called "Whole Host Replacement".

Co-authored-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

mgr/dashboard: force TLS 1.3

Fixes: https://tracker.ceph.com/issues/58942
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

qa: fixup quota values exceeding 64 bit signed int value

The test does the following:

        setfattr -n ceph.quota.max_bytes -v 8388608Ti .

This failes since 8388608Ti exceeds 64 bit signed integer value.
Change that to the approprivate value and expect faliure when
quota is set for this exceeded value.

Fixes: http://tracker.ceph.com/issues/55940
Introduced-by: 733065148250e892fca941cdeaad3e3a8193b0d8
Signed-off-by: Venky Shankar <vshankar@redhat.com>