Merge pull request #47244 from ceph/fix-release-order

doc/releases: Fix quincy order

Merge pull request #47124 from tchaikov/spec-mibs

ceph.spec.in: correct "%files mib" section

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Justin Caratzas <jcaratza@redhat.com>

Merge pull request #47243 from tchaikov/wip-gantt

doc/_ext: sort the release date

Reviewed-by: David Galloway <dgallowa@redhat.com>

doc/releases: Fix quincy order

Signed-off-by: David Galloway <dgallowa@redhat.com>

doc/_ext: sort the release date

so the release in reverse chronologically order, so that the newer
one is listed at the top.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #47237 from rosinL/fix-move

crimson/osd/osd: fix sigsegv caused by reusing the moved object

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

crimson/osd/osd: fix sigsegv caused by reusing the moved objcect

In some case, `m` will be moved and setted nullptr before
`m->get_spg()` be called, causing a segmentation fault.

Signed-off-by: luo rixin <luorixin@huawei.com>

Merge pull request #47206 from cyx1231st/wip-seastore-cleaner-tail-committed

crimson/os/seastore: simplify journal tails

Reviewed-by: Myoungwon Oh <myoungwon.oh@samsung.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: simplify journal tails

* append latest dirty tail as journal delta, in order to simplify and
  drop journal_tail_committed in the cleaner.
* simplify misc journal tails into alloc_tail and dirty_tail, with
  proper renaming.
* move journal-tail recovery logic from cleaner to journal.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: misc cleanups

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: store valid journal tail during mkfs

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #47219 from ceph/sec

16.2.10 and 17.2.2 Release Notes

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #47212 from tchaikov/wip-aligned-free

include/compat: define aligned_free() as a function not a macro

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #46890 from aclamk/wip-fix-54547

os/bluestore: Fix collistion between BlueFS and BlueStore deferred writes

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #36282 from rzarzynski/wip-bl-drop-clone

common/bl, kv, tests: drop MemDB and simplify buffer::ptr and buffer::raw

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #47213 from Matan-B/wip-matanb-snapmapper-pendingnote

PendingReleaseNotes: add a note about SnapMapper key conversion

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #46941 from Matan-B/wip-matanb-make-writeable

osd/PrimaryLogPG: make_writeable nit fixes

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #46561 from NitzanMordhai/wip-nitzan-add-pglog-dups-length

osd, mon: add pglog dups length

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

PendingReleaseNotes: add a note about SnapMapper key coversion

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

doc: 16.2.10 Release notes

Signed-off-by: David Galloway <dgallowa@redhat.com>

doc: 17.2.2 Release notes

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #47001 from yaarith/fix-telemetry-changed-warning

mgr/telemetry: reset health warning after re-opting-in

Merge pull request #47209 from yuvalif/wip-yuval-fix-56170

rgw/lua: make assignment of char* to std::string explicit

reviewed-by: tchaikov, cbodley

include/compat: define aligned_free() as a function not a macro

so it does not get substituted by the preprocessor unconditionally.
this change helps to address the compiling failure when using boost
1.79 using MinGW, like:

../build.deps/mingw/boost/include/boost/align/aligned_allocator.hpp: In member function 'void boost::alignment::aligned_allocator<T, Alignment>::deallocate(boost::alignment::aligned_allocator<T, Alignment>::pointer, boost::alignment::aligned_allocator<T, Alignment>::size_type)':
../src/include/compat.h:352:27: error: '_aligned_free' is not a member of 'boost::alignment'; did you mean 'aligned_free'?
  352 | #define aligned_free(ptr) _aligned_free(ptr)
      |                           ^~~~~~~~~~~~~

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

doc: CVE-2022-0670

Signed-off-by: David Galloway <dgallowa@redhat.com>

doc: Add missing 16.2.9 release

Signed-off-by: David Galloway <dgallowa@redhat.com>

rgw/lua: make assignment of char* to std::string explicit

issue is happening on RHEL8, when using: g++ (GCC) 8.5.0

Fixes: https://tracker.ceph.com/issues/56170
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #47203 from SUSE/wip-buildrequires-ndctl-suse

ceph.spec.in: use BuildRequires libndctl-devel for SUSE distros

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

crimson/os/seastore: drop journal tail info from segment_tail_t

We will write a segment header right after writing a segment tail during
rolling, so there is no need to store journal tail in segment tail.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: move Transaction::src_t to seastore_types.h

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

ceph.spec.in: use BuildRequires libndctl-devel for SUSE distros

Fixes: b6b24852a6d1c6d8e785e15661f3bf29d662417f
Signed-off-by: Tim Serong <tserong@suse.com>

Merge pull request #46917 from ceph/releases-always-main

.readthedocs.yml: Always build latest doc/releases pages

Merge pull request #47002 from rzarzynski/wip-mgr-fix-daemon-server-race

mgr: fix a race condition in DaemonServer::handle_report()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

.readthedocs.yml: Always build latest doc/releases pages

We don't backport PRs merged into doc/releases.  Therefore, when one browses to an older Ceph release version on docs.ceph.com (e.g., https://docs.ceph.com/en/pacific/), the information is out of date at best.

The doc/releases page is only accurate if browsing https://docs.ceph.com/en/latest/, for example.

So this post_checkout command will make sure we've checked out doc/releases from main before building and publishing.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #46933 from clwluvw/s3-website-empty-bucket

rgw: s3website check for bucket before retargeting

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #47009 from adamemerson/wip-chatty-bucket-sync-run

rgw: User friendly progress report on bucket sync run

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #47163 from rhcs-dashboard/get-storage-class-metrics

mgr/dashboard:Get different storage class metrics in Prometheus dashboard

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #46993 from rishabh-d-dave/qa-caps-helper-multiple-errmsgs

qa/cephfs: allow checking for multiple error messages while...

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #47185 from cyx1231st/wip-seastore-backref-block-size

crimson/os/seastore/backref: decouple SegmentManagerGroup

Reviewed-by: Myoungwon Oh <myoungwon.oh@samsung.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #47183 from cyx1231st/wip-seastore-cleaner-journal-head

crimson/os/seastore/async_cleaner: re-introduce journal_header member

Reviewed-by: Myoungwon Oh <myoungwon.oh@samsung.com>

Merge pull request #47125 from xxhdx1985126/wip-56535

crimson/os/seastore/async_cleaner: avoid trimming backrefs when reclaiming space

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

qa/cephfs: allow checking for multiple error messages while...

while negative testing.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #47013 from rishabh-d-dave/qa-caps-method-authorize

qa/cephfs: upgrade method authorize in class Filesystem

Reviewed-by: Nikhilkumar Shelke <nshelke@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #46991 from rishabh-d-dave/qa-caps-helper-rm-path

qa/cephfs: delete path from cmd args after use

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #46998 from ceph/dependabot-npm_and_yarn-src-pybind-mgr-dashboard-frontend-moment-2.29.4

mgr/dashboard: bump moment from 2.29.3 to 2.29.4 in /src/pybind/mgr/dashboard/frontend

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #42335 from rishabh-d-dave/qa-get-keyring

qa: add a method to get keyring for a client_id

Reviewed-by: Venky Shankar <vshankar@redhat.com>

mgr/dashboard:Get "Different Storage Class" metrics in Prometheus dashboard

Get metrics of the different "HDDRule" and "MixedUse" classes of the "Raw Storage" for their ceph VMs. So that Prometheus can scrape the data and display it to them in grafana

Fixes: https://tracker.ceph.com/issues/56625
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

crimson/os/seastore/backref: decouple SegmentManagerGroup

Backref tree is not specific to segments.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: drop unused code

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #47137 from rhcs-dashboard/cephadm-e2e-more-fixes

mgr/dashboard: fix dashboard cephadm e2e errors

Reviewed-by: Pedro González <pegonzal@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

crimson/os/seastore/async_cleaner: re-introduce journal_header member

RBM won't have segments_info_t, it has to maintain the journal_header by
itself.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #47173 from alimaredia/wip-rgw-s3tests-cloudtransitions-iam-section-add

qa: add iam section to cloud transitions config

Reviewed-by: Soumya Koduri <skoduri@redhat.com>

crimson/os/seastore/backref_manager: rename some methods for the sake of readability

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore/async_cleaner: don't trim backref when reclaiming space

Since the current backref cache doesn't invalidate duplicated backrefs any more
and backrefs get trimmed in the exact order of journal seqs, there's no need to trim
backrefs when reclaiming space

Fixes: https://tracker.ceph.com/issues/56535
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #47140 from cyx1231st/wip-seastore-split-fix

crimson/os/seastore: do not assign gen/hint when split a clean extent

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Zhang Song <zhangsong325@gmail.com>

qa: add iam section to cloud transitions config

Signed-off-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #46736 from baergj/fix-log-buffer-resize

log: Make log_max_recent have an effect again

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #46666 from MrFreezeex/batch-blockdb-fix

ceph-volume: fix fast device alloc size on mulitple device

Merge pull request #46928 from linuxbox2/wip-rgwlc-azone

rgwlc: permit lifecycle to reduce data conditionally in archive zone

Merge pull request #47019 from badone/wip-get_or_fail-debug-louder

msg: Log at higher level when Throttle::get_or_fail() fails

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #46866 from myoungwon/wip-fix-53294

osd: return ENOENT if pool information is invalid during tier-flush

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #47116 from chrisphoffman/wip-rbd-56549

librbd: bail from schedule_request_lock() if already lock owner

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #47139 from rishabh-d-dave/cephfs-top-man

doc/man/cephfs-top.rst: add missing options: --delay, --conffile

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

Merge pull request #46889 from rhcs-dashboard/osd-followup-iops

mgr/dashboard: do not recommend throughput for ssd's only cluster

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
Reviewed-by: sunilangadi2 <NOT@FOUND>

Merge pull request #47135 from rhcs-dashboard/doc-default-main

mgr/dashboard: change doc service default release from master to main

Merge pull request #47034 from tchaikov/wip-mon-signness-cleanup

mon: make paxos_size() unsigned

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Kamoltat Sirivadhna <ksirivad@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #47089 from athanatos/sjust/wip-pg-shard-manager

crimson: introduce pg_shard_manager, begin to separate osd-wide from core-local state

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

doc/man/cephfs-top.rst: add missing options: --delay, --conffile

add missing options: --delay, --conffile

Signed-off-by: wangxinyu <wangxinyu@inspur.com>
Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #47152 from tchaikov/wip-crimson-close-later

crimson/net: postpone the close() using yield()

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #46493 from adk3798/tuned-profiles

mgr/cephadm: support for os tuning profiles

Reviewed-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #46924 from hookak/loki-support

mgr/cephadm: fix the loki address in grafana, promtail configuration file

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #46620 from rzarzynski/wip-tools-cot-force-pg-import

tools: COT ignores fsid mismatch when importing PG with --force

Reviewed-by: Nitzan Mordechai <nmordech@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #47044 from alimaredia/wip-rgw-suite-s3test-refactor

add s3tests-brach.yaml for rgw teuthology suites that run s3tests

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #46615 from selvakumaar5496/main

rgw: fix api response in case of get and delete object tagging apis

Reviewed-by: Ali Maredia <amaredia@redhat.com>

crimson/net: postpone the close() using yield()

otherwise we'd erase an element in a container when we are still
iterating through it.

Fixes: https://tracker.ceph.com/issues/56589
Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #47136 from idryomov/wip-48038

qa/suites/rbd: disable workunit timeout for dynamic_features_no_cache

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

librbd: bail from schedule_request_lock() if already lock owner

Race condition may be hit if there are multiple pending locks for the
same image and pending callbacks. Abort exclusive lock process if
already exclusive lock owner.

Fixes: https://tracker.ceph.com/issues/56549
Signed-off-by: Christopher Hoffman <choffman@redhat.com>

Merge pull request #46911 from ifed01/wip-ifed-fix-mempool-cache-other

os/bluestore: fix AU accounting in bluestore_cache_other mempool.

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #47106 from idryomov/wip-56561

rbd: don't default empty pool name unless namespace is specified

Reviewed-by: Christopher Hoffman <choffman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #46741 from Matan-B/wip-matanb-docker-debug

script: CentOS 8 EOL, use archived mirror in ceph-debug-docker.sh

Reviewed-by: Nitzan Mordechai nmordech@redhat.com
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #46375 from ceph/cv-loop-devs

ceph-volume: Optionally consume loop devices

crimson/os/seastore: do not assign gen/hint when split a clean extent

Gen/hint is only required when need to write the extent to disk.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

mgr/dashboard: do not recommend throughput for ssd's only cluster

This is just a bug fix where we recommend the throughput option even if
there are only ssd's are present in the cluster.

Fixes: https://tracker.ceph.com/issues/56413
Signed-off-by: Nizamudeen A <nia@redhat.com>

crimson/os/seastore: introduce PLACEMENT_HINT_NULL

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #47113 from zhscn/fix-split-test

crimson/os/seastore: fix bugs in test_map_existing_extent_concurrent

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

rgwlc: return std::string_view from sal::Zone::get_tier_type()

Valid values are all small strings, often static.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgwlc:  count LCFilter flags towards multi-condition

Found by Soumya Koduri <skoduri@redhat.com>

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgwlc: return real default zone type from sal_dbstore and sal_motr

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgwlc:  update LCFilter::dump_xml(...) to add flags/ArchiveZone

Also updates the location of Prefix, which is supposed to *generate*
as <Filter><Prefix/></Filter>, regardless of how we parsed it.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgwlc: activate lifecycle processing on non-master zones

The basic idea of this change is the same as the proposal by
Ilsoo Byun <ilsoobyun@linecorp.com>, but some details have changed.

The main differences are to use the existing
RGWLC::set(remove)_bucket_config methods, and to use the
RGWBucketInstanceMetadataHandler infrastructue to dispatch
the corresponding calls.  Thank you!

Fixes: https://tracker.ceph.com/issues/44268
Related PR: #33524

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgwlc:  zone-conditional lifecycle processing

Lifecycle rules with the ArchiveZone flag must execute on archive zones,
but must not execute on others.

Fixes: https://tracker.ceph.com/issues/56440
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgwlc: represent lc filter flags as XML elements

Suggested by Casey in review, this makes the XML prettier.

also: fix filter parsing, remove unused code

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgwlc: introduce lifecycle config flags extension

rgwlc: add uint32_t flags bitmap to LCFilter

This is intended to support a concise set of extensions to S3
LifecycleConfiguration, initially, just a flag that indicates a
rule is intended for execution on RGW ArchiveZone.

rgwlc: add machinery to define and recognize LCFilter flags

Add a concept of filter flags to lifecycle filter rules, an RGW
extension.  The initial purpose of flags is to permit marking
specific lifecycle rules as specific to an RGW archive zone, but
other flags could be added in future.

rgwlc: add new unittest_rgw_lc to run internal checks, add a few
valid and invalid lifecycle configuration xml  parses for now.

Fixes: https://tracker.ceph.com/issues/53361
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

mgr/dashboard: fix dashboard cephadm e2e errors

https://tracker.ceph.com/issues/56079
Signed-off-by: Nizamudeen A <nia@redhat.com>

qa/suites/rbd: disable workunit timeout for dynamic_features_no_cache

The I/O workload in this test is xfstests (qa/run_xfstests_qemu.sh)
which isn't subjected to any timeout other than global max_job_time
limit in any other subsuite (e.g. qemu/workloads/qemu_xfstests.yaml).
But here, there is a parallel "op" workload defined as a workunit.
The workunit task has a default timeout of 3 hours which is effectively
imposed on the entire job.  In the "rbd cache = false" configuration,
it's sometimes exceeded.

Fixes: https://tracker.ceph.com/issues/48038
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

mgr/dashboard: change doc service default release from master to main

Signed-off-by: Nizamudeen A <nia@redhat.com>

rgw: Print syncing objects during `bucket sync run`

When --extra-info is provided to radosgw-admin.

This is a bit minimal but they weren't too clear in precisely what
information they wanted, and this should give a definite indication of
forward progress.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Print user friendly progress on `bucket sync init`

When `--extra-info` is provided

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: RGWBucketPipeSyncStatusManger takes optional ostream

`radosgw-admin` provides it when the `--extra-info` option is provided.

It gets stashed in the SyncEnv for later use.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: RGWBucketPipeSyncStatusManager gets zone name on construction

So we can print it in the user friendly messages to come.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>