Merge pull request #46749 from ibm-s390-cloud/fsmap_switch_to_iterator

fsmap: switch to using iterator based loop

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

Merge pull request #47691 from myoungwon/wip-cbjournal-integration

crimson/os/seastore: CBJournal integration works

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #47597 from NitzanMordhai/wip-nitzan-crimson-watch-list-empty-return-code

crimson/osd: watch list empty return incorrect

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Sam Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #47670 from rzarzynski/wip-common-ftbfs-systemd

common: fix a FTBFS in Journald.cc when building without systemd

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #47680 from tchaikov/wip-crimson-cleanups

crimson: cleanups to use `IsFullKey` concept

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: introduce finish_commit to update tails after trim transaction

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore/cbjournal: rename journal_tail to dirty_tail

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore/cbjournal: make cbjournal to store alloc_tail

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore/cbjournal: make journal_tail and written_to to use journal_seq_t instead of rbm_abs_addr

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore: divide the role of open_device_read_header into device.mount()
and open_for_mount()

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

Merge pull request #47615 from adamemerson/wip-objv-doc

rgw: Document `RGWOvjVersionTracker`

Reviewed-by: Casey Bodley <cbodley@redhat.com>

crimson/os: make _node_fields_013_t::insert_at() use IsFullKey

less repeatings this way

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #47288 from batrick/i56274

debian: add libsqlite3-mod-ceph dependency for ceph-mgr

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

crimson/os: define *::from_key() with IsFullKey

more compact this way.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

crimson: move helper classes down

so we can use the concept of `IsFullKey` for defining accessors in the
incoming commit.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

crimson: define encode_key() with IsFullKey constrait

also, let the compiler to deduce the type of the template parameter,
less repeating and less error-prone this way.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

common: fix a FTBFS in Journald.cc when building without systemd

This came from Josh Salomon:

```
/home/jsalomon/src/prim_score/src/common/Journald.h:93:24: error: ‘SubsystemMap’ does not name a type
   93 |   JournaldLogger(const SubsystemMap *) {}
      |                        ^~~~~~~~~~~~
/home/jsalomon/src/prim_score/src/common/Journald.h:94:23: error: ‘Entry’ does not name a type
   94 |   int log_entry(const Entry &) {
      |
```

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

Merge pull request #47633 from bosc0/alerts_multicluster

prometheus: add multicluster support to alerts

Merge pull request #47554 from Matan-B/wip-matanb-c-rados-python

qa/suite/crimson-rados: Fix rados_python test

Reviewed by: Samuel Just <sjust@redhat.com>
Reviewed by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed by: Nitzan Mordechai <nmordech@redhat.com>

Merge pull request #47653 from Matan-B/wip-matanb-snapmapper-pending-typo

PendingReleaseNotes: fix typo in 15.2.17

Reviewed-by: Nitzan Mordechai <nmordech@redhat.com>

Merge pull request #47392 from Matan-B/wip-matanb-memstore-subsys

common: MemStore debug subsystem

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #47643 from aisakaki/wip-omap-merge

crimson/os/seastore/omap: optimize merge policy

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #47558 from rzarzynski/wip-crimson-extradbgs

crimson/{net,osd}: add debugs after bughunt's investigations

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge PR #47600 into main

* refs/pull/47600/head:
doc: document debugging for libcephsqlite

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #47523 from adk3798/oracle-linux-support

cephadm: support for Oracle Linux 8

Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Redouane Kachach <rkachach@redhat.com>

PendingReleaseNotes: fix typo in 15.2.17

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #47422 from adamemerson/wip-coroutine-cleaver

rgw: Break Full and Incremental sync out into their own classes

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/suites: Reduce rados_python time out

This test runs for few minutes,
reducing timeout from 3h to 1h to avoid hanging jobs.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

qa/crimson-rados: add 'wait' attribute

wait_for_complete tests currently hanging with crimson-osd
and are temporary disabled in Crimson suite.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

ceph-mixin: Remove jsonnet building

Signed-off-by: Aswin Toni <aswin.toni@cern.ch>

prometheus: add multicluster support to alerts

Signed-off-by: Aswin Toni <aswin.toni@cern.ch>

crimson/os/seastore/omap: optimize merge policy

in some corner cases on single key erase sence,
for e.g. we have three nodes which are brothers:
1 2 3 4 5 | 6 7 8 | 9 10 11, node min size is 3.
When erasing 11 will trigger balance instead of
merge, the middle nodes will adjust to 6 7, this
will not ensure the all nodes above min size.
This corner case might not be harmful but if we
want to erase 10,11 at one time (range remove wip
need this), this will trigger balance in past
policy which will make both two nodes below min
size and we might need at least one more merge.
This is actually because we cannot ensure a node
to merge in tree is above min size.

Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>

crimson/os/seastore/omap: code cleanup

Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>

Merge pull request #47491 from cyx1231st/wip-seastore-fix-return-future

crimson/os/seastore/transaction_manager: return future when enoent

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #46855 from ronen-fr/wip-rf-pg-updates

osd/scrub: Scrubbing PGs now periodically update their stats

Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Aishwarya Mathuria <amathuri@redhat.com>

Merge pull request #47553 from cyx1231st/wip-seastore-cleanup-reclaim

crimson/os/seastore: cleanups to gc reclaim

Reviewed-by: Myoungwon Oh <myoungwon.oh@samsung.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>
Reviewed-by: Zhang Song <zhangsong325@gmail.com>

Merge pull request #47477 from mchangir/doc-clarify-start-time-in-snap-schedule

doc/cephfs: note regarding start time time zone

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

Merge pull request #47099 from joscollin/wip-B56537-wrong-rsp-wsp-values

cephfs-top: fix the rsp/wsp display

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Neeraj Pratap Singh <neesingh@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

Merge pull request #47365 from adk3798/explicit-host-no-schedule

mgr/cephadm: fix handling of draining hosts with explicit placement specs

Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #47529 from hookak/container-args

cephadm: Remove duplicated process args in promtail and loki

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #47545 from adk3798/grafana-password

mgr/cephadm: set dashboard grafana-api-password when user provides one

Reviewed-by: Michael Fritch <mfritch@suse.com>

osd/scrub: PGs that are scrubbing now update stats periodically

Added periodic calls to all PGs for which the OSD is the primary,
asking for a scrub statistics update.

This allows operator queries (e.g. 'pg dump pgs') to present
up-to-date scrub duration, "scrub is blocked" duration, etc'.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #47565 from adk3798/unit-test-runtime

mgr/cephadm: mock CephadmHttpServer run in unit tests

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Redouane Kachach <rkachach@redhat.com>

doc/cephfs: note regarding start time time zone

Fixes: https://tracker.ceph.com/issues/56730
Signed-off-by: Milind Changire <mchangir@redhat.com>

Merge pull request #47119 from rhcs-dashboard/image-refs-bug

mgr/dashboard: fix _rbd_image_refs caching

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #47609 from cbodley/wip-install-deps-rhel

install-deps: remove outdated rhel8 devtool packages

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #45628 from PepperJo/comparev_and_writev_v7

librbd: add compare-and-write compare length limit, remove sector size restriction

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #47617 from tchaikov/wip-cmake-jsonnet

cmake: set $PATH for tests using jsonnet tools

Reviewed-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@cern.ch>

Merge pull request #47618 from xxhdx1985126/wip-journal-move

crimson/os/seastore/journal: move bufferlists to reduce the amount of memory copies

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/journal: move bufferlists to reduce the amount of memory copies

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

cmake: set $PATH for tests using jsonnet tools

otherwise they would not able to find executables installed into
${CMAKE_CURRENT_BINARY_DIR}.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #47594 from tchaikov/wip-cmake-win32

cmake: enforce gcc-10 for win32 port

Reviewed-by: Casey Bodley <cbodley@redhat.com>

cephadm: Remove duplicated process args in promtail and loki(config.file)

Signed-off-by: jinhong.kim <jinhong.kim0@navercorp.com>

rgw: Document `RGWOvjVersionTracker`

Give some guidelines as to what it's for and how to use it.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #47612 from cbodley/wip-rgw-revert-lua-filter

Revert "rgw/lua: allow read access to object data"

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Revert "rgw/lua: allow read access to object data"

This reverts commit f4501f69701d74caa5c0da0b7247eed80c1f2a3f.

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #47564 from cbodley/wip-build-disable-motr

build: temporarily disable WITH_RADOSGW_MOTR in make check

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

install-deps: remove outdated rhel8 devtool packages

these were accidentally resurrected in 65b1a13139873c1525456e05cd00aeb64da4e881

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #47513 from Dazul/cephadm-allow-multiple-vips-keepalived

Cephadm: Allow multiple virtual IP addresses for keepalived and haproxy

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

doc: document debugging for libcephsqlite

Fixes: https://tracker.ceph.com/issues/57127
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #47539 from guits/fix-zap-msg-cephadm

cephadm: fix incorrect warning

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #47562 from guits/c-v-fix_ceph_device

cephadm/ceph-volume: fix rm-cluster --zap

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #46550 from yuvalif/wip-yuval-lua-filter

rgw/lua: allow access to object data

reviwed-by: cbodley, anthonyeleven

Merge pull request #47420 from yuvalif/wip-yuval-remove-metadata-entry

rgw: remove metadata entry when remote fetch reply with ENOENT

reviewed-by: cbodley

Merge pull request #47362 from dang/wip-dang-zipper-cleanup

RGW - Zipper - Init cleanup

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #47010 from yuvalif/wip-yuval-fix-lua-post-ctx

rgw/lua: post request context does not work

reviewed-by: cbodley, zenomri

Merge pull request #46304 from yuvalif/wip-yuval-fix-53810

rgw/lua: don't create the script manager per request

reviewed-by: cbodley, dang, mkogan1

Merge pull request #47411 from linuxbox2/wip-56997

rgwlc/sync: avoid calling merge-and-store-attrs from remove_bucket_co…

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

crimson/osd: watch list empty return incorrect
fixing retun respon encode when watcher list is empty

Signed-off-by: Nitzan Mordechai <nmordec@redhat.com>

test/librbd: various fixups for compare-and-write test cases

- missing EILSEQ asserts in IO and IOWithIOHint
- AioCompletion leak in CompareTooSmall and WriteTooSmall
- skip StripeUnitUnaligned and TooLarge on the PWL cache as it wasn't
  limited by the 512-byte sector limit before and isn't limited by the
  stripe unit limit now

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd/cache/pwl: adjust compare-and-write compare substringing

cmp_bl can no longer be shorter than read_bl (i.e. image extent).
This is caught very early at the API level, see commit af96e6dae3f4
("librbd: make C++ cmp&write semantics equal to C API").

However, cmp_bl can still be longer than read_bl and in that case
it should be capped by the image extent length.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd: discard cache state if compare-and-write writes to disk

Otherwise this is obviously broken with rbd_cache_policy = writeback or
rbd_cache_policy = writethrough as the write is done on the OSD, beneath
ObjectCacher.  This went unnoticed because the original (and so far the
only known) compare-and-write user, tcmu-runner rbd driver, disables the
cache altogether.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

cmake: enforce gcc-10 for win32 port

we only have GCC-10 targeting MinGW at the time of writing, and it
is good enough for compiling the Winows port.

so let's relax the requirement to enable the windows build.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #47402 from NitzanMordhai/wip-nitzan-crimson-rbd-cls-watcher-list

crimson/osd: Port rgw watcher list

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #47552 from myoungwon/wip-fix-seastore-trm-test

test/crimson/seastore: unittest-transaction-manager fixes

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/transaction_manager: don't return invalid extent in get_extents_if_live()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/async_cleaner: cleanup and add logs to gc trim

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: document how get_extents_if_live deals with extents split

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/transaction_manager: cleanups to get_extents_if_live

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/async_cleaner: cleanup, introduce do_gc_reclaim()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/async_cleaner: move retrieve_backref_extents_in_range to weak transaction

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/async_cleaner: cleanup gc_reclaim_space()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/async_cleaner: use weak transaction to get backref mappings

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #47580 from zdover23/wip-doc-2022-08-14-rados-operations-operating-prompt-update

doc/rados/operations: add prompts to operating.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #47570 from idryomov/wip-57107

rbd: find_action() should sort actions first

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #46891 from ronen-fr/wip-rf-redo-noscrub

osd/scrub: modify scrub behaviour under no-scrub

Reviewed-by: Aishwarya Mathuria <amathuri@redhat.com>

doc/rados/operations: add prompts to operating.rst

This commit adds ".. prompt:: bash $"-style prompts to operating.rst.
This brings this file up to the standard established in 2020 when
Kefu added support for the ".. prompt::" directive.

This commit is a part of an initiative to modernize the presentation
of all BASH commands in the RADOS documentation.

The progress of this project can be tracked here:
https://tracker.ceph.com/issues/57108

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #44436 from nh2/patch-2

doc/man/rbd: Mention changed `bluestore_min_alloc_size`

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

doc/man/rbd: Mention changed `bluestore_min_alloc_size`

This change was done in: https://github.com/ceph/ceph/pull/34588

Signed-off-by: Niklas Hambüchen <mail@nh2.me>

Merge pull request #47272 from dheart-joe/master

test/osd/TestPGLog: Fix confusing description between log and olog.

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #46318 from rzarzynski/wip-mon-fix-mgrmap-rc

mon: fix a race between `mgr fail` and MgrMonitor::prepare_beacon()

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #44836 from NitzanMordhai/wip-nitzan-osd-status-format-json

pybind/mgr: fix "osd status" command output with format json

Reviewed-by: Kotresh HR <khiremat@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

PendingReleaseNotes: add rbd compare-and-write notes

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd: make C++ cmp&write semantics equal to C API

The compare and write C++ API (both sync and async) does not check
compare bufferlist length and executes compare ops of bufferlist length
size, even if (write) "len" is smaller than bufferlist length.
This patch changes this behavior by always issuing compares and
writes of "len" size to match the C API. The bufferlist length
can be greater than "len" for both compare and write but only
"len" bytes are compared and written. If the bufferlist length
is smaller than "len" the call will fail.

Signed-off-by: Jonas Pfefferle <pepperjo@japf.ch>

librbd: remove cmp&write sector size restriction

This patch removes the compare and write max sector size len
restriction. We can allow up to stripe unit size accesses if
the access is aligned properly.
To allow larger size compare and write requests in the
journal we split the buffers like we do for writes now.

Signed-off-by: Jonas Pfefferle <pepperjo@japf.ch>

librbd: new cmp&write C API tests

Test correct mismatch offset is returned.
Test no data is written when compare fails.
Test correct data is written when compare succeeds.

Signed-off-by: Jonas Pfefferle <pepperjo@japf.ch>

Merge pull request #47568 from tchaikov/wip-dashboard-more-itertools

mgr/dashboard: bump up more-itertools

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #44710 from joscollin/wip-F40633-dump-recent-extraordinary-events

mds: dump recent log events for extraordinary events

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Milind Changire <mchangir@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #47067 from phlogistonjohn/jjm-cephfs-clone-config-1

mgr/volumes: prevent intermittent ParsingError failure in "clone cancel"

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>
Reviewed-by: Nikhilkumar Shelke <nshelke@redhat.com>

rbd: remove incorrect use of std::includes()

- std::includes() requires sorted ranges but command specs aren't
  sorted
- std::includes() purpose is to check whether the second range is
  a subsequence of the first range but here the size of the second
  range is always equal to the size of the first range, which means
  that, had the ranges been sorted, std::includes() would have checked
  straight equality

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>