crimson/osd: inform in logs that TMAP is not supported

Simpler this way than to investigate for `EOPNOTSUPP`.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/net: add debug on SocketMessenger's destruction

The `logic_name` will hopefully help if the assertion in the dtor
fails (which actually started happing).

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #47334 from s0nea/wip-osd-objectstore-types-fix

monitoring/ceph-mixin: OSD overview typo fix

Reviewed-by: MrFreezeex <NOT@FOUND>
Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #45602 from soumyakoduri/wip-skoduri-dbstore-obj-versioning

rgw/dbstore: Object versioning feature support

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #47345 from cyx1231st/wip-seastore-log-improvements

crimson/os/seastore: misc fixes and log improvements

Reviewed-by: Myoungwon Oh <myoungwon.oh@samsung.com>

Merge pull request #47280 from adk3798/remove-error-upgrade-unpause

mgr/cephadm: clear error message when resuming upgrade

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #47318 from adk3798/rescan-parser-name

cephadm: give disk rescan its own parser object

Reviewed-by: Paul Cuzner <pcuzner@redhat.com>

Merge pull request #47338 from ksperis/master

cephadm: Fix repo_gpgkey should return 2 vars

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #47361 from guits/fix-simple-scan

ceph-volume: fix simple scan

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #47367 from tchaikov/wip-osd-capture-this

osd/PG: capture this explicitly

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

osd/PG: capture variables explicitly

to silence warning if compiled with -std=c++20:

src/osd/PG.h:571:7: warning: implicit capture of ‘this’ via ‘[=]’ is
deprecated in C++20 [-Wdeprecated]
  571 |       [=](auto &history, auto &stats) {
      |       ^

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #47300 from kotreshhr/fips-md5-issue

mgr/volumes: Fix subvolume creation in FIPS enabled system.

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #47202 from SUSE/wip-fix-56658

cephfs-shell: move source to separate subdirectory

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Nikhilkumar Shelke <nshelke@redhat.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #47246 from liu-chunmei/crimson-test-case

crimson/test: comment out EC test cases and some cases not supported now

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

crimson/test: skip EC class SetUp

otherwise crimson will hang

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

qa: Fix use of 'sudo' args

https://tracker.ceph.com/issues/56727
Signed-off-by: Kotresh HR <khiremat@redhat.com>

mgr/cephadm: clear error message when resuming upgrade

the message field in the output of "ceph orch upgrade status"
will first take the value of the error field of the UpgradeState,
and if only if it blank/None, display an info string we periodically
update throughout the upgrade with useful info such as that
we're upgrading a daemon of a particular type or pulling an image
on a certain host. When an upgrade fails, we set the error field
of the UpgradeState, pause the upgrade and raise a health warning.
Sometimes, the user is able to resolve the issue and simply resume
the upgrade. The issue here is, in that case, the error field of
the UpgradeState is still set, so instead of seeing the useful info
messages, it will continue to display an error message that may
no longer be relevant. By emptying the error field of the UpgradeState
when upgrades are resumed, we return to normal behavior of
displaying the info string, and will only show another error message
if another error actually occurs.

Fixes: https://tracker.ceph.com/issues/56714
Signed-off-by: Adam King <adking@redhat.com>

Merge pull request #46248 from linuxbox2/wip-rgwlc-tryharder

rgwlc: try hard to get lc locks when saving policy

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #47363 from tchaikov/boost-phoenix

mgr/MgrCap: include <boost/phoenix.hpp>

Reviewed-by: Neha Ojha <nojha@redhat.com>

mgr/volumes: Fix subvolume creation in FIPS enabled system.

The md5 checksum is used in the construction of legacy
subvolume config filename. It's not used for security reason.
Hence marking the 'usedforsecurity' flag to false to
make it FIPs compliant.

The usage of md5 was always in there. The commit 373a04cf734
made it to get exercised in 'open_subvol' which is pre-requisite
for all the subvolume operations and hence subvolume
creation has failed.

Fixes: https://tracker.ceph.com/issues/56727
Signed-off-by: Kotresh HR <khiremat@redhat.com>

rgwlc:  define RGWLC::thr_name() and use it to name lc lock cookies

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #47336 from tchaikov/wip-ceph-dencoder-for-dev

tools/ceph-dencoder: register dencoders in "lib" in dev env

Reviewed-by: Xiubo Li <xiubli@redhat.com>

rgwlc: try hard to get lc locks when saving policy

During active lifecycle processing, there can be high contention for
lifecycle shard locks.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #47187 from cfsnyder/wip-cfsnyder-56646

rgw: optimizations for handling ECANCELED errors from within get_obj_state

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #47343 from ceph/ljflores-patch-1

doc/releases: write a note about the "invalid regex" mgr bug

doc/releases: add release note about "invalid regex" bug in mgr

Signed-off-by: Laura Flores <lflores@redhat.com>

mgr/MgrCap: include <boost/phoenix.hpp>

instead of include `spirit/include/phoenix.hpp`, include
`<boost/phoenix.hpp>` to silence the warning from boost, like:

/opt/ceph/include/boost/spirit/include/phoenix.hpp:12:1: note: ‘#pragma
message: This header is deprecated. Use <boost/phoenix.hpp> instead.’
   12 | BOOST_HEADER_DEPRECATED("<boost/phoenix.hpp>")
      | ^~~~~~~~~~~~~~~~~~~~~~~

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

ceph-volume: fix simple scan

When the class `Device` is instantiated with a path instead of a
block device, it fails like following.

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.6/site-packages/ceph_volume/util/device.py", line 130, in __init__
    self._parse()
  File "/usr/lib/python3.6/site-packages/ceph_volume/util/device.py", line 233, in _parse
    self.ceph_device = disk.has_bluestore_label(self.path)
  File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 906, in has_bluestore_label
    with open(device_path, "rb") as fd:
IsADirectoryError: [Errno 21] Is a directory: '/var/lib/ceph/osd/ceph-0/'
```

passing a path instead of a block device is valid, `simple scan` needs it.

Fixes: https://tracker.ceph.com/issues/56969
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #47190 from dang/wip-dang-zipper-filter

RGW - Zipper - Base filter implementation

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #47347 from anthonyeleven/anthonyeleven/essthree

doc/radosgw: Uppercase s3

Reviewed-by: Zac Dover <zac.dover@gmail.com>

crimson/os/seastore: misc fixes and log adjustments

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: unify segment info formatting

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: collect stats during replay

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

cephadm: ceph.manual.gpg cleanup

Cleanup file /etc/apt/trusted.gpg.d/ceph.manual.gpg when repo is removed

Signed-off-by: Laurent Barbe <laurent@ksperis.com>

doc/radosgw: Uppercase s3

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

Merge pull request #47260 from rhcs-dashboard/fix-frontend-monitor-port-fields

mgr/dashboard: add required validation for frontend and monitor port

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #47284 from anthonyeleven/anthonyeleven/prom-localrules-tweaks

monitoring/ceph-mixin: clean up prometheus_alerts.yml

crimson/os/seastore/async_cleaner: improve log gc status

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson/seastore/transaction_manager_test_state: improve logs

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #47328 from liu-chunmei/crimson-cmpext

crimson: for cmp_fail error, return -MAX_ERRNO-index

Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/test: check env variable "CRIMSON_COMPAT"

when "CRIMSON_COMPAT" is set, to skip EC and some test cases.

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

monitoring/ceph-mixin: clean up prometheus_alerts.yml

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

Merge pull request #47291 from cyx1231st/wip-seastore-journal-head

crimson/os/seastore: fixes to trimming

Reviewed-by: Myoungwon Oh <myoungwon.oh@samsung.com>

Merge pull request #47275 from myoungwon/wip-rbm-cleanup

seastore: RBM clean-ups

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson: for cmp_fail error, return -MAX_ERRNO-index

instead of cmp_fail error code.

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

Merge pull request #47308 from mdw-at-linuxbox/wip-master-implanon

rgw: better tenant id from the uri on anonymous access

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #47170 from guits/dep-dev-inventory

ceph-volume: report slave devices in inventory

ceph-volume/tests: migrate to pyfakefs

ceph-volume unit tests shouldn't actually create contents on the
filesystem from where it runs (even though they are written in a tmp
dir), let's use pyfakefs.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

ceph-volume/tests: fix test_path_is_valid()

When ceph-volume tests are run from a host where there's no `/dev/sda`
device, this test fails.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

ceph-volume: report slave devices in inventory

`ceph-volume inventory` currently reports the following:

```

Device Path               Size        rotates available Model name
/dev/mapper/mpatha        50.00 GB    True    True
/dev/mapper/mpathaa       6.00 GB     True    True
/dev/mapper/mpathab       6.00 GB     True    True
/dev/mapper/mpathac       6.00 GB     True    True
/dev/mapper/mpathad       6.00 GB     True    True
/dev/mapper/mpathae       6.00 GB     True    True
```

whereas something like following would be useful:

```

Device Path               Size         Device nodes    rotates available Model name
/dev/mapper/mpatha        50.00 GB     sdf,sde         True    True
/dev/mapper/mpathaa       6.00 GB      sdbe,sdat       True    True
/dev/mapper/mpathab       6.00 GB      sdav,sdbf       True    True
/dev/mapper/mpathac       6.00 GB      sdbb,sdbl       True    True
/dev/mapper/mpathad       6.00 GB      sdas,sdbc       True    True
/dev/mapper/mpathae       6.00 GB      sdax,sdbh       True    True
```

Fixes: https://tracker.ceph.com/issues/56624
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #47169 from guits/improve-mpath-reporting

ceph-volume: improve mpath devices reporting

Merge pull request #46893 from guits/ceph-volume-symlinks-guits

ceph-volume: allow symlinks as devices

cephadm: Fix repo_gpgkey should return 2 vars

when option --gpg-url is specified, the name used for the gpg filename is missing and throws an exception
this adds the string "manual" to the gpg key : /etc/apt/trusted.gpg.d/ceph.manual.gpg

Fixes: https://tracker.ceph.com/issues/56950
Signed-off-by: Laurent Barbe <laurent@ksperis.com>

tools/ceph-dencoder: register dencoders in "lib" in dev env

if "CMakeCache.txt" is found in current directory, try to load
dencoder shared libraries  from ./lib. this heuristics is used by
`ceph.in` also for relaunching itself to get access to python
bindings.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #47310 from tchaikov/ceph-context-formatting

common/ceph_context: fix the formatting

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

monitoring/ceph-mixin: OSD overview typo fix

Correct a wrongly set bracket on ceph-dashboard -> OSD Overview ->
OSD Objectstore Types resulting in a parser error.

Fixes: https://tracker.ceph.com/issues/56948
Signed-off-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #46996 from rhcs-dashboard/html-injection-bug

mgr/dashboard: added pattern validaton for form input

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #46970 from ronen-fr/wip-rf-unlocked-logs

osd/scrub: pre-prepare the description text of the current state

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #46024 from sirtoobii/patch-2

doc/radosgw/STS: sts_key and user capabilities

Reviewed-by: Zac Dover <zac.dover@gmail.com>

seastore/rbm: rename nvme_device to random_block_device

rename nvme_device namespace to random_block_device, and
introduce random_block_device::nvme

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

Merge pull request #44039 from richael02/update-isal-crypto-source

.gitmodules: update isa-l_crypto source address

Reviewed-by: Yibo Cai <yibo.cai@arm.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

cephadm: give disk rescan its own parser object

It was using the same name as the agent. Missed this
in https://github.com/ceph/ceph/pull/46420/commits/6584c49bdf254e8750d23ed529478e6a9289e12f

Signed-off-by: Adam King <adking@redhat.com>

Merge pull request #46420 from pcuzner/add-disk-rescan

mgr/cephadm: Add disk rescan feature to the orchestrator

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

Merge pull request #47286 from tchaikov/wip-fmt-ranges

osd: use fmt/ranges.h for printing ranges

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #46883 from adk3798/custom-config

mgr/cephadm: support for miscellaneous config files for daemons

Reviewed-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #46955 from hookak/monitoring-stack

cephadm: Add promtail and loki when bootstrap

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #47088 from adk3798/cephadm-logging

cephadm: reduce spam to cephadm.log

Reviewed-by: John Mulligan <jmulligan@redhat.com>

common/ceph_context: fix the formatting

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #47178 from adk3798/logrotate-owner

cephadm: add "su root root" to cephadm.log logrotate config

Reviewed-by: Guillaume Abrioux gabrioux@redhat.com

Merge pull request #46618 from aclamk/wip-dout-for-alien

crimson/alienstore: Add douts

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #47269 from adk3798/fix-repo-test

qa/workunits/cephadm: update test_repos master -> main

Reviewed-by: Guillaume Abrioux gabrioux@redhat.com

.gitmodules: update isa-l_crypto source address

Update address to https://github.com/intel/isa-l_crypto for it's
moved under intel repo.

Signed-off-by: Richael Zhuang <richael.zhuang@arm.com>

Merge pull request #44499 from SMIL-Infra/fix-missed-lease-ack

osd/PeeringState: fix missed recheck_readable from laggy

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #45135 from lmlg/fix-32-bit

rgw,test: Fix build errors in 32 bit platforms

Reviewed-by: Soumya Koduri <skoduri@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #43005 from wxypro/timeout_conn_msg_opt

msg: Add target_addr to log when connections timed out

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #44040 from richael02/update-isal-crypto-version

crypto: update isa-l_crypto version

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #44762 from Zhiwei-Dai/wip-zlib-compressor-isal-for-aarch64

compressor/zlib: isa-l optimization for zlib algorithm on aarch64

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

rgw: better tenant id from the uri on anonymous access

When anonymous tries access public bucket, it gets 404,
because rgw doesn't check tenant correctly.

A previous fix for this broke legacy implicit tenants,
because it didn't check for anonymous access.  This version
restricts its behavior to the anonymous user.

Fixes: https://tracker.ceph.com/issues/48001 https://tracker.ceph.com/issues/48382
Original fix by
Author: Rafał Wądołowski <rafal@rafal.net.pl>
Signed-off-by: Rafał Wądołowski <rwadolowski@cloudferro.com>
This fix
Signed-off-by: Marcus Watts <mwatts@redhat.com>

RGW - Zipper - Add temporary Filter testing

In order to test filters, we need to be able to instantiate one and
stack it over a store.  Add a temporary config option to specify a
filter that can be used for testing.  This will be replaced with a
better system once the loadable module work is done.

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

RGW - Zipper - Filter base implementation

This is the base implementation for filters.  It passes each call on
each object through to the next layer down.  This allows filters to
subclass these, and only implement the bits they care about, leaving the
rest as pass-through.

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

RGW - Zipper - make Serializer use unique_ptr

Serializer needs to use a unique_ptr like the rest of the SAL API.

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

ceph-volume: support symlinks as devices

This makes ceph-volume support passing symlinks as devices.

Fixes: https://tracker.ceph.com/issues/49103
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

ceph-volume: drop self.abspath in Device()

seems useless to have both self.path and self.abspath attributes.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #47167 from guits/drop-bs-tool-call

ceph-volume: drop `ceph-bluestore-tool` call in Device._parse()

seastore/rbm: rename works

1. rename NVMeBlockDevice to RBMDevice
2. rename PosixNVMeDevice to NVMeBlockDevice

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

Merge pull request #47290 from aisakaki/wip-omap-clear

crimson/os/seastore/omap: remove unnecessary extent loads when clearing

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

crimson/os/seastore: set tail to the last-known journal head under the full trim

Rather than setting to the start_seq of the current record during replay
time.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/async_cleaner: abort when the alloc_tail goes backwards

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: distinguish from commit-time and submit-time sequences

Also, the committed journal head should be updated atomically with the
rest of the complete-commit logics.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

ceph-volume: fix is_ceph_disk_member()

`dev['NAME']` can't match `part` given that it's the name of the
parent device being compared to the partition name.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

osd/scrub: pre-prepare the description text of the current state

The existing code accesses the states of the scrub FSM via the
standard "state iterator", which is definitely not thread-safe.
It is also a bit time consuming.

As it happened, new code that tried to create the state string
while not holding the PG lock - crashed the OSD.
This PR changes the relevant lines to allow for safe and fast
access to the state description text.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

seastore/rbm: rename NVMeManager to BlockRBManager

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

Merge pull request #46755 from aravind-wdc/zns

Crimson: Update zns support

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

seastore/rbm: use DEBUG instead of logger().debug()

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore/omap: remove unnecessary extent loads when clearing.

Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>

Merge pull request #47285 from rzarzynski/wip-crimson-monc-discard_result

crimson/monc: replace then_unpack() with discard_result() if possible

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

rgw,test: Fix build errors in 32 bit platforms

When building on 32 bit platforms (tested on armhf), 2 errors were detected:
The first involves the compiler being unable to disambiguate a call to
deserialize an integer (apparently because there isn't a specialization for it).
The second involves invalid usage of the builtin reinterpret_cast.

Signed-off-by: Luciano Lo Giudice <luciano.logiudice@canonical.com>

osd: use fmt/ranges.h for printing ranges

now that fmt/ranges.h provides a generic mechinary for printing
ranges. let's just use it.

the output format is changed a little bit when it comes to
set and map, but as long as it makes sense, and we don't use
a tool to parse the logging messages for extracting information
from the messages printed by fmt::format(), we should be safe.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #47287 from tchaikov/wip-crimson-cleanup

crimson/osd: remove unused variable

Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/osd: remove unused variable

Signed-off-by: Kefu Chai <tchaikov@gmail.com>