osd/PG: remove unused function.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #36935 from mgfritch/cephadm-ceph-service

mgr/cephadm: introduce `CephService` class

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #37070 from zdover23/wip-doc-3-hacking-into-docs

doc/dev: add HACKING.rst to Dev Guide

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #37089 from cbodley/wip-cmake-osd-fifo

cmake: add cls_fifo as a dependency of osd

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Soumya Koduri <skoduri@redhat.com>
Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #36980 from adk3798/fix-iscsi

cephadm: fix iSCSI unit.run file

Reviewed-by: Matthew Oliver <moliver@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #36747 from varshar16/wip-doc-update-about-priority-check

doc/developer_guide: Update about '--force-priority' flag

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.de>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #36313 from changchengx/fix_warning

rgw: fix vector index warning

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

cmake: add cls_fifo as a dependency of osd

otherwise cls_fifo is not built by 'make vstart'

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #37075 from tchaikov/wip-doc-dev

doc/dev: fix broken link and fix the codeblocks' formatting

Reviewed-by: Zac Dover <zac.dover@gmail.com>

doc/dev: add HACKING.rst to Dev Guide

This commit exposes HACKING.rst through the
docs website. Formerly, HACKING.rst was just
a file that existed at /src/pybind/mgr/dashboard/
HACKING.rst. This commit also updates README.rst
to point to the new location of HACKING.rst.

Fixes: https://tracker.ceph.com/issues/47230
Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #37071 from votdev/cleanup_servicespec_code

cephadm: Remove assignment to member variable in ServiceSpecs

Reviewed-by: Joshua Schmid <jschmid@suse.de>

Merge PR #37046 into master

* refs/pull/37046/head:
cephadm: unit.run: create /var/run/ceph/$FSID before doing anything else

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #36766 from jmolmo/cephadm_log

cephadm: log to file

Merge pull request #36798 from dang/wip-dang-zipper-8

Wip dang zipper 8

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #37039 from jan--f/c-v-no-fstrings

ceph-volume: avoid format strings for now

Merge pull request #36855 from yaarith/fix-devid-split

mgr/telemetry: fix device id splitting when anonymizing serial

Reviewed-by: Neha Ojha <nojha@redhat.com>

ceph-volume: avoid format strings for now

Fixes: https://tracker.ceph.com/issues/47354
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #37005 from tchaikov/wip-buffer-hyprcombine

common/buffer: check for null before deref

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc/dev: reformat the sample for tree structure of a test suite

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/dev: use appropriate directive for code blocks

* use "code-block:: yaml" for qa suite samples
* use "prompt:: bash $" for shell commands using a non-root user

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/dev: update supported distros of ceph

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/dev: drop unused label

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/dev: fix broken link

otherwise "make check" would be a link pointing to
https://wiki.sepia.ceph.com/doku.php

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37050 from idryomov/wip-krbd-namespace-symlinks

rbd: include RADOS namespace in krbd symlinks

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37073 from tchaikov/wip-crimson-test-config

test/crimson: fix racing in testing

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #36961 from jdurgin/wip-auth-perf

mon/AuthMonitor: speed up caps updates

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: add unittest_seastar_config as a ceph test

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/crimson: fix racing in testing

before this change, unittest_seastar_config set the specified option to
the shard id in parallel and expects that the values of the option are
consistent across all shards after setting the value on all shards. but
sharded::invoke_on_all() is executed in parallel, and so does
ConfigProxy::do_change(), so there are actually "n x n" continuations
racing each other. the order of the continuations storing the setting
cannot be determined. so we cannot expected that the last batch of
continuations hitting shareds store the same value.

in this change, only a single `conf.set_val()` call is performed with
a known value. and this value is checked with the values stored on all
shards.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/crimson: use seastar::make_ready_future()

instead of seastar::now(). as seastar::now() was moved into
seastar/util/laster.hh.

this change is a leftover of aa28b51f3c5434253c1788927100fb0b6e41bdbb

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36962 from tchaikov/wip-qa-py3-cleanup

qa: py3 cleanups

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #37054 from cbodley/wip-47306

rgw: ObjectCache::put() clears stale objv

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Mark Kogan <mkogan@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #36619 from cbodley/wip-46734

dmclock: update submodule to fix hang on thread shutdown

Merge pull request #35548 from adamemerson/wip-datalog-fifo

FIFO the Datalog!

cephadm: Remove assignment to member variable in ServiceSpecs

Remove unnecessary assignment to member variable 'preview_only', this is done in the constructor of the derived ServiceSpec class.

Fixes: https://tracker.ceph.com/issues/47384
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #36964 from trociny/wip-47275

mgr: don't update pending service map epoch on receiving map from mon

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #34777 from nSedrickm/grafana-iframe

mgr/dashboard: increase Grafana iframe height to avoid scroll bar

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #37057 from tchaikov/wip-bump-seastar

seastar,crimson: bump seastar to use upstream master

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

qa: add test for krbd symlinks created by udev

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #37042 from xxhdx1985126/wip-47357

crimson/osd: avoid accessing WaitForObjectRecovery::pi when no pulling occurred

Reviewed-by: Kefu Chai <kchai@redhat.com>

seastar: bump seastar to use upstream master

before this change we are using a fork which includes a fix to address
the FTBFS with GnuTLS < 3.5.6

after this change, we are using the lastest upstream which already
contains this fix.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: print notify_id instead of its address

this addresses the FTBFS like

    /usr/include/fmt/core.h:902:19: error: static assertion failed: formatting of non-void pointers is disallowed
      902 |     static_assert(!sizeof(T), "formatting of non-void pointers is disallowed");
          |                   ^~~~~~~~~~

also, i think it is more helpful in this context to have the
notify id instead of the address, as we can use the id to reference
the notification printed by other logging messages.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: cast pointers to void* before printing them

to address the FTBFS like:

../src/crimson/os/seastore/lba_manager/btree/btree_range_pin.cc:110:14:   required from here
/usr/include/fmt/core.h:902:19: error: static assertion failed: formatting of non-void pointers is disallowed
  902 |     static_assert(!sizeof(T), "formatting of non-void pointers is disallowed");
      |                   ^~~~~~~~~~

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: instantiate future_type<> with future_stored_type

to accommodate the change on seastar's side, which started to use
monostate for representing `void` return type, and hide this convention
using future_stored_type_t.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/common: move continuation_base_from_future out of internal namespace

otherwise we have

../src/crimson/common/errorator.h:1090:8: error: specialization of ‘template<class Future> struct seastar::continuation_base_from_future’ in different namespace [-fpermissive]
 1090 | struct continuation_base_from_future<Container<::crimson::errorated_future_marker<Values...>>> {
      |        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

since seastar has moved its continuation_base_from_future() out
of seastar::internal into seastar namespace

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/net: print entity_addr_t not the in4_addr in it

for two reasons:

* we don't have operator<<(ostream&, in4_addr&) defined.
* we don't have any fmt::format() facility specialized for in4_addr
* to print entity_addr_t instead of in4_addr is more correct in
  the sense that we should support IPv6

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/crimson: include seastar/util/later.hh for seastar::now()

seastar::now() is extracted into seastar/util/later.hh in a recent
change in seastar, so update accordingly

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: include seastar/core/when_all.hh for when_all()

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: use temporary RecoveryInfo for recovering object delete ops

Right now, recover_delete use WaitForObjectRecovery::pi, which may not be available
if there were no pulling, to pass recovery_info, so use temporary recovery_info instead

Fixes: https://tracker.ceph.com/issues/47357
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/osd: set push future an exception when there are errors during pushing

Fixes: https://tracker.ceph.com/issues/47357
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/net: do not print non-void* pointer

to address the failure like

fmt/core.h:902:19: error: static assertion failed: formatting of non-void pointers is disallowed
  902 |     static_assert(!sizeof(T), "formatting of non-void pointers is disallowed");
      |                   ^~~~~~~~~~

and i searched around, couldn't find any other
statement printing the address of message, so
i think it's not of much use to print their
addresses in a single place, unless we can cross
check them in different places.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: use seastar::make_ready_future()

instead of seastar::now(). as seastar::now() was moved into
seastar/util/laster.hh.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cls/fifo: Add 'exclusive' option to Trim

To support RGW's MetadataLog, add a flag to FIFO::trim that when true
trims up to but not including the given entry.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Datalog with selectable backends, FIFO and Omap

By default, use whatever is present (and use FIFO if nothing is
present.)

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Fix async caller into Datalog

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Remove superfluous wrapper around datalog

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Normalize DataLog function signatures

Pull out the stuff we aren't using so FIFO and Timelog Datalog can use
the same interface.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common: Make error conversion functions `nodiscard`

As suggested by Casey, so the compiler can catch failures to return
errors.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Get rid of BucketFilter class, use a std::function

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Remove date from REST log trim/list

Also only accept one marker.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Remove some TimeLog options from radosgw-admin

Remove date-oriented trim/list functionality from MDLOG, DataLog, and
Sync-Error-Log.

Don't provide start/end markers. Support only a start marker for list
and an end marker for trim.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

gitignore: Add vstart_environment.sh

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: FIFO for legacy RADOS client

Use optional_yield for most operations and provide an AioCompletion*
overload for trim.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

cls/fifo: Use neorados interface

Rewrite around the asynchronous ASIO-based RADOS interface, and
include support for calls from multiple threads.

`allocate_unique` pattern contributed by
Casey Bodley <cbodley@redhat.com>

Co-authored-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

cls/fifo: FIFO over RADOS

This is an implementation of fifo queue over rados. Data is appended
to rados object until it's full. At that point data will be written to
a new object. Data is read from the tail sequentially (can be iterated
using marker). Data can be trimmed (up to and including marker).

Queue has a header object (meta), and zero or more data objects (parts).

The software has two layers: the higher level client operations side
that deals with the application apis, and manages the meta and parts,
and there’s the objclass level that deals with the rados layout of
meta and part objects. There are different objclass methods that deal
with reading and modifying each of these entities.

A single part has max possible size, however, it may become full once
a certain smaller size is reached (full_size_threshold). It is
imperative that once a part has reached its capacity, it will not
allow any more writes into it. For this reason, it is important that
data being written to the queue does not exceed max_entry_size . This
is enforced, by the higher level client api.

Written entries go to the current head object, and when it’s full, a
new head part is created. When listing entries, data is iterated from
tail to the current head. Trim can either change the pointer within
the current tail object, and if needed it removes tail objects.

A unitest has been created to test functionality.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
Squashed-by: Adam C. Emerson <aemerson@redhat.com>

strtol: Add parse/consume for string_view friendly interface

Also these don't have the stringstream overhead.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common/Thread: Don't store pointer to thread_name

Having Thread::create store a pointer to a string that is passed to
ceph_pthread_setname in Thread::entry_wrapper can lead to using a
pointer in the calling thread's stack that gets freed before use.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common/ceph_json: Support en/decoding Boost's flat containers

Specifically boost::container::flat_map and boost::container::flat_set

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common/async: Add handler for librados::AioCompletion

Support using a librados::AioCompletion as a completion handler in
Boost.Asio when the arguments are void or only an error code.

(To support more arguments we'd have to wrap up the AioCompletion with
pointers to hold them.)

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common: add ceph::allocate_unique()

adds an allocator-aware version of std::make_unique(). this is similar to
std::allocate_shared(), though it's slightly less convenient because,
unlike std::shared_ptr<T>, the Deleter has to be specified as a template
parameter in std::unique_ptr<T, Deleter>

Signed-off-by: Casey Bodley <cbodley@redhat.com>

common: bind_like

A template to ensure that one completion has the same executor and
allocator as another.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common: Add bind_allocator template

Bind an allocator to a completion, analogously to
boost::async::bind_executor.

Based on bind_executor.hpp which is copyright 2003-2019
Christopher M. Kohlhoff <chris@kohlhoff.com> and under
the Boost Software License, Version 1.0

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

include: Import (modified) uses_allocator.h

From https://github.com/phalpern/uses-allocator

To handle uses_allocator construction

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #37049 from yuriw/wip-yuriw-crontab-master

qa/tests: removed extra line

Reviewed-by: Neha Ojha <nojha@redhat.com>

rgw: ObjectCache::put() clears stale objv

if an existing object is cached with an object version, but it's
mutated without updating that version number, clear the OBJV flag so
that later cache reads asking for an object version result in a miss and
re-read the version from the osd

Fixes: https://tracker.ceph.com/issues/47306
Signed-off-by: Casey Bodley <cbodley@redhat.com>

rbd: include RADOS namespace in krbd symlinks

Fixes: https://tracker.ceph.com/issues/40247
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/tests: removed extra line

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Zipper - More getRados removal

- read_usage

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Zipper - Replace RGWAttrs with a using

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Zipper - Make s->user a unique_ptr

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Zipper - cleanup get_obj_head and get_multipart_info

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Zipper - remove get_obj_attrs and last user

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Zipper - More getRados removal

- zonegroup
- obj policy

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Zipper - Split RadosStore from base SAL

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Zipper - implement copy_object

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Zipper - Remove more instances of getRados() from rgw_op.cc

- RGWGetObj and RGWPutObj

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #36230 from trociny/wip-43174-2

os/bluestore: fix collection_list ordering

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #36955 from agayev/zbc-to-zbd

os/bluestore: Switch from libzbc library to libzbd library

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36950 from ceph/wip-bz1872983

ceph-volume: simple scan should ignore tmpfs

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

cephadm: unit.run: create /var/run/ceph/$FSID before doing anything else

This ensures the /var/run/ceph/$FSID directory exists before any
other commands are run.

Fixes: https://tracker.ceph.com/issues/47360
Signed-off-by: Tim Serong <tserong@suse.com>

Merge pull request #36976 from rhcs-dashboard/fix-47285-master

mgr/dashboard: fix duplicate logs

Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #37003 from xxhdx1985126/wip-47310

crimson/osd: drop repop if osdmap has increased

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36974 from jan--f/snap-schedule-keep-with-empty-retention

snap-schedule: don't prune all on empty retention

crimson/osd: add can_discard_op method for detecting "discardable" client requests

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/osd: drop repop if it can be discarded

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/osd: rename is_valid_rep_op_reply to can_discard_replica_op

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #36903 from sebastian-philipp/python-common-simplify-hsp-to-json

python-common: Make HostPlacementSpec.to_json() more friendly

Reviewed-by: Adam King <adking@redhat.com>

common/buffer: add hint to compiler

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/buffer: do not deref _raw ptr when checking for hypercombind buf

_raw could be null when the buffer::ptr is std::moved away. and ASan
complains when we try to dereference a nullptr. so use a little awkward
way to calculate the address instead

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36994 from ktdreyer/boost-context-s390x

rpm: enable WITH_BOOST_CONTEXT for s390x

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #36432 from sebastian-philipp/cephadm-repo_digest

mgr/cephadm: convert tags to repo_digest

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Varsha Rao <varao@redhat.com>