osd/: clean up PeeringState::write_if_dirty

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: Move init into PeeringState

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move stat updates and publishing to PeeringState

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move more state helpers to PeeringState

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move split/merge helpers into PeeringState

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: fix try_mark_clean/finish_recovery interface boundary

Signed-off-by: Samuel Just <sjust@redhat.com>

PeeringState: remove PG references and include

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move calc_min_last_complete_ondisk to PeeringState

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: pass OSDService::logger and log_enter/exit through PeeringListener

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: pass merge state control through PeeringListener

Signed-off-by: Samuel Just <sjust@redhat.com>

PeeringState: pass dump_recovery_info through to pl for recovery state

Signed-off-by: Samuel Just <sjust@redhat.com>

PeeringState: rearrange and fix remaining pg->gen_prefix call

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move fulfill_info etc

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: clarify PeeringState deletion interfaces

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move min_peer_*_features and helpers

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move proc_* and log handling methods

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move share_pg_info to PeeringState

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move waiting_for_flushed|peering handling into on_activate_complete

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move search_for_missing, discover_all_unfound, build_might_have_unfound

Signed-off-by: Samuel Just <sjust@redhat.com>

PeeringState: restructure activate to avoid snap_trimq

In particular, don't use snap_trimq when manipulating
info.purged_snaps.  Pass the resulting to_trim set
into on_activate.

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: start moving activate to PeeringState

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move try_mark_clean/finish_recovery to PG::on_clean

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move choose_acting and callees into PeeringState

Also marks IsPGRecoverablePredicate users const to allow
simply reusing the instance on MissingLoc.

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: remove OSDService::check_osdmap_full, fix caller

PeeringState was the only remaining caller -- both the interface in
PGBackend and the pass-through in PrimaryLogPG were actually unused.
Further, the WaitLocalRecoveryReserved user does not appear to actually
require the fresh-off-of-the-wire osdmap available from OSDService.
As such, moved the logic into OSDMap itself and simply used the
PG local osdmap.

Note, the above is a change in behavior that probably could use a second
opinion.

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move the backfill space reservation back into PG

PeeringState really only needs an interface for requesting
it.

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move needs_recover/backfill, all_unfound_are_queried_or_lost

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: clarify recovery queueing from PeeringState

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move Active purged_snaps handling back to PG

Note: this patch only sets dirty_info/dirty_big_info when
info.purged_snaps is actually changed.  I *think* that's
right, but that part deserves particular attention during
review.

Signed-off-by: Samuel Just <sjust@redhat.com>

PeeringState: mechanically rename a ton of pg-> to ps->

Signed-off-by: Samuel Just <sjust@redhat.com>

PeeringState: introduce a macro for the ps, pl, and pg declarations

This way I can mechanically rename all of the PeeringState member
variable references.

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move build_prior and related state to PeeringState

Signed-off-by: Samuel Just <sjust@redhat.com>

PeeringState: clean up references in Primary, Start, Reset

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move io reservation machinery into PeeringState

This patch recasts the reservation and backoff interfaces
on PeeringListener in terms of events to queue rather than
explicit callbacks (PG handles implementing via callbacks).

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: remove remaining scrubber references from PeeringState

Signed-off-by: sjust@redhat.com <sjust@redhat.com>

osd/: rephrase [un]reg_next_scrub as on_info_history_change

As PeeringState won't be as careful about usage of
on_info_history_change, add a scrub_registered flag
to protect against double-(un)registers.

Signed-off-by: sjust@redhat.com <sjust@redhat.com>

osd/: merge update_store_with_options with on_pool_change

Signed-off-by: sjust@redhat.com <sjust@redhat.com>

osd/: simply requeue_map_waiters in handle_activate, remove take_waiters

This seems clearer.

Signed-off-by: sjust@redhat.com <sjust@redhat.com>

osd/: move check_past_interval_bounds into PeeringState

Signed-off-by: sjust@redhat.com <sjust@redhat.com>

osd/: move start_peering_interval and callees into PeeringState

Signed-off-by: sjust@redhat.com <sjust@redhat.com>

osd/: move should_restart_peering to PeeringState

Signed-off-by: sjust@redhat.com <sjust@redhat.com>

osd/: move check_full_transition to PeeringState

Signed-off-by: sjust@redhat.com <sjust@redhat.com>

PeeringState: route recoverystate_perf calls through PeeringListener

Signed-off-by: Samuel Just <rexludorum@gmail.com>

osd/: move flush state into PeeringState

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move check_recovery_sources and missing_loc into PeeringState

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: extract MissingLoc into its own file

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move update_history and purge_strays into PeeringState

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move proc_replica_info and some callees into PeeringState

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move the is_* and acting/up/backfill helpers into PeeringState

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: Move osdmap handling into PeeringState

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move most peering state to PeeringState

This patch moves the 40 something peering state variables over into
PeeringState while leaving references to them in PG.  The following
patches will move over the users until all users are in PeeringState.
Then, the PG references will be removed.  A subsequent patch will also
move the recovery_state member to be the last initialize and first
destructed.

Signed-off-by: sjust@redhat.com <sjust@redhat.com>

PeeringState: begin abstracting out PG, add dpp for logging

Signed-off-by: Samuel Just <sjust@redhat.com>

PGStateUtils: remove PG*, move into PGStateHistory into PeeringState

I don't think there's any need to worry about the pg locking from
PGStateHistory.  NamedState::begin/exit and dump_pgstate_history are the
only users, so the only calls should be under the peering event handler,
the asok command, or the PG constructor and destructor.  The first two
already have the lock.  The last should be safe as well as long as
the state machine states are constructed and destructed after and
before the PGStateHistory instance. As such, this patch removes most
of that state leaving the epoch generation as an interface
implemented by PG.

The snap trimming state machine was already excluded, so that this
patch leaves it disabled.

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: pull NamedState and PGStateHistory out into PGStateUtils

Signed-off-by: Samuel Just <sjust@redhat.com>

PeeringState: fix pg->info->pgid.pgid references

Signed-off-by: Samuel Just <sjust@redhat.com>

PeeringState: adjust logging to avoid using pg

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: mechanically rip PeeringState and friends out into their own file

The next step is to rationalize the PeeringState<->PG relationship.

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: mechanically rename RecoveryMachine/State/Ctx to Peering*

I'm going to extract this logic and reuse it in crimson.  Recovery* has
always been a confusing name as it implements neither log-based recovery
nor backfill.  Rather, it's mainly the buisiness logic for agreeing on
an authoritative log and some ancillary things such as scrub/backfill
reservation.

$ for i in $(git grep -l 'RecoveryMachine'); do sed -i 's/RecoveryMachine/PeeringMachine/g' $i; done
$ for i in $(git grep -l 'RecoveryState'); do sed -i 's/RecoveryState/PeeringState/g' $i; done
$ for i in $(git grep -l 'RecoveryCtx'); do sed -i 's/RecoveryCtx/PeeringCtx/g' $i; done

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #27867 from tchaikov/wip-ceph-object-corpus

ceph-object-corpus: pick up 15.0.0-539-g191ab33faf

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #27730 from cbodley/wip-rgw-boost-170

rgw: fix compile errors with boost 1.70

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #27861 from tchaikov/wip-utime-cleanup

include/utime: do not cast sec to time_t

Reviewed-by: Samuel Just <sjust@redhat.com>

ceph-object-corpus: pick up 15.0.0-539-g191ab33faf

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27784 from Devp00l/wip-39483

mgr/dashboard: Make all columns sortable

Merge PR #27590 into master

* refs/pull/27590/head:
messages: #include necessary header

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #27804 into master

* refs/pull/27804/head:
common, osd: remove erroneous 'section' in dump functions
dencoder: include some missed types
common: make utime_t dencoder-compatible
common: make uuid_d dencoder-compatible
osd: make watch_item_t dencoder-compatible

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #27832 into master

* refs/pull/27832/head:
qa/suites/upgrade/mimic-x-singleton: suppress legacy statfs warning
osd: reenable authorizer on hb messenger for nautilus+
mon/MonClient: do not crash on empty auth request buffer
msg: set_require_authorizer on messenger, not dispatcher
mon/Monitor: allow probe if MMonProbe::mon_release == 0
mon: fix min_mon_release print

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27788 from cyx1231st/wip-seastar-msgr-batch-send

crimson/net: improve batching in the write path

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #27776 from JunHe77/install-deps

install-deps.sh: add option to skip prebuilt boost-* pkgs installation

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Make all columns sortable

Fixes: https://tracker.ceph.com/issues/39483
Signed-off-by: Stephan Müller <smueller@suse.com>

include/utime: do not cast sec to time_t

strictly speaking, time_t is an opaque time, and in this context, we
need a uint32_t, so it is not necessary and wrong. let's remove it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: auth: fix user lastUpdate timestamp generation (#27604)

mgr/dashboard: auth: fix user lastUpdate timestamp generation

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

monitoring: Prometheus default alerts (#27596)

monitoring: Prometheus default alerts

Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #27837 from cyx1231st/wip-seastar-msgr-fix-read

crimson/net: misc fixes in v1 read path

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/net: misc fixes in v1 read path

* abort when decoding/signature-check failed;
* change log-level to warn;
* free the decoded message when failed;
* add unlikely to read path;

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

crimson/net: increase out buffer to 65536

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

crimson/net: group write-related general implementations

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

crimson/net: use optional keepalive_ack to notify writes

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

crimson/net: gather message buffers when send

Gather buffers from pending messages/keepalive and send them together.

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

Merge pull request #27809 from tchaikov/wip-crimson-auth

crimson,auth: fix FTBFS of crimson-osd and fix v1/v2 auth

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #27831 from tchaikov/wip-crimson-silence-warning

crimson/net: prefer <fmt/chrono.h> over <fmt/time.h>

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

qa/suites/upgrade/mimic-x-singleton: suppress legacy statfs warning

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #27756 from dillaman/wip-39025

librbd: initial consolidation of internal locks

Reviewed-by: Mykola Golub <mgolub@suse.com>

crimson/osd: silence -Wunused-variable warning

silence warning like

/home/kchai/ceph/src/crimson/osd/heartbeat.cc:314:24: warning: unused
variable ‘peer’ [-Wunused-variable]
       auto& [peer, info] = item;
                        ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/mon: log when a monitor is fully connected

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/mon: hold rotating_keyring using a pointer

we could have kept rotating_keyring as a plain member variable, but
`AuthClientHandler` keeps a weak reference to it, we are not able to
update it to point it to the new rotating_keyring in the newly created
active_con created from the connected pending connection -- the move
ctor of ceph::mon::Connection does create a new RotatingKeyring. so this
has two consequences:

- the raw pointer held by AuthClientHandler is not valid anymore, after
  the pending connection is destroyed when it is promoted to active_con.
  so we are writing to freed memory when renewing the rotating keyring.
- we won't have access to the updated keyring.

if we use a std::vector<unique_ptr<Connection>> for pending_conns, this
change won't be needed. we could make this change in a future change
though.

Signed-off-by: Kefu Chai <kchai@redhat.com>

auth,mon,crimson: pass KeyStore by const reference

AuthAuthorizeHandler::verify_authorizer() neither changes the keystore,
nor expects a nullptr. so we should pass the keystore by const reference
for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/crimson: do not link against ceph-common

crimson-common and ceph-common have subtle difference which is
conditionalized by WITH_SEASTAR macro. and the different implementations
share the same name, so we should not link crimson executables against ceph-common .

Signed-off-by: Kefu Chai <kchai@redhat.com>

auth: do not use cct unless i have to

crimson messenger's tests does not use ConfigProxy at the time of writing,
and ldout() cannot tell if a log message should be written to the
logfile or not without querying config subsystem, so, to avoid involving
config system in crimson messenger's tests, the log line is disabled if
WITH_SEASTAR, we will kill this workaround once we ditch crimson/net/Config.h.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/mon: renew rotating secret if necessary

Signed-off-by: Kefu Chai <kchai@redhat.com>

auth/Auth: pass utime_t by const reference

Signed-off-by: Kefu Chai <kchai@redhat.com>

auth/cephx: do not require a cct for accessing config

crimson/msgr does not depend on ConfigProxy at the time of writing. we
want to decouple it from config subsystem in hope to have less cross
subsystem dependencies and speed up the compiling speed.

but, to introduce cephx to msgr, we need to enable msgr to read from
ConfigProxy at runtime. so, to avoid keeping a CephContext instance
simply for accessing the ConfigProxy reference in it, it would be more
efficient to define a free function `conf()` which hides the difference
between crimson and classic user of auth/cephx.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/net: learn my address from peer

OSD always relies on its server to tell its address, without this
feature, it will advertise a blank ip address to monitor in MOSDBoot.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/mon: pass cct to verify_authorizer()

CephxAuthorizeHandler::verify_authorizer() uses it for querying config
and for a RNG, so we have to pass it a CephContext which has `_conf` and
`random()`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: pass `Connection*` to Dispatch::ms_dispatch()

it's a regression introduced by c69bf0c002f1b62283b3eb5df62fdcb65963b4ae

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/net: sign message and check it if authorizer is available

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/mon: set entity_name before trying to connect to mon

otherwise Monitor.cc or AuthMonitor.cc won't be able to find it in the
keyring.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/mgr: connect mgr with v2 protocol

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: remove ms_get_authorizer() and friends

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/crimson: disable authorizer for tests

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: sync msgr's require_authorizer with osdmap

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/mon: tolerate lack of authorizer for some messengers

also, we should not crash on empty auth request buffer.

see also 64eddc4d and b99daab1

Signed-off-by: Kefu Chai <kchai@redhat.com>