osd: Fix merge_log_dups() not remove dup corresponding to log.tail

The log tail is not inclusive of the log, so not an overlap
Fix unittest_pglog to verify proper operation

Signed-off-by: David Zafman <dzafman@redhat.com>

test: unittest_pglog: Additional test cases for trim, copy_up_to and copy_after

Signed-off-by: David Zafman <dzafman@redhat.com>

test: unittest_pglog: Remove irrelevant config values

osd_min_pg_log_entries and osd_max_pg_log_entries are not involved
in the log handling code being tested.

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Adding standalone test of log copy handling

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Include dups in copy_after() and copy_up_to()

Client saw out of order results in a test with a very small
pg_log because a backfill/recovery sent a small number of
log entries for duplicate checks.

Since these copy operations are equivalent to both a copy
and trim (an argument controls how many log entries
transfer), we need to include any new dups with enough
existing dups up to the configured maximum.

Fixes: http://tracker.ceph.com/issues/39304
Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Remove unused pg_log_t::copy_range()

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Fix off by 1 error in trim handling of osd_pg_log_dups_tracked

Move log message out of loop so it doesn't repeat for no reason

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #27897 from ivancich/wip-add-editor-comments-rgw-services

rgw: add editor directive comments to rgw services source files

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27820 from smithfarm/wip-39361

rgw: drop cloud sync module logs attrs from the log

Reviewed-By: Casey Bodley <cbodley@redhat.com>
Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

Merge PR #27871 into master

* refs/pull/27871/head:
ceph_test_objectstore: add very_large_write test
os/bluestore: fix aio pwritev lost data problem.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #27844 from wjwithagen/wjw-fix-src_common_bit_vector.hpp.diff

common: Clang requires a default constructor, but it can be empty

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #27874 from athanatos/sjust/wip-peering-refactor-forreview

Extract peering logic into a module for use in crimson

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #27763 into master

* refs/pull/27763/head:
common/PriorityCache: fix over-aggressive assert when mem limited

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

common/PriorityCache: fix over-aggressive assert when mem limited

Fixes: https://tracker.ceph.com/issues/39437
Signed-off-by: Mark Nelson <mnelson@redhat.com>

PGStateUtils: improvements for PGStateHistory

Signed-off-by: Samuel Just <sjust@redhat.com>

PeeringState: don't zero backfill target num_bytes on activation

834d3c19a774f1cc93903447d91d182776e12d18 preserves num_bytes
on backfill targets in order to estimate space required to complete
backill.  However, from activation until backfill reservation,
info.stats.stats.sum.num_bytes is persisted to disk as 0 messing
up future intervals.  Instead, preserve it in the info sent during
recovery and leave it alone in RequestBackfillPrio.

Additionally, it's possible for backfill to be preempted between
last_backfill=MAX being sent to the replica and Backfilled being
queued occuring.  In that case, the stats get on reservation
and the replica ends up with invalid stats.

Fixes: https://tracker.ceph.com/issues/39401
Signed-off-by: sjust@redhat.com <sjust@redhat.com>

PeeringState: use ceph_assert

Signed-off-by: Samuel Just <sjust@redhat.com>

admin/build-doc: use PeeringState* for gen_state_diagram.py

Signed-off-by: Samuel Just <sjust@redhat.com>

PeeringState: add explanations for public interface methods

Also rearranges the methods a little for clarity.

Signed-off-by: Samuel Just <sjust@redhat.com>

PG,PeeringState: Fix initialization order

PeeringState needs to be initialized last and destructed
first.

Signed-off-by: sjust@redhat.com <sjust@redhat.com>

PeeringState: mark state and helpers private

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: clean up remaining info mutators

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: condense missing mutations for recovery/repair/errors

At a high level, this patch attempts to unify the various
sites at which some combination of
- mark object missing in one or more pg_missing_t
- mark object needs_recovery in missing_loc
- manipulate the locations map based on external information
occur.  It seems to me that the pg_missing_t and missing_loc
should be in sync except for the mark_unfound_lost_revert
case and the case where we are about to do a backfill push.

This patch also cleans up repair_object.  It sort of worked by accident
for non-ec non-primary bad peers.  It didn't update missing_loc, so
needs_recovery() returns the wrong answer.  However, is_unfound() does
as well, so ReplicatedBackend is nevertheless happy as the object would
be present on the primary.  This patch updates the behavior to be
uniform as in the other force_obejct_missing cases.

Signed-off-by: Samuel Just <sjust@redhat.com>

test-erasure-eio: first eio may be fixed during recovery

The changes to the way EC/ReplicatedBackend communicate read
t showerrors had a side effect of making first eio on the object in
TEST_rados_get_subread_eio_shard_[01] repair itself depending
on the timing of the killed osd recovering.  The test should
be improved to actually test that behavior at some point.

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move pg_log, missing_loc  mutations to PeeringState

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: unify PGBackend pull error pathways

This patch narrows the PGBackend -> PrimaryLogPG recovery
cancel/error interface to on_failed_pull and cancel_pull.

This patch requires careful review.

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move peer_info mutators into PeeringState

Signed-off-by: sjust@redhat.com <sjust@redhat.com>

osd/: Move log version pointer updates to PeeringState

Signed-off-by: sjust@redhat.com <sjust@redhat.com>

osd/: refactor to avoid mutable peer_missing refs in PG

Signed-off-by: sjust@redhat.com <sjust@redhat.com>

PG: remove might_have_unfound, peer_*_requested

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: fix upset, actingset, acting_backfill_recovery references

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: fix primary/up_primary references

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move last_..._to_applied, backfill_targets, async_recovery_targets to PeeringState

Signed-off-by: Samuel Just <sjust@redhat.com>

PG: fix last_peering_reset and past_intervals references

Signed-off-by: Samuel Just <sjust@redhat.com>

PeeringState::PeeringMachine::Deleting: rollfoward before resetting backfill

This looked wrong when I was moving it over.

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: add helpers to add remaining info dirtiers into PeeringState

Signed-off-by: sjust@redhat.com <sjust@redhat.com>

osd/: move append_log into PeeringState

Signed-off-by: sjust@redhat.com <sjust@redhat.com>

osd/: move append_log_entries_update_missing and merge_new_log_entries to PeeringState

Signed-off-by: sjust@redhat.com <sjust@redhat.com>

osd/: clean up PG deleted and deleting references

Signed-off-by: sjust@redhat.com <sjust@redhat.com>

PG: remove direct acting and up references

Signed-off-by: sjust@redhat.com <sjust@redhat.com>

PG: begin cleaning up scrub stat and history mutations

Signed-off-by: sjust@redhat.com <sjust@redhat.com>

PG: remove first batch of unused references

Signed-off-by: sjust@redhat.com <sjust@redhat.com>

osd/: clarify interface for introducing disk state to PeeringState

Signed-off-by: sjust@redhat.com <sjust@redhat.com>

osd/: move ostream<< and dump logic into PeeringState

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: clean up PeeringState::write_if_dirty

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: Move init into PeeringState

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move stat updates and publishing to PeeringState

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move more state helpers to PeeringState

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move split/merge helpers into PeeringState

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: fix try_mark_clean/finish_recovery interface boundary

Signed-off-by: Samuel Just <sjust@redhat.com>

PeeringState: remove PG references and include

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move calc_min_last_complete_ondisk to PeeringState

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: pass OSDService::logger and log_enter/exit through PeeringListener

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: pass merge state control through PeeringListener

Signed-off-by: Samuel Just <sjust@redhat.com>

PeeringState: pass dump_recovery_info through to pl for recovery state

Signed-off-by: Samuel Just <sjust@redhat.com>

PeeringState: rearrange and fix remaining pg->gen_prefix call

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move fulfill_info etc

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: clarify PeeringState deletion interfaces

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move min_peer_*_features and helpers

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move proc_* and log handling methods

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move share_pg_info to PeeringState

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move waiting_for_flushed|peering handling into on_activate_complete

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move search_for_missing, discover_all_unfound, build_might_have_unfound

Signed-off-by: Samuel Just <sjust@redhat.com>

PeeringState: restructure activate to avoid snap_trimq

In particular, don't use snap_trimq when manipulating
info.purged_snaps.  Pass the resulting to_trim set
into on_activate.

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: start moving activate to PeeringState

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move try_mark_clean/finish_recovery to PG::on_clean

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move choose_acting and callees into PeeringState

Also marks IsPGRecoverablePredicate users const to allow
simply reusing the instance on MissingLoc.

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: remove OSDService::check_osdmap_full, fix caller

PeeringState was the only remaining caller -- both the interface in
PGBackend and the pass-through in PrimaryLogPG were actually unused.
Further, the WaitLocalRecoveryReserved user does not appear to actually
require the fresh-off-of-the-wire osdmap available from OSDService.
As such, moved the logic into OSDMap itself and simply used the
PG local osdmap.

Note, the above is a change in behavior that probably could use a second
opinion.

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move the backfill space reservation back into PG

PeeringState really only needs an interface for requesting
it.

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move needs_recover/backfill, all_unfound_are_queried_or_lost

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: clarify recovery queueing from PeeringState

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move Active purged_snaps handling back to PG

Note: this patch only sets dirty_info/dirty_big_info when
info.purged_snaps is actually changed.  I *think* that's
right, but that part deserves particular attention during
review.

Signed-off-by: Samuel Just <sjust@redhat.com>

PeeringState: mechanically rename a ton of pg-> to ps->

Signed-off-by: Samuel Just <sjust@redhat.com>

PeeringState: introduce a macro for the ps, pl, and pg declarations

This way I can mechanically rename all of the PeeringState member
variable references.

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move build_prior and related state to PeeringState

Signed-off-by: Samuel Just <sjust@redhat.com>

PeeringState: clean up references in Primary, Start, Reset

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move io reservation machinery into PeeringState

This patch recasts the reservation and backoff interfaces
on PeeringListener in terms of events to queue rather than
explicit callbacks (PG handles implementing via callbacks).

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: remove remaining scrubber references from PeeringState

Signed-off-by: sjust@redhat.com <sjust@redhat.com>

osd/: rephrase [un]reg_next_scrub as on_info_history_change

As PeeringState won't be as careful about usage of
on_info_history_change, add a scrub_registered flag
to protect against double-(un)registers.

Signed-off-by: sjust@redhat.com <sjust@redhat.com>

osd/: merge update_store_with_options with on_pool_change

Signed-off-by: sjust@redhat.com <sjust@redhat.com>

osd/: simply requeue_map_waiters in handle_activate, remove take_waiters

This seems clearer.

Signed-off-by: sjust@redhat.com <sjust@redhat.com>

osd/: move check_past_interval_bounds into PeeringState

Signed-off-by: sjust@redhat.com <sjust@redhat.com>

osd/: move start_peering_interval and callees into PeeringState

Signed-off-by: sjust@redhat.com <sjust@redhat.com>

osd/: move should_restart_peering to PeeringState

Signed-off-by: sjust@redhat.com <sjust@redhat.com>

osd/: move check_full_transition to PeeringState

Signed-off-by: sjust@redhat.com <sjust@redhat.com>

PeeringState: route recoverystate_perf calls through PeeringListener

Signed-off-by: Samuel Just <rexludorum@gmail.com>

osd/: move flush state into PeeringState

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move check_recovery_sources and missing_loc into PeeringState

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: extract MissingLoc into its own file

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move update_history and purge_strays into PeeringState

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move proc_replica_info and some callees into PeeringState

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move the is_* and acting/up/backfill helpers into PeeringState

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: Move osdmap handling into PeeringState

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move most peering state to PeeringState

This patch moves the 40 something peering state variables over into
PeeringState while leaving references to them in PG.  The following
patches will move over the users until all users are in PeeringState.
Then, the PG references will be removed.  A subsequent patch will also
move the recovery_state member to be the last initialize and first
destructed.

Signed-off-by: sjust@redhat.com <sjust@redhat.com>

PeeringState: begin abstracting out PG, add dpp for logging

Signed-off-by: Samuel Just <sjust@redhat.com>

PGStateUtils: remove PG*, move into PGStateHistory into PeeringState

I don't think there's any need to worry about the pg locking from
PGStateHistory.  NamedState::begin/exit and dump_pgstate_history are the
only users, so the only calls should be under the peering event handler,
the asok command, or the PG constructor and destructor.  The first two
already have the lock.  The last should be safe as well as long as
the state machine states are constructed and destructed after and
before the PGStateHistory instance. As such, this patch removes most
of that state leaving the epoch generation as an interface
implemented by PG.

The snap trimming state machine was already excluded, so that this
patch leaves it disabled.

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: pull NamedState and PGStateHistory out into PGStateUtils

Signed-off-by: Samuel Just <sjust@redhat.com>

PeeringState: fix pg->info->pgid.pgid references

Signed-off-by: Samuel Just <sjust@redhat.com>

PeeringState: adjust logging to avoid using pg

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: mechanically rip PeeringState and friends out into their own file

The next step is to rationalize the PeeringState<->PG relationship.

Signed-off-by: Samuel Just <sjust@redhat.com>