ceph_test_rados_api_snapshots_pp: drop unnecessary assert

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: record last_purged_snaps_scrub from beacon to osdmap

Signed-off-by: Sage Weil <sage@redhat.com>

osd: report last_purged_snaps_scrub as part of beacon

Signed-off-by: Sage Weil <sage@redhat.com>

osd: log purged_snaps scrub to cluster log

Signed-off-by: Sage Weil <sage@redhat.com>

osd: record last_purged_snaps_scrub in superblock

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMap: add last_purged_snaps_stamp to osd_xinfo_t

Record OSDs' last scrub of purged_snaps in the OSDMap.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: fix bug in try_prune_purged_snaps

If 'begin' isn't found, we'll get a [pbegin,pend) range back that was
nearby.  Only if it overlaps the [begin,end) range do we want to shorten
our range to [begin,pbegin); the old assert was making the assumption
that the lookup would only return a range that was after 'begin', but in
reality it can return was that comes before it too.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: record snap removal seq as purged

When we delete a selfmanaged snap we have to bump seq.  Record this as
purged so that we avoid discontinuities in the history and so our storage
is a bit more efficient.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: do not bother reporting gaps in removed_snaps

Signed-off-by: Sage Weil <sage@redhat.com>

osdc/Objecter: don't worry about gap_removed_snaps from map gaps

This was an attempt to ensure that we didn't let removed_snaps slip by
when we had a discontiguous stream of OSDMaps.  In octopus, this can still
happen, but it's mostly harmless--the OSDs will periodically scrub to
clean up any resulting stray clones.  It's not worth the complexity.

Signed-off-by: Sage Weil <sage@redhat.com>

mds/SnapServer: make not about pre-octopus compat code

Signed-off-by: Sage Weil <sage@redhat.com>

osd: implement scrub_purged_snaps command

This a naive one-shot implementation that does the full scan synchronously
in the command thread.  It shouldn't block any IO except to the extent
that it will compete for IO reading the underlying snapmapper omap object.

When we discover mapped objects that are covered by ranges of snaps that
should be purged, we requeue the snapid for trim on the relevant PG(s).
For these 'repeat' trims we skip the final step(s) to mark the snapid as
purged, since that presumably already happened some time ago.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: always remove the snap we are trimming

This doesn't change behavior (yet), since we only trim snaps that are
marked for removal in the OSDMap.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_rados_api_snapshots_pp: (partial) test to reproduce stray clones

The test creates a snap, removes it, waits for it to (hopefully) purge,
and then uses that snapid in a snapc to generate a clone.

This isn't a complete test because (1) it doesn't wait for the purge to
happen (e.g., by watching the osdmaps go by), and (2) it doesn't trigger
an osd scrub_purged_snaps afterwards.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: sync old purged_snaps on startup after upgrade or osd creation

This path only triggers after an upgrade or osd creation, when
purged_snaps_last < current_epoch.  When that happens, we slurp down the
old purged snaps so that we have a full history recorded locally.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: record purged_snaps when we store new maps

When we get a new map, record the (new) purged_snaps.

Only do this if the OSD has purged_snaps that are in sync with the latest
OSDMap.  That means that after an upgrade, if the OSD didn't sync the
old purged_snaps on startup, it won't sync anything until it *next* starts
up.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: add messages to get past purged_snaps

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: record pre-octopus purged snaps with first octopus map

When we public our first require_osd_release >= octopus osdmap, record
all prior purged snaps in a key linked to the previous osdmap.  We assume
this will encode and fit into a single key and transaction because the
even larger set of removed_snaps is already a member of pg_pool_t, which
is included in every osdmap.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: record purged_snaps for each epoch

Only do this if the mons are all running octopus (and thus there is also
a record for all the pre-octopus purged snaps).

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: make_snap_epoch_key -> make_removed_snap_epoch_key

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: add purged_snaps_last to OSDSuperblock

Make this 0 for old encodings or new superblocks.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: clean up initial values for OSDSuperblock

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: make {removed,purged}_snap storage more efficient

Merge adjacent extents to keep the database smaller and lookups more
efficient.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: move (removed, purged) snap update into a helper

This is trivial at the moment, but we'll make it clever next.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: generalize/refactor lookup_*_snap

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: refactor snap key and value helpers

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: make_snap_key -> make_removed_snap_key, make_purged_snap_key

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: fix lookup_purged_snap implementation

- look at purged, not removed snap keys
- fix the key check to look at the *key name* prefix, not the overall
  prefix (the one implemented by the KeyValueDB interface).

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: lookup_pruned_snap -> lookup_purged_snap

This confused me and sent me into the weeds.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: adjust snapmapper keys on first start as octopus

Convert snapmapper keys to the new form the first time we start up running
octopus.

This is an incompat feature--once you start as octopus you can't go back.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/SnapMapper: include poolid in snap index

We want to sort starting with (pool, snapid, ...) so that we align with
the structure of the purged_snaps.  Simply flattening all snaps across
pools is less than ideal because the purge records are intervals (with the
snap in the key the last snap for the interval); flattening means we'd have
to look at many records (across pools) to conclude anything.  Putting
these in the form we really want them simplifies things going forward.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: document osd snap metadata format

Signed-off-by: Sage Weil <sage@redhat.com>

osd/SnapMapper: document stored keys and values

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: use structured binding for prepare_remove_snaps

No functional change

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: send MRemoveSnaps back to octopus MDS

The octopus MDS wants an explicit ack when snaps are removed.

Signed-off-by: Sage Weil <sage@redhat.com>

mds/SnapServer: handle MRemoveSnaps acks from mon

Signed-off-by: Sage Weil <sage@redhat.com>

CMakeLists: include 'cephfs' (which includes libcephfs) in 'vstart' target

This ensures 'make vstart' will build libcephfs, which lets the mgr volumes
module start, such that we can successfully vstart.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PaxosService: add C_ReplyOp

Generic helper context to reply to an op on (successful) finish.

Signed-off-by: Sage Weil <sage@redhat.com>

vnewosd.sh: add script to add a new osd to an existing vstart

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: remove useless auth add for osds

'osd new' installs the key.

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: wait for mgr volume module to start up

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: make snap removal handle dups safely

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: only update removed_snaps when pre-octopus

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_rados: stop doing long object names

This was there to test filestore's long file name handling, which (1)
works, and (2) we don't care that much about anymore.  Meanwhile, the
long names make the OSD log files *really* painful to read.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_rados_api_tier_pp: fix osd version checks

These need to exclude old releases but run the test for newer, unnamed
releases.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: use get_ssc_as_of for snapc for flushing clones

We will stop maintaining SnapSet::snaps shortly.  Instead, generate this
snapc using the existing SnapSet::get_ssc_as_of() method, which will now
derive the snap list from the clone_snaps member.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: only maintain SnapSet::snaps for pre-octopus compat

Once we have all octopus or newer OSDs, we can stop maintaining
SnapSet::snaps.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: only maintain pg_pool_t::removed_snaps for pre-octopus

- do not examine removed_snaps
- do not add new items to removed_snaps unless we need pre-octopus compat
- clear removed_snaps in first octopus epoch

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: mark SnapSet::snaps as legacy

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: SnapSet::get_ssc_as_of: use clone_snaps

Fabricate a SnapContext from the clone_snaps instead of snaps (which we
are trying to kill).

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: change fabrication of promoted clone snaps

Instead of using SnapSet::snaps (which we are trying to kill), use the
clone_snaps[] snaps.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: only filter SnapSet::snaps for flush for pre-octopus compat

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: trim_objects: only filter SnapSet::snaps for pre-octopus

For octopus and later, we don't need SnapSet::snaps.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: make best effort to sanitize clones on copy-from

Signed-off-by: Sage Weil <sage@redhat.com>

mds/SnapServer: int -> int32_t for encoded type

Signed-off-by: Sage Weil <sage@redhat.com>

messages/MRemoveSnaps: int -> int32_t on encoded type

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: find_object_context: trust SnapSet's clone_snaps

Instead of checking the OSDMap pg_pool_t whether a snap exists, instead

1- Look at the clone_snaps more carefully.  If the snap didn't exist when
the clone was last touched (created or partially-trimmed) then it still
doesn't exist now (snaps aren't resurrected).

2- Check in the OSDMap's removed snaps queue.  This will catch anything
that is still being removed but hasn't been reflected by the clone_snaps
yet.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: use osdmap removed_snaps_queue for snap trimming

No need to use the pg_pool_t member now--the osdmap has a queue
specifically for the snaps we are in the process of trimming.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: avoid is_removed_snap()

Instead, consult the OSDMap's removed_snap_queue and the mon's record of
purged snaps.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PeeringState: drop some mimic conditionals

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: drop pre-mimic snap_trimq code

This is no longer needed for octopus.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PeeringState: removed pre-mimic removed snap tracking

Signed-off-by: Sage Weil <sage@redhat.com>

osd: move snap_interval_set_t to osd_types

Signed-off-by: Sage Weil <sage@redhat.com>

mon: drop mon_debug_no_require_mimic

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: remove pre-mimic snap behavior support

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: remove support for pre-mimic conversion

Cluster must already be mimic to run octopus.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: remove build_removed_snaps(), maybe_update_removed_snaps()

Signed-off-by: Sage Weil <sage@redhat.com>

osd: remove luminous compat code for removed_snaps

This is no longer necessary with peers >= mimic.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #28752 from xiexingguo/wip-reset-hb-session

osd/OSD: auto mark heartbeat sessions as stale and tear them down

Reviewed-by: yanjun <yan.jun8@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28731 from tchaikov/wip-40533

qa/tasks/ceph_manager.py: ignore errors in test_pool_min_size

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #28744 from tchaikov/wip-monitor-cleanup

mon/Monitor: no need to create a local variable for capturing it

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #28581 from Aran85/fix-mtime

osd: copyfrom omitted to set mtime

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

mgr/dashboard: Separate file for icons for refering and reusing (#27376)

mgr/dashboard: Separate file for icons for refering and reusing

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #28827 from Rjerk/fix-mgr-ansible-doc

doc/mgr/ansible.rst: fix typo

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28798 from tchaikov/wip-crimson-net

crimson/net: print tx/rx messages using logger().info()

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxincheng@gmail.com>

doc/mgr/ansible.rst: fix typo

Signed-off-by: Lan Liu <liulan@umcloud.com>

Merge pull request #28828 from iotcg/crimson

crimson/common: remove unused file .#log.cc

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28818 from tchaikov/wip-ceph-conf-cli-test

test/cli/ceph-conf: fix test

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #28805 from Rjerk/fix-doc

doc/rados/configuration: fix typos in osd-config-ref.rst

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/common: remove unused file .#log.cc

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

crimson/net: print tx/rx messages using logger().debug()

so we can find them with less efforts.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/net: print trivial logging using "trace"

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/net: note down server's peer_id as client

the peer_id is not used at this moment. this change is made just for
the sake of completeness.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/net: use Connection::peer_is_mon()

always prefer using interface over its implementation

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: store ticket.peer_global_id in connection

peer_global_id is used by `ProtocolV2::_handle_auth_request()` when
encoding an `AuthDoneFrame`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/rados/configuration: fix typos in osd-config-ref.rst

Signed-off-by: Lan Liu <liulan@umcloud.com>

Merge pull request #28795 from tchaikov/wip-cmake-rbd

CMakeLists.txt: fix typo in error message

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Milind Changire <mchangir@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #28750 from neha-ojha/wip-40488

doc/rados/operations/erasure-code.rst: allow recovery below min_size

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #27073 into master

* refs/pull/27073/head:
qa/tasks: Check MDS failover during mon_thrash
qa/tasks: Compare two FSStatuses
qa/suites/fs: renamed default.yaml to mds.yaml
qa/suites/fs: mon_thrash test for fs
qa/tasks: Fix typo in the comment

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #28459 into master

* refs/pull/28459/head:
mds: wake up lock waiters after forcibly changing lock state

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #28642 into master

* refs/pull/28642/head:
mds: check last laggy before marking unresponsive client stale
mds: remove the code that skip evicting the only client

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #28632 from linuxbox2/wip-rgwfile-advexp

rgw_file: advance_mtime() should consider namespace expiration

Merge pull request #27794 from cbodley/wip-39487

rgw: data/bilogs are trimmed when no peers are reading them

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #28743 from dillaman/wip-40417

librbd: prevent concurrent AIO callbacks to external clients

Reviewed-by: Mykola Golub <mgolub@suse.com>

test/cli/ceph-conf: fix test

turns out the libstdc++ on fc30 does not include error_code.message() in
filesystem_error.what(). let's make it optional

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28800 from tchaikov/wip-crimson-mkfs

crimson: use given osd_fsid when mkfs

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #28774 from falcon78921/wip-doc-40575

doc: fixed broken link in Swift Settings section

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: fixed broken link in Swift Settings section

Fixes: https://tracker.ceph.com/issues/40575
Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>

Merge pull request #28786 from tchaikov/wip-const-head

osd/PrimaryLogPG: more constness

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

crimson: use given osd_fsid when mkfs

MemStore does not have this feature, but CyanStore needs it. because
`qa/tasks/ceph.py` uses following steps when creating/starting an OSD

1. ceph-osd --mkfs --mkkey -i <osd_id> --monmap <monmap>
2. read the osd_fsid of osd.<osd_id>
3. ceph osd new <osd_fsid> <osd_id>

when we mkfs for the OSD, the osd_fsid is still unknown. so we cannot
use the configured one, as it is always empty. in that case, we need to
use a random uuid, and persist it to both ${osd_data}/fsid and
superblock.osd_fsid

Signed-off-by: Kefu Chai <kchai@redhat.com>