mon/OSDMonitor: record purged_snaps for each epoch

Only do this if the mons are all running octopus (and thus there is also
a record for all the pre-octopus purged snaps).

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: make_snap_epoch_key -> make_removed_snap_epoch_key

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: add purged_snaps_last to OSDSuperblock

Make this 0 for old encodings or new superblocks.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: clean up initial values for OSDSuperblock

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: make {removed,purged}_snap storage more efficient

Merge adjacent extents to keep the database smaller and lookups more
efficient.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: move (removed, purged) snap update into a helper

This is trivial at the moment, but we'll make it clever next.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: generalize/refactor lookup_*_snap

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: refactor snap key and value helpers

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: make_snap_key -> make_removed_snap_key, make_purged_snap_key

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: fix lookup_purged_snap implementation

- look at purged, not removed snap keys
- fix the key check to look at the *key name* prefix, not the overall
  prefix (the one implemented by the KeyValueDB interface).

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: lookup_pruned_snap -> lookup_purged_snap

This confused me and sent me into the weeds.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: adjust snapmapper keys on first start as octopus

Convert snapmapper keys to the new form the first time we start up running
octopus.

This is an incompat feature--once you start as octopus you can't go back.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/SnapMapper: include poolid in snap index

We want to sort starting with (pool, snapid, ...) so that we align with
the structure of the purged_snaps.  Simply flattening all snaps across
pools is less than ideal because the purge records are intervals (with the
snap in the key the last snap for the interval); flattening means we'd have
to look at many records (across pools) to conclude anything.  Putting
these in the form we really want them simplifies things going forward.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: document osd snap metadata format

Signed-off-by: Sage Weil <sage@redhat.com>

osd/SnapMapper: document stored keys and values

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: use structured binding for prepare_remove_snaps

No functional change

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: send MRemoveSnaps back to octopus MDS

The octopus MDS wants an explicit ack when snaps are removed.

Signed-off-by: Sage Weil <sage@redhat.com>

mds/SnapServer: handle MRemoveSnaps acks from mon

Signed-off-by: Sage Weil <sage@redhat.com>

CMakeLists: include 'cephfs' (which includes libcephfs) in 'vstart' target

This ensures 'make vstart' will build libcephfs, which lets the mgr volumes
module start, such that we can successfully vstart.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PaxosService: add C_ReplyOp

Generic helper context to reply to an op on (successful) finish.

Signed-off-by: Sage Weil <sage@redhat.com>

vnewosd.sh: add script to add a new osd to an existing vstart

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: remove useless auth add for osds

'osd new' installs the key.

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: wait for mgr volume module to start up

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: make snap removal handle dups safely

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: only update removed_snaps when pre-octopus

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_rados: stop doing long object names

This was there to test filestore's long file name handling, which (1)
works, and (2) we don't care that much about anymore.  Meanwhile, the
long names make the OSD log files *really* painful to read.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_rados_api_tier_pp: fix osd version checks

These need to exclude old releases but run the test for newer, unnamed
releases.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: use get_ssc_as_of for snapc for flushing clones

We will stop maintaining SnapSet::snaps shortly.  Instead, generate this
snapc using the existing SnapSet::get_ssc_as_of() method, which will now
derive the snap list from the clone_snaps member.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: only maintain SnapSet::snaps for pre-octopus compat

Once we have all octopus or newer OSDs, we can stop maintaining
SnapSet::snaps.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: only maintain pg_pool_t::removed_snaps for pre-octopus

- do not examine removed_snaps
- do not add new items to removed_snaps unless we need pre-octopus compat
- clear removed_snaps in first octopus epoch

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: mark SnapSet::snaps as legacy

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: SnapSet::get_ssc_as_of: use clone_snaps

Fabricate a SnapContext from the clone_snaps instead of snaps (which we
are trying to kill).

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: change fabrication of promoted clone snaps

Instead of using SnapSet::snaps (which we are trying to kill), use the
clone_snaps[] snaps.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: only filter SnapSet::snaps for flush for pre-octopus compat

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: trim_objects: only filter SnapSet::snaps for pre-octopus

For octopus and later, we don't need SnapSet::snaps.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: make best effort to sanitize clones on copy-from

Signed-off-by: Sage Weil <sage@redhat.com>

mds/SnapServer: int -> int32_t for encoded type

Signed-off-by: Sage Weil <sage@redhat.com>

messages/MRemoveSnaps: int -> int32_t on encoded type

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: find_object_context: trust SnapSet's clone_snaps

Instead of checking the OSDMap pg_pool_t whether a snap exists, instead

1- Look at the clone_snaps more carefully.  If the snap didn't exist when
the clone was last touched (created or partially-trimmed) then it still
doesn't exist now (snaps aren't resurrected).

2- Check in the OSDMap's removed snaps queue.  This will catch anything
that is still being removed but hasn't been reflected by the clone_snaps
yet.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: use osdmap removed_snaps_queue for snap trimming

No need to use the pg_pool_t member now--the osdmap has a queue
specifically for the snaps we are in the process of trimming.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: avoid is_removed_snap()

Instead, consult the OSDMap's removed_snap_queue and the mon's record of
purged snaps.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PeeringState: drop some mimic conditionals

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: drop pre-mimic snap_trimq code

This is no longer needed for octopus.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PeeringState: removed pre-mimic removed snap tracking

Signed-off-by: Sage Weil <sage@redhat.com>

osd: move snap_interval_set_t to osd_types

Signed-off-by: Sage Weil <sage@redhat.com>

mon: drop mon_debug_no_require_mimic

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: remove pre-mimic snap behavior support

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: remove support for pre-mimic conversion

Cluster must already be mimic to run octopus.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: remove build_removed_snaps(), maybe_update_removed_snaps()

Signed-off-by: Sage Weil <sage@redhat.com>

osd: remove luminous compat code for removed_snaps

This is no longer necessary with peers >= mimic.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #28752 from xiexingguo/wip-reset-hb-session

osd/OSD: auto mark heartbeat sessions as stale and tear them down

Reviewed-by: yanjun <yan.jun8@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28731 from tchaikov/wip-40533

qa/tasks/ceph_manager.py: ignore errors in test_pool_min_size

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #28744 from tchaikov/wip-monitor-cleanup

mon/Monitor: no need to create a local variable for capturing it

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #28581 from Aran85/fix-mtime

osd: copyfrom omitted to set mtime

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

mgr/dashboard: Separate file for icons for refering and reusing (#27376)

mgr/dashboard: Separate file for icons for refering and reusing

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #28827 from Rjerk/fix-mgr-ansible-doc

doc/mgr/ansible.rst: fix typo

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28798 from tchaikov/wip-crimson-net

crimson/net: print tx/rx messages using logger().info()

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxincheng@gmail.com>

doc/mgr/ansible.rst: fix typo

Signed-off-by: Lan Liu <liulan@umcloud.com>

Merge pull request #28828 from iotcg/crimson

crimson/common: remove unused file .#log.cc

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28818 from tchaikov/wip-ceph-conf-cli-test

test/cli/ceph-conf: fix test

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #28805 from Rjerk/fix-doc

doc/rados/configuration: fix typos in osd-config-ref.rst

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/common: remove unused file .#log.cc

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

crimson/net: print tx/rx messages using logger().debug()

so we can find them with less efforts.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/net: print trivial logging using "trace"

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/net: note down server's peer_id as client

the peer_id is not used at this moment. this change is made just for
the sake of completeness.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/net: use Connection::peer_is_mon()

always prefer using interface over its implementation

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: store ticket.peer_global_id in connection

peer_global_id is used by `ProtocolV2::_handle_auth_request()` when
encoding an `AuthDoneFrame`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/rados/configuration: fix typos in osd-config-ref.rst

Signed-off-by: Lan Liu <liulan@umcloud.com>

Merge pull request #28795 from tchaikov/wip-cmake-rbd

CMakeLists.txt: fix typo in error message

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Milind Changire <mchangir@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #28750 from neha-ojha/wip-40488

doc/rados/operations/erasure-code.rst: allow recovery below min_size

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #27073 into master

* refs/pull/27073/head:
qa/tasks: Check MDS failover during mon_thrash
qa/tasks: Compare two FSStatuses
qa/suites/fs: renamed default.yaml to mds.yaml
qa/suites/fs: mon_thrash test for fs
qa/tasks: Fix typo in the comment

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #28459 into master

* refs/pull/28459/head:
mds: wake up lock waiters after forcibly changing lock state

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #28642 into master

* refs/pull/28642/head:
mds: check last laggy before marking unresponsive client stale
mds: remove the code that skip evicting the only client

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #28632 from linuxbox2/wip-rgwfile-advexp

rgw_file: advance_mtime() should consider namespace expiration

Merge pull request #27794 from cbodley/wip-39487

rgw: data/bilogs are trimmed when no peers are reading them

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #28743 from dillaman/wip-40417

librbd: prevent concurrent AIO callbacks to external clients

Reviewed-by: Mykola Golub <mgolub@suse.com>

test/cli/ceph-conf: fix test

turns out the libstdc++ on fc30 does not include error_code.message() in
filesystem_error.what(). let's make it optional

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28800 from tchaikov/wip-crimson-mkfs

crimson: use given osd_fsid when mkfs

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #28774 from falcon78921/wip-doc-40575

doc: fixed broken link in Swift Settings section

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: fixed broken link in Swift Settings section

Fixes: https://tracker.ceph.com/issues/40575
Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>

Merge pull request #28786 from tchaikov/wip-const-head

osd/PrimaryLogPG: more constness

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

crimson: use given osd_fsid when mkfs

MemStore does not have this feature, but CyanStore needs it. because
`qa/tasks/ceph.py` uses following steps when creating/starting an OSD

1. ceph-osd --mkfs --mkkey -i <osd_id> --monmap <monmap>
2. read the osd_fsid of osd.<osd_id>
3. ceph osd new <osd_fsid> <osd_id>

when we mkfs for the OSD, the osd_fsid is still unknown. so we cannot
use the configured one, as it is always empty. in that case, we need to
use a random uuid, and persist it to both ${osd_data}/fsid and
superblock.osd_fsid

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28719 from majianpeng/remove-unused-code-XioMessenger

msg/Message: Remove used code about XioMessenger.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28794 from tchaikov/wip-crimson-option-help-seastar

crimson/osd: add "--help-seastar" command line option

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

CMakeLists.txt: s/Remote block storage/RADOS Block Device/

RBD is short for RADOS Block Device, not Remote Block Storage.

Signed-off-by: Kefu Chai <kchai@redhat.com>

CMakeLists.txt: fix typo in error message

krbd depends on rbd, so we cannot build krbd without rbd.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/dev/crimson.rst: update with stuff related to seastar options

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28792 from tchaikov/wip-crimson-logging

common/config_values: set seastar logging level per that of ceph

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #28790 from tchaikov/wip-crimson-unknown-objectstore

crimson/osd: abort on unsupported objectstore type

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: add "--help-seastar" command line option

so we can

* have access to the available command line options offered by Seastar.
* tell if the executable we are playing around is ceph-osd or crimson-osd.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/dev/crimson.rst: update logging related doc

as crimson's logging levels are now connected those of ceph, there is no
need to note down the difference between them anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/config_values: set seastar logging level as well

when the logging level of a subsys changes, update seastar accordingly

before this change, seastar logging levels of different logger are not
connected to ceph's logging levels of different subsystem.

after this change, they are connected. whenever a subsys's logging level
changes, the corresponding seastar logger is updated as well.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/dout.h: extract log level mapping out

so we can reuse it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: revert 9813f7d6

since we have a proper PeeringState implementation, there is no need to
fake an active+clean PG anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: abort on unsupported objectstore type

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: add editor variables/options

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/common: implement "__ceph_abortf()"

the macro of `ceph_abort_msgf()` is implemented using it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: add a space before function name

for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: use get_osdmap_epoch()

always prefer using the interface.

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/PrimaryLogPG: more constness

use `hobject_t::get_head()` instead of relying on the fact that head
object's snap is `CEPH_NOSNAP`.

Signed-off-by: Kefu Chai <kchai@redhat.com>