crimson/osd: shutdown services in the right order

we should stop config service *after* osd is stopped, as osd depends on
a working and alive config subsystem when stopping itself. for instance,
the destructor of AuthRegistry unregisters itself from the ObserverMgr,
which is in turn a member variable of ConfigProxy, so if ConfigProxy is
destroyed before we destroy mon::Client, we will have a segfault with
following backtrace

ObserverMgr<ceph::md_config_obs_impl<ceph::common::ConfigProxy>
>::remove_observer(ceph::md_config_obs_impl<ceph::common::ConfigProxy>*)
at /var/ssd/ceph/build/../src/common/config_obs_mgr.h:78
AuthRegistry::~AuthRegistry() at
/var/ssd/ceph/build/../src/crimson/common/config_proxy.h:101
 (inlined by) AuthRegistry::~AuthRegistry() at
/var/ssd/ceph/build/../src/auth/AuthRegistry.cc:28
ceph::mon::Client::~Client() at
/var/ssd/ceph/build/../src/crimson/mon/MonClient.h:44
ceph::mon::Client::~Client() at
/var/ssd/ceph/build/../src/crimson/mon/MonClient.h:44
OSD::~OSD() at /usr/include/c++/9/bits/unique_ptr.h:81

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26152 from noonedeadpunk/mgr/zabbix_discovery

mgr/zabbix Added pools discovery and per-pool statistics

Reviewed-by: Wido den Hollander <wido@42on.com>

Apply suggestions from code review

Signed-off-by: Dmitriy Rabotjagov <noonedeadpunk@ya.ru>

Merge pull request #27967 from liewegas/wip-fix-rgw-bias

rgw/rgw_tools: fix 'osd pool set' json syntax

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: Allow the decrease of pg's of an existing pool (#27785)

mgr/dashboard: Allow the decrease of pg's of an existing pool

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #27977 from cbodley/wip-39597

doc/rgw: document CreateBucketConfiguration for s3 PUT Bucket api

Reviewed-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #27968 from iotcg/master

vstart.sh: correct ceph-run path

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/rgw: document CreateBucketConfiguration for s3 PUT Bucket api

Fixes: http://tracker.ceph.com/issues/39597
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27970 from Liuchang0812/duplicate-tags-bl

rgw: eliminates duplicated tags_bl var

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc: fix small typo in dashboard documentation (#27850)

doc: fix small typo in dashboard documentation

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

vstart.sh: correct ceph-run path

ceph-run is in the same directory as vstart.sh. It's often that
vstart.sh is run under build directory. Without giving the right
directory, ceph-run file can't be found.

Signed-off-by: Changcheng Liu <changcheng.liu@intel.com>

rgw: eliminates duplicated tags_bl var

Signed-off-by: Chang Liu <liuchang0812@gmail.com>

Merge pull request #27538 from iotcg/radosgw

init-radosgw: use ceph-conf to get cluster configuration value

Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw/rgw_tools: fix 'osd pool set' json syntax

Fixes af30e3a512dd1e5c51639d1f823b99ed9b43fb50

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #27885 from b-ranto/wip-restful-perf-counters

restful: Expose perf counters

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27891 from tchaikov/wip-mgr-pgp-num-change-throttling

mgr/DaemonServer: refactor pgp_num changes throttling

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #27960 from yuriw/wip-yuriw-crontab-master

qa/tests: removed all runs on ovh

Merge pull request #27896 from ivancich/wip-add-rgw-period-comments

rgw: add some comments to rgw code to help explain functionality

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27956 from yuriw/wip-yuriw-smoke-master

qa/tests - cleaned up distro settings

qa/tests: removed all runs on ovh

and consolidated `smoke` runs re: https://github.com/ceph/ceph/pull/27956

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

rgw: add some comments to rgw code to help explain functionality

Comments to explain the role of RGWPeriod and the use of prefixes in
gc log entries are added. Some other comments are reformatted for 80
columns.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #27838 from yuvalif/wip-yuval-pubsub_teuthology-new

rgw/pubsub: add multisite pubsub tests to teuthology

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/tests - cleaned up distro settings

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #27448 from ricardoasmarques/upgrade-ceph-iscsi-version

mgr/dashboard: Support ceph-iscsi config v9

Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #27887 from smithfarm/wip-bci-len255plus

script/backport-create-issue: handle long Redmine issue names

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #27895 from dillaman/wip-librbd-zero-copy

librbd: support zero-copy writes via the C API

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #16929 from amitkumar50/cov-rgw-1

rgw: Initialize member variables in rgw_sync.h, rgw_rados.h

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #27927 from tchaikov/wip-util.collect_sys_info-is-optional

test/common/test_util: skip it if /etc/os-release does not exist

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #27897 from ivancich/wip-add-editor-comments-rgw-services

rgw: add editor directive comments to rgw services source files

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27820 from smithfarm/wip-39361

rgw: drop cloud sync module logs attrs from the log

Reviewed-By: Casey Bodley <cbodley@redhat.com>
Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

test: do not conditionize on GTEST_HAS_PARAM_TEST

value-parameterized is supported on all platforms, so drop this check

Signed-off-by: Kefu Chai <kchai@redhat.com>

test: s/INSTANTIATE_TEST_CASE_P/INSTANTIATE_TEST_SUITE_P/

the former is deprecated

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/valgrind.supp: added false-positive in boost::queue::lockfree

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test/rbd_mirror: add "using ::testing::DoAll"

we should `using` it before calling it using `DoAll`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/common/test_util: skip it if /etc/os-release does not exist

some GNU/Linux distros do not ship this file, and we should not fail the
test on them.

inspired by
http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/patches/ceph-skip-collect-sys-info-test.patch?id=48f19e60c4677e392ee2c23f28098cfcaf9d1710

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not install googletest

the new googletest added an option to allow its consumer to skip this,
so instead of patching googletest, let's use this option.

Signed-off-by: Kefu Chai <kchai@redhat.com>

googletest: pick up GTEST_SKIP() changes

so we can use it to disable test at runtime

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw/pubsub: fix more test issues with teuthology failures

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>

librbd: use zero-copy on write via the C API when cache is disabled

It will also be disabled when using synchronous write API methods.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

common/deleter: added missing headers

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: allow AioCompletion objects to be blocked

This will be used when user-provided memory is wrapped into a
ceph::buffer::raw pointer to prevent its release prior to the
drop of its last reference internally.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: AioCompletion fail path should complete in clean thread

Ensure that no internal librbd locks are being held when the
callbacks are invoked for AioCompletion::fail.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: simplify AioCompletion reference counting for 'fail' case

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: switch to lock-free queue for event poll IO interface

'perf' shows several percent of CPU being wasted on lock contention
in the event poll interface. The 'fio' RBD engine uses this poll
IO interface by default when available.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: avoid using lock within AIO completion where possible

'perf' shows several percent of CPU is being utilized handling the
heavyweight locking semantics of AIO completion. With these changes,
the lock contention disappears.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: remove special case for starting AioCompletion ops

All ops can be immediately started now that flush ops won't
accidentally block themselves.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: simplify IO flush handling through AsyncOperation

Allow ImageFlushRequest to directly execute a flush call through
AsyncOperation. This will allow the flush to be directly linked
to its preceeding IOs.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #27871 into master

* refs/pull/27871/head:
ceph_test_objectstore: add very_large_write test
os/bluestore: fix aio pwritev lost data problem.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

rgw/pubsub: fix test issue with 3 zones

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>

Merge pull request #27844 from wjwithagen/wjw-fix-src_common_bit_vector.hpp.diff

common: Clang requires a default constructor, but it can be empty

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #27874 from athanatos/sjust/wip-peering-refactor-forreview

Extract peering logic into a module for use in crimson

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #27763 into master

* refs/pull/27763/head:
common/PriorityCache: fix over-aggressive assert when mem limited

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

common/PriorityCache: fix over-aggressive assert when mem limited

Fixes: https://tracker.ceph.com/issues/39437
Signed-off-by: Mark Nelson <mnelson@redhat.com>

PGStateUtils: improvements for PGStateHistory

Signed-off-by: Samuel Just <sjust@redhat.com>

PeeringState: don't zero backfill target num_bytes on activation

834d3c19a774f1cc93903447d91d182776e12d18 preserves num_bytes
on backfill targets in order to estimate space required to complete
backill.  However, from activation until backfill reservation,
info.stats.stats.sum.num_bytes is persisted to disk as 0 messing
up future intervals.  Instead, preserve it in the info sent during
recovery and leave it alone in RequestBackfillPrio.

Additionally, it's possible for backfill to be preempted between
last_backfill=MAX being sent to the replica and Backfilled being
queued occuring.  In that case, the stats get on reservation
and the replica ends up with invalid stats.

Fixes: https://tracker.ceph.com/issues/39401
Signed-off-by: sjust@redhat.com <sjust@redhat.com>

PeeringState: use ceph_assert

Signed-off-by: Samuel Just <sjust@redhat.com>

admin/build-doc: use PeeringState* for gen_state_diagram.py

Signed-off-by: Samuel Just <sjust@redhat.com>

PeeringState: add explanations for public interface methods

Also rearranges the methods a little for clarity.

Signed-off-by: Samuel Just <sjust@redhat.com>

PG,PeeringState: Fix initialization order

PeeringState needs to be initialized last and destructed
first.

Signed-off-by: sjust@redhat.com <sjust@redhat.com>

PeeringState: mark state and helpers private

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: clean up remaining info mutators

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: condense missing mutations for recovery/repair/errors

At a high level, this patch attempts to unify the various
sites at which some combination of
- mark object missing in one or more pg_missing_t
- mark object needs_recovery in missing_loc
- manipulate the locations map based on external information
occur.  It seems to me that the pg_missing_t and missing_loc
should be in sync except for the mark_unfound_lost_revert
case and the case where we are about to do a backfill push.

This patch also cleans up repair_object.  It sort of worked by accident
for non-ec non-primary bad peers.  It didn't update missing_loc, so
needs_recovery() returns the wrong answer.  However, is_unfound() does
as well, so ReplicatedBackend is nevertheless happy as the object would
be present on the primary.  This patch updates the behavior to be
uniform as in the other force_obejct_missing cases.

Signed-off-by: Samuel Just <sjust@redhat.com>

test-erasure-eio: first eio may be fixed during recovery

The changes to the way EC/ReplicatedBackend communicate read
t showerrors had a side effect of making first eio on the object in
TEST_rados_get_subread_eio_shard_[01] repair itself depending
on the timing of the killed osd recovering.  The test should
be improved to actually test that behavior at some point.

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move pg_log, missing_loc  mutations to PeeringState

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: unify PGBackend pull error pathways

This patch narrows the PGBackend -> PrimaryLogPG recovery
cancel/error interface to on_failed_pull and cancel_pull.

This patch requires careful review.

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move peer_info mutators into PeeringState

Signed-off-by: sjust@redhat.com <sjust@redhat.com>

osd/: Move log version pointer updates to PeeringState

Signed-off-by: sjust@redhat.com <sjust@redhat.com>

osd/: refactor to avoid mutable peer_missing refs in PG

Signed-off-by: sjust@redhat.com <sjust@redhat.com>

PG: remove might_have_unfound, peer_*_requested

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: fix upset, actingset, acting_backfill_recovery references

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: fix primary/up_primary references

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move last_..._to_applied, backfill_targets, async_recovery_targets to PeeringState

Signed-off-by: Samuel Just <sjust@redhat.com>

PG: fix last_peering_reset and past_intervals references

Signed-off-by: Samuel Just <sjust@redhat.com>

PeeringState::PeeringMachine::Deleting: rollfoward before resetting backfill

This looked wrong when I was moving it over.

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: add helpers to add remaining info dirtiers into PeeringState

Signed-off-by: sjust@redhat.com <sjust@redhat.com>

osd/: move append_log into PeeringState

Signed-off-by: sjust@redhat.com <sjust@redhat.com>

osd/: move append_log_entries_update_missing and merge_new_log_entries to PeeringState

Signed-off-by: sjust@redhat.com <sjust@redhat.com>

osd/: clean up PG deleted and deleting references

Signed-off-by: sjust@redhat.com <sjust@redhat.com>

PG: remove direct acting and up references

Signed-off-by: sjust@redhat.com <sjust@redhat.com>

PG: begin cleaning up scrub stat and history mutations

Signed-off-by: sjust@redhat.com <sjust@redhat.com>

PG: remove first batch of unused references

Signed-off-by: sjust@redhat.com <sjust@redhat.com>

osd/: clarify interface for introducing disk state to PeeringState

Signed-off-by: sjust@redhat.com <sjust@redhat.com>

osd/: move ostream<< and dump logic into PeeringState

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: clean up PeeringState::write_if_dirty

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: Move init into PeeringState

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move stat updates and publishing to PeeringState

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move more state helpers to PeeringState

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move split/merge helpers into PeeringState

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: fix try_mark_clean/finish_recovery interface boundary

Signed-off-by: Samuel Just <sjust@redhat.com>

PeeringState: remove PG references and include

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move calc_min_last_complete_ondisk to PeeringState

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: pass OSDService::logger and log_enter/exit through PeeringListener

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: pass merge state control through PeeringListener

Signed-off-by: Samuel Just <sjust@redhat.com>

PeeringState: pass dump_recovery_info through to pl for recovery state

Signed-off-by: Samuel Just <sjust@redhat.com>

PeeringState: rearrange and fix remaining pg->gen_prefix call

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move fulfill_info etc

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: clarify PeeringState deletion interfaces

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move min_peer_*_features and helpers

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move proc_* and log handling methods

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move share_pg_info to PeeringState

Signed-off-by: Samuel Just <sjust@redhat.com>