osd: do not authenticate heartbeat connections until nautilus

Some (currently) pre-nautilus OSDs will crash if you try to authenticate
a heartbeat connection but they are not expecting it:

src/auth/Crypto.h: 109: FAILED assert(ckh)
 ceph version 12.2.8-457-gccd69ef (ccd69ef36aafebab964a2e47e249fdb95e083e46) luminous (stable)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110) [0x5618b04aaea0]
 2: (()+0x41cbec) [0x5618afe2cbec]
 3: (CephxSessionHandler::_calc_signature(Message*, unsigned long*)+0x8c5) [0x5618b0777ba5]
 4: (CephxSessionHandler::check_message_signature(Message*)+0x7d) [0x5618b077800d]
 5: (AsyncConnection::process()+0x1b44) [0x5618b0761a04]
 6: (EventCenter::process_events(int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0x359) [0x5618b0546079]
 7: (()+0xb38c3e) [0x5618b0548c3e]
 8: (()+0xb5070) [0x7ff04faf5070]
 9: (()+0x7dd5) [0x7ff050168dd5]
 10: (clone()+0x6d) [0x7ff04f259b3d]

See http://tracker.ceph.com/issues/36443

It won't be fixed in all clusters before upgrade to nautilus, though, so
we also need to work around it here.

Signed-off-by: Sage Weil <sage@redhat.com>

msg,osd: enable unauthenticated Dispatcher for pre-nautilus OSD compat

Before nautilus, osd heartbeats are sent over an unauthenticated channel.
We need support here to allow these connections when they are necessary
for upgrade compatibility.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: add missing space in heartbeat debug output

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/DaemonServer: add missing return

Fixes: http://tracker.ceph.com/issues/36110
Signed-off-by: Sage Weil <sage@redhat.com>

msg/async: drop verify_authorizer wrapper

Signed-off-by: Sage Weil <sage@redhat.com>

osd: authenticate ping sessions

Do not set up a Session object, though--nobody cares (currently!).

This avoids having to special-case the generic authorizer validation
code in msg/* to have to handle non-authenticated sessions.  Also, it
seems like a good idea to authenticate these sessions!

Signed-off-by: Sage Weil <sage@redhat.com>

msg/simple: remove verify_authorizer wrapper

Signed-off-by: Sage Weil <sage@redhat.com>

msg: remove unused ms_verify_authorizer

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async: remove get_authorizer wrapper

Signed-off-by: Sage Weil <sage@redhat.com>

msg/simple: remove get_authorizer wrapper

Signed-off-by: Sage Weil <sage@redhat.com>

msg/Messenger: pull authenticator validation into Messenger

This code is essentially identical across the OSD and MDS.  The
monitor is annoyingly different, but in a msgr1 specific way that
we can handle carrying here until msgr1 gets ripped out in a
couple years.

Signed-off-by: Sage Weil <sage@redhat.com>

msg/Messenger: uninline ms_deliver_verify_authorizer

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/DaemonServer: expose keyring for authenticator verification

Signed-off-by: Sage Weil <sage@redhat.com>

mon: expose keyring for msgr1 authentication

Signed-off-by: Sage Weil <sage@redhat.com>

mds: expose keyring for authenticater verification

Signed-off-by: Sage Weil <sage@redhat.com>

osd: expose keyring for authenticater verification

Signed-off-by: Sage Weil <sage@redhat.com>

msg/Dispatcher: add ms_get_auth1_authorizer_keystore

This is there to provide the keyring used for authenticating msgr1
authorizers.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: fix ref cycle breakage in handle_forward

We now rely on the session -> connection ref for printing
remote addr, peer_global_id, and so on.  Change this code to
break the ref cycle instead by removing the con->session link,
which is only needed by the MonOpRequest ctor called at the top
of _ms_dispatch.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: use MonOpRequest get_session() instead of PaxosServiceMessage's

The PaxosServiceMessage method relies on the msg -> con -> session linkage,
and the con -> session link is not present for forwarded messages.  Also,
the message path is redundant and unnecessary.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: get session from MonOpRequest in handle_command

We should have made this switchover a long time ago.

Signed-off-by: Sage Weil <sage@redhat.com>

messages/MForward: drop unused ctor

Signed-off-by: Sage Weil <sage@redhat.com>

mon: use ms_handle_authentication to parse caps

The situation is a bit different here than the MDS and OSD because the
authentication happens from MAuth instead of ms_verify_authorizer, but
we are moving toward being more consistent.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: kill Session::global_id and use Connection member instead

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/DaemonServer: move session setup into ms_handle_authentication

Signed-off-by: Sage Weil <sage@redhat.com>

mds: move session setup into ms_handle_authentication

Signed-off-by: Sage Weil <sage@redhat.com>

osd: move session setup into ms_handle_authentication

Signed-off-by: Sage Weil <sage@redhat.com>

msg: new ms_handle_authentication, add fields to Connection

Lay some groundwork to eliminate ms_verify_authorizer.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #24146 into master

* refs/pull/24146/head:
mgr/orchestrator: Fix Python 3 issues

Reviewed-by: John Spray <john.spray@redhat.com>

Merge PR #24247 into master

* refs/pull/24247/head:
PendingReleaseNotes: add note about increased mon memory footprint
doc/start/hardware-recommendations: refresh recommendations for RAM
rocksdb: increase default cache size to 512 MB
mon: mon_osd_cache_size = 500 (from 10)

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #24495 into master

* refs/pull/24495/head:
kv: apply the move semantics in KeyValueDB::get().

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #24535 into master

* refs/pull/24535/head:
mon: consider AUTH_NONE clients which "authenticate" to be finished

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #24563 from dillaman/wip-36410

test: move OpenStack devstack test to rocky release

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #24564 from neha-ojha/wip-21931

osd: do not overestimate the size of the object for reads with trimtrunc

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed By: Xie Xingguo <xie.xingguo@zte.com.cn>

Merge PR #24204 into master

* refs/pull/24204/head:
qa/suites/rgw/tempest: valgrind on centos only

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge PR #24249 into master

* refs/pull/24249/head:
doc/rados/troubleshooting-mon: update mondb recovery script

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #24296 into master

* refs/pull/24296/head:
osd: Handle is_stopping() by discarding item so it doesn't loop
osd: Simplify _process() logic recently added

Reviewed-by: Jianpeng Ma <jianpeng.ma@intel.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #24131 into master

* refs/pull/24131/head:
tests, common: introduce bufferlist::apennd benchmark.
test, common: add BufferListIterator.BenchDeref benchmark.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #24270 into master

* refs/pull/24270/head:
osd: make 'cache drop' command require 'executable' permission
osd: rename 'drop cache' and 'get cache stats' to group them by component
doc: add documentation for 'drop cache' and 'get cache stats'
osd: don't print osdmap cache stats in 'get cache stats' command
osd: do not clear osdmap cache on 'drop cache' command
osd: offload dumping cache stats to the object store
osd: pass a stream to flush_cache commands for more verbosity
osd: implement flush_cache() method for Filestore
osd: add clear_cache and get_cache_object_count commands

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #24312 into master

* refs/pull/24312/head:
osd: kill the std::stringstream in ReplicatedBackend::do_repop_reply.
osd: bump-up the dout level in PGLog::write_log_and_missing.

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #24520 into master

* refs/pull/24520/head:
install-deps.sh: dashboard frontend needs git

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge PR #24521 into master

* refs/pull/24521/head:
ceph.in: reinstate interactive mode output

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

osd/PrimaryLogPG.cc: reassign size only when object size > truncate_size

Before setting size equal to op.extent.truncate_size, we need to check
if the size of the object is greater than the truncate_size. We do not
need to set size to op.extent.truncate_size, in the case where the size of
the object is less than op.extent.truncate_size.

Without this change, we were always setting size =
op.extent.truncate_size, when (seq < op.extent.truncate_seq) and
(op.extent.offset + op.extent.length > op.extent.truncate_size), were both
true. This ended up in:

1. overestimating the size of the object
2. not considering the correct size of the object, for
   the later checks, which calculate op.extent.length for the read ops
3. causing crashes when trying to read more data than what was present

Fixes: http://tracker.ceph.com/issues/21931
Fixes: http://tracker.ceph.com/issues/22330
Signed-off-by: Neha Ojha <nojha@redhat.com>

osd: print offset and length to track trimtrunc

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #24341 from joke-lee/WIP_FIX_RGWPOSTOBJ_VERSIONING

rgw: list bucket can not show the object uploaded by RGWPostObj when enable bucket versioning

qa/tasks/workunit: use suite branch/SHA1 when cloning workunits

Right now it's using the Ceph branch/SHA1 but it's using the suite
Git URL.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits/rbd: switch devstack to rocky branch and tempest to 19.0.0 tag

Fixes: http://tracker.ceph.com/issues/36410
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #24550 from dillaman/wip-36409

qa/workunits: replace 'realpath' with 'readlink -f' in fsstress.sh

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #24517 from jcsp/wip-23970-test

mgr/selftest: fix disabled module selection

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #24426 from Miouge1/grafana-fix1

monitoring/grafana: Fix OSD Capacity Utlization Grafana graph

Merge pull request #24511 from a2batic/grafana-doc

mgr/dashboard: Improves documentation for Grafana Setting

Reviewed-by: Kai Wagner <kwagner@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #24513 from rhcs-dashboard/27047-landing-page-info-visibility

mgr/dashboard: Landing Page: info visibility

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #24222 from votdev/bug_36109

mgr/dashboard: The RGW backend doesn't handle IPv6 properly

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

qa/tasks: qemu task now uses a relative path in suite repo for test

This makes it easier to re-run tests against a suite branch without
requiring a full ceph-ci build and repo.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #24402 from cbodley/wip-36302

librgw: initialize curl and http client for multisite

Merge PR #24514 into master

* refs/pull/24514/head:
mgr/restful: do not print warning message at seeing unknown request

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

qa/workunits: replace 'realpath' with 'readlink -f' in fsstress.sh

Fixes: http://tracker.ceph.com/issues/36409
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #24264 into master

* refs/pull/24264/head:
tools/rados/rados.cc: fix rados rm --force-full blocking problem

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #24287 into master

* refs/pull/24287/head:
osd/OSD: drop osd_lock only if pg osdmaps are truly laggy

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #24331 into master

* refs/pull/24331/head:
common: fix memory leaks in WeightedPriorityQueue.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #24365 into master

* refs/pull/24365/head:
erasure-code: var-sized array init is done separatly.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #24506 into master

* refs/pull/24506/head:
doc/rados/troubleshooting: update notes on rebuilding mon db
ceph-objectstore-tool: update-mon-db: remove pgmap references
ceph-monstore-tool: remove remaining pgmap references, inflate-pgmap command
ceph-monstore-tool: rebuild creating_pgs
ceph-monstore-tool: rebuild: remove obsolete pgmap_meta fields

Reviewed-by: Kefu Chai <kchai@redhat.com>

mon: consider AUTH_NONE clients which "authenticate" to be finished

We check for AUTH_NONE success by examining the side effect of the caps being
filled in with allow_all, and we need that to be "finished" for purposes of
replying with a monmap as well.

Fixes: http://tracker.ceph.com/issues/36300
Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #24409 from alimaredia/wip-rgw-dpp-auth

rgw: thread DoutPrefixProvider into RGW::Auth_S3::authorize

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc/rados/troubleshooting-mon: update mondb recovery script

- some cleanup (e.g., use $ms throughput)
- behave if the local host is in the $hosts list (use $ms.remote)
- be clear about updating all mons
- mon.0 -> mon.foo

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #24428 from trociny/wip-rbd-config-global

rbd: add 'config global' command to get/store overrides in mon config db

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #24541 into master

* refs/pull/24541/head:
common/config.h: fix spelling error in this file

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/selftest: fix disabled module selection

This was broken by always on modules -- it was also
unnecessary complication, let's just pick some
hardcoded module to use.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge PR #24305 into master

* refs/pull/24305/head:
msg/async: ProtocolV1: use continuations instead of direct function calls
msg/async: ProtocolV1 state machine ascii diagram

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

install-deps.sh: dashboard frontend needs git

Running `npm install` for the dashboard pulls dependenies with git.
Under some setups, git was not installed when building the frontend.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #24404 from ceph/wip-batch-report-idempotent

ceph-volume: make `lvm batch` idempotent

Reviewed-by: Alfredo Deza <adeza@redhat.com>

common/config.h: fix spelling error in this file

Signed-off-by: xiaomanh <huangxiaoman@cmss.chinamobile.com>

Merge pull request #24425 from b-ranto/wip-ld-hardening

rpm: Use hardened LDFLAGS

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

mgr/dashboard: Improves documentation for Grafana Setting

Fixes: http://tracker.ceph.com/issues/36371
Signed-off-by: Kanika Murarka <kmurarka@redhat.com>

Merge PR #23069 into master

* refs/pull/23069/head:
tests/libcephfs: add simple reclaim test
mds: check auth name before reclaiming session
mds: reclaim session before allowing mds to become active
mds: allow client to specify its session timeout
mds: initial code for client states reclaim

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-volume: extracts batch.filter_devices from Batch._get_strategy

This allows us to easily provide tests for that method.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: failing to get block db size from conf logs an exception

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: when all devices are filtered exit gracefully

Even if all devices are filtered we want to return a 0 exit code and
make sure the json reporting still works.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: filter devices used by journals/block.db

If after filterering of data/block devices there are only
one device left it can not be used if it is an SSD and
has been used previously as a journal or block.db

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: add rotational property to Device class

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: add info about filtered devices to batch pretty reports

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: remove the used_by_ceph key in the json output

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: fix idempotency checks for lvm batch tests

The mixed type tests will change strategy after the idempotency test so
we need to handle that in test playbook.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: fix bluestore strategy json reporting and type

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: ignore failure to load ceph configuration for block.db size

If we fail to load a ceph configureation file when trying to get the
block.db size then just use defaults instead of throwing an error.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: fix strategy comparison in 'lvm batch'

This also fixes some small json reporting issues with the
filestore MixedType strategy

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: raise a non zero exit code if strategy changes with batch

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: add functional tests to ensure lvm batch is idempotent

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: add tests for util.device.Device.used_by_ceph

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: update tests to account for filtered_devices in batch

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: consider block and data devices used_by_ceph

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: add filtered_devices and used_by_ceph to all batch reports

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: pick strategy for batch with only the unused devices

This will pick a strategy, filter out any devices already been used by
ceph and then pick a strategy again. If the strategy has changed the
call should error, if the strategy is the same proceed. If there are no
unused devices then the command is a noop.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: adds a 'changed' key to lvm batch --report

This will indicate if the command would result in any OSDs being created
or not. Other tooling can use that key for idempotency checks.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: adds used_by_ceph to filestore singletype batch report

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: adds a used_by_ceph property to the Device class

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: adds a lvs property to the Device class

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: add vg_name to the Device class

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #24525 from jtlayton/k8s-no-module-warn

mgr/orchestrator: clarify error message about kubernetes python module

Reviewed-by: John Spray <john.spray@redhat.com>

osd: make 'cache drop' command require 'executable' permission

Signed-off-by: Mohamad Gebai <mgebai@suse.com>

osd: rename 'drop cache' and 'get cache stats' to group them by component

Rename 'drop cache' to 'cache drop' and 'get cache stats' to 'cache
status'.

Signed-off-by: Mohamad Gebai <mgebai@suse.com>