Merge PR #24151 into master

* refs/pull/24151/head:
mgr/devicehealth: use is_valid_daemon_name helper
mgr/devicehealth: generalize to mon and osd daemons
mon: implement 'smart [devid]' tell command
mgr: parse mon metadata properly
mon: report device id used by mon
common/blkdev: add get_device_by_path
common/blkdev: migrate block_device_run_smartctl from OSD.cc

Reviewed-by: John Spray <john.spray@redhat.com>

Merge PR #24566 into master

* refs/pull/24566/head:
osd,mon: keep last_epoch_started along with last_epoch_clean premerge

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #24598 into master

* refs/pull/24598/head:
.github/stale.yml: configure probot/stale to automatically close stale issues

Reviewed-by: Erwan Velu <erwan@redhat.com>
Reviewed-by: Piotr Dałek <piotr.dalek@corp.ovh.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

Merge pull request #24619 from tchaikov/wip-crimson-mon-client-better-encapsulation

crimson/mon: move mon::Connection into .cc

Reviewed-by: Casey Bodley <cbodley@redhat.com>

crimson/mon: move mon::Connection into .cc

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23849 from tchaikov/wip-crimson-monc

crimson: add MonClient

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #24505 into master

* refs/pull/24505/head:
mds: wait shorter intervals if beacon not sent

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #24373 from mogeb/build-cls-rbd

osd: add required cls_* libraries as dependencies of osd

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

mds: wait shorter intervals if beacon not sent

MDS beacon upkeep always waits mds_beacon_interval seconds even when laggy.
Check more frequently for when we stop being laggy to reduce likelihood that
the MDS is removed.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #24095 into master

* refs/pull/24095/head:
osd: do not authenticate heartbeat connections until nautilus
msg,osd: enable unauthenticated Dispatcher for pre-nautilus OSD compat
osd: add missing space in heartbeat debug output
mgr/DaemonServer: add missing return
msg/async: drop verify_authorizer wrapper
osd: authenticate ping sessions
msg/simple: remove verify_authorizer wrapper
msg: remove unused ms_verify_authorizer
msg/async: remove get_authorizer wrapper
msg/simple: remove get_authorizer wrapper
msg/Messenger: pull authenticator validation into Messenger
msg/Messenger: uninline ms_deliver_verify_authorizer
mgr/DaemonServer: expose keyring for authenticator verification
mon: expose keyring for msgr1 authentication
mds: expose keyring for authenticater verification
osd: expose keyring for authenticater verification
msg/Dispatcher: add ms_get_auth1_authorizer_keystore
mon: fix ref cycle breakage in handle_forward
mon: use MonOpRequest get_session() instead of PaxosServiceMessage's
mon: get session from MonOpRequest in handle_command
messages/MForward: drop unused ctor
mon: use ms_handle_authentication to parse caps
mon: kill Session::global_id and use Connection member instead
mgr/DaemonServer: move session setup into ms_handle_authentication
mds: move session setup into ms_handle_authentication
osd: move session setup into ms_handle_authentication
msg: new ms_handle_authentication, add fields to Connection

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge PR #24579 into master

* refs/pull/24579/head:
qa/osd: fixup osd-rep-recov-eio.sh fails to parse pg dump

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #24597 from batrick/i36450

qa: fix run call args

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

Merge pull request #23983 from theanalyst/vstart-rgw-asok

vstart: set admin socket for RGW in conf

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24576 from cyx1231st/wip-seastar-msgr-refactor

crimson/net: seastar-msgr refactoring

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24255 from gregsfortytwo/wip-crush-doc

doc: explain 'firstn v indep' in the CRUSH docs

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge PR #24286 into master

* refs/pull/24286/head:
client: fix fuse client can't read or write data due its caps is invalid

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #24292 into master

* refs/pull/24292/head:
qa: add test for rctime on root inode
mds: set rctime on new system inode
mds: small refactor

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #24486 into master

* refs/pull/24486/head:
client: explicitly show blacklisted state via asok status command

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #24508 into master

* refs/pull/24508/head:
cephfs-shell: fixup 'str' object has no attribute 'decode'

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #24596 into master

* refs/pull/24596/head:
ptl-tool.py: move githubmap update into merge commit

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

.github/stale.yml: configure probot/stale to automatically close stale issues

Initially only warn about closing issues, but do not close them.

Signed-off-by: Sage Weil <sage@redhat.com>

qa: fix run call args

Fixes: http://tracker.ceph.com/issues/36450
Introduced-by: 95746ecce9215c8428a02f1745d03e10536a4129
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #24562 into master

* refs/pull/24562/head:
removed warning for resolved errata

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #24186 from libingyang-zte/master

doc: fix typos in doc/releases

Merge pull request #24238 from LenzGr/doc-leads-update

doc/dev: Updated component leads table

crimson/net: clean seastar-msgr class dependencies

Remove protocol-specific interfaces from Messenger/Connection classes,
and let SocketMessenger manage SocketConnection instead of Connection.

Signed-off-by: Yingxin <yingxin.cheng@intel.com>

osd: do not authenticate heartbeat connections until nautilus

Some (currently) pre-nautilus OSDs will crash if you try to authenticate
a heartbeat connection but they are not expecting it:

src/auth/Crypto.h: 109: FAILED assert(ckh)
 ceph version 12.2.8-457-gccd69ef (ccd69ef36aafebab964a2e47e249fdb95e083e46) luminous (stable)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110) [0x5618b04aaea0]
 2: (()+0x41cbec) [0x5618afe2cbec]
 3: (CephxSessionHandler::_calc_signature(Message*, unsigned long*)+0x8c5) [0x5618b0777ba5]
 4: (CephxSessionHandler::check_message_signature(Message*)+0x7d) [0x5618b077800d]
 5: (AsyncConnection::process()+0x1b44) [0x5618b0761a04]
 6: (EventCenter::process_events(int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0x359) [0x5618b0546079]
 7: (()+0xb38c3e) [0x5618b0548c3e]
 8: (()+0xb5070) [0x7ff04faf5070]
 9: (()+0x7dd5) [0x7ff050168dd5]
 10: (clone()+0x6d) [0x7ff04f259b3d]

See http://tracker.ceph.com/issues/36443

It won't be fixed in all clusters before upgrade to nautilus, though, so
we also need to work around it here.

Signed-off-by: Sage Weil <sage@redhat.com>

msg,osd: enable unauthenticated Dispatcher for pre-nautilus OSD compat

Before nautilus, osd heartbeats are sent over an unauthenticated channel.
We need support here to allow these connections when they are necessary
for upgrade compatibility.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: add missing space in heartbeat debug output

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/DaemonServer: add missing return

Fixes: http://tracker.ceph.com/issues/36110
Signed-off-by: Sage Weil <sage@redhat.com>

msg/async: drop verify_authorizer wrapper

Signed-off-by: Sage Weil <sage@redhat.com>

osd: authenticate ping sessions

Do not set up a Session object, though--nobody cares (currently!).

This avoids having to special-case the generic authorizer validation
code in msg/* to have to handle non-authenticated sessions.  Also, it
seems like a good idea to authenticate these sessions!

Signed-off-by: Sage Weil <sage@redhat.com>

msg/simple: remove verify_authorizer wrapper

Signed-off-by: Sage Weil <sage@redhat.com>

msg: remove unused ms_verify_authorizer

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async: remove get_authorizer wrapper

Signed-off-by: Sage Weil <sage@redhat.com>

msg/simple: remove get_authorizer wrapper

Signed-off-by: Sage Weil <sage@redhat.com>

msg/Messenger: pull authenticator validation into Messenger

This code is essentially identical across the OSD and MDS.  The
monitor is annoyingly different, but in a msgr1 specific way that
we can handle carrying here until msgr1 gets ripped out in a
couple years.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/osd: fixup osd-rep-recov-eio.sh fails to parse pg dump

Fixes: http://tracker.ceph.com/issues/36418
Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

Merge pull request #24577 from p-na/fix-saves-invalid-config

dashboard/mgr: Save button doesn't prevent saving an invalid form

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

removed warning for resolved errata

Signed-off-by: Scoots Hamilton <scoots@redhat.com>

mgr/devicehealth: use is_valid_daemon_name helper

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/devicehealth: generalize to mon and osd daemons

Signed-off-by: Sage Weil <sage@redhat.com>

mon: implement 'smart [devid]' tell command

Just like the OSD!

Signed-off-by: Sage Weil <sage@redhat.com>

mgr: parse mon metadata properly

Identify the "device_ids" like we do with OSD metadata by using the
helper.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: report device id used by mon

This will feed into the same device tracking that OSDs currently use.

Signed-off-by: Sage Weil <sage@redhat.com>

common/blkdev: add get_device_by_path

Signed-off-by: Sage Weil <sage@redhat.com>

common/blkdev: migrate block_device_run_smartctl from OSD.cc

Slight change in behavior here in that we return the error message in
the result.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #24473 into master

* refs/pull/24473/head:
common: drop get_contiguous() from ceph::bufferlist.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #24493 into master

* refs/pull/24493/head:
mgr/DaemonState: clean up device life_expectancy values
mgr/devicehealth: warn based on life_expectancy_max
mgr/devicehealth: warn on failing devices at 6 weeks

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #24523 from s0nea/wip-dashboard-configs-table-cleanup

mgr/dashboard: config options table cleanup

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: config options table cleanup

Remove columns 'tags', 'enum_values', 'long_desc', 'type', 'flags',
'daemon_default', 'desc', 'level', 'can_update_at_runtime', 'services',
'max', 'see_also', 'min' and 'source' from table view and add them to
the details.
The table contains 'name', 'value' and 'default' only.

Fixes: http://tracker.ceph.com/issues/34533
Signed-off-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #24560 from sebastian-philipp/orchestrator-fix-rook-cluster-in-name

mgr/rook: Fix Rook cluster name detection

Reviewed-by: John Spray <john.spray@redhat.com>

mgr/dashboard: Save button doesn't prevent saving an invalid form

Fixes: http://tracker.ceph.com/issues/36426
Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

Merge PR #24494 into master

* refs/pull/24494/head:
ceph-kvstore-tool: rename repair -> destructive-repair

Reviewed-by: Neha Ojha <nojha@redhat.com>

msg/Messenger: uninline ms_deliver_verify_authorizer

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/DaemonServer: expose keyring for authenticator verification

Signed-off-by: Sage Weil <sage@redhat.com>

mon: expose keyring for msgr1 authentication

Signed-off-by: Sage Weil <sage@redhat.com>

mds: expose keyring for authenticater verification

Signed-off-by: Sage Weil <sage@redhat.com>

osd: expose keyring for authenticater verification

Signed-off-by: Sage Weil <sage@redhat.com>

msg/Dispatcher: add ms_get_auth1_authorizer_keystore

This is there to provide the keyring used for authenticating msgr1
authorizers.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: fix ref cycle breakage in handle_forward

We now rely on the session -> connection ref for printing
remote addr, peer_global_id, and so on.  Change this code to
break the ref cycle instead by removing the con->session link,
which is only needed by the MonOpRequest ctor called at the top
of _ms_dispatch.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: use MonOpRequest get_session() instead of PaxosServiceMessage's

The PaxosServiceMessage method relies on the msg -> con -> session linkage,
and the con -> session link is not present for forwarded messages.  Also,
the message path is redundant and unnecessary.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: get session from MonOpRequest in handle_command

We should have made this switchover a long time ago.

Signed-off-by: Sage Weil <sage@redhat.com>

messages/MForward: drop unused ctor

Signed-off-by: Sage Weil <sage@redhat.com>

mon: use ms_handle_authentication to parse caps

The situation is a bit different here than the MDS and OSD because the
authentication happens from MAuth instead of ms_verify_authorizer, but
we are moving toward being more consistent.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: kill Session::global_id and use Connection member instead

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/DaemonServer: move session setup into ms_handle_authentication

Signed-off-by: Sage Weil <sage@redhat.com>

mds: move session setup into ms_handle_authentication

Signed-off-by: Sage Weil <sage@redhat.com>

osd: move session setup into ms_handle_authentication

Signed-off-by: Sage Weil <sage@redhat.com>

msg: new ms_handle_authentication, add fields to Connection

Lay some groundwork to eliminate ms_verify_authorizer.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #24146 into master

* refs/pull/24146/head:
mgr/orchestrator: Fix Python 3 issues

Reviewed-by: John Spray <john.spray@redhat.com>

ceph-kvstore-tool: rename repair -> destructive-repair

This is shown to corrupt otherwise healthy rocksdb databases.  Rename to
make it clear that it is generally not safe to run and shoud only be used
as a last resort.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #24247 into master

* refs/pull/24247/head:
PendingReleaseNotes: add note about increased mon memory footprint
doc/start/hardware-recommendations: refresh recommendations for RAM
rocksdb: increase default cache size to 512 MB
mon: mon_osd_cache_size = 500 (from 10)

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #24495 into master

* refs/pull/24495/head:
kv: apply the move semantics in KeyValueDB::get().

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #24535 into master

* refs/pull/24535/head:
mon: consider AUTH_NONE clients which "authenticate" to be finished

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #24563 from dillaman/wip-36410

test: move OpenStack devstack test to rocky release

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #24564 from neha-ojha/wip-21931

osd: do not overestimate the size of the object for reads with trimtrunc

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed By: Xie Xingguo <xie.xingguo@zte.com.cn>

Merge PR #24204 into master

* refs/pull/24204/head:
qa/suites/rgw/tempest: valgrind on centos only

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge PR #24249 into master

* refs/pull/24249/head:
doc/rados/troubleshooting-mon: update mondb recovery script

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd,mon: keep last_epoch_started along with last_epoch_clean premerge

We need to populate both the last_epoch_clean and last_epoch_started
accurately when fabricating a pg merge target.  In the case where the
premerge PG peers, recovers (e.g., backfill), then becomes ready to
merge, the last_epoch_clean may be > the last_epoch_started, and using
the later last_epoch_clean will prevent the PG from subsequently
peering (it'll go to incomplete state instead).

Fix by keeping both values, separately.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #24296 into master

* refs/pull/24296/head:
osd: Handle is_stopping() by discarding item so it doesn't loop
osd: Simplify _process() logic recently added

Reviewed-by: Jianpeng Ma <jianpeng.ma@intel.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #24131 into master

* refs/pull/24131/head:
tests, common: introduce bufferlist::apennd benchmark.
test, common: add BufferListIterator.BenchDeref benchmark.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #24270 into master

* refs/pull/24270/head:
osd: make 'cache drop' command require 'executable' permission
osd: rename 'drop cache' and 'get cache stats' to group them by component
doc: add documentation for 'drop cache' and 'get cache stats'
osd: don't print osdmap cache stats in 'get cache stats' command
osd: do not clear osdmap cache on 'drop cache' command
osd: offload dumping cache stats to the object store
osd: pass a stream to flush_cache commands for more verbosity
osd: implement flush_cache() method for Filestore
osd: add clear_cache and get_cache_object_count commands

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #24312 into master

* refs/pull/24312/head:
osd: kill the std::stringstream in ReplicatedBackend::do_repop_reply.
osd: bump-up the dout level in PGLog::write_log_and_missing.

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #24520 into master

* refs/pull/24520/head:
install-deps.sh: dashboard frontend needs git

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge PR #24521 into master

* refs/pull/24521/head:
ceph.in: reinstate interactive mode output

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

osd/PrimaryLogPG.cc: reassign size only when object size > truncate_size

Before setting size equal to op.extent.truncate_size, we need to check
if the size of the object is greater than the truncate_size. We do not
need to set size to op.extent.truncate_size, in the case where the size of
the object is less than op.extent.truncate_size.

Without this change, we were always setting size =
op.extent.truncate_size, when (seq < op.extent.truncate_seq) and
(op.extent.offset + op.extent.length > op.extent.truncate_size), were both
true. This ended up in:

1. overestimating the size of the object
2. not considering the correct size of the object, for
   the later checks, which calculate op.extent.length for the read ops
3. causing crashes when trying to read more data than what was present

Fixes: http://tracker.ceph.com/issues/21931
Fixes: http://tracker.ceph.com/issues/22330
Signed-off-by: Neha Ojha <nojha@redhat.com>

osd: print offset and length to track trimtrunc

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #24341 from joke-lee/WIP_FIX_RGWPOSTOBJ_VERSIONING

rgw: list bucket can not show the object uploaded by RGWPostObj when enable bucket versioning

qa/tasks/workunit: use suite branch/SHA1 when cloning workunits

Right now it's using the Ceph branch/SHA1 but it's using the suite
Git URL.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge branch 'master' into orchestrator-fix-rook-cluster-in-name

mgr/DaemonState: clean up device life_expectancy values

Instead of "0.000000" for empty time values, use "".

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/devicehealth: warn based on life_expectancy_max

The failure interval needs to be sufficiently precise that it establishes
an approximate upper bound on the device life expectancy.

Also, deal with the fact that the max value may be '0.000000' intead of ''.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/rook: Fix cluster name detection

`ROOK_CLUSTER_NAME` vanished from environment. Change to
'POD_NAMESPACE'.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

qa/workunits/rbd: switch devstack to rocky branch and tempest to 19.0.0 tag

Fixes: http://tracker.ceph.com/issues/36410
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #24550 from dillaman/wip-36409

qa/workunits: replace 'realpath' with 'readlink -f' in fsstress.sh

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #24517 from jcsp/wip-23970-test

mgr/selftest: fix disabled module selection

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #24426 from Miouge1/grafana-fix1

monitoring/grafana: Fix OSD Capacity Utlization Grafana graph

Merge pull request #24511 from a2batic/grafana-doc

mgr/dashboard: Improves documentation for Grafana Setting

Reviewed-by: Kai Wagner <kwagner@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #24513 from rhcs-dashboard/27047-landing-page-info-visibility

mgr/dashboard: Landing Page: info visibility

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>