Merge pull request #24453 from cbodley/wip-rgw-putobjprocessor

rgw: refactor PutObjProcessor stack

Merge pull request #23304 from smithfarm/wip-23776

 tools: ceph-authtool: report correct number of caps when creating keyring

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #19571 from Fusl/patch-1

doc: doc/radosgw/s3.rst: Adding AWS S3 `Storage Class` as `Not Supported`

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24660 from tchaikov/wip-crimson-cleanup

crimson: crimson/mon: remove timeout support from mon::Client::authenticate()

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24626 from jtlayton/wip-mgr-urlparse

mgr/rook: fix urljoin import

Merge pull request #24604 from yangdongsheng/token_fix

common/Throttle: TokenBucketThrottle: use reference to m_blockers.front()

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #24653 from mynaramana/claydoc

doc: erasure-code-clay fixes typos

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22162 from liuzhanhao/master

doc: Update cpp.rst to accommodate the new APIs in libs3

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: erasure-code-clay fixes typos

Signed-off-by: Myna <mynaramana@gmail.com>

tests/ceph-authtool: add reproducer for "with 0 caps" issue

Signed-off-by: Nathan Cutler <ncutler@suse.com>

tools: ceph-authtool: report correct number of caps when creating

Fixes: https://tracker.ceph.com/issues/23776
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge PR #24503 into master

* refs/pull/24503/head:
qa: increase timeout for cleanup

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #24455 into master

* refs/pull/24455/head:
qa: use timeout for fs asok operations

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #24615 into master

* refs/pull/24615/head:
cmake: mark gcc 7 as minimum C++ compiler (due to std++17)

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24617 from p-na/fix-python3-issue

mgr/dashboard: Fix Python3 issue

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

crimson/mon: remove timeout support from mon::Client::authenticate()

as the timeout parameter is not used by OSD.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: reuse common_mountcephfs_objs in crimson-common

so we won't built them twice.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #24102 into master

* refs/pull/24102/head:
global: fix err_to_stderr can't work when it was configured to false
osd: replace cout/derr with dout(0) to avoid writing trivial logs to syslog file

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #24236 into master

* refs/pull/24236/head:
osd: Log error and abort if fails to pick address

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #24359 into master

* refs/pull/24359/head:
qa/tests: update ansible version to 2.6 for master branch testing.
qa/tests: use lvm as default for ceph-ansible testing, this should also work with raw devices

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge PR #24603 into master

* refs/pull/24603/head:
crush: get "ceph osd crush class create/rm" back

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #24613 into master

* refs/pull/24613/head:
OSD: fix a typo err when increase recvoery bytes statistics

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #24586 from mynaramana/claydoc

erasure-code: fixes alignment issue when clay code is used with jerasure, cauchy_orig

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/rook: fix urljoin import

This is the correct syntax to ensure that we get urljoin, no matter
the python version.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #24608 from cyx1231st/wip-seastar-msgr-accepting

crimson/net: implement accepting/connecting states

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/net: implement accepting/connecting states

- introduce the new accepting/connecting connection states.
- return ConnectionRef immediately when connect().
- manage the ownership of the accepting connections.
- manage the ownership of the registered connections.
- encapsulate a Socket class because it is not created when constructing
  a SocketConnection, and allow it to be replaced in the future.
- refactor related interfaces.

Signed-off-by: Yingxin <yingxin.cheng@intel.com>

Merge PR #24624 into master

* refs/pull/24624/head:
client: Mutex::Locker -> std::lock_guard
mds: Mutex::Locker -> std::lock_guard

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

erasure-code: fixing bugs in clay-code plugin

* Fixes alignment issue while using clay code with jerasure, cauchy_orig
* Fixes ceph_erasure_code_non_regression to adapt to new decode

Signed-off-by: Myna <mynaramana@gmail.com>

mgr/dashboard: Fix Python3 issue

Which results in a 500 error when trying to access the `Performance
Counter` tab on the OSD list.

Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

Merge pull request #24629 from cyx1231st/wip-seastar-fix

crimson/net: fix compile errors in test_alien_echo.cc

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/net: fix compile errors

test_alien_echo.cc is using common_init_finish, but it is not available
if defined WITH_SEASTAR.

Dispatcher::ms_verify_authorizer() is removed by PR#24095.

Signed-off-by: Yingxin <yingxin.cheng@intel.com>

Merge pull request #24607 from p-na/fix-osd-list-status-labels

mgr/dashboard: Fix spaces around status labels on OSD list

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #24614 from s0nea/wip-dashboard-configs-textarea-vertical-resize

mgr/dashboard: configs textarea disallow horizontal resize

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #24631 from Devp00l/wip-issue-36466

mgr/dashboard: Add left padding to helper icon

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Add left padding to helper icon

Fixes: https://tracker.ceph.com/issues/36466
Signed-off-by: Stephan Müller <smueller@suse.com>

doc: erasure-code cleanup of clay plugin documentation

Signed-off-by: Myna <mynaramana@gmail.com>

TokenBucketThrottle: use reference to m_blockers.front() and then update it

As we need to update the blocker below, then we need to use reference to front object,
rather than a new object of blocker.

Fixes: http://tracker.ceph.com/issues/36475
Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

Merge PR #24578 into master

* refs/pull/24578/head:
pybind/ceph_argparse.py: do not create file for validating CephFilepath

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #24565 into master

* refs/pull/24565/head:
mgr: python 3 compat fixes

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #24558 into master

* refs/pull/24558/head:
pybind/mgr: Fix Python 3 imports in diskprediction & insights

Reviewed-by: Noah Watkins <nwatkins@redhat.com>

Merge PR #24528 into master

* refs/pull/24528/head:
common/condition_variable_debug: fix wait hooks
common/mutex_debug: remove no-op before/after hooks
common/mutex_debug: do lockdep post-lock step in caller
os/bluestore: {Mutex,Cond} -> ceph::{mutex,condition_variable}
os/bluestore: std::recursive_mutex -> ceph::recursive_mutex
os/bluestore: re-add is_locked assert
os/bluestore: std::{mutex,condition_variable} -> ceph::{mutex,condition_variable}
os/bluestore: use deduction for lock_guard<>, unique_lock<>

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>

Merge PR #24151 into master

* refs/pull/24151/head:
mgr/devicehealth: use is_valid_daemon_name helper
mgr/devicehealth: generalize to mon and osd daemons
mon: implement 'smart [devid]' tell command
mgr: parse mon metadata properly
mon: report device id used by mon
common/blkdev: add get_device_by_path
common/blkdev: migrate block_device_run_smartctl from OSD.cc

Reviewed-by: John Spray <john.spray@redhat.com>

Merge PR #24566 into master

* refs/pull/24566/head:
osd,mon: keep last_epoch_started along with last_epoch_clean premerge

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

client: Mutex::Locker -> std::lock_guard

Signed-off-by: Sage Weil <sage@redhat.com>

mds: Mutex::Locker -> std::lock_guard

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #24598 into master

* refs/pull/24598/head:
.github/stale.yml: configure probot/stale to automatically close stale issues

Reviewed-by: Erwan Velu <erwan@redhat.com>
Reviewed-by: Piotr Dałek <piotr.dalek@corp.ovh.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

Merge pull request #24619 from tchaikov/wip-crimson-mon-client-better-encapsulation

crimson/mon: move mon::Connection into .cc

Reviewed-by: Casey Bodley <cbodley@redhat.com>

crimson/mon: move mon::Connection into .cc

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23849 from tchaikov/wip-crimson-monc

crimson: add MonClient

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: remove old PutObjProcessor stack

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: copy_obj uses AtomicObjectProcessor

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove RGWPutObj::select_processor

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWBulkUploadOp uses AtomicObjectProcessor

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: PostObj uses AtomicObjectProcessor

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: fetch_remote_obj() uses AtomicObjectProcessor

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: prepare RGWRadosPutObj for fetch_remote_obj

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWObjectCtxImpl::invalidate takes const ref

also removes the default implementation, which is duplicated in the
specialization for rgw_obj

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: rgw_file uses AtomicObjectProcessor

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: PutObj::execute() uses new ObjectProcessors

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: compression filter uses rgw::putobj::Pipe

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: encryption filter uses rgw::putobj::Pipe

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: temporarily disable putobj filters

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: read multipart params in RGWPutObj_ObjStore_S3::get_params

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: PutObj avoids some static_casts to RGWObjectCtx

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove gotos from PutObj

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: PutObj uses unique_ptr to manage PutObjProcessor

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: PutObj uses scope_guard to update perf counter on return

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add rgw::putobj::MultipartObjectProcessor

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add rgw::putobj::AtomicObjectProcessor

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add const to some RGWBucketInfo helpers

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add rgw::putobj::ManifestObjectProcessor

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add const to some manifest generator functions

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add rgw::putobj::RadosWriter adapter

implements the DataProcessor interface by writing its buffers with Aio,
and tracks the set of successful writes so they can be deleted on
failure/cancelation

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge PR #24505 into master

* refs/pull/24505/head:
mds: wait shorter intervals if beacon not sent

Reviewed-by: Zheng Yan <zyan@redhat.com>

cmake: mark gcc 7 as minimum C++ compiler (due to std++17)

Ceph requires C++17 support from compiler. This means gcc 7.x being
minimal version supported.

This also allows to fail quick on Debian 'stretch' with it's gcc 6.3
compiler.

Signed-off-by: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>

Merge pull request #24373 from mogeb/build-cls-rbd

osd: add required cls_* libraries as dependencies of osd

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

mds: wait shorter intervals if beacon not sent

MDS beacon upkeep always waits mds_beacon_interval seconds even when laggy.
Check more frequently for when we stop being laggy to reduce likelihood that
the MDS is removed.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

global: fix err_to_stderr can't work when it was configured to false

function `set_stderr_level(-1, -1)` set m_stderr_log and m_stderr_crash to -1,
regardless of whether `err_to_stderr` is set to false or not, so logs will be
 always written to stderr. fix it as the same way as handle_conf_change does.

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

osd: replace cout/derr with dout(0) to avoid writing trivial logs to syslog file

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

Merge PR #24095 into master

* refs/pull/24095/head:
osd: do not authenticate heartbeat connections until nautilus
msg,osd: enable unauthenticated Dispatcher for pre-nautilus OSD compat
osd: add missing space in heartbeat debug output
mgr/DaemonServer: add missing return
msg/async: drop verify_authorizer wrapper
osd: authenticate ping sessions
msg/simple: remove verify_authorizer wrapper
msg: remove unused ms_verify_authorizer
msg/async: remove get_authorizer wrapper
msg/simple: remove get_authorizer wrapper
msg/Messenger: pull authenticator validation into Messenger
msg/Messenger: uninline ms_deliver_verify_authorizer
mgr/DaemonServer: expose keyring for authenticator verification
mon: expose keyring for msgr1 authentication
mds: expose keyring for authenticater verification
osd: expose keyring for authenticater verification
msg/Dispatcher: add ms_get_auth1_authorizer_keystore
mon: fix ref cycle breakage in handle_forward
mon: use MonOpRequest get_session() instead of PaxosServiceMessage's
mon: get session from MonOpRequest in handle_command
messages/MForward: drop unused ctor
mon: use ms_handle_authentication to parse caps
mon: kill Session::global_id and use Connection member instead
mgr/DaemonServer: move session setup into ms_handle_authentication
mds: move session setup into ms_handle_authentication
osd: move session setup into ms_handle_authentication
msg: new ms_handle_authentication, add fields to Connection

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge PR #24579 into master

* refs/pull/24579/head:
qa/osd: fixup osd-rep-recov-eio.sh fails to parse pg dump

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #24597 from batrick/i36450

qa: fix run call args

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

Merge pull request #23983 from theanalyst/vstart-rgw-asok

vstart: set admin socket for RGW in conf

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: configs textarea disallow horizontal resize

The textarea allows horizontal and vertical resize by default. Only the
vertical resize is appropriate for this form.

Fixes: http://tracker.ceph.com/issues/36452
Signed-off-by: Tatjana Dehler <tdehler@suse.com>

mgr/dashboard: Fix spaces around status labels on OSD list

Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

OSD: fix a typo err when increase recvoery bytes statistics

Signed-off-by: kungf <yang.wang@easystack.cn>

Merge pull request #24576 from cyx1231st/wip-seastar-msgr-refactor

crimson/net: seastar-msgr refactoring

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24255 from gregsfortytwo/wip-crush-doc

doc: explain 'firstn v indep' in the CRUSH docs

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge PR #24286 into master

* refs/pull/24286/head:
client: fix fuse client can't read or write data due its caps is invalid

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #24292 into master

* refs/pull/24292/head:
qa: add test for rctime on root inode
mds: set rctime on new system inode
mds: small refactor

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #24486 into master

* refs/pull/24486/head:
client: explicitly show blacklisted state via asok status command

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #24508 into master

* refs/pull/24508/head:
cephfs-shell: fixup 'str' object has no attribute 'decode'

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

crush: get "ceph osd crush class create/rm" back

This reverts a27fd9d25cb2819e25cc48b790c40afac0250464 and
b863883ca783487401fde4f4480ed1d9b093363e.

Quote form Sébastien Han:
> IIRC at some point, we were able to create a device class from the CLI.
Now it seems that the device class gets created when at least one OSD
of a particular class starts.
In ceph-ansible, we create pools after the initial monitors are up and
we want to assign a device crush class on some of them.
That's not possible at the moment since there no device class available yet.
Also, someone might want to create its own device class.
Something as crazy as running Filestore with a tmpfs osd store and
might want to isolate them.
I know it's a very limited use case, but still, it could be desired.

See also https://www.spinics.net/lists/ceph-devel/msg41152.html

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #24596 into master

* refs/pull/24596/head:
ptl-tool.py: move githubmap update into merge commit

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

.github/stale.yml: configure probot/stale to automatically close stale issues

Initially only warn about closing issues, but do not close them.

Signed-off-by: Sage Weil <sage@redhat.com>

qa: fix run call args

Fixes: http://tracker.ceph.com/issues/36450
Introduced-by: 95746ecce9215c8428a02f1745d03e10536a4129
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #24562 into master

* refs/pull/24562/head:
removed warning for resolved errata

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #24186 from libingyang-zte/master

doc: fix typos in doc/releases

Merge pull request #24238 from LenzGr/doc-leads-update

doc/dev: Updated component leads table

rgw: add rgw::putobj::HeadObjectProcessor

a striped object processor with special handling for the first chunk of
data

Signed-off-by: Casey Bodley <cbodley@redhat.com>