osd/PeeringState: recover_got - add special handler for empty log

If we use a small osd_min[max]_pg_log_entries setting, say, 10, and
do splitting, a child pg may only be able to keep some divergent log
entries in the list. Any follow-up peering cycle on that child pg
would then rewind all the divergent log entries and lead to an empty
log list, which as a result could reliably fire the

  ceph_assert(info.last_complete == info.last_update)

assert if we start to recover any of the missing divergent objects
later.

Fixes: https://tracker.ceph.com/issues/41816
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #30440 from dzafman/wip-41900

Revert "common: default pg_autoscale_mode=on for new pools"

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #30458 from tchaikov/wip-crimson-cbt-test

test/crimson: lower the bar for cbt test

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #30433 from tchaikov/wip-crimson-pgls

crimson/osd: add pgls support

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: add pgls_filter support

it's modeled after pgnls_filter. and use seastar::map_reduce() for
filtering the objects into the response.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: return hobject_t from pgls_filter()

so we can use it as the mapper function in seastar::map_reduce().

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: add pgls support

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: handle pg ops with do_pg_ops()

currently, OpsExecuter tries to load the object info for all ops to be
executed, but the PG ops do not require the existence of object
specified by op. so we need to specialize for them.

in this change,

* `OpsExecuter::execute_pg_op()` is added to handle pg ops without
  loading object info first.
* `pgnls` and `pgnls_filter` are handled by
  `OpsExecuter::execute_pg_op()`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd/OpsExecuter: pass nspace in msg to do_pg_op()

as pg ops should not require the object info, as there is chance that
the object specified in pg ops does not exist at all.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd/OpsExecuter: s/do_osd_op/execute_osd_op/

will add execute_pg_op() for performing pg specific ops, which do not
need the object state.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/crimson: lower the bar for cbt test

sometimes we have

13:27:59 - WARNING  - cbt      -prefill/incerta08.front.sepia.ceph.com/0: iops_stddev: (or (less) (near 0.05)):: 13.0/9.53939  => rejected

i think it's acceptable

Signed-off-by: Kefu Chai <kchai@redhat.com>

github: add "jenkins test crimson perf" to triggers doc

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30452 from idryomov/wip-krbd-discard-direct

qa: avoid page cache for krbd discard round off tests

Reviewed-by: Dongsheng Yang <dongsheng.yang@easystack.cn>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #30386 from tchaikov/wip-run_tox.sh-cleanup

script/run_tox.sh: remove unused code

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #29734 from hselin/rados_bench_rand_read

tools/rados: use num ops instead of num objs for tracking outstanding IO

Reviewed-by: Mark Nelson <mnelson@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29767 from votdev/issue_41314

mgr/dashboard: Editing RGW bucket fails because of name is already in use

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #30313 from cyx1231st/wip-seastar-msgr-test-racing-crimson

crimson/net: v2 racing tests, stall tests and bug fixes

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/net: log out_q size when execute_ready()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: set exit_open atomically with write_dispatching

exit_open should be set atomically with state checks and
write_dispatching changes, or there would be chances to be blocked
forever by exit_open.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: add timeout feature for blockers to fail

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: improve loggings for test operations

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: chain non-discardable futures

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: discard and add note to discardable futures

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: discard future returned by close()

It's OK to discard the returned future of Connection::close() because
Messenger::shutdown() will wait for all connections closed.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: explicitly discard future from with_gate()

We are using with_gate() to wrap a slow operation when we want to start
it but don't want/need to wait it to be finished in the same function.
In this case, the future returned from with_gate() is discarded, and we
need to make sure there is another continuation responsible to waiting
for the gate closed.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: test connection stall cases for v2

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: implement STALL action for breakpoint

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: test connect-reconnect racing cases for v2

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: always dispatch accept when replaced/accepted

Fix issues of missing dispatches of accept event during connection
racing.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: test connect racing cases for v2

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: fix tests with the new ESTABLISHING state

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: add ESTABLISHING state

With the new ESTABLISHING state, connection lookup and acceptance can be
atomic, solving the issues related to racing connect.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: test reconnect racing cases for v2

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: implement BLOCK action for breakpoint

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: implement different ways to wait in tests

wait_replaced(count): wait for connection replaced;
wait_established(): wait for all pending messages sent and received;

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: log intercepted breakpoint with counter

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: improve loggings related to state change

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: always validate state when switch to another

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: use bp_type_t for breakpoint type

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: don't send msg from lossy server

It is possible to lose message at server side due to lossy policy,
causing unit test blocked and waiting for lost replies.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: add SocketConnection::peer_wins()

Also expose Connection::peer_wins() to unit tests.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Revert "common: default pg_autoscale_mode=on for new pools"

This reverts commit 91e4fc24e785a25884125df6762ab2304bf8908b.

Fixes: https://tracker.ceph.com/issues/41900
Signed-off-by: David Zafman <dzafman@redhat.com>

qa: avoid page cache for krbd discard round off tests

rbd rounds off and ignores discards that are smaller than alloc_size,
but the page cache is invalidated according to the original offset and
length, before calling into the driver.  For a 512~4193792 discard, all
pages except for the first will be invalidated, but the first page may
remain cached after being partially zeroed out.  This results in an
inconsistency when alloc_size != 512.  E.g. for 64k:

                0       512     4k      64k             4M
                |       |       |       |               |
  page cache:   xxxxxxxx00000000xxxxxxxx0000000000000000
       RADOS:   xxxxxxxxxxxxxxxxxxxxxxxx0000000000000000

Given that these tests asserts on-disk state, we need to read directly
from disk.  krbd_discard_512b.t and krbd_zeroout.t are fine -- there is
no rounding there.

Fixes: https://tracker.ceph.com/issues/41753
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #29839 from hualongfeng/master

doc/radosgw/admin:fix how to modify subuser info

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30392 from zhangsw/rgw-cleanup-trimusage

rgw: change MAX_USAGE_TRIM_ENTRIES value from 128 to 1000.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30393 from tianshan/fix_data_sync_noent

rgw: fix data sync start delay if remote haven't init data_log

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30381 from ajarr/fix-40689

doc: add ceph fs volumes and subvolumes documentation

Merge pull request #30053 from mxdInspur/fix_20190901

librbd: adjust the else-if conditions in validate_striping()

Merge pull request #30314 from Songweibin/wip-rbd-pybind

rbd/pybind: fix unsupported format character of %lx

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

Merge pull request #30429 from smithfarm/wip-rn-14.2.4

doc/releases: Ceph Nautilus v14.2.4 Release Notes

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

doc/releases: Ceph Nautilus v14.2.4 Release Notes

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #30193 from cbodley/wip-qa-s3test-website

qa/rgw: add new rgw/website suite for static website tests

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30389 from batrick/i41835

mds: drive cap recall while dropping cache

qa/rgw: rgw/website suite uses subdomain calling format

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: add new rgw/website suite for static website tests

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: s3test tasks sets s3website_domain to match dns-s3website-name

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/s3tests: use correct endpoint from rgw_server

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/ragweed: use correct endpoint from rgw_server

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: fix for extra_args in s3tests task

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30218 from cbodley/wip-qa-rgw-barbican

qa/rgw: add integration test for sse-kms with barbican

Reviewed-by: Ali Maredia <amaredia@redhat.com>

doc: add ceph fs volumes and subvolumes documentation

Fixes: https://tracker.ceph.com/issues/40689
Signed-off-by: Ramana Raja <rraja@redhat.com>

mgr/dashboard: Improve workaround to redraw datatables (#30335)

mgr/dashboard: Improve workaround to redraw datatables

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Patrick Seidensal <pnawracay@suse.com>

mgr/dashboard: Firefox ngx-datatable performance issue (#30316)

mgr/dashboard: Firefox ngx-datatable performance issue

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #30351 from dillaman/wip-41780

rbd-mirror: cannot restore deferred deletion mirrored images

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #29817 into master

* refs/pull/29817/head:
doc: document that the kcephfs mount helper will search keyring files for secrets
mount.ceph: fork a child to get info from local configuration
mount.ceph: track mon string and path inside ceph_mount_info
mount.ceph: add name and secret to ceph_mount_info
mount.ceph: add ceph_mount_info structure
mount.ceph: clean up debugging output and error messages
mount.ceph: clean up return codes
mount.ceph: add comment explaining why we need to modprobe
mount.ceph: use bools for flags
common: have read_secret_from_file return negative error codes

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29735 into master

* refs/pull/29735/head:
mgr/volumes: cluster log when a purge thread bails out
mgr/volumes: retry purging trash entries on intermittent error

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29705 into master

* refs/pull/29705/head:
client: fix lazyio_synchronize() to update file size

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #29569 into master

* refs/pull/29569/head:
mds: Reorganize class members in DamageTable header

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29888 into master

* refs/pull/29888/head:
mds: Reorganize class members in JournalPointer header

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #30029 into master

* refs/pull/30029/head:
mds: use set to store to evict client

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #30058 into master

* refs/pull/30058/head:
qa/tasks: test for prevent negative subvolume size
mgr/volumes: prevent negative subvolume size

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge PR #30265 into master

* refs/pull/30265/head:
doc: add a new document on CephFS distributed metadata cache

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc: add a new document on CephFS distributed metadata cache

Fixes: https://tracker.ceph.com/issues/41451
Signed-off-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #30399 from tchaikov/wip-crimson-peering-query

crimson: handle MOSDPGQuery2 properly

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #28830 from albin-antony/indv2

rgw: fix indentation for listobjectsv2

Reviewed-by: Ali Maredia <amaredia@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #29672 into master

* refs/pull/29672/head:
common/RefCountedObj: cleanup con/des
common: add make_ref factory for RefCountedObject
common: refactor RefCountedObj and reduce header dependencies
include: convert FunctionContext usage to generic LambdaContext
osd: add missing header
cls/rgw: add missing header dependency

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30377 from tspmelo/wip-e2e-hosts-mds

mgr/dashboard: Fix e2e host test

Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #29819 from Devp00l/wip-41396

mgr/dashboard: Document UiApiController with ApiController usage

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

common/RefCountedObj: cleanup con/des

Also, don't allow children to set nref (to 0). This is the more significant
change as it required fixing various code to not do this:

    <reftype> ptr = new RefCountedObjectFoo(..., 0);

as a way to create a starting reference with nref==1. This is a pretty
bad code smell so I've converted all the code doing this to use the new
factory method which produces the reference safely:

    auto ptr = ceph::make_ref<RefCountedObjectFoo>(...);

libradosstriper was particularly egregious in its abuse of setting the starting
nref. :(

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

common: add make_ref factory for RefCountedObject

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

common: refactor RefCountedObj and reduce header dependencies

- Protect constructors/destructors; RefCountedObj is an abstract class.

- Move some helpers to separate unit to avoid circular dependency with
  CephContext which could otherwise use RefCountedObj.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

include: convert FunctionContext usage to generic LambdaContext

The main motivation for this change is to avoid copies due to the use of
boost::function/std::function where captures of std::unique_ptr (in
subsequent commits) would fail to compile.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

osd: add missing header

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

cls/rgw: add missing header dependency

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #30368 from liewegas/wip-snapmapper-cleanup

osd/SnapMapper: remove pre-octopus snapmapper keys after conversion

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #30356 from majianpeng/msg-reduece-duplicated-code

msg: add func is_blackhole to reduce duplicated code.

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #30350 from liewegas/wip-bluestore-poolstats-configs

os/bluestore: simplify per-pool-stat config options

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

crimson/osd: change log level to warn if pg is absent

it's not fatal or indicating a bug, but it's unusual. so let's make it a
warning.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: reply if pg is absent when handling MQuery

in previous commit, the support of `MOSDPGQuery` was dropped. but
if we want to handle `MOSDPGQuery2` properly, we should take
`on_pg_absent()` into consideration when handling `MOSDPGQuery2`.

in this change, we handle `MOSDPGQuery2` by checking the boost event
against `MQuery`, so we don't need to create a new class derived from
`RemotePeeringEvent` for specializing the behavior.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: do not handle MOSDPGQuery anymore

this is a follow-up of 260ab1a5, as MOSDPGQuery is deprecated in favor
of MOSDPGQuery2 in octopus and up. let's drop the support of
MOSDPGQuery.

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/OSD.h: remove unused OSD::discard_context()

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: do not use __func__ in a lambda for logging

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: verbose logging in OSD::_send_alive()

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Firefox ngx-datatable performance issue

Fixes: https://tracker.ceph.com/issues/41667
Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

Merge pull request #30398 from tspmelo/wip-backport-head

script/ceph-backport.sh: Fix verification of git repository

script/ceph-backport.sh: Fix verification of git repository

In git 2.16 running "git show-ref HEAD" returns an error and
prevents the script to procced.

Signed-off-by: Tiago Melo <tmelo@suse.com>

rgw: fix indentation for listobjectsv2

https://tracker.ceph.com/issues/41840

Signed-off-by: Albin Antony <aantony@redhat.com>

rgw: fix data sync start delay if remote haven't init data_log

Fixes: https://tracker.ceph.com/issues/41839
Signed-off-by: Tianshan Qu <tianshan@xsky.com>

mds: drive cap recall while dropping cache

STEADY does not imply TRIM. We want to always trim as many caps as
possible.

Introduced-by: be49866a150e04438cef971dec7948eaf34852fc
Fixes: https://tracker.ceph.com/issues/41835
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

rgw: change MAX_USAGE_TRIM_ENTRIES value from 128 to 1000.

Signed-off-by: zhang Shaowen <zhangshaowen@cmss.chinamobile.com>