osd/PrimaryLogPG: do not append outdata to TMAPUP ops

Write operations should not have data but TMAPUP incorrectly appends
the previously read TMAP object to the outdata buferlist.

Fixes: https://tracker.ceph.com/issues/41908
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #30452 from idryomov/wip-krbd-discard-direct

qa: avoid page cache for krbd discard round off tests

Reviewed-by: Dongsheng Yang <dongsheng.yang@easystack.cn>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #30386 from tchaikov/wip-run_tox.sh-cleanup

script/run_tox.sh: remove unused code

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #29734 from hselin/rados_bench_rand_read

tools/rados: use num ops instead of num objs for tracking outstanding IO

Reviewed-by: Mark Nelson <mnelson@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29767 from votdev/issue_41314

mgr/dashboard: Editing RGW bucket fails because of name is already in use

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #30313 from cyx1231st/wip-seastar-msgr-test-racing-crimson

crimson/net: v2 racing tests, stall tests and bug fixes

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/net: log out_q size when execute_ready()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: set exit_open atomically with write_dispatching

exit_open should be set atomically with state checks and
write_dispatching changes, or there would be chances to be blocked
forever by exit_open.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: add timeout feature for blockers to fail

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: improve loggings for test operations

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: chain non-discardable futures

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: discard and add note to discardable futures

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: discard future returned by close()

It's OK to discard the returned future of Connection::close() because
Messenger::shutdown() will wait for all connections closed.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: explicitly discard future from with_gate()

We are using with_gate() to wrap a slow operation when we want to start
it but don't want/need to wait it to be finished in the same function.
In this case, the future returned from with_gate() is discarded, and we
need to make sure there is another continuation responsible to waiting
for the gate closed.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: test connection stall cases for v2

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: implement STALL action for breakpoint

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: test connect-reconnect racing cases for v2

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: always dispatch accept when replaced/accepted

Fix issues of missing dispatches of accept event during connection
racing.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: test connect racing cases for v2

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: fix tests with the new ESTABLISHING state

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: add ESTABLISHING state

With the new ESTABLISHING state, connection lookup and acceptance can be
atomic, solving the issues related to racing connect.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: test reconnect racing cases for v2

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: implement BLOCK action for breakpoint

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: implement different ways to wait in tests

wait_replaced(count): wait for connection replaced;
wait_established(): wait for all pending messages sent and received;

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: log intercepted breakpoint with counter

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: improve loggings related to state change

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: always validate state when switch to another

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: use bp_type_t for breakpoint type

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: don't send msg from lossy server

It is possible to lose message at server side due to lossy policy,
causing unit test blocked and waiting for lost replies.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: add SocketConnection::peer_wins()

Also expose Connection::peer_wins() to unit tests.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

qa: avoid page cache for krbd discard round off tests

rbd rounds off and ignores discards that are smaller than alloc_size,
but the page cache is invalidated according to the original offset and
length, before calling into the driver.  For a 512~4193792 discard, all
pages except for the first will be invalidated, but the first page may
remain cached after being partially zeroed out.  This results in an
inconsistency when alloc_size != 512.  E.g. for 64k:

                0       512     4k      64k             4M
                |       |       |       |               |
  page cache:   xxxxxxxx00000000xxxxxxxx0000000000000000
       RADOS:   xxxxxxxxxxxxxxxxxxxxxxxx0000000000000000

Given that these tests asserts on-disk state, we need to read directly
from disk.  krbd_discard_512b.t and krbd_zeroout.t are fine -- there is
no rounding there.

Fixes: https://tracker.ceph.com/issues/41753
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #29839 from hualongfeng/master

doc/radosgw/admin:fix how to modify subuser info

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30392 from zhangsw/rgw-cleanup-trimusage

rgw: change MAX_USAGE_TRIM_ENTRIES value from 128 to 1000.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30393 from tianshan/fix_data_sync_noent

rgw: fix data sync start delay if remote haven't init data_log

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30381 from ajarr/fix-40689

doc: add ceph fs volumes and subvolumes documentation

Merge pull request #30053 from mxdInspur/fix_20190901

librbd: adjust the else-if conditions in validate_striping()

Merge pull request #30314 from Songweibin/wip-rbd-pybind

rbd/pybind: fix unsupported format character of %lx

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

Merge pull request #30429 from smithfarm/wip-rn-14.2.4

doc/releases: Ceph Nautilus v14.2.4 Release Notes

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

doc/releases: Ceph Nautilus v14.2.4 Release Notes

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #30193 from cbodley/wip-qa-s3test-website

qa/rgw: add new rgw/website suite for static website tests

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30389 from batrick/i41835

mds: drive cap recall while dropping cache

qa/rgw: rgw/website suite uses subdomain calling format

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: add new rgw/website suite for static website tests

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: s3test tasks sets s3website_domain to match dns-s3website-name

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/s3tests: use correct endpoint from rgw_server

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/ragweed: use correct endpoint from rgw_server

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: fix for extra_args in s3tests task

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30218 from cbodley/wip-qa-rgw-barbican

qa/rgw: add integration test for sse-kms with barbican

Reviewed-by: Ali Maredia <amaredia@redhat.com>

doc: add ceph fs volumes and subvolumes documentation

Fixes: https://tracker.ceph.com/issues/40689
Signed-off-by: Ramana Raja <rraja@redhat.com>

mgr/dashboard: Improve workaround to redraw datatables (#30335)

mgr/dashboard: Improve workaround to redraw datatables

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Patrick Seidensal <pnawracay@suse.com>

mgr/dashboard: Firefox ngx-datatable performance issue (#30316)

mgr/dashboard: Firefox ngx-datatable performance issue

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #30351 from dillaman/wip-41780

rbd-mirror: cannot restore deferred deletion mirrored images

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #29817 into master

* refs/pull/29817/head:
doc: document that the kcephfs mount helper will search keyring files for secrets
mount.ceph: fork a child to get info from local configuration
mount.ceph: track mon string and path inside ceph_mount_info
mount.ceph: add name and secret to ceph_mount_info
mount.ceph: add ceph_mount_info structure
mount.ceph: clean up debugging output and error messages
mount.ceph: clean up return codes
mount.ceph: add comment explaining why we need to modprobe
mount.ceph: use bools for flags
common: have read_secret_from_file return negative error codes

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29735 into master

* refs/pull/29735/head:
mgr/volumes: cluster log when a purge thread bails out
mgr/volumes: retry purging trash entries on intermittent error

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29705 into master

* refs/pull/29705/head:
client: fix lazyio_synchronize() to update file size

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #29569 into master

* refs/pull/29569/head:
mds: Reorganize class members in DamageTable header

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29888 into master

* refs/pull/29888/head:
mds: Reorganize class members in JournalPointer header

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #30029 into master

* refs/pull/30029/head:
mds: use set to store to evict client

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #30058 into master

* refs/pull/30058/head:
qa/tasks: test for prevent negative subvolume size
mgr/volumes: prevent negative subvolume size

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge PR #30265 into master

* refs/pull/30265/head:
doc: add a new document on CephFS distributed metadata cache

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc: add a new document on CephFS distributed metadata cache

Fixes: https://tracker.ceph.com/issues/41451
Signed-off-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #30399 from tchaikov/wip-crimson-peering-query

crimson: handle MOSDPGQuery2 properly

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #28830 from albin-antony/indv2

rgw: fix indentation for listobjectsv2

Reviewed-by: Ali Maredia <amaredia@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #29672 into master

* refs/pull/29672/head:
common/RefCountedObj: cleanup con/des
common: add make_ref factory for RefCountedObject
common: refactor RefCountedObj and reduce header dependencies
include: convert FunctionContext usage to generic LambdaContext
osd: add missing header
cls/rgw: add missing header dependency

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30377 from tspmelo/wip-e2e-hosts-mds

mgr/dashboard: Fix e2e host test

Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #29819 from Devp00l/wip-41396

mgr/dashboard: Document UiApiController with ApiController usage

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

common/RefCountedObj: cleanup con/des

Also, don't allow children to set nref (to 0). This is the more significant
change as it required fixing various code to not do this:

    <reftype> ptr = new RefCountedObjectFoo(..., 0);

as a way to create a starting reference with nref==1. This is a pretty
bad code smell so I've converted all the code doing this to use the new
factory method which produces the reference safely:

    auto ptr = ceph::make_ref<RefCountedObjectFoo>(...);

libradosstriper was particularly egregious in its abuse of setting the starting
nref. :(

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

common: add make_ref factory for RefCountedObject

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

common: refactor RefCountedObj and reduce header dependencies

- Protect constructors/destructors; RefCountedObj is an abstract class.

- Move some helpers to separate unit to avoid circular dependency with
  CephContext which could otherwise use RefCountedObj.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

include: convert FunctionContext usage to generic LambdaContext

The main motivation for this change is to avoid copies due to the use of
boost::function/std::function where captures of std::unique_ptr (in
subsequent commits) would fail to compile.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

osd: add missing header

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

cls/rgw: add missing header dependency

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #30368 from liewegas/wip-snapmapper-cleanup

osd/SnapMapper: remove pre-octopus snapmapper keys after conversion

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #30356 from majianpeng/msg-reduece-duplicated-code

msg: add func is_blackhole to reduce duplicated code.

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #30350 from liewegas/wip-bluestore-poolstats-configs

os/bluestore: simplify per-pool-stat config options

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

crimson/osd: change log level to warn if pg is absent

it's not fatal or indicating a bug, but it's unusual. so let's make it a
warning.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: reply if pg is absent when handling MQuery

in previous commit, the support of `MOSDPGQuery` was dropped. but
if we want to handle `MOSDPGQuery2` properly, we should take
`on_pg_absent()` into consideration when handling `MOSDPGQuery2`.

in this change, we handle `MOSDPGQuery2` by checking the boost event
against `MQuery`, so we don't need to create a new class derived from
`RemotePeeringEvent` for specializing the behavior.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: do not handle MOSDPGQuery anymore

this is a follow-up of 260ab1a5, as MOSDPGQuery is deprecated in favor
of MOSDPGQuery2 in octopus and up. let's drop the support of
MOSDPGQuery.

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/OSD.h: remove unused OSD::discard_context()

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: do not use __func__ in a lambda for logging

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: verbose logging in OSD::_send_alive()

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Firefox ngx-datatable performance issue

Fixes: https://tracker.ceph.com/issues/41667
Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

Merge pull request #30398 from tspmelo/wip-backport-head

script/ceph-backport.sh: Fix verification of git repository

script/ceph-backport.sh: Fix verification of git repository

In git 2.16 running "git show-ref HEAD" returns an error and
prevents the script to procced.

Signed-off-by: Tiago Melo <tmelo@suse.com>

rgw: fix indentation for listobjectsv2

https://tracker.ceph.com/issues/41840

Signed-off-by: Albin Antony <aantony@redhat.com>

rgw: fix data sync start delay if remote haven't init data_log

Fixes: https://tracker.ceph.com/issues/41839
Signed-off-by: Tianshan Qu <tianshan@xsky.com>

mds: drive cap recall while dropping cache

STEADY does not imply TRIM. We want to always trim as many caps as
possible.

Introduced-by: be49866a150e04438cef971dec7948eaf34852fc
Fixes: https://tracker.ceph.com/issues/41835
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

rgw: change MAX_USAGE_TRIM_ENTRIES value from 128 to 1000.

Signed-off-by: zhang Shaowen <zhangshaowen@cmss.chinamobile.com>

tools/rados: Fix rand & seq read bench to use num ops instead of num objs when issuing and tracking outstanding IO

ObjBencher::seq_read_bench() is using "num_objects > data.started" to make sure
we don't issue more reads than what was written during ObjBencher::write_bench().
However, this does not work op_size != object_size as data.started is number of read
ops issued, not number of objects read.
This fix modifies ObjBencher::seq_read_bench() to use "num_ops > data.started" instead.
Where "num_ops" is metadata saved at the end of ObjBencher::write_bench().

ObjBencher::rand_read_bench() is using "rand() % num_objects" for rand_id and
"rand_id / reads_per_object" to generate object name.
This will not work correctly when reads_per_object != 1 (i.e. when op_size != object_size).
For example, if reads_per_object = 100 and num_objects = 2, then all generated
reads will be directed towards the first object, with no read for the second object.
This fix modifies ObjBencher::rand_read_bench() to use "rand() % num_ops" for rand_id instead.
Where "num_ops" is metadata saved at the end of ObjBencher::write_bench().

This patch also modifies ObjBencher::write_bench() to save number of write operations issued
(num_ops above), rather than number of objects written (num_objects). We can always derive
num_objects based on "(num_ops + ops_per_object - 1) / ops_per_object" where "ops_per_object"
is simply object_size / op_size.

Signed-off-by: Albert H Chen <hselin.chen@gmail.com>

Merge pull request #29549 from smanjara/wip-multisite-metadata

rgw: Allow admin APIs that write metadata to be executed first on the mast…

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30174 from mkogan1/wip-rgw-40700

rgw: fix memory growth while deleting objects with

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

script/run_tox.sh: remove unused code

it's a follow-up of 793308f8, where the code to use python2/python3 to
determine env_list was removed already

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #30379 from smithfarm/wip-41821

tools/setup-virtualenv.sh: do not default to python2.7

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #30362 from uweigand/fix-unaligned-check

cmake: Fix unaligned check on big-endian systems

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30374 from tchaikov/wip-cmake-brotli

cmake: s/bortli_libs/brotli_libs/

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30366 from liu-chunmei/ceph_seastar_MonClient

crimson/mon: use shared_future for waiting MauthReply

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30380 from rzarzynski/wip-crimson-cyanread-returns-bl

crimson/os: Object::read() returns bufferlist instead of never used errcode

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30339 from rzarzynski/wip-crimson-cls-2

crimson: serve basic RBD traffic coming from fio

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/mon:use shared_future for waiting MauthReply

when monitor-side close the connection, msgr call MonClient
ms_handle_reset cause reply.get_future be called twice then
assert happen in promise.get_future.
promise<T...>::get_future() noexcept {
    assert(!this->_future && this->_state && !this->_task);
    return future<T...>(this);
}
use shared_promise instead of promise to solve it.

Signed-off-by: chunmei Liu <chunmei.liu@intel.com>

Merge pull request #30363 from idryomov/wip-krbd-huge-osdmap

qa: add script to test how libceph handles huge osdmaps

Reviewed-by: Jason Dillaman <dillaman@redhat.com>