cls/rbd: sanitize entity instance messenger version type

The entity_addr_t::type field does not uniquely identify the connection
and might switch back and forth between LEGACY, V2, and ALL depending on
the client, the OSD, and the OSD map feature bits. Therefore, always
fix the entity address type to ANY prior to any comparisons.

Fixes: https://tracker.ceph.com/issues/41833
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #30393 from tianshan/fix_data_sync_noent

rgw: fix data sync start delay if remote haven't init data_log

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30381 from ajarr/fix-40689

doc: add ceph fs volumes and subvolumes documentation

Merge pull request #30053 from mxdInspur/fix_20190901

librbd: adjust the else-if conditions in validate_striping()

Merge pull request #30314 from Songweibin/wip-rbd-pybind

rbd/pybind: fix unsupported format character of %lx

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

Merge pull request #30429 from smithfarm/wip-rn-14.2.4

doc/releases: Ceph Nautilus v14.2.4 Release Notes

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

doc/releases: Ceph Nautilus v14.2.4 Release Notes

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #30193 from cbodley/wip-qa-s3test-website

qa/rgw: add new rgw/website suite for static website tests

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30389 from batrick/i41835

mds: drive cap recall while dropping cache

qa/rgw: rgw/website suite uses subdomain calling format

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: add new rgw/website suite for static website tests

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: s3test tasks sets s3website_domain to match dns-s3website-name

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/s3tests: use correct endpoint from rgw_server

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/ragweed: use correct endpoint from rgw_server

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: fix for extra_args in s3tests task

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30218 from cbodley/wip-qa-rgw-barbican

qa/rgw: add integration test for sse-kms with barbican

Reviewed-by: Ali Maredia <amaredia@redhat.com>

doc: add ceph fs volumes and subvolumes documentation

Fixes: https://tracker.ceph.com/issues/40689
Signed-off-by: Ramana Raja <rraja@redhat.com>

mgr/dashboard: Improve workaround to redraw datatables (#30335)

mgr/dashboard: Improve workaround to redraw datatables

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Patrick Seidensal <pnawracay@suse.com>

mgr/dashboard: Firefox ngx-datatable performance issue (#30316)

mgr/dashboard: Firefox ngx-datatable performance issue

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #30351 from dillaman/wip-41780

rbd-mirror: cannot restore deferred deletion mirrored images

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #29817 into master

* refs/pull/29817/head:
doc: document that the kcephfs mount helper will search keyring files for secrets
mount.ceph: fork a child to get info from local configuration
mount.ceph: track mon string and path inside ceph_mount_info
mount.ceph: add name and secret to ceph_mount_info
mount.ceph: add ceph_mount_info structure
mount.ceph: clean up debugging output and error messages
mount.ceph: clean up return codes
mount.ceph: add comment explaining why we need to modprobe
mount.ceph: use bools for flags
common: have read_secret_from_file return negative error codes

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29735 into master

* refs/pull/29735/head:
mgr/volumes: cluster log when a purge thread bails out
mgr/volumes: retry purging trash entries on intermittent error

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29705 into master

* refs/pull/29705/head:
client: fix lazyio_synchronize() to update file size

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #29569 into master

* refs/pull/29569/head:
mds: Reorganize class members in DamageTable header

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29888 into master

* refs/pull/29888/head:
mds: Reorganize class members in JournalPointer header

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #30029 into master

* refs/pull/30029/head:
mds: use set to store to evict client

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #30058 into master

* refs/pull/30058/head:
qa/tasks: test for prevent negative subvolume size
mgr/volumes: prevent negative subvolume size

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge PR #30265 into master

* refs/pull/30265/head:
doc: add a new document on CephFS distributed metadata cache

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc: add a new document on CephFS distributed metadata cache

Fixes: https://tracker.ceph.com/issues/41451
Signed-off-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #30399 from tchaikov/wip-crimson-peering-query

crimson: handle MOSDPGQuery2 properly

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #28830 from albin-antony/indv2

rgw: fix indentation for listobjectsv2

Reviewed-by: Ali Maredia <amaredia@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #29672 into master

* refs/pull/29672/head:
common/RefCountedObj: cleanup con/des
common: add make_ref factory for RefCountedObject
common: refactor RefCountedObj and reduce header dependencies
include: convert FunctionContext usage to generic LambdaContext
osd: add missing header
cls/rgw: add missing header dependency

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30377 from tspmelo/wip-e2e-hosts-mds

mgr/dashboard: Fix e2e host test

Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #29819 from Devp00l/wip-41396

mgr/dashboard: Document UiApiController with ApiController usage

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

common/RefCountedObj: cleanup con/des

Also, don't allow children to set nref (to 0). This is the more significant
change as it required fixing various code to not do this:

    <reftype> ptr = new RefCountedObjectFoo(..., 0);

as a way to create a starting reference with nref==1. This is a pretty
bad code smell so I've converted all the code doing this to use the new
factory method which produces the reference safely:

    auto ptr = ceph::make_ref<RefCountedObjectFoo>(...);

libradosstriper was particularly egregious in its abuse of setting the starting
nref. :(

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

common: add make_ref factory for RefCountedObject

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

common: refactor RefCountedObj and reduce header dependencies

- Protect constructors/destructors; RefCountedObj is an abstract class.

- Move some helpers to separate unit to avoid circular dependency with
  CephContext which could otherwise use RefCountedObj.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

include: convert FunctionContext usage to generic LambdaContext

The main motivation for this change is to avoid copies due to the use of
boost::function/std::function where captures of std::unique_ptr (in
subsequent commits) would fail to compile.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

osd: add missing header

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

cls/rgw: add missing header dependency

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #30368 from liewegas/wip-snapmapper-cleanup

osd/SnapMapper: remove pre-octopus snapmapper keys after conversion

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #30356 from majianpeng/msg-reduece-duplicated-code

msg: add func is_blackhole to reduce duplicated code.

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #30350 from liewegas/wip-bluestore-poolstats-configs

os/bluestore: simplify per-pool-stat config options

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

crimson/osd: change log level to warn if pg is absent

it's not fatal or indicating a bug, but it's unusual. so let's make it a
warning.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: reply if pg is absent when handling MQuery

in previous commit, the support of `MOSDPGQuery` was dropped. but
if we want to handle `MOSDPGQuery2` properly, we should take
`on_pg_absent()` into consideration when handling `MOSDPGQuery2`.

in this change, we handle `MOSDPGQuery2` by checking the boost event
against `MQuery`, so we don't need to create a new class derived from
`RemotePeeringEvent` for specializing the behavior.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: do not handle MOSDPGQuery anymore

this is a follow-up of 260ab1a5, as MOSDPGQuery is deprecated in favor
of MOSDPGQuery2 in octopus and up. let's drop the support of
MOSDPGQuery.

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/OSD.h: remove unused OSD::discard_context()

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: do not use __func__ in a lambda for logging

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: verbose logging in OSD::_send_alive()

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Firefox ngx-datatable performance issue

Fixes: https://tracker.ceph.com/issues/41667
Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

Merge pull request #30398 from tspmelo/wip-backport-head

script/ceph-backport.sh: Fix verification of git repository

script/ceph-backport.sh: Fix verification of git repository

In git 2.16 running "git show-ref HEAD" returns an error and
prevents the script to procced.

Signed-off-by: Tiago Melo <tmelo@suse.com>

rgw: fix indentation for listobjectsv2

https://tracker.ceph.com/issues/41840

Signed-off-by: Albin Antony <aantony@redhat.com>

rgw: fix data sync start delay if remote haven't init data_log

Fixes: https://tracker.ceph.com/issues/41839
Signed-off-by: Tianshan Qu <tianshan@xsky.com>

mds: drive cap recall while dropping cache

STEADY does not imply TRIM. We want to always trim as many caps as
possible.

Introduced-by: be49866a150e04438cef971dec7948eaf34852fc
Fixes: https://tracker.ceph.com/issues/41835
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #29549 from smanjara/wip-multisite-metadata

rgw: Allow admin APIs that write metadata to be executed first on the mast…

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30174 from mkogan1/wip-rgw-40700

rgw: fix memory growth while deleting objects with

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #30379 from smithfarm/wip-41821

tools/setup-virtualenv.sh: do not default to python2.7

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #30362 from uweigand/fix-unaligned-check

cmake: Fix unaligned check on big-endian systems

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30374 from tchaikov/wip-cmake-brotli

cmake: s/bortli_libs/brotli_libs/

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30366 from liu-chunmei/ceph_seastar_MonClient

crimson/mon: use shared_future for waiting MauthReply

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30380 from rzarzynski/wip-crimson-cyanread-returns-bl

crimson/os: Object::read() returns bufferlist instead of never used errcode

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30339 from rzarzynski/wip-crimson-cls-2

crimson: serve basic RBD traffic coming from fio

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/mon:use shared_future for waiting MauthReply

when monitor-side close the connection, msgr call MonClient
ms_handle_reset cause reply.get_future be called twice then
assert happen in promise.get_future.
promise<T...>::get_future() noexcept {
    assert(!this->_future && this->_state && !this->_task);
    return future<T...>(this);
}
use shared_promise instead of promise to solve it.

Signed-off-by: chunmei Liu <chunmei.liu@intel.com>

Merge pull request #30363 from idryomov/wip-krbd-huge-osdmap

qa: add script to test how libceph handles huge osdmaps

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

crimson/osd: add dummy support for CEPH_OSD_OP_WATCH.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: implement cls_get_client_features().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: switch get_orig_source_inst() to get_message() in OpsExecuter.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: implement cls_cxx_map_set_val().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: implement cls_get_request_origin().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: add support for CEPH_OSD_OP_OMAPGETVALS.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #28560 into master

* refs/pull/28560/head:
cephfs-shell: handle du's arguments elsewhere outside do_du()
cephfs-shell: reuse code
cephfs-shell: rewrite call to perror in do_du
pybind/cephfs: define variable for hexcode used in stat()
test_cephfs_shell: test cephfs-shell command at invocation
cephfs-shell: refactor do_du()
cephfs-shell: option -r is not for reverse
cephfs-shell: extend to_bytes()
test_cephfs_shell: test du with no args
test_cephfs_shell: test du with multiple paths in args
test_cephfs_shell: test behaviour of "du -r"
test_cephfs_shell: test du's output for softlinks
qa/cephfs: add convenience method lstat()
qa/cephfs: add option to make stat() unfollow symlinks
test_cephfs_shell: test du's output for hardlinks
test_cephfs_shell: test du's output for directories
test_cephfs_shell: test du's output for regular files
test_cephfs_shell: add a method to get command output
test_cephfs_shell: allow cmd as list too
test_cephfs_shell: rename and rewrite _cephfs_shell()
test_cephfs_shell: copy humanize() from cephfs-shell
cephfs-shell: print disk usage for non-directory files too
pybind/cephfs: add method that stats symlinks without following
cephfs-shell: Fix 'du' command error

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29907 into master

* refs/pull/29907/head:
doc: add a doc for vstart_runner.py

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/keystone: add --debug to openstack commands

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/keystone: save keystone config to archive dir

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/tempest: update url for upper-constraints.txt in tox.ini

the tox command fails to invoke tools/tox_install.sh because it passes
a stale url for its extra requirements. patch that url in tox.ini

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/barbican: pin barbican to 5.0.1 release

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/keystone: hardcode --os-identity-api-version 2.0

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/keystone: add requirements.txt with python-openstackclient

works around a VersionConflict error by including the requirements on
jsonschema when installing python-openstackclient

also pin the version of python-openstackclient, because newer clients
don't seem to recognize --os-url

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/keystone: log to a file in the archive dir

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/keystone: pip install bindep

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/keystone: use bindep to discover keystone's dependencies

downloads the keystone repository first, because we need
keystone/bindep.txt to discover the binary dependencies

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/keystone: add missing ConfigError import

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa: barbican task doesn't look for ceph release branches

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa: remove unnecessary imports in barbican task

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: crypt suite uses clusters/fixed-1.yaml

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: abstract out barbican under 2-kms/

Signed-off-by: Casey Bodley <cbodley@redhat.com>

tasks/barbican: Added task to deploy and configure barbican

Fixed keystone, rgw and s3tests tasks to work with it.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

Merge PR #30185 into master

* refs/pull/30185/head:
mgr/volumes: drop unnecessary size

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

crimson/os: fix ENODATA handling in CyanStore::get_attrs().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: add support for CEPH_OSD_OP_OMAPGETKEYS.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: PGBackend::write() alters object size now.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: implement cls_cxx_write2().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: add support for CEPH_OSD_OP_OMAPSETVALS.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: add support for CEPH_OSD_OP_OMAPGETVALSBYKEYS.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #29037 from zhangsw/rgw-reshard-numshards

rgw: shard number must be non-negative when resharding the bucket

Reviewed-by: Casey Bodley <cbodley@redhat.com>

crimson/osd: add support for CEPH_OSD_OP_CREATE.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: ignore CEPH_OSD_OP_TMAPUP like ceph-osd does for EC pools.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

qa: add script to test how libceph handles huge osdmaps

That code will also handle moderately-sized osdmaps when the memory is
fragmented.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #29623 from majianpeng/bluestore-combine-deferred_finisher-and-finisher

os/bluestore: Merge deferred_finisher and finisher.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>