qa: avoid hexdump skip and length options

On Debian-based distros, hexdump(1) comes from FreeBSD in bsdmainutils
package (other distros ship a Linux version from util-linux).  It turns
out that '-s skip' for anything but a regular file has been broken for
ages in FreeBSD -- instead of calling fseek(), it consumed the input
byte by byte with getchar().  This was fixed in commit 7cd5ae492557
"hexdump(1): Speed up -s flag on devices" last year, but it hasn't made
it to the Debian world yet.

bsdmainutils_9.0.6ubuntu3 in xenial had a custom patch for that, but
it's no longer there in bsdmainutils_11.1.2ubuntu1 in bionic.  hexdump
-s <large offset> is now a fancy infinite loop!

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #30429 from smithfarm/wip-rn-14.2.4

doc/releases: Ceph Nautilus v14.2.4 Release Notes

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

doc/releases: Ceph Nautilus v14.2.4 Release Notes

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #30193 from cbodley/wip-qa-s3test-website

qa/rgw: add new rgw/website suite for static website tests

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30389 from batrick/i41835

mds: drive cap recall while dropping cache

qa/rgw: rgw/website suite uses subdomain calling format

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: add new rgw/website suite for static website tests

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: s3test tasks sets s3website_domain to match dns-s3website-name

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/s3tests: use correct endpoint from rgw_server

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/ragweed: use correct endpoint from rgw_server

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: fix for extra_args in s3tests task

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30218 from cbodley/wip-qa-rgw-barbican

qa/rgw: add integration test for sse-kms with barbican

Reviewed-by: Ali Maredia <amaredia@redhat.com>

mgr/dashboard: Improve workaround to redraw datatables (#30335)

mgr/dashboard: Improve workaround to redraw datatables

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Patrick Seidensal <pnawracay@suse.com>

mgr/dashboard: Firefox ngx-datatable performance issue (#30316)

mgr/dashboard: Firefox ngx-datatable performance issue

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #30351 from dillaman/wip-41780

rbd-mirror: cannot restore deferred deletion mirrored images

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #29817 into master

* refs/pull/29817/head:
doc: document that the kcephfs mount helper will search keyring files for secrets
mount.ceph: fork a child to get info from local configuration
mount.ceph: track mon string and path inside ceph_mount_info
mount.ceph: add name and secret to ceph_mount_info
mount.ceph: add ceph_mount_info structure
mount.ceph: clean up debugging output and error messages
mount.ceph: clean up return codes
mount.ceph: add comment explaining why we need to modprobe
mount.ceph: use bools for flags
common: have read_secret_from_file return negative error codes

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29735 into master

* refs/pull/29735/head:
mgr/volumes: cluster log when a purge thread bails out
mgr/volumes: retry purging trash entries on intermittent error

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29705 into master

* refs/pull/29705/head:
client: fix lazyio_synchronize() to update file size

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #29569 into master

* refs/pull/29569/head:
mds: Reorganize class members in DamageTable header

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29888 into master

* refs/pull/29888/head:
mds: Reorganize class members in JournalPointer header

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #30029 into master

* refs/pull/30029/head:
mds: use set to store to evict client

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #30058 into master

* refs/pull/30058/head:
qa/tasks: test for prevent negative subvolume size
mgr/volumes: prevent negative subvolume size

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge PR #30265 into master

* refs/pull/30265/head:
doc: add a new document on CephFS distributed metadata cache

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc: add a new document on CephFS distributed metadata cache

Fixes: https://tracker.ceph.com/issues/41451
Signed-off-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #30399 from tchaikov/wip-crimson-peering-query

crimson: handle MOSDPGQuery2 properly

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #28830 from albin-antony/indv2

rgw: fix indentation for listobjectsv2

Reviewed-by: Ali Maredia <amaredia@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #29672 into master

* refs/pull/29672/head:
common/RefCountedObj: cleanup con/des
common: add make_ref factory for RefCountedObject
common: refactor RefCountedObj and reduce header dependencies
include: convert FunctionContext usage to generic LambdaContext
osd: add missing header
cls/rgw: add missing header dependency

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30377 from tspmelo/wip-e2e-hosts-mds

mgr/dashboard: Fix e2e host test

Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #29819 from Devp00l/wip-41396

mgr/dashboard: Document UiApiController with ApiController usage

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

common/RefCountedObj: cleanup con/des

Also, don't allow children to set nref (to 0). This is the more significant
change as it required fixing various code to not do this:

    <reftype> ptr = new RefCountedObjectFoo(..., 0);

as a way to create a starting reference with nref==1. This is a pretty
bad code smell so I've converted all the code doing this to use the new
factory method which produces the reference safely:

    auto ptr = ceph::make_ref<RefCountedObjectFoo>(...);

libradosstriper was particularly egregious in its abuse of setting the starting
nref. :(

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

common: add make_ref factory for RefCountedObject

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

common: refactor RefCountedObj and reduce header dependencies

- Protect constructors/destructors; RefCountedObj is an abstract class.

- Move some helpers to separate unit to avoid circular dependency with
  CephContext which could otherwise use RefCountedObj.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

include: convert FunctionContext usage to generic LambdaContext

The main motivation for this change is to avoid copies due to the use of
boost::function/std::function where captures of std::unique_ptr (in
subsequent commits) would fail to compile.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

osd: add missing header

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

cls/rgw: add missing header dependency

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #30368 from liewegas/wip-snapmapper-cleanup

osd/SnapMapper: remove pre-octopus snapmapper keys after conversion

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #30356 from majianpeng/msg-reduece-duplicated-code

msg: add func is_blackhole to reduce duplicated code.

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #30350 from liewegas/wip-bluestore-poolstats-configs

os/bluestore: simplify per-pool-stat config options

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

crimson/osd: change log level to warn if pg is absent

it's not fatal or indicating a bug, but it's unusual. so let's make it a
warning.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: reply if pg is absent when handling MQuery

in previous commit, the support of `MOSDPGQuery` was dropped. but
if we want to handle `MOSDPGQuery2` properly, we should take
`on_pg_absent()` into consideration when handling `MOSDPGQuery2`.

in this change, we handle `MOSDPGQuery2` by checking the boost event
against `MQuery`, so we don't need to create a new class derived from
`RemotePeeringEvent` for specializing the behavior.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: do not handle MOSDPGQuery anymore

this is a follow-up of 260ab1a5, as MOSDPGQuery is deprecated in favor
of MOSDPGQuery2 in octopus and up. let's drop the support of
MOSDPGQuery.

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/OSD.h: remove unused OSD::discard_context()

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: do not use __func__ in a lambda for logging

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: verbose logging in OSD::_send_alive()

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Firefox ngx-datatable performance issue

Fixes: https://tracker.ceph.com/issues/41667
Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

Merge pull request #30398 from tspmelo/wip-backport-head

script/ceph-backport.sh: Fix verification of git repository

script/ceph-backport.sh: Fix verification of git repository

In git 2.16 running "git show-ref HEAD" returns an error and
prevents the script to procced.

Signed-off-by: Tiago Melo <tmelo@suse.com>

rgw: fix indentation for listobjectsv2

https://tracker.ceph.com/issues/41840

Signed-off-by: Albin Antony <aantony@redhat.com>

mds: drive cap recall while dropping cache

STEADY does not imply TRIM. We want to always trim as many caps as
possible.

Introduced-by: be49866a150e04438cef971dec7948eaf34852fc
Fixes: https://tracker.ceph.com/issues/41835
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #29549 from smanjara/wip-multisite-metadata

rgw: Allow admin APIs that write metadata to be executed first on the mast…

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30174 from mkogan1/wip-rgw-40700

rgw: fix memory growth while deleting objects with

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #30379 from smithfarm/wip-41821

tools/setup-virtualenv.sh: do not default to python2.7

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #30362 from uweigand/fix-unaligned-check

cmake: Fix unaligned check on big-endian systems

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30374 from tchaikov/wip-cmake-brotli

cmake: s/bortli_libs/brotli_libs/

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30366 from liu-chunmei/ceph_seastar_MonClient

crimson/mon: use shared_future for waiting MauthReply

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30380 from rzarzynski/wip-crimson-cyanread-returns-bl

crimson/os: Object::read() returns bufferlist instead of never used errcode

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30339 from rzarzynski/wip-crimson-cls-2

crimson: serve basic RBD traffic coming from fio

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/mon:use shared_future for waiting MauthReply

when monitor-side close the connection, msgr call MonClient
ms_handle_reset cause reply.get_future be called twice then
assert happen in promise.get_future.
promise<T...>::get_future() noexcept {
    assert(!this->_future && this->_state && !this->_task);
    return future<T...>(this);
}
use shared_promise instead of promise to solve it.

Signed-off-by: chunmei Liu <chunmei.liu@intel.com>

Merge pull request #30363 from idryomov/wip-krbd-huge-osdmap

qa: add script to test how libceph handles huge osdmaps

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

crimson/osd: add dummy support for CEPH_OSD_OP_WATCH.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: implement cls_get_client_features().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: switch get_orig_source_inst() to get_message() in OpsExecuter.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: implement cls_cxx_map_set_val().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: implement cls_get_request_origin().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: add support for CEPH_OSD_OP_OMAPGETVALS.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #28560 into master

* refs/pull/28560/head:
cephfs-shell: handle du's arguments elsewhere outside do_du()
cephfs-shell: reuse code
cephfs-shell: rewrite call to perror in do_du
pybind/cephfs: define variable for hexcode used in stat()
test_cephfs_shell: test cephfs-shell command at invocation
cephfs-shell: refactor do_du()
cephfs-shell: option -r is not for reverse
cephfs-shell: extend to_bytes()
test_cephfs_shell: test du with no args
test_cephfs_shell: test du with multiple paths in args
test_cephfs_shell: test behaviour of "du -r"
test_cephfs_shell: test du's output for softlinks
qa/cephfs: add convenience method lstat()
qa/cephfs: add option to make stat() unfollow symlinks
test_cephfs_shell: test du's output for hardlinks
test_cephfs_shell: test du's output for directories
test_cephfs_shell: test du's output for regular files
test_cephfs_shell: add a method to get command output
test_cephfs_shell: allow cmd as list too
test_cephfs_shell: rename and rewrite _cephfs_shell()
test_cephfs_shell: copy humanize() from cephfs-shell
cephfs-shell: print disk usage for non-directory files too
pybind/cephfs: add method that stats symlinks without following
cephfs-shell: Fix 'du' command error

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29907 into master

* refs/pull/29907/head:
doc: add a doc for vstart_runner.py

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/keystone: add --debug to openstack commands

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/keystone: save keystone config to archive dir

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/tempest: update url for upper-constraints.txt in tox.ini

the tox command fails to invoke tools/tox_install.sh because it passes
a stale url for its extra requirements. patch that url in tox.ini

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/barbican: pin barbican to 5.0.1 release

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/keystone: hardcode --os-identity-api-version 2.0

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/keystone: add requirements.txt with python-openstackclient

works around a VersionConflict error by including the requirements on
jsonschema when installing python-openstackclient

also pin the version of python-openstackclient, because newer clients
don't seem to recognize --os-url

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/keystone: log to a file in the archive dir

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/keystone: pip install bindep

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/keystone: use bindep to discover keystone's dependencies

downloads the keystone repository first, because we need
keystone/bindep.txt to discover the binary dependencies

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/keystone: add missing ConfigError import

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa: barbican task doesn't look for ceph release branches

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa: remove unnecessary imports in barbican task

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: crypt suite uses clusters/fixed-1.yaml

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: abstract out barbican under 2-kms/

Signed-off-by: Casey Bodley <cbodley@redhat.com>

tasks/barbican: Added task to deploy and configure barbican

Fixed keystone, rgw and s3tests tasks to work with it.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

Merge PR #30185 into master

* refs/pull/30185/head:
mgr/volumes: drop unnecessary size

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

crimson/os: fix ENODATA handling in CyanStore::get_attrs().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: add support for CEPH_OSD_OP_OMAPGETKEYS.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: PGBackend::write() alters object size now.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: implement cls_cxx_write2().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: add support for CEPH_OSD_OP_OMAPSETVALS.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: add support for CEPH_OSD_OP_OMAPGETVALSBYKEYS.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #29037 from zhangsw/rgw-reshard-numshards

rgw: shard number must be non-negative when resharding the bucket

Reviewed-by: Casey Bodley <cbodley@redhat.com>

crimson/osd: add support for CEPH_OSD_OP_CREATE.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: ignore CEPH_OSD_OP_TMAPUP like ceph-osd does for EC pools.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

qa: add script to test how libceph handles huge osdmaps

That code will also handle moderately-sized osdmaps when the memory is
fragmented.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #29623 from majianpeng/bluestore-combine-deferred_finisher-and-finisher

os/bluestore: Merge deferred_finisher and finisher.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #30342 from ifed01/wip-ifed-fix-blues-alerts

mon/pgmap: fix bluestore alerts output.

Reviewed-by: Sage Weil <sage@redhat.com>

crimson/os: Object::read() returns bufferlist instead of never used errcode.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #30372 from tchaikov/wip-crimson-single-pg-peering

crimson/osd: use single-pg peering ops

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #24794 into master

* refs/pull/24794/head:
mds: move BatchOp to separate translation unit
mds: use auto ref during iteration
mds: answering all pending getattr/lookups targeting the same inode in one go

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #28702 into master

* refs/pull/28702/head:
qa: update json format from session listing
mds: recall caps from quiescent sessions
mds: use Session::dump method uniformly
mds: use auto to deduce iterator type
mds: simplify method definition
mds: remove useless debug message
mds: use const get_request_count
mds: use session_info_t socket inst for addr
mds: refactor session lookup
mds: add explicit trim flag
mds: alphabetize tracked config keys
common: provide method to get half-life
common: correct variable name

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #29005 into master

* refs/pull/29005/head:
client: nfs-ganesha with cephfs client, removing dir reports not empty

Reviewed-by: Jeff Layton <jlayton@redhat.com>