Merge pull request #40656 from tchaikov/wip-qa-upgrade-focal

qa/suites: test upgrade/octopus-x on focal instead bionic

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #40811 from tchaikov/wip-gen-corpus

script/gen-corpus.sh: set CEPH_CONF

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #40795 from wjwithagen/wjw-fix-ceph-dencoder

tools: do not unload plugins during destruction.

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/workunits/cls: add executable bit to script

all the scripts except for test_cls_cas.sh under qa/workunits/cls
are executable. to be more consistent, add the executable bit to
test_cls_cas.sh as well.

also, these scripts are launched by src/script/gen-corpus.sh directly,
so it's convenient just call them.

Signed-off-by: Kefu Chai <kchai@redhat.com>

script/gen-corpus.sh: set CEPH_CONF

if we happen to run this script on a host where /etc/ceph/ceph.conf is
available, ceph CLI would use it instead. so, point it to $PWD/ceph.conf
instead.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40797 from wjwithagen/wjw-fix-monmap-retval

test: Undo the FreeBSD specific retval test

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40794 from wjwithagen/wjw-fix-promtool

test: Run Dockers only on Linux platforms

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38120 from kiizawa/wip-cls-remote-read

osd: allow remote read by calling cls method from within cls context

Reviewed-by: Samuel Just <sjust@redhat.com>

test: Undo the FreeBSD specific retval test

Changes to the socket code now result in returning EINVAL
In the past ENOENT was returned which is the FreeBSD error code
if DNS lookup does not work.

And that change is probably because somewhere in the code that
errorcode is not passed verbatim from the systemcall, but is
rewritten in extra evaluation.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

tools: do not unload plugins during destruction.

FreeBSD ceph-dencoder crashes in the exit() calls, due to
invalid pointer references during the release process of
the loaded libraries.

Often this is signaled by libc reporting:
  __cxa_thread_call_dtors: dtr 0x47efc0 from unloaded dso, skipping

The cause for this is different behaviour between FreeBSD and Linux:
https://groups.google.com/g/bsdmailinglist/c/22ncTZAbDp4/m/Dii_pII5AwAJ
_The FreeBSD implementation here looks racy. If one thread dlcloses an
object while another thread is exiting, we can end up calling a
function at an invalid memory address. It also looks as if it may
be possible to unload one library, load another at the same address,
and end up executing entirely the wrong code, which would have some
serious security implications.

The GNU/Linux equivalent of this function locks the DSO in memory
until all references to it have gone away. A call to dlclose() on
GNU/Linux will not actually unload the library until all threads
with destructors in that library have been unloaded. I believe
that this reuses the same reference counting mechanism that
allows the same library to be dlopened and dlclosed multiple times.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #37016 from zhangdaolong/subcommon-bulefs-import

os/bluestore:Add subcommand bluefs-import in ceph-bluestore-tool.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40644 from SMIL-Infra/cleanup-slash

cephadm: cleanup extra slash in runtime dir

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #40658 from tchaikov/wip-systemd

cmake: s/HAVE_MSGHDR/WITH_SYSTEMD/

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #40635 from pcuzner/prometheus_add_pool_metadata

mgr/prometheus:Improve the pool metadata

Merge pull request #39944 from yuvalif/wip-yuval-fix-49650

rgw/notifications: delete bucket notification object when empty

test: Run Dockers only on Linux platforms

Running a docker alternative only works if the platform
   is Linux

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #40786 from tchaikov/wip-script-bit

build-integration-branch: retry when running into network failures

Reviewed-by: Sage Weil <sage@redhat.com>

build-integration-branch: retry when running into network failures

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore/bluestore_tool: Add subcommand blufs-import

Examples
 ceph-bluestore-tool bluefs-import --path /var/lib/ceph/osd/ceph-1 --input-file ./db/CURRENT --dest-file db/CURRENT

Signed-off-by: zhang daolong <zhangdaolong@fiberhome.com>

build-integration-branch: define variables for pr_number and friends

so they can be reused later.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #40577 into master

* refs/pull/40577/head:
cephadm: normalize unqualified repo digests to docker.io
mgr/cephadm/upgrade: normalize unqualified target image

Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>

Merge PR #40537 into master

* refs/pull/40537/head:
cephadm:persist the grafana.db file

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #40653 into master

* refs/pull/40653/head:
mon: check mdsmap is resizeable before promoting standby-replay

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #40642 into master

* refs/pull/40642/head:
client: don't allow access to MDS-private inodes

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #40481 into master

* refs/pull/40481/head:
qa: test standby-replay with fs:workloads

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #40431 into master

* refs/pull/40431/head:
qa/cephfs: remove create_keyring_file from cephfs_test_case.py
qa/cephfs: don't use sudo to write files in /tmp

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #40389 into master

* refs/pull/40389/head:
mds: reject lookup ino requests for mds dirs
test: add test for invalid lookup of mdsdir

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #40723 from zdover23/wip-doc-second-attempt-mclock-rewrite-second-half-2021-Apr-09

doc/rados: rewrite mclock docs (2 of 2)

Reviewed-by: Sridhar Seshasayee <sseshasa@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #40738 from jdurgin/wip-librados-docs

include/librados: fix doxygen syntax for docs build

Reviewed-by: Neha Ojha <nojha@redhat.com>

include/librados: fix doxygen syntax for docs build

The docs build is now warning about these like:

WARNING: Unparseable C cross-reference: '[in]'
Invalid C declaration: Expected identifier in nested name.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

cephadm: normalize unqualified repo digests to docker.io

A RepoDigests returned by docker|podman image inspect can either include
the docker.io/ prefix or not.  For reasons that aren't entirely clear,
this may vary between hosts in a cluster.  However, ceph/ceph@sha256:abc...
is the same thing as docker.io/ceph/ceph@sha256:abc..., and should be
treated as such.  Otherwise, upgrade can get into a loop where it pulls
the image on a new host, finds the other variant of the repodigests,
sees no overlap, updates target_digests, and restarts.  (It will then
find the first variant again on the first host and loop.)

Avoid this by normalizing any docker.io digests by always including the
docker.io/ prefix.

Note that it is technically possible that this assumption is wrong: it
may be that the image that already exists on the local host is from a
different registry in registries.conf's unqualified-search-registries.
However, we don't know which, since this is a search list.  In practice,
it should be exceeding rare that an image that *we* are installing using
a fully-qualified image name will end up having an unqualified repodigest
in the local registry.  Hopefully!

Fixes: https://tracker.ceph.com/issues/50114
Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm/upgrade: normalize unqualified target image

If we get an unqualified target image, assume it's docker.io.  This
ensures that we're passing a fully-qualified target to docker|podman on
the various hosts and don't end up with something different based on the
per-host search path for unqualified image names.

Signed-off-by: Sage Weil <sage@newdream.net>

doc/rados: rewrite mclock docs (2 of 2)

This is my second attempt to rewrite the
second half of the mclock docs. The first attempt
is enshrined in https://github.com/ceph/ceph/pull/40571,
in which I got cute with git and got burned.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #39216 from myoungwon/wip-manifest-dedup-test

osd, test: reworks for manifest dedup test cases

Reviewed-by: Samuel Just <sjust@redhat.com>

qa/suites: test upgrade/octopus-x on focal instead bionic

so we can solely build on focal in future once all other bionic facets
in qa/ is removed or replaced.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40623 from ronen-fr/wip-ronenf-revert-40077

osd: Revert "osd: Try other PGs when reservation failures occur"

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #40606 from myoungwon/wip-49427-2

osd: recover unreadable snapshot before reading refcount info

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #40510 from aclamk/wip-bluestore-sharding-rst

doc: Add BlueStore sharding documentation

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mds: reject lookup ino requests for mds dirs

Fixes: https://tracker.ceph.com/issues/49922
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: test standby-replay with fs:workloads

Fixes: https://tracker.ceph.com/issues/50045
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #40486 into master

* refs/pull/40486/head:
mds: trim cache regularly for standby-replay
mds: remove extra heap release

Reviewed-by: Sidharth Anupkrishnan <sanupkri@redhat.com>

Merge PR #40520 into master

* refs/pull/40520/head:
mds/scrub: background scrub error fixes

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #40633 into master

* refs/pull/40633/head:
mds: ensure export_pin rank < max_mds

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #40638 into master

* refs/pull/40638/head:
mds: do not show the default auth if it's unambiguous
mds: switch to rank number instead

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mon: check mdsmap is resizeable before promoting standby-replay

If any MDS is up:creating, some rank data structures may not exist yet.

Fixes: https://tracker.ceph.com/issues/50215
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #40617 from tchaikov/wip-system-pmem

install-deps.sh: install libpmem libraries if WITH_PMEM is set

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #40467 into master

* refs/pull/40467/head:
doc: detail `fs snapshot mirror daemon status` mgr command
doc: s/<fs>/<fs_name>/g

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sébastien Han <seb@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #40654 from rzarzynski/wip-crimson-notify-lifetime

crimson/osd: fix the lifetime of Notify during timeouts

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40641 from idryomov/wip-require-ceph-common-for-ioc

packaging: require ceph-common for immutable object cache daemon

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: improve debugs around Watch / Notify.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: fix the lifetime of Notify during timeouts.

This fixes a segfault during `LibRadosWatchNotify.WatchNotify2Timeout`.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc: detail `fs snapshot mirror daemon status` mgr command

@leseb noticed that this interface did not have any documentation. Thanks!

Fixes: http://tracker.ceph.com/issues/50229
Signed-off-by: Venky Shankar <vshankar@redhat.com>

doc: s/<fs>/<fs_name>/g

Signed-off-by: Venky Shankar <vshankar@redhat.com>

run-make-check: enable WITH_RBD_RWL when WITH_PMEM is true

so we can at least build-test the rwl cache plugin in "make check" run.

this is an opt-in option which is only enabled if WITH_PMEM is "true",
we will set it in the corresponding ceph-build job.

Signed-off-by: Kefu Chai <kchai@redhat.com>

install-deps.sh: install libpmem libraries if WITH_PMEM is set

Signed-off-by: Kefu Chai <kchai@redhat.com>

do_freebsd.sh: disable WITH_SYSTEMD

as FreeBSD does not has systemd at the time of writing, so it does not
make sense to build or install systemd support.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: s/HAVE_MSGHDR/WITH_SYSTEMD/

as FreeBSD also has msghdr but it does not have systemd, or
flags like MFD_ALLOW_SEALING, O_TMPFILE or F_SEAL_GROW. so
use WITH_SYSTEMD for enabling journald backend of logging system.

also move the option of "WITH_SYSTEMD" up so that the src/CMakeLists.txt
is able to see the variable of WITH_SYSTEMD defined by it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: do not assert() in the case of no obc

Upon rollback, we should handle ENOENT case,
so what we should do here is to return NOOP.

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

osd: remove unnecessary ref handling in _delete_oid

Let's consider the following case when handling a delete op.
1. Delete --> whiteouted
2. Make clone

In this case, current code clears chunk_map and calls dec_all_manifest_refcount()
in _delete_oid() even if the clone still has the references.

To fix this, This commit remove unnecessary ref handling in _delete_oid, and
makes finish_ctx() to handle ref handling, which can aware of whether the
clone is created or not.

Also, remove oi.size == 0 condition in finish_ctx() to handle ref. counting
upon a delete op with whitedouted clone.

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

doc: Add BlueStore sharding documentation

Provide sharding explanation, configuration parameters explanation.
Provide resharding tool example into Pacific format.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
Co-authored-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #40596 from tchaikov/wip-cmake-clean

cmake: remove created directory when "clean" target is run

Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

qa/cephfs: remove create_keyring_file from cephfs_test_case.py

Since teuthology.orchestra.remote.mktemp() can write a temporary file
and not just create it, create_keyring_file() is now redundant.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa/cephfs: don't use sudo to write files in /tmp

Files in /tmp cannot be written by any user( including the root user)
other than the file owner even if the permission mode on the file is
777.

Fixes: https://tracker.ceph.com/issues/49466
Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #40342 from batrick/snap_schedule-qa-fixes

qa: snap_schedule test debugging

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #40625 from TRYTOBE8TME/wip-rgw-notifications

rgw/test: Changing force-branch to master

Merge pull request #40622 from batrick/dev-make

doc: update some docs still referring to make

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40647 from rzarzynski/wip-crimson-notify-send_comp-broken-debug

crimson/osd: fix broken debug in Notify::send_completion().

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge PR #40204 into master

* refs/pull/40204/head:
client: cleanup _preadv_pwritev_locked()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #40427 into master

* refs/pull/40427/head:
mds: skip the buffer in UnknownPayload::decode()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #40611 into master

* refs/pull/40611/head:
qa: retry cloning the ceph.git

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #40636 from tchaikov/wip-50196

qa/tasks/mgr: skip test_diskprediction_local on python>=3.8

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

mds: ensure export_pin rank < max_mds

Signed-off-by: Yanhu Cao <gmayyyha@gmail.com>

osd: recover unreadable snapshot before reading ref. count info

Manifest objects needs adjacent clones when incrementing/decrementing
refcount. This commit makes the current code to call get_manifest_ref_count
before reading ref. count info.

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

mgr/prometheus:Improve the pool metadata

Adds percent_used and used_bytes to the per pool
metrics group, and add additional labels to the ceph_pool_metadata
metric
- compression_mode - so consumers can tell whether compression
  is active
- description - provide a text string showing the protection
  scheme of the pool (e.g. replica2, or ec:4+2
- type - text string showing replicated or erasure enabling
  easy filtering across the pools

These additional fields allow compression savings to be more
easily shown. The inclusion of percent_used ensures that any
prometheus based view of pool usage will ALWAYS match the CLI.

Fixes: https://tracker.ceph.com/issues/49049
Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

Merge pull request #40632 from adamemerson/wip-warnless-4now

Warning Cleanup and Clang Compile Fix

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

rgw/test: Changing force-branch to master

This PR includes 2 things:
1. Changing force-branch to master and removing the git-remote. This change was forgetten for PR #39139.
2. Proper cleanup/removal after completion of commands more precisely removing the kafka logs directory.

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>

test/osd: Change const copy in loop to const reference

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Fix brace-enclosed-scalar warning

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Fix warnings from s3select

Since this is a submodule, just wrap the include in a pragma.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Remove unused variables

(Also mark the file as C++ rather than C)

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

test/librados: Remove unused capture

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rbd: Fix ambiguous call to addr.parse

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Fix signed comparison warning

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

test/librados: Disambiguate function call

Cleaning up a warning.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Add missing extension to file in sources list

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #38737 from IlsooByun/objmeta-key-underscore

rgw: keep underscore in metatdata key

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #39749 from linuxbox2/wip-olock-fixes

rgw: objectlock: improve client error messages

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #40263 from ivancich/wip-update-listing-comments

rgw: update bucket listing log output

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #40570 from zdover23/wip-doc-mclock-rewrite-first-half-2021-Apr-2

doc/rados: rewrite mclock-config-ref

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sridhar Seshasayee <sseshasa@redhat.com>

Merge pull request #40639 from yuvalif/wip-yuval-fix-50201

rgw/multisite: return correct error code when op fails

Reviewed-by: Casey Bodley <cbodley@redhat.com>

crimson/osd: fix broken debug in Notify::send_completion().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

qa/tasks/mgr: skip test_diskprediction_local on python>=3.8

query the python version before trying to test diskprediction_local

Fixes: https://tracker.ceph.com/issues/50196
Signed-off-by: Kefu Chai <kchai@redhat.com>

cephadm: cleanup extra slash in runtime dir

%t already contains a slash, no need to add an extra one

Signed-off-by: 胡玮文 <huww98@outlook.com>

Merge pull request #40581 from weixinwei/master

os/bluestore/BlueFS: do not _flush_range deleted files

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #40601 from cbodley/wip-50147

qa/rgw: move ignore-pg-availability.yaml out of suites/rgw

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

client: don't allow access to MDS-private inodes

Fixes: https://tracker.ceph.com/issues/50112
Signed-off-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #39907 from guits/fix_get_first_lv

ceph-volume: `get_first_lv()` refactor

packaging: require ceph-common for immutable object cache daemon

This daemon has a systemd service which starts it with --setuser ceph
--setgroup ceph.  "ceph" user and group are created by ceph-common and
won't be there unless ceph-common is installed.

Fixes: https://tracker.ceph.com/issues/50207
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #40624 from rhcs-dashboard/read-only-config-access-disable

mgr/dashboard: Revoke read-only user's access to Manager modules

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

mgr/selftest: add a command for querying python version

so the test driver can skip certain tests based on the version of python
runtime on the test node

Fixes: https://tracker.ceph.com/issues/50196
Signed-off-by: Kefu Chai <kchai@redhat.com>