Merge pull request #39653 from zdover23/wip-doc-dev-t8y-workflow-viewing-tests-results-2021-Feb-24

doc/dev: s/getting tests r.../getting test r...

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc/dev: s/getting tests r.../getting test r...

s/getting tests results/getting test results/

This commit changes a title so that it is grammatically
correct. The adjective "test" now properly modifies the
noun "results". Another way of thinking of this is that
the complex noun "test results" has had a typo in it
corrected.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #39564 from tchaikov/wip-doc-layout

doc: consolidate the layout settings using CSS

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #39288 from cybozu/fix-the-procedure-of-recovery-mon-quorum-using-osd

doc: fix the procedure of recovery mon quorum using osd

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39632 from jdurgin/wip-crontab

qa: adjust scheduled jobs to fit better in the lab

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #39615 from zdover23/wip-doc-dev-basic-workflow-intro-adding-ceph-ceph-2021-Feb-22

doc/dev: adding upstream repo name

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39606 from idryomov/wip-rbd-map-sanity-check

krbd: make sure the device node is accessible after the mapping

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #39596 into master

* refs/pull/39596/head:
qa/tasks/mgr/dashboard: add wear_level to devices schema

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

doc/dev: adding upstream repo name

This commit makes one sentence slightly less
ambiguous by designating the location from
which the "ceph fetch" command fetches objects.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #39608 from tchaikov/wip-crimson-fix-sequencer

crimson/osd: drop pending ops when pg interval changes

Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #39519 from petrutlucian94/minor_fixes

common, rbd: Minor Windows fixes

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

rbd: fix rbd-wnbd log messages

Previously, rbd-wnbd was called "rbd-nbd" as it was using NBD
to communicate with the driver.

The tool was then renamed to "rbd-wnbd" but some messages were
not updated.

This patch will simply replace "rbd-nbd" with "rbd-wnbd" in some
log messages.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

rbd: fix rbd-wnbd device status

The "rbd-wnbd show" command will always report the device status
as "inactive". This patch adds the missing check, similar to the
one used by the "list" command.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

common: fix win32 event log source

The Windows "get_process_name" function uses the input buffer
to store the entire executable path, while the caller only
expects the filename.

The "get_process_name_cpp" function is using an insufficient
buffer, for which reason it will return "(unknown)" when the
executable path exceeds 32 characters.

Windows event log entries have the wrong source because of this.

We'll update "get_process_name" to use a separate buffer for the
full executable path and avoid requesting a larger buffer than
actually needed.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Merge pull request #39594 from ronen-fr/wip-ronenf-peerreplayer

tools/cephfs-mirror: fix a dangling pointer

crimson/osd: s/opSequencer/op_sequencer/

to be more consistent with the naming convention in this project

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: drop pending ops when pg interval changes

fullfill the promises of pending ops when pg interval changes, so
we can drop them if the primary osd is changed as well.

note, this behavior depends on interruptable future.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39634 from tchaikov/wip-crimson-octopus-quincy

crimson/osd: assume an octopus+ osd peer

Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/osd: assume an octopus+ osd peer

no need to handle cases talking to pre-octopus osd peers in a post
quincy era.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39604 from inspur-wyq/inspur-wyq-patch-1

doc/radosgw/: fix typos

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39572 from kotreshhr/version-auth-metadata-mgr

mgr/volumes: Bump up AuthMetadataManager's version

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge PR #39630 into master

* refs/pull/39630/head:
mgr/cephadm/upgrade: fix typo

Reviewed-by: Sebastian Wagner <swagner@suse.com>

qa: use --newest 100 to find a recent build

master and stable branches are only built a couple times a day now, so
we need to backtrack further.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

qa: deprioritize master and upgrade runs

Keep pacific at high for now. Since teuthology-dispatcher locks nodes
before starting jobs, the upgrade jobs no longer need high priority to
run on more machines than average.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

qa: schedule a wider variety of jobs

Use the day of the year instead of compressing everything into
quarters. This lets us get through more breadth of combinations in a
year.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

mgr/cephadm/upgrade: fix typo

Signed-off-by: Sage Weil <sage@newdream.net>

qa/crontab: adjust scheduled frequencies

smoke: 3x a week on master, weekly on other branches
pacific: every 2 days
master, octopus, nautilus, upgrades: weekly

This should be a net reduction in tests each week. Currently
we schedule far more than the lab can run in a given week.

Try to balance the scheduling over the course of the week.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #39138 into master

* refs/pull/39138/head:
qa: valgrind test for cephfs-mirror daemon
cephfs-mirror: use preforker for daemonizing
test: adjust sleep time to account for valgrind runs
cephfs-mirror: gracefully shutdown threads, timers, etc..
cephfs-mirror: call ceph_release() to cleanup mount alloc
cephfs-mirror: shutdown filesystem/cluster connections on shutdown
cephfs-mirror: set init failed flag on FSMirror::init() failure

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #39618 into master

* refs/pull/39618/head:
doc: fix broken links multimds and kcephfs

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/tasks/mgr/dashboard: add wear_level to devices schema

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #39620 from rzarzynski/wip-crimson-monc-failure-handling

crimson: multiple fixes for connection failure handling in MonClient

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39405 from rhcs-dashboard/cherrypy-security

mgr/dashboard: set security headers

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

crimson/monc: drop the pending connection on ms_handle_reset().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/monc: don't assume reopen_session() shall establish active_con.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc: fix broken links multimds and kcephfs

Fixes: https://tracker.ceph.com/issues/49372
Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #39432 from dillaman/wip-49238

rbd-mirror: delay update snapshot mirror image state

Reviewed-by: Mykola Golub <mgolub@suse.com>

krbd: make sure the device node is accessible after the mapping

We have always assumed this to be the case and users' scripts and
orchestration tools have grown to depend on this.  Let's add some
enforcement, prompted by [1]:

  "I am running my Kubernetes worker node inside of an LXC container
   which doesn't benefit from the device node created by the kernel, so
   I'm using udev to create the /dev/rbd* device nodes inside of the LXC
   container."

which, through the unfortunate interaction with ceph-csi rbd plugin,
results in data loss for "volumeMode: Filesystem" PVs because it ends
up recreating the filesystem every time the PV is attached to the pod:

  "When deleting the pod and re-creating it, I can see that the RBD
   image is indeed being reformatted. This seems to be because when
   blkid is being run to check if the image is formatted, the /dev/rbd*
   device has not yet been created by udev. By the time the code gets
   down to running mkfs, the device is there and the damage is done."

[1] https://github.com/ceph/ceph-csi/issues/1820

Fixes: https://tracker.ceph.com/issues/49410
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

doc/radosgw/: fix typos

Signed-off-by: wangyunqing wangyunqing@inspur.com

tools/cephfs-mirror: fix a dangling pointer

stringify(snap_id) is ephemeral:

tools/cephfs_mirror/PeerReplayer.cc:916:62: warning: object backing the
pointer will be destroyed at the end of the full-expression

  snap_metadata snap_meta[] = {{PRIMARY_SNAP_ID_KEY.c_str(), stringify(snap_id).c_str()}};

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge PR #39328 into master

* refs/pull/39328/head:
tools/rados: replace omap_key_valid with std::optional.
tools/rados: replace omap_key_pretty with prettify().
tools/rados: add support for binary object names.

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge PR #39394 into master

* refs/pull/39394/head:
tools/kvstore-tool: implement 'dissect' command
kv/KeyValueHistogram: make Bluestore's DBHistogram class reusable.

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>

Merge PR #39460 into master

* refs/pull/39460/head:
msg/async/dpdk: be compatible with DPDK 19.11
common/Tub: remove Tub.h
cmake: use NUMA for package name in FindNUMA.cmake
msg/async/dpdk: define MARKER in .hh
msg/async/dpdk: use optional<> instead of Tub<>
compressor/lz4: use optional<> instead of Tub<>
compressor/lz4: use range-based loop

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #39495 into master

* refs/pull/39495/head:
mgr/cephadm: remove spec from CephadmDaemonDeploySpec

Reviewed-by: Adam King <adking@redhat.com>

Merge PR #39535 into master

* refs/pull/39535/head:
osd: Don't clear m_planned_scrub until scrub finishes successfully

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #39478 from orozery/rbd-encryption-doc

doc/rbd: rbd encryption

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #39504 into master

* refs/pull/39504/head:
mgr/cephadm/upgrade: restart mgr after mons upgrade to pacific
mgr: rename config_prefix -> mgr_store_prefix
mgr: fix config option prefix
mgr/MgrStandby: simplify weird code
mgr/cephadm: use get_foreign_ceph_option() instead of 'config get' mon command
mgr_module: get_foreign_ceph_option(entity, name)
mgr: maintain a ConfigMap copy
mon/ConfigMap: factor out parse_key helper
mon/ConfigMap: fix stray option leak
mgr: use new kv subscription for mgr/, device/, config/
mon: allow subscription to kv/config-key data
mon: convert ConfigKeyService -> KVMonitor

Reviewed-by: João Eduardo Luís <joao@suse.de>

Merge pull request #39348 from xxhdx1985126/wip-crimson-client-request-sequencer

crimson/osd: client request sequencer

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39552 from smithfarm/wip-49356

rpm: openSUSE Tumbleweed has lua54 now

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: log when obc lock is acquired

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/osd: add I/O sequencer to preserve client_requests' order across PG interval change

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/osd: interrupt all client_requests that are waiting for acquiring obc locks

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

mgr/volumes: Bump up AuthMetadataManager's version

With ceph_volume_client and mgr-volumes co-existing
for sometime, the version of both needs to be same.
The ceph_volume_client version <=5 can't decode
'subvolumes' key in auth-metadata file. Hence to
handle version in-compatibility, the version of
ceph_volume_client is bumped up to 6 and the same
needs to be done in mgr-volume's AuthMetadataManager

Fixes: https://tracker.ceph.com/issues/49374
Signed-off-by: Kotresh HR <khiremat@redhat.com>

doc/rbd: rbd encryption

This commit adds documentation of the new rbd encryption feature

Signed-off-by: Or Ozeri <oro@il.ibm.com>

Merge pull request #39586 from liewegas/bug-45441

qa/suites/rados/singletone: ignorelist MON_DOWN when injecting msgr errors

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #39587 from dillaman/wip-test-neorados

cmake: libneoradostest-support should be static

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39591 from tchaikov/wip-mgr-diskpred-ann

mgr/diskprediction_local: add typing annotations and cleanups

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm/upgrade: restart mgr after mons upgrade to pacific

If we are upgrading to pacific, we need to restart the mgr *after* the
mons have a pacific quorum so that they can get the kv subscriptions.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #39539 from lixiaoy1/enable_pwl

cmake: enable write-back cache in spec

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39567 from lixiaoy1/fix_read_ios

librbd/cache/pwl: wait for the completion of reads

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39522 from trociny/wip-49284

mgr/rbd_support: mirror snapshot schedule should skip non-primary images

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39451 from trociny/wip-49282

librbd: allow disabling journaling for snapshot based mirroring image

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39481 from trociny/wip-luks-encryption-test-sudo

qa/workunits/rbd: make luks-encryption test work on vstart cluster

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #39582 into master

* refs/pull/39582/head:
mon: fix INCOMPAT_QUINCY ondisk compatset feature bit

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39590 from tchaikov/wip-49395

ceph.spec.in: s/%cmake/cmake/

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39592 from tchaikov/wip-crimson-read-exactly

crimson/net: throw read_eof if short read

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/net: throw read_eof if short read

as per the implementation and the document of input_stream::read_exactly():

/// stream and returns them. If the end of stream is reached before n
/// bytes were read, fewer than n bytes will be returned - so despite
/// the method's name, the caller must not assume the returned buffer
/// will always contain exactly n bytes.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/diskprediction_local: add typing annotations

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/diskprediction_local: raise if Predictor cannot be initialized

simpler this way.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/diskprediction_local: add Predictor abstract class

to make the interface more explicit, also pave the road to mypy.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/diskprediction_local: drop unused COMMANDS

the base class of `MgrModule` already has it

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/diskprediction_local: use "int" for interval options

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/diskprediction_local: define options using Option

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/diskprediction_local: add to flake8

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39588 from athanatos/sjust/wip-fix-store-nbd

crimson/tools/store-nbd: fix invalid buffer access on connection termination

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph.spec.in: s/%cmake/cmake/

this change partially reverts da7030db79c5dbd480040eb249a76638caf52707
which use %cmake rpm macro in the place of "cmake". but

%cmake sets BUILD_SHARED_LIBS=ON. so quite a few internal libraries
defined using add_library() are now compiled into shared libraries which
are not installed or packagesd. when we are installing the rpm packages
compiled with this option, rpm compiles because the linked libraries are
missing, for instance, `libgmock.so.1.10.0` was compiled as a static
library before da7030db79c5dbd480040eb249a76638caf52707, and was
included by the test executables. but after that change it's compiled
as a shared library.

so we need to either package the linked shared libraries or just link
against them statically. at this moment, the latter approach is simpler,
albeit larger size of exectuable and dbg symbols.

Fixes: https://tracker.ceph.com/issues/49395
Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/tools/store-nbd: actually check magic

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #39573 from weixinwei/master

librbd: memory is not preallocated in rbd c++ read api

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

crimson/tools/store-nbd: check length of buffer

read_exactly does not return an error if the stream
has ended.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #39585 from ivancich/wip-clean-rgw-warnings-1

rgw: clean up some compiler warnings

Reviewed-by: Ali Maredia <amaredia@redhat.com>

cmake: libneoradostest-support should be static

Some builds seem to be creating a libneoradostest-support.so dynamic
library instead, which results in unresolved dependencies when installing
ceph-test.

See: https://tracker.ceph.com/issues/38611
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cmake: enable write-back cache in spec

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

qa/suites/rados/singletone: whitelist MON_DOWN when injecting msgr errors

add a 'none.yaml' facet which doesn't whitelist so we hopefully capture
other causes of MON_DOWN.

This fixes at least one source of https://tracker.ceph.com/issues/45441

Fixes: https://tracker.ceph.com/issues/45441
Signed-off-by: Sage Weil <sage@newdream.net>

rgw: clean up some compiler warnings

Minimal changes to clean up a few compiler warnings.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge PR #39561 into master

* refs/pull/39561/head:
qa/valgrind.supp: suppress free[] warnings
qa/suites: do not use notcmalloc flavor

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mon: fix INCOMPAT_QUINCY ondisk compatset feature bit

This was overlapping with pacific.

Fixes: https://tracker.ceph.com/issues/49143
Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #39501 from rhcs-dashboard/fix-tooltip-position

mgr/dashboard: fix tooltip for Provisioned/Total Provisioned fields

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge PR #38763 into master

* refs/pull/38763/head:
doc/cephfs: improve fs-nfs-exports.rst
doc/cephfs: improve nfs.rst

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

librbd/mirror: leave non-primary snapshot images in creating state

The creating state is a special case in rbd-mirror where it will
automatically delete the image since it assumes it's malformed.
A non-primary, snapshot-based mirror image needs to have at least
one non-primary snapshot and the first one is not created until
after replay has started. Now rbd-mirror will update the mirror
image state to the enabled state after creating the first
non-primary snapshot but before attempting the sync.

Fixes: https://tracker.ceph.com/issues/49238
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: ensure that the last non-primary snapshot cannot be pruned

Tweak the normal pruning behavior to ensure that an incomplete initial
non-primary snapshot is not included in the prune set since we know
it will be complete since otherwise the image would have been deleted
due to not updating the mirror-image-state to enabled. Also ensure
we cannot prune a non-primary mirror snapshot if we don't have a
predecessor.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39575 from zdover23/wip-doc-dev-t8y-intro-remove-upgrade-link-19-Feb-2021

doc/dev: intro: removing ceph-deploy mention

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #39569 from yanghonggang/wip-multisite-etag-misidentify

rgw: multisite: fix single-part-MPU object etag misidentify problem

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc/dev: intro: removing ceph-deploy mention

This removes a mention of the ceph-deploy manpage.
We also no longer direct people to use horses as
a primary form of transportation.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

librbd: memory is not preallocated in rbd c++ read api

Signed-off-by: weixinwei <weixw3@lenovo.com>

Merge PR #39566 into master

* refs/pull/39566/head:
cmake: build static libs if they are internal ones

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #39472 from sebastian-philipp/doc-cephadm-host-labels

doc/orch: Host labels

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #39571 from sebastian-philipp/doc-remove-upstart-from-freebsd

doc/install: Remove Upstart from Freebsd deployment

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #39546 from zdover23/wip-doc-dev-teuthology-intro-suites-inventory-rewrite-18-Feb-2021

doc/dev: t5logy: rewrite "suites inventory"

Reviewed-by: Neha Ojha <nojha@redhat.com>

doc/install: Remove Upstart from Freebsd deployment

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

rpm: openSUSE Tumbleweed has lua54 now

Until someone figures out an easier way, we are stuck with the minor Lua version
in the package names.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

mgr/dashboard: set security headers

Fixes: https://tracker.ceph.com/issues/49243
Signed-off-by: Avan Thakkar <athakkar@redhat.com>