Merge pull request #39617 from tchaikov/wip-mgr-diskprediction-local-flake8

mgr/diskprediction_local: wrap line longer than 100 chars

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #39638 from Rethan/fix-description-throttle-val

common/Throttle: change description about throttle val

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39667 from tchaikov/wip-tools-rados-optional

tools/rados: assign to optional<> without deref'ing it

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #39562 from Daniel-Pivonka/cephadmversionauthregerror

cephadm: version command hide traceback when login is needed

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39474 from sebastian-philipp/doc-cephadm-manual-mgr

doc/cephadm: troubleshooting: manually deploy MGR

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge pull request #39420 from sebastian-philipp/cephadm-enable-ubuntu-again

qa/suites/rados/cephadm: Add 20.04 podman:testing

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39421 from p-se/wip-pse-cephadm-port-in-use

cephadm: fix port_in_use when IPv6 is disabled

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39415 from jmolmo/paths_in_all_osd_components

cephadm: Allow to use paths in all <_devices> drivegroup sections

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39537 from adk3798/emsg-redeploy

mgr/cephadm: error if service action called with daemonless service

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39536 from liewegas/cephadm-drain-weight

mgr/cephadm: fix and improve osd draining

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

tools/rados: assign to optional<> without deref'ing it

this change addresses a regression introduced by
d333b35aa10bf03a8bc047994d5cf3fed019b49a

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39463 from trociny/wip-49115

librbd: use on-disk image name when storing mirror snapshot state

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39625 from trociny/wip-49418

rbd-mirror: reset update_status_task pointer in timer thread

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39622 from dillaman/wip-librbd-disable-qcow

librbd: permit disabling QCOW migration format support

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #38170 from rhcs-dashboard/feature-47375-select-rgw-daemon

mgr/dashboard: Select any object gateway on local cluster

Merge pull request #38940 from rhcs-dashboard/tenant-user

mgr/dashboard: Splitting tenant$user when creating rgw user

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #39322 from rzarzynski/wip-librados-nul-objname

librados, pybind, tests: allow to list objects with the NUL character in names

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #39570 from tchaikov/wip-getprogname

common: s/prctl/pthread_getname_np/ for better portability

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #39427 from ktdreyer/rpm-whitespace

ceph.spec.in: remove trailing whitespace

Reviewed-by: Andrew Schoen <aschoen@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #39621 from kshtsk/wip-sle-luarocks

ceph.spec.in: use lua53 for openSUSE 15.2

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

mgr/dashboard: select any object gateway on local cluster.

Dashboard backend settings:
- Refactoring: now accepting more than 1 type of value.
- RGW_API_ACCESS_KEY & RGW_API_SECRET_KEY accept string (backward compatibility: legacy behavior) as well as dictionary of strings for connecting multiple daemons.
- Ease of use: deprecated: mgr/dashboard/RGW_API_USER_ID: not useful anymore (kept for backward compatibility).

UI/UX:
- Created context component (to be shown only on rgw-related routes) for selecting operating daemon.
- Daemon selector only shown if there is more than 1 daemon running on a local cluster (to reduce cognitive load).

Fixes: https://tracker.ceph.com/issues/47375
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #39655 from cyx1231st/wip-seastore-onode-fix-cursor-cmp

crimson/onode-staged-tree: fix Cursor operator==()

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/onode-staged-tree: add missing type in Cache::alloc_new_extent_by_type()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: fix Cursor operator==

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #39653 from zdover23/wip-doc-dev-t8y-workflow-viewing-tests-results-2021-Feb-24

doc/dev: s/getting tests r.../getting test r...

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc/dev: s/getting tests r.../getting test r...

s/getting tests results/getting test results/

This commit changes a title so that it is grammatically
correct. The adjective "test" now properly modifies the
noun "results". Another way of thinking of this is that
the complex noun "test results" has had a typo in it
corrected.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #39564 from tchaikov/wip-doc-layout

doc: consolidate the layout settings using CSS

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #39288 from cybozu/fix-the-procedure-of-recovery-mon-quorum-using-osd

doc: fix the procedure of recovery mon quorum using osd

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39632 from jdurgin/wip-crontab

qa: adjust scheduled jobs to fit better in the lab

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #39615 from zdover23/wip-doc-dev-basic-workflow-intro-adding-ceph-ceph-2021-Feb-22

doc/dev: adding upstream repo name

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39606 from idryomov/wip-rbd-map-sanity-check

krbd: make sure the device node is accessible after the mapping

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #39596 into master

* refs/pull/39596/head:
qa/tasks/mgr/dashboard: add wear_level to devices schema

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

doc/dev: adding upstream repo name

This commit makes one sentence slightly less
ambiguous by designating the location from
which the "ceph fetch" command fetches objects.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #39608 from tchaikov/wip-crimson-fix-sequencer

crimson/osd: drop pending ops when pg interval changes

Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #39519 from petrutlucian94/minor_fixes

common, rbd: Minor Windows fixes

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

rbd: fix rbd-wnbd log messages

Previously, rbd-wnbd was called "rbd-nbd" as it was using NBD
to communicate with the driver.

The tool was then renamed to "rbd-wnbd" but some messages were
not updated.

This patch will simply replace "rbd-nbd" with "rbd-wnbd" in some
log messages.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

common: s/prctl/pthread_getname_np/ for better portability

prctl() is linux specific, let's use pthread_getname_np for thread names.

Signed-off-by: Kefu Chai <kchai@redhat.com>

librbd: permit disabling QCOW migration format support

Downstream Red Hat products do not support the older QCOW format. This
will allow the support for the legacy QCOW format to be disabled for the
new RBD import-only migration support.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

common/Throttle: change description about throttle val

l_throttle_val indicates current used count, not available.

Signed-off-by: haoyixing <haoyixing@kuaishou.com>

rbd: fix rbd-wnbd device status

The "rbd-wnbd show" command will always report the device status
as "inactive". This patch adds the missing check, similar to the
one used by the "list" command.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

common: fix win32 event log source

The Windows "get_process_name" function uses the input buffer
to store the entire executable path, while the caller only
expects the filename.

The "get_process_name_cpp" function is using an insufficient
buffer, for which reason it will return "(unknown)" when the
executable path exceeds 32 characters.

Windows event log entries have the wrong source because of this.

We'll update "get_process_name" to use a separate buffer for the
full executable path and avoid requesting a larger buffer than
actually needed.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Merge pull request #39594 from ronen-fr/wip-ronenf-peerreplayer

tools/cephfs-mirror: fix a dangling pointer

crimson/osd: s/opSequencer/op_sequencer/

to be more consistent with the naming convention in this project

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: drop pending ops when pg interval changes

fullfill the promises of pending ops when pg interval changes, so
we can drop them if the primary osd is changed as well.

note, this behavior depends on interruptable future.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39634 from tchaikov/wip-crimson-octopus-quincy

crimson/osd: assume an octopus+ osd peer

Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/osd: assume an octopus+ osd peer

no need to handle cases talking to pre-octopus osd peers in a post
quincy era.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/librbd: TestLibRBD.RenameViaLockOwner doesn't require journaling now

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #39604 from inspur-wyq/inspur-wyq-patch-1

doc/radosgw/: fix typos

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39572 from kotreshhr/version-auth-metadata-mgr

mgr/volumes: Bump up AuthMetadataManager's version

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge PR #39630 into master

* refs/pull/39630/head:
mgr/cephadm/upgrade: fix typo

Reviewed-by: Sebastian Wagner <swagner@suse.com>

qa: use --newest 100 to find a recent build

master and stable branches are only built a couple times a day now, so
we need to backtrack further.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

qa: deprioritize master and upgrade runs

Keep pacific at high for now. Since teuthology-dispatcher locks nodes
before starting jobs, the upgrade jobs no longer need high priority to
run on more machines than average.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

qa: schedule a wider variety of jobs

Use the day of the year instead of compressing everything into
quarters. This lets us get through more breadth of combinations in a
year.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

mgr/cephadm/upgrade: fix typo

Signed-off-by: Sage Weil <sage@newdream.net>

qa/crontab: adjust scheduled frequencies

smoke: 3x a week on master, weekly on other branches
pacific: every 2 days
master, octopus, nautilus, upgrades: weekly

This should be a net reduction in tests each week. Currently
we schedule far more than the lab can run in a given week.

Try to balance the scheduling over the course of the week.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #39138 into master

* refs/pull/39138/head:
qa: valgrind test for cephfs-mirror daemon
cephfs-mirror: use preforker for daemonizing
test: adjust sleep time to account for valgrind runs
cephfs-mirror: gracefully shutdown threads, timers, etc..
cephfs-mirror: call ceph_release() to cleanup mount alloc
cephfs-mirror: shutdown filesystem/cluster connections on shutdown
cephfs-mirror: set init failed flag on FSMirror::init() failure

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #39618 into master

* refs/pull/39618/head:
doc: fix broken links multimds and kcephfs

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/tasks/mgr/dashboard: add wear_level to devices schema

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #39620 from rzarzynski/wip-crimson-monc-failure-handling

crimson: multiple fixes for connection failure handling in MonClient

Reviewed-by: Kefu Chai <kchai@redhat.com>

rbd-mirror: reset update_status_task pointer in timer thread

To avoid a time window when m_update_status_task is invalid. If
during this time the cancel_update_mirror_image_replay_status is
called, it may cancel some other's ImageReplayer task, if it
happened to add the task with the same address.

Fixes: https://tracker.ceph.com/issues/49418
Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #39405 from rhcs-dashboard/cherrypy-security

mgr/dashboard: set security headers

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

ceph.spec.in: use lua53 for openSUSE 15.2

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

crimson/monc: drop the pending connection on ms_handle_reset().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/monc: don't assume reopen_session() shall establish active_con.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc: fix broken links multimds and kcephfs

Fixes: https://tracker.ceph.com/issues/49372
Signed-off-by: Jos Collin <jcollin@redhat.com>

mgr/diskprediction_local: wrap line longer than 100 chars

to appease flake8 and autopep8

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39432 from dillaman/wip-49238

rbd-mirror: delay update snapshot mirror image state

Reviewed-by: Mykola Golub <mgolub@suse.com>

librbd: use on-disk image name when storing mirror snapshot state

Fixes: https://tracker.ceph.com/issues/49115
Signed-off-by: Mykola Golub <mgolub@suse.com>

test/librbd: extend TestLibRBD.RenameViaLockOwner

To cover the following case:

- Client A has image opened but does not owns the lock.
- Client B renames the image (client A is not aware of it).
- Client A becomes the lock owner.
- Client B requests rename, which is proxied to the client A.

Signed-off-by: Mykola Golub <mgolub@suse.com>

mgr/dashboard: Splitting tenant$user when creating rgw user

Fixes: https://tracker.ceph.com/issues/47378
Signed-off-by: Nizamudeen A <nia@redhat.com>

krbd: make sure the device node is accessible after the mapping

We have always assumed this to be the case and users' scripts and
orchestration tools have grown to depend on this.  Let's add some
enforcement, prompted by [1]:

  "I am running my Kubernetes worker node inside of an LXC container
   which doesn't benefit from the device node created by the kernel, so
   I'm using udev to create the /dev/rbd* device nodes inside of the LXC
   container."

which, through the unfortunate interaction with ceph-csi rbd plugin,
results in data loss for "volumeMode: Filesystem" PVs because it ends
up recreating the filesystem every time the PV is attached to the pod:

  "When deleting the pod and re-creating it, I can see that the RBD
   image is indeed being reformatted. This seems to be because when
   blkid is being run to check if the image is formatted, the /dev/rbd*
   device has not yet been created by udev. By the time the code gets
   down to running mkfs, the device is there and the damage is done."

[1] https://github.com/ceph/ceph-csi/issues/1820

Fixes: https://tracker.ceph.com/issues/49410
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

doc/radosgw/: fix typos

Signed-off-by: wangyunqing wangyunqing@inspur.com

tools/cephfs-mirror: fix a dangling pointer

stringify(snap_id) is ephemeral:

tools/cephfs_mirror/PeerReplayer.cc:916:62: warning: object backing the
pointer will be destroyed at the end of the full-expression

  snap_metadata snap_meta[] = {{PRIMARY_SNAP_ID_KEY.c_str(), stringify(snap_id).c_str()}};

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge PR #39328 into master

* refs/pull/39328/head:
tools/rados: replace omap_key_valid with std::optional.
tools/rados: replace omap_key_pretty with prettify().
tools/rados: add support for binary object names.

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge PR #39394 into master

* refs/pull/39394/head:
tools/kvstore-tool: implement 'dissect' command
kv/KeyValueHistogram: make Bluestore's DBHistogram class reusable.

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>

Merge PR #39460 into master

* refs/pull/39460/head:
msg/async/dpdk: be compatible with DPDK 19.11
common/Tub: remove Tub.h
cmake: use NUMA for package name in FindNUMA.cmake
msg/async/dpdk: define MARKER in .hh
msg/async/dpdk: use optional<> instead of Tub<>
compressor/lz4: use optional<> instead of Tub<>
compressor/lz4: use range-based loop

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #39495 into master

* refs/pull/39495/head:
mgr/cephadm: remove spec from CephadmDaemonDeploySpec

Reviewed-by: Adam King <adking@redhat.com>

Merge PR #39535 into master

* refs/pull/39535/head:
osd: Don't clear m_planned_scrub until scrub finishes successfully

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #39478 from orozery/rbd-encryption-doc

doc/rbd: rbd encryption

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #39504 into master

* refs/pull/39504/head:
mgr/cephadm/upgrade: restart mgr after mons upgrade to pacific
mgr: rename config_prefix -> mgr_store_prefix
mgr: fix config option prefix
mgr/MgrStandby: simplify weird code
mgr/cephadm: use get_foreign_ceph_option() instead of 'config get' mon command
mgr_module: get_foreign_ceph_option(entity, name)
mgr: maintain a ConfigMap copy
mon/ConfigMap: factor out parse_key helper
mon/ConfigMap: fix stray option leak
mgr: use new kv subscription for mgr/, device/, config/
mon: allow subscription to kv/config-key data
mon: convert ConfigKeyService -> KVMonitor

Reviewed-by: João Eduardo Luís <joao@suse.de>

Merge pull request #39348 from xxhdx1985126/wip-crimson-client-request-sequencer

crimson/osd: client request sequencer

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39552 from smithfarm/wip-49356

rpm: openSUSE Tumbleweed has lua54 now

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: log when obc lock is acquired

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/osd: add I/O sequencer to preserve client_requests' order across PG interval change

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/osd: interrupt all client_requests that are waiting for acquiring obc locks

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

mgr/volumes: Bump up AuthMetadataManager's version

With ceph_volume_client and mgr-volumes co-existing
for sometime, the version of both needs to be same.
The ceph_volume_client version <=5 can't decode
'subvolumes' key in auth-metadata file. Hence to
handle version in-compatibility, the version of
ceph_volume_client is bumped up to 6 and the same
needs to be done in mgr-volume's AuthMetadataManager

Fixes: https://tracker.ceph.com/issues/49374
Signed-off-by: Kotresh HR <khiremat@redhat.com>

doc/rbd: rbd encryption

This commit adds documentation of the new rbd encryption feature

Signed-off-by: Or Ozeri <oro@il.ibm.com>

Merge pull request #39586 from liewegas/bug-45441

qa/suites/rados/singletone: ignorelist MON_DOWN when injecting msgr errors

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #39587 from dillaman/wip-test-neorados

cmake: libneoradostest-support should be static

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39591 from tchaikov/wip-mgr-diskpred-ann

mgr/diskprediction_local: add typing annotations and cleanups

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm/upgrade: restart mgr after mons upgrade to pacific

If we are upgrading to pacific, we need to restart the mgr *after* the
mons have a pacific quorum so that they can get the kv subscriptions.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #39539 from lixiaoy1/enable_pwl

cmake: enable write-back cache in spec

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39567 from lixiaoy1/fix_read_ios

librbd/cache/pwl: wait for the completion of reads

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39522 from trociny/wip-49284

mgr/rbd_support: mirror snapshot schedule should skip non-primary images

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39451 from trociny/wip-49282

librbd: allow disabling journaling for snapshot based mirroring image

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39481 from trociny/wip-luks-encryption-test-sudo

qa/workunits/rbd: make luks-encryption test work on vstart cluster

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #39582 into master

* refs/pull/39582/head:
mon: fix INCOMPAT_QUINCY ondisk compatset feature bit

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39590 from tchaikov/wip-49395

ceph.spec.in: s/%cmake/cmake/

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39592 from tchaikov/wip-crimson-read-exactly

crimson/net: throw read_eof if short read

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/net: throw read_eof if short read

as per the implementation and the document of input_stream::read_exactly():

/// stream and returns them. If the end of stream is reached before n
/// bytes were read, fewer than n bytes will be returned - so despite
/// the method's name, the caller must not assume the returned buffer
/// will always contain exactly n bytes.

Signed-off-by: Kefu Chai <kchai@redhat.com>