qa: run fs:verify on all distros

It's believed this is no longer a problem now that we use tcmalloc.

Fixes: https://tracker.ceph.com/issues/49391
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #39649 from Daniel-Pivonka/fixorphaninitialdaemons

cephadm: fix bug in orphan-initial-daemons logic

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #39613 from sebastian-philipp/DaemonDescriptionStatus

mgr/orch: Add DaemonDescriptionStatus

Reviewed-by: Daniel-Pivonka <dpivonka@redhat.com>
Reviewed-by: Varsha Rao <varao@redhat.com>

Merge pull request #39672 from ceph/wip-159-notes

doc/releases/octopus.rst: 15.2.9 Release Notes

Merge pull request #39598 from tchaikov/wip-crimson-op-sequencer

crimson/osd: refactor OperationRepeatSequencer

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #39659 from tchaikov/wip-pybind-rados-cleanup

pybind/rados: remove rados_nobjects_list_next() from .pxi

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #38937 from myoungwon/fix-snap-refcount

src/test: fix to avoid fail notification when testing manifest refcount

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #39610 from tchaikov/wip-mgr-set-option

mgr: raise if set_module_option() fails

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #39320 from ifed01/wip-ifed-fix-huge-bluefs-write

os/bluestore: fix huge(>4GB) writes from RocksDB to BlueFS.

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>

Merge pull request #39600 from tchaikov/wip-mgr-zabbix-desc

mgr/zabbix: add desc to commands

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #39676 from liewegas/fix-kv

mon/KVMonitor: fix 'osd new' cross-service commit

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #38914 into master

* refs/pull/38914/head:
qa: bump osd heartbeat grace for ffsb workload

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge PR #39498 into master

* refs/pull/39498/head:
test: use std::atomic<bool> instead of volatile for cb_done var

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #39183 into master

* refs/pull/39183/head:
mon: simplify fs 'services' view of 'ceph status'
mds: add volumes + status to data: section of 'ceph status'

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

pybind/rados: remove rados_nobjects_list_next() from .pxi

this is a follow-up of da5d4c813ffdd391da54dcf5022763616bab4b21

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39668 from zdover23/wip-doc-dev-t8y-rewrite-naming-ceph-ci-branch-2021-Feb-25

doc/dev: t8y-naming-ceph-ci-branch rewrite

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39140 from sseshasa/wip-dmclock-profiles-improvements

osd: Refinements to mclock built-in profiles implementation.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sunny Kumar <sunkumar@redhat.com>

Merge pull request #38921 from lixiaoy1/pwl_teuthology

qa: add tests for persistent writeback cache

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mon/KVMonitor: fix 'osd new' cross-service commit

When we converted ConfigKeyService to KVMonitor, we didn't correctly
change this to propose_pending(), which mean that the kv change wasn't
captured in the paxos transaction.

Fixes: bb7ebc41532aeb23cff2241ab07b3f01c2f57ddd
Fixes: https://tracker.ceph.com/issues/49460
Signed-off-by: Sage Weil <sage@newdream.net>

doc/releases/octopus.rst: 15.2.9 Release Notes

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge PR #39607 into master

* refs/pull/39607/head:
qa/cephfs: check for Ceph FS before mounting
qa/cephfs: remove parameter createfs from mount classes

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #39609 into master

* refs/pull/39609/head:
doc/ceph-fuse: add some options to man page

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #39617 from tchaikov/wip-mgr-diskprediction-local-flake8

mgr/diskprediction_local: wrap line longer than 100 chars

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #39638 from Rethan/fix-description-throttle-val

common/Throttle: change description about throttle val

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39667 from tchaikov/wip-tools-rados-optional

tools/rados: assign to optional<> without deref'ing it

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #39562 from Daniel-Pivonka/cephadmversionauthregerror

cephadm: version command hide traceback when login is needed

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/dev: t8y-naming-ceph-ci-branch rewrite

This PR rewrites the "Naming the Ceph Branch"
section of the Teuthology docs, improving its
elegance and unburying the ledes.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #39474 from sebastian-philipp/doc-cephadm-manual-mgr

doc/cephadm: troubleshooting: manually deploy MGR

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge pull request #39420 from sebastian-philipp/cephadm-enable-ubuntu-again

qa/suites/rados/cephadm: Add 20.04 podman:testing

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39421 from p-se/wip-pse-cephadm-port-in-use

cephadm: fix port_in_use when IPv6 is disabled

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39415 from jmolmo/paths_in_all_osd_components

cephadm: Allow to use paths in all <_devices> drivegroup sections

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39537 from adk3798/emsg-redeploy

mgr/cephadm: error if service action called with daemonless service

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39536 from liewegas/cephadm-drain-weight

mgr/cephadm: fix and improve osd draining

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

tools/rados: assign to optional<> without deref'ing it

this change addresses a regression introduced by
d333b35aa10bf03a8bc047994d5cf3fed019b49a

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39463 from trociny/wip-49115

librbd: use on-disk image name when storing mirror snapshot state

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39625 from trociny/wip-49418

rbd-mirror: reset update_status_task pointer in timer thread

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/cephfs: check for Ceph FS before mounting

If the given Ceph FS, or the default Ceph FS when no Ceph FS is given,
is absent, abort the execution with AsssertionError and an error
message.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #39622 from dillaman/wip-librbd-disable-qcow

librbd: permit disabling QCOW migration format support

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #38170 from rhcs-dashboard/feature-47375-select-rgw-daemon

mgr/dashboard: Select any object gateway on local cluster

Merge pull request #38940 from rhcs-dashboard/tenant-user

mgr/dashboard: Splitting tenant$user when creating rgw user

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #39322 from rzarzynski/wip-librados-nul-objname

librados, pybind, tests: allow to list objects with the NUL character in names

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #39570 from tchaikov/wip-getprogname

common: s/prctl/pthread_getname_np/ for better portability

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #39427 from ktdreyer/rpm-whitespace

ceph.spec.in: remove trailing whitespace

Reviewed-by: Andrew Schoen <aschoen@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #39621 from kshtsk/wip-sle-luarocks

ceph.spec.in: use lua53 for openSUSE 15.2

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

crimson/osd: s/OperationRepeatSequencer/OpSequencers/

as pointed out by Radoslaw Zarzynski,

> this is actually a registry of sequencers.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: refactor OperationRepeatSequencer

* extract the OpSequencer out from OperationRepeatSequencer
* refactor OpSequencer so we don't need to track the ops using a map,
  only track the last op and last pg interval for better performance
  and smaller memory footprint.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: select any object gateway on local cluster.

Dashboard backend settings:
- Refactoring: now accepting more than 1 type of value.
- RGW_API_ACCESS_KEY & RGW_API_SECRET_KEY accept string (backward compatibility: legacy behavior) as well as dictionary of strings for connecting multiple daemons.
- Ease of use: deprecated: mgr/dashboard/RGW_API_USER_ID: not useful anymore (kept for backward compatibility).

UI/UX:
- Created context component (to be shown only on rgw-related routes) for selecting operating daemon.
- Daemon selector only shown if there is more than 1 daemon running on a local cluster (to reduce cognitive load).

Fixes: https://tracker.ceph.com/issues/47375
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

crimson/common: let the first id to be 1

it'd be simpler if we can use 0 as a sentry for sequencer. 0 would be
id of the last completed operation.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39655 from cyx1231st/wip-seastore-onode-fix-cursor-cmp

crimson/onode-staged-tree: fix Cursor operator==()

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/onode-staged-tree: add missing type in Cache::alloc_new_extent_by_type()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: fix Cursor operator==

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #39653 from zdover23/wip-doc-dev-t8y-workflow-viewing-tests-results-2021-Feb-24

doc/dev: s/getting tests r.../getting test r...

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc/dev: s/getting tests r.../getting test r...

s/getting tests results/getting test results/

This commit changes a title so that it is grammatically
correct. The adjective "test" now properly modifies the
noun "results". Another way of thinking of this is that
the complex noun "test results" has had a typo in it
corrected.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #39564 from tchaikov/wip-doc-layout

doc: consolidate the layout settings using CSS

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #39288 from cybozu/fix-the-procedure-of-recovery-mon-quorum-using-osd

doc: fix the procedure of recovery mon quorum using osd

Reviewed-by: Neha Ojha <nojha@redhat.com>

cephadm: fix bug in orphan-initial-daemons logic

Signed-off-by: Daniel Pivonka <dpivonka@redhat.com>

Merge pull request #39632 from jdurgin/wip-crontab

qa: adjust scheduled jobs to fit better in the lab

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #39615 from zdover23/wip-doc-dev-basic-workflow-intro-adding-ceph-ceph-2021-Feb-22

doc/dev: adding upstream repo name

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39606 from idryomov/wip-rbd-map-sanity-check

krbd: make sure the device node is accessible after the mapping

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #39596 into master

* refs/pull/39596/head:
qa/tasks/mgr/dashboard: add wear_level to devices schema

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

doc/dev: adding upstream repo name

This commit makes one sentence slightly less
ambiguous by designating the location from
which the "ceph fetch" command fetches objects.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #39608 from tchaikov/wip-crimson-fix-sequencer

crimson/osd: drop pending ops when pg interval changes

Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #39519 from petrutlucian94/minor_fixes

common, rbd: Minor Windows fixes

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

rbd: fix rbd-wnbd log messages

Previously, rbd-wnbd was called "rbd-nbd" as it was using NBD
to communicate with the driver.

The tool was then renamed to "rbd-wnbd" but some messages were
not updated.

This patch will simply replace "rbd-nbd" with "rbd-wnbd" in some
log messages.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

common: s/prctl/pthread_getname_np/ for better portability

prctl() is linux specific, let's use pthread_getname_np for thread names.

Signed-off-by: Kefu Chai <kchai@redhat.com>

librbd: permit disabling QCOW migration format support

Downstream Red Hat products do not support the older QCOW format. This
will allow the support for the legacy QCOW format to be disabled for the
new RBD import-only migration support.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

common/Throttle: change description about throttle val

l_throttle_val indicates current used count, not available.

Signed-off-by: haoyixing <haoyixing@kuaishou.com>

rbd: fix rbd-wnbd device status

The "rbd-wnbd show" command will always report the device status
as "inactive". This patch adds the missing check, similar to the
one used by the "list" command.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

common: fix win32 event log source

The Windows "get_process_name" function uses the input buffer
to store the entire executable path, while the caller only
expects the filename.

The "get_process_name_cpp" function is using an insufficient
buffer, for which reason it will return "(unknown)" when the
executable path exceeds 32 characters.

Windows event log entries have the wrong source because of this.

We'll update "get_process_name" to use a separate buffer for the
full executable path and avoid requesting a larger buffer than
actually needed.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Merge pull request #39594 from ronen-fr/wip-ronenf-peerreplayer

tools/cephfs-mirror: fix a dangling pointer

crimson/osd: s/opSequencer/op_sequencer/

to be more consistent with the naming convention in this project

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: drop pending ops when pg interval changes

fullfill the promises of pending ops when pg interval changes, so
we can drop them if the primary osd is changed as well.

note, this behavior depends on interruptable future.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39634 from tchaikov/wip-crimson-octopus-quincy

crimson/osd: assume an octopus+ osd peer

Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/osd: assume an octopus+ osd peer

no need to handle cases talking to pre-octopus osd peers in a post
quincy era.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/librbd: TestLibRBD.RenameViaLockOwner doesn't require journaling now

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #39604 from inspur-wyq/inspur-wyq-patch-1

doc/radosgw/: fix typos

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39572 from kotreshhr/version-auth-metadata-mgr

mgr/volumes: Bump up AuthMetadataManager's version

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge PR #39630 into master

* refs/pull/39630/head:
mgr/cephadm/upgrade: fix typo

Reviewed-by: Sebastian Wagner <swagner@suse.com>

qa: use --newest 100 to find a recent build

master and stable branches are only built a couple times a day now, so
we need to backtrack further.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

qa: deprioritize master and upgrade runs

Keep pacific at high for now. Since teuthology-dispatcher locks nodes
before starting jobs, the upgrade jobs no longer need high priority to
run on more machines than average.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

qa: schedule a wider variety of jobs

Use the day of the year instead of compressing everything into
quarters. This lets us get through more breadth of combinations in a
year.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

mgr/cephadm/upgrade: fix typo

Signed-off-by: Sage Weil <sage@newdream.net>

qa/crontab: adjust scheduled frequencies

smoke: 3x a week on master, weekly on other branches
pacific: every 2 days
master, octopus, nautilus, upgrades: weekly

This should be a net reduction in tests each week. Currently
we schedule far more than the lab can run in a given week.

Try to balance the scheduling over the course of the week.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #39138 into master

* refs/pull/39138/head:
qa: valgrind test for cephfs-mirror daemon
cephfs-mirror: use preforker for daemonizing
test: adjust sleep time to account for valgrind runs
cephfs-mirror: gracefully shutdown threads, timers, etc..
cephfs-mirror: call ceph_release() to cleanup mount alloc
cephfs-mirror: shutdown filesystem/cluster connections on shutdown
cephfs-mirror: set init failed flag on FSMirror::init() failure

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #39618 into master

* refs/pull/39618/head:
doc: fix broken links multimds and kcephfs

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/tasks/mgr/dashboard: add wear_level to devices schema

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #39620 from rzarzynski/wip-crimson-monc-failure-handling

crimson: multiple fixes for connection failure handling in MonClient

Reviewed-by: Kefu Chai <kchai@redhat.com>

rbd-mirror: reset update_status_task pointer in timer thread

To avoid a time window when m_update_status_task is invalid. If
during this time the cancel_update_mirror_image_replay_status is
called, it may cancel some other's ImageReplayer task, if it
happened to add the task with the same address.

Fixes: https://tracker.ceph.com/issues/49418
Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #39405 from rhcs-dashboard/cherrypy-security

mgr/dashboard: set security headers

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

ceph.spec.in: use lua53 for openSUSE 15.2

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

crimson/monc: drop the pending connection on ms_handle_reset().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/monc: don't assume reopen_session() shall establish active_con.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc: fix broken links multimds and kcephfs

Fixes: https://tracker.ceph.com/issues/49372
Signed-off-by: Jos Collin <jcollin@redhat.com>

mgr/diskprediction_local: wrap line longer than 100 chars

to appease flake8 and autopep8

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39432 from dillaman/wip-49238

rbd-mirror: delay update snapshot mirror image state

Reviewed-by: Mykola Golub <mgolub@suse.com>

mgr/orch: drop __all__ from __init__.py

`__all__` is only required if you want to modify the `form .. import *`
behavior. We don't need it.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: add DaemonDescriptionStatus

Signed-off-by: Daniel Pivonka <dpivonka@redhat.com>
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

librbd: use on-disk image name when storing mirror snapshot state

Fixes: https://tracker.ceph.com/issues/49115
Signed-off-by: Mykola Golub <mgolub@suse.com>

test/librbd: extend TestLibRBD.RenameViaLockOwner

To cover the following case:

- Client A has image opened but does not owns the lock.
- Client B renames the image (client A is not aware of it).
- Client A becomes the lock owner.
- Client B requests rename, which is proxied to the client A.

Signed-off-by: Mykola Golub <mgolub@suse.com>

mgr/influx: handle ValueError when setting options

Signed-off-by: Kefu Chai <kchai@redhat.com>