qa/tasks: add exception in do_thrash()

Fixes the bug during DaemonWatchdog testing:

Traceback (most recent call last):
  File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-daemonwatchdog-testing13/qa/tasks/daemonwatchdog.py", line 38, in _run
    self.watch()
  File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-daemonwatchdog-testing13/qa/tasks/daemonwatchdog.py", line 112, in watch
    if thrasher.e is not None:
AttributeError: MonitorThrasher instance has no attribute 'e'

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #28879 from Devp00l/wip-40580

mgr/dashboard: Fix the table mouseenter event handling test

Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #28916 from p-na/wip-pna-fix-tag

mgr/dashboard: Add linter for unclosed HTML tags

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Add missing closing tag for container

Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

Merge pull request #27463 from rjfd/wip-dashboard-ssl-cert

mgr/dashboard: commands to set SSL certificate and key

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #28213 from sebastian-philipp/orchestrator_cache

mgr/orchestrator: Add cache for Inventory and Services

Reviewed-by: Tim Serong <tserong@suse.com>

Merge pull request #28700 from batrick/i39620

doc/cephfs: improve add/remove MDS section

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #29033 from neha-ojha/wip-add-yt

doc/dev/developer_guide/index.rst: add youtube references

Reviewed-by: Noah Watkins <nwatkins@redhat.com>

Merge pull request #28649 from votdev/issue_40427

mgr/dashboard: Pool form uses different loading spinner

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

doc/mgr/dashboard: update SSL configuration instructions

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: added CLI commands to set SSL certificate and key

Fixes: https://tracker.ceph.com/issues/39123
Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: Pool form uses different loading spinner

Use the 'cd-loading-panel' element to display the loading progress.

Fixes: https://tracker.ceph.com/issues/40427
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #29043 from votdev/issue_40778

mgr/dashboard: Form fields do not show error messages/hints.

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge PR #28951 into master

* refs/pull/28951/head:
mds: apply configuration changes through MDSRank
common/TrackedOp: make settings atomic

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28964 from cbodley/wip-qa-workunit-s3tests

qa/rgw: update run-s3tests.sh

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

mgr/dashboard: Add linter for unclosed HTML tags

Fixes: http://tracker.ceph.com/issues/40686
Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

mgr/dashboard: Form fields do not show error messages/hints.

Fixes: https://tracker.ceph.com/issues/40778
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #28682 from cyx1231st/wip-seastar-msgr-socket2

crimson/net: implement preemptive shutdown/close

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #26104 into master

* refs/pull/26104/head:
client/Client: change dir's size delta to current write file size delta

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #28683 into master

* refs/pull/28683/head:
mon/MDSMonitor: use stringstream instead of dout for mds repaired

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #28710 into master

* refs/pull/28710/head:
cephfs-shell: Remove undefined variable files in do_rm()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #28831 into master

* refs/pull/28831/head:
client: support the fallocate() when fuse version >= 2.9

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #28906 into master

* refs/pull/28906/head:
cephfs-shell: Fix TypeError in poutput()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mds: apply configuration changes through MDSRank

This avoids the need to acquire the mds_lock prior to responding to config
changes. The MDSRank can create a finisher context that acquires the lock
later.

Fixes: https://tracker.ceph.com/issues/40694
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

common/TrackedOp: make settings atomic

To avoid locks on configuration changes.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc/cephfs: improve add/remove MDS section

Include hardware details and update language for modern tools.

Fixes: http://tracker.ceph.com/issues/39620
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc/dev/developer_guide/index.rst: add youtube references

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge PR #28978 into master

* refs/pull/28978/head:
qa/tasks/mgr/dashboard/test_health: fix test_full_health test
mon: use per-pool stats only when all OSDs are reporting
osd: report whether we have per-pool stats
osd/osd_types: osd_stat_t: include num_per_pool_osds member

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #29003 into master

* refs/pull/29003/head:
qa/standalone/scrub/osd-scrub-snaps.sh: adjust expected output
osd/SnapMapper: to_raw_key: use hobject pool member, not SnapMapper:pool
ceph-objectstore-tool: fix use of SnapSet::snaps
qa/standalone/scrub/osd-scrub-snaps: fix kv grep

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #29013 into master

* refs/pull/29013/head:
qa/tasks/ceph_manager: 5s -> 15s for 'osd out' to be visible

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #28998 from alfredodeza/wip-rm40664

ceph-volume  zap always skips block.db, leaves them around

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

mgr/dashboard: controllers/grafana is not Python3 compatible (#28956)

mgr/dashboard: controllers/grafana is not Python3 compatible

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #28979 from falcon78921/wip-doc-18197

doc: updated OpenStack rbd documentation

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/standalone/scrub/osd-scrub-snaps.sh: adjust expected output

SnapSet now dumps just seq, not a (fake) SnapContext.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/SnapMapper: to_raw_key: use hobject pool member, not SnapMapper:pool

Some callers (ceph-objectstore-tool) instantiate SnapMapper with a bogus
pool id (0) and want to use it to adjust mappings.  We have the hobj
available, so use its pool member instead.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-objectstore-tool: fix use of SnapSet::snaps

Instead, use clone_snaps to identify clones.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Minor fixes (#29009)

mgr/dashboard: Minor fixes

- Fix radio input display
- Fix the close button on initiators card
- Fix missing 'fa-' prefix in spinner icon
- Fix label font-weight

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge PR #28945 into master

* refs/pull/28945/head:
os/bluestore: StupidAllocator stucks on 4GB allocations
os/bluestore: avoid length overflow in extents returned by Stupid

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #28990 into master

* refs/pull/28990/head:
qa: vstart_runner fails because of string index out of range

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

qa/tasks/ceph_manager: 5s -> 15s for 'osd out' to be visible

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-volume tests: verify that wipefs tries several times

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.zap: retry wipefs several times to prevent race condition failures

Signed-off-by: Alfredo Deza <adeza@redhat.com>

qa/tasks/mgr/dashboard/test_health: fix test_full_health test

New fields!

Signed-off-by: Sage Weil <sage@redhat.com>

qa/standalone/scrub/osd-scrub-snaps: fix kv grep

SnapMapper keys are now SNA_, not MAP_.

Fixes: http://tracker.ceph.com/issues/40725
Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Remove button and badge radius (#29001)

mgr/dashboard: Remove button and badge radius

Reviewed-by: Patrick Seidensal <pnawracay@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

mgr/dashboard: Fix form validation problems in RGW user form (#29000)

mgr/dashboard: Fix form validation problems in RGW user form

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #28267 from flumm/ceph-volume-zap-fix

ceph-volume lvm.zap fix cleanup for db partitions

Reviewed-by: Alfredo Deza <adeza@redhat.com>

mgr/dashboard: Fix label font-weight

Some labels were not bold, because the css rule didn't apply to them.
I have improved the css rule and also add missing classes in some labels.

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Fix missing 'fa-' prefix in spinner icon

This was causing the submit button to not show the spinner.

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Add nolockdep option to e2e-script (#28981)

mgr/dashboard: Add nolockdep option to e2e-script

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

mgr/dashboard: Fix the close button on initiators card

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Fix radio input display

We were using the wrong class for the labels.

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #28957 from mgerstner/sfw2

rpm,etc/sysconfig: remove SuSEfirewall2 support

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

packaging: remove SuSEfirewall2 support

removal of SuSEfirewall2 service, since SuSEfirewall2 has been replaced
by firewalld, see [1].

[1]: https://lists.opensuse.org/opensuse-factory/2019-01/msg00490.html

Fixes: http://tracker.ceph.com/issues/40738
Signed-off-by: Matthias Gerstner <matthias.gerstner@suse.de>

mgr/deepsea: gracefully handle nonexistent nodes in {service,device} ls

Signed-off-by: Tim Serong <tserong@suse.com>

mgr/deepsea: rejig service cache

The previous implementation didn't work properly when there was more
than once service on a node (e.g. mgr and mon on the same node).

Signed-off-by: Tim Serong <tserong@suse.com>

mgr/orchestrator: fix some minor typos/kinks in inventory/service cache

Signed-off-by: Tim Serong <tserong@suse.com>

mgr/orchestrator: Impove type hint for describe_service

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: Introduce OutdatableDictMixin

No need for a persistent cache in DeepSea

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: Add cache for Inventory and Services

mgr/mgr_module: Added persistent dict
  basically a pythonic interface for a k-v store
mgr/orchestrator: Added common code to implement a cache
mgr/ssh: The cache works by manually adding and removing hosts.
  only the data is invalidated
mgr/deepsea: The cache invalidates all at once and only
  valid objects are cache

Fixes https://tracker.ceph.com/issues/39990

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #28802 from lixiaoy1/cmake_pmdk

cmake: pmem/pmdk changes to cmake

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28984 from tchaikov/wip-cmake-rebuild-cleanup

cmake: use BUILD_ALWAYS for rebuilding external project

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #29002 into master

* refs/pull/29002/head:
qa/tasks/ceph_deploy: do not rely on ceph-create-keys

Reviewed-by: Vasu Kulkarni <vasu@redhat.com>

Merge PR #28643 into master

* refs/pull/28643/head:
doc: removed old EOL Ceph versions from the list
doc: added openSUSE distros to the list and updated the releases
doc: updated the OS recommendations to reflect the Nautilus release

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc: updated OpenStack rbd documentation

Removed references to older OpenStack releases. Updated
documentation in regards to using QCOW2 as a virtual machine
disk type.

Fixes: https://tracker.ceph.com/issues/18197
Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>

Merge PR #28944 into master

* refs/pull/28944/head:
qa/workunits/rados/test_cache_pool.sh: s/forward/proxy/
qa/workunits/cephtool/test.sh: cache forward mode is no longer allowed
mon/OSDMonitor: forward -> proxy, readforward -> readproxy on octopus
mon/OSDMonitor: disallow FORWARD and READFORWARD cache modes
osd: do proxy when in forward/readforward cache modes

Reviewed-by: Neha Ojha <nojha@redhat.com>

mgr/dashboard: Fix form validation problems in RGW user form

ng-bootstrap-form-validation requires that every "form-control" has
an associated "formControlName".

Since these didn't had one, we had to either removed the "form-control" class
or rename it to "cd-form-control" (which will apply the same style without
triggering the validation).

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Remove button and badge radius

Fixes: http://tracker.ceph.com/issues/38953
Signed-off-by: Tiago Melo <tmelo@suse.com>

qa/tasks/ceph_deploy: do not rely on ceph-create-keys

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #28426 from mkogan1/wip-rgw-40183

rgw: save an unnecessary copy of RGWEnv

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge PR #28969 into master

* refs/pull/28969/head:
qa/tasks/ceph_manager: make is_{clean,recovered,active_or_down} less racy

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #28943 into master

* refs/pull/28943/head:
qa: stop testing on 16.04 xenial

Reviewed-by: Kefu Chai <kchai@redhat.com>

mon: use per-pool stats only when all OSDs are reporting

Previously, we would start using the per-pool stat sums as soon as *any*
OSDs were reporting.  For a legacy cluster, that meant that as soon as
one bluestore instance is updated or one new bluestore OSD is created, the
usage stats per pool would become useless.

Instead, only use the per-pool stats once *all* OSDs are reporting the new
values.  This mostly aligns with the health warning when one more bluestore
OSDs are not reporting; once they are update the warning goes away.  (That
does not factor in fielstore OSDs, though; all OSDs need to be new *and*
bluestore.)

Signed-off-by: Sage Weil <sage@redhat.com>

qa: vstart_runner fails because of string index out of range

Fix problem in LocalRemote::_perform_checks_and_return_list_of_args() when LocalCephManager::raw_cluster_cmd() is called with a empty command parameter.

Fixes: https://tracker.ceph.com/issues/40729
Signed-off-by: Volker Theile <vtheile@suse.com>

cmake: do not always build brotli or zstd

these two libraries are relatively stable, and Ceph developers are not
likely to hack them. so no need to rebuild them every time.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake/modules/BuildRocksDB.cmake: use BUILD_ALWAYS

this option was introduced by cmake 3.1, and we require cmake 3.5.1, so
we can use it without using the homebrew solution.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Add nolockdep option to e2e-script

Adding the --nolockdep option to the vstart command
seems to fix the current issue where vstart just
hangs at the dashboard ac-user-create command.
Hopefully we can get the e2e tests to run again
in Jenkins by applying this change.
Note that this might be a workaround only.

Fixes: https://tracker.ceph.com/issues/40688
Signed-off-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #28766 from liewegas/wip-clean-out-nibblewise

osd/osd_types: drop last_backfill_bitwise member

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28801 from ideepika/wip-update-dump-historic-ops

osd: add duration field to dump_historic_ops method

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>

mgr/dashboard: fix icons variable assignment (#28961)

mgr/dashboard: fix icons variable assignment

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Update bootstrap (#25188)

mgr/dashboard: Update bootstrap

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #28797 from simon-rock/simon_bug_fix

mgr/status: fix 'ceph osd status' ZeroDivisionError

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28687 from hsiang41/mgr/diskprediction_cloud_service_unavailable

mgr/diskprediction_cloud: Service unavailable

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #28950 from tchaikov/wip-corpus-gen

script,doc: add gen-corpus.sh

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc/dev/corpus.rst: updates to match gen-corpus.sh

* to reference the script to generate corpus
* use qa/workunits/rgw/run-s3tests.sh for producing rgw corpus

Signed-off-by: Kefu Chai <kchai@redhat.com>

script: add gen-corpus.sh

to automate generation of ceph-object-corpus objects. it facilitate the
process of cutting a new release

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28070 from badone/wip-lazy-omap-test

Wip lazy omap test

Reviewed-by: Greg Farnum gfarnum@redhat.com

qa/workunits/rados/test_cache_pool.sh: s/forward/proxy/

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/cephtool/test.sh: cache forward mode is no longer allowed

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: forward -> proxy, readforward -> readproxy on octopus

Adjust cache modes on upgrade.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: disallow FORWARD and READFORWARD cache modes

Signed-off-by: Sage Weil <sage@redhat.com>

osd: report whether we have per-pool stats

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: osd_stat_t: include num_per_pool_osds member

Count OSDs that are reporting per-pool stats.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #28949 from adamemerson/wip-40416

Don't crash on copy when metadata directive not supplied

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/tasks/ceph_manager: make is_{clean,recovered,active_or_down} less racy

Currently these can be thrown off if the cluster is creating or removing
pools at the same time.  Fix by taking a single snapshot of the pg stats
and based our judgement on that.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #28940 into master

* refs/pull/28940/head:
common/config: Remove unused code.

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/rgw: run-s3tests.sh runs in s3tests directory

something looks for request_decision_graph.yml in the working directory

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cmake: Add the support to use system PMDK library

Signed-off-by: Scott Peterson <scott.d.peterson@intel.com>
Signed-off-by: Xiaoyan Li <xiaoyan.li@intel.com>

cmake: move pmdk build script to Buildpmem.cmake

Signed-off-by: Scott Peterson <scott.d.peterson@intel.com>
Signed-off-by: Xiaoyan Li <xiaoyan.li@intel.com>

cmake: update HAVE_PMEM to HAVE_BLUESTORE_PMEM

Signed-off-by: Scott Peterson <scott.d.peterson@intel.com>
Signed-off-by: Xiaoyan Li <xiaoyan.li@intel.com>