ceph_argparse: increment matchcnt on kwargs

Currently when you pass a param in on the ceph cli as a kwarg
(--<param_name>) the matchcnt isn't incremented in the validate method
which is used to choose the right command signature.

This means currently things fail like:

  ceph orchestrator rgw rm --realm_name=default --zone_name=default
  no valid command found; 1 closest matches:
  orchestrator rgw rm <realm_name> <zone_name>
  Error EINVAL: invalid command

The '--realm_name' and '--zone_name' isn't counted to the matchcnt, so
'orchestrator rgw rm' isn't picked as the valid command.

This patch simply corrects this by incrementing matchcnt on the kwarg
validate path before calling shortcircuiting the loop.

Fixes: https://tracker.ceph.com/issues/43803
Signed-off-by: Matthew Oliver <moliver@suse.com>

Merge pull request #32531 from zdover23/wip-doc-landing-page-update

doc: Added the crisp getting started guide to index.rst

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #32977 into master

* refs/pull/32977/head:
qa/workunits/cephadm/test_cephadm.sh: add missing monitoring tests
cephadm: simplify Monitoring.components structure
cephadm: add proper tox type for monitoring components

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

Merge PR #33012 into master

* refs/pull/33012/head:
mgr/cephadm: prefix daemon ids with hostname
cephadm: bootstrap: name mgr with $hostname.$random

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #32155 from yuvalif/wip-yuval-multidelete-notifications

rgw/pubsub: send notifications from multi-delete op

mgr/cephadm: prefix daemon ids with hostname

This is friendlier to a human operator since they can immediately see
where an instance is located, as with the legacy scheme, while still
keeping the unique random suffix.  Use a . to separate so that we can
set per-host options.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32859 from liewegas/wip-cephadm-services

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #33030 from smithfarm/wip-luminous-12.2.13-release-notes

doc: release notes for Luminous v12.2.13

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #33031 from smithfarm/wip-nautilus-14.2.7-release-notes

doc: release notes for Nautilus 14.2.7

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #33038 from tchaikov/wip-librados-reinterpret-cast

librados: prefer reinterpret_cast over c-style cast

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #32933 from DimStar77/obs-mini

ceph.spec.in: Use pkgconfig() style BuildRequires for udev/libudev-devel

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32553 from tspmelo/wip-ts-strict

mgr/dashboard: Enable compiler options used by Angular --strict flag

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

librados: prefer reinterpret_cast over c-style cast

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33016 from tchaikov/wip-do-cmake

do_cmake.sh: bail out if something goes wrong

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

qa/workunits/cephadm/test_cephadm.sh: add missing monitoring tests

add tests for node-exporter, prometheus, and grafana

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: simplify Monitoring.components structure

by removing the nested image/metadata dict

Signed-off-by: Michael Fritch <mfritch@suse.com>

More commits to the landing page project.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

cephadm: bootstrap: name mgr with $hostname.$random

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32570 into master

* refs/pull/32570/head:
cephfs-shell: Add tests for setxattr, getxattr and listxattr
cephfs-shell: Add listxattr command
cephfs-shell: Add getxattr command
cephfs-shell: Add setxattr command
doc: Update about extended attributes

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31633 into master

* refs/pull/31633/head:
cephfs-shell: Instead of assert use stat for tests in rmdir
cephfs-shell: Add function for common rmdir test code
cephfs-shell: Add rmdir test for non empty directory
cephfs-shell: Add rmdir -p test for non empty directory
cephfs-shell: Add rmdir -p test for non existing dir
cephfs-shell: Add rmdir -p test to delete all dirs in given path
cephfs-shell: Add rmdir -p test for root directory with empty directories
cephfs-shell: Add rmdir test for valid file
cephfs-shell: Add rmdir test for invalid directory
cephfs-shell: Add rmdir test for valid directory
cephfs-shell: Fix rmdir '-p' issues

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32986 into master

* refs/pull/32986/head:
qa/tasks/ceph_manager: fix movement of cot exports with cephadm

Reviewed-by: Neha Ojha <nojha@redhat.com>

doc: release notes for Nautilus 14.2.7

Signed-off-by: Nathan Cutler <ncutler@suse.com>

doc: release notes for Luminous v12.2.13

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #32030 from vshankar/wip-mgr-volumes-clone

mgr/volumes: clone from snapshot

Merge PR #33014 into master

* refs/pull/33014/head:
mgr/dashboard: pass a list of drive_group to create_osds

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #32973 into master

* refs/pull/32973/head:
mgr/insights: make 'insights prune-health-history 0' zap current state

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #32987 into master

* refs/pull/32987/head:
qa/tasks/ceph_manager: make fix_pgp_num behave when no pool is found

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #32989 into master

* refs/pull/32989/head:
qa/tasks/ceph_manager: add --log-early to raw_cluster_cmd

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #32830 into master

* refs/pull/32830/head:
global: ensure CEPH_ARGS is decoded before early arg processing

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32806 into master

* refs/pull/32806/head:
common/bl: fix accessibility of bptr's _off and _len fields.
common/bl: drop get_raw() from the public buffer::ptr interface.
common: drop sharing of buffer::raw outside bufferlist.

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/tasks/ceph_manager: fix movement of cot exports with cephadm

I think this will finally work...

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33017 from linuxbox2/wip-rgw-beastc

rgw: improve beast

Merge pull request #33010 from tchaikov/wip-cmd-getval

src/: drop cct from cmd_getval()

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

do_cmake.sh: bail out if something goes wrong

so we can be aware if some updated submodule reference is missing in our
repo

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: pass a list of drive_group to create_osds

as orchestrator backends expect a list of drive_groups.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Unable to remove an iSCSI gateway that is alrea… (#32984)

mgr/dashboard: Unable to remove an iSCSI gateway that is already in use

Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Add Modified filter to Configuration table (#32923)

mgr/dashboard: Add Modified filter to Configuration table

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

rgw: improve beast

Avoid leaking connections that had partially-consumed
client data on unexpected disconnect.

Resolves CVE-2020-1700 (moderate impact flaw).

Fixes: https://tracker.ceph.com/issues/42531
Signed-off-by: Or Friedmann <ofriedma@redhat.com>
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

cephadm: add proper tox type for monitoring components

instead of `type: ignore`

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge branch 'octopus'

mgr/dashboard: Enable noImplicitAny TS compiler option

Fixes: https://tracker.ceph.com/issues/43525
Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Enable noImplicitReturns TS compiler option

Fixes: https://tracker.ceph.com/issues/43525
Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Enable noImplicitThis TS compiler option

Fixes: https://tracker.ceph.com/issues/43525
Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Enable noFallthroughCasesInSwitch TS compiler option

Fixes: https://tracker.ceph.com/issues/43525
Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Move TS compile rules into tsconfig.json

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #32999 from liewegas/fix-found

doc/foundation: fix amihan

mgr/cephadm: Don't duplicate cache entries in _create_daemon

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: test_cephadm: always set cluster fsid

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: less noisy pytest failue messages

Let's hide the `import pydevd` exception, as it doesn't provide any useful info

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

src/: drop cct from cmd_getval()

it's not used.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: add mgr/volumes clone commands

Fixes: http://tracker.ceph.com/issues/24880
Signed-off-by: Venky Shankar <vshankar@redhat.com>

test: add subvolume clone tests

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: allow force removal of incomplete failed clones

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: asynchronous cloner module

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: purge thread uses new async interface

This also makes `_cancel_jobs()` thread safe, which was not the
case earlier (with `_cancel_purge_job()`) -- this also makes the
code simpler by sharing the lock betweent two condition variables.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: fetch oldest clone entry

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: add clone specific commands

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: interface for fetching cloned subvolume status

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: add protect/unprotect and snap clone interface

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: handle transient subvolume states

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: interface for creating a cloned subvolume

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: get/set property for subvolume mode/uid/gid

This will be required when creating a clone as the clone would
inherit source subvolumes creation mode and uid/gid.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: module to track pending clone operations

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: add operation state machine table

... and fetch creation state from state machine table.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #33001 from tchaikov/wip-drop-dpdk-submodule

dpdk: drop dpdk submodule

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #32814 from mty22/au_ceph_branch_correct_docs

docs: Update au.ceph.com maintainers, update README.md.

Reviewed-by: Kefu Chai <kchai@redhat.com>

dpdk: drop dpdk submodule

we are using spdk/dpdk for async msgr's dpdk backend since 01a9f178. so
no need to keep dpdk submodule anymore.

Fixes: http://tracker.ceph.com/issues/24032
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #32947 into master

* refs/pull/32947/head:
OSD: Allow 64-char hostname to be added as the "host" in CRUSH

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #32974 into master

* refs/pull/32974/head:
mon: dump json from 'sessions' asok/tell command

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32976 into master

* refs/pull/32976/head:
cls_hello: fix typo

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #32981 into master

* refs/pull/32981/head:
mon: elector: return after triggering a new election

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #32988 into master

* refs/pull/32988/head:
qa/tasks/ceph: only re-request scrub on unscrubbed pgs

Reviewed-by: Neha Ojha <nojha@redhat.com>

doc/foundation: fix amihan

I misunderstood the status of their foundation membership.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32992 from ceph/wip-revert-dpdk

Revert "dpdk: drop dpdk submodule"

Revert "dpdk: drop dpdk submodule"

This reverts commit cb8087dfac31b8490fefdfca28d389b7b9901ef8.

The seastar submodule's .gitmodules links to `../dpdk` which is no longer present after removing dpdk from ceph.git's .gitmodules.

```
<dwfreed> the ceph/seastar repo uses awful URLs for the submodules
<dwfreed> and those awful URLs are the real reason it's failing
<dwfreed> dgalloway: ^^^
<dwfreed> seastar's .gitmodules references repos in the parent directory, so that when it's checked out as a submodule of ceph, you don't download the repos twice (and git will probably also use references instead of duplicating the local .git); however, ceph doesn't have a submodule for dpdk anymore
<dwfreed> so seastar's referencing a dpdk repo that doesn't exist
<dgalloway> i think i follow.  so you're suggesting revert https://github.com/ceph/ceph/commit/cb8087dfac31b8490fefdfca28d389b7b9901ef8 ?
<dwfreed> yep
<dwfreed> that'd be one way to fix it
...
<joshd> dgalloway: I'd suggest revert for now, and let the crimson folks figure out the longer term fix when they're back
```

Signed-off-by: David Galloway <dgallowa@redhat.com>

mgr/insights: make 'insights prune-health-history 0' zap current state

The self._health_slot value needs to be updated too if the current key
is removed.

Fixes: https://tracker.ceph.com/issues/43886
Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: fix style of placeholder * services entry

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: prime cached service state with newly created services

If we create a service, add the expected state to our cache, so that it
immediately appears in the get_services result.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: by default, return current service state

In the normal case, return whatever state we currently have.

In the serve loop, refresh if the state is old.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32958 into master

* refs/pull/32958/head:
qa/suites/rados/singleton/all/lost-unfound*: whitelist SLOW_OPS

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #32789 into master

* refs/pull/32789/head:
test: fix unused assert's variable in ceph_test_osd_stale_read.cc

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #32961 into master

* refs/pull/32961/head:
qa/standalone/osd/osd-bench: debug bluestore

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #32968 into master

* refs/pull/32968/head:
qa/suites/rados/verify: debug monc = 20

Reviewed-by: Neha Ojha <nojha@redhat.com>

qa/tasks/ceph_manager: add --log-early to raw_cluster_cmd

This is harmless if logging is low, but adds useful info when it is turned
up.

Hunting bug https://tracker.ceph.com/issues/43914

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph: only re-request scrub on unscrubbed pgs

If we haven't scrubbed everything, we occasinoally re-request scrub in case
the request was missed by the OSD (this can happen).  But we were
re-requesting scrub on ALL pgs, and if they are done in a
semi-deterministic order and are slow, then we may never get to the final
ones.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32771 from xxcs/xxcs07

rgw: fix multipart upload's error response

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-By: Casey Bodley <cbodley@redhat.com>

Merge PR #32878 into master

* refs/pull/32878/head:
cephadm: share code between 'pull' and 'inspect-image'
mgr/cephadm: upgrade: pull image after upgrade start, and for each host
cephadm: add inspect-image command

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #30969 into master

* refs/pull/30969/head:
mds : clean up data written to unsafe inodes

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32397 into master

* refs/pull/32397/head:
mds: Move StrayManager initializations to its header
mds: Remove extra spaces in StrayManager header.
mds: Reorganize structure members in StrayManager header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/tasks/ceph_manager: make fix_pgp_num behave when no pool is found

Fixes:

2020-01-30T04:41:24.697 INFO:tasks.thrashosds.thrasher:fixing pg num pool None
2020-01-30T04:41:24.698 INFO:tasks.thrashosds.thrasher:Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-sage-testing-2020-01-29-1034/qa/tasks/ceph_manager.py", line 1070, in wrapper
    return func(self)
  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-sage-testing-2020-01-29-1034/qa/tasks/ceph_manager.py", line 1200, in _do_thrash
    self.choose_action()()
  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-sage-testing-2020-01-29-1034/qa/tasks/ceph_manager.py", line 768, in fix_pgp_num
    if self.ceph_manager.set_pool_pgpnum(pool, force):
  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-sage-testing-2020-01-29-1034/qa/tasks/ceph_manager.py", line 2088, in set_pool_pgpnum
    assert isinstance(pool_name, six.string_types)
AssertionError

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32969 into master

* refs/pull/32969/head:
qa/suites/rados/cephadm: explicitly test many distros

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge PR #32854 into master

* refs/pull/32854/head:
qa: fix testing kernel branch link

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32801 into master

* refs/pull/32801/head:
cephfs-shell: Fix ls -l

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

mgr/dashboard: Change project name to "Ceph Dashboard" (#32959)

mgr/dashboard: Change project name to "Ceph Dashboard"

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Patrick Seidensal <pnawracay@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Unable to remove an iSCSI gateway that is already in use

Fixes: https://tracker.ceph.com/issues/43900
Signed-off-by: Ricardo Marques <rimarques@suse.com>

mgr/cephadm: tolerate unknown service state

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: periodically refresh services in the background

Pass them into check_for_strays to avoid double-work.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: periodically check hosts, and warn if the host check fails

If we manually run the check, wake up the serve thread to recheck things.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: remove notify hook

We don't want to wake up serve so often.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: fix 'cephadm check-host' error output

Signed-off-by: Sage Weil <sage@redhat.com>