cephadm: ceph-iscsi remove pool from cap

When we create a ceph-iscsi daemon/continer in cephadm we create a user
and set some caps. Turns out we were a little too restrictive.

We were locking down to only access the pool that was given in the spec,
which happens to be the pool the iscsi config is stored. But in reality
we need to be able  to attach any rbd images which could exist in other
pools.

So this patch removes the `pool=` from the osd cap, so from:

   osd = allow rwx pool={spec.pool}

To:

   osd = allow rwx

Fixes: https://tracker.ceph.com/issues/46138
Signed-off-by: Matthew Oliver <moliver@suse.com>

Merge pull request #35736 from ivancich/wip-orphan-list-timestamp-fix

rgw: orphan-list timestamp fix

Reviewed-by: Ali Maredia <amaredia@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge PR #35752 into master

* refs/pull/35752/head:
qa: fix flake8 error

Reviewed-by: Neha Ojha <nojha@redhat.com>

qa: fix flake8 error

    ./tasks/vstart_runner.py:33:1: F401 'io.BytesIO' imported but unused

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #35665 into master

* refs/pull/35665/head:
vstart_runner.py: improve a conditional in _do_run

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #33879 into master

* refs/pull/33879/head:
client: fix snap directory atime

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #35518 into master

* refs/pull/35518/head:
mds: fix filelock state when Fsc is issued

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #35522 into master

* refs/pull/35522/head:
vstart_runner: set default values of stdout and stderr to None

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #35532 into master

* refs/pull/35532/head:
Revert "Revert "qa/suites/rados/mgr/tasks/module_selftest: whitelist mgr client getting backlisted""

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge PR #35540 into master

* refs/pull/35540/head:
qa/cephfs: don't pass cmd args from run_as_user as str
qa/cephfs: refactor run_as_root() to user run_as_user()

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #35582 into master

* refs/pull/35582/head:
mds: reset heartbeat in EMetaBlob replay

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #35619 into master

* refs/pull/35619/head:
mds: reset sequence number when new rank0 mds is available
mds: dump mds name when processing mds map

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

rgw: orphan-list timestamp fix

When creating intermediate and output files, the rgw-orphan-list
script uses a timestamp using the `date` command. The hour was
inserted with "%k" but that padds with a space rather than a zero. So
that's changed to "%H".

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #35208 from prateek2211/refactor

rgw: Replace boost::string_ref/view with std::string_view

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #35331 from yuvalif/fix_issue_43536

rgw/pubsub: remove deprecated radogw-admin pubsub creation commands

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #35604 into master

* refs/pull/35604/head:
CODEOWNERS: added doc-writers team

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Alexandra Settle <asettle@suse.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #35739 into master

* refs/pull/35739/head:
mgr/volumes/nfs: Reorder imports

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35716 from rzarzynski/wip-bl-rval-claim_append

common, msgr/async: bufferlist::claim_append() doesn't require intermediaries anymore

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35643 from sebastian-philipp/cephadm-fix-already-in-use-msg

mgr/cephadm: fix broken "already in use" exception message

Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #35633 from matthewoliver/cephadm_bootstap_ipv6

cephadm: Set ms bind ipv6 when mon-ip is ipv6

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #35589 from jschmid1/osd_servicedescription_fix

mgr/cephadm: exclude osds without osdspec_affinity

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #35655 from batrick/i46081

cephadm: restrict mds caps to cephfs pools

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #35651 from sebastian-philipp/cephadm-unit-run-set-e

cephadm: unit.run: add `set -e`

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>

Merge pull request #35654 from neha-ojha/wip-44186

pybind/mgr/pg_autoscaler/module.py: do not update event if ev.pg_num== ev.pg_num_target

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #35710 from rhcs-dashboard/wip-46144-fix-sort-ui

mgr/dashboard: fix UI for Device class column in OSDs list

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #35742 from theanalyst/doc/releases/oct-update

doc: releases: updates for octopus

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #35721 from bk201/wip-43971

mgr/dashboard: fix Source column i18n issue in RBD configuration tables

Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #35734 from tspmelo/wip-fix-tooltip

mgr/dashboard: Fix display of tooltips inside tables

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

doc: releases: octopus: fixup changelog

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

doc: releases: update timeline for octopus

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

mgr/volumes/nfs: Reorder imports

This patch reorders import to fix "ImportError: cannot import name 'TimedOut'".
As flake8 requires similar import types to be grouped together.

Signed-off-by: Varsha Rao <varao@redhat.com>

CODEOWNERS: added doc-writers team

Signed-off-by: Lenz Grimmer <lgrimmer@suse.com>

Merge pull request #35473 from aclamk/fix-45903-bluefs-log-growth

os/bluestore: fix bluefs log growth.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

doc: update releases table

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

doc: releases: octopus: add v15.2.3 release notes

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

doc: releases: add v15.2.2 release notes

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #35572 from p-se/wip-pse-prometheus-mod-cache

mgr/prometheus: improve cache

Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #35155 from yuvalif/verify_wraparound_in_cls_q

cls/queue: add unit test to verify behaviour when queue wraps

Reviewed-by: Pritha Srivastava <prsivas@redhat.com>

Merge pull request #35674 from tchaikov/wip-qa-tox

qa,cmake: enable flake8 and import-tasks tests

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>
Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Varsha Rao <varao@redhat.com>

Merge pull request #35580 from xiexingguo/wip-46024

common, osd: add sanity checks around osd_scrub_max_preemptions

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #35662 from neha-ojha/wip-42820

mgr/DaemonServer.cc: make 'config show' on fsid work

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #35656 from ifed01/wip-ifed-fix-spillover-again

os/bluestore: fix spillover test case

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #35416 from yanghonggang/master

os/bluestore: add blkin traces to BlueStore

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #35689 from changchengx/options

common: use template shortcuts like is_unsigned_v

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35660 from mgfritch/orch-yaml-safe-load

mgr/orch: read untrusted input using the yaml SafeLoader

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Stephan Müller <smueller@suse.com>

qa/tasks/cephfs: flake8 fixes

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35621 from rzarzynski/wip-crimson-alien_memory

crimson: experimentally run AlienStore with Seastar's default allocator

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35727 from rzarzynski/wip-crimson-no-msgrv2.1

crimson/msgr: don't advertise the on-wire format v2.1.

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35719 from tchaikov/wip-cephadm-sans-el7

qa/suites/rados: do not test with el7

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #32855 from jdurgin/wip-message-cap

osd: make message cap option usable again

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Vikhyat Umrao <vikhyat@redhat.com>

mgr/dashboard: Fix display of tooltips inside tables

Hover event was not being propagated into the components inside the table.

Fixes: https://tracker.ceph.com/issues/45968
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #35732 from neha-ojha/wip-perf-distro

qa/suites/perf-basic: remove supported-all-distro

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

qa/suites/perf-basic: remove supported-all-distro

d4a04809fd7fd8aaf447005f76a1090db99d75c5 did not serve it's purpose, we
need to remove supported-all-distro to only run on ubuntu_latest

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #35678 from sebastian-philipp/cephadm-availabe-ssh-set

mgr/cephadm: make `orch status` fail, if ssh key is not set

Reviewed-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #35645 from sebastian-philipp/python-common-remove-old-cephadm-orch.py

python-common: remove old `cephadm_orchestrator.py`

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #35242 from bk201/wip-38234

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

qa/tasks/cephfs/test_nfs.py: flake8 fixes

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits/mon: flake8 fixes

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/standalone: flake8 fixes

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/vstart_runner: flake8 fixes

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/ragweed: flake8 fixes

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/mgr/test_orchestrator_cli: flake8 fixes

Signed-off-by: Kefu Chai <kchai@redhat.com>

tasks/mgr/dashboard: flake8 fixes

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks: flake8 fixes

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa: drop flake8-py2 test

as we've dropped py2 support, no need to run flake8 with python2
anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add TOX_ENVS keyword to test with listed test envs

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #35702 into master

* refs/pull/35702/head:
vstart: just use ganesha.nfsd that's in $PATH

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge PR #35724 into master

* refs/pull/35724/head:
vstart: Add ip value to nfs mount information

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #35646 into master

* refs/pull/35646/head:
qa/tasks/cephfs/nfs: Poll for max 60 seconds to ensure removal of ganesha services
pybind/mgr/volumes/nfs: Remove unused cephfs module
mgr/volumes/nfs: Ignore if ganesha disconnects

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

crimson/msgr: don't advertise the on-wire format v2.1.

We don't support it yet.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/os/alienstore: make the thread pool private.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson, common: validate the BlueStore's config for AlienStore.

This commit bases on the idea from Kefu Chai. Thanks!

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

vstart: Add ip value to nfs mount information

Signed-off-by: Varsha Rao <varao@redhat.com>

Merge pull request #35668 from bk201/wip-46105

mgr/dashboard: fix browserTarget for zh-Hans and zh-Hant

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

qa/suites/rados/thrash-old-clients: disable verify_ceph_hash

in this test, older ceph clients are installed on el7, but the ceph
cluster is deployed using cephadm, which in turn pulls ceph container
images built using the ceph being tested on el8.

since we've dropped the build of master on el7, there is no need to
verify if ceph package is available if cephadm is used for deploying the
cluster.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: fix Source column i18n issue in RBD configuration tables

Fixes: https://tracker.ceph.com/issues/43971
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

Merge pull request #35653 from ideepika/fix-cpatch

src/cstart: fix typo in cstart script

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Varsha Rao <varao@redhat.com>

common/Thread: configure Seastar's memory allocator.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

os/bluestore: configure Seastar's allocator for RocksDB's threads.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/os/alienstore: configure Seastar's memory allocator.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Revert "crimson/os: disable alienized bluestore if using seastar allocator"

This reverts commit db84151ed728f6951f891690e3bb704139e2ed9a.
It's not necessary as further commits will bring experimental
(and restricted) support for running AlienStore on top of the
default Seastar's allocator.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

cmake: clean-up the alienstore's CMakeLists.txt.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

cmake: restrict crush_objs in libcrimson-alienstore.a.

The problem this commit deals with is that alienized
set of some `src/common` has included all `crush_srcs`:

  ```
  set(crush_srcs
    builder.c
    mapper.c
    crush.c
    hash.c
    CrushWrapper.cc
    CrushCompiler.cc
    CrushTester.cc
    CrushLocation.cc)
  ```

while only `CrushLocation.cc` has been altered with
the `TOPNSPC` macro -- other files lack the namespace
differentiation. This leads to a conflict with
`libcrimson-common.a` which includes the same files:

  ```
  add_library(crimson-common STATIC
  ...
    ${PROJECT_SOURCE_DIR}/src/crush/builder.c
    ${PROJECT_SOURCE_DIR}/src/crush/mapper.c
    ${PROJECT_SOURCE_DIR}/src/crush/crush.c
    ${PROJECT_SOURCE_DIR}/src/crush/hash.c
    ${PROJECT_SOURCE_DIR}/src/crush/CrushWrapper.cc
    ${PROJECT_SOURCE_DIR}/src/crush/CrushCompiler.cc
    ${PROJECT_SOURCE_DIR}/src/crush/CrushTester.cc
    ${PROJECT_SOURCE_DIR}/src/crush/CrushLocation.cc
  ```

Another, potential problem is linking with a variant
built `WITH_ALIEN` setting.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

cmake: move buffer_seastar.cc into libcrimson-common.a.

Before the commit it was a part of libcrimson.a. However, this
assignment misses the dependency MonMac.cc has towards it:

  ```
  ../../../lib/libcrimson-common.a(MonMap.cc.o): In function `operator()':
  /work/ceph-2/src/mon/MonMap.cc:671: undefined reference to `ceph::buffer::v15_2_0::create(seastar::temporary_buffer<char>&&)'
  /work/ceph-2/src/mon/MonMap.cc:671: undefined reference to `ceph::buffer::v15_2_0::create(seastar::temporary_buffer<char>&&)'
  ```

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

qa/suites/rados: do not test with el7

since we stopped building master on el7, there is no need to test
cephadm with el7 anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35676 from ishanrai05/autocomplete-background

mgr/dashboard: fix autocomplete input backgrounds in chrome and firefox

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge PR #35627 into master

* refs/pull/35627/head:
qa: fix type error in stderr processing

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

msg/async: drop intermediary variables in frame_v2.cc.

They were required only because we lacked rvalue-taking
variant of `bufferlist::claim_append()`.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common/bl: introduce rvalue taking claim_append().

The motivation is to stop prohibiting callers from
directly passing temporaries. It was not infrequent
to see constructs like:

  ```cpp
  {
    // ...
    auto tmp = produce_bl();
    other_bl.claim_append(tmp);
  }
  ```

They were necessary because `claim_append(bufferlist&)`
requires lvalue reference to which an rvalue can't bind.

This patch fixes the problem by introducing `&&`-taking
variant of `claim_append()`. In the future it could
actually supersede the original one.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common/bl: deduplicate code between claim_append() and clear().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mgr/dashboard: fix UI for Device class column in OSDs list

Fixes: https://tracker.ceph.com/issues/46144
Signed-off-by: Avan Thakkar <athakkar@redhat.com>

Revert "ceph_osd: remove client message cap limit"

This reverts commit 45d5ac3ea040d6a7213b63d1c582e3a1bbaae8d4.

Without a msg throttler, we can't change osd_client_message_cap cap.
The throttler is designed to work with 0 as a max, so change the
default to 0 to disable it by default instead.

This doesn't affect the default behavior, it only lets us use this
option again.

Fixes: https://tracker.ceph.com/issues/46143
Conflicts:
src/ceph_osd.cc - new style of gconf() access

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #35635 from zdover23/wip-doc-man-deploy-mon-bootstrapping-2020-jun

doc/install: add sudo to command

Merge pull request #35595 from jan--f/c-v-fix-lvm-functests

ceph-volume: fix lvm functional tests

mgr/DaemonServer.cc: make 'config show' on fsid work

This follows 5661dd75efd48251b7f433b1ffe01c9c52906e96, which added special
handling for "config get" to work on fsid.

Fixes: https://tracker.ceph.com/issues/46123
Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #35027 from yuvalif/add_doxygen_target

cmake: add doxygen target for c++ code under src

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: fix autocomplete input backgrounds in chrome and firefox

Fixes: https://tracker.ceph.com/issues/46109
Signed-off-by: Ishan Rai <ishanrai05@gmail.com>

mgr/prometheus: enable mypy type checking for prometheus module

Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

Merge pull request #35688 from tchaikov/wip-doc-sphinx

admin/doc-requirements: pin sphinx-autodoc-typehints to 1.10.3

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

vstart: just use ganesha.nfsd that's in $PATH

In an earlier patch, I added a --ganesha-path option, but that left out
ganesha-rados-grace, which also could be in an alternate path. Instead,
just change vstart to run whatever ganesha.nfsd it finds in $PATH. We
already do that for ganesha-rados-grace so it should be fine to do that
for ganesha.nfsd as well.

Signed-off-by: Jeff Layton <jlayton@redhat.com>