Merge pull request #35416 from yanghonggang/master

os/bluestore: add blkin traces to BlueStore

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #35689 from changchengx/options

common: use template shortcuts like is_unsigned_v

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35660 from mgfritch/orch-yaml-safe-load

mgr/orch: read untrusted input using the yaml SafeLoader

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #35621 from rzarzynski/wip-crimson-alien_memory

crimson: experimentally run AlienStore with Seastar's default allocator

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35727 from rzarzynski/wip-crimson-no-msgrv2.1

crimson/msgr: don't advertise the on-wire format v2.1.

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35719 from tchaikov/wip-cephadm-sans-el7

qa/suites/rados: do not test with el7

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #32855 from jdurgin/wip-message-cap

osd: make message cap option usable again

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Vikhyat Umrao <vikhyat@redhat.com>

Merge pull request #35732 from neha-ojha/wip-perf-distro

qa/suites/perf-basic: remove supported-all-distro

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

qa/suites/perf-basic: remove supported-all-distro

d4a04809fd7fd8aaf447005f76a1090db99d75c5 did not serve it's purpose, we
need to remove supported-all-distro to only run on ubuntu_latest

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #35678 from sebastian-philipp/cephadm-availabe-ssh-set

mgr/cephadm: make `orch status` fail, if ssh key is not set

Reviewed-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #35645 from sebastian-philipp/python-common-remove-old-cephadm-orch.py

python-common: remove old `cephadm_orchestrator.py`

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #35242 from bk201/wip-38234

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge PR #35702 into master

* refs/pull/35702/head:
vstart: just use ganesha.nfsd that's in $PATH

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge PR #35724 into master

* refs/pull/35724/head:
vstart: Add ip value to nfs mount information

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #35646 into master

* refs/pull/35646/head:
qa/tasks/cephfs/nfs: Poll for max 60 seconds to ensure removal of ganesha services
pybind/mgr/volumes/nfs: Remove unused cephfs module
mgr/volumes/nfs: Ignore if ganesha disconnects

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

crimson/msgr: don't advertise the on-wire format v2.1.

We don't support it yet.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/os/alienstore: make the thread pool private.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson, common: validate the BlueStore's config for AlienStore.

This commit bases on the idea from Kefu Chai. Thanks!

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

vstart: Add ip value to nfs mount information

Signed-off-by: Varsha Rao <varao@redhat.com>

Merge pull request #35668 from bk201/wip-46105

mgr/dashboard: fix browserTarget for zh-Hans and zh-Hant

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

qa/suites/rados/thrash-old-clients: disable verify_ceph_hash

in this test, older ceph clients are installed on el7, but the ceph
cluster is deployed using cephadm, which in turn pulls ceph container
images built using the ceph being tested on el8.

since we've dropped the build of master on el7, there is no need to
verify if ceph package is available if cephadm is used for deploying the
cluster.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35653 from ideepika/fix-cpatch

src/cstart: fix typo in cstart script

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Varsha Rao <varao@redhat.com>

common/Thread: configure Seastar's memory allocator.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

os/bluestore: configure Seastar's allocator for RocksDB's threads.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/os/alienstore: configure Seastar's memory allocator.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Revert "crimson/os: disable alienized bluestore if using seastar allocator"

This reverts commit db84151ed728f6951f891690e3bb704139e2ed9a.
It's not necessary as further commits will bring experimental
(and restricted) support for running AlienStore on top of the
default Seastar's allocator.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

cmake: clean-up the alienstore's CMakeLists.txt.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

cmake: restrict crush_objs in libcrimson-alienstore.a.

The problem this commit deals with is that alienized
set of some `src/common` has included all `crush_srcs`:

  ```
  set(crush_srcs
    builder.c
    mapper.c
    crush.c
    hash.c
    CrushWrapper.cc
    CrushCompiler.cc
    CrushTester.cc
    CrushLocation.cc)
  ```

while only `CrushLocation.cc` has been altered with
the `TOPNSPC` macro -- other files lack the namespace
differentiation. This leads to a conflict with
`libcrimson-common.a` which includes the same files:

  ```
  add_library(crimson-common STATIC
  ...
    ${PROJECT_SOURCE_DIR}/src/crush/builder.c
    ${PROJECT_SOURCE_DIR}/src/crush/mapper.c
    ${PROJECT_SOURCE_DIR}/src/crush/crush.c
    ${PROJECT_SOURCE_DIR}/src/crush/hash.c
    ${PROJECT_SOURCE_DIR}/src/crush/CrushWrapper.cc
    ${PROJECT_SOURCE_DIR}/src/crush/CrushCompiler.cc
    ${PROJECT_SOURCE_DIR}/src/crush/CrushTester.cc
    ${PROJECT_SOURCE_DIR}/src/crush/CrushLocation.cc
  ```

Another, potential problem is linking with a variant
built `WITH_ALIEN` setting.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

cmake: move buffer_seastar.cc into libcrimson-common.a.

Before the commit it was a part of libcrimson.a. However, this
assignment misses the dependency MonMac.cc has towards it:

  ```
  ../../../lib/libcrimson-common.a(MonMap.cc.o): In function `operator()':
  /work/ceph-2/src/mon/MonMap.cc:671: undefined reference to `ceph::buffer::v15_2_0::create(seastar::temporary_buffer<char>&&)'
  /work/ceph-2/src/mon/MonMap.cc:671: undefined reference to `ceph::buffer::v15_2_0::create(seastar::temporary_buffer<char>&&)'
  ```

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

qa/suites/rados: do not test with el7

since we stopped building master on el7, there is no need to test
cephadm with el7 anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35676 from ishanrai05/autocomplete-background

mgr/dashboard: fix autocomplete input backgrounds in chrome and firefox

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge PR #35627 into master

* refs/pull/35627/head:
qa: fix type error in stderr processing

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Revert "ceph_osd: remove client message cap limit"

This reverts commit 45d5ac3ea040d6a7213b63d1c582e3a1bbaae8d4.

Without a msg throttler, we can't change osd_client_message_cap cap.
The throttler is designed to work with 0 as a max, so change the
default to 0 to disable it by default instead.

This doesn't affect the default behavior, it only lets us use this
option again.

Fixes: https://tracker.ceph.com/issues/46143
Conflicts:
src/ceph_osd.cc - new style of gconf() access

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #35635 from zdover23/wip-doc-man-deploy-mon-bootstrapping-2020-jun

doc/install: add sudo to command

Merge pull request #35595 from jan--f/c-v-fix-lvm-functests

ceph-volume: fix lvm functional tests

Merge pull request #35027 from yuvalif/add_doxygen_target

cmake: add doxygen target for c++ code under src

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: fix autocomplete input backgrounds in chrome and firefox

Fixes: https://tracker.ceph.com/issues/46109
Signed-off-by: Ishan Rai <ishanrai05@gmail.com>

Merge pull request #35688 from tchaikov/wip-doc-sphinx

admin/doc-requirements: pin sphinx-autodoc-typehints to 1.10.3

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

vstart: just use ganesha.nfsd that's in $PATH

In an earlier patch, I added a --ganesha-path option, but that left out
ganesha-rados-grace, which also could be in an alternate path. Instead,
just change vstart to run whatever ganesha.nfsd it finds in $PATH. We
already do that for ganesha-rados-grace so it should be fine to do that
for ganesha.nfsd as well.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #35698 from tchaikov/wip-man-print-file-which-is-non-ascii

man/conf.py: print out path to file which contains undecodable chars

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #35701 from tchaikov/wip-doc-sans-utf8

doc/man/8/ceph-objectstore-tool.rst: drop UTF-8 chars

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/man/8/ceph-objectstore-tool.rst: drop UTF-8 chars

it's a regression introduced by 5a09d7a1ca2e93811505345c31367627bedbea0e

Signed-off-by: Kefu Chai <kchai@redhat.com>

man/conf.py: print out path to file which contains undecodable chars

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35640 from sebastian-philipp/doc-cephadm-adopt-placment

doc/cephadm: adopt: fix misleading `apply` examples

Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #35587 from sebastian-philipp/doc-orch-stateless-cli

doc/mgr/orch: Update documentation to the new CLI

Reviewed-by: Zac Dover <zac.dover@gmail.com>

ceph-volume: fix lvm functional tests

This is related to https://github.com/ceph/ceph-ansible/pull/5413 and
adjusts the tests to work with the related ansible fix.

Fixes: https://tracker.ceph.com/issues/46131
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

qa/tasks/cephfs/nfs: Poll for max 60 seconds to ensure removal of ganesha services

Fixes: https://tracker.ceph.com/issues/46104
Signed-off-by: Varsha Rao <varao@redhat.com>

pybind/mgr/volumes/nfs: Remove unused cephfs module

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/volumes/nfs: Ignore if ganesha disconnects

Fixes: https://tracker.ceph.com/issues/46046
Signed-off-by: Varsha Rao <varao@redhat.com>

Merge pull request #35675 from smithfarm/wip-doc-backport-labelling

doc/SubmittingPatches-backports.rst: clarify backport PR labels

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc/install: add sudo to command

This commit adds "sudo" to a ceph-authtool command
that is missing the "sudo".

Fixes: https://tracker.ceph.com/issues/46065
Signed-off-by: Zac Dover <zac.dover@gmail.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35691 from bk201/wip-46130

doc/man: fix ceph-objectstore-tool description

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35682 from tchaikov/wip-cmake-cxx

cmake: only apply C++ compile options to C++ language

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc/man: fix ceph-objectstore-tool description

There should be a `--` between the name and the description.

Fixes: https://tracker.ceph.com/issues/46130
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

doc/conf.py: s/add_javascript/add_js_file/

to address following warning:

jenkins-build/build/workspace/ceph-pr-docs/doc/conf.py:102: RemovedInSphinx40Warning: The app.add_javascript() is deprecated. Please use app.add_js_file() instead.

Signed-off-by: Kefu Chai <kchai@redhat.com>

admin/doc-requirements: pin sphinx-autodoc-typehints to 1.10.3

to silence following error:

ERROR: sphinx-autodoc-typehints 1.11.0 has requirement Sphinx>=3.0, but you'll have sphinx 2.4.3 which is incompatible.

* update breathe to the latest release, 4.14.2, which supports sphinx <
  3.0
* pin sphinx to 2.4.4 the last sphinx before 3.0.0. as if sphinx >= 3.0
is used, we have errors like:

Invalid C declaration: Expected identifier in nested name, got keyword: int [error at 18]
  CEPH_RADOS_API int rados_aio_append (rados_ioctx_t io, const char *oid, rados_completion_t completion, const char *buf, size_t len)
  ------------------^

Signed-off-by: Kefu Chai <kchai@redhat.com>

common: short variable template with suffix_v/_t

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

common: typo fix for readability

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

Merge pull request #35658 from dillaman/wip-46083

librbd: race when disabling object map with overlapping in-flight writes

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #35588 from jschmid1/osdspec_affinity_to_metadata

osd: expose osdspec_affinity to osd_metadata

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #35569 from sungjunyoung/master

install_deps.sh: Add debian based elementary os

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35294 from agayev/punch-hole

os/bluestore: Add missing punch_hole call so that object offsets are …

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #34189 from changchengx/bufferlist

common/buffer: refine bufferlist implementation

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35592 from tchaikov/wip-ceph-in-de2

ceph.in: fallback to install path of "ceph" when "ceph-conf" is not found

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #35078 from idryomov/wip-msgr21

msg/async: New msgr2 crc and secure modes (msgr2.1)

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #35607 from wuxingyi/schedulerlogger

osd/scheduler: add white space for better readability

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #35584 from aclamk/wip-mempool-bufferlist-c_str

common/mempool: copies bufferlist assignment to mempool through rebuild

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #32271 from toliu/disable-restful-basic-auth

mgr/restful: allow users to disable http authentication

Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

librbd: race when disabling object map with overlapping in-flight writes

The block guard that protects against overlapping updates to the object
map needs to be flushed prior to closing the object map instance.

Fixes: https://tracker.ceph.com/issues/46083
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #33823 from zdover23/wip-doc-ceph-objectstore-tool-2020-mar

doc/man: add manpage for ceph-objectstore-tool

cmake: only apply C++ compile options to C++ language

we have sources written in C language, and they are compiled along with
C++ source files which uses Seastar library. so when the C compiler sees
the options exposed by seastar, it complains like

1: warning: command line option '-fconcepts' is valid for C++/ObjC++ but
not for C

with this change, only C++ options is added for C++ language.

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: add blkin traces to BlueStore

Signed-off-by: Yang Honggang <yanghonggang@kuaishou.com>

Merge pull request #35681 from yuriw/wip-yuriw-crontab-master

qa/tests: removed mimic runs as EOL

Reviewed-by: Neha Ojha <nojha@redhat.com>

qa/tests: removed mimic runs as EOL

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #35673 from tspmelo/wip-chromium

mgr/dashboard: Fix e2e chromium binary validation

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

mgr/cephadm: make `orch status` fail, if ssh key is not set

ceph-salt relies on the output of `orch status` to determine the
end of the bootstrap process.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: Fix e2e chrome and chromium binaries verification

Fixes: https://tracker.ceph.com/issues/46110
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #35223 from smithfarm/wip-releases-ordering

doc/releases: list archived releases in correct order

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

doc/SubmittingPatches-backports.rst: clarify backport PR labels

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #35586 from tspmelo/wip-service-root

mgr/dashboard: Provide all services in root

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #30670 from trociny/wip-41736

mgr: fix race between module load and notify

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Tim Serong <tserong@suse.com>

Merge pull request #35141 from matthewoliver/cephadm_iscsi_lib_module_mount

cephadm: Give iscsci a RO /lib/modules bind mounted

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #35190 from cybozu/ceph-volume-fix-rejected-reason-in-device-type-mismatch

ceph-volume: report correct rejected reason in inventory if device type is invalid

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

mgr/dashboard: fix browserTarget for zh-Hans and zh-Hant

Fixes: https://tracker.ceph.com/issues/46105
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

ceph-volume: show correct rejected reason in inventory if device type is not acceptable

If device type is not acceptable in `c-v inventory`, its rejected reason
becomes "Insufficient space (<5GB)" by mistake. It's because sys_api is
empty due to skipping devices that are neither `disk` nor `device`. We
should report the target device is not acceptable in this case.

Fixes: https://tracker.ceph.com/issues/46102
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>

ceph-volume: cleanup code

Simplify the logic and fix a typo.

Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>

incorporating dzafman's suggestions

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #35652 from rzarzynski/wip-crimson-thread-cleanup

crimson: clean-up the crimson/thread directory

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35632 from dzafman/wip-46064

tools: Add statfs operation to ceph-objecstore-tool

Reviewed-by: Neha Ojha <nojha@redhat.com>

mgr/orch: read untrusted input using the yaml SafeLoader

https://msg.pyyaml.org/load

Signed-off-by: Michael Fritch <mfritch@suse.com>

tools: Add statfs operation to ceph-objecstore-tool

Fixes: https://tracker.ceph.com/issues/46064
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #35329 from trociny/wip-quiesce_complete

librbd: quiesce/unquiesce API improvements

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #35649 into master

* refs/pull/35649/head:
vstart: allow specifying a different ganesha binary
vstart: handle multiple ganesha's more sanely

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
Reviewed-by: Varsha Rao <varao@redhat.com>

vstart: allow specifying a different ganesha binary

Signed-off-by: Jeff Layton <jlayton@redhat.com>

vstart: handle multiple ganesha's more sanely

Currently, if you have more than one ganesha daemon, it clobbers the
log, config and pidfiles from the previous daemons. Give them each
their own files.

While we're in here, move the log and pidfile to the out/ directory
with all of the other logfiles/pidfiles.

Fixes: https://tracker.ceph.com/issues/46079
Signed-off-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #35650 from trociny/wip-46076

test/rbd_mirror: fix race on test shut down

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #35631 from tchaikov/wip-cmake-cleanup

cmake: use add_compile_options() and add_definitions()

Reviewed-by: Casey Bodley <cbodley@redhat.com>

src/cstart: fix typo in cstart script

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #35630 from tchaikov/wip-systemd-subvars

cmake: move include(GNUInstallDirs) up

Reviewed-by: Neal Gompa <ngompa13@gmail.com>
Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson: move Throttle to src/crimson/common.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>