script/run-cbt.sh: set fs.aio-max-nr for seastar

seastar requires 11027 for each reactor, so we need at least

fs.aio-max-nr = 11027 * n_osd

but to make it simple and to take the needs of other applications using
aio on the machine, we increase the number of io context to 32768,
and always set this setting no matter if we are starting a vstart
cluster with classic osd or crimson osd

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31523 from sebastian-philipp/ssh-make-check

mgr/ssh: add make check integration

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>

Merge PR #31509 into master

* refs/pull/31509/head:
client: remove unused variable

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

client: remove unused variable

the flock and cond are useless variables in Client::_read_sync,remove useless variables

Fixes: https://tracker.ceph.com/issues/42720
Signed-off-by: su_nan@inspur.com <su_nan@inspur.com>

Merge pull request #31643 from liewegas/fix-empty-crash

mgr/crash: behave when posted crash has no backtrace

Merge PR #31626 into master

* refs/pull/31626/head:
test_cephfs_shell: initialize stderr for run_cephfs_shell_cmd()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31460 into master

* refs/pull/31460/head:
cephfs-shell: print helpful message when conf file is not found

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #30385 from ZVampirEM77/acl-detailed-err-msg

rgw: add detailed error message for PutACLs

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30470 from dongbula/bf-fix-usage-keyiter-update

RGW: fix an endless loop error when to show usage

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30574 from zhangsw/rgw-fix-compress

rgw: compression info should be same during multipart uploading

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30575 from mkogan1/wip-rgw-resharding-mem

rgw: when resharding store progress json

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
Reviewed-by: Yuval Lifshitz <yuvalif@yahoo.com>

Merge pull request #31632 from tspmelo/wip-rbd-timezone

mgr/dashboard: Fix error in unit test caused by timezone

Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #29004 from zhangsw/rgw-cleanup-taggingop

rgw: remove re-defined is_tagging_op in RGWHandler_REST_Bucket_S3

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #31582 into master

* refs/pull/31582/head:
global: disable THP for Ceph daemons

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge PR #31584 into master

* refs/pull/31584/head:
common/options: osd_crush_chooseleaf_type is CLUSTER_CREATE
mon/ConfigMonitor: do not assimilate CLUSTER_CREATE options
common/ceph_context: observe container_image so we don't get warnings

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #31621 into master

* refs/pull/31621/head:
os/bluestore: consolidate extents from the same device only

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/crash: behave when posted crash has no backtrace

This is the case when the qa/tasks/mgr/test_crash.py test runs, and could
happen in the real world too, perhaps, if someone fed us wonky input.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: add make check integration

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #31436 from dang/wip-dang-zipper-3

rgw: Project Zipper - Bucket

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #30754 into master

* refs/pull/30754/head:
doc/cephfs: merge fstab doc with respective mount docs
doc: add systemd unit part for FUSE mounts in fstab doc
doc: update and improve "mount using kernel driver" doc
doc: update and improve "mount using FUSE" doc

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31631 into master

* refs/pull/31631/head:
ceph-daemon: Add more type hints

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #31561 from sebastian-philipp/orchestrator-remove-cli-tox

mgr/orchestrator_cli: remove tox and move test to parent dir

Reviewed-by: Stephan Müller <smueller@suse.com>

mgr/orchestrator: remove tox and move test to parent dir

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: Fix error in unit test caused by timezone

Unit test was only accepting positive timezones

Fixes: https://tracker.ceph.com/issues/42812
Signed-off-by: Tiago Melo <tmelo@suse.com>

ceph-daemon: Add more type hints

Commit 7ce26be7a3448 started to add type hints. This commit continues
that and adds the missing type hints.

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>

test_cephfs_shell: initialize stderr for run_cephfs_shell_cmd()

Since teuthology initializes stderr to None by default, absence of this
breaks the tests accessing stderr of commands executed within the test
when the execution is using teuthology.

Fixes: https://tracker.ceph.com/issues/42806
Signed-off-by: Rishabh Dave <ridave@redhat.com>

doc/cephfs: merge fstab doc with respective mount docs

Fixes: https://tracker.ceph.com/issues/42601
Signed-off-by: Rishabh Dave <ridave@redhat.com>

global: disable THP for Ceph daemons

Ceph is known to suffer from memory fragmentation due to transparent
huge pages (THP). This is indicated by RSS usage above configured memory
targets and is only observable when the distribution default for THP is
"always", which is the default in the upstream kernel if
CONFIG_TRANSPARENT_HUGEPAGE_MADVISE is unset.

For now, enabling THP is currently discouraged until selective use of
THP by Ceph is implemented via madvise. We will need to consider both
defaults for THP so madvise calls to both enable and disable THP will
need implemented.

All credit to Mark Nelson for doing the legwork identifying this issue
and potential solutions.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
Signed-off-by: Mark Nelson <mnelson@redhat.com>

Merge PR #31614 into master

* refs/pull/31614/head:
ceph-daemon: use `-e` instead of `--env`

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: consolidate extents from the same device only

Presumably Fixes: https://tracker.ceph.com/issues/42223

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #31613 into master

* refs/pull/31613/head:
mgr/ssh: fix redeploy

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #31609 into master

* refs/pull/31609/head:
ceph-daemon: Add basic mypy support

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #31601 from tchaikov/wip/install-newer-binutils-on-xenial

install-deps.sh: install binutils 2.28 for xenial

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge PR #31602 into master

* refs/pull/31602/head:
mgr/ssh: annotate object representation

Reviewed-by: Sebastian Wagner <swagner@suse.com>

ceph-daemon: use `-e` instead of `--env`

Older versions of podman don't allow intermixing `-e` and `--env`

Signed-off-by: Michael Fritch <mfritch@suse.com>

install-deps.sh: remove the cleanup for kitware repo

it was used for removing the kitware repo, and in
7265b55d094a639be50a567d3be92fba94c04786, we switched to a rebuilt
version hosted in chacra, and it was more than two months ago,
presumably, none builder has kitware repo anymore. so, let's remove
the cleanup code.

Signed-off-by: Kefu Chai <kchai@redhat.com>

install-deps.sh: install binutils 2.28 for xenial

ld 2.26 segment faults when linking librados.so in
https://github.com/ceph/ceph/pull/16715 .

while ld 2.28 always survives.
binutils 2.28 is rebuilt and uploaded to chacra
binutils/master/7fa393306ed8b93019d225548474c0540b8928f7/ubuntu/xenial,

so let's use it instead.

the source deb package comes from
http://ppa.launchpad.net/jonathonf/binutils/ubuntu

i tried to use the binutils 2.30 source package from bionic,
but it has build dependency of dpkg-dev (>= 1.19.0.5) which cannot be
fulfilled on xenial. so without more changes, we cannot get
binutils 2.30 built on xenial using this source package.

Fixes: https://tracker.ceph.com/issues/42596
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31592 from jan--f/c-v-fix-mimic-functional-tests

ceph-volume: assume msgrV1 for all branches containing mimic

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge PR #31603 into master

* refs/pull/31603/head:
ceph-daemon: fix prompt

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Thomas Bechtold <tbechtold@suse.com>

ceph-daemon: Add basic mypy support

Add mypy support and some type hints to be able to do static code
analysis.

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>

mgr/ssh: fix redeploy

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Add E2E tests for tasks and notifications sideb… (#31128)

mgr/dashboard: Add E2E tests for tasks and notifications sidebar

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

ceph-daemon: fix prompt

[ceph: root@gnit /]#
not
[ceph: root@gnit /]$

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: annotate object representation

* InventoryNode
* ServiceDescription

Signed-off-by: Joshua Schmid <jschmid@suse.de>

Merge PR #31583 into master

* refs/pull/31583/head:
ceph-daemon: -v|--verbose, not -d|--debug

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #31585 into master

* refs/pull/31585/head:
ceph-daemon: add `--legacy-dir` arg to `ls` command

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #31533 from tspmelo/wip-bc-redmine-key

backport-create-issue: read redmine key from file

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #31544 from ifed01/wip-ifed-gettid2

test/fio: bump to fio-3.15

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-volume: assume msgrV1 for all branches containing mimic

With nautilus and newer OSDs listen on v1 ports and v2 ports. Assume
that if mimic (or luminous) occur in the branch name, the OSDs are
running msgrv1 only.

Fixes: https://tracker.ceph.com/issues/42791
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #31426 from cyx1231st/rfc-seastar-fix-build-dpdk

crimson: fix build seastar with dpdk

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30662 from sebastian-philipp/orchestrator-c-v-inventory

python-common, mgr/orchestrator, mgr/dashboard: Use common "Devices"

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

crimson: build seastar dpdk from src/seastar/dpdk

src/spdk/dpdk and src/seastar/dpdk are both at their private branches
with project-specific modifications, so select proper dpdk source
directory according to flags WITH_SPDK and Seastar_DPDK.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson: link required dpdk components for seastar

see https://github.com/scylladb/seastar/blob/
809b3ef875e7edb7c56643812dc8347ff9a82cee/cmake/Finddpdk.cmake#L60-L85

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

doc: add systemd unit part for FUSE mounts in fstab doc

To make FUSE-mounted CephFS persist across reboots, user also needs to
start and enable the systemd units. Add that part to the document for
fstab, instead of mentioning it in "Mount CephFS using FUSE" doc. Also,
wrap few lines and rename mountpoint to /mnt/mycephfs in examples to
keep them same across docs.

Fixes: https://tracker.ceph.com/issues/42298
Signed-off-by: Rishabh Dave <ridave@redhat.com>

doc: update and improve "mount using kernel driver" doc

Move the examples of mount command for Ceph cluster with CephX enabled
to the top of page, since it is enabled by default, improve explanation
around Ceph with multiple FSs, get rid of hash symbols before every
command (without them it's clear that the text is command and with them
the reader cannot use the commands directly), link fstab page, add how
mount in general looks, add prerequisites required for kernel mounts and
expand explanation wherever possible.

Fixes: https://tracker.ceph.com/issues/42220
Signed-off-by: Rishabh Dave <ridave@redhat.com>

doc: update and improve "mount using FUSE" doc

Recommend keyring permission to be 600 instead of 644, show examples
for `-k`, `-r`, `-m` and `--client_mds_namespace` options, move
references to the bottom of the page,  show how to unmount FUSE-mounted
CephFS, copy the tip about unmounting from "mount using kernel" page to
"mount using FUSE" page, correct commands for automating FUSE mounts,
add sub-headings to the document and add how ceph-fuse command looks in
general.

Fixes: https://tracker.ceph.com/issues/42205
Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #31571 from rhcs-dashboard/42775-fix-typo

common: fix typo in rgw_user_max_buckets option long description.

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31510 from yuvalif/wip-yuval-remove-flaky-amqp-test

rgw/amqp: remove flaky amqp test

Reviewed-by: Kefu Chai <kchai@redhat.com>

common/options: osd_crush_chooseleaf_type is CLUSTER_CREATE

Signed-off-by: Sage Weil <sage@redhat.com>

mon/ConfigMonitor: do not assimilate CLUSTER_CREATE options

These are only useful when the mon first bootstrapped--no reason to pull
them into the cluster config.

Signed-off-by: Sage Weil <sage@redhat.com>

common/ceph_context: observe container_image so we don't get warnings

This gets rid of messages like

2019-11-12T23:46:28.578+0000 7f9ab2b70700 -1 set_mon_vals failed to set container_image = cephci/daemon-base:wip-sage2-testing-2019-11-11-1737-4ea2bc7-centos-7-x86_64-devel: Configuration option 'container_image' may not be modified at runtime

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: add `--legacy-dir` arg to `ls` command

Signed-off-by: Michael Fritch <mfritch@suse.com>

ceph-daemon: -v|--verbose, not -d|--debug

-v is more natural IMO

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31575 into master

* refs/pull/31575/head:
ceph-daemon: 'logs' command

Reviewed-by: Michael Fritch <mfritch@suse.com>

ceph-daemon: 'logs' command

This just invokes podman logs with the appropriate argument(s).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31577 into master

* refs/pull/31577/head:
pybind/mgr/CMakeLists: exclude tox.ini, requirements.txt from install

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #31542 from neha-ojha/wip-42756

osd/OSDMap.cc: don't output over/underfull messages to lderr

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Xie Xingguo <xie.xingguo@zte.com.cn>

pybind/mgr/CMakeLists: exclude tox.ini, requirements.txt from install

Signed-off-by: Sage Weil <sage@redhat.com>

backport-create-issue: read redmine key from file

Now you don't need to provide redmine key or user/password as a flag, as the
script will read it from "~/.redmine_key".

This makes it more consistent with other scripts.

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge PR #31157 into master

* refs/pull/31157/head:
doc: update inf/infinite option for subvolume resize
qa/tasks: fs subvolume resize inf command
mgr/volumes: fs subvolume resize inf/infinite command

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31503 into master

* refs/pull/31503/head:
Revert "os/bluestore: add kv_drain_preceding_waiters indicate drain_preceding."

Reviewed-by: Jianpeng Ma <jianpeng.ma@intel.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #31537 into master

* refs/pull/31537/head:
mgr/orchestrator: move split_host into shared helpers
mgr/ssh: allow names to be specified for stateless services
mgr/ssh: optionally specify names for mon daemons
mgr/ssh: optionally specify names for mgr daemons

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #31280 from zy751713126/snap_get_name_and_id

librbd: add snap_get_name and snap_get_id method API

Merge pull request #31491 from zy751713126/rbd_bench

rbd/bench: outputs bytes/s format dynamically

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #31497 from trociny/wip-mock-warnings

test/librbd: fix mock warnings in TestMockIoImageRequest

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #31487 into master

* refs/pull/31487/head:
ceph-daemon: add `--base-dir` arg to `adopt` command

Reviewed-by: Sage Weil <sage@redhat.com>

common: fix typo in rgw_user_max_buckets option long description.

https://tracker.ceph.com/issues/42775
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

mgr/orchestrator: move split_host into shared helpers

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: allow names to be specified for stateless services

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: optionally specify names for mon daemons

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: optionally specify names for mgr daemons

Use '=' as a separator.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Manage PG autoscaling (#31417)

Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge PR #31559 into master

* refs/pull/31559/head:
mgr: Python cleanup and type check

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #31539 into master

* refs/pull/31539/head:
mgr/orchestrator_cli: sort by service name
mgr/orchestrator_cli: collapse TYPE and ID columns

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #31394 into master

* refs/pull/31394/head:
mgr/crash: Calculate and add stack_sig to metadata

Reviewed-by: Sage Weil <sage@redhat.com>

mgr: Python cleanup and type check

- Add/remove missing/obsolete empty lines
- Add type check doc
- Rename inner function args to make pylint happy

Signed-off-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Support RBD namespaces (#30935)

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #31357 from votdev/improve_timediff_unittest

Reviewed-by: Tatjana Dehler <tdehler@suse.com>

cmake: detect gettid() presense

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #30534 from jiahuizeng/osd_dev_node

mgr/prometheus: assign a value to osd_dev_node when obj_store is not filestore or bluestore

Fix backend testing error due to attribute rename

`id` in Device is renamed to `path`.

Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

Merge pull request #31431 from jschmid1/fix_cherrypy_imports

mgr/dashboard: cheroot moved into a separate project

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr: Run python unit tests with tox in the mgr (#30364)

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

mgr/dashboard: Remove compression mode unset in pool from (#31376)

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #29944 from xxhdx1985126/wip-fix-blustore-fadvice_flags-leak

osd: copy ObjectOperation::BufferUpdate::Write::fadvise_flag to ceph::os::Transaction

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #31333 from adamemerson/wip-who-warns-the-warnings

src: use un-deprecated version of aio_create_completion

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #31416 from sseshasa/wip-41666-replicaSizeWarn

osd/OSDMap: Show health warning if a pool is configured with size 1

Reviewed-by: Sage Weil <sweil@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #31475 from zdover23/wip-doc-community-calendar

doc: Added a link to Ceph Community Calendar.

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

cephfs-shell: print helpful message when conf file is not found

Fixes: https://tracker.ceph.com/issues/42508
Signed-off-by: Rishabh Dave <ridave@redhat.com>

doc: Added a link to Ceph Community Calendar.

Co-Authored by: Neha Ojha <nojha@redhat.com>
Signed-off-by: Zac Dover <zac.dover@gmail.com>