mgr/dashboard: Update dev documentation URL

Update the developer documentation URL because the
old one is only returning "403 Forbidden".

Signed-off-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #31546 from zjcmszh/execute

pybind/rados: add WriteOp::execute()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30968 from majianpeng/osd-optimze-send_message_osd_cluster

osd: optimize send_message to peers

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #31550 from wido/issue-42761

systemd: Wait 5 seconds before attempting a restart of an OSD

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #31623 from badone/wip-tracker-42387-extend-admin-socket-timeout

common/admin_socket: Increase socket timeouts

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #31241 from zjcmszh/add_rados_create_pool2

pybind/rados/rados.pyx: improve Rados.create_pool()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31617 from Devp00l/wip-42776

mgr/dashboard: Fix pool renaming functionality

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #31515 from drunkard/master

doc: typo fix in doc/dev/dev_cluster_deployement.rst: s/hostanme/hostname/

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31541 from BlackLotus/master

added a remark to always use powers of two for pg_num

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31685 from liewegas/fix-mgr-shutdown

mgr: _exit(0) from signal handler even if we are standby

Merge pull request #31695 from liewegas/wip-cd-simpler-bootstrap

ceph-daemon: bootstrap: make --output-* args optional

mgr/dashboard: Remove title from sparkline tooltips (#31645)

mgr/dashboard: Remove title from sparkline tooltips

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #31689 from tchaikov/wip/workunits/cephtool/test.sh

qa/workunits/cephtool/test.sh: a handful fixes

Reviewed-by: Sage Weil <sweil@redhat.com>

Merge pull request #30397 from MahatiC/wip-ref-counting

librbd: add reference counting

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/suites/rados: whitelist health warnings

in cephtool/test.sh, we

ceph fs set cephfs inline_data {1,0}

so the health check fails when the test ends, like

mon.a (mon.0) 3498 : cluster [WRN] Health check failed: 1 filesystem
with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED)" in
cluster log

so, before we remove the test, we need to whitelist this warning

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31688 from smithfarm/wip-bci-clean-print

backport-create-issue: flush line before overprinting

Reviewed-by: Tiago Melo <tmelo@suse.com>

qa/workunits/cephtool/test.sh: set min_{bytes,pgs}_per_osd for reweight

otherwise ceph-mgr will refuse to perform reweight

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits/cephtool/test.sh: create pool with power-of-2 pg num

otherwise wait_for_health() fails like:

wait_for_health:  ceph health detail
HEALTH_WARN 1 pool(s) have non-power-of-two pg_num
[WRN] POOL_PG_NUM_NOT_POWER_OF_TWO: 1 pool(s) have non-power-of-two pg_num
    pool 'rbd' pg_num 10 is not a power of two
../qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1613: wait_for_health:  return 1

the failure was found when testing test_mon_pg().

this behavior was introduced by 6e46b1c0e50ad377d5db058e0452b0d956b3fad6

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits/cephtool/test.sh: use "ceph compact" in test_mon_deprecated_commands()

both "compact" and "scrub" are now cli command, so we cannot issue them
using "tell" anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31710 from votdev/ignore_ssh_config

mgr/ssh: Ignore ssh-config file

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/ssh: Ignore ssh-config file

Ignore the ssh-config file that is created when working with the Vagrant box.

Signed-off-by: Volker Theile <vtheile@suse.com>

qa/workunits/cephtool/test.sh: use "sessions" in test_mon_tell()

"version" is not an asok command anymore in the sense that it's served
by registered asock hook. so in this change, we replace "version" with
"sessions", so we can verify that audit channel does not the dispatched
"sessions" command sent from ceph cli.

also, restructure the test as a loop for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits/cephtool/test.sh: use tell instead of --admin-daemon for asok commands

ceph cli will be able to route the command to the target daemon when
handling "tell" commands.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits/cephtool/test.sh: remove heap cli tests

"heap" cli commands are removed in the big asok/tell cleanup, they are now
asock commands.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/man/ceph: "heap" is now an asok command

after the big asok/tell cleanup, "heap" is now an asok command, so
update the manpage accordingly

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits/cephtool/test.sh: fix iec/si tests

tests all IEC and SI units related test with a tier pool. as
`target_max_objects` and `target_size_bytes` only apply to tier
pools. so, for the sake of simplicity, tests all of them using
a tier pool.

introduced by 9095f67e

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-daemon: bootstrap: fix example shell command

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: default output file names

- These are reasonable names that a user would probably want
- If they run bootstrap in /etc/ceph then they are already installed in
  the right location
- This means that the bootstrap command has only 1 (!) required argument
  (--mon-ip or equivalent)

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #31637 from carletes/patch-1

src/msg/async/net_handler.cc: Fix compilation

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31543 from tchaikov/wip/cmake/librt

cmake: detect librt for POSIX time functions

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #31532 from tchaikov/wip-42742

msg/async/rdma: do not init mutex before lockdeps is ready

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Changcheng Liu <changcheng.liu@aliyun.com>

Merge pull request #31506 from Yan-waller/wip-walle-191109-fixomapentries

common/options: decrease the default max_omap_entries_per_request

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #31310 from badone/wip-tracker-42477-reinstate-o-outfile

tools/rados: Unmask '-o' to restore original behaviour

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #31469 from p-se/wip-pse-summary-reload-traceback

mgr/dashboard: KeyError on dashboard reload

Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #31399 from zy751713126/osd_stat

mon/OSDMonitor: fix format error ceph osd stat --format json

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31427 from zy751713126/osd_status

pybind/mgr: Cancel output color control

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #31560 from rosinL/wip-fix-incorrect-pg-pool-sum

mon/PGMap: fix incorrect pg_pool_sum when delete pool

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31158 from zjcmszh/test_remove

test/pybind/test_rados.py: test test_operate_aio_write_op()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30717 from rzarzynski/wip-bug-42026

msg/async: perform the v2 resets in proper EventCenter

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #31517 from changchengx/rdma_gid

msg/async/rdma: add an option for choosing different RoCE protocol

Reviewed-by: Kefu Chai <kchai@redhat.com>

backport-create-issue: flush line before overprinting

When iterating over "Pending Backport" issues, the script displays
a message for each issue, like so:

    Examining issue#53432 (345/543)

Since a newline is not printed, each subsequent message overwrites
the previous one. But the last one remained on the screen and was
being overwritten by a shorter log message, leaving behind an
unwanted artifact:

    INFO:root:Processed 334 issues)

Fixes: 8495d37695d43ea38e1b70851e4a6127ebb4fc76
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge PR #31620 into master

* refs/pull/31620/head:
mgr: handle race with finisher after shutdown

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

mgr: _exit(0) from signal handler even if we are standby

In 3d360b97ed7dead8e9a5f602cfaf61ab7b41e531 a signal handler was added to
shut down modules and associated clients cleanly, but the standby case
was left ignoring the signal completely.

Fixes: https://tracker.ceph.com/issues/42744
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #31681 from ifed01/wio-ifed-fix-fio

test/fio: handle nullptr when parsing throttle params

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #31025 from scarvalhojr/ssevault2

rgw: improvements to SSE-KMS with Vault

Reviewed-by: Casey Bodley <cbodley@redhat.com>

test/fio: handle nullptr when throttle params parsing

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #31671 from sebastian-philipp/pybind-mgr-exclude-tests

pybind/mgr: Exclude tests/

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge PR #31486 into master

* refs/pull/31486/head:
ceph-daemon: add standalone `adopt` tests

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #31618 from dsavineau/ceph_daemon_rename_podman_vars

ceph-daemon: replace podman variables by container

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

pybind/mgr: Exclude tests/

Fixes: d6c10192c61a59f6a77d04a9d601c9df2f72d07f
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #31666 from hrchu/origin/patch-2

doc: Update mailing lists

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31667 from tchaikov/wip/run-cbt-sh/fs.aio-max-nr

script/run-cbt.sh: set fs.aio-max-nr for seastar

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

script/run-cbt.sh: set fs.aio-max-nr for seastar

seastar requires 11027 for each reactor, so we need at least

fs.aio-max-nr = 11027 * n_osd

but to make it simple and to take the needs of other applications using
aio on the machine, we increase the number of io context to 32768,
and always set this setting no matter if we are starting a vstart
cluster with classic osd or crimson osd

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31523 from sebastian-philipp/ssh-make-check

mgr/ssh: add make check integration

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>

doc: Update mailing lists

Signed-off-by: hrchu <petertc.chu@gmail.com>

Merge PR #31509 into master

* refs/pull/31509/head:
client: remove unused variable

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

client: remove unused variable

the flock and cond are useless variables in Client::_read_sync,remove useless variables

Fixes: https://tracker.ceph.com/issues/42720
Signed-off-by: su_nan@inspur.com <su_nan@inspur.com>

Merge pull request #31643 from liewegas/fix-empty-crash

mgr/crash: behave when posted crash has no backtrace

src/msg/async/net_handler.cc: Fix compilation

On a Cray system I'm working on, it seems that `SO_PRIORITY` is defined,
but `IPTOS_CLASS_CS6` is not. Without this patch, compilation fails at
line 150:

```
r = ::setsockopt(sd, SOL_SOCKET, SO_PRIORITY, &prio, sizeof(prio));
```

because the variable `r` is not defined (originally it is defined inside
the `#ifdef IPTOS_CLASS_CS6` block).

Fixes: https://tracker.ceph.com/issues/42821
Signed-off-by: Carlos Valiente <carlos.valiente@ecmwf.int>

Merge PR #31626 into master

* refs/pull/31626/head:
test_cephfs_shell: initialize stderr for run_cephfs_shell_cmd()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31460 into master

* refs/pull/31460/head:
cephfs-shell: print helpful message when conf file is not found

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #30385 from ZVampirEM77/acl-detailed-err-msg

rgw: add detailed error message for PutACLs

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30470 from dongbula/bf-fix-usage-keyiter-update

RGW: fix an endless loop error when to show usage

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30574 from zhangsw/rgw-fix-compress

rgw: compression info should be same during multipart uploading

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30575 from mkogan1/wip-rgw-resharding-mem

rgw: when resharding store progress json

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
Reviewed-by: Yuval Lifshitz <yuvalif@yahoo.com>

mgr/dashboard: Remove title from sparkline tooltips

Title was never being defined in sparklines and
was always displayed as 'undefined'.

Since they are not needed for this type of chart, we are simply going to remove
them.

Fixes: https://tracker.ceph.com/issues/41656
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #31632 from tspmelo/wip-rbd-timezone

mgr/dashboard: Fix error in unit test caused by timezone

Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #29004 from zhangsw/rgw-cleanup-taggingop

rgw: remove re-defined is_tagging_op in RGWHandler_REST_Bucket_S3

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #31582 into master

* refs/pull/31582/head:
global: disable THP for Ceph daemons

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge PR #31584 into master

* refs/pull/31584/head:
common/options: osd_crush_chooseleaf_type is CLUSTER_CREATE
mon/ConfigMonitor: do not assimilate CLUSTER_CREATE options
common/ceph_context: observe container_image so we don't get warnings

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #31621 into master

* refs/pull/31621/head:
os/bluestore: consolidate extents from the same device only

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/crash: behave when posted crash has no backtrace

This is the case when the qa/tasks/mgr/test_crash.py test runs, and could
happen in the real world too, perhaps, if someone fed us wonky input.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: replace podman variables by container

There's no logic to use variables with podman prefix when we can use both
docker and podman.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

mgr/ssh: add make check integration

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #31436 from dang/wip-dang-zipper-3

rgw: Project Zipper - Bucket

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #30754 into master

* refs/pull/30754/head:
doc/cephfs: merge fstab doc with respective mount docs
doc: add systemd unit part for FUSE mounts in fstab doc
doc: update and improve "mount using kernel driver" doc
doc: update and improve "mount using FUSE" doc

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31631 into master

* refs/pull/31631/head:
ceph-daemon: Add more type hints

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #31561 from sebastian-philipp/orchestrator-remove-cli-tox

mgr/orchestrator_cli: remove tox and move test to parent dir

Reviewed-by: Stephan Müller <smueller@suse.com>

cmake: detect librt for POSIX time functions

per clock_gettime(3),

> On POSIX systems on which these functions are available, the symbol
> _POSIX_TIMERS is defined in <unistd.h> to a value greater than 0.

and

> Most systems require the program be linked with the librt library to
> use these functions.

so, we should detect this symbol and link against librt for using these
functions.

in this change, librt is linked for checking the existence of
clock_gettime(), if it exists. RT_LIBRARY is defined.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/orchestrator: remove tox and move test to parent dir

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: Fix error in unit test caused by timezone

Unit test was only accepting positive timezones

Fixes: https://tracker.ceph.com/issues/42812
Signed-off-by: Tiago Melo <tmelo@suse.com>

ceph-daemon: Add more type hints

Commit 7ce26be7a3448 started to add type hints. This commit continues
that and adds the missing type hints.

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>

test_cephfs_shell: initialize stderr for run_cephfs_shell_cmd()

Since teuthology initializes stderr to None by default, absence of this
breaks the tests accessing stderr of commands executed within the test
when the execution is using teuthology.

Fixes: https://tracker.ceph.com/issues/42806
Signed-off-by: Rishabh Dave <ridave@redhat.com>

common/admin_socket: Increase socket timeouts

With the move of the 'bench' command to the admin socket the recv
timeout is being exceeded in testing due to the duration of the command.

Fixes: https://tracker.ceph.com/issues/42387
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

doc/cephfs: merge fstab doc with respective mount docs

Fixes: https://tracker.ceph.com/issues/42601
Signed-off-by: Rishabh Dave <ridave@redhat.com>

ceph-daemon: add standalone `adopt` tests

Signed-off-by: Michael Fritch <mfritch@suse.com>

global: disable THP for Ceph daemons

Ceph is known to suffer from memory fragmentation due to transparent
huge pages (THP). This is indicated by RSS usage above configured memory
targets and is only observable when the distribution default for THP is
"always", which is the default in the upstream kernel if
CONFIG_TRANSPARENT_HUGEPAGE_MADVISE is unset.

For now, enabling THP is currently discouraged until selective use of
THP by Ceph is implemented via madvise. We will need to consider both
defaults for THP so madvise calls to both enable and disable THP will
need implemented.

All credit to Mark Nelson for doing the legwork identifying this issue
and potential solutions.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
Signed-off-by: Mark Nelson <mnelson@redhat.com>

mgr: handle race with finisher after shutdown

The module is deleted when modules is cleared. The notify method is
called on a destructed class.

Fixes: https://tracker.ceph.com/issues/42744
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31614 into master

* refs/pull/31614/head:
ceph-daemon: use `-e` instead of `--env`

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: consolidate extents from the same device only

Presumably Fixes: https://tracker.ceph.com/issues/42223

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #31613 into master

* refs/pull/31613/head:
mgr/ssh: fix redeploy

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #31609 into master

* refs/pull/31609/head:
ceph-daemon: Add basic mypy support

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #31601 from tchaikov/wip/install-newer-binutils-on-xenial

install-deps.sh: install binutils 2.28 for xenial

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

mgr/dashboard: Fix pool renaming functionality

The problem was that, if a pool was renamed the routine to update tried
to work on the old pool name and didn't used the info that the pool got
renamed, therefore an error occurred in the UI saying that the pool
could not be found.

Fixes: https://tracker.ceph.com/issues/42776
Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Fix pg_num counts in pool API tests

To prevent the not a number of two cluster health warning, all created
pools now have a pg_num that is a power of two.

Fixes: https://tracker.ceph.com/issues/42776
Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Refactor pool API tests

Renamed "__create_pool" to "__yield_pool" to make it's purpose clearer
and added a description.

Split up create pool test into three different tests.
Used "__yield_pool" for creation tests, too.

Fixes: https://tracker.ceph.com/issues/42776
Signed-off-by: Stephan Müller <smueller@suse.com>

Merge PR #31602 into master

* refs/pull/31602/head:
mgr/ssh: annotate object representation

Reviewed-by: Sebastian Wagner <swagner@suse.com>

ceph-daemon: use `-e` instead of `--env`

Older versions of podman don't allow intermixing `-e` and `--env`

Signed-off-by: Michael Fritch <mfritch@suse.com>

install-deps.sh: remove the cleanup for kitware repo

it was used for removing the kitware repo, and in
7265b55d094a639be50a567d3be92fba94c04786, we switched to a rebuilt
version hosted in chacra, and it was more than two months ago,
presumably, none builder has kitware repo anymore. so, let's remove
the cleanup code.

Signed-off-by: Kefu Chai <kchai@redhat.com>

install-deps.sh: install binutils 2.28 for xenial

ld 2.26 segment faults when linking librados.so in
https://github.com/ceph/ceph/pull/16715 .

while ld 2.28 always survives.
binutils 2.28 is rebuilt and uploaded to chacra
binutils/master/7fa393306ed8b93019d225548474c0540b8928f7/ubuntu/xenial,

so let's use it instead.

the source deb package comes from
http://ppa.launchpad.net/jonathonf/binutils/ubuntu

i tried to use the binutils 2.30 source package from bionic,
but it has build dependency of dpkg-dev (>= 1.19.0.5) which cannot be
fulfilled on xenial. so without more changes, we cannot get
binutils 2.30 built on xenial using this source package.

Fixes: https://tracker.ceph.com/issues/42596
Signed-off-by: Kefu Chai <kchai@redhat.com>