script/run_tox.sh: do not use python2 if we have python3

always use python3 for bootstrapping tox environment unless the tox-env
contains "py2" but not "py3"

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #30698 into master

* refs/pull/30698/head:
common/options: Set bluestore min_alloc size to 4K

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #31638 into master

* refs/pull/31638/head:
mgr/pg_autoscaler: only generate target_* health warnings if targets set

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #31572 into master

* refs/pull/31572/head:
mgr/diskprediction_local: some debug output during predict (and self-test)

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #31721 into master

* refs/pull/31721/head:
mgr/ssh: set UserKnownHostsFile

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #31660 from ivancich/wip-add-some-const

rgw: add const correctness to some rest functions

Merge pull request #16715 from adamemerson/wip-I-Object!

osdc/Objecter: Boost.Asio (I object!)

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #31672 into master

* refs/pull/31672/head:
mgr/orch: adapt update_mgr/mon funcs to new parsing
mgr/orch: improve commandline parsing

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #31679 into master

* refs/pull/31679/head:
ceph-daemon: add osd create test(s)

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #31731 from varshar16/wip-doc-update-orchestrator_cli-url

mgr/orchestrator_cli: Update doc link in README

mgr/ssh: set UserKnownHostsFile

Set this to /dev/null, since we have StrictHostKeyChecking
off anyway.  This gets rid of messages to stderr every time we connect
to a new host.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #29399 from ofriedma/master

rgw: Fix expiration header does not return the earliest rule

Merge pull request #29937 from ofriedma/wip-ofriedma-days0

Add days0 to rgw lc

Merge pull request #31642 from trociny/wip-rbd-mirror-service-daemon-ns-support

rbd-mirror: add namespace support to service daemon

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #31627 from trociny/wip-pybind-rbd-snap_remove

pybind/rbd: provide snap remove flags

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #31608 from trociny/wip-mock-warnings

test/rbd_mirror: fix mock warnings

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #31606 from trociny/wip-test_rbd.py-tabs

test/pybind: inconsistent use of tabs and spaces in indentation

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/orchestrator_cli: Update doc link in README

Update the orchestrator cli doc link to fix page not found error. Also remove
some unnecessary blank lines.

Signed-off-by: Varsha Rao <varao@redhat.com>

Merge pull request #31594 from smithfarm/wip-bri-regex-fix

backport-resolve-issue: narrow regular expression and read key/token from files

Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #30336 from joscollin/wip-F40959-subvolume-uid-gid

mgr/volumes: uid, gid for subvolume create and subvolumegroup create commands

Merge pull request #31546 from zjcmszh/execute

pybind/rados: add WriteOp::execute()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30968 from majianpeng/osd-optimze-send_message_osd_cluster

osd: optimize send_message to peers

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #31550 from wido/issue-42761

systemd: Wait 5 seconds before attempting a restart of an OSD

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #31623 from badone/wip-tracker-42387-extend-admin-socket-timeout

common/admin_socket: Increase socket timeouts

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #31241 from zjcmszh/add_rados_create_pool2

pybind/rados/rados.pyx: improve Rados.create_pool()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31617 from Devp00l/wip-42776

mgr/dashboard: Fix pool renaming functionality

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

ceph-daemon: add osd create test(s)

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge pull request #31515 from drunkard/master

doc: typo fix in doc/dev/dev_cluster_deployement.rst: s/hostanme/hostname/

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31541 from BlackLotus/master

added a remark to always use powers of two for pg_num

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31685 from liewegas/fix-mgr-shutdown

mgr: _exit(0) from signal handler even if we are standby

Merge pull request #31695 from liewegas/wip-cd-simpler-bootstrap

ceph-daemon: bootstrap: make --output-* args optional

mgr/orch: adapt update_mgr/mon funcs to new parsing

update_mgr and update_mon (and their correspinding orch-backend
implementation) are adapted to the new parsing parsing method
introduced in ff9093873.

Signed-off-by: Joshua Schmid <jschmid@suse.de>

mgr/dashboard: Remove title from sparkline tooltips (#31645)

mgr/dashboard: Remove title from sparkline tooltips

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #31689 from tchaikov/wip/workunits/cephtool/test.sh

qa/workunits/cephtool/test.sh: a handful fixes

Reviewed-by: Sage Weil <sweil@redhat.com>

Merge pull request #30397 from MahatiC/wip-ref-counting

librbd: add reference counting

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/suites/rados: whitelist health warnings

in cephtool/test.sh, we

ceph fs set cephfs inline_data {1,0}

so the health check fails when the test ends, like

mon.a (mon.0) 3498 : cluster [WRN] Health check failed: 1 filesystem
with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED)" in
cluster log

so, before we remove the test, we need to whitelist this warning

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31688 from smithfarm/wip-bci-clean-print

backport-create-issue: flush line before overprinting

Reviewed-by: Tiago Melo <tmelo@suse.com>

backport-resolve-issue: read Redmine key and GitHub token from file

Thanks to Tiago Melo for inspiring me to do this.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

qa/workunits/cephtool/test.sh: set min_{bytes,pgs}_per_osd for reweight

otherwise ceph-mgr will refuse to perform reweight

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits/cephtool/test.sh: create pool with power-of-2 pg num

otherwise wait_for_health() fails like:

wait_for_health:  ceph health detail
HEALTH_WARN 1 pool(s) have non-power-of-two pg_num
[WRN] POOL_PG_NUM_NOT_POWER_OF_TWO: 1 pool(s) have non-power-of-two pg_num
    pool 'rbd' pg_num 10 is not a power of two
../qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1613: wait_for_health:  return 1

the failure was found when testing test_mon_pg().

this behavior was introduced by 6e46b1c0e50ad377d5db058e0452b0d956b3fad6

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits/cephtool/test.sh: use "ceph compact" in test_mon_deprecated_commands()

both "compact" and "scrub" are now cli command, so we cannot issue them
using "tell" anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

backport-resolve-issue: narrow regular expression

The regular expression used to extract tracker URLs from the PR body
was too generous. When there are two URLs on the same line and the
second is a tracker URl, the regex picked up both of them. For example,
the following string matched the regex:

"https://github.com/ceph/ceph/pull/26538, https://tracker.ceph.com/issues/41452"

This commit fixes the issue.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #31710 from votdev/ignore_ssh_config

mgr/ssh: Ignore ssh-config file

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/ssh: Ignore ssh-config file

Ignore the ssh-config file that is created when working with the Vagrant box.

Signed-off-by: Volker Theile <vtheile@suse.com>

qa/workunits/cephtool/test.sh: use "sessions" in test_mon_tell()

"version" is not an asok command anymore in the sense that it's served
by registered asock hook. so in this change, we replace "version" with
"sessions", so we can verify that audit channel does not the dispatched
"sessions" command sent from ceph cli.

also, restructure the test as a loop for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits/cephtool/test.sh: use tell instead of --admin-daemon for asok commands

ceph cli will be able to route the command to the target daemon when
handling "tell" commands.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits/cephtool/test.sh: remove heap cli tests

"heap" cli commands are removed in the big asok/tell cleanup, they are now
asock commands.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/man/ceph: "heap" is now an asok command

after the big asok/tell cleanup, "heap" is now an asok command, so
update the manpage accordingly

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits/cephtool/test.sh: fix iec/si tests

tests all IEC and SI units related test with a tier pool. as
`target_max_objects` and `target_size_bytes` only apply to tier
pools. so, for the sake of simplicity, tests all of them using
a tier pool.

introduced by 9095f67e

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-daemon: bootstrap: fix example shell command

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: default output file names

- These are reasonable names that a user would probably want
- If they run bootstrap in /etc/ceph then they are already installed in
  the right location
- This means that the bootstrap command has only 1 (!) required argument
  (--mon-ip or equivalent)

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #31637 from carletes/patch-1

src/msg/async/net_handler.cc: Fix compilation

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31543 from tchaikov/wip/cmake/librt

cmake: detect librt for POSIX time functions

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #31532 from tchaikov/wip-42742

msg/async/rdma: do not init mutex before lockdeps is ready

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Changcheng Liu <changcheng.liu@aliyun.com>

Merge pull request #31506 from Yan-waller/wip-walle-191109-fixomapentries

common/options: decrease the default max_omap_entries_per_request

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #31310 from badone/wip-tracker-42477-reinstate-o-outfile

tools/rados: Unmask '-o' to restore original behaviour

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #31469 from p-se/wip-pse-summary-reload-traceback

mgr/dashboard: KeyError on dashboard reload

Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #31399 from zy751713126/osd_stat

mon/OSDMonitor: fix format error ceph osd stat --format json

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31427 from zy751713126/osd_status

pybind/mgr: Cancel output color control

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #31560 from rosinL/wip-fix-incorrect-pg-pool-sum

mon/PGMap: fix incorrect pg_pool_sum when delete pool

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31158 from zjcmszh/test_remove

test/pybind/test_rados.py: test test_operate_aio_write_op()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30717 from rzarzynski/wip-bug-42026

msg/async: perform the v2 resets in proper EventCenter

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #31517 from changchengx/rdma_gid

msg/async/rdma: add an option for choosing different RoCE protocol

Reviewed-by: Kefu Chai <kchai@redhat.com>

backport-create-issue: flush line before overprinting

When iterating over "Pending Backport" issues, the script displays
a message for each issue, like so:

    Examining issue#53432 (345/543)

Since a newline is not printed, each subsequent message overwrites
the previous one. But the last one remained on the screen and was
being overwritten by a shorter log message, leaving behind an
unwanted artifact:

    INFO:root:Processed 334 issues)

Fixes: 8495d37695d43ea38e1b70851e4a6127ebb4fc76
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge PR #31620 into master

* refs/pull/31620/head:
mgr: handle race with finisher after shutdown

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

mgr: _exit(0) from signal handler even if we are standby

In 3d360b97ed7dead8e9a5f602cfaf61ab7b41e531 a signal handler was added to
shut down modules and associated clients cleanly, but the standby case
was left ignoring the signal completely.

Fixes: https://tracker.ceph.com/issues/42744
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #31681 from ifed01/wio-ifed-fix-fio

test/fio: handle nullptr when parsing throttle params

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #31025 from scarvalhojr/ssevault2

rgw: improvements to SSE-KMS with Vault

Reviewed-by: Casey Bodley <cbodley@redhat.com>

test/fio: handle nullptr when throttle params parsing

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #31671 from sebastian-philipp/pybind-mgr-exclude-tests

pybind/mgr: Exclude tests/

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge PR #31486 into master

* refs/pull/31486/head:
ceph-daemon: add standalone `adopt` tests

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #31618 from dsavineau/ceph_daemon_rename_podman_vars

ceph-daemon: replace podman variables by container

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

pybind/mgr: Exclude tests/

Fixes: d6c10192c61a59f6a77d04a9d601c9df2f72d07f
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #31666 from hrchu/origin/patch-2

doc: Update mailing lists

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/orch: improve commandline parsing

We have to parse options like hostname, network and optional name

Examples:

"myhost"
"myhost=name"
"myhost:1.2.3.4"
"myhost:1.2.3.4=name"
"myhost:1.2.3.0/24"
"myhost:1.2.3.4/24=name"
"myhost:v2:1.2.3.4:3000/24=name"
"myhost:v2:1.2.3.4:3000,v1:1.2.3.4:6789/24=name"

This patch improves the durability of the code by using
regexes and adds validations for ip_addresses and subnets.

This will now also act as the single parser for all inputs.

Signed-off-by: Joshua Schmid <jschmid@suse.de>
(cherry picked from commit 4812167228813adc2fa45ff502257ca38ed53530)

Merge pull request #31667 from tchaikov/wip/run-cbt-sh/fs.aio-max-nr

script/run-cbt.sh: set fs.aio-max-nr for seastar

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

script/run-cbt.sh: set fs.aio-max-nr for seastar

seastar requires 11027 for each reactor, so we need at least

fs.aio-max-nr = 11027 * n_osd

but to make it simple and to take the needs of other applications using
aio on the machine, we increase the number of io context to 32768,
and always set this setting no matter if we are starting a vstart
cluster with classic osd or crimson osd

Signed-off-by: Kefu Chai <kchai@redhat.com>

rbd-mirror: add namespace support to service daemon

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #31523 from sebastian-philipp/ssh-make-check

mgr/ssh: add make check integration

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>

doc: Update mailing lists

Signed-off-by: hrchu <petertc.chu@gmail.com>

Merge PR #31509 into master

* refs/pull/31509/head:
client: remove unused variable

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

client: remove unused variable

the flock and cond are useless variables in Client::_read_sync,remove useless variables

Fixes: https://tracker.ceph.com/issues/42720
Signed-off-by: su_nan@inspur.com <su_nan@inspur.com>

rgw: add const correctness to some rest functions

Also, clean up some namespace declarations. Also, convert to using
`#pragma once`.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #31643 from liewegas/fix-empty-crash

mgr/crash: behave when posted crash has no backtrace

src/msg/async/net_handler.cc: Fix compilation

On a Cray system I'm working on, it seems that `SO_PRIORITY` is defined,
but `IPTOS_CLASS_CS6` is not. Without this patch, compilation fails at
line 150:

```
r = ::setsockopt(sd, SOL_SOCKET, SO_PRIORITY, &prio, sizeof(prio));
```

because the variable `r` is not defined (originally it is defined inside
the `#ifdef IPTOS_CLASS_CS6` block).

Fixes: https://tracker.ceph.com/issues/42821
Signed-off-by: Carlos Valiente <carlos.valiente@ecmwf.int>

Merge PR #31626 into master

* refs/pull/31626/head:
test_cephfs_shell: initialize stderr for run_cephfs_shell_cmd()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31460 into master

* refs/pull/31460/head:
cephfs-shell: print helpful message when conf file is not found

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #30385 from ZVampirEM77/acl-detailed-err-msg

rgw: add detailed error message for PutACLs

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30470 from dongbula/bf-fix-usage-keyiter-update

RGW: fix an endless loop error when to show usage

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30574 from zhangsw/rgw-fix-compress

rgw: compression info should be same during multipart uploading

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30575 from mkogan1/wip-rgw-resharding-mem

rgw: when resharding store progress json

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
Reviewed-by: Yuval Lifshitz <yuvalif@yahoo.com>

mgr/dashboard: Remove title from sparkline tooltips

Title was never being defined in sparklines and
was always displayed as 'undefined'.

Since they are not needed for this type of chart, we are simply going to remove
them.

Fixes: https://tracker.ceph.com/issues/41656
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #31632 from tspmelo/wip-rbd-timezone

mgr/dashboard: Fix error in unit test caused by timezone

Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #29004 from zhangsw/rgw-cleanup-taggingop

rgw: remove re-defined is_tagging_op in RGWHandler_REST_Bucket_S3

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #31582 into master

* refs/pull/31582/head:
global: disable THP for Ceph daemons

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge PR #31584 into master

* refs/pull/31584/head:
common/options: osd_crush_chooseleaf_type is CLUSTER_CREATE
mon/ConfigMonitor: do not assimilate CLUSTER_CREATE options
common/ceph_context: observe container_image so we don't get warnings

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #31621 into master

* refs/pull/31621/head:
os/bluestore: consolidate extents from the same device only

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/crash: behave when posted crash has no backtrace

This is the case when the qa/tasks/mgr/test_crash.py test runs, and could
happen in the real world too, perhaps, if someone fed us wonky input.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/pg_autoscaler: only generate target_* health warnings if targets set

We may get into a situation where the pool stats appear to exceed the
cluster capacity.  If that happens, don't generate confusing health
messages about the target_{ratio,bytes} pool settings when they aren't
actually set at all.

Fixes: https://tracker.ceph.com/issues/42301
Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: replace podman variables by container

There's no logic to use variables with podman prefix when we can use both
docker and podman.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>