Merge pull request #41929 from sebastian-philipp/fix-qa-tox

qa: Various make check fixes

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42419 from dsavineau/fix_typo_radosgw_admin_max_entries

doc/man/8/radosgw-admin: fix typo on max entries

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42289 from mgfritch/cephadm-test-create-daemon-dirs-prom

cephadm: use pyfakefs during test_create_daemon_dirs_prometheus

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

Merge pull request #42028 from mgfritch/cephadm-infer-mismatch-fsid

cephadm: infer fsid from ceph.conf

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #41280 from sebastian-philipp/test_cephadm-stdin

qa/workunits/test_cephadm: Also test stdin

Reviewed-by: Michael Fritch <mfritch@suse.com>

doc/man/8/radosgw-admin: fix typo on max entries

This fixes a typo for the --max-entries option.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #42413 from galsalomon66/fix_s3tests_yaml

RGW:revert force-branch to be ceph-master (s3-tests)

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42304 from sebastian-philipp/doc-dameon-add-note

doc/cephadm: add notes to `orch daemon add`

Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #42372 from sebastian-philipp/doc-cephadm-rgw-ssl

doc/cephadm: Add RGW ssl

Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>

revert force-branch to be ceph-master

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>

Merge pull request #42396 from ronen-fr/wip-ronenf-debug-docker-1

script: fix a bug in naming the source branch in ceph-debug-docker.sh

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42355 from yuvalif/wip-yuval-fix-51681

rgw/notifications: send correct size in case of delete marker creation

Merge pull request #42358 from yuvalif/wip-yuval-fix-51687

rgw/notification: send current time in complete multipart upload event

Merge pull request #42189 from yuvalif/wip-yuval-fix-51530

rgw/http/notifications: support content type in HTTP POST messages

Merge pull request #42388 from tchaikov/wip-crimson-alien

crimson/osd: use app.alien() to initialize AlienStore::alien

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #42392 from tchaikov/wip-test-test_ipaddr

test/test_ipaddr: always set ifa_flags

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #42188 from votdev/issue_51408_motd

mgr/dashboard: Add configurable MOTD or wall notification

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Paul Cuzner <pcuzner@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: sebastian-philipp <NOT@FOUND>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #42102 from Matan-B/lua-request-userId

rgw: Added request field to lua scripting

Merge pull request #41927 from Matan-B/rfe-51193-add-luarocks-version

rgw: add package version support to lua scripting

Merge pull request #42074 from ljflores/wip-lflores-perf-channel

mgr/telemetry: add new 'perf' channel that shares aggregated perf counter metrics of a cluster

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Yaarit Hatuka <yaarithatuka@gmail.com>

script: fix a bug in naming the source branch in ceph-debug-docker.sh

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #42391 from kelubo/master

doc/cephadm: positions of words are interchanged

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #42393 from tchaikov/wip-cmake-cleanups

cmake: require CMake v3.16 and cleanups

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

cmake: do not check policies introduced before 3.16

see

- https://cmake.org/cmake/help/latest/policy/CMP0065.html
- https://cmake.org/cmake/help/latest/policy/CMP0074.html
...
- https://cmake.org/cmake/help/latest/policy/CMP0093.html

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: use add_compile_options() to pass compile options

add_definitions() is for adding "-D" define flags, while "-include" does
not define flags, so let's be explicit and use add_compile_options()
instead.

this change partially reverts 5fb90bb79671b8e22f507728e4b4b948a591f98f.
back then, we were using ubuntu bionic for building windows packages.
but since we've switched to ubuntu focal, where cmake 3.16 is available,
there is no reason to stick with add_definitions() anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/cephadm: add missing "ceph"

The word ceph is missing.

Signed-off-by: "Wang,Fei" <wf.ab@126.com>

doc/cephadm: correct a transposed word error

The positions of two words are interchanged:
scans each cluster in the host ----> scans each host in the cluster

Signed-off-by: "Wang,Fei" <wf.ab@126.com>

cmake: require CMake v3.16

since we dropped the support of bionic, we now have the luxury of using
newer CMake! and by using CMake 3.16, we now have access to

* add_compile_definitions(), which was introduced in cmake 3.12

also, drop the comment of

> remove cmake/modules/FindPython* once 3.12 is required

because we need to support python 3.10 in future, better off keeping
the FindPython files around, so we can backport cmake changes from
upstream when necessary.

we are allowed to bump up the CMake's required to 3.16, because:

- ubuntu focal ships with cmake 3.16.3
- RHEL/CentOS8's AppStream has cmake 3.18

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/test_ipaddr: avoid reinterpret cast

drop couple occurrences of C-style cast, for better readability.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/test_ipaddr: always set ifa_flags

do not rely on the uninitialized bits on stack for testing, otherwise
there is chance that we have following test failure:

64: ../src/test/test_ipaddr.cc:226: Failure
64: Expected equality of these values:
64:   two.ifa_addr
64:     Which is: 127.0.0.1:57677
64:   result
64:     Which is: 127.0.0.1:57677

the root cause is just that the "result" is actually `one.ifa_addr`,
which is marked UP as that's what the uninitialized bits on stack
indicate.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: s/FuturizedStore*/FuturizedStore&/ in OSDMeta and PGMeta's ctors

since we always pass a valid store reference to these ctors, there is no
reason to use a pointer. let's use a reference instead.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: create store in main()

to initialize an alienstore with a reference of
seastar::alien::instance is but the internal of AlienStore, would be
better if we could avoid exposing it as a part of the interface of
crimson::osd::OSD().

so, in this change, instead of creating FuturizedStore in OSD(), we
create it in main() where the app is available, so we can just create
FuturizedStore without passing the alien instance all the way down to
OSD().

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: use app.alien() to initialize AlienStore::alien

in e53ea0886fba9073904f59ea85fb73d854565921, the new alien::submit_to() API
is used in the place of the old one. but seastar::alien::instance::_qs
should be initialized before we are able to use the alien instance, just
creating an instance of alien::instance is not enough.

in this change, instead of creating an instance of alien::instance using
make_unique<>, the return value of app.alien() is used to initialize the
alien member variable of AlienStore. app.alien() is always properly
initialized by reactor.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42360 from noviicee/noviicee

mgr/devicehealth: replace CLICommand with CLIReadCommand

Reviewed-by: Varsha Rao <varao@redhat.com>

Merge pull request #42379 from liewegas/cephadm-keyring-help

mgr/cephadm: add help strings for 'orch client-keyring ...' commands

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #42036 from kamoltat/wip-pg-autoscaler-overlap

pybind/mgr/autoscaler: don't scale pools with overlapping roots

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #42357 from cbodley/wip-cmake-legacy-option-headers-dep

cmake: add transitive dependency on legacy-option-headers

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42283 from orozery/op_target_t_constructor

common: optimize op_target_t and hobject_t constructors

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42366 from liewegas/mon-down-grace

mon: allow a MON_DOWN grace period after cluster mkfs

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #42315 from TheJJ/fix-op-duration-calculation

osd: fix op event duration calculation

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42385 from tchaikov/wip-crimson-test-alien

test/crimson/test_alien_echo: fixes

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #42351 from SoftIron/softiron-distro

build/qa: Add SoftIron Linux as valid distro in build and test scripts

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

test/crimson/test_alien_echo: msgr->stop() before shutdown

it is required to remove all registered dispatchers before shutting down a
messenger. otherwise assert(dispatchers.empty()) in SocketMessenger::shutdown() fails.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/crimson/test_alien_echo: drop msgr v1 support

crimson msgr dropped the v1 support back in
7a7e869c0a7d96e1e242929755c3af73085d00ac. so there is no point to offer
an option to use v1 protocol.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/cephadm: add help strings for 'orch client-keyring ...' commands

Signed-off-by: Sage Weil <sage@newdream.net>

rgw/notifications: send correct size in case of delete marker creation

Fixes: https://tracker.ceph.com/issues/51681
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

doc/cephadm: Add RGW ssl

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

rgw/http/notifications: support content type in HTTP POST messages

Fixes: https://tracker.ceph.com/issues/51530
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge PR #42338 into master

* refs/pull/42338/head:
mon/LogMonitor: tolerate external_log_to in the future

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/devicehealth:replace CLICommand with CLIReadCommand

Fixes: https://tracker.ceph.com/issues/51631
Signed-off-by: Anamika <cseanamika3@gmail.com>

mon: allow a MON_DOWN grace period after cluster mkfs

During teuthology tests, the initial cluster bootstrap often starts up
the mon sbut doesn't include all mons in the initial quorum, due to
mon startup misalignment and random delays.  Provide a short grace period
where we will not raise a MON_DOWN alert even though the quorum is not
complete.

Fixes: https://tracker.ceph.com/issues/43584
Signed-off-by: Sage Weil <sage@newdream.net>

Merge PR #42341 into master

* refs/pull/42341/head:
doc/cephadm: operations.rst typo

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #42200 from cbodley/wip-47781

rgw: RadosBucket::get_bucket_info() updates RGWBucketEnt

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #36213 from joke-lee/BulkUpload_md5

rgw: fix md5 not match for RGWBulkUploadOp upload when enable rgw com…

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42126 from paulreece42/wip-copyobject-urldecode

rgw: url_decode before parsing copysource in copyobject

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #41579 from trociny/wip-50932

rgw: allow to set ssl options and ciphers for beast frontend

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42195 from xrmeng8756/master

rgw: fail as expected when set/delete-bucket-website attempted on a non-exis…

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42359 from idryomov/wip-15.2.13-linkage

doc/releases: add linkage for 15.2.13

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

doc/releases: add linkage for 15.2.13

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

rgw/notification: send current time in complete multipart upload event

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>
Fixes: https://tracker.ceph.com/issues/51687

Merge PR #42343 into master

* refs/pull/42343/head:
qa/suites/upgrade/pacific-x: do distro before tasks
qa/suites/orch/cephadm/smoke: centos 8.2 -> 8.3

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #42073 into master

* refs/pull/42073/head:
doc/mgr/nfs: fix 'export apply', pool name
PendingReleaseNotes: document workaround for NFS storage change
qa/tasks/mgr/test_orchestrator_cli: fix test
qa/suites/orch/cephadm/mgr-nfs-upgrade: add test for nfs migration
mgr/cephadm: migrate nfs grace file
mgr/nfs: migrate pre-pacific nfs.ganesha-foo clusters to nfs.foo
doc/cephfs/fs-nfs-exports: document new export apply capabilities
qa/tasks/cephfs/test_nfs: define NFS_POOL_NAME
mgr/nfs: use NFS_POOL_NAME in test_nfs.py
mgr/nfs: test export apply on JSON list
mgr/nfs: add test for ganesha conf apply/import
qa/tasks/cephfs/test_nfs: retry mount a few times
mgr/cephadm: migrate all legacy nfs exports to new .nfs pool
mgr/nfs: adjust cephfs export caps if necessary
python-common: don't accept pool/ns for NFSServiceSpec
mgr/orchestrator: drop rados_config_location ServiceDescription property
mgr/cephadm: move rados_config_location() out of NFSServiceSpec
mgr/nfs: change nfs pool to .nfs
mgr/nfs/export: accept a JSON or ganesha EXPORT config
mgr/nfs: allow 'nfs export apply' to take a list of exports
python-common: remove pool + namespace from NFSServiceSpec
mgr/nfs: used fixed pool + ns
mgr/rook: used fixed pool + ns
mgr/dashboard: use fixed pool + ns
mgr/cephadm: always use fixed pool and namespace
mgr/nfs: adjust test to match pool name
mgr/nfs: always create ganesha pool with well-defined name

Reviewed-by: Varsha Rao <varao@redhat.com>

cmake: add transitive dependency on legacy-option-headers

the legacy-option-headers target is only marked as a dependency of the
common-objs and common-common-objs. because those targets are OBJECT
libraries, their dependencies aren't inherited by the targets that link
common-objs or common-common-objs

this adds the dependencies manually, so that changes to the config
yaml files will cause legacy-option-headers to regenerate the headers

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42290 from javicacheiro/fix_cephadm_selinux

cephadm: fetch the real selinux status

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Daniel Pivonka <dpivonka@redhat.com>

Merge pull request #42220 from dsavineau/issue_51546

mgr/cephadm/iscsi: check if dashboard is enabled

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

doc/mgr/nfs: fix 'export apply', pool name

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #42339 from cbodley/wip-revert-vstart-system

Revert "src/vstart: create rgw user with system flag"

Reviewed-by: Varsha Rao <varao@redhat.com>

Merge pull request #41300 from rhcs-dashboard/force-maintenance-test-error

mgr/dashboard: Fix test_error force maintenance dashboard check

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

qa: Add SoftIron Linux as valid debian-based distro in test scripts

Signed-off-by: Danny Abukalam <danny@softiron.com>

build: Add SoftIron Linux as valid debian-based distro in install-deps

Signed-off-by: Danny Abukalam <danny@softiron.com>

common: optimize op_target_t and hobject_t constructors

Switch to passing on arguments by reference instead of copying them.

Signed-off-by: Or Ozeri <oro@il.ibm.com>

Merge pull request #42334 from tchaikov/wip-crimson-logging

crimson/common/log: print out logger.debug() when log level >=6

Reviewed-by: Mark Nelson <mnelson@readhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #42337 from rzarzynski/wip-crimson-alienstore-dedicated-subsys

crimson/os: give AlienStore its own debug subsystem.

Reviewed-by: Mark Nelson <mnelson@readhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42251 from adamemerson/wip-robustly-invalidate

rgw: Robustly Invalidate

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

qa/suites/upgrade/pacific-x: do distro before tasks

Otherwise, we isntall new podman at the end, and the
container-selinux-policy package install triggers a bunch of selinux
errors.

Fixes: https://tracker.ceph.com/issues/50151
Signed-off-by: Sage Weil <sage@newdream.net>

PendingReleaseNotes: document workaround for NFS storage change

Signed-off-by: Sage Weil <sage@newdream.net>

qa/tasks/mgr/test_orchestrator_cli: fix test

Signed-off-by: Sage Weil <sage@newdream.net>

qa/suites/orch/cephadm/mgr-nfs-upgrade: add test for nfs migration

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: migrate nfs grace file

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs: migrate pre-pacific nfs.ganesha-foo clusters to nfs.foo

In octopus, the service is nfs.ganesha-$id instead of nfs.$id

Signed-off-by: Sage Weil <sage@newdream.net>

doc/cephfs/fs-nfs-exports: document new export apply capabilities

- import list
- import ganesha conf

Signed-off-by: Sage Weil <sage@newdream.net>

qa/tasks/cephfs/test_nfs: define NFS_POOL_NAME

We can't import from mgr_module.py from here, sadly.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs: use NFS_POOL_NAME in test_nfs.py

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs: test export apply on JSON list

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs: add test for ganesha conf apply/import

Signed-off-by: Sage Weil <sage@newdream.net>

qa/tasks/cephfs/test_nfs: retry mount a few times

It may take a moment for a ganesha to (re)configure itself with a new
export.  If a mount fails, retry a couple times.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: migrate all legacy nfs exports to new .nfs pool

Migrate all past NFS pools, whether they were created by mgr/nfs or by
the dashboard, to the new mgr/nfs .nfs pool.

Since this migrations relies on RADOS being available, we have to be a bit
careful here: we only attempt the migration from serve(), not during
module init.

After the exports are re-imported, we destroy existing ganesha daemons so
that new ones will get recreated.  This ensures the (new) daemons have
cephx keys to access the new pool.

Note that no attempt is made to clean up the old NFS pools.  This is out
of paranoia: if something goes wrong, the old NFS configuration data will
still be there.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs: adjust cephfs export caps if necessary

If we are importing an old export, we may find that the cephx user
existed but with the wrong caps.  Adjust caps in that case!

Signed-off-by: Sage Weil <sage@newdream.net>

python-common: don't accept pool/ns for NFSServiceSpec

All users are now gone.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/orchestrator: drop rados_config_location ServiceDescription property

I'm not sure what this was intended to be used for...

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: move rados_config_location() out of NFSServiceSpec

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs: change nfs pool to .nfs

This is a new pool that we can migrate all past NFS configuration to,
simplifying the migration process (and also allowing us to pick a
.-prefixed name).

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs/export: accept a JSON or ganesha EXPORT config

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs: allow 'nfs export apply' to take a list of exports

Signed-off-by: Sage Weil <sage@newdream.net>

python-common: remove pool + namespace from NFSServiceSpec

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs: used fixed pool + ns

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/rook: used fixed pool + ns

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/dashboard: use fixed pool + ns

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: always use fixed pool and namespace

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs: adjust test to match pool name

Signed-off-by: Sage Weil <sage@newdream.net>