mgr/dashboard: run cephadm-backend e2e tests with KCLI

Fixes: https://tracker.ceph.com/issues/51300
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #41919 from agayev/zoned-more-cleaning-support

os/bluestore: More support for cleaning zones.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #42100 from rzarzynski/wip-crimson-assert_moveable

crimson: introduce assert_moveable().

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41976 from tchaikov/wip-crimson-alienstore-lockless

crimson/os: use lockfree queue for sharded queue

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge PR #42101 into master

* refs/pull/42101/head:
cephsqlite: add comment on atexit

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/os/alienstore: use semaphore to manage tasks in thread pool

* implement std::counting_semaphore in C++17
* use the homebrew counting_semaphore as the synchronization primitive
  to access the pending tasks, for better performance than the
  implementation using mutex and condition_variable.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cephsqlite: add comment on atexit

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

crimson: introduce assert_moveable().

In C++ `std::moving` a `const`-qualified value yields a constant
r-value reference (`const T&&`) which won't be matched with a callable
taking non-constant r-value reference (like move constructors) but
can play with one taking a constant l-value reference (like copy
constructors do). This behaviour is surprising especially in lambas
where adding or removing the `mutable` specifier may lead to different
behaviour. The problem isn't obvious and it's easy to wrongly drop
the `mutable` druing a clean-up. Therefore introducing a tool for
developers to fail at compile-time if that happens seems desired.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #42002 from tchaikov/wip-common-armor

common/armor: mark dst_end a const pointer

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #42035 from batrick/i51372

libcephsqlite: shutdown RADOS in destructor

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42094 from zdover23/wip-doc-upgrading-ceph-monitoring-the-upgrade-2021-06-29

doc/cephadm: improving "Monitoring the Upgrade"

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #42079 from zdover23/wip-doc-upgrading-ceph-first-section-2021-06-29

doc/cephadm: improve "Upgrading Ceph" (main)

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

doc/cephadm: improving "Monitoring the Upgrade"

This PR improves the section "Monitoring the Upgrade"
in the "Upgrading Ceph" chapter of the cephadm documentation.

This PR introduces a couple of section breaks with signposting
information in their titles, and rewrites some sentences in order
to reduce the cognitive load of the reader.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #42042 from tchaikov/wip-51375

common/options: convert a millisecs opt to a chrono::milliseconds and cleanups

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/os: use lockfree queue for sharded queue

each sharded queue has multiple producers and a single consumer queue,
but the producer side is in seastar world, so ideally, we should not
guard the queue using a POSIX lock.

in this change, the lockfree multiple-writer/multiple-reader queue is
used to replace the std::queue to drop the lock guarding it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr,mon: s/boost::optional/std::optional/

since the encoding schemes of boost::optional and std::optional are
identical, they can be used interchangeably.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/config: s/boost::variant/std::variant/

we should use standard library for more well defined behavior, and
less dependencies on 3rd party libraries.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/options: s/boost::variant/std::variant/

we should use standard library for more well defined behavior, and
less dependencies on 3rd party libraries.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42013 from ronen-fr/wip-ronenf-scrubs-config

qa/suites/rados: add simultaneous scrubs to the thrasher

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #42031 from sebastian-philipp/normalize_image_digest_local-registry

cephadm: Fix normalize_image_digest for local registries

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #41933 from liewegas/fix-48311

mgr: set debug_mgr=2/5 (so INFO goes to mgr log by default)

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41999 from tchaikov/wip-rpm-deb

rpm,debian: cleanups related to python3-setuptools dependencies

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #42076 from liu-chunmei/crimson-pgnls

crimson: fix pgnls exception

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42039 from liu-chunmei/crimson-cmpxattr

crimson: add cmp_xatt support

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson: fix pgnls exception

has_pg_op is always false, since m->ops is empty at that time.
so pgnls operation will go to process_op and report unknown operations.
move m->finish_decode ahead to fill m->ops.

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

crimson: add cmp_xatt support

fix Exceptional future ignored: std::runtime_error (op 'cmpxattr' not supported)

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

doc/cephadm: improve "Upgrading Ceph" (main)

This PR makes a couple of minor improvements to the text under the
top-level section "Upgrading Ceph" in the "Upgrading Ceph" chapter of
the cephadm documentation.

This one, mercifully, contains only a couple of changes.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge PR #42011 into master

* refs/pull/42011/head:
mds: just respawn mds daemon when osd op requests timeout

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #41988 into master

* refs/pull/41988/head:
logrotate: include cephfs-mirror daemon
cephfs-mirror: reopen logs on SIGHUP

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #41917 into master

* refs/pull/41917/head:
mgr/mgr_util: switch using unshared cephfs connections whenever possible

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>

Merge PR #41849 into master

* refs/pull/41849/head:
mds: try to flush the mdlog when requesting the rdlock

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #42038 into master

* refs/pull/42038/head:
mds: fix compile warning

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42049 from zdover23/wip-doc-cephadm-serve-man-disable-auto-deploy-of-daemons

doc/cephadm: enrich "Disabling Automatic Deploy..."

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #41681 from TRYTOBE8TME/wip-rgw-dpp

src/rgw: DPP addition

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #42050 from rzarzynski/wip-crimson-alienstore-fix-attrs-conv

crimson/os: fix memory corruption in AlienStore::get_attrs().

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/os: fix memory corruption in AlienStore::get_attrs().

`FuturizedStore` and `ObjectStore` use different memory layout for
conveying object attributes: map of `bufferlists` and map of `bptrs`
respectively. Unfortunately, `AlienStore` was trying to solve this
mismatch with just a `reinterpret_cast`.

Very likely this problem was the root cause behind the observed
crashes in `PGBackend::load_matadata` like the following one:

```
2021-06-15T09:25:07.511 INFO:journalctl@ceph.osd.3.smithi100.stdout:Jun 15 09:24:19 smithi100 conmon[54917]: DEBUG 2021-06-15 09:24:19,199 [shard 0] osd - peering_event(id=412, detail=PeeringEvent(from=7 pgid=5.14 sent=49 requested=49 evt=epoch_sent: 49 epoch_requested: 49 MInfoRec from 7 info: 5.14( v 45'2 (0'0,45'2] local-lis/les=48/49 n=0 ec=44/44 lis/c=48/44 les/c/f=49/45/0 sis=48) pg_lease_ack(ruub 19.176788330s))): complete
2021-06-15T09:25:07.511 INFO:journalctl@ceph.osd.3.smithi100.stdout:Jun 15 09:24:19 smithi100 conmon[54917]: Segmentation fault on shard 0.
2021-06-15T09:25:07.511 INFO:journalctl@ceph.osd.3.smithi100.stdout:Jun 15 09:24:19 smithi100 conmon[54917]: Backtrace:
2021-06-15T09:25:07.511 INFO:journalctl@ceph.osd.3.smithi100.stdout:Jun 15 09:24:19 smithi100 conmon[54917]:  0# 0x000055C99757FFBF in /usr/bin/ceph-osd
2021-06-15T09:25:07.511 INFO:journalctl@ceph.osd.3.smithi100.stdout:Jun 15 09:24:19 smithi100 conmon[54917]:  1# FatalSignal::signaled(int, siginfo_t const*) in /usr/bin/ceph-osd
2021-06-15T09:25:07.511 INFO:journalctl@ceph.osd.3.smithi100.stdout:Jun 15 09:24:19 smithi100 conmon[54917]:  2# FatalSignal::install_oneshot_signal_handler<11>()::{lambda(int, siginfo_t*, void*)#1}::_FUN(int, siginfo_t*, void*) in /usr/bin/ceph-osd
2021-06-15T09:25:07.512 INFO:journalctl@ceph.osd.3.smithi100.stdout:Jun 15 09:24:19 smithi100 conmon[54917]:  3# 0x00007F34BB632B20 in /lib64/libpthread.so.0
2021-06-15T09:25:07.512 INFO:journalctl@ceph.osd.3.smithi100.stdout:Jun 15 09:24:19 smithi100 conmon[54917]:  4# 0x000055C99263D4D2 in /usr/bin/ceph-osd
2021-06-15T09:25:07.512 INFO:journalctl@ceph.osd.3.smithi100.stdout:Jun 15 09:24:19 smithi100 conmon[54917]:  5# 0x000055C992740E47 in /usr/bin/ceph-osd
2021-06-15T09:25:07.512 INFO:journalctl@ceph.osd.3.smithi100.stdout:Jun 15 09:24:19 smithi100 conmon[54917]:  6# seastar::continuation<seastar::internal::promise_base_with_type<std::unique_ptr<PGBackend::loaded_object_md_t, std::default_delete<PGBackend::loaded_object_md_t> > >, seastar::noncopyable_function<crimson::errorator<crimson::unthrowable_wrapper<std::error_code const&, crimson::ec<(std::errc)84> > >::_future<crimson::errorated_future_marker<std::unique_ptr<PGBackend::loaded_object_md_t, std::default_delete<PGBackend::loaded_object_md_t> > > > (seastar::future<std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<void>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > > >&&)>, seastar::future<std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<void>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > > >::then_wrapped_nrvo<crimson::errorator<crimson::unthrowable_wrapper<std::error_code const&, crimson::ec<(std::errc)84> > >::_future<crimson::errorated_future_marker<std::unique_ptr<PGBackend::loaded_object_md_t, std::default_delete<PGBackend::loaded_object_md_t> > > >, seastar::noncopyable_function<crimson::errorator<crimson::unthrowable_wrapper<std::error_code const&, crimson::ec<(std::errc)84> > >::_future<crimson::errorated_future_marker<std::unique_ptr<PGBackend::loaded_object_md_t, std::default_delete<PGBackend::loaded_object_md_t> > > > (seastar::future<std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<void>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > > >&&)> >(seastar::noncopyable_function<crimson::errorator<crimson::unthrowable_wrapper<std::error_code const&, crimson::ec<(std::errc)84> > >::_future<crimson::errorated_future_marker<std::unique_ptr<PGBackend::loaded_object_md_t, std::default_delete<PGBackend::loaded_object_md_t> > > > (seastar::future<std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<void>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > > >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<std::unique_ptr<PGBackend::loaded_object_md_t, std::default_delete<PGBackend::loaded_object_md_t> > >&&, seastar::noncopyable_function<crimson::errorator<crimson::unthrowable_wrapper<std::error_code const&, crimson::ec<(std::errc)84> > >::_future<crimson::errorated_future_marker<std::unique_ptr<PGBackend::loaded_object_md_t, std::default_delete<PGBackend::loaded_object_md_t> > > > (seastar::future<std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<void>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > > >&&)>&, seastar::future_state<std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<void>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > > >&&)#1}, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<void>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > > >::run_and_dispose() in /usr/bin/ceph-osd
2021-06-15T09:25:07.512 INFO:journalctl@ceph.osd.3.smithi100.stdout:Jun 15 09:24:19 smithi100 conmon[54917]:  7# 0x000055C99CFD195F in /usr/bin/ceph-osd
2021-06-15T09:25:07.513 INFO:journalctl@ceph.osd.3.smithi100.stdout:Jun 15 09:24:19 smithi100 conmon[54917]:  8# 0x000055C99CFD6EA0 in /usr/bin/ceph-osd
2021-06-15T09:25:07.513 INFO:journalctl@ceph.osd.3.smithi100.stdout:Jun 15 09:24:19 smithi100 conmon[54917]:  9# 0x000055C99D188F0B in /usr/bin/ceph-osd
2021-06-15T09:25:07.513 INFO:journalctl@ceph.osd.3.smithi100.stdout:Jun 15 09:24:19 smithi100 conmon[54917]: 10# 0x000055C99CCE698A in /usr/bin/ceph-osd
2021-06-15T09:25:07.513 INFO:journalctl@ceph.osd.3.smithi100.stdout:Jun 15 09:24:19 smithi100 conmon[54917]: 11# 0x000055C99CCF0AAE in /usr/bin/ceph-osd
2021-06-15T09:25:07.513 INFO:journalctl@ceph.osd.3.smithi100.stdout:Jun 15 09:24:19 smithi100 conmon[54917]: 12# main in /usr/bin/ceph-osd
2021-06-15T09:25:07.513 INFO:journalctl@ceph.osd.3.smithi100.stdout:Jun 15 09:24:19 smithi100 conmon[54917]: 13# __libc_start_main in /lib64/libc.so.6
2021-06-15T09:25:07.514 INFO:journalctl@ceph.osd.3.smithi100.stdout:Jun 15 09:24:19 smithi100 conmon[54917]: 14# _start in /usr/bin/ceph-osd
2021-06-15T09:25:07.514 INFO:journalctl@ceph.osd.3.smithi100.stdout:Jun 15 09:24:19 smithi100 conmon[54917]: Fault at location: 0x31dfff8000
2021-06-15T09:25:07.514 INFO:journalctl@ceph.osd.3.smithi100.stdout:Jun 15 09:24:20 smithi100 podman[55356]: 2021-06-15 09:24:20.230341885 +0000 UTC m=+0.072958807 container died a3ea2a1d0a176286b93b8f5b94458982b9038e70d09128fb55f53b92976f0c42 (image=quay.ceph.io/ceph-ci/ceph@sha256:13ae953e3f83ee011d784d6eb9126fdc692f5bb688fe7d918be61ca7a7282b3c, name=ceph-43579b90-cdba-11eb-8c13-001a4aab830c-osd.3)
```

The fix deals with the issue by wrapping the `bptrs` in `bufferlists`.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #41989 from zdover23/wip-doc-cephadm-serve-man-deploy-of-daemons-2021-06-24

doc/cephadm: enrich "deployment of daemons"

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

doc/cephadm: enrich "Disabling Automatic Deploy..."

This PR rewrites and reformats the section "Disabling Automatic
Deployment of Daemons" in the "Service Management" chapter of the
cephadm guide.

I've rewritten some sentences, removed some "please"s, and added
some section titles so that the content in this is better
signposted.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #41998 from kevinzs2048/arm64-rwl-cache-optional

ceph.spec.in, debian/rules: enable rbd-rwl-cache by default only on x86_64

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42021 from tchaikov/wip-rpm-memory-constraint

ceph.spec.in: increase memory per core to 3000MB on SUSE distros

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

include/uuid: define operator>() for uuid_d

it can be used by Option::validate() if
operator>(const variant<>&, const variant<>&) requires this.

Signed-off-by: Kefu Chai <kchai@redhat.com>

msg/msg_types: add operator>() for entity_addrvec_t

it can be used by Option::validate() if
operator>(const variant<>&, const variant<>&) requires this.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/options: s/boost::lexical_cast/std::stoull/

we should use standard library for more well defined behavior, and
less dependencies on 3rd party libraries.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/options: convert a millisecs opt to a chrono::milliseconds when paring it

Option always parses a new string value and convert it to a value_t
before validating it. and value_t is an alias of boost::variant<...>.
and we use "new_value < min" to tell if the new_value is out of the
bound or not, where both "new_value" and "min" are instances of
value_t. so it is critcal that these two values contain the same type
of value, otherwise boost::variant::operator< would

> Returns:
> If which() == rhs.which() then: content_this < content_rhs, where content_this is the content of *this and content_rhs is the content of rhs. Otherwise: which() < rhs.which().

where which() indicates which type of value is contained in the value_t.

before this change, instead of converting a new value of milliseconds to
std::chrono::milliseconds, we convert it to an uint64_t, whose index in
the value_t is 2, while the milliseconds value's index is 9, so
"new_value < min" evaluates to true even if "new_value" is 100 and "min"
is 30.

after this change, the new value of a milliseconds option is converted
to std::chrono::milliseconds, so it is comparable with its min value and
max value.

a minimal test is added to reproduce this issue.

the change which added the support of millisec to option was
29690a338ba4482d187e6036903e138437ae3bb4 which is not included by any
LTS branches, so no need to backport this fix.

Fixes: https://tracker.ceph.com/issues/51375
Signed-off-by: Kefu Chai <kchai@redhat.com>

libcephsqlite: register atexit handler for cleanup

We need to tear down the ceph vfs when sqlite3 (or other binaries) call
exit(). Otherwise global state gets destructed which can cause library
threads to segfault or raise exceptions.

Also pull vfs struct out of appdata. We need to be able to detect double
calls to the atexit handler which, sadly, can happen.

Fixes: https://tracker.ceph.com/issues/50503
Fixes: https://tracker.ceph.com/issues/51372
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

libcephsqlite: shutdown RADOS in destructor

Fixes: https://tracker.ceph.com/issues/50503
Fixes: https://tracker.ceph.com/issues/51372
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #41574 into master

* refs/pull/41574/head:
qa/tasks/vstart_runner: add LocalCluster.run
qa/tasks/cephfs/test_nfs: fiddle with sudo
mgr/nfs/export: some cleanup, minor refactoring
mgr/nfs/cluster: remove unused @cluster_setter
nfs/mgr: fix help message case
doc/cephfs/fs-nfs-export: add note about export update behavior
mgr/nfs: move user create/delete into helper
mgr/nfs: refactor _delete_user helper
mgr/nfs: refactor create_export_from_dict() helper
mgr/nfs: keep 'nfs export get' around for backward-compat
mgr/nfs: rename method
qa/tasks/cephfs/test_nfs: test new export via apply
doc/cephfs/fs-nfs-export: be consistent with cluster_id and _ vs -
mgr/nfs: addr -> client_addr for 'nfs export create ...'
mgr/nfs: fix tests
mgr/nfs: 'nfs export get' -> 'nfs export info'
mgr/nfs: binding -> pseudo_path
mgr/nfs: more revisions based on review
mgr/nfs: adjust NFSExceptoin errno arg
doc/cephfs: update 'nfs export {get,apply}' docs
mgr/nfs: merge FSExport back into ExportMgr
doc/radosgw/nfs: document mgr/nfs way to add/remove rgw exports
mgr/nfs: merge 'nfs export {update,import}' -> 'nfs export apply'
mgr/nfs: test export creation and list
mgr/nfs: test export_update (+ fixes)
mgr/nfs: test Export.validate(); several fixes
mgr/nfs: test that export <-> block+dict conversions go both ways
mgr/nfs: clean up test a bit
mgr/nfs/export: fix export validation
mgr/nfs/export: fix tests
mgr/nfs: handle option addr/client block in create_export()
mgr/nfs: allow multiple addrs for new exports
mgr/nfs: fix/finish rgw export
mgr/nfs/module: clusterid -> cluster_id
mgr/nfs/export: fix export_update_1 to type check
mgr/nfs/cluster: fix type error
mgr/nfs/export: wrap long lines
mgr/nfs: ExportMgr._delete_export only works for cephfs for now
mgr/nfs: Remove pool_ns from NFSCluster
mgr/nfs: Remove ExportMgr.rados_namespace
mgr/nfs: flake8
mgr/nfs: Add type checking
mgr/nfs: Add __eq__ method to Export
mgr/nfs: Add some compatibility to mgr/dashboard
mgr/nfs: Fix whitespace handling
mgr/nfs: Copy unit tests from mgr/dashboard
mgr/nfs: partially implement rgw export support
mgr/nfs: abstract FSAL; add RGWFSAL
mgr/nfs: refactor to merge 'update' and 'import' code
mgr/nfs: add 'nfs export import' command
mgr/nfs: refactor 'nfs export update' and export validation
mgr/nfs: fix _fetch_export to distinguish between clusters
mgr/nfs: move export ganesha conf translation into caller
mgr/nfs: name nfs cephfs client key 'nfs.{cluster_id}.{export_id}'
mgr/nfs: add --addr to 'nfs export create'
mgr/nfs: add --squash to 'nfs export create'
mgr/nfs/export_utils: include false but non-None items in config
vstart.sh: enable nfs module
mgr/cephadm: nfs: drop attr_expiration_time from top-level config
mgr/cephadm: remove Dir_Chunk = 0

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #41937 from liewegas/mgr-crash

mgr: generate crash dumps for Python exceptions in mgr modules

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41946 from liewegas/fix-51294

mgr/devicehealth: fix _get_device_metrics ValueError

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/tasks/vstart_runner: add LocalCluster.run

Signed-off-by: Sage Weil <sage@newdream.net>

qa/tasks/cephfs/test_nfs: fiddle with sudo

- no sudo for 'ceph' commands
- explicit sudo for _sys_cmd (things like 'rados' don't need sudo!)

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs/export: some cleanup, minor refactoring

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs/cluster: remove unused @cluster_setter

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #41977 from rzarzynski/wip-crimson-common-print-more-on-crash

crimson/common: dump more on faults

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mds: fix compile warning

    ../src/mds/Server.cc: In member function ‘void Server::handle_set_vxattr(MDRequestRef&, CInode*)’:
    ../src/mds/Server.cc:5703:18: warning: unused variable ‘realm’ [-Wunused-variable]
           SnapRealm *realm = cur->find_snaprealm();
                      ^~~~~

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

nfs/mgr: fix help message case

Signed-off-by: Sage Weil <sage@newdream.net>

doc/cephfs/fs-nfs-export: add note about export update behavior

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs: move user create/delete into helper

- Do user create or delete via a helper
- Defer until after we have validated the Export (on create or update)
- Support updates to user_id, which is needed to keep the naming consistent
and to also support changing the bucket, since the user_id is derived
from that.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #41838 from p-se/grafana-clean-up

monitoring: Clean up Grafana dashboards

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: jan--f <NOT@FOUND>
Reviewed-by: p-se <NOT@FOUND>
Reviewed-by: Paul Cuzner <pcuzner@redhat.com>

qa/suites/rados/mgr: whitelist module crash during selftest

One of the selftests triggers an exception from serve().

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #41721 from aaryanporwal/telemetry-ident-fix

mgr/dashboard: telemetry activate: show ident fields when checked

Reviewed-by: aaryanporwal <NOT@FOUND>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #41991 from dang/wip-dang-bucket-delete

RGW - Bucket Remove Op: Pass in user

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #41993 from ronen-fr/wip-ronenf-50346

osd/scrub: replace a ceph_assert() with a test

Reviewed-by: Neha Ojha <nojha@redhat.com>

cephadm: Fix normalize_image_digest for local registries

Cause they typically don't have dots in it.

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #42024 from rzarzynski/wip-crimson-load_obc_nocpy

crimson/osd: don't extra copy hobject in PG::load_head_obc().

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: don't extra copy hobject in PG::load_head_obc().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

ceph.spec.in: increase memory per core to 3000MB on SUSE distros

in the KVM instance offered by OBS, we have

[  346s] + cat /proc/meminfo
[  347s] MemTotal:       10167736 kB
[  347s] MemFree:         4983964 kB
[  347s] MemAvailable:    9826800 kB
[  347s] Buffers:           85856 kB
[  347s] Cached:          4615192 kB
[  347s] SwapCached:            0 kB
...
[  347s] SwapTotal:       2097148 kB

and its number of hardware threads is

[  346s] ++ /usr/bin/getconf _NPROCESSORS_ONLN
[  346s] + _threads=8

so ($MemTotal+$SwapTotal)/1024/2600 = 4.6, which is less
than the # of threads, so "4" was used for the number of jobs.

but per our recent observation in
38be14bc0fa32be6877dea08ebd35495d39e464f, some compiling jobs could
take up to 3GB. in the OOM failure in OBS, we had

[24915s] [24848.843594] Out of memory: Killed process 16894 (cc1plus) total-vm:4293756kB, anon-rss:2970012kB, file-rss:0kB, shmem-rss:0kB, UID:399 pgtables:8324kB oom_score_adj:0

where 4GiB memory was allocated, in which 3GiB was mapped into
memory. this matches with our findings.

in this change, the memory per core is bumped up to 3000MB
in hope to address the OOB. the downside of this change is
that it would take even longer to finish the build if the
building host is limited in memory.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41615 from tchaikov/wip-avl-alloc-ff

os/bluestore/AvlAllocator: introduce bluestore_avl_alloc_ff_max_* options

Reviewed-by: Igor Fedotov <ifedotov@suse,com>
Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #38939 from ronen-fr/wip-ronenf-scrub-blocked

osd: issue a warning if the scrubber blocks for too long on an object

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #40850 from varshar16/wip-vstart-support-cephadm-rgw

src/vstart: deploy rgw service with cephadm and create rgw user with system flag

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42020 from athanatos/sjust/wip-cache-assert

crimson/os/seastore: transaction conflict handling improvements

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #42003 from cyx1231st/wip-seastore-fix-onode-tree

crimson/onode-staged-tree: fix ref-counter assert failures

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/common: dump entire siginfo on segmentation fault.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/common: FatalSignal::signaled() takes siginfo by a reference.

There is no point in having the distincted `nullptr` value.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/common: dump /proc/self/maps on crash.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mds: just respawn mds daemon when osd op requests timeout

Fixes: https://tracker.ceph.com/issues/51280
Signed-off-by: Xiubo Li <xiubli@redhat.com>

ceph.spec.in, debian/rules: Set rbd-rwl-cache optional on arm64 and ppc64le

set rwl cache option on arm64 and ppc64le as PMDK is not well supported.
Currently, only 64-bit Linux* and Windows* on x86 are supported PMDK

Reference:
1. Experimental support on Arm64, but lacking of librpmem:
See: https://github.com/pmem/pmdk#experimental-support-for-64-bit-arm
2. No RPM for PMDK on Arm64:
See: https://bugzilla.redhat.com/show_bug.cgi?id=1340635
3. > Does PMDK support ARM64*?
   > Currently only 64-bit Linux* and Windows* on x86 are supported.
See: https://software.intel.com/content/www/us/en/develop/articles/persistent-memory-faq.html
4. Make check fail on Arm64
See: https://github.com/pmem/pmdk/issues/5255

Fixes: https://tracker.ceph.com/issues/51339
Signed-off-by: Kevin Zhao <kevin.zhao@linaro.org>

Merge pull request #41889 from ChenFanTony/mkfs_wait_complete

osd/OSD: mkfs need wait for transcation completely finish

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/onode-staged-tree: reset root node after lookup

Otherwise there could be unexpected references that will break the
asserts when remove nodes during insert/delete.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: add missing mutable keyword

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #42004 from tchaikov/wip-crimson-osd-fsm

crimson/osd: shutdown if osdmap forces us to do so

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

seastore/.../staged_fltree/node: check for conflict in Node::load

This will be unnecessary once converted to interruptible_future.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager/btree/lba_btree_node_impl: add debugging

Signed-off-by: Samuel Just <sjust@redhat.com>

seastore/.../node_extent_manager/seastore: detect transaction conflicts in read_extent

This won't be necessary once converted to interruptible_future.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: mark conflict in get_extent

After wait_io, the extent may have been mutated again, so it may be
invalid.  Check in the caller and mark the transaction conflicted as
needed.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/transasction: expose is_conflicted

Useful for components not yet converted to use interruptible_future.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #41963 from athanatos/sjust/wip-interruptible-tm

crimson/os/seastore: refactor transaction_manager and below to use interruptible_future

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/devicehealth: fix _get_device_metrics ValueError

This appears to have broken with abd35d47696c208990355395d48c1c1e261de95c

The SQL OR doesn't work because in the case that sample is passed,
_t2epoch(min_sample) is 0 and the 0 <= time portion of the expression
is always true.

Fixes: https://tracker.ceph.com/issues/51294
Signed-off-by: Sage Weil <sage@newdream.net>

test/crimson/test_interruptible_future: disable handle_error

Seems to cause a linker hang with gcc-9 in bionic.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/transaction_manager: pass t by ref to submit_transaction

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #39934 from Jeegn-Chen/wip-tracker-49128

rgw: write meta of a MP part to a correct pool

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #41739 from liewegas/rgw-realm-metadata

radosgw: include realm_{id,name} in service map

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/suites/rados: add simultaneous scrubs (multiple options) to the thrasher

Setting osd-max-scrubs to either 2 or 3.

Triggered by https://tracker.ceph.com/issues/50346

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

RGW - Bucket Remove Op: Pass in user

When a bucket remove op is called on the non-master zone, the op is
forwarded to the master zone, but this needs a user, so pass the user
in.

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #41994 from anthonyeleven/anthonyeleven/adjust-rados-operations-pools

doc/rados/operations: Update pools.rst

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #42005 from trociny/wip-51342

test/librbd: use really invalid domain

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

ceph.spec.in: enable --with-rbd_ssd_cache by default

unlike rbd_rwl_cache, rbd_ssd_cache does not depend on pmdk (libpmem),
so let's enable it on all supported architecture and rpm based distros.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/librbd: use really invalid domain

in TestMockMigrationHttpClient.OpenResolveFail

Fixes: https://tracker.ceph.com/issues/51342
Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #41828 from tchaikov/wip-btree-alloc

os/bluestore: add BtreeAllocator

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

crimson/osd: document fsm of crimson osd

Signed-off-by: Kefu Chai <kchai@redhat.com>