crimson/net: Use out_q instead of pending_q

pending_q contains the same messages as in out_q and it is only used
for creating a bytestream out of these messages. We can just use out_q for that.

Signed-off-by: Amnon Hanuhov <ahanukov@redhat.com>

Merge pull request #40831 from vshankar/wip-cephfs-mirror-incremental-sync

cephfs-mirror: incremental sync

Reviewed-by: Milind Changire <mchangir@redhat.com>

Merge pull request #41279 from pkalever/promote-attach

rbd: promote rbd-nbd attach and detach at rbd integrated cli

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #41378 from varshar16/wip-check-file-inputs-nfs

pybind/mgr: generalize CLICheckNonemptyFileInput() error msg

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Waad Alkhoury <walkhour@redhat.com>

Merge pull request #41381 from AmnonHanuhov/wip-Refactor_PeeringState

crimson/osd: Refactor PeeringState

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #41516 from tchaikov/wip-47380

mon/OSDMonitor: drop stale failure_info even if can_mark_down()

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #41546 from tchaikov/wip-crush-alignment

crush/crush: ensure alignof(crush_work_bucket) is 1

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #41517 from tchaikov/wip-osd-osd-types

osd/osd_type: use f->dump_unsigned() when appropriate

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #41527 from t-msn/cleanup-peeringstate-init

osd/PeeringState: cleanup dead code in PeeringState::init

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41565 from anthonyeleven/anthonyeleven/update-rgw-chunk

doc/radosgw: modernize reference to rgw_max_chunk_size

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/radosgw: modernize reference to rgw_max_chunk_size

The value changed from 512KB to 4MB in Kraken.  Reference the prevailing
option default instead of embedding the current value.

Signed-off-by: Anthony D'Atri anthony.datri@gmail.com

Merge pull request #41564 from tchaikov/wip-dmclock-seastar

dmclock: pick up change to support seastar

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #41560 from athanatos/sjust/wip-clang-linker-problem

crimson/os/seastore: resolve clang build problems, misc cleanups

Reviewed-by: Kefu Chai <kchai@redhat.com>

dmclock: pick up change to support seastar

so if WITH_SEASTAR is defined, the POSIX synchronous primitives
are either replaced with seastar counterparts or disabled.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os/seastore/seastore: add helpers to simplify omap usage

Add _omap_get_values and _omap_get_value to clarify omap_get_values and
get_attr.  Also resolves a clang linker error.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: use tuple return for omap_list throughout

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/seastore.h: remove unncessary whitespace

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/seastore.h: remove non-const repeat_with_onode

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/futurized_store: use ceph::bufferlist to match

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge PR #41351 into master

* refs/pull/41351/head:
cephadm: clean-up error message
cephadm: raise an error when `--config` file is not found

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #41283 into master

* refs/pull/41283/head:
cephadm: manage cephadm log with logrotated

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #41465 from adamemerson/wip-50169

rgw: Simplify log shard probing and err on the side of omap

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #41554 from rzarzynski/wip-crimson-simplify-ox-lt-mgmt

crimson/osd: simplify the management of OpsExecuter's life-time.

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: simplify the management of OpsExecuter's life-time.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #41510 into master

* refs/pull/41510/head:
doc/cephfs/nfs: remove documented limitation

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #41547 from t-msn/wip-update-cephspec

ceph.spec.in: install gcc-toolset-9-gcc-c++ for rhel only

Reviewed-by: Kefu Chai <kchai@redhat.com>

crush/crush: ensure alignof(crush_work_bucket) is 1

in do_rule(), we allocate the space for crush_work_bucket using

char work[crush_work_size(crush, maxout)];

where crush_work_size() calculate the size like:

map->working_size + result_max * 3 * sizeof(__u32);

so work is allocated on stack, but the alignment of the
crush_work_bucket struct is not taken into consideration, so in
crush_init_workspace(), point could point to an address which is not
aligned to 8 bytes, which is the alignment of crush_work_bucket by
default. so is its member variables, all of them are uint32_t, and hence
are also 8-bytes aligned.

to ensure the compiler generate the correct assembly for accessing
the member variables without assuming that the struct is 8-byte
aligned, we should specify the alignment explicitly.

in this change, `__attribute__ ((packed))` is specified for
crush_work_bucket, so that its alignment is 1.

this issue is spotted by ASan, it complains like:

../src/crush/mapper.c:881:22: runtime error: member access within misaligned address 0x7ffe051f90dc for type 'struct crush_work_bucket', which requires 8 byte alignment
0x7ffe051f90dc: note: pointer points here
  1d e5 77 3d 68 55 00 00  00 00 00 00 00 00 00 00  20 93 1f 05 fe 7f 00 00  10 91 1f 05 fe 7f 00 00
              ^
../src/crush/mapper.c:882:22: runtime error: member access within misaligned address 0x7ffe051f90dc for type 'struct crush_work_bucket', which requires 8 byte alignment
0x7ffe051f90dc: note: pointer points here
  1d e5 77 3d 00 00 00 00  00 00 00 00 00 00 00 00  20 93 1f 05 fe 7f 00 00  10 91 1f 05 fe 7f 00 00
              ^
../src/crush/mapper.c:883:20: runtime error: member access within misaligned address 0x7ffe051f90dc for type 'struct crush_work_bucket', which requires 8 byte alignment
0x7ffe051f90dc: note: pointer points here
  1d e5 77 3d 00 00 00 00  00 00 00 00 00 00 00 00  20 93 1f 05 fe 7f 00 00  10 91 1f 05 fe 7f 00 00
              ^

Fixes: https://tracker.ceph.com/issues/50978
Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph.spec.in: install gcc-toolset-9-gcc-c++ for rhel only

Otherwise fedora 33 complains there is no gcc-toolset-9-gcc-c++
when running "WITH_SEASTAR=true ./install_deps.sh"

Related to: 36759b53635
Signed-off-by: Misono Tomohiro <misono.tomohiro@jp.fujitsu.com>

Merge pull request #41545 from tchaikov/wip-vstart-fix

vstart.sh: pass the addition option to parse_block_devs()

Reviewed-by: Samuel Just <sjust@redhat.com>

mgr/dashboard/access_control: fix flake8 expected 2 blank lines error

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/nfs: use CLICheckNonemptyFileInput decorator

Fixes: https://tracker.ceph.com/issues/50858
Signed-off-by: Varsha Rao <varao@redhat.com>

pybind/mgr: generalize CLICheckNonemptyFileInput() error msg

Signed-off-by: Varsha Rao <varao@redhat.com>

pybind/mgr: check if file contains only spaces

Signed-off-by: Varsha Rao <varao@redhat.com>

vstart.sh: use || instead of "-o"

to silence the warning like:

 SC2166: Prefer [ p ] || [ q ] as [ p -o q ] is not well defined.

see also
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/test.html

Signed-off-by: Kefu Chai <kchai@redhat.com>

vstart.sh: pass the addition option to parse_block_devs()

to address the regression introduced by
3ea5242e381a850c080ee9edbaeea28059ad4da9

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41543 from runsisi/wip-fix-clay-doc

doc: add missing crush-device-class={device-class} pair for clay code profile

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41542 from tchaikov/wip-vstart-cleanup

vstart: cleanups

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

qa/workunits/rbd: use rbd cli for device attach/detach commands

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>

rbd: improve conditional compilation specific checks

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>

rbd: promote rbd-nbd attach and detach at rbd integrated cli

Example:
$ rbd device attach rbd-pool/image --device /dev/nbd0 --device-type nbd --force
$ rbd device detach rpool/image --device-type nbd

for now returning EOPNOTSUPP with krbd, ggate and wnbd

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>

crimson/os/seastore/logging.h: use ##__VA_ARGS__ rather than __VA_OPT__

This seems to work with both clang and gcc for now.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../staged-fltree/tree_utils: fix cursor binding

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crimson/test_backfill: fix captured bindings

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crimson/seastore/transaction_manager_test_state: mark TestSegmentManagerWrapper final

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #41536 from rzarzynski/wip-crimson-fix-ox-lifetime

crimson/osd: extend lifetime of OpsExecuter to match all_completed

Reviewed-by: Xuehan Xu <xuxuehan@360.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

vstart: define helper for parsing block_devs options

for better readability, and to simplify the code

Signed-off-by: Kefu Chai <kchai@redhat.com>

vstart: use block_devs for {blue,sea}store_dev

so the logic handling bluestore_dev and seastore_dev can be merged.

Signed-off-by: Kefu Chai <kchai@redhat.com>

vstart: use here document to define usage

for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: add missing crush-device-class={device-class} pair for clay code profile

`crush-device-class` is optional for `ceph osd erasure-code-profile set`,
add it for the sake of completeness

Signed-off-by: luo.runbing <luo.runbing@zte.com.cn>

Merge pull request #41484 from liewegas/cephadm-progress

mgr/cephadm: progress item for service apply

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>
Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>

Merge pull request #41452 from ifed01/wip-ifed-fix-no-track

os/bluestore: track bluestore_warn_on_no_per_pg_omap cfg parameter ch…

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #41453 from ifed01/wip-ifed-stray-omap

os/bluestore: improve stray omap logging during fsck.

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #41398 from aclamk/wip-aclamk-avl-block-picker-improve

os/bluestore: Improve _block_picker function

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41488 from liewegas/cephadm-conf

config,mgr: expose ceph.conf path to mgr modules

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41459 from liewegas/devid-underscores

common/blkdev: remove double _'s from device_id

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41491 from tchaikov/wip-ceph-daemon-ann

pybind/ceph_daemon: add type annotations

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #41493 from tchaikov/wip-pybind-impport-from-future

pybind: do not import from "__future__" anymore

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #41498 from tchaikov/wip-cmake-libasan.6

cmake/modules/FindSanitizers: prefer libasan.6

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #41401 from rzarzynski/wip-crimson-injectdataerr

crimson/osd, common: implement the inject{m,}dataerr admin commands

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41541 from liu-chunmei/seastore-add-devs

crimson/seastore: add --seastore-devs in vstart.sh

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41533 from tchaikov/wip-doc-rgw-conf

doc/radosgw: use confval directive to define options

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

crimson/osd: implement the injectmdataerr admin command.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: implement the injectdataerr admin command.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/os: implement inject_{m,}data_error in AlienStore.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #41434 from tchaikov/wip-cmd-getval

common/cmdparse: use string_view for the key and return val by retval

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/seastore: add --seastore-devs in vstart.sh

to support /dev/xxx as seastore device

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

rgw: Simplify log shard probing and err on the side of omap

In the multigeneration version we no longer care whether entries
exist, since we never delete and recreate empty logs. Remove logic
that marked entirely empty shards as DNE under the assumption that
they would be deleted if so.

Fixes: https://tracker.ceph.com/issues/50169
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge PR #41007 into master

* refs/pull/41007/head:
qa/tasks/cephfs/test_nfs: fix info test
doc/cephfs/fs-nfs-exports: document --ingress --virtual-ip
mgr/nfs: move ingress vs virtual_ip check to cluster interface
PendingReleaseNotes: clarify deprecated
PendingReleaseNotes: note breaking CLI changes
doc/cephadm/nfs: document nfs+ingress
qa/suites/rados/cephadm/smoke-roleless: test nfs, nfs + ingress
mgr/nfs: take --ingress argument to 'nfs cluster create'
mgr/cephadm: adjust debug output for device refresh
mgr/cephadm: ingress: fix log msg
mgr/cephadm: fix logging of config/placement errors
common/options: enable nfs module for new clusters
cephadm: --stop-signal=SIGTERM
mgr/orchestrator: default nfs pool, namespaces
mgr/cephadm: nfs: create pool if it doesn't yet exist
doc/cephadm/nfs: update
mgr/nfs: change 'nfs cluster info'
mgr/nfs: take optional virtual_ip for deploying ingress
mgr/nfs: remove 'nfs cluster update'
mgr/nfs: factor out ganesha pool creation
mgr/nfs: delete -> rm for CLI
mgr/nfs: add some type annotations
python-common: fix IngressSpec yaml dump
mgr/cephadm: ingress: remove eth0 default
qa/tasks/cephadm: allow mounting volumes in shell
cephadm: add -v arg to shell
qa/tasks/vip: add 'vip.exec' task
mgr/orchestrator: add --port arg to 'orch apply nfs'
mgr/cephadm: nfs: add purge
mgr/cephadm: ingress: support nfs
mgr/cephadm: do not reconfigure daemons on deleted services
mgr/cephadm: nfs: shell out to rados tool for conf creation
mgr/cephadm: nfs: add rank to grace file from mgr module
mgr/cephadm: nfs: bind ganesha to appropriate ip:port
mgr/cephadm: enable ranked daemons for nfs
mgr/cephadm: support creation of daemons with ranks
mgr/cephadm: make _plan show removed daemon names
mgr/cephadm/schedule: assign/map ranks
mgr/cephadm: add rank[_generation] properties
mgr/cephadm/inventory: store optional rank_map along with specs
mgr/cephadm: include service_name is generated DaemonDescription
mgr/orchestrator: include service_name in DaemonDescription dump
mgr/cephadm/inventory: fix deleted check
mgr/cephadm: simplify
mgr/cephadm/schedule: make placement shuffle deterministic
mgr/cephadm: document CephadmService flags

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Varsha Rao <varao@redhat.com>

Merge PR #41539 into master

* refs/pull/41539/head:
doc/cephadm: fix prompts in service-management.rst

Reviewed-by: Sage Weil <sage@redhat.com>

doc/cephadm: fix prompts in service-management.rst

This PR formats the prompts in service-managment.rst
properly.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

doc/radosgw: use confval directive to define options

less repeating this way

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: extend lifetime of OpsExecuter to match all_completed.

f7181ab2f65803ecd8204f8f4f5aad4713b747f3 has optimized the client
parallelism. To achieve that `PG::do_osd_ops()` were converted to
return basically future pair of futures. Unfortunately, the life-
time management of `OpsExecuter` was kept intact. In the result,
the object was valid only till fullfying the outer future while,
due to the `rollbacker` instances, it should be available till
`all_completed` becomes available.

This issue can explain the following problem has been observed
in a Teuthology job [1].

```
DEBUG 2021-05-20 08:03:22,617 [shard 0] osd - do_op_call: method returned ret=-17, outdata.length()=0 while num_read=1, num_write=0
DEBUG 2021-05-20 08:03:22,617 [shard 0] osd - rollback_obc_if_modified: object 19:e17d4708:test-rados-api-smithi095-38404-2::foo:head got erro
r generic:17, need_rollback=false
=================================================================
==33626==ERROR: AddressSanitizer: heap-use-after-free on address 0x60d0000b9320 at pc 0x560f486b8222 bp 0x7fffc467a1e0 sp 0x7fffc467a1d0
READ of size 4 at 0x60d0000b9320 thread T0
    #0 0x560f486b8221  (/usr/bin/ceph-osd+0x2c610221)
    #1 0x560f4880c6b1 in seastar::continuation<seastar::internal::promise_base_with_type<boost::intrusive_ptr<MOSDOpReply> >, seastar::noncopy
able_function<crimson::interruptible::interruptible_future_detail<crimson::osd::IOInterruptCondition, crimson::errorator<crimson::unthrowable_
wrapper<std::error_code const&, crimson::ec<(std::errc)11> > >::_future<crimson::errorated_future_marker<boost::intrusive_ptr<MOSDOpReply> > >
 > ()>, seastar::future<void>::then_impl_nrvo<seastar::noncopyable_function<crimson::interruptible::interruptible_future_detail<crimson::osd::
IOInterruptCondition, crimson::errorator<crimson::unthrowable_wrapper<std::error_code const&, crimson::ec<(std::errc)11> > >::_future<crimson:
:errorated_future_marker<boost::intrusive_ptr<MOSDOpReply> > > > ()>, crimson::interruptible::interruptible_future_detail<crimson::osd::IOInte
rruptCondition, crimson::errorator<crimson::unthrowable_wrapper<std::error_code const&, crimson::ec<(std::errc)11> > >::_future<crimson::error
ated_future_marker<boost::intrusive_ptr<MOSDOpReply> > > > >(seastar::noncopyable_function<crimson::interruptible::interruptible_future_detail
<crimson::osd::IOInterruptCondition, crimson::errorator<crimson::unthrowable_wrapper<std::error_code const&, crimson::ec<(std::errc)11> > >::_
future<crimson::errorated_future_marker<boost::intrusive_ptr<MOSDOpReply> > > > ()>&&)::{lambda(seastar::internal::promise_base_with_type<boos
t::intrusive_ptr<MOSDOpReply> >&&, seastar::noncopyable_function<crimson::interruptible::interruptible_future_detail<crimson::osd::IOInterruptCondition, crimson::errorator<crimson::unthrowable_wrapper<std::error_code const&, crimson::ec<(std::errc)11> > >::_future<crimson::errorated_future_marker<boost::intrusive_ptr<MOSDOpReply> > > > ()>&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>::run_and_dispose() (/usr/bin/ceph-osd+0x2c7646b1)
    #2 0x560f5352c3ae  (/usr/bin/ceph-osd+0x374843ae)
    #3 0x560f535318ef  (/usr/bin/ceph-osd+0x374898ef)
    #4 0x560f536e395a  (/usr/bin/ceph-osd+0x3763b95a)
    #5 0x560f532413d9  (/usr/bin/ceph-osd+0x371993d9)
    #6 0x560f476af95a in main (/usr/bin/ceph-osd+0x2b60795a)
    #7 0x7f7aa0af97b2 in __libc_start_main (/lib64/libc.so.6+0x237b2)
    #8 0x560f477d2e8d in _start (/usr/bin/ceph-osd+0x2b72ae8d)

```

[1]: http://pulpito.front.sepia.ceph.com/rzarzynski-2021-05-20_07:28:16-rados-master-distro-basic-smithi/6124735/

The commit deals with the problem by repacking the outer future.
An alternative could be in switching from `std::unique_ptr` to
`seastar::shared_ptr` for managing `OpsExecuter`.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

qa/tasks/cephfs/test_nfs: fix info test

Signed-off-by: Sage Weil <sage@newdream.net>

doc/cephfs/fs-nfs-exports: document --ingress --virtual-ip

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs: move ingress vs virtual_ip check to cluster interface

Signed-off-by: Sage Weil <sage@newdream.net>

PendingReleaseNotes: clarify deprecated

Signed-off-by: Sage Weil <sage@newdream.net>

PendingReleaseNotes: note breaking CLI changes

Signed-off-by: Sage Weil <sage@newdream.net>

doc/cephadm/nfs: document nfs+ingress

Signed-off-by: Sage Weil <sage@newdream.net>

qa/suites/rados/cephadm/smoke-roleless: test nfs, nfs + ingress

Still missing a full client mount test, though!

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs: take --ingress argument to 'nfs cluster create'

It is likely that the rook/k8s variation of ingress will not take a
virtual_ip argument.  We want to make sure that ingress yes/no can be
specified independent of the virtual_ip.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: adjust debug output for device refresh

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: ingress: fix log msg

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: fix logging of config/placement errors

Signed-off-by: Sage Weil <sage@newdream.net>

common/options: enable nfs module for new clusters

Signed-off-by: Sage Weil <sage@newdream.net>

cephadm: --stop-signal=SIGTERM

haproxy's container image tells docker|podman to send SIGUSR1 for a "clean"
shutdown.  For NFS, the connections never close, so we will always hit the
podman|docker 10s timeout and get a SIGKILL.  That, in turn, causes haproxy
to exit with 143, and puts the systemd unit in a failed state.

This highlights a general problem(?) with stopping containers: if they don't
do it quickly then we'll end up in this error state.  We don't directly
address that here.

Avoid this problem by always stopping containers with SIGTERM.  In the
haproxy case, that means an immediate shutdown (no graceful drain of
open connections).  In theory we could do this only for haproxy with
NFS, but we can easily imagine RGW connections that don't close in 10s
either, and we don't want containers exiting in error state--we just
want the proxy to stop quickly.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/orchestrator: default nfs pool, namespaces

Apply nfs default pool (currently 'nfs-ganesha'), and default the
namespace to the service_id.

There is no practical reason for users to ever need to change this, and
requiring them to provide this informaiton at config/apply time just
complicates life.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: nfs: create pool if it doesn't yet exist

Signed-off-by: Sage Weil <sage@newdream.net>

doc/cephadm/nfs: update

- leave off pool/ns, since they should almost never be necessary.
- add port

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs: change 'nfs cluster info'

- include the virtual_ip and port at top level
- move backend server list into a sub-item
- include (haproxy) monitoring port

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs: take optional virtual_ip for deploying ingress

For 'nfs cluster create', optionally take a virtual_ip to deploy ingress.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs: remove 'nfs cluster update'

This command is very awkward to implement unless all service spec fields
are always required.  That will soon mean both the placement *and*
virtual_ip (if any), making it much less useful for a human to make use
of.

Instead, let them update yaml, or adjust the nfs and/or ingress specs
directly.  I don't think this command is needed.

Signed-off-by: Sage Weil <sage@newdream.net>

doc/_ext: render :example: field of an option

some options have this fields in their document, let's render it as
well.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test: add test to verify incremental snapshot updates

Fixes: http://tracker.ceph.com/issues/49939
Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: synchronize file mode

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: transfer snapshot diffs whenever possible

Use incremental transfer when the data on the remote file system
for a given directory root is belongs to the snapshot which can
be used for local comparison.

Files are chosen based on mtime changes.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: remove hardcoded metadata count when creating snapshot

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: adjust PeerReplayer::cleanup_remote_dir() for purging any remote (sub)directory

Right now, the only path to cleanup (purge) is the remote file
system directory root (for configured snapshot directories).
With incremental sync, we would need the ability to cleanup
selective directories under configured snapshot directories.
Furthermore, the path passed to the function ise used to checking
if the mirror daemon needs to backoff due to mirroring being
disabled, daemon shutdown etc.

So, adjust the function prototype to accept the directory root
and the relative path to be cleaned up under this directory.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mds: introduce ceph.mirror.dirty_snap_id vxattr

Purpose of this xattr is to identify if the "next" snapshot to be
synchronized can be incrementally transferred. Value of the xattr
is the snap-id of a snapshot in the primary cluster (for a given
directory root).

CephFS mirror daemon sets this xattr on the directory root once
the data for a snapshot is synchronized, thereby signifying that
the data under this directory root is the data of the snapshot
identified by the snap-id value. This allows the mirror daemon
to use the much efficient selective synchronzation (based on
mtime/ctime) by scanning the two snapshots locally and only
transferring the inodes that have changed.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

osd: Overload send_cluster_message() in PeeringState

One for crimson-osd taking std::unique_ptr and one for ceph-osd
taking boost::intrusive_ptr

Signed-off-by: Amnon Hanuhov <ahanukov@redhat.com>

msg: Change crimson's make_message() namespace to crimson

instead of crimson::net so we can use TOPNSPC namespace in shared parts
between crimson-osd and ceph-osd

Signed-off-by: Amnon Hanuhov <ahanukov@redhat.com>