rgw: Robust notify invalidates on cache timeout

This avoids a potential race condition in which updates are delayed.

Fixes: https://tracker.ceph.com/issues/51674
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Rename REMOVE_OBJ to INVALIDATE_OBJ

Also rename ObjectCache::remove to ObjectCache::invalidate_remove

Since we're depending on these message types/functions having
invalidate semantics but NOT caching a negative result, rename and
leave a comment for clarity.

Fixes: https://tracker.ceph.com/issues/51674
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: distribute() takes RGWCacheNotifyInfo

So we don't have to parse the bufferlist back out to find what object
to throw out of the cache.

Fixes: https://tracker.ceph.com/issues/51674
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #42327 from zdover23/wip-doc-cephadm-troubleshooting-1-of-x-2021-07-15

doc/cephadm: rewrite troubleshooting 1 of x

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #42324 from adamemerson/wip-51661

rgw: Don't segfault on datalog trim

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

rgw: Don't segfault on datalog trim

Synchronous (or yielded, basically other-than AioCompletion trim)
would try to dereference the past-the-end iterator if we were trimming
to a point in the most recent generation.

https://tracker.ceph.com/issues/51661
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #41490 from idryomov/wip-rbd-pwl-ssd-tailp

librbd/cache/pwl/ssd: fix first_free_entry and m_first_free_entry corruption

Reviewed-by: Mahati Chamarthy <mahati.chamarthy@intel.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #42145 from hualongfeng/fix_get_context_error

librbd/cache/pwl/ssd: fix use-after-free on C_BlockIORequest

Reviewed-by: Mahati Chamarthy <mahati.chamarthy@intel.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #42244 from zdover23/wip-doc-cephadm-operations-data-location-and-health-checks-2021-07-08

doc/cephadm: operations: Data location & ...

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #42288 from mgfritch/cephadm-test-fixup-mock

cephadm: use CephadmContext rather than MagicMock

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #42310 from rzarzynski/wip-crimson-remotepeeringevent-waits-on-active

crimson/osd: remote peering requests wait for OSD activation.

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

librbd/cache/pwl/ssd: fix use-after-free on C_BlockIORequest

In setup_schedule_append() function, its first expression
will cause the req to be deleted, and subsequent use of
the variable req becomes an illegal operation. And due to
delete, rep->m_image_ctx will be empty, so it lead to
segfault in AbstractWriteLog::get_context().
So pass the `req` into `schedule_append()` function.

Fixes: https://tracker.ceph.com/issues/50951
Signed-off-by: Hualong Feng <hualong.feng@intel.com>

crimson/osd: move the wait-for-active logic into OSDState.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #42314 from rzarzynski/wip-crimson-single-do_peering_event

crimson/osd: consider epoch_sent when ignoring old peering event.

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42271 from xxhdx1985126/wip-seastore-writev

crimson/os/seastore: use writev to avoid extra memory copy

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42311 from rzarzynski/wip-crimson-fix-premature-activation

crimson/osd: prevent premature OSD activation.

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42202 from cyx1231st/wip-seastore-metrics

crimson/os/seastore: implement label-based profiling with metrics

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/cephadm: rewrite troubleshooting 1 of x

This PR improves the readability and format
of the troubleshooting.rst file. This also
makes a change to the markdown of one of the
sub-subsections so that it is made of tildes
(~) instead of carets (^), because that's
the RST standard.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

crimson/os/seastore: use writev to avoid extra memory copy

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore/cache: improve logs for transaction conflict/reset

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: count invalidated transactions

Labeled by source and extent-type.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: count committed transactions

Labeled by source.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: count created transactions

Labeled by source.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: introduce src_t to classify transactions

* INIT - READ            => INIT
* INIT - MUTATE          => INIT
* SEASTORE/TEST - READ   => READ
* SEASTORE/TEST - MUTATE => MUTATE
* CLEANER - READ         => CLEANER
* CLEANER - MUTATE       => CLEANER

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: consolidate the static member CachedExtent::TYPE

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #42279 from myoungwon/wip-fix-51627

osd: fix to recover adjacent clone when set_chunk is called

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42285 from orozery/append-frame-claim-append

msg/async/ProtocolV2: optimize append_frame

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42218 from ifed01/wip-ifed-compact-after-upgrade

os/bluestore: compact db after bulk omap naming upgrade.

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #42197 from ifed01/wip-ifed-fix-spurious-read-errors-warn

os/bluestore: respect bluestore_warn_on_spurious_read_errors setting

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42291 from dsavineau/issue_51620

cephadm: ensure sysctl_dir exist

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42276 from tchaikov/wip-common-LogEntry

common/LogEntry: drop support of LogSummary v2 encoding scheme

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #42319 into master

* refs/pull/42319/head:
qa/tasks/rebuild_mondb: fix rebuild vs logmonitor external_log_to

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #42294 from neha-ojha/wip-51101

qa/*/test_envlibrados_for_rocksdb.sh: install libarchive-3.3.3

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #42278 into master

* refs/pull/42278/head:
doc/mgr/nfs: update about RGW exports
doc/cephfs/nfs: update about nfs module
doc/mgr/nfs: update cephfs export create command about client and squash arguments
doc/mgr/nfs: update nfs links
doc/mgr/nfs: add missing cluster_id to export info command
doc/cephfs: move nfs doc under mgr docs

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/tasks/rebuild_mondb: fix rebuild vs logmonitor external_log_to

Signed-off-by: Sage Weil <sage@newdream.net>

common/LogEntry: drop support of LogSummary v2 encoding scheme

LogSummary's v3 encoding scheme was introduced in
648aaf271cb02c647f046288656c11f15a7799b2, which was in turn included
by Ceph v13.1.0 and all newer releases. since LogSummary is persistented
by monitor, and it is trimmed regularly by monitor, there is no need
to read a LogSummary encoded by 2 releases older monitor.

in this change, the support of LogSummary v2 encoding scheme is dropped.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42301 from tchaikov/wip-doc-crimson-nbd

doc/dev/crimson: add a section for testing with crimson-store-nbd

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #42305 from rhcs-dashboard/remove-rgw-frontend-ssl-key

mgr/dashboard: remove usage of 'rgw_frontend_ssl_key'

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

doc/dev/crimson: add a section for testing with crimson-store-nbd

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: consider epoch_sent when ignoring old peering event.

The classical OSD does:

```
bool old_peering_evt(PGPeeringEventRef evt) {
  return old_peering_msg(evt->get_epoch_sent(), evt->get_epoch_requested());
}
```

```
bool PG::old_peering_msg(epoch_t reply_epoch, epoch_t query_epoch)
{
  if (auto last_reset = get_last_peering_reset();
      last_reset > reply_epoch || last_reset > query_epoch) {
    dout(10) << "old_peering_msg reply_epoch " << reply_epoch << " query_epoch "
             << query_epoch << " last_peering_reset " << last_reset << dendl;
    return true;
  }
  return false;
}
```

See also: https://gist.github.com/rzarzynski/e60750c279ae9d234728b0b199562c40#gistcomment-3808289.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: flip the negation in PG::do_peering_event().

Less operations, lower mental load on reader's mind.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: unify the variants of PG::do_peering_event().

Simpler and more comprehensible that way.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #42227 into master

* refs/pull/42227/head:
doc/man/8/cephadm: add --log-to-file (and --single-host-defaults)
cephadm: add bootstrap --log-to-file option

Reviewed-by: Adam King <adking@redhat.com>

Merge PR #42014 into master

* refs/pull/42014/head:
mon/LogMonitor: make 'ceph log last' work with new storage
mon/LogMonitor: transition log entries on upgrade to quincy
mon/LogMonitor: store logentries in separate keys
mon/LogMonitor: replace semi-broken pending_summary with pending_keys
mon/LogMonitor: externally log events from when we were out of quorum
mon/LogMonitor: factor logging to file out of update_from_paxos
mon: funnel SIGHUP handler through handle_mon_signal
common/LogEntry: add some constness
mon/LogMonitor: log mkfs in cluster channel
common/LRUSet: combine lru and hash-based lookup
mon/LogMonitor: remove stray derr

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/dev/crimson: add a missing not

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: improve debugs and clean up OSD activation.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #42312 from rzarzynski/wip-crimson-boot_epoch-in-MOSDBoot

crimson/osd: use boot_epoch for MOSDBoot.

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42313 from rzarzynski/wip-crimson-drop-temporary-from-pg

crimson/osd: drop a temporary in PG::handle_initialize().

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/*/test_envlibrados_for_rocksdb.sh: install libarchive-3.3.3

To workaround the libarchive dependency issue seen with centos 8, which
has been causing consistent failures like

```
2021-06-04T04:52:51.147 INFO:tasks.workunit.client.0.smithi071.stdout:Installed:
2021-06-04T04:52:51.148 INFO:tasks.workunit.client.0.smithi071.stdout:  cmake-3.18.2-9.el8.x86_64                 cmake-data-3.18.2-9.el8.noarch
...
2021-06-04T04:52:57.554 INFO:tasks.workunit.client.0.smithi071.stderr:+ cmake -DCMAKE_BUILD_TYPE=Debug -DWITH_TESTS=ON -DWITH_LIBRADOS=ON -DWITH_SNAPPY=ON -DWITH_GFLAGS=OFF -DFAIL_ON_WARNINGS=OFF ..
2021-06-04T04:52:57.579 DEBUG:teuthology.orchestra.run:got remote process result: 127
2021-06-04T04:52:57.580 INFO:tasks.workunit.client.0.smithi071.stderr:cmake: symbol lookup error: cmake: undefined symbol: archive_write_add_filter_zstd
```
More details in https://tracker.ceph.com/issues/51101#note-5

Fixes: https://tracker.ceph.com/issues/51101
Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #42303 from tchaikov/wip-crimson-nbd-graceful-shutdown

crimson/tools/store_nbd: fixes to implement graceful shutdown

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge PR #42029 into master

* refs/pull/42029/head:
vstart_runner: use FileNotFoundError when os.stat() fails

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge PR #42030 into master

* refs/pull/42030/head:
vstart_runner: maintain log level when --debug is passed

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #42033 into master

* refs/pull/42033/head:
vstart_runner: add log messages to vstart_runner.py

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

crimson/osd: remote peering requests wait for OSD activation.

Before the patch `RemotePeeringRequest` instances were not
waiting for OSD activation. This was eluding the protection
from handling old, outdated peering events the `MOSDBoot`
machinery offers. The net results are crashes like this one
(`OSDState is booting` has been produced by a custom debug):

```
2021-07-07T18:20:23.293 INFO:journalctl@ceph.osd.2.smithi145.stdout:Jul 07 18:16:30 smithi145 conmon[71083]: DEBUG 2021-07-07 18:16:30,535 [shard 0] ms - [osd.2(cluster) v2:172.21.15.145:6802/2@62336 >> osd
.1 v2:172.21.15.145:6809/2] <== #19 === pg_lease(4.9 pg_lease(ru 60.120281219s ub 68.121276855s int 16.000000000s) e86/86) v1 (133)
2021-07-07T18:20:23.293 INFO:journalctl@ceph.osd.2.smithi145.stdout:Jul 07 18:16:30 smithi145 conmon[71083]: DEBUG 2021-07-07 18:16:30,536 [shard 0] osd - handle_peering_op on 4.9 from 1
2021-07-07T18:20:23.293 INFO:journalctl@ceph.osd.2.smithi145.stdout:Jul 07 18:16:30 smithi145 conmon[71083]: DEBUG 2021-07-07 18:16:30,536 [shard 0] osd - peering_event(id=125, detail=PeeringEvent(from=1 pg
id=4.9 sent=86 requested=86 evt=epoch_sent: 86 epoch_requested: 86 MLease epoch 86 from osd.1 pg_lease(ru 60.120281219s ub 68.121276855s int 16.000000000s))): start
2021-07-07T18:20:23.293 INFO:journalctl@ceph.osd.2.smithi145.stdout:Jul 07 18:16:30 smithi145 conmon[71083]: DEBUG 2021-07-07 18:16:30,536 [shard 0] osd - peering_event(id=125, detail=PeeringEvent(from=1 pg
id=4.9 sent=86 requested=86 evt=epoch_sent: 86 epoch_requested: 86 MLease epoch 86 from osd.1 pg_lease(ru 60.120281219s ub 68.121276855s int 16.000000000s))): got map 93
2021-07-07T18:20:23.294 INFO:journalctl@ceph.osd.2.smithi145.stdout:Jul 07 18:16:30 smithi145 conmon[71083]: DEBUG 2021-07-07 18:16:30,536 [shard 0] osd - peering_event(id=125, detail=PeeringEvent(from=1 pgid=4.9 sent=86 requested=86 evt=epoch_sent: 86 epoch_requested: 86 MLease epoch 86 from osd.1 pg_lease(ru 60.120281219s ub 68.121276855s int 16.000000000s))): OSDState is booting
2021-07-07T18:20:23.294 INFO:journalctl@ceph.osd.2.smithi145.stdout:Jul 07 18:16:30 smithi145 conmon[71083]: ERROR 2021-07-07 18:16:30,536 [shard 0] none - /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-5007-g3a9abb02/rpm/el8/BUILD/ceph-17.0.0-5007-g3a9abb02/src/crimson/osd/osd_operations/peering_event.cc:165 : In function 'crimson::osd::RemotePeeringEvent::get_pg()::<lambda()>', ceph_assert(%s)
2021-07-07T18:20:23.294 INFO:journalctl@ceph.osd.2.smithi145.stdout:Jul 07 18:16:30 smithi145 conmon[71083]: osd.state.is_active()
2021-07-07T18:20:23.294 INFO:journalctl@ceph.osd.2.smithi145.stdout:Jul 07 18:16:30 smithi145 conmon[71083]: Aborting on shard 0.
```

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: drop a temporary in PG::handle_initialize().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: use boot_epoch for MOSDBoot.

For the sake of compliance and similarity with the classcial OSD.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: prevent premature OSD activation.

In contrast to the classical OSD:

```
int OSD::init()
{
  // ...

  {
    epoch_t bind_epoch = osdmap->get_epoch();
    service.set_epochs(NULL, NULL, &bind_epoch);
  }

  // ...

  // load up pgs (as they previously existed)
  load_pgs();
```

crimson doesn't set the `bind_epoch` when initializing. The net
result is going active prematurely which happens because the 3rd
condition (`bind_epoch < osdmap->get_up_from(whoami)`) is always
true.

```
    if (osdmap->is_up(whoami) &&
        osdmap->get_addrs(whoami) == public_msgr->get_myaddrs() &&
        bind_epoch < osdmap->get_up_from(whoami)) {
      if (state.is_booting()) {
        logger().info("osd.{}: activating...", whoami);
```

Nullifying it translates the "is it activated?" check basically
into "is it up?" verification. This is problematic in a situation
like:

1. Primary got new OSDMap but replica has not.
2. Replica restarts, sends `MOSDBoot` and receives the newer map
   from the previous point.
3. Primary sends a message that is unexpected by replica.
4. Monitor publishes a new OSDMap diven by the `MOSDBoot`.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #42267 from ifed01/wip-fix-missing-shared-blob

os/bluestore: fix erroneous SharedBlob record removal during repair.

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42209 from sebastian-philipp/doc-dev-cephadm-define-vars

doc/dev/cephadm: Define variables

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: remove usage of 'rgw_frontend_ssl_key'

Fixes: https://tracker.ceph.com/issues/51643
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
Removing the usage of rgw_frontend_ssl_key from the rgw service form.

Merge pull request #42274 from liewegas/cleanup-blkdev

common/blkdev: remove stray debug output

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42206 from ronen-fr/wip-ronenf-list-object

common/hobject: a minor fix and performance gain to hobjects listing

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async/ProtocolV2: optimize append_frame

The commonly used append_frame function currently copies
frame data, incurring expensive heap allocation and data copying.
Instead, switch to claiming the frame data, re-using it without copying.

Signed-off-by: Or Ozeri <oro@il.ibm.com>

Merge pull request #42210 from sebastian-philipp/options-ms-bind-port-max

common/options: global.yaml: change ms_bind_port_max to 7568

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42268 from neha-ojha/wip-health-cleanup

mon/PGMap: remove get_stuck_counts because there are no callers

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42235 from tchaikov/tools/kvstore-tool

tools/kvstore_tool: add "std::" before ostream and string

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/tools/store_nbd: handle ECONNABORTED returned by accept()

if we abort accept() call, an ECONNABORTED is expected. and we should
handle it, otherwise unhandled exception will be noticed by seastar's
reactor. and it complains in that case.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/tools/store_nbd: call segment_manager->close() after tm->close()

TransactionManager::close() calls into journal->close(), which in turn
calls BlockSegmentManager::segment_close(). and
SegmentStateTracker::write_out() is then called by
BlockSegmentManager::segment_close().

but BlockSegmentManager::close() closes the underlying seastar::file,
we are not able to write to the file after closing it.

in this change, to ensure that we can close a segment correctly in
TMDriver::close(), tm->close() is called before
segment_manager->close().

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: fix to recover adjacent clone when set_chunk is called

set_chunk needs adjacent clones to calculate reference count

fixes: https://tracker.ceph.com/issues/51627

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

doc/mgr/nfs: update about RGW exports

This patch just moves the RGW exports created using nfs module to mgr/nfs
document. The RGW requirements will be updated in a different PR.

Signed-off-by: Varsha Rao <varao@redhat.com>

doc/cephfs/nfs: update about nfs module

Signed-off-by: Varsha Rao <varao@redhat.com>

doc/mgr/nfs: update cephfs export create command about client and squash arguments

Signed-off-by: Varsha Rao <varao@redhat.com>

doc/mgr/nfs: update nfs links

Signed-off-by: Varsha Rao <varao@redhat.com>

doc/mgr/nfs: add missing cluster_id to export info command

Signed-off-by: Varsha Rao <varao@redhat.com>

doc/cephfs: move nfs doc under mgr docs

Fixes: https://tracker.ceph.com/issues/51428
Signed-off-by: Varsha Rao <varao@redhat.com>

Merge pull request #42269 from zdover23/wip-doc-dev-essentials-irc-2021-07-10

doc/dev: add IRC information to dev guide

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #42113 from zdover23/wip-doc-upgrading-ceph-potential-problems-2021-06-30

doc/cephadm: improve "Potential Problems"

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #42092 from zdover23/wip-doc-upgrading-ceph-starting-the-upgrade-2021-06-29

doc/cephadm: improving "Starting the Upgrade"

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

doc/man/8/cephadm: add --log-to-file (and --single-host-defaults)

Signed-off-by: Sage Weil <sage@newdream.net>

cephadm: ensure sysctl_dir exist

For some reason, the sysctl directory could not exist if no packages dropping
a custom sysctl file is installed on the host.
Instead we create the directory if it doesn't exist.

Closes: https://tracker.ceph.com/issues/51620
Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #42217 from clwluvw/osd-device-details-grafana

monitoring: fix Physical Device Latency unit

Reviewed-by: Waad Alkhoury <walkhour@redhat.com>
Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: mykaul <NOT@FOUND>
Reviewed-by: p-se <NOT@FOUND>

Merge pull request #41034 from nSedrickm/auth-storage-directive

mgr/dashboard: create directive for AuthStorage service

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: create directive for AuthStorage service

This commit adds a directive that can be used to conditionally display elements based on authorization/scopes criteria

Fixes: https://tracker.ceph.com/issues/47355
Signed-off-by: Ngwa Sedrick Meh <nsedrick101@gmail.com>

common/blkdev: remove stray debug output

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #41834 from grajoria/master

doc: Correction and improvisation for Timestamp part of the doc

Merge pull request #42272 from smithfarm/wip-51622

rpm: remove macro invocation from comment line

Reviewed-by: Kefu Chai <kchai@redhat.com>

rpm: remove macro invocation from comment line

In RPM spec files, comment lines should not include macro invocations,
because RPM can and will expand them, with unpredictable results.

Fixes: https://tracker.ceph.com/issues/51622
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #42212 from wjwithagen/wjw-fix-signal

rgw: Use signaling compatible with POSIX

doc/dev: add IRC information to dev guide

In days of yore, the Developer Guide linked to the
IRC page at ceph.io. After the 2021 rewriting of
ceph.io, a new era began, and that page was no
longer easily accessible (it could still be found
at old.ceph.com, but this was deprecated).

Anyway, that IRC information should be included in
the docs. This PR makes sure that the IRC
information is included in the docs.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

mon/PGMap: remove get_stuck_counts because there are no callers

The only callers were removed in 729a08923fb4403b88baafa0ae75643cfefe6510,
as a part of the broader health reporting improvements.

Signed-off-by: Neha Ojha <nojha@redhat.com>

cephadm: use CephadmContext rather than MagicMock

MagicMock hides attribute errors:

```
ctx = <cephadm.CephadmContext object at 0x7f0a12f58eb0>, container_id = 'container_id', daemon_type = 'node-exporter'

    @staticmethod
    def get_version(ctx, container_id, daemon_type):
        # type: (CephadmContext, str, str) -> str
        """
        :param: daemon_type Either "prometheus", "alertmanager" or "node-exporter"
        """
        assert daemon_type in ('prometheus', 'alertmanager', 'node-exporter')
        cmd = daemon_type.replace('-', '_')
        code = -1
        err = ''
        version = ''
        if daemon_type == 'alertmanager':
            for cmd in ['alertmanager', 'prometheus-alertmanager']:
                _, err, code = call(ctx, [
                    ctx.container_engine.path, 'exec', container_id, cmd,
                    '--version'
                ], verbosity=CallVerbosity.DEBUG)
                if code == 0:
                    break
            cmd = 'alertmanager'  # reset cmd for version extraction
        else:
            _, err, code = call(ctx, [
>               ctx.container_engine.path, 'exec', container_id, cmd, '--version'
            ], verbosity=CallVerbosity.DEBUG)
E           AttributeError: 'NoneType' object has no attribute 'path'
```

Signed-off-by: Michael Fritch <mfritch@suse.com>

os/bluestore: fix erroneous SharedBlob record removal during repair.

Fixes: https://tracker.ceph.com/issues/51619
Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #42246 from ceph/1625

doc: 16.2.5 Release Notes

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

doc: 16.2.5 Release Notes

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #42253 from liewegas/doc-linode

doc/foundation: add linode

rgw: Use signaling compatible with POSIX

FreeBSD does not have signal_t.
Linux specifies also to use `sigval` as POSIX compliant.

tracker: https://tracker.ceph.com/issues/51610
fixes: https://github.com/ceph/ceph/pull/36266
Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #42112 from CongMinYin/fix-common-buffer-SIGABRT

common/buffer: fix stack corruption in rebuild_aligned_size_and_memory()

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

common/buffer: fix SIGABRT in  rebuild_aligned_size_and_memory

There is such a bl, which needs to satisfy two conditions:
1)all ptrs' length sum except last ptr is aligned with 4K;
2)the length of last ptr is 0.
This bl will cause stack corruption when calling
bufferlist::rebuild_aligned_size_and_memory().

Deal with this special scenario in rebuild_aligned_size_and_memory() to
solve the bug. And added a specialtest-case to reproduce this scenario.

Fixes: https://tracker.ceph.com/issues/51419
Signed-off-by: Yin Congmin <congmin.yin@intel.com>

os/bluestore: respect bluestore_warn_on_spurious_read_errors setting

Fixes: https://tracker.ceph.com/issues/51540
Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: compact db after bulk omap naming upgrade.

Omap naming scheme upgrade introduced recently might perform bulk data
removal and hence leave DB in a "degraded" state. Let's compact it.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>