Merge PR #38266 into master

* refs/pull/38266/head:
client: add ceph.{cluster_fsid/client_id} vxattrs suppport

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #38367 into master

* refs/pull/38367/head:
pybind/cephfs: fix missing terminating NULL char in readlink()'s C string

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #36127 into master

* refs/pull/36127/head:
client: more reasonable statfs ffree

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #37971 from lixiaoy1/rwl_plugin

librbd/cache: make image persistent writeback cache as plugin

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38484 from rhcs-dashboard/update-ssl-error-1

mgr/dashboard: Updating the inbuilt ssl providers error

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #38459 from ronen-fr/wip-ronenf-lua-include

rgw/lua: fixing incompatible declarations of CephContext

Merge pull request #37519 from pcuzner/docs-cephadm-config-drift

doc/dev/cephadm: high level design for a compliance check feature

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #38498 from bk201/wip-48506

mgr/dashboard: remove pyOpenSSL version pinning

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38415 from insujang/master

msg/async: refine worker creation in NetworkStack

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38457 from tchaikov/wip-mon-scrub-interval

mon: use TYPE_SEC for mon_scrub_interval

Reviewed-by: Joao Eduardo Luis <joao@suse.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

mgr/dashboard: remove pyOpenSSL version pinning

Tests require a newer version of pyOpenSSL.

Fixes: https://tracker.ceph.com/issues/48506
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

Merge pull request #38497 from tchaikov/wip-seastar

seastar: update seastar submodule

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #38491 from agayev/fix-hmsmr-compilation

os/bluestore: Fix HMSMRDevice.cc compilation.

Reviewed-by: Kefu Chai <kchai@redhat.com>

seastar: update seastar submodule

to pick up recent fixes in file io

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38437 from ifed01/wip-ifed-alloc-replay-dump

os/bluestore: add more means to troubleshoot allocator(s)

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #38454 from tchaikov/wip-compressor-cleanup

compressor: put Compressor into TOPNSPC namespace and use string_view

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #38458 from tchaikov/wip-librados-cct

librados: move definition of cct_deleter to .cc

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #38418 from anthonyeleven/anthonyeleven/clarify-op-priorities

doc: clarify osd recovery op priority and fix a couple of typos

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

os/bluestore: Fix HMSMRDevice.cc compilation.

Signed-off-by: Abutalib Aghayev <agayev@psu.edu>

Merge PR #38419 into master

* refs/pull/38419/head:
vstart_runner: print result line unless opt_rotate_log is set

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/dashboard: Adding the alert bad certificate error to the ssl providers error

upstream tracked in https://github.com/cherrypy/cheroot/pull/348
Fixes: https://tracker.ceph.com/issues/48490
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: Updating the inbuilt ssl providers error

Fixes: https://tracker.ceph.com/issues/48490
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #37956 from dorindabassey/example-notific

examples/boto3/README: examples notification

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #38374 from rzarzynski/wip-crimson-backfill-remove

crimson: add support for backfill, part 2 -- MOSDPGBackfillRemove

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38392 from rzarzynski/wip-crimson-backfill-virtual_pgfacade

crimson: use explicit, abstract interfaces for backfill's facades.

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38208 from sebastian-philipp/mgr-cephadm-type-annotations-mandatory

mgr/cephadm: make type annotations mandatory for some modules

Reviewed-by: Michael Fritch <mfritch@suse.com>

crimson: use explicit, abstract interfaces for backfill's facades.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #37130 from pcuzner/cephadm-exporter

cephadm:Add a daemon mode for cephadm to provide a metadata endpoint

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #38483 from tchaikov/wip-crimson-cleanup

crimson: cleanups

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mgr/cephadm: make annotations for schedule.py mandatory

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: make annotations for inventory.py mandatory

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: make annotations for serve.py mandatory

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: make annotations for upgrade.py mandatory

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38106 from varshar16/wip-fix-rook-orch-ps-img-id-name

mgr/rook: Fix container id and image id in 'orch ps'

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Travis Nielsen <tnielsen@redhat.com>

Merge pull request #38346 from varshar16/wip-remove-container-id-field-orch-ps

mgr/orchestrator: Remove 'CONTAINER ID' col from orch ps if unknown

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #37952 from mgfritch/cephadm-json-decode

mgr/cephadm: handle JSONDecodeError

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38032 from mgfritch/cephadm-last-local-image-name

cephadm: reference the last local image by digest

Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38064 from mgfritch/cephadm-rm-host-key-err

mgr/cephadm: verify the `host` key exists during refresh

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

crimson/test: make the backfill testing quieter.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/test: make use of maybe_flush() in the backfill testing.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: add support for MOSDPGBackfillRemove at primary.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: add support for MOSDPGBackfillRemove at replicas.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #38130 from sebastian-philipp/cephadm-upgrade-fail-daemon

mgr/cephadm: upgrade: fail gracefully, if daemon redeploy fails

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #38317 from sebastian-philipp/cephadm-reduce-log-dg-noise

mgr/cephadm: OSD service: reduce log noise

Reviewed-by: Michael Fritch <mfritch@suse.com>

crimson/osd: return something in PG::do_delete_work()

this silences the warning of

crimson/osd/pg.cc:316:1: warning: no return statement in function returning non-void [-Wreturn-type]
  316 | }
      | ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: use a constexpr function for sanity test

since match_stage_t is an alias of uint_8. it is an unsigned type, so an
instance of this type should be always greater or equal to 0.

this silences warning like:

stage_types.h:37:50: warning: comparison is always true due to limited range of data type [-Wtype-limits]
   37 |     static_assert(STAGE >= STAGE_BOTTOM && STAGE <= STAGE_TOP);
      |                                            ~~~~~~^~~~~~~~~~~~

also, GCC complains if it is able to assure that the compare always
returns true. so a function helps to silence it.

this warning is meaningless per-se, as we are using static_assert() to
perform sanity test at *compile-time*.

Signed-off-by: Kefu Chai <kchai@redhat.com>

librbd/cache: move image writeback cache to plugin

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

Merge pull request #38460 from ronen-fr/wip-ronenf-lua-move

rgw: removing a pessimizing move

doc: clarify osd recovery op priority and fix a couple of typos

Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>

pybind/cephfs: fix missing terminating NULL char in readlink()'s C string

The error is in test_readlink() in src/test/pybind/test_cephfs.py:

...
test_cephfs.test_readlink ... FAIL

======================================================================
FAIL: test_cephfs.test_readlink
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/tmhoang/test_cephfs.py", line 99, in test_readlink
    assert_equal(d, b"/file-1")
AssertionError: b'/file-1\xe8' != b'/file-1'
...

Fixes: https://tracker.ceph.com/issues/48408
Signed-off-by: Tuan Hoang <tmhoang@linux.ibm.com>

librbd/plugin: enable hook points for exclusive lock

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

Merge pull request #36986 from cyx1231st/wip-seastar-onode-tree-upstream

crimson/seastore: add initial onode staged-fltree

Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/test: fix test/tool make target names

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/staged-onode-tree: use LT, EQ, GT for comparison results

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: add and fix comments

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #38356 from icepic/patch-2

doc/radosgw: Fix misspelled word in index.rst

Reviewed-by: Zac Dover <zac.dover@gmail.com>

rgw: removing a pessimizing move

"warning: moving a temporary object prevents copy elision"

Co-authored-by: Kefu Chai <kefu@redhat.com>
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #38324 from tchaikov/wip-mon-cleanups

mon: drop QuorumService and cleanups.

Reviewed-by: Joao Eduardo Luis <joao@suse.com>

mon/ConfigKeyService: drop ConfigKeyService::dispatch()

this method is basically an alias of
ConfigKeyService::service_dispatch(), so get rid of it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/ConfigKeyService: drop ConfigKeyService::get_name()

the only caller of this function is prefix(), so hardcode the service
name there instead.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/ConfigKeyService: remove unused bits

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/ConfigKeyService: store mon and paxos using reference

before this change, we always have a valid pointer as member variable.
so, a reference serve this purpose better.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon: drop QuorumService

so far we only implemented ConfigKeyService, so move QuorumService into
the its only child class -- ConfigKeyService.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/ConfigKeyService: move private member variable declaration to the end of class

to follow https://google.github.io/styleguide/cppguide.html

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon: store a reference as member variable

before this change, we always have a valid pointer as member variable.
so, a reference serve this purpose better.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon: use unique_ptr<> to manage lifecycle of Monitor::paxos

it is less error-prone, and has better readability.

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw/lua: fixing incompatible declarations of CephContext

Existing forward declaration breaks OSD compilation (clang)

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

librados: move definition of cct_deleter to .cc

so cct->get() and cct->put() are visually closer. this improves the
readability.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon: use TYPE_SEC for mon_scrub_interval

* common/legacy_config_opts.h: drop mon_scrub_interval from legacy
  settings. as this option is not in critical path. so we can afford
  the overhead of look its value up in the map.
* common/options.cc: change mon_scrub_interval's type from TYPE_INT
  to TYPE_SECS, so we can use human friendly representations like
  "3 days" when specifying its value
* mon/Monitor.cc: adapt to the changes above.
* doc/rados: update the doc for "mon scrub interval" accordingly

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/Timer: add SafeTimer::add_event_after() which accepts timespan

easier to use when we are moving from the number of seconds to a
C++ timespan.

Signed-off-by: Kefu Chai <kchai@redhat.com>

compressor: pass string_view instead of string

this allows us to find by string without creating a std::string
instance, even if this string is not nul terminated.

Signed-off-by: Kefu Chai <kchai@redhat.com>

compressor: put Compressor into TOPNSPC namespace

we want to add the support for on-the-wire compression to msgr v2, and
this feature will be shared by classic osd and crimson. but
Compressor.cc is also used by bluestore which is linked against by
crimson also. Compressor depends on CephContext which has two different
implementation in two namespaces for classic osd and crison.

so to avoid violating ODR, we should also put Compressor into two
different namespaces so it can be shared by alienstore and crimson in
the same executable.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38442 from badone/wip-bump-fmt-package-min-version

cmake: require fmt 6.0.0 and up

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38389 from trociny/wip-48412

librbd: fix regression in object map diff request

Reviewed-by: Jason Dillaman <dillaman@hotmail.com>

Merge pull request #35821 from Aran85/fix-ec-read

osd/ECBackend: optimize remaining read as readop contain multiple objects

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #37543 from ianwatsonrh/master

docs: With the addition of STS x-amz-security-token is now supported

Merge pull request #38401 from a16bitsysop/alpine-tests

test/lazy-omap-stats: Fix compilation on alpine linux

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38314 from badone/wip-monitor-reset-abort-if-is_shutdown

mon: paxos: Delete logger in destructor

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38410 from badone/wip-monitor-logger-deelete-in-destructor

mon: Delete logger in destructor

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37314 from ifed01/wip-ifed-faster-rm-p1

osd: optimize PG removal (part1)

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #38438 from ifed01/wip-ifed-fix-avl

os/bluestore: fix inappropriate ENOSPC from avl/hybrid allocator

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38436 from ktdreyer/prometheus-err

mgr/prometheus: don't store exception as e

Reviewed-by: Boris Ranto <branto@redhat.com>

cmake: require fmt 6.0.0 and up

The cmake search for a local package has lagged in regard to our usage
of it.

Fixes: https://tracker.ceph.com/issues/48453
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #38400 from tchaikov/wip-pybind-use-non-deprecated-functions

pybind/rados: use rados_set_pool_full_try instead of the deprecated one

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #36384 from linuxbox2/wip-doc-ippolicy

rgw:doc: document support for (Not)IpAddress in S3 policy

os/bluestore: bring back probing available space for main dev allocator.

This feature was mistakenly removed while making a single allocator
for main device (https://github.com/ceph/ceph/pull/30838).

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #37546 from myoungwon/wip-revamp-tier-flush

osd: refactoring tier_flush()

Reviewed-by: Samuel Just <sjust@redhat.com>

doc/osd-config-ref: adjust default value for osd_delete_sleep_hybrid

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

osd: adjust osd_delete_sleep_ssd[_hybrid]

1) Do not differentiate all-flash and hybrid(ssd+hdd) deployments since
they both bound to SSD drive backing DB volume.
2) Drop osd_delete_sleep_hybrid from 2s to 1s
3) Raise osd_delete_sleep_ssd to 1s since 0 is a way too low - having
it that low tend to cause a significant negative imact on OSD's regular
operation.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: introduce removal perf counter

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: track kv utilization in kv_sync_thread

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: perf counters for omap_get_keys[_get_values]

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

osd/pg: use next when calling collection_list for pg removal

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #38449 from trociny/wip-rbd-flush-finisher

librbd: flush task finisher queue on unregistering watcher

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38399 from es-gyt/master

rbd: current pc value should be greater than last_pc

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38403 from zdover23/wip-doc-CLT-update-02-Dec-2020-1

doc/governance.rst - updating CLT list

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #38440 from bk201/wip-48454

mgr/dashboard: fix trusted_ip_list in iSCSI service form

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #38398 from tchaikov/wip-48210

pybind/mgr: unpin urllib3

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #38298 from bk201/wip-43165

mgr/dashboard: refactor /api/orchestrator/* endpoints

Reviewed-by: Courtney Caldwell <ccaldwel@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #38432 from rhcs-dashboard/orch-not-available-incorrect-status

mgr/dashboard: Fix for misleading "Orchestrator is not available" error

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>