Merge pull request #40451 from tchaikov/wip-dencoder-plugins

tools/ceph-dencoder: build dencoders as plugins

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #39761 from aclamk/wip-stop-sh-less-kill

stop.sh: prevent forced kill

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40348 from rhcs-dashboard/fix-49606-master

mgr/dashboard: improve telemetry opt-in reminder notification message

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Yaarit Hatuka <yaarithatuka@gmail.com>

tools/ceph-dencoder: build dencoders as plugins

to reduce the memory footprint when linking ceph-dencoder.

* src/tools/ceph-dencoder:
  * build dencoders as shared libraries named with the prefix of
    "den-mod-". so ceph-dencoder can find them
  * install dencoders into $prefix/lib/ceph/denc, so ceph-dencoder
    can find them
  * only expose "register_dencoders()" function from plugins.
  * load plugins in specified directory
* ceph.spec.in: package plugins
* debian: package plugins

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40519 from rhcs-dashboard/50077-vstart-runner-max-required-mgrs

qa/tasks/vstart_runner.py: start max required mgrs

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: rishabh-d-dave <NOT@FOUND>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40055 from yuvalif/wip-yuval-fix-49738

librgw/notifications: initialize kafka and amqp

Merge pull request #40532 from liu-chunmei/seastore_journal_fix

crimson/seastore:fix scan_extents in journal.cc

Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/seastore:fix scan_extents in journal.cc

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

Merge pull request #40265 from ceph/wip-1610

doc: Add Pacific release notes

Merge pull request #40483 from gregsfortytwo/wip-stretch-mon-location

Improve mon location handling for stretch clusters

Reviewed-by: Sam Just <sjust@redhat.com>

doc/releases/pacific.rst: Spacing

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #40521 from tchaikov/wip-mon-log

mon/Monitor: add a space before "*m" in logging message

Reviewed-by: Sage Weil <sage@redhat.com>

doc: rgw release notes for pacific

Signed-off-by: Casey Bodley <cbodley@redhat.com>

doc/releases/pacific.rst: add rbd highlights

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

doc: add links for pacific cephfs features

Note: cephfs-mirror does not yet have user-facing documentation to link
to.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc/releases/pacific.rst: add CEPHX_V2 release note

Landed in pacific in commit a6651bad8f1f ("auth: require CEPHX_V2 by
default").

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

doc/releases/pacific.rst: add RADOS highlights

Signed-off-by: Neha Ojha <nojha@redhat.com>

doc/releases/pacific.rst: add more release notes

Some of these were under >=15.0.0 in master PendingReleaseNotes.

Signed-off-by: Neha Ojha <nojha@redhat.com>

doc: include more CephFS updates

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc/releases/pacific: add Dashboard notes

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

doc: add more CephFS release notes for Pacific

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc/releases/pacific: a few adjustments

Signed-off-by: Sage Weil <sage@newdream.net>

doc/releases/index: add pacific to index

Signed-off-by: Sage Weil <sage@newdream.net>

doc/releases/pacific: draft outline, and upgrade instructions

Signed-off-by: Sage Weil <sage@newdream.net>

doc: Add release note for mclock scheduler refinements and profiles

Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>

doc: add release note for libcephsqlite

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc: remove aggressive scrubbing from pacific release notes

This was reverted in pacific by 673b5ea4ae988096d7165f7ff166f6c556345386

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

doc: Add Pacific release notes

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge PR #40509 into master

* refs/pull/40509/head:
qa/tasks/cephfs: create enough subvolumes

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #40515 from tchaikov/wip-system-zstd

cmake, script/run-make: enable WITH_SYSTEM_ZSTD on focal

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #40527 from ceph/wip-14219

doc/release/nautilus.rst: Correct regressed version

Merge pull request #40368 from zdover23/wip-doc-cephadm-osd-deploy-osd-rewrite-2021-Mar-25

doc/cephadm: rewrite part of "deploy osds"

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #40374 from zdover23/wip-doc-cephadm-osd-dry-run-rewrite-2021-Mar-25

doc/cephadm: rewrite "dry run" section in osd.rst

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc/release/nautilus.rst: Correct regressed version

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #40370 from MrFreezeex/fix-perfdump-rbd-mirror

rbd-mirror: fix UB while registering perf counters

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #40525 from tchaikov/wip-50079

cmake: do not pass extra param to crimson tests

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

cmake: do not pass extra param to crimson tests

Fixes: https://tracker.ceph.com/issues/50079
Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/Monitor: add a space before "*m" in logging message

for better readability, otherwise the log reads like:

handle_probe_probe mon.5 v2:192.168.1.234:3300/0mon_probe(probe 77f26cbf-918c-44c5-b2ff-63a1ae713839 ...

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/vstart_runner.py: start max required mgrs

Pass environment copy with max required mgrs when shell kwarg is True.

Fixes: https://tracker.ceph.com/issues/50077
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

osd: osdmap: do not assert target_v when encoding under stretch mode

As in dd63a3e21fb6d63b8951a6542fb09f98b79b8cd6 for the OSDMap, this
caused crashes when encoding for kernel clients, and is unnecessary
for servers because they are separately gated.

I did a full audit of every instance of "assert" I added to the codebase
to make sure this is the very last one of these issues.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mon: let users specify a crush location on boot, and send it in MMonJoin

Go to some effort to look at our location in the monmap and update it the
same way we update names or IP addresses when necessary. Let users
pass in the location on the CLI via "--set-crush-location".

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mon: messages: Extend MMonJoin so it can provide a crush_location on join

This will let newly-created monitors auto-join on startup in stretch mode, by
providing the needed location.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

script/run-make: enable WITH_SYSTEM_ZSTD on focal

to speed up the build for "make check"

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: allow use libzstd in system

since we are moving the test nodes from bionic to focal, we are able to
use the prebuilt libzstd libraries when running "make check". to speed
up the build and test, in this change:

* add FindZstd.cmake which allows us to use the libzstd in system
* extract BuildZstd.cmake for better readability
* add an option named "WITH_SYSTEM_ZSTD", which defaults to "OFF",
  so user can enable it on demand.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40449 from tchaikov/wip-mds-autoscaler-ann

pybind/mgr/mds_autoscaler: add type annotation

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #40505 from tchaikov/wip-progress

mgr/PyModule: put mgr_module_path before Py_GetPath()

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #40471 into master

* refs/pull/40471/head:
qa/suites/rados/cephadm: stop testing on broken focal kubic podman

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/tasks/cephfs: create enough subvolumes

... in a subvolume group so that the desired number of ephmerally pinned
subtrees are created for the subvolume group, which has the distributed
ephemeral pin policy set on it.

Fixes: https://tracker.ceph.com/issues/48411
Signed-off-by: Ramana Raja <rraja@redhat.com>

Merge pull request #40503 from ceph/wip-14219

doc: 14.2.19 Release Note

mgr/PyModule: put mgr_module_path before Py_GetPath()

pip comes with _vendor/progress. so there is chance to import the vendored
version of "progress" module instead of the "progress" mgr module, and
fail to import the latter.

in this change, the order of paths are rearranged so the configured
`mgr_module_path` is put before the return value of `Py_GetPath()`.

Fixes: https://tracker.ceph.com/issues/50058
Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: 14.2.19 Release Note

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #40496 from tchaikov/wip-cmake-test

cmake: pass unparsed args to add_ceph_test()

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

cmake: pass unparsed args to add_ceph_test()

this allows us to pass extra argument to the script or executable to
further customize its behavior when testing.

also fix all the callers of add_ceph_unittest(), as there is no need to
repeat the executable, which is also used as the name of the test.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: pass PARALLEL to add_ceph_unittest()

and parse it using cmake_parse_arguments(), more flexible this way.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: drop duplicated fmt::fmt in ceph_common_deps

we already have it in ceph_common_deps, and the already-existing one is
located after the new one. so linker should be able to find it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: improve telemetry opt-in reminder notification message

Added activition button and linked the word telemetry to telemetry documentation

Fixes: https://tracker.ceph.com/issues/49606
Signed-off-by: Waad Alkhoury <walkhour@redhat.com>

Merge pull request #40453 from tchaikov/wip-doc-confval

doc/rados/configuration: format confval with "confval" directive

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #40462 from cyx1231st/wip-seastore-fix-onode-tree-eagain

crimson/onode-staged-tree: allow non-empty DeltaRecorder to be destructed

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #40472 into master

* refs/pull/40472/head:
mgr/cephadm/upgrade: ignore deployed_by until mgr is upgraded

Reviewed-by: Adam King <adking@redhat.com>

Merge PR #40443 into master

* refs/pull/40443/head:
doc/rbd: missed blacklist -> blocklist

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #40390 into master

* refs/pull/40390/head:
cstart.sh: some improvements
cpatch: allow python 3.6 or 3.8
cpatch: behave with ninja (instead of make)

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #40409 into master

* refs/pull/40409/head:
qa/workunits/cephadm/test_cephadm: specify image separately

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #40428 into master

* refs/pull/40428/head:
win32*.sh: disable libcephsqlite when targeting Windows

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/cephadm/upgrade: ignore deployed_by until mgr is upgraded

Until we upgrade the mgr itself, we will never be able to make our
deployed daemons have a deployed_by == target_digests.  Ignore those
daemons until the mgr is the right version.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #40470 from idryomov/wip-msgr1-extra-tickets

auth/CephxClientHandler: explain why skipping extra_tickets is needed

Reviewed-by: Sage Weil <sage@redhat.com>

crimson/onode-staged-tree: allow non-empty DeltaRecorder to be destructed

Fixes: http://tracker.ceph.com/issues/50028
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/.../transaction_manager: add log for eagain

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

qa/suites/rados/cephadm: stop testing on broken focal kubic podman

See https://tracker.ceph.com/issues/49633

Signed-off-by: Sage Weil <sage@newdream.net>

auth/CephxClientHandler: explain why skipping extra_tickets is needed

Since nautilus, CEPHX_GET_AUTH_SESSION_KEY shares both the auth ticket
and the service tickets, sparing an extra round-trip to get the service
tickets via CEPHX_GET_PRINCIPAL_SESSION_KEY.  This applies to msgr1 as
well, but we don't take advantage of it on the client side.

However, fixing CephxClientHandler to do the right thing breaks msgr1.
Since msgr1 is on its way out, rather than also fixing MonClient just
document the bug and the current behaviour.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #40466 from inspur-wyq/patch-2

blk/spdk/NVMEDevice.cc:  remove unused variables

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40188 from rhcs-dashboard/fix-49869-master

mgr/dashboard: fix broken feature toggles

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38679 from ceph/dependabot/npm_and_yarn/src/pybind/mgr/dashboard/frontend/node-notifier-8.0.1

build(deps): bump node-notifier from 8.0.0 to 8.0.1 in /src/pybind/mgr/dashboard/frontend

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #39908 from rhcs-dashboard/fix-49645-master

mgr/dashboard: Remove username, password fields from Manager Modules/dashboard,influx

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #40023 from rhcs-dashboard/fix-49354-master

mgr/dashboard: Device health status is not getting listed under hosts section

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #40332 from rhcs-dashboard/prometheus-alert-notification-fix

mgr/dashboard: Fix for alert notification message being undefined

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>

Merge pull request #39917 from rhcs-dashboard/fix-43058-master

mgr/dashboard: warn password expiration in User Management

blk/spdk/NVMEDevice.cc:  remove unused variables

Signed-off-by: wangyunqing <wangyunqing@inspur.com>

Merge pull request #40240 from rhcs-dashboard/clear-cookies-on-logout

mgr/dashboard: Unable to login to ceph dashboard until clearing cookies manually

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #40458 from lxbsz/typo

client: fix typo in the comments

crimson/onode-staged-tree: fix typo in compare_full_key()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

client: fix typo in the comments

Signed-off-by: Xiubo Li <xiubli@redhat.com>

doc/rados/configuration: format confval with "confval" directive

for better formatting

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/dev/cephadm: quote command with backquotes

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: add a new directive "confval"

* so we can define configuration in a more structured way
* so the confvals can be cross referenced

see also https://www.sphinx-doc.org/en/master/extdev/appapi.html#sphinx.application.Sphinx.add_object_type

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40445 from liewegas/cephadm-upgrade-progress-version

mgr/cephadm: make upgrade progress bar mention target version, not repo digest

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #40447 from Daniel-Pivonka/cephadmdocapplyrgwflags

mgr/orchestrator: fix rgw realm and zone flags

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #40379 from adk3798/no-overwrite

mgr/cephadm: --no-overwite flag for apply command

Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #39969 from batrick/i49684

qa: wait for daemons to come up via cephadm

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #40396 from maydaycc/wip-49968

mgr: attr_list reassigned before use in diskprediction_local module

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40421 from badone/wip-fix-typo-get_device_id

common/blkdev: Remove trailing single quote

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40450 from smithfarm/wip-50007-alt

rpm: drop extraneous explicit sqlite-libs runtime dependency

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

rpm: drop extraneous explicit sqlite-libs runtime dependency

Commit 75980798f19b8c11efd75ba4aae3e491d4c99f98 introduced a new package,
libcephsqlite, with a hard RPM dependency on a package "sqlite-libs" which
does not exist in openSUSE.

Since the runtime library dependencies of libcephsqlite are handled by RPM
transparently, this line is not needed.

Fixes: https://tracker.ceph.com/issues/50007
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #40442 from Thingee/update-foundation-page

doc: Update Foundation page

Reviewed-by: Sage Weil <sage@redhat.com>

doc: Updating Foundation membership page

Signed-off-by: Mike Perez <thingee@gmail.com>

Merge pull request #40408 from varshar16/wip-doc-update-ninja

doc/install: update about ninja

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40345 from rzarzynski/wip-crimson-doc-waitstates-follow-up

doc/crimson: improve wording about the pipeline / wait states.

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/mds_autoscaler: add type annotation

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #38732 into master

* refs/pull/38732/head:
client: fire the finish_cap_snap() after buffer being flushed
client: simplify the iterating code
client: remove unused _flushed_cap_snap
client: clean up the code

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #40326 into master

* refs/pull/40326/head:
client: rebuild bl to avoid too many vector(> IOV_MAX)

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge PR #40394 into master

* refs/pull/40394/head:
osd: handle MOSDPGInfo
osd/PeeringState: init last_require_osd_release in ctor

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>