ms/async/ProtocolV2: add ms_die_on_bug and assert rxbuf/txbuf don't get big

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async/ProtocolV2: do not reenable pre_auth buffering on from reset_recv_state

This is specifically bad because we call reset_recv_state from
reuse_connection, which turns buffering back on on an already-authenticated
session.

Instead, reenable it only when we set the state to START_CONNECT.  (On
the accepting side, it is a fresh connection, so it starts out true.)
Also, we want to *disable* it on the connection we are reusing, which
might be in a pre-auth state, while we are in a post-auth state.

Fixes: http://tracker.ceph.com/issues/38746
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #26905 into nautilus

* refs/pull/26905/head:
mgr: Cleanup get_localized_module_option function

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge PR #26875 into nautilus

* refs/pull/26875/head:
common: implement HMACs on top of OpenSSL.
msg/async, v2: switch the pre-auth mechanism to HMAC-SHA256.
include/types: beef sha_digest_t up with encode and compare.
auth: add hmac_sha256() to CryptoKey.
msg/async, v2: introduce pre_auth exchanges with CRC32.
msg/async, v2: introduce pre_auth buffers.
msg/async, v2: rectify the encapsulation of rx_segments_{desc,data}.
msg/async, v2: rework decoding of MessageFrame.
msg/async, v2: limit the num_segments to non-empty segments.
msg/async, v2: drop the bl onwire space optimization in ControlFrames.
msg/async, v2: clean up ret handling in ProtocolV2::write().
msg/async, v2: drop next_payload_len as we don't need anymore.
msg/async, v2: drop temp_buffer and limitations driven by it.
msg/async, v2: switch to rx_buffer_t entirely.
msg/async, v2: rx continuations use buffer::ptr_node.
msg/async, v2: use bptr continuation for segment reading.
msg/async: introduce bptr-carrying continuations.
msg/async: replace CONTINUATION_PARAM() with specialized types.
msg/async, v2: ::_banner_exchange() takes CtRef instead of CtPtr.
msg/async: avoid extra pointers in continuation definitions.
msg/async, v2: dissect setting stream handlers into ::finish_auth().
msg/async, v2: drop ceph_msg_header2 handling from ControlFrames.
msg/async, v2: drop the SignedEncryptedFrame entirely.
msg/async, v2: reintroduce segment aligment. It's compile-time now.
msg/async, v2: generalize Frame about number of segments.
msg/async, v2: rework and generalize Frame encryption.
msg/async, v2: rework the class hierarchy - introduce MessageFrame.
msg/async, v2: rework the class hierarchy - introduce ControlFrame.
msg/async/ProtocolV2: remove obsolete AuthFlags

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #26918 from trociny/wip-rbd-bench-wait-on-interrupted

rbd: wait for all io complete when bench is interrupted

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr: Cleanup get_localized_module_option function

Address comments from https://github.com/ceph/ceph/pull/26736.

Signed-off-by: Volker Theile <vtheile@suse.com>

Merge PR #26948 into nautilus

* refs/pull/26948/head:
rbd-nbd: fix segfault for list-mapped command

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #26864 into nautilus

* refs/pull/26864/head:
rados: update to latest ver. of dmclock lib w performance enhancements

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: yanjun <yan.jun8@zte.com.cn>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #26921 into nautilus

* refs/pull/26921/head:
mgr/telemetry: fix 'telemetry {on,off}'

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #26924 into nautilus

* refs/pull/26924/head:
auth/cephx: fix decode_encrypt error handling
auth/cephx: fix encode_encrypt error handling

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #26951 from ricardoasmarques/validate-ceph-iscsi-config-version-nautilus

nautilus: mgr/dashboard: Validate `ceph-iscsi` config version

Reviewed-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: Validate `ceph-iscsi` config version

Signed-off-by: Ricardo Marques <rimarques@suse.com>
(cherry picked from commit b9b2b562ccf5dc6469d11912ee28dd2d932fdb48)

rbd-nbd: fix segfault for list-mapped command

The global ceph context is not initialized when list-mapped command
is issued.

The regression was introduced by a92628f2857390460e35ae60b8e06f16296da481.

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge PR #26935 into nautilus

* refs/pull/26935/head:
qa: extend MDS heartbeat grace for valgrind

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #26916 into nautilus

* refs/pull/26916/head:
qa: ignore MON_DOWN for volume-client testing

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #26894 into nautilus

* refs/pull/26894/head:
qa/standalone/erasure-code/test-erasure-code: adjust test to avoid m=0
erasure-code: ensure m >= 1
mon/OSDMonitor: set ec min_size to k + min(1, m - 1)

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #26919 into nautilus

* refs/pull/26919/head:
doc: add CephFS notes for nautilus

Reviewed-by: Sébastien Han <seb@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

doc: add CephFS notes for nautilus

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge remote-tracking branch 'gh/nautilus'

qa/standalone/erasure-code/test-erasure-code: adjust test to avoid m=0

_DD is k=2 m=0, which we don't allow.  Switch it to cDD.

I confess I don't fully understand why this was _DD to begin with, but
I'm pretty sure mapping is there to control the order of results so that
it can be mapped to the CRUSH rule output sanely, and the coding portion
is not relevant to the test.

Signed-off-by: Sage Weil <sage@redhat.com>

qa: extend MDS heartbeat grace for valgrind

Valgrind makes the MDS slowwwww. The newish mds_heartbeat_grace config allows
us to keep sending beacons to the mons even if the internal heartbeat is slow.
This avoids the laggy messages which are useful to grep for unrelated messaging
issues.

Fixes: http://tracker.ceph.com/issues/38723
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #26790 from ricardoasmarques/iscsi-passwords-with-slash

mgr/dashboard: iSCSI - Support iSCSI passwords with '/'

Reviewed-by: Patrick Nawracay <pnawracay@suse.com>

mgr/telemetry: fix 'telemetry {on,off}'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Always get iSCSI passwords decrypted

Fixes: https://tracker.ceph.com/issues/38630
Signed-off-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: Support iSCSI password with '/'

Signed-off-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #26765 from p-na/wip-pna-rbd-conf-table-search

mgr/dashboard: Fix search in `Source` column of RBD configuration list

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #26766 from p-na/wip-pna-fix-table-search

mgr/dashboard: Search broken for entries with null values

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #26800 from linuxbox2/wip-rgw-limitcheck-trunc

rgw: bucket limit check misbehaves for > max-entries buckets (usually…

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #26920 into master

* refs/pull/26920/head:
qa/tasks/mgr/test_module_selftest: fix localized value test
mgr/BaseMgrStandbyModule: parse prefix properly

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #26756 from badone/wip-f28-broken-Cython-dep

rpm: Install python2-Cython on f28

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

qa/tasks/mgr/test_module_selftest: fix localized value test

When mgr/selftest/testkey = foo and mgr/selftest/x/testkey is not set,
then get_localized() should return foo.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #26886 into master

* refs/pull/26886/head:
crush/CrushWrapper: ensure crush_choose_arg_map.size == max_buckets

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #26917 into master

* refs/pull/26917/head:
osd/osd_types: include omap and hit_set_archive bytes in pool usage

Reviewed-by: Neha Ojha <nojha@redhat.com>

auth/cephx: fix decode_encrypt error handling

**decode_encrypt** returns 0 on success, 1 otherwise

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #26788 from ricardoasmarques/iscsi-docs-password

doc/rbd/iscsi-target-cli: Update auth command

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

auth/cephx: fix encode_encrypt error handling

**encode_encrypt** returns 0 on success, 1 otherwise

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

rpm: Install python2-Cython in python2 environment

do_cmake.sh tests for Cython module version in python2

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

common: implement HMACs on top of OpenSSL.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: switch the pre-auth mechanism to HMAC-SHA256.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

include/types: beef sha_digest_t up with encode and compare.

BEWARE: sha_digest_t is part of our public. The commit tries to
preserve it as the cost of avoiding a clean-up.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

auth: add hmac_sha256() to CryptoKey.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: introduce pre_auth exchanges with CRC32.

CRC32 is just scaffolding and will be replaced with
HMAC-SHA256 or other msg auth code.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #26880 from pcuzner/prom-add-recovery-metrics

mgr/prometheus: Add recovery metrics

Reviewed-by: Boris Ranto <branto@redhat.com>

mgr/BaseMgrStandbyModule: parse prefix properly

Fixes: http://tracker.ceph.com/issues/38705
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #26712 from badone/wip-b64decode-decorator-traceback

pybind/mgr/restful: Decode the output of b64decode

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #26899 from dzafman/wip-38678

Minor cleanups in tests and log output

Reviewed-by: Neha Ojha <nojha@redhat.com>

rbd: wait for all io complete when bench is interrupted

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge PR #26890 into master

* refs/pull/26890/head:
ceph.spec.in: add libstoragemgmt to mgr package

Reviewed-by: Sage Weil <sage@redhat.com>

erasure-code: ensure m >= 1

Fixes: http://tracker.ceph.com/issues/38682
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #26893 into master

* refs/pull/26893/head:
qa: unmount clients prior to marking fs down

Reviewed-by: Zheng Yan <zyan@redhat.com>

osd/osd_types: include omap and hit_set_archive bytes in pool usage

- Include the hit_set_archive bytes if we're in legacy mode.

- Add in (estimated) omap bytes for both nautilus+ and legacy mode, since
we didn't break out omap usage by pool (oops!).

Fixes: http://tracker.ceph.com/issues/38703
Signed-off-by: Sage Weil <sage@redhat.com>

qa: ignore MON_DOWN for volume-client testing

The test restarts the monitors.

Fixes: http://tracker.ceph.com/issues/38704
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #26892 into master

* refs/pull/26892/head:
qa: stop testing simple messenger in CephFS suites

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #26503 from ricardoasmarques/iscsi-support-dots-in-name

mgr/dashboard: iSCSI - Adds support for pool/image names with dots

Reviewed-by: Ricardo Dias <rdias@suse.com>

crush/CrushWrapper: ensure crush_choose_arg_map.size == max_buckets

The crush/builder.c crush_add_bucket method resizes the max_buckets array
but a power of 2 when it has to expand, but the code in CrushWrapper was
assuming that if the array grew the pos for the new bucket would be the
last position in the new array.  This led to a situation where the
crush_choose_arg_map args array size didn't match max_buckets, and
eventually caused a crash.

Fixes: http://tracker.ceph.com/issues/38664
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #26891 from rjfd/wip-dashboard-iscsi-ssl

mgr/dashboard: iscsi: adds CLI command to enable/disable API SSL verification

Reviewed-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #26658 from ivancich/wip-lc-prefix-fix

rgw: fix unordered bucket listing when object names are adorned

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #26545 from ivancich/wip-rgw-gc-stall-min-fixes

rgw: resolve bug where marker was not advanced during garbage collection

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #26601 from ivancich/wip-rgw-gc-stall-fixes

rgw: resolve bugs and clean up garbage collection code

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #26599 from cbodley/wip-rgw-beast-default

rgw: make beast the default for rgw_frontends

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #26896 from dillaman/wip-rbd-cls-warning

cls/rbd: fixed uninitialized variable compiler warning

Reviewed-by: Mykola Golub <mgolub@suse.com>

ceph.spec.in: add libstoragemgmt to mgr package

The libstoragemgmt package is needed so we can activate the drive
blink-led feature so we should pull that package.

Signed-off-by: Sébastien Han <seb@redhat.com>

Merge pull request #26854 from ricardoasmarques/fix-mutual-password-input-type

mgr/dashboard: Fix iSCSI mutual password input type

Reviewed-by: Tiago Melo <tmelo@suse.com>

mon/OSDMonitor: set ec min_size to k + min(1, m - 1)

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Fix iSCSI mutual password input type

Signed-off-by: Ricardo Marques <rimarques@suse.com>

Merge PR #26906 into master

* refs/pull/26906/head:
Typo in Nautilus release notes: not -> now

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #26904 from SUSE/wip-orchestrator-deepsea-sharedsecret

mgr/deepsea: always use 'password' parameter for salt-api auth

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #26851 from rhcs-dashboard/36176-enable-api-tests-coverage

mgr/dashboard: enable coverage for API tests

Merge pull request #26669 from ricardoasmarques/iscsi-docs-disk-slash

doc/rbd/iscsi-target-cli: Update disk separator

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: tests: move helper classes to __init__.py

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: tests: add iscsi CLI commands testing

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: tests: update tests that mock mons kv store

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: tests: add mixin classes to mock mons KV store

Signed-off-by: Ricardo Dias <rdias@suse.com>

doc/mgr/dashboard: update iscsi documentation

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: iscsi: use `ssl_verify` configuration option in restclient

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: iscsi: add CLI command to disable API SSL verification

Signed-off-by: Ricardo Dias <rdias@suse.com>

Adds support for pool/image names with dots

According to https://github.com/ceph/ceph-iscsi/pull/29

Signed-off-by: Ricardo Marques <rimarques@suse.com>

Typo in Nautilus release notes: not -> now

Signed-off-by: Yaniv Kaul <ykaul@redhat.com>

mgr/deepsea: always use 'password' parameter for salt-api auth

Prior to https://github.com/saltstack/salt/commit/71d5601507, the
salt-api expected the password to be sent using the 'sharedsecret'
parameter if using shared secrets, and the 'password' parameter
for other authentication types.  The above commit unifies this so
that we always only need to use the 'password' parameter.

Signed-off-by: Tim Serong <tserong@suse.com>

Merge pull request #26902 from tchaikov/wip-doc-erasure-clay

doc/rados/operations: add clay to erasure-code-profile

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

doc/rados/operations: add clay to erasure-code-profile

so it's more visible.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/test-erasure-code.sh: Don't grep entire bluestore directory

Bluestore caused grep crash with "grep: memory exhausted" due to
size of "block" storage.

Fixes: http://tracker.ceph.com/issues/38678
Signed-off-by: David Zafman <dzafman@redhat.com>

qa: Don't create rbd pool because it creates an object

This also reverts commit 10b9626ea7b09e7c124067a2ce08a76eea073c9c.

Fixes: http://tracker.ceph.com/issues/38631
Signed-off-by: David Zafman <dzafman@redhat.com>

qa: Can't wait for clean when there aren't any pools/PGs.

Fixes: http://tracker.ceph.com/issues/38678
Signed-off-by: David Zafman <dzafman@redhat.com>

mon, osd: Fix log messages (missing space, convert int to string)

Binary data was ending up in the log file which could cause
terminals to produce garbage output.

Fixes: http://tracker.ceph.com/issues/38678
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge PR #26763 into master

* refs/pull/26763/head:
mgr/diskprediction_cloud: Fixed cannot import string.maketrans in the python 3.x

Reviewed-by: Sebastian Wagner <swagner@suse.com>

cls/rbd: fixed uninitialized variable compiler warning

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #26885 into master

* refs/pull/26885/head:
doc/releases/nautilus: add msgr2 refs

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #26736 into master

* refs/pull/26736/head:
mgr: push localized config handling into BaseMgr*Module
mgr: Fix broken get_localized_module_option function

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge PR #26856 into master

* refs/pull/26856/head:
os/bluestore: make osd_memory_limit default to .8x the cgroup limit
os/bluestore: observe osd_memory_target config options
common/config: set osd_memory_target based on POD_MEMORY_REQUEST
common/util: include group limit in daemon metadata
common/util: get_cgroup_memory_limit() helper
common/config: pass entity_type down into parse_env()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sébastien Han <seb@redhat.com>

Merge PR #26879 into master

* refs/pull/26879/head:
vstart.sh: make 'db' volume 1G instead of 64M
os/bluestore: make bluefs spillover messages nicer

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #26878 from dillaman/wip-pool-init-validation

librbd: bypass pool validation if "rbd_validate_pool" is false

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #26888 into master

* refs/pull/26888/head:
doc/releases/nautilus: add 'ceph-volume simple scan' to upgrade instructions

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Andrew Schoen <aschoen@redhat.com>

os/bluestore: make osd_memory_limit default to .8x the cgroup limit

We do this at runtime in _set_cache_sizes so that changes to these values
at runtime will be picked up.

Signed-off-by: Sage Weil <sage@redhat.com>

14.1.1

mgr: push localized config handling into BaseMgr*Module

We need both (1) the raw key name (in order to look up the correct option
name and transform the result into the correct type) and (2) the optional
prefix (in order to look up both possible keys).  This simplifies the
mgr_module implementation.

There is a weird mismatch between the way that BaseMgrModule and
BaseMgrStandbyModule implement this method.  I suspect they could be
unified to work the same, but I'll leave that for another day.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr: Fix broken get_localized_module_option function

Fixes: https://tracker.ceph.com/issues/38560
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge PR #26818 into master

* refs/pull/26818/head:
qa/cephfs: relax min_caps_per_client check

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa: unmount clients prior to marking fs down

Evicted RHEL7.5 clients may hang.

Fixes: http://tracker.ceph.com/issues/38677
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: stop testing simple messenger in CephFS suites

Simple messenger is on it's way out and it doesn't work with msgr2.

Fixes: http://tracker.ceph.com/issues/38676
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>