msg/async/ProtocolV2: clear dispatch throttle on connection stop

This aligns us with the ProtocolV1 implementation.

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async: fix should_use_msgr2 behavior (including monc)

Be consistent about whether we should use v2 to connect or not, and fix
teh monclient check to use the same logic.

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async/AsyncMessenger: clear need_addr *after* we set our new addr

We check need_addr at the top without a lock held, so we need to be sure
we finished our work before we clear it, or else when there are two racing
threads the first will get the clock and clear the value and the second
will do nothing and see the unlearned value before the first finishes.

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async/ProtocolV2: fix handling for v2 client connection with v1 addr

Switch it to be v2.  Reject the case where the client sends and addrvec, though;
that should only happen for clients that did_bind, and they should only connect to
v2 if they have a v2 bound addr.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_msgr: do not connect_to on the client side

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async: do not connect from server

We could have a fault on a server-side of a non-lossy connectoin where
there is a fault and we have outgoing data queued.  Since we are a server,
we cannot connect; we should just go into standby and wait for the other
end to reconnect, or for someone to mark us down.

This fixes a failure reproduced by Messenger/MessengerTest.SyntheticInjectTest/0
where it would assert(!policy.server) in the connect code.

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async: do not use peer to addr detection; use getsockname()

If of relying on the peer to tell us what address we are connecting from,
look at how our local socket is bound, and use that address.

This removes the possibility for error because we will infer our address
locally and that will be the one place it is decide; the server will just
use our value.  As things were previously, we had to make the local and
remote inference match, which was fragile.

This does take away the client's ability to discover if it is traversing
NAT to reach the server and learning its public/external address.  I
don't think anybody has ever tested this, so it probably didn't even work,
and I've never heard it come up as a requirement.

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async/ProtocolV2: always send non-empty addrvec for self

If we don't know our address yet, send the peer a 0.0.0.0 or :: address with an empty
port and a populated nonce.  That way the peer can infer our final addr the same way
we do from learned_addr.

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async: never fill out port in myaddr if we didn't bind

If we are a client and didn't bind, then we should not fill in the port for our
address.  The one the peer sent us is just the random port our outgoing connection
happened to land on!

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_msgr: use v2 addrs for simplemessenger

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async: msgr2: don't force write event on every message received

Signed-off-by: Ricardo Dias <rdias@suse.com>

msg/async/ProtocolV2: be forgiving in server identity check

If we parse something out of mon_host or are given an addr by a user,
we should succeed as long as our target_addr is one of the destination's
addrs.

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async/ProtocolV2: fault if we connect to the wrong peer

Someone else may have bound to the same port.  If they identify as someone
else, fault and back off.

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async: msgr2: clean cookie if connection failed in ACCEPT_SESSION

Signed-off-by: Ricardo Dias <rdias@suse.com>

msg/async/ProtocolV2: do not bump connect_seq for fault during ACCEPTING_SESSION

If we have a connection race, and we lose, we may end up with outgoing
messages *and* be in ACCEPTING_SESSION.  If we then fault, we want to
leave connect_seq at 0 to avoid triggering a reset.

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async: msgr2: don't send SESSION_RETRY_GLOBAL in handle_existing_connection

Signed-off-by: Ricardo Dias <rdias@suse.com>

msg/async: msgr2: organizing log messages

Signed-off-by: Ricardo Dias <rdias@suse.com>

msg/async: msgr2: fix connection fault when replacing

Signed-off-by: Ricardo Dias <rdias@suse.com>

msg/async: msgr2: fix replacing race handling

Signed-off-by: Ricardo Dias <rdias@suse.com>

msg/async: msgr2: fix connection race when existing connection is newer

Signed-off-by: Ricardo Dias <rdias@suse.com>

msg/async: msgr2: assign recv_stamp in handle_message

Signed-off-by: Ricardo Dias <rdias@suse.com>

msg/async: msgr2: fix peer_addrs discovery

Signed-off-by: Ricardo Dias <rdias@suse.com>

msg/async: msgr2: keep authorizer bufferlist across reconnects

Signed-off-by: Ricardo Dias <rdias@suse.com>

msg/async: msgr2: fix connection secret problems for WITH_SEASTAR builds

Signed-off-by: Ricardo Dias <rdias@suse.com>

msg/async: msgr2: send keepalive on connection race winner

Msgr2 version of PR #25754

Signed-off-by: Ricardo Dias <rdias@suse.com>

msg/async: msgr2: fix client address learning

Signed-off-by: Ricardo Dias <rdias@suse.com>

msg/async: msgr2: fix keepalive_ack message

Signed-off-by: Ricardo Dias <rdias@suse.com>

msg/async: msgr2: do not force updating rotating keys inline

Msgr2 version of PR #25859

Signed-off-by: Ricardo Dias <rdias@suse.com>

msg/async: msgr2: fix mark_down vs accept race

Msgr2 version of PR #25823

Signed-off-by: Ricardo Dias <rdias@suse.com>

msg/async: msgr2: unregister con from accept vs mark_down race

Msgr2 version of PR #25755

Signed-off-by: Ricardo Dias <rdias@suse.com>

auth/cephx/CephxSessionHandler: use connection_secret for encryption

Signed-off-by: Sage Weil <sage@redhat.com>

msg,cephx: establish a unique connection_secret for every connection

The session_key is unique for each time we newly authenticate with the
mon--e.g., for each client instantiation.  This is not ideal for
encryption, since we don't want to encrypt all connections with the same
key.

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async: msgr2: use sha256_digest_t to print signature hex strings

Signed-off-by: Ricardo Dias <rdias@suse.com>

types.h,rgw: merge sha*_digest_t definitions

Signed-off-by: Ricardo Dias <rdias@suse.com>

msg/async: msgr2: close connection when no authorizer is given

Signed-off-by: Ricardo Dias <rdias@suse.com>

msg/async: msgr2: formatting fixes

Signed-off-by: Ricardo Dias <rdias@suse.com>

msg/async: msgr2: send client v2 address when only v1 address is defined

Signed-off-by: Ricardo Dias <rdias@suse.com>

msg/async: msgr2: add payload length to banner

Signed-off-by: Ricardo Dias <rdias@suse.com>

msg/async: msgr2: check protocol state after fast dispatch

Signed-off-by: Ricardo Dias <rdias@suse.com>

msg/async: msgr2: reduce log level for sending messages event

Signed-off-by: Ricardo Dias <rdias@suse.com>

msg/async: msgr2: call verify authorizer when CEPH_AUTH_NONE is used

Signed-off-by: Ricardo Dias <rdias@suse.com>

msg/async: msgr2: store peer entity name in the protocol

Signed-off-by: Ricardo Dias <rdias@suse.com>

msg/async: msgr2: apply sign/encrypt to messages data payload

Signed-off-by: Ricardo Dias <rdias@suse.com>

msg/async: msgr2: encryption/decryption of frames

Signed-off-by: Ricardo Dias <rdias@suse.com>

cephx: added encrypt/decrypt bufferlist method to session handler

Signed-off-by: Ricardo Dias <rdias@suse.com>

msg/async: msgr2: refactored the frame structures

Signed-off-by: Ricardo Dias <rdias@suse.com>

cephx: add sign bufferlist method

Signed-off-by: Ricardo Dias <rdias@suse.com>

options: msgr2 enable/disable signing and encrytion options

Signed-off-by: Ricardo Dias <rdias@suse.com>

msg/async: msgr2: cephx authentication

Signed-off-by: Ricardo Dias <rdias@suse.com>

msg/async: msgr2: implement reconnect

Signed-off-by: Ricardo Dias <rdias@suse.com>

msg/async: msgr2: fault handling

Signed-off-by: Ricardo Dias <rdias@suse.com>

msg/async: msgr2: messange exchange phase

Signed-off-by: Ricardo Dias <rdias@suse.com>

msg/async: msgr2: message flow handshake

Signed-off-by: Ricardo Dias <rdias@suse.com>

msg/async: msgr2: authentication phase

Signed-off-by: Ricardo Dias <rdias@suse.com>

msg/async: msgr2: exchange peer_type in banner phase

Signed-off-by: Ricardo Dias <rdias@suse.com>

test/msgr: cloned test_msgr test for testing msgr2 protocol

Signed-off-by: Ricardo Dias <rdias@suse.com>

msg/async: msgr2: banner exchange

Signed-off-by: Ricardo Dias <rdias@suse.com>

msg/async: asyncconnection: update the source address info

Signed-off-by: Ricardo Dias <rdias@suse.com>

msg/async: move base class Protocol its own source file

Signed-off-by: Ricardo Dias <rdias@suse.com>

Merge pull request #25991 from jtlayton/wip-dashboard

mgr/dashboard: add a script to run an API request on a rook cluster

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #25774 from yehudasa/wip-rgw-tiering-3

rgw: rados tiering

Merge pull request #26073 from rhcs-dashboard/add-tox-ini-command

mgr/dashboard: added command to tox.ini

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #26020 from rhcs-dashboard/issue-37854

mgr/dashboard: Fixed performance details context for host list row selection

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kanika Murarka <kmurarka@redhat.com>

Merge pull request #26007 from rhcs-dashboard/upgrade-python-dev-deps

mgr/dashboard: upgraded python dev dependencies

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

mgr/dashboard: added command to tox.ini

* Added 'run' command to run arbitrary commands so
  you don't have to run the whole test/lint suite.
  Use case examples:
  - Run only 1 test:
  tox -e py3-run pytest tests/test_rgw_client.py::RgwClientTest::test_ssl_verify
  - Run pycodestyle for only 1 file:
  tox -e py3-run pycodestyle tests/test_rgw_client.py

Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #26032 from jtlayton/wip-rook

mgr/rook: remove dead code and fix bug in url fetching code

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #25895 from jan--f/c-v-func-test-local

ceph-volume: functional tests, add libvirt customization

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #26044 from tchaikov/wip-build-doc-pyyaml

doc: bump up sphinx and pyyaml versions

Reviewed-by: Alfredo Deza <adeza@redhat.com>

doc: bump up sphinx and pyyaml versions

* bump up pyyaml version to address CVE-2017-18342, see
  https://nvd.nist.gov/vuln/detail/CVE-2017-18342
* bump up sphinx to the latest stable

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26052 from tchaikov/wip-37976

mgr/zabbix: drop "total_objects" field

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #26054 from tchaikov/wip-crimson-dispatcher-fix-ftbfs

crimson/osd: remove "force_new" from ms_get_authorizer()

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #26045 from hsiang41/Ticket_33736

mgr/diskprediction_cloud: fix divide by zero when total_size is 0

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: remove "force_new" from ms_get_authorizer()

it was removed in e62f45d1, so update crimson-osd accordingly

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26046 from liewegas/wip-37976

mgr/telemetry: fix total_objects

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/zabbix: drop "total_objects" field

This field was removed from df output a while back in
342f309

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25712 from familyuu/configuration

mgr/dashboard: Add a 'clear filter' button to configuration page

Reviewed-by: Sebastian Krah <skrah@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/diskprediction_cloud: fix divide by zero when total_size is 0

Fix devide by zero when total_size is o.

Ref ticket: https://tracker.ceph.com/issues/37736

Signed-off-by: Rick Chen <rick.chen@prophetstor.com>

mgr/telemetry: fix total_objects

This field was removed from df output a while back in
342f309645df886fb96eb401634e38376553e6d9

Fixes: http://tracker.ceph.com/issues/37976
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #26031 into master

* refs/pull/26031/head:
mgr/BaseMgrModule: fix mon commands targetting a specific monitor
mgr/ActivePyModules: fix RestoreThread calls for get('devices')

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25490 from Aran85/fix-ec-using-cache

osd: keep using cache even if op will invalid cache

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25920 from wjwithagen/wjw-fix-freebsd-numa

common/numa: Add shim routines for NUMA on FreeBSD.

Merge pull request #25938 from majianpeng/msg-remove-memcopy-message-header

msg/async: reduce additional ceph_msg_header copy.

Reviewed-by: Xie Xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26021 from majianpeng/osd-remove-unused-code

osd: remove unused fields

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26040 from tchaikov/wip-blkdev-stat

common/blkdev: check retval of stat()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #26041 from tchaikov/wip-kvstore-tool-gcc-ice

tools/ceph_kvstore_tool: extract StoreTool into kvstore_tool.cc

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #26012 into master

* refs/pull/26012/head:
qa: add test that down fs does not ERR
mon/MDSMonitor: skip offline ERR for down fs

Reviewed-by: Douglas Fuller <dfuller@redhat.com>

Merge PR #25993 into master

* refs/pull/25993/head:
mds/FSMap: simplify encode/decode
mds: alias reference type for Filesystem
mds: add missing memory header include
include/encoding: eliminate smart_ptr temporaries

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #25976 into master

* refs/pull/25976/head:
Objecter::calc_op_budget: Fix invalid access to extent union member

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #25974 into master

* refs/pull/25974/head:
MDSMonitor: add missing OSDMonitor writeable check
MDSMonitor: codify constraint for writeable OSDMap

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #25973 into master

* refs/pull/25973/head:
qa: use simpler fs fail to bring fs down
MDSMonitor: add fs fail command

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Douglas Fuller <dfuller@redhat.com>

Merge PR #25971 into master

* refs/pull/25971/head:
qa: test hex dump_inode
pybind/ceph_argparse: accept int with non-decimal base

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #25650 into master

* refs/pull/25650/head:
client: add ceph_stat/ceph_lstat/ceph_fstat api

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

tool/ceph_kvstore_tool: extract duration calc out

simplify the code a little bit

Signed-off-by: Kefu Chai <kchai@redhat.com>

tools/ceph_kvstore_tool: extract StoreTool into kvstore_tool.cc

to workaround n bug in GCC 7.3, which leads to ICE like

build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.0.1-2751-gb268822/rpm/el7/BUILD/ceph-14.0.1-2751-gb268822/src/tools/ceph_kvstore_tool.cc:625:1:
internal compiler error: Segmentation fault
 }
 ^
Please submit a full bug report,
with preprocessed source if appropriate.
See <http://bugzilla.redhat.com/bugzilla> for instructions.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/BaseMgrModule: fix mon commands targetting a specific monitor

We just need to use the right variant of start_mon_command().

Fixes: http://tracker.ceph.com/issues/37967
Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ActivePyModules: fix RestoreThread calls for get('devices')

Fixes 61aa7e2e0230bff49e757c32932d1db4bcad5f67

Fixes: http://tracker.ceph.com/issues/37736
Signed-off-by: Sage Weil <sage@redhat.com>

common/blkdev: check retval of stat()

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa: use simpler fs fail to bring fs down

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

rgw: align head object size with pool alignment

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #25919 from yuriw/wip-yuriw-distros-master

qa/tests: added rhel 7.6