qa/tasks/ceph: only use monmaptool --addv if addr has [,:v]

Otherwise, we want the --add path, which has the logic to infer ports,
v2+v1, etc.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph_manager: make get_mon_status use mon addr

We don't have the 'mon addr' config property any more.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph: keep mon addrs in ctx namespace

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: log all osd addrs on boot

Signed-off-by: Sage Weil <sage@redhat.com>

msg/simple: behave when v2 and v1 addrs are present at target

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonClient: warn if global_id changes

Shouldn't happen!

Signed-off-by: Sage Weil <sage@redhat.com>

msg/Connection: add warning/note on get_peer_global_id

This field isn't populated for loopback connections because the msgr
doesn't have any insight into what global_id its user has.

Signed-off-by: Sage Weil <sage@redhat.com>

mds/MDSDaemon: clean up handle_mds_map debug output a bit

The old wording was misleading!

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/upgrade: debug mds

Signed-off-by: Sage Weil <sage@redhat.com>

mds/MDSRank: improve is_stale_message to handle addrvecs

If we get a connection on a loopback from ourselves, get_source_addrs()
will have everything we bound to, but the mdsmap may only have the v1
address.  Avoid the addrvec comparison by instead comparing the
ConnectionRefs.

NOTE: this implementation is a stopgap.  We should really maintain a map
of ConnectionRefs for the current up set and compare the ConnectionRefs
directly instead of comparing addr(vecs).

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async: make loopback detect when sending to one of our many addrs

Drop the assert just because it's inefficient and not necessary.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/upgrade: no aggressive pg num changes

We now run with mixed mons and old mgrs, so this won't work.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: require nautilus mons for require_osd_release=nautilus

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: require mimic mons for require_osd_release=mimic

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/thrash-old-clients: use legacy addr syntax in ceph.conf

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async: preserve peer features when replacing a connection

The features are now stored in the protocol implementation.  When we replace
an existing connection, copy those features so that our connect_msg_reply
calculates the correct features for the session.

This fixes an issue where a 3-mon cluster, after upgrading the two followers
but not the leader, was unable to include the (luminous) leader in the
quorum because it was seeing missing features in the connect reply, because
the new mons were replacing an old instance of the connection and weren't
copying the features, and that old instance had connect_msg.features == 0.

Add some debug lines that helped (finally) identify the problem.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph.py: move methods from teuthology.git into ceph.py directly; support mon bind * options

Having these live in teuthology.git is silly, since they are only consumed
by the ceph task, and it is hard to revise the behavior.

Revise the behavior by adding mon_bind_* options.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonMap: adjust build_initial behavior for mkfs vs probe

For the mkfs case, interpret an ambiguous port as a v2 address.  For probe,
try both.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonMap: improve ambiguous addr behavior

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/upgrade: spread mons a bit

This will mean 2/3 mons have default ports.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/rados/thrash-old-clients: keep mons on separate hosts

This ensures the mons can use default ports, ceph.conf won't have v1: or
v2: prefixes, and old clients will be happy.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/standalone/mon/misc.sh: tweak test to be more robust

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/mon_seesaw: expect v1/v2 prefix in addr

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMap: fix is_blacklisted() check to assume type ANY

Note that this still does a copy of the addr struct (as it did before).
This could be more efficient...

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: use ANY addr type for blacklisting

Client addresses are untyped in that they can connect to v1 or v2 server
endpoints, so blacklist them as TYPE_ANY.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/msg_types: TYPE_V1ORV2 -> TYPE_ANY

..and allow us to parse it.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/cephtool: fix blacklist test

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/upgrade: install old version with only v1 addrs

v1+v2 support is new in nautilus.

Signed-off-by: Sage Weil <sage@redhat.com>

common/options: by default, bind to both msgr v1 and v2 addresses

This will be the case for some time, until v1 clients are fully phased out.

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: add --msgr1, --msgr2, --msgr21 options

..and clean up the monmap IP generation.

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async/ProtocolV2: be flexible with server identity check

Signed-off-by: Sage Weil <sage@redhat.com>

msg/msg_types: fix entity_addrvec_t::parse() with null end arg

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/basic/msgr: no msgr2 addrs in initial monmaps

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph: add 'mon_bind_addrvec' and 'mon_bind_msgr2' options

- Sometimes we don't want to use v2 addrs
- Sometimes we don't want addrvecs at all (e.g., upgrades)

Signed-off-by: Sage Weil <sage@redhat.com>

monmaptool: add --addv argument to pass in addrvec directly

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/basic/msgr: do not use msgr2 with simplemessenger

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/basic/msgr: async is not experimental

Signed-off-by: Sage Weil <sage@redhat.com>

messages/MOSDBoot: fix compat with pre-nautilus

We can't make an addr -> addrvec transition without a compat encoding,
unfortunately (I screwed this one up back when then ADDR2 feature was
introduced).

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonMap: allow v1 or v2 to be explicitly specified along with part

If there is a v1: or v2: prefix, that's the protocol/addr type.  Otherwise,
we assume msgr2.

Signed-off-by: Sage Weil <sage@redhat.com>

msg/msg_types: allow parsing of IPs without assuming v1 vs v2

Signed-off-by: Sage Weil <sage@redhat.com>

msg/msg_types: default parse to v2 addrs

If we parse "1.2.3.4", parse to v2:1.2.3.4:0/0, not a v1 legacy addr.

Signed-off-by: Sage Weil <sage@redhat.com>

msg: standarize on v1: and v2: prefixes for *all* entity_addr_t's

This is more consistent than 1.2.3.4:0/0 being v1 and msgr2:1.2.3.4:0/0
being v2.

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: use msgr2 by default

This is useful because we tend to bind to random ports

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonMap: remove get_addr() methods

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-mon: adjust startup/bind/join sequence to use addrs

- Use the addrvecs in the monmap, and bind accordingly
- If we are joining the cluster, generate sane addrvec for ourselves
- Adjsut warning about mismatched ceph.conf.  Note that this may become
noisy for old clusters, mostly because the conf-based mon addr options
are ancient and crufty and should not be used.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: use MonMap::get_addrs() (instead of get_addr())

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonClient: change pending_cons to addrvec-based map

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonMap: fix set_addr() caller, kill wrapper

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonMap: remove addr-based add()

Signed-off-by: Sage Weil <sage@redhat.com>

monmaptool: fix --add to do either legacy or msgr2+legacy

Base the decision on the monmap's required_features vs FEATURE_NAUTILIUS.

Signed-off-by: Sage Weil <sage@redhat.com>

monmaptool: clean up iterator use a bit

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonMap: handle ambiguous mon addrs by trying both legacy and msgr

If we are given a mon addr (a raw IP or hostname) we don't know whether
it is a legacy or msgr2 entity_addr_t.  Either infer from the port number
or probe both protocol types.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonMap: take addrvec for set_initial_members

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonMap: use addrvecs for test instances

Signed-off-by: Sage Weil <sage@redhat.com>

mon: pass addrvec via MMonJoin

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonmapMonitor: fix 'mon add' to populate addrvec

If a port is not specified (or the legacy port is specified), add both
the new IANA and legacy addrs to the map.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonMap: addr -> addrvec

Modify MonMap to store an addrvec instead of just an addr.

Add temporary helper/wrapper methods to avoid changing all of the callers
(yet).

No big semantic changes yet!

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async/ProtocolV2: only update socket_addr if we learned our addr

Signed-off-by: Sage Weil <sage@redhat.com>

osd: go active even if mon only accepted our v1 addr

We may bind to v1 and v2 addrs, but we need to behave if the mon only
recognized our v1 addr and still go active.  That's because whether they
see our v2 addr depends on whether we connected to the mon via the v1 or
v2 port, and the mon may not be binding to v2 (yet, or ever).

Add a legacy_equals to entity_addrvec_t, and use that instead of
probably_equals for the OSD boot checks.  The probably_equals returns true
if the IP address portion of the address is empty.. which should never
happen in the OSD boot case since we have learned our real IP long before
we try to send osd_boot.

Signed-off-by: Sage Weil <sage@redhat.com>

test/msgr: add test for msgr2 protocol

Signed-off-by: Ricardo Dias <rdias@suse.com>

msg/async/ProtocolV2: share socket_addr and all addrs during handshake

Also, drop the sloppy addr checks.. I'm not sure we need them?

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async: print socket_addr for the connection

When we bind to multiple addrs, print the one the current connection is
using for the given connection.

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async: msgr2 protocol placeholder

This commit adds messenger V2 protocol. This V2 protocol is a clone of
the V1 protocol with the additional support for address vectors.
After the banner exchange the peers exchange their address vectors.

Signed-off-by: Ricardo Dias <rdias@suse.com>

msg/async: move ProtocolV1 class to its own source file

Signed-off-by: Ricardo Dias <rdias@suse.com>

msg/async: keep listen addr in ServerSocket, pass to new connections

When we accept a connection, we want to know what listening addr we
accepted on.  Because the addr can change after we create teh listening socket
(when we learn the addr and fill in the IP portion), instead store the position
in our myaddrs addrvec.

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async/AsyncMessenger: fix set_addr_unknowns

Fill in IP address from a v1 or v2 address into any v1 or v2 addresses.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #25620 into master

* refs/pull/25620/head:
os/bluestore: more robust handling for lack of per-pool stats cases.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #25632 into master

* refs/pull/25632/head:
test: add a json tester
ceph_json: parser sets data on all types when parsing

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #22800 from wjwithagen/wjw-ctest-timeout

cmake: allow setting of the CTest timeout during building.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25661 from joscollin/wip-doc-changing-monitors

doc: fix typo in add-or-rm-mons

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25639 from oliveiradan/wip-krb_pr25566_cleanup_oliveiradan

auth/krb: Fix Kerberos build warnings

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #25537 from jecluis/dev/wip-mgr-module-error

mgr: make module error message more descriptive

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #25595 from rzarzynski/wip-ec-drop_bl_crosstalk_dependency

ec: lrc doesn't depend on crosstalks between bufferlists anymore.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #25503 from ceph/rbd-mirror-ceph-base

spec: requires ceph base instead of common

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc: fix typo in add-or-rm-mons

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #25655 from gregsfortytwo/wip-rbd-snap-doc

doc: Clean up the snapshot consistency note

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #25577 into master

* refs/pull/25577/head:
test: allow ceph_kvstorebench to be build on FreeBSD

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #25508 into master

* refs/pull/25508/head:
kv/rocksdb_cache: Remove C style struct hack undefined behavior

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #25236 into master

* refs/pull/25236/head:
ceph-volume: Add venv to vstart cmake target
mgr/orchestrator: Add test orchestrator

Reviewed-by: Jeff Layton <jlayton@redhat.com>

doc: Clean up the snapshot consistency note

The old note made it sound like we weren't crash-consistent, and had a
confusing section about needing to clone before mounting.
You *do* need to clone the snapshot before mounting it, but that has
nothing to do fs freezing or consistency.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

auth/krb: Fix Kerberos build warnings

Signed-off-by: Daniel Oliveira <doliveira@suse.com>

Merge pull request #25207 from cyx1231st/wip-crimson-msgr-dispatch-events

crimson/net: encapsulate protocol implementations with states (remaining part)

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/net: let conn directly verify the authorizer with dispatcher

Signed-off-by: Yingxin <yingxin.cheng@intel.com>

crimson/net: remove the messenger gate

There is no need to use messenger gate if SocketConnection themselves
are properly gated.

Signed-off-by: Yingxin <yingxin.cheng@intel.com>

crimson/net: encapsulate protocol implementations with accept/connect

Signed-off-by: Yingxin <yingxin.cheng@intel.com>

crimson/net: no open connections in accepting_conns

It should an atomic operation to move the connection from
accepting_conns to connections when switch to the open state.

Signed-off-by: Yingxin <yingxin.cheng@intel.com>

crimson/net: don't execute_open() if exception is thrown

Signed-off-by: Yingxin <yingxin.cheng@intel.com>

crimson/net: encapsulate protocol implementations with open state

Signed-off-by: Yingxin <yingxin.cheng@intel.com>

crimson/net: remove unecessary future dependencies for accept/dispatch

Signed-off-by: Yingxin <yingxin.cheng@intel.com>

crimson/net: dispatch events in SocketConnection

* move dispatch(), and exception handling logics in accept() and
  connect() from SocketMessenger into SocketConnection, so we can manage
  the state transition in the same class and at the same abstraction
  level.
* gate the dangling futures in SocketConnection, because the
  connection's smart_ptr won't be hold by messenger any more during
  exception handling.
* don't return close() inside SocketConnection to prevent recursive
  gating -- dead lock.

Signed-off-by: Yingxin <yingxin.cheng@intel.com>

Merge pull request #25647 from tchaikov/wip-ftbfs-aarch64-gcc-8.1

test/crimson: fix FTBFS of unittest_seastar_perfcounters on arm64

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #25641 from tchaikov/cmake-rbd-tracing

cmake: let rbd_api depend on librbd-tp

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #25340 from sebastian-philipp/orchestrator-json-output

mgr/orchestrator: Add JSON output to CLI commands

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Tim Serong <tserong@suse.com>

Merge PR #25630 into master

* refs/pull/25630/head:
osd/OSDMap: cancel mapping if target osd is out

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #25598 into master

* refs/pull/25598/head:
common: avoid bl::get_current_ptr()-driven ref counting in denc.h.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #25585 from SUSE/wip-remove-cython-subinterpreter-check

cmake,deb,rpm: remove cython 0.29's subinterpreter check, re-enable build with cython 0.29+

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

test/crimson: fix FTBFS of unittest_seastar_perfcounters on arm64

this should address the GCC bug which causes following failure:

/home/jenkins-build/build/workspace/ceph-pull-requests-arm64/src/test/crimson/test_perfcounters.cc:35:9:
internal compiler error: in tsubst_decomp_names, at cp/pt.c:16537
         for (const auto &[path, perf_counter_ref] : by_path) {
         ^~~

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-volume: Add venv to vstart cmake target

Adds new add_custom_target `ceph-volume-venv-setup`
that creates a virtual environment, and
calls `python setup.py develop`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: Add test orchestrator

1. To be able to run the cli without an external orchestrator.
2. Run the CLI in Teuthology.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

cmake: let rbd_api depend on librbd-tp

whoever includes tracing/librbd.h should depends on librbd-tp.

this fix the FTBFS of:

ceph/src/librbd/librbd.cc:50:10: fatal error: tracing/librbd.h: No such
file or directory
 #include "tracing/librbd.h"
          ^~~~~~~~~~~~~~~~~~
compilation terminated.
src/librbd/CMakeFiles/rbd_api.dir/build.make:62: recipe for target
'src/librbd/CMakeFiles/rbd_api.dir/librbd.cc.o' failed

Signed-off-by: Kefu Chai <kchai@redhat.com>