Sage Weil [Fri, 8 Feb 2019 14:59:52 +0000 (08:59 -0600)]
osd/OSD: respect osdmap message limits
We restructure the function significantly to let us work forwards instead
of backwards through epochs. We also make the assumption that the OSD
will have the maps it is supposed to have. If we for some reason fail to
load a map, we fall back to something minimal, but in general there is
little point to sending a more complete message when local maps are missing
since it shouldn't ever happen anyway and the receiver can always go get
maps from the mon.
Fixes: http://tracker.ceph.com/issues/38040 Signed-off-by: Sage Weil <sage@redhat.com>
Kefu Chai [Fri, 8 Feb 2019 08:09:11 +0000 (16:09 +0800)]
Merge pull request #26301 from tchaikov/wip-cxx17-aggre-init
changes to address FTBFS on fc30
Reviewed-by: Brad Hubbard <bhubbard@redhat.com> Reviewed-by: Kaleb S. KEITHLEY <kkeithle@redhat.com> Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Sage Weil [Thu, 7 Feb 2019 22:51:39 +0000 (16:51 -0600)]
Merge PR #26059 into master
* refs/pull/26059/head:
mon/MonClient: fix keepalive with v2 auth
msg/async/ProtocolV2: reject peer_addrs of -
msg/async/ProtocolV2: clean up feature management
mon/MonClient: set up rotating_secrets, etc before msgr ready
msg/async: let client specify preferred order of modes
msg/async/ProtocolV2: include entity_name, features in reconnect
msg/async/ProtocolV2: fix write_lock usage around AckFrame
qa/suites/rados/verify/validator/valgrind: debug refs = 5
qa/standalone/ceph-helpers: fix health_ok test
auth/AuthRegistry: only complain about disabling cephx if cephx was enabled
auth/AuthRegistry: fix locking for get_supported_methods()
auth: remove AUTH_UNKNOWN weirdness, hardcoded defaults.
msg/async/ProtocolV2: remove unused get_auth_allowed_methods
osd: set up messener auth_* before setting dispatcher (and going 'ready')
mon/AuthMonitor: request max_global_id increase from peon in tick
mon: prime MgrClient only after messengers are initialized
qa/suites/rados/workloads/rados_api_tests.yaml: debug mgrc = 20 on mon
auth: document Auth{Client,Server} interfaces
auth: future-proof AUTH_MODE_* a bit in case we need to change the encoding byte
mon/MonClient: request monmap on open instead of ping
mgr/PyModuleRegistry: add details for MGR_MODULE_{DEPENDENCY,ERROR}
crimson: fix build
mon/MonClient: finsih authenticate() only after we get monmap; fix 'tell mgr'
mon: add auth_lock to protect auth_meta manipulation
ceph-mon: set up auth before binding
mon: defer initial connection auth attempts until initial quorum is formed
mon/MonClient: make MonClientPinger an AuthCleint
ceph_test_msgr: use DummyAuth
auth/DummyAuth: dummy auth server and client for test code
mon/Monitor: fix leak of auth_handler if we error out
doc/dev/cephx: re-wordwrap
doc/dev/cephx: document nautilus change to cephx
vstart.sh: fix --msgr2 option
msg/async/ProtocolV2: use shared_ptr to manage auth_meta
auth/Auth{Client,Server}: pass auth_meta in explicitly
mon/MonClient: behave if authorizer can't be built (yet)
osd: set_auth_server on client_messenger
common/ceph_context: get_moduel_type() for seastar cct
auth: make connection_secret a std::string
auth,msg/async/ProtocolV2: negotiate connection modes
auth/AuthRegistry: refactor handling of auth_*_requred options
osd,mgr,mds: remove unused authorize registries
switch monc, daemons to use new msgr2 auth frame exchange
doc/dev/msgr2: update docs to match implementation for auth frames
auth/AuthClientHandler: add build_initial_request hook
msg/Messenger: attach auth_client and/or auth_server to each Messenger
auth: introduce AuthClient and AuthServer handlers
auth: codify AUTH_MODE_AUTHORIZER
msg/Connection: track peer_id (id portion of entity_name_t) for msgr2
auth/AuthAuthorizeHandler: add get_supported_methods()
auth/AuthAuthorizeHandler: fix args for verify_authorizer()
auth: constify bufferlist arg to AuthAuthorizer::add_challenge()
auth/cephx: share all tickets and connection_secret in initial reply
msg/async,auth: add AuthConnectionMeta to Protocol
auth/AuthClientHandler: pass in session_key, connection_secret pointers
auth/AuthServiceHandler: take session_key and connection_secret as args
auth/cephx: pass more specific type into build_session_auth_info
mon/Session: separate session creation, peer ident, and registration
mon/AuthMonitor: bump max_global_id from on_active() and tick()
mon/AuthMonitor: be more careful with max_global_id
mon: only all ms_handle_authentication() if auth method says we're done
mon/AuthMonitor: fix "finished with auth" condition check
auth: clean up AuthServiceHandler::handle_request() args
auth: clean up AuthServiceHandler::start_session()
mon/AuthMonitor: drop unused op arg to assign_global_id()
msg/async: separate TAG_AUTH_REQUEST_MORE and TAG_AUTH_REPLY_MORE
msg/async: consolidate authorizer checks
msg/async: move get_auth_allowed into ProtocolV2.cc
mon/MonClient: trivial cleanup
Sage Weil [Wed, 6 Feb 2019 03:41:14 +0000 (21:41 -0600)]
msg/async/ProtocolV2: include entity_name, features in reconnect
- A connects to B
- A sends client_ident
- fault before A gets server_ident, so A doesn't know B's features or name
- B reconnects to A
- connection established
A thinks B is unknown.0 and has not idea what the featurs are.
Fix this by including id and featurs in reconnect. We don't know the type, but that is
included in TAG_HELLO in another branch, which will be merged separately; add a
This is what the old code does so I kept it but I don't think it makes any sense.
Same with the defaults; let's just set the config option to something valid.
Sage Weil [Sun, 3 Feb 2019 13:41:26 +0000 (07:41 -0600)]
qa/suites/rados/workloads/rados_api_tests.yaml: debug mgrc = 20 on mon
Seeing some hangs when the mon is forwarding mgr commands (pg deep-scrub)
to the mgr. This is a buggy test (it should send it to the mgr directly)
but it is helpful to verify the mon forwarding behavior works.
Sage Weil [Fri, 1 Feb 2019 17:09:42 +0000 (11:09 -0600)]
mon/MonClient: request monmap on open instead of ping
The ping is useless. The MMonGetMap ensures we get a monmap (and finish
authenticate()) before we get any other maps/messages, like mgr_map.
Getting other maps sooner rather than later can be confuse to MonClient
users because they will get dispatched MMgrMap before the authenticate()
call has returned.
Sage Weil [Thu, 31 Jan 2019 20:05:37 +0000 (14:05 -0600)]
mon/MonClient: finsih authenticate() only after we get monmap; fix 'tell mgr'
We used to get a valid monmap before we finished the MAuth exchange and
returned from authenticate(). Now, we finish authenticating before we even
send or receive a message, so authenticate() returns quickly. This
confuses many callers, and is probably a bad idea. So, rejigger the
_finish_auth and _finish_hunting callers so that we finish hunting as soon
as we have picked a mon but don't finish_auth if we have not gotten our
first monmap.
Sage Weil [Thu, 31 Jan 2019 19:10:31 +0000 (13:10 -0600)]
mon: add auth_lock to protect auth_meta manipulation
In particular, we could be handling a get_auth_request() on a reconnect
while also running handle_auth_request() on a racing connection between
monitors.
Sage Weil [Tue, 29 Jan 2019 17:57:55 +0000 (11:57 -0600)]
msg/async/ProtocolV2: use shared_ptr to manage auth_meta
When we reconnect a session, we need to move the new connection's auth_meta
over to the existing connection. However, the existing connection may
have a thread that is unlocked and calling into an AuthClient or AuthServer
method making good use of the old auth_meta.
Resolved this by making auth_meta a shared_ptr and taking a local ref
before dropping the connection lock. This way we are free to move the
auth_meta over to the new connection as long as we are holding the lock,
and at the same time the existing connection can fiddle with the old
auth_meta without being disturbed. (That old auth_meta is about to get
discarded, but we still need to prevent the two threads from stomping on
each other.)
This also cleans up the reset_recv_state() a bit since we can simply
replace the old auth_meta with a totally fresh one without worrying about
what kind of state might be lurking in there.
Sage Weil [Wed, 23 Jan 2019 16:14:16 +0000 (10:14 -0600)]
auth: make connection_secret a std::string
Move connection mode decision to initial auth_request point so that it
can inform auth implementation how big the connection secret should be.
Pass that value through where appropriate.
The connection_secret is now a std::string filled with random bytes.
For now the v2 protocol just uses the session_key CryptoKey to encrypt,
but this is about to change.
- crc: crc32c checksums to protect against bit errors. No secrecy or
authenticity guarantees, so a MITM could alter traffic in flight.
- secure: cryptographic secrecy and authenticity proection (i.e, encrypted
and signed).
We do not include a 'signed' mode that provides authenticity without
secrecy because the cryptographic protocols appear to be faster than
SHA-2.
New settings:
- ms_cluster_mode : mode(s list) for intra-cluster connections
- ms_service_mode : mode(s list) for daemons to allow
- ms_client_mode : mode(s list) for clients to allow
The msgr2 protocol is expanded slightly to negotiate a mode. Client
shares it's allowed/preferred modes, and server picks one as auth finishes.
Negotiation is independent of the authentication, except that the
authentiction mode may precluse certain choices. Specifically, AUTH_NONE
does not support 'secure', only 'crc'.
Sage Weil [Mon, 21 Jan 2019 16:22:26 +0000 (10:22 -0600)]
switch monc, daemons to use new msgr2 auth frame exchange
- MonClient implements AuthClient to authenticate as a client
- MonClient implements AuthServer to allow daemons to verify authorizers
- Monitor implements AuthServer to allow clients to authenticate with
an exchange of msgr2 frames
- Monitor implements AuthClient to authenticate with other monitors
After this change ProtocolV1 and SimpleMessenger still use all of the
old Dispatcher-based callbacks, but ProtocolV2 doesn't need them at
all (except for ms_handle_authentication when we finish).
Sage Weil [Thu, 7 Feb 2019 15:49:54 +0000 (09:49 -0600)]
Merge PR #26306 into master
* refs/pull/26306/head:
ceph_test_msgr: fix server->client addr discovery
msg/{async,simple}: make learned_addr a bit smarter
msg/async: very protocol type when looking up existing connections
common/LogEntry: use as_legacy_addr()
msg/msg_types: add entity_addrvec_t::as_legacy_addr()
msg/async/AsyncMessenger: drop weird assert
msg/Messenger: be less verbose
msg/simple: learn client addr as type any
msg/simple: set_addr_unknowns(): use front(), not legacy_addr()
msg/msg_types: drop any: prefix
remove all Messenger::get_myaddr() users
test/crimson: fix DummyAuthAuthorizer
mon/MonClient: kill get_myaddr()
msg/async/ProtocolV1: use legacy v1 address, not any, in protocol
msg/Messenger: add get_myaddr_legacy()
msg/msg_types: add entity_addr_t::is_any()
msg/async: msgr2: send entity type and peer_address in Tag::HELLO frame
msg/async: identify client using any: addr
Kefu Chai [Thu, 7 Feb 2019 09:51:55 +0000 (17:51 +0800)]
mgr: install tox==2.9.1 for tox tests
* pybind/mgr/ansible/requirements.txt:
in src/pybind/mgr/ansible/tox.ini, tox>=2.8.1 is required. to be
consistent with other requirements.txt in Ceph, tox 2.9.1 will be
installed for these tox environments.
* pybind/mgr/orchestrator_cli/requirements.txt:
in src/pybind/mgr/orchestrator_cli: tox>=2.5 is required. to be
consistent with other requirements.txt in Ceph, tox 2.9.1 will be
installed for these tox environments.
With msgr2 the initial kickoff of an authentication handshake is client ->
server, while with msgr1 it was server -> client. So existing
implementations have an empty initial message (outside of the messenger's
envelope). Future auth implementations that are msgr2 only (e.g., krb)
may want to make use of this initial payload.
Sage Weil [Sun, 20 Jan 2019 23:03:18 +0000 (17:03 -0600)]
auth: introduce AuthClient and AuthServer handlers
These will be the primary interfaces consumed by the messenger and
implemented by either MonClient (regular client, or service daemon) or
Monitor for doing authentication.
Sage Weil [Sun, 20 Jan 2019 22:51:12 +0000 (16:51 -0600)]
auth: codify AUTH_MODE_AUTHORIZER
The AuthAuthorizer encoding always begins with byte 0x01. Codify that
as AUTH_MODE_AUTHORIZER so that we can distinguish an authorizer from
something else (e.g., an attempt to authenticate and get an initial auth
ticket with the mon).
Sage Weil [Wed, 16 Jan 2019 16:46:34 +0000 (10:46 -0600)]
auth/cephx: share all tickets and connection_secret in initial reply
Previously, we would give the client the auth ticket, like a rbd TGT
(ticket granting ticket), and the client would then ask for all of the
other tickets it wants in a separate message.
Instead, have the client specify which tickets it wants up front and pass
them all at the same time.
Also, generate and share the connection_secret, which will be used for
encryption.
Sage Weil [Wed, 16 Jan 2019 20:57:13 +0000 (14:57 -0600)]
msg/async,auth: add AuthConnectionMeta to Protocol
This will hold all of the authentication-related state in an easy-to-find
section that can be accessed via a Connection* or by the protocol stack
(as needed).
Sage Weil [Mon, 14 Jan 2019 23:18:13 +0000 (17:18 -0600)]
mon: only all ms_handle_authentication() if auth method says we're done
Previously we would call ms_handle_authentication() possibly multiple
times, and without knowning whether it might succeed. Instead, only call
it when start_session() or handle_request() returns >0 to indicate that
we should.
Sage Weil [Wed, 6 Feb 2019 22:01:01 +0000 (16:01 -0600)]
ceph_test_msgr: fix server->client addr discovery
The client's myaddr will be an ANY address, but the internel connection table
will use a v1: or v2: address. Use the get_peer_addrs() to figure out how to
connect instead.
Sage Weil [Wed, 6 Feb 2019 12:23:16 +0000 (06:23 -0600)]
msg/{async,simple}: make learned_addr a bit smarter
Only set type ANY if we are a pure client; otherwise, preserve the
type. Also, only populate the addr if we have a blank ip (sometimes
we already know it from learn_addr_unknowns).
Sage Weil [Tue, 5 Feb 2019 11:08:00 +0000 (05:08 -0600)]
msg/async: very protocol type when looking up existing connections
Since we register client connections as any:, we may have either a ProtocolV1 or V2
connection. This happens when clients have an imprecise mon search list and connect
to the same mon via both v1 and v2, for example when you do something like
If we do encounter the other protocol type than what we expect, just mark it down and
proceed. This is only a temporarily case that happens during mon discovery, the client
is always prepared to retry, and it doesn't actually matter which one succeeds since
it will return a monmap and the client will adapt accordingly.
projects / ceph.git / log