crimson/net: set policy from peer_type

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

crimson/net: fix Socket::read_exactly(0) case

Do not raise read_eof when try to read 0 bytes.

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

test/crimson: fix perf_crimson_msgr to support v2

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

test/crimson: msgr should wait before shutdown

test_concurrent_dispatch() should wait for server->wait() before
shutting down its messengers.

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

test/crimson: unit test for v2 protocol

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

crimson/net: message frame exchanges (lossy part)

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

crimson/net: add keepalive timestamps to connection

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

crimson/net: protocolv2 handshake frame exchanges

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

crimson/net: ProtocolV2 auth part

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

crimson/net: dummy auth for protocol v2

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

crimson/net: add AuthConnectionMeta to Protocol

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

crimson/net: add peer_global_id to Connection

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

crimson/net: banner exchange and HelloFrame

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

crimson/net: support frame read/write for ProtocolV2

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

crimson/net: add target_addr to SocketConnection

represent the peer target_addr accepted/connected

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

crimson/net: add get_mytype() to Messenger

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

crimson/net: add global_seq in ProtocolV2

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

crimson/net: skeleton code for ProtocolV2 logic

crimson ProtocolV2 class is following a state-machine design style:
* states are defined in ProtocolV2::state_t;
* call `execute_<state_name>()` methods to trigger different states;
* V2 logics are implemented in each execute_<state_name>() methods, and
  with explicit transitions to other states at the end of the execute_*;
* each state is associated with a write state defined in Protocol.h:
  - none: not allowed to send;
  - delay: messages can be queued, but will be delayed to send;
  - open: dispatch queued message/keepalive/ack;
  - drop: not send any messages, drop them all.

crimson ProtocolV2 is alike async ProtocolV2, with some considerations:
* explicit and encapsulated client/server handshake workflow.
* futurized-exception-based fault handling, which can interrupt protocol
  workflow at any time in each state.
* introduced SERVER_WAIT state, meaning to wait for peer-client's socket
  to reset or be replaced, and expect no further reads.
* introduced an explicit REPLACING state, async-msgr would be at the
  NONE state during replacing.

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

crimson/net: apple-to-apple server async-msgr

The simplest async-msgr server which will have the same behavior with
crimson-msgr server for apple-to-apple performance test.

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

test/crimson: improved perf tool for crimson-msgr

New features:
* --jobs: start multiple client messengers from core #1 ~ #jobs;
* --core: can assign server core to get away from busy client cores;
* --rounds: a client will send <rounds>/<jobs> messages;

Improved:
* Better configuration report;
* Report individual client results plus a summary;
* Validate if CPU number is sufficient before running;
* Sleep 1 second while connecting, so it won't hurt performance;
* Simplify client logic and bug fixes;

Removed unecessary features:
* finish_decode() for MOSDOp;
* keepalive ratio;

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

crimson/net: fix messenger creation

sharded data structures should only be allocated in core#0, or the
program will hang during exit.

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

crimson/net: create SocketConnection based on address type

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

crimson/net: use entity_addrvec_t::front() inside messenger

Do not expect legacy-addr in order to bind v2 address.

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

crimson/net: added msgr2 setting for SocketConnection

Provide entrypoint for the future ProtocolV2 class.

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

crimson/net: implement factory method of Socket

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

crimson/net: introduce SocketFRef

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

crimson/net: introduce protocol-level abstraction

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

crimson/net: batch messages instead of chaining futures

Instead of chaining writes with send_ready, connection will batch
messages in out_q, and will reap them by write_events() in the open
state.

The performance of pingpong is 3.7 times better from observation.

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

crimson/net: centralized write_event()

* introduce write_state_t to decouple write behaviors from states.
* replace `h.promise` with `state_changed`, with a more general way to
  change write behaviors according to state switches.
* centralize write_event() to dispatch writes in the open state.
* friendly interface for v1/v2 protocol abstraction.

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

Merge PR #27044 into master

* refs/pull/27044/head:
common: add bool log_to_file option

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #27048 into master

* refs/pull/27048/head:
doc: add changelog for nautilus

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #27099 from tchaikov/wip-crimson-misc-logging

crimson/{net,mon,osd}: misc logging changes

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #27089 from tchaikov/wip-cmake-with-ninja

cmake: do not assume ${CMAKE_GENERATOR} == make

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27100 from ddiss/trivial_rgw_auth_keystone_gcc_warning

[trivial] rgw: fix minor compiler warning in keystone auth

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27094 from tchaikov/wip-cmake-cmake-ubsan

cmake: workaround of false alarm from ubsan

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27088 from tchaikov/wip-update-seastar

cmake,crimson: pick up latest seastar

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #26731 from wjwithagen/wjw-fix-cmake-os-release

build: FreeBSD does not have /etc/os-release

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #26607 from tchaikov/wip-cython-language_level

pybind: set language_level for cythonize explicitly

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

rgw: fix minor compiler warning in keystone auth

Signed-off-by: David Disseldorp <ddiss@suse.de>

Merge pull request #27093 from tchaikov/wip-crimson-test-alien-echo

test/crimson: update to accomodate Dispatcher changes

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: more verbose logging

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/{mon,osd}: don't log ms_dispatch messages

msgr already logs them

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/net: log tx/rx messages

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #27092 into master

* refs/pull/27092/head:
mailmap: sort entries
mailmap: sort organizationmap
mailmap: updates for nautilus
doc: mailmap updates for mimic
mailmap fixes for 12.2.0

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #26404 from sebastian-philipp/orchestrator-error-handling

mgr/orchestrator: Add error handling to interface

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Tim Serong <tserong@suse.com>

cmake: workaround of false alarm from ubsan

to workaround a GCC bug, see
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=88684

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/crimson: update to accomodate Dispatcher changes

Dispatcher::ms_get_authorizer() was changed in f22d433a, so update the
test accordingly

Signed-off-by: Kefu Chai <kchai@redhat.com>

mailmap: sort entries

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

mailmap: sort organizationmap

since this will make it easier for editing entries in the future

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

mailmap: updates for nautilus

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

cmake: disable -Werror when building seastar

when building seastar with a recent fmt, fmt complains:

src/fmt/include/fmt/time.h:13:2: error: #warning fmt/time.h is
deprecated, use fmt/chrono.h instead [-Werror=cpp]
 #warning fmt/time.h is deprecated, use fmt/chrono.h instead
  ^~~~~~~

Signed-off-by: Kefu Chai <kchai@redhat.com>

seastar: pickup the latest seastar

for couple improvements

* dbf7b6b0: add fast path for small writes
* 2f357317: IPv6 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: link against libfmt

Seastar::seastar does pull in libfmt library as a PUBLIC linkage. but we
should not depend on this. if we use fmt directly, we should link against it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: include headers if necessary

we should not rely on seastar or other headers to do this. if we use
fmt or std functions, we should include corresponding header(s).

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not assume ${CMAKE_GENERATOR} == make

* modules/BuildDPDK.cmake: always use "make"
* modules/BuildSPDK.cmake: always use "make"
  DPDK only support "make". if user wants to use other generator than
  "make", we will fail to build DPDK support. so we should always use
  "make" here. and before using "make", we need to detect it.
* cmake/modules/BuildRocksDB.cmake: use cmake for build
* src/compressor/zstd/CMakeLists.txt: use cmake for build
  just use whatever generator use specifies, and use ${CMAKE_COMMAND}
  for building the specified target, no need to repeat
  ${CMAKE_GENERATOR} for building the target.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: build fmt submodule if libfmt is not found

the recent seastar dropped fmt submodule, so we need to build from our
own fmt submodule.

Signed-off-by: Kefu Chai <kchai@redhat.com>

fmt: add fmt submodule back

seastar does not bring fmt as a submodule, so let's ready it by
ourselves.

This reverts commit 53a38bf57e10b681268378737955e50f6239e925.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27057 from yuriw/wip-yuriw-crontab

qa/tests: reduced frequency for luminous and mimic runs...

qa/tests: reduced frequency for luminous and mimic runs
added all suites for nauitlus

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #26786 from aclamk/fix-rocksdb-compaction-ranges

common/kv/rocksdb: Fixed async compations.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27046 from smithfarm/wip-38812

rpm: refrain from building ceph-resource-agents on SLE

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Tim Serong <tserong@suse.com>

Merge pull request #27054 from cbodley/wip-37770

doc/rgw: update civetweb rgw_frontends config example

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27035 from cbodley/wip-test-rgw-reshard-wait-clock

test/rgw: test_rgw_reshard_wait uses same clock for timing

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26853 from nathan-weinberg/more-cluster-tests

mgr/dashboard: Added breadcrumb tests to Manager modules and Alerts menu

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #27052 from rzarzynski/wip-rgw-drop_rgw_decode_pki_token

rgw: drop unused rgw_decode_pki_token().

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27025 from yangdongsheng/poll_timeout

rbd: krbd: return -ETIMEDOUT in polling

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

doc/rgw: update civetweb rgw_frontends config example

all of these civetweb options have to be on the same line as rgw_frontends

Fixes: https://tracker.ceph.com/issues/37770
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27050 from theanalyst/build-doc-warns

admin/build-doc: keep-going when finding warnings

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26534 from majianpeng/skip-calc-crc-header-for-async-msg

msg/async: skip repeat calc crc header in Message::encode

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #27009 into master

* refs/pull/27009/head:
ceph_release: the next release will be octopus

Reviewed-by: Kai Wagner <kwagner@suse.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #26514 from dillaman/wip-38381

pybind/rados: fixed Python3 string conversion issue on get_fsid

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26985 from rhcs-dashboard/38768-fix-sparkline-component

mgr/dashboard: fix sparkline component

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

Merge pull request #27034 from jan--f/mgr-prometheus-remove-scrape-duration

pybind/mgr/prometheus: remove scrape_duration metric

Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #26701 from adamemerson/wip-trip-the-light-clangtastic

fix clang build failures, and clean up warnings

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26836 from p-na/wip-pna-fix-osd

 mgr/dashboard: "1 osds exist in the crush map but not in the osdmap" breaks OSD page

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #26717 from ifed01/wip-ifed-objectstore-tool-fix

tools: no-mon-config switch for ceph-objectstore-tool.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26926 from rouming/epoll-lost-epollet

msg/async/EventEpoll: set EPOLLET flag on del_event()

Reviewed-by: Kefu Chai <kchai@redhat.com>

rbd: krbd: return -ETIMEDOUT in polling

We don't want to wait on uevent forever, but the return value
of polling in timeout is 0 rather than a negative value.

Fixes: http://tracker.ceph.com/issues/38792
Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

Merge pull request #26934 from sebastian-philipp/doc-rados-mon_command

doc/rados/api/python: Add documentation for mon_command

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26562 from familyuu/log

mgr/dashboard: Add date range and log search functionality

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #27065 from tchaikov/wip-crimson-mgr-client

crimson/osd: report pg_stats to mgr

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: connect OSD to mgr

so OSD is able to report to mgr periodically using mgr::Client

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: include MonClient.h in .cc

mon::Client's definition is only necessary in .cc

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: add facilities for reporting pg stats

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/mgr: add mgr client

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd/pg: accessors for pg stats/state

so OSD can collect pg stats and report them to mgr

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: create msgrs in main.cc

update PG to hold references of messengers instead of pointers to them,
as their lifecycle will be managed in main.cc

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27045 from smithfarm/wip-38810

rpm: fix "rhel <= 7" conditional

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Tim Serong <tserong@suse.com>

Merge pull request #27066 from tchaikov/wip-crimson-sharded-stop

crimson/osd: create msgrs in main.cc

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/os: use transparent comparator in xattr

so we don't need to allocate a temporary string for looking up OI or SS
attribute

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: cancel timers before restart

otherwise we will rearm already-armed timer when booting up OSD again.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/net: call do_shutdown() in SocketMessenger::stop()

so sharded<SocketMessenger> can stop the messenger service with stop()

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: create msgrs in main.cc

messengers are sharded<Service>. we should not create them in another
sharded service's start() method. to ensure the ordering of stop of
sharded services, we should create the sharded services in main().
and register their stop() method in the proper order.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27064 from tchaikov/wip-crimson-pg-init

crimson/osd: init PG with more info

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: init primary state in PG::read_state()

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: add PG::read_state()

to load pg_info and past_intervals

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: skip fast_info if it is not around

fast_info is optional. for instance, there is chance that it
is the first time the info is written to store, in that case,
there is no delta, i.e. fast info yet.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: pass pgid, osdmap and msgr to PG's ctor

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27062 from tchaikov/wip-crimson-ceph-opts-in-argv

crimson/osd: pass unknown args to ConfigProxy::parse_args()

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #27003 from tchaikov/wip-crimson-for-classic

osd,messages: changes for preparing for crimson-osd

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>