Merge pull request #27098 from ming416/master

mgr/dashboard: auth ttl expired error

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge PR #27070 into master

* refs/pull/27070/head:
osd/OSDMap: add 'zone' to default crush map

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #27020 into master

* refs/pull/27020/head:
common/blkdev: handle devices with ID_MODEL as "LVM PV ..." but valid ID_MODEL_ENC

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #27021 into master

* refs/pull/27021/head:
msg: remove XioMessenger
qa/suites/rados/thrash-old-clients: add nautilus
qa/suites/rados/thrash-old-clients: add mimic v1 variant
qa/suites/rados/thrash-old-clients: add mimic
qa/suites/rados/thrash-old-clients: collapse msgr and client choice
qa: remove simplemessenger tests
ceph_test_msgr: remove simple
msg: remove SimpleMessenger

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #27101 into master

* refs/pull/27101/head:
mon/MgrStatMonitor: ensure only one copy of initial service map

Reviewed-by: Neha Ojha <nojha@redhat.com>

mgr/dashboard: auth ttl expired error

Fixes: https://tracker.ceph.com/issues/38428
Signed-off-by: ming416 <geenature@163.com>

Merge pull request #26940 from xiexingguo/wip-monc-add-con

mon/MonClient: weight-based mon selection

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26955 from liewegas/wip-slow-add

crush: various fixes for weight-sets, the osd_crush_update_weight_set option, and tests

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #26950 from xiexingguo/wip-mgr-fixes

mgr/Mgr: kill redundant sub_unwant call

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27068 from hjwsm1989/upmap-filter-out-underfull-buckets

crush: add root_bucket to identify underfull buckets

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #27067 from SUSE/wip-fix-38788

cmake: remove cython 0.29's subinterpreter check during install

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26923 from majianpeng/msg-optimize-check-loopback-con

msg/async/AsyncConnection: optimize check loopback connection.

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #27076 from MariusSchiffer/ceph-mgr-python3-fix

mgr/balancer: Python 3 compatibility fix

Reviewed-by: Kai Wagner <kwagner@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27106 from dalgaaf/dalgaaf-wip-fix-lrc-doc

doc: fix LRC documentation

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27071 from tchaikov/wip-crimson-osd-peering

crimson/osd: add minimal state machine for PG peering

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/osd: add minimal PG recovery FSM

after this change, PG is able to go clean as primary

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: advance pg in consume_map()

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd/pg: update info.stats

PG.info.stats is sent to mgr periodically for PG stats reporting,
also, it is sent to peer OSD in MOSDPGLog message when PG performs
peering. we need to keep it updated.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: add more peering facilities

* add PG::should_restart_peering() for telling if we should start
  a peering at seeing a new mapping
* add PG::start_peering_interval() for starting a peering
* add PG::activate() for activating a PG
* add PG::on_activated() to be called once a PG is activated
* add PG::send_to_osd() for sending message to given OSD

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: add operator<< for PG

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd/pg: add peer_activated

add facilities for updating peer_activated. it is used for tracking the
peers has activated and committed. once all of them ack the activation
proposed by primary, the PG is considered activated.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: add facilities for sending notify

we need to notify the primary osd that a PG hosted by it has changed,
and it needs to start peering for it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: maintain PG::want_acting

* add PG::choose_acting()
* add PG::proc_replica_info()

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: add PG::is_{acting,up}()

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: wait osdmap before processing peering evt

the constraint that

> peering messages/events from peers must remain ordered by peer

is still enforced. because, the peering message from a certain peer
will block the succeeding messages from this peer, if the latest osdmap
possessed by OSD is not new enough.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: add OSD::_send_active()

so PG can request monitor to update the OSD's up_thru in osdmap

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: handle MOSDPGLog in OSD

will handle it in PG in follow-up changes.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: handle MOSDPGQuery in OSD

will handle it in PG in follow-up changes.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: handle MOSDPGInfo in OSD

will handle it in PG in follow-up changes.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: handle MOSDPGNotify in OSD

will handle it in PG in follow-up changes.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: keep track of last_peering_reset

last_peering_reset is used to track the epoch when the most recent peering
started. it is used to tell if a message is targeting a changed PG. the
sender of a peering message will attach it. and the recipient will check
it against its own last_peering_reset. for instance, if a message claims
that that its map epoch is 16, while our last_peering_reset is 18, we will
consider this message stale, and probably drop it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: fix LRC documentation

Recovery from a failure in jerasure need only k reads and
not k+m-1.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

common/blkdev: handle devices with ID_MODEL as "LVM PV ..." but valid ID_MODEL_ENC

Some devices appear like so:

E: ID_MODEL=LVM PV LUClYG-Oyte-jcM6-npfZ-ncsl-ycL0-bkOH0m on /dev/sdn
E: ID_MODEL_ENC=WDC\x20WDS200T2B0A-00SM50\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20
E: ID_SERIAL=WDC_WDS200T2B0A-00SM50_183503800168
E: ID_SERIAL_SHORT=183503800168

To avoid upsetting our prioritization, defer to ID_MODEL_ENC over ID_MODEL
if ID_MODEL has "LVM PV" in it (as opposed to just using ID_SERIAL).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #27044 into master

* refs/pull/27044/head:
common: add bool log_to_file option

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

osd/OSDMap: add 'zone' to default crush map

This lets use have a root -> region -> zone -> host hierarchy, like you
will see in the public cloud.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MgrStatMonitor: ensure only one copy of initial service map

It's possible for create_pending to be called multiple times.

Fixes: http://tracker.ceph.com/issues/38839
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #27048 into master

* refs/pull/27048/head:
doc: add changelog for nautilus

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #27099 from tchaikov/wip-crimson-misc-logging

crimson/{net,mon,osd}: misc logging changes

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #27089 from tchaikov/wip-cmake-with-ninja

cmake: do not assume ${CMAKE_GENERATOR} == make

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27100 from ddiss/trivial_rgw_auth_keystone_gcc_warning

[trivial] rgw: fix minor compiler warning in keystone auth

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27094 from tchaikov/wip-cmake-cmake-ubsan

cmake: workaround of false alarm from ubsan

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27088 from tchaikov/wip-update-seastar

cmake,crimson: pick up latest seastar

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #26731 from wjwithagen/wjw-fix-cmake-os-release

build: FreeBSD does not have /etc/os-release

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #26607 from tchaikov/wip-cython-language_level

pybind: set language_level for cythonize explicitly

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

rgw: fix minor compiler warning in keystone auth

Signed-off-by: David Disseldorp <ddiss@suse.de>

Merge pull request #27093 from tchaikov/wip-crimson-test-alien-echo

test/crimson: update to accomodate Dispatcher changes

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: more verbose logging

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/{mon,osd}: don't log ms_dispatch messages

msgr already logs them

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/net: log tx/rx messages

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #27092 into master

* refs/pull/27092/head:
mailmap: sort entries
mailmap: sort organizationmap
mailmap: updates for nautilus
doc: mailmap updates for mimic
mailmap fixes for 12.2.0

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #26404 from sebastian-philipp/orchestrator-error-handling

mgr/orchestrator: Add error handling to interface

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Tim Serong <tserong@suse.com>

cmake: workaround of false alarm from ubsan

to workaround a GCC bug, see
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=88684

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/crimson: update to accomodate Dispatcher changes

Dispatcher::ms_get_authorizer() was changed in f22d433a, so update the
test accordingly

Signed-off-by: Kefu Chai <kchai@redhat.com>

mailmap: sort entries

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

mailmap: sort organizationmap

since this will make it easier for editing entries in the future

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

mailmap: updates for nautilus

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

cmake: disable -Werror when building seastar

when building seastar with a recent fmt, fmt complains:

src/fmt/include/fmt/time.h:13:2: error: #warning fmt/time.h is
deprecated, use fmt/chrono.h instead [-Werror=cpp]
 #warning fmt/time.h is deprecated, use fmt/chrono.h instead
  ^~~~~~~

Signed-off-by: Kefu Chai <kchai@redhat.com>

seastar: pickup the latest seastar

for couple improvements

* dbf7b6b0: add fast path for small writes
* 2f357317: IPv6 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: link against libfmt

Seastar::seastar does pull in libfmt library as a PUBLIC linkage. but we
should not depend on this. if we use fmt directly, we should link against it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: include headers if necessary

we should not rely on seastar or other headers to do this. if we use
fmt or std functions, we should include corresponding header(s).

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not assume ${CMAKE_GENERATOR} == make

* modules/BuildDPDK.cmake: always use "make"
* modules/BuildSPDK.cmake: always use "make"
  DPDK only support "make". if user wants to use other generator than
  "make", we will fail to build DPDK support. so we should always use
  "make" here. and before using "make", we need to detect it.
* cmake/modules/BuildRocksDB.cmake: use cmake for build
* src/compressor/zstd/CMakeLists.txt: use cmake for build
  just use whatever generator use specifies, and use ${CMAKE_COMMAND}
  for building the specified target, no need to repeat
  ${CMAKE_GENERATOR} for building the target.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: build fmt submodule if libfmt is not found

the recent seastar dropped fmt submodule, so we need to build from our
own fmt submodule.

Signed-off-by: Kefu Chai <kchai@redhat.com>

fmt: add fmt submodule back

seastar does not bring fmt as a submodule, so let's ready it by
ourselves.

This reverts commit 53a38bf57e10b681268378737955e50f6239e925.

Signed-off-by: Kefu Chai <kchai@redhat.com>

msg: remove XioMessenger

There is a lot of good stuff going on here, but nobody is investing in xio
and it is not expected to be the path forward for RDMA.  If that ever
changes, we can resurrect the code.  Until then, let's clean up the tree
and reduce friction for changes going forward.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #27057 from yuriw/wip-yuriw-crontab

qa/tests: reduced frequency for luminous and mimic runs...

qa/tests: reduced frequency for luminous and mimic runs
added all suites for nauitlus

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #26786 from aclamk/fix-rocksdb-compaction-ranges

common/kv/rocksdb: Fixed async compations.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27046 from smithfarm/wip-38812

rpm: refrain from building ceph-resource-agents on SLE

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Tim Serong <tserong@suse.com>

mgr/balancer: Python 3 compatibility fix
Fixes: http://tracker.ceph.com/issues/38831
Signed-off-by: Marius Schiffer <marius@mschiffer.de>

Merge pull request #27054 from cbodley/wip-37770

doc/rgw: update civetweb rgw_frontends config example

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27035 from cbodley/wip-test-rgw-reshard-wait-clock

test/rgw: test_rgw_reshard_wait uses same clock for timing

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26853 from nathan-weinberg/more-cluster-tests

mgr/dashboard: Added breadcrumb tests to Manager modules and Alerts menu

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #27052 from rzarzynski/wip-rgw-drop_rgw_decode_pki_token

rgw: drop unused rgw_decode_pki_token().

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27025 from yangdongsheng/poll_timeout

rbd: krbd: return -ETIMEDOUT in polling

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

qa/suites/rados/thrash-old-clients: add nautilus

v2+v1, v2, and v1 variants.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/thrash-old-clients: add mimic v1 variant

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/thrash-old-clients: add mimic

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/thrash-old-clients: collapse msgr and client choice

Hammer doesn't have async, and nautilus doesn't have simple.  Expand the
async default (v2v1) and v1-only choice and the client choice.

Signed-off-by: Sage Weil <sage@redhat.com>

doc/rgw: update civetweb rgw_frontends config example

all of these civetweb options have to be on the same line as rgw_frontends

Fixes: https://tracker.ceph.com/issues/37770
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27050 from theanalyst/build-doc-warns

admin/build-doc: keep-going when finding warnings

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26534 from majianpeng/skip-calc-crc-header-for-async-msg

msg/async: skip repeat calc crc header in Message::encode

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #27009 into master

* refs/pull/27009/head:
ceph_release: the next release will be octopus

Reviewed-by: Kai Wagner <kwagner@suse.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #26514 from dillaman/wip-38381

pybind/rados: fixed Python3 string conversion issue on get_fsid

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26985 from rhcs-dashboard/38768-fix-sparkline-component

mgr/dashboard: fix sparkline component

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

Merge pull request #27034 from jan--f/mgr-prometheus-remove-scrape-duration

pybind/mgr/prometheus: remove scrape_duration metric

Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #26701 from adamemerson/wip-trip-the-light-clangtastic

fix clang build failures, and clean up warnings

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26836 from p-na/wip-pna-fix-osd

 mgr/dashboard: "1 osds exist in the crush map but not in the osdmap" breaks OSD page

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #26717 from ifed01/wip-ifed-objectstore-tool-fix

tools: no-mon-config switch for ceph-objectstore-tool.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26926 from rouming/epoll-lost-epollet

msg/async/EventEpoll: set EPOLLET flag on del_event()

Reviewed-by: Kefu Chai <kchai@redhat.com>

rbd: krbd: return -ETIMEDOUT in polling

We don't want to wait on uevent forever, but the return value
of polling in timeout is 0 rather than a negative value.

Fixes: http://tracker.ceph.com/issues/38792
Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

qa: remove simplemessenger tests

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #26934 from sebastian-philipp/doc-rados-mon_command

doc/rados/api/python: Add documentation for mon_command

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26562 from familyuu/log

mgr/dashboard: Add date range and log search functionality

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

crush/CrushWrapper: make update_choose_args less chatty

These were now exposed due to cct being passed down to more
places.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/standalone/crush/crush-choose-args: add weight-set tests

Verify we have the expected behavior for creates and moves that
maintain bucket summation, both with and without the
osd_crush_update_weight_set option enabled.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/standalone/crush/crush-choose-args: fix test

- Make the initial weight-set actually consistent (summing)
- Fix the intermediate state so that it reflects a correctly
  maintained summation.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: apply osd_crush_update_weight_set for reweight, create-or-move

Since CrushWrapper no longer applies this setting at a low level,
where it can't tell what the real intention is, we instead apply
it at the top command level where we do.

Specifically, we use it to control whether the weight-set weights
are set for the commands

 osd crush reweight
 osd crush reweight-subtree
 osd crush move
 osd crush create-or-move

Note that this (indirectly) affects the way weight-set weights
are initialized for newly created OSDs, since those are added to
the crush map via the 'osd crush create-or-move' command.

Signed-off-by: Sage Weil <sage@redhat.com>

crush/CrushWrapper: move_item: do not clobber weight-set weights

Signed-off-by: Sage Weil <sage@redhat.com>

crush/CrushWrapper: create_or_move: make weight-set update optional

Signed-off-by: Sage Weil <sage@redhat.com>

crush/CrushWrapper: use adjust_item_weight_in_bucket for subtree reweight

This avoids clobbering weight-sets when you reweight, if the
adjust_weight_sets=false.

Signed-off-by: Sage Weil <sage@redhat.com>