tools: format headers for object cache daemon

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

tools: create local utils for object cache daemon

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

tools: clean up immutable object cache

- drop "ceph_" prefix
- prefix all options with immutable_object_cache_
- adding configurable immutable_object_cache_max_inflight_ops

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

tools: adding ceph level immutable obj cache daemon

The daemon is built for future integration with both RBD and RGW cache.
The key components are:
- domain socket based simple IPC
- simple LRU policy based promotion/demotion for the cache
- simple file based caching store for RADOS objs with sync IO interface
- systemd service/target files for the daemon

Signed-off-by: Dehao Shang <dehao.shang@intel.com>
Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

Merge pull request #27089 from tchaikov/wip-cmake-with-ninja

cmake: do not assume ${CMAKE_GENERATOR} == make

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27100 from ddiss/trivial_rgw_auth_keystone_gcc_warning

[trivial] rgw: fix minor compiler warning in keystone auth

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27094 from tchaikov/wip-cmake-cmake-ubsan

cmake: workaround of false alarm from ubsan

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27088 from tchaikov/wip-update-seastar

cmake,crimson: pick up latest seastar

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #26731 from wjwithagen/wjw-fix-cmake-os-release

build: FreeBSD does not have /etc/os-release

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #26607 from tchaikov/wip-cython-language_level

pybind: set language_level for cythonize explicitly

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

rgw: fix minor compiler warning in keystone auth

Signed-off-by: David Disseldorp <ddiss@suse.de>

Merge pull request #27093 from tchaikov/wip-crimson-test-alien-echo

test/crimson: update to accomodate Dispatcher changes

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #27092 into master

* refs/pull/27092/head:
mailmap: sort entries
mailmap: sort organizationmap
mailmap: updates for nautilus
doc: mailmap updates for mimic
mailmap fixes for 12.2.0

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #26404 from sebastian-philipp/orchestrator-error-handling

mgr/orchestrator: Add error handling to interface

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Tim Serong <tserong@suse.com>

cmake: workaround of false alarm from ubsan

to workaround a GCC bug, see
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=88684

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/crimson: update to accomodate Dispatcher changes

Dispatcher::ms_get_authorizer() was changed in f22d433a, so update the
test accordingly

Signed-off-by: Kefu Chai <kchai@redhat.com>

mailmap: sort entries

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

mailmap: sort organizationmap

since this will make it easier for editing entries in the future

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

mailmap: updates for nautilus

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

cmake: disable -Werror when building seastar

when building seastar with a recent fmt, fmt complains:

src/fmt/include/fmt/time.h:13:2: error: #warning fmt/time.h is
deprecated, use fmt/chrono.h instead [-Werror=cpp]
 #warning fmt/time.h is deprecated, use fmt/chrono.h instead
  ^~~~~~~

Signed-off-by: Kefu Chai <kchai@redhat.com>

seastar: pickup the latest seastar

for couple improvements

* dbf7b6b0: add fast path for small writes
* 2f357317: IPv6 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: link against libfmt

Seastar::seastar does pull in libfmt library as a PUBLIC linkage. but we
should not depend on this. if we use fmt directly, we should link against it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: include headers if necessary

we should not rely on seastar or other headers to do this. if we use
fmt or std functions, we should include corresponding header(s).

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not assume ${CMAKE_GENERATOR} == make

* modules/BuildDPDK.cmake: always use "make"
* modules/BuildSPDK.cmake: always use "make"
  DPDK only support "make". if user wants to use other generator than
  "make", we will fail to build DPDK support. so we should always use
  "make" here. and before using "make", we need to detect it.
* cmake/modules/BuildRocksDB.cmake: use cmake for build
* src/compressor/zstd/CMakeLists.txt: use cmake for build
  just use whatever generator use specifies, and use ${CMAKE_COMMAND}
  for building the specified target, no need to repeat
  ${CMAKE_GENERATOR} for building the target.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: build fmt submodule if libfmt is not found

the recent seastar dropped fmt submodule, so we need to build from our
own fmt submodule.

Signed-off-by: Kefu Chai <kchai@redhat.com>

fmt: add fmt submodule back

seastar does not bring fmt as a submodule, so let's ready it by
ourselves.

This reverts commit 53a38bf57e10b681268378737955e50f6239e925.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27057 from yuriw/wip-yuriw-crontab

qa/tests: reduced frequency for luminous and mimic runs...

qa/tests: reduced frequency for luminous and mimic runs
added all suites for nauitlus

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #26786 from aclamk/fix-rocksdb-compaction-ranges

common/kv/rocksdb: Fixed async compations.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27046 from smithfarm/wip-38812

rpm: refrain from building ceph-resource-agents on SLE

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Tim Serong <tserong@suse.com>

Merge pull request #27054 from cbodley/wip-37770

doc/rgw: update civetweb rgw_frontends config example

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27035 from cbodley/wip-test-rgw-reshard-wait-clock

test/rgw: test_rgw_reshard_wait uses same clock for timing

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26853 from nathan-weinberg/more-cluster-tests

mgr/dashboard: Added breadcrumb tests to Manager modules and Alerts menu

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #27052 from rzarzynski/wip-rgw-drop_rgw_decode_pki_token

rgw: drop unused rgw_decode_pki_token().

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27025 from yangdongsheng/poll_timeout

rbd: krbd: return -ETIMEDOUT in polling

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

doc/rgw: update civetweb rgw_frontends config example

all of these civetweb options have to be on the same line as rgw_frontends

Fixes: https://tracker.ceph.com/issues/37770
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27050 from theanalyst/build-doc-warns

admin/build-doc: keep-going when finding warnings

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26534 from majianpeng/skip-calc-crc-header-for-async-msg

msg/async: skip repeat calc crc header in Message::encode

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #27009 into master

* refs/pull/27009/head:
ceph_release: the next release will be octopus

Reviewed-by: Kai Wagner <kwagner@suse.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #26514 from dillaman/wip-38381

pybind/rados: fixed Python3 string conversion issue on get_fsid

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26985 from rhcs-dashboard/38768-fix-sparkline-component

mgr/dashboard: fix sparkline component

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

Merge pull request #27034 from jan--f/mgr-prometheus-remove-scrape-duration

pybind/mgr/prometheus: remove scrape_duration metric

Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #26701 from adamemerson/wip-trip-the-light-clangtastic

fix clang build failures, and clean up warnings

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26836 from p-na/wip-pna-fix-osd

 mgr/dashboard: "1 osds exist in the crush map but not in the osdmap" breaks OSD page

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #26717 from ifed01/wip-ifed-objectstore-tool-fix

tools: no-mon-config switch for ceph-objectstore-tool.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26926 from rouming/epoll-lost-epollet

msg/async/EventEpoll: set EPOLLET flag on del_event()

Reviewed-by: Kefu Chai <kchai@redhat.com>

rbd: krbd: return -ETIMEDOUT in polling

We don't want to wait on uevent forever, but the return value
of polling in timeout is 0 rather than a negative value.

Fixes: http://tracker.ceph.com/issues/38792
Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

Merge pull request #26934 from sebastian-philipp/doc-rados-mon_command

doc/rados/api/python: Add documentation for mon_command

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26562 from familyuu/log

mgr/dashboard: Add date range and log search functionality

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #27065 from tchaikov/wip-crimson-mgr-client

crimson/osd: report pg_stats to mgr

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: connect OSD to mgr

so OSD is able to report to mgr periodically using mgr::Client

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: include MonClient.h in .cc

mon::Client's definition is only necessary in .cc

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: add facilities for reporting pg stats

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/mgr: add mgr client

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd/pg: accessors for pg stats/state

so OSD can collect pg stats and report them to mgr

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: create msgrs in main.cc

update PG to hold references of messengers instead of pointers to them,
as their lifecycle will be managed in main.cc

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27045 from smithfarm/wip-38810

rpm: fix "rhel <= 7" conditional

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Tim Serong <tserong@suse.com>

Merge pull request #27066 from tchaikov/wip-crimson-sharded-stop

crimson/osd: create msgrs in main.cc

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/os: use transparent comparator in xattr

so we don't need to allocate a temporary string for looking up OI or SS
attribute

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: cancel timers before restart

otherwise we will rearm already-armed timer when booting up OSD again.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/net: call do_shutdown() in SocketMessenger::stop()

so sharded<SocketMessenger> can stop the messenger service with stop()

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: create msgrs in main.cc

messengers are sharded<Service>. we should not create them in another
sharded service's start() method. to ensure the ordering of stop of
sharded services, we should create the sharded services in main().
and register their stop() method in the proper order.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27064 from tchaikov/wip-crimson-pg-init

crimson/osd: init PG with more info

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: init primary state in PG::read_state()

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: add PG::read_state()

to load pg_info and past_intervals

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: skip fast_info if it is not around

fast_info is optional. for instance, there is chance that it
is the first time the info is written to store, in that case,
there is no delta, i.e. fast info yet.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: pass pgid, osdmap and msgr to PG's ctor

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27062 from tchaikov/wip-crimson-ceph-opts-in-argv

crimson/osd: pass unknown args to ConfigProxy::parse_args()

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #27003 from tchaikov/wip-crimson-for-classic

osd,messages: changes for preparing for crimson-osd

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #27026 from tchaikov/wip-remove-MPGStats/had_map_for

messages,osd: remove MPGStats::had_map_for

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #27059 from tchaikov/wip-crimson-heartbeat

crimson/{osd,heartbeat}: allow heartbeat to have access to authorizer

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: pass unknown args to ConfigProxy::parse_args()

these args are not recognized by seastar's app_template's option parser,
so they are likely to be the ceph options and should be fed to
ConfigProxy::parse_args()

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: do not pass ceph options to seastar

the program_option parser used by seastar::app_template does not allow
unrecognized options. but ceph options can be specfied by being passed
to ceph applications as command line options, for instance, we can
specify the "key" or "keyfile" when creating an objectstore using
"--mkfs", like:

  ceph-osd --mkfs --key <key>

in this change, all options known by seastar's app_template are
enumerated and stored into a separated vector. so it can be passed to
"app". and the unknown ones are passed to ceph functions.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/common: add ConfigProxy::parse_argv()

it's only used by crimson-osd for parsing ceph options.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27061 from tchaikov/wip-crimson-mkfs

crimson/osd: call engine().exit(0) after mkfs

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

ceph_release: the next release will be octopus

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #26987 into master

* refs/pull/26987/head:
msg/msg_types: use inet_ntop(3) to render IP addresses

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #26672 into master

* refs/pull/26672/head:
osd: correct a local variable type

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

crimson/osd: init OSD::store in its ctor

less repeating this way.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: unmount store in OSD::stop()

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: print message after done with mkfs

so user is aware of what's accomplished.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: call engine().exit(0) after mkfs

* crimson/osd/main.cc:
  if crimson-osd is launched as a daemon, we can always stop it using
  SIGINT, and seastar::reactor does stop itself when handling this signal.

  but if we start crimson-osd as an ordinary command line tool, we should
  stop it explicitly after it is done with its job. so call
  seastar::engine().exit().
* crimson/osd/osd.cc:
  do not stop unitialized services in OSD::stop(). OSD is initialized as
  a sharded service in main.cc, so we have to stop it before stopping
  engine. if OSD is used for mkfs, the internal services like heartbeat,
  monc, mgrc are not initialized at all, so we should not stop them in
  this case. and in theory, user could stop crimson-osd when it boots,
  so we need to check the pointers for null before dereferencing them.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #26652 into master

* refs/pull/26652/head:
osd: clear osd op reply output only when writes success

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #26604 into master

* refs/pull/26604/head:
qa/workunits/rest: Better detection of rest url

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #27010 into master

* refs/pull/27010/head:
mon/OSDMonitor: allow 'osd pool set pgp_num_actual'

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #26812 into master

* refs/pull/26812/head:
msg/async V2: optimize check state by replace.

Reviewed-by: Ricardo Dias <rdias@suse.com>

crimson/osd: correct the osd_ping's message type

s/CEPH_MSG_PING/MSG_OSD_PING/

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: implement Heartbeat::ms_handle_reset()

reconnect to peer if the connection is reset

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: add AuthService

AuthService is introduced to allow Dispatchers to access to
authorizers, even if it is not chained with a Dispatcher implementing
Dispatcher::ms_get_authorizer(). in this case, we need to grant access to
Heartbeat class. it has its own messengers dedicated for heartbeat
traffic. it's mon::Client which provides the facilities of authorization
via Dispatcher interface.

we could just cast mon::Client to ceph::common::Dispatch for accessing
Dispatcher::ms_get_authorizer(), but i want to make this explicit using
AuthService. as the consumers of Dispatch inteface is messenger and
ChainedDispatcher not the domain specific classes.

in future, we need to either implement Auth{Client,Server} or adapt to
this machinery for msgr V2.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/mon: implement Dispatcher::ms_get_authorizer()

we need this method for supporting cephx authentication

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/{net,osd}: make ms_get_authorizer() sync

the authorizer manager does not perform (significant) i/o for building
an authorizer. see CephXTicketHandler::build_authorizer(). what it does
is but read random bytes using getentropy(3) which uses getrandom(2).
getrandom(2) could potentially block if the system just boots and does
not have enough randomness. but i think it's safe to assume that we have
enough entrophy when crimson-osd starts.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27006 from tchaikov/wip-crimson-os

crimson/os: do not fail if fsid file exists when mkfs

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #26996 from neha-ojha/wip-2-36739

osd/PG: skip rollforward when !transaction_applied during append_log()

Reviewed-by: Xie Xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #26056 into master

* refs/pull/26056/head:
mds: check earlier if directories are already exported
mds: dont print auth trees if they are too many
mds: dont print subtrees if they are too many/big

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #26662 into master

* refs/pull/26662/head:
qa: update kclient testing to RHEL 7.6

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #26817 into master

* refs/pull/26817/head:
mds: avoid revoking Fsx from loner during directory fragmentation

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #26895 into master

* refs/pull/26895/head:
mds: convert unnecessary list usage to vector
mds: convert get_*dirfrags to use vector
mds: convert iterator loop to generic loop
mds: list to std::list
mds: constantize dump_loads

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #26931 from trociny/wip-38706

librbd: make flush be queued by QOS throttler

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #27049 from trociny/wip-rbd-deep-copy-cleanup

librbd: fix typo in deep_copy::ObjectCopyRequest::compute_read_ops

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

crimson/os: do not fail if fsid file exists when mkfs

instead of passing a random uuid down to CyanStore::mkfs(), let it
create one if necessary.

Signed-off-by: chunmei Liu <chunmei.liu@intel.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>