tools: make cache watermark configurable

Signed-off-by: Dehao Shang <dehao.shang@intel.com>

tools: make dedicated thread configurable for RO

At cache client side of RO, make dedicated woker thread configurable
Also, cleanup useless member of CacheClient class

Signed-off-by: Dehao Shang <dehao.shang@intel.com>

tools: adjust code style of RO

Signed-off-by: Dehao Shang <dehao.shang@intel.com>

tools: clean up data structure of immutable obj cache daemon

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

tools: refactor ObjectCacheRequest of RO

Signed-off-by: Dehao Shang <dehao.shang@intel.com>

tools: return real cache status on lookup in immutable obj cache

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

tools: fix librados compatibility

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

tools: clean up cache dir for immutable obj cache

- user defined full path for cache dir
- hash objects into subfolders with crc32 of its name

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

tools: maintain cache size counter for immutable obj cache daemon

Do not assume 4M object size since we allow to cache any-sized objects.

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

tools: templatize create_rados_callback

Also cleanup unused template codes

Signed-off-by: Dehao Shang <dehao.shang@intel.com>

tools: use specific message for different ops in immutable obj cache daemon

use different types of message for different ops to be more efficient

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

tools: cleanup IPC message for immutable obj cache daemon

using ceph serialization(encode/decode) framework instead of building
new data structure

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

tools: set namespace & snap_id before promote from RADOS in immutable obj cache

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

tools: format code and clean up

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

tools: adding signal handling for immutable obj cache daemon

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

tests: reduce testing resources occupied for immutable obj cache

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

tools: follow current codes to modify UT

Signed-off-by: Dehao Shang <dehao.shang@intel.com>

tools: release dedicated thread resource

Addtional, enable all UT

Signed-off-by: Dehao Shang <dehao.shang@intel.com>

tools: add connect fails to fault

Signed-off-by: Dehao Shang <dehao.shang@intel.com>

tools: clean up object cache request struture

adding version and compat_version info for serailization

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

common/buffer: assign error message to caller on short read in pread_file

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

tools: add m_cache_path

1) : enable ObjectCacheRequest.m_data.m_cache_path
2) : add generate_cache_file_path to create cache_path
    , which will be used by CacheClient.

Signed-off-by: Dehao Shang <dehao.shang@intel.com>

tools: clean up dead code in immutable obj cache

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

tools: kill ObjectCacheFile in immutable obj cache

instead we could use bufferlist::write_file/pread_file to write/read on cache

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

tools: clean up immutable obj cache

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

common/options: move immutable obj cache options out of rbd section

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

common/buffer: return 0 on short read in pread_file

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

tools: some modification at RO

1) : add new data member to ObjectCacheRequest,
     including m_pool_namepace, m_pool_id, m_snap_id,
     m_cache_path
2) : add some log at CacheSession
3) : change pool name to pool_nspace + poo_id + snap_id
4) : change cache file name
5) : add generate_cache_file_name method
6) : modify UT interface

Signed-off-by: Dehao Shang <dehao.shang@intel.com>

tool: break down receive_message implements

Also, enable fault at CacheClient side.

Signed-off-by: Dehao Shang <dehao.shang@intel.com>

common/buffer: fix pread_file to generate local bufferlist

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

tools: clean up types for immutable obj cache daemon

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

tools: refuse to start if obj cache daemon fail to initialize cache dir

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

test: enable asio tests for immutable obj cache controller

cleanups for immutable obj cache controller

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

tools: immutable_object_cache code format cleanup

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

common/buffer: adding pread_file method

pread_file allows to read length of contents at a given offset

using pread_file instead of read_file to reduce extra reads in ObjectCacheFile

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

tools: refactor asio domain socket of RO

Signed-off-by: Dehao Shang <dehao.shang@intel.com>
Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

tools: fix redudent updates on cache metdata

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

build/ops: adding build spec for immutable object cache daemon

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

test: add object cache message unite test

Signed-off-by: Dehao Shang <dehao.shang@intel.com>

tools: add new object cache message

Signed-off-by: Dehao Shang <dehao.shang@intel.com>

tools: use bufferlist to implement ObjectCacheFile

1) : modify unite testing of object cache file
2) : remove open and create internface
3) : use bufferlist interface to read/write file

Signed-off-by: Dehao Shang <dehao.shang@intel.com>

test: add ObjectCacheStore unit test

Signed-off-by: Dehao Shang <dehao.shang@intel.com>

test: add multi-session unit testing

Signed-off-by: Dehao Shang <dehao.shang@intel.com>

test: move WaitEvent class to test_common.h file

Signed-off-by: Dehao Shang <dehao.shang@intel.com>

tools: fix immutable obj cache max size

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

tools: remove RBD specific logic in immutable obj cache

Make the daemon more generic(pool name + object id) so it
can service both RBD and RGW.
Also supports caching partial object(even empty), the caller
needs to append_zero to read buffer if read on those objects.

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

tools: remove background eviction

do eviction check in promotion callback simply

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

tools: format headers for object cache daemon

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

tools: create local utils for object cache daemon

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

tools: clean up immutable object cache

- drop "ceph_" prefix
- prefix all options with immutable_object_cache_
- adding configurable immutable_object_cache_max_inflight_ops

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

tools: adding ceph level immutable obj cache daemon

The daemon is built for future integration with both RBD and RGW cache.
The key components are:
- domain socket based simple IPC
- simple LRU policy based promotion/demotion for the cache
- simple file based caching store for RADOS objs with sync IO interface
- systemd service/target files for the daemon

Signed-off-by: Dehao Shang <dehao.shang@intel.com>
Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

Merge pull request #27089 from tchaikov/wip-cmake-with-ninja

cmake: do not assume ${CMAKE_GENERATOR} == make

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27100 from ddiss/trivial_rgw_auth_keystone_gcc_warning

[trivial] rgw: fix minor compiler warning in keystone auth

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27094 from tchaikov/wip-cmake-cmake-ubsan

cmake: workaround of false alarm from ubsan

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27088 from tchaikov/wip-update-seastar

cmake,crimson: pick up latest seastar

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #26731 from wjwithagen/wjw-fix-cmake-os-release

build: FreeBSD does not have /etc/os-release

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #26607 from tchaikov/wip-cython-language_level

pybind: set language_level for cythonize explicitly

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

rgw: fix minor compiler warning in keystone auth

Signed-off-by: David Disseldorp <ddiss@suse.de>

Merge pull request #27093 from tchaikov/wip-crimson-test-alien-echo

test/crimson: update to accomodate Dispatcher changes

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #27092 into master

* refs/pull/27092/head:
mailmap: sort entries
mailmap: sort organizationmap
mailmap: updates for nautilus
doc: mailmap updates for mimic
mailmap fixes for 12.2.0

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #26404 from sebastian-philipp/orchestrator-error-handling

mgr/orchestrator: Add error handling to interface

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Tim Serong <tserong@suse.com>

cmake: workaround of false alarm from ubsan

to workaround a GCC bug, see
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=88684

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/crimson: update to accomodate Dispatcher changes

Dispatcher::ms_get_authorizer() was changed in f22d433a, so update the
test accordingly

Signed-off-by: Kefu Chai <kchai@redhat.com>

mailmap: sort entries

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

mailmap: sort organizationmap

since this will make it easier for editing entries in the future

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

mailmap: updates for nautilus

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

cmake: disable -Werror when building seastar

when building seastar with a recent fmt, fmt complains:

src/fmt/include/fmt/time.h:13:2: error: #warning fmt/time.h is
deprecated, use fmt/chrono.h instead [-Werror=cpp]
 #warning fmt/time.h is deprecated, use fmt/chrono.h instead
  ^~~~~~~

Signed-off-by: Kefu Chai <kchai@redhat.com>

seastar: pickup the latest seastar

for couple improvements

* dbf7b6b0: add fast path for small writes
* 2f357317: IPv6 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: link against libfmt

Seastar::seastar does pull in libfmt library as a PUBLIC linkage. but we
should not depend on this. if we use fmt directly, we should link against it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: include headers if necessary

we should not rely on seastar or other headers to do this. if we use
fmt or std functions, we should include corresponding header(s).

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not assume ${CMAKE_GENERATOR} == make

* modules/BuildDPDK.cmake: always use "make"
* modules/BuildSPDK.cmake: always use "make"
  DPDK only support "make". if user wants to use other generator than
  "make", we will fail to build DPDK support. so we should always use
  "make" here. and before using "make", we need to detect it.
* cmake/modules/BuildRocksDB.cmake: use cmake for build
* src/compressor/zstd/CMakeLists.txt: use cmake for build
  just use whatever generator use specifies, and use ${CMAKE_COMMAND}
  for building the specified target, no need to repeat
  ${CMAKE_GENERATOR} for building the target.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: build fmt submodule if libfmt is not found

the recent seastar dropped fmt submodule, so we need to build from our
own fmt submodule.

Signed-off-by: Kefu Chai <kchai@redhat.com>

fmt: add fmt submodule back

seastar does not bring fmt as a submodule, so let's ready it by
ourselves.

This reverts commit 53a38bf57e10b681268378737955e50f6239e925.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27057 from yuriw/wip-yuriw-crontab

qa/tests: reduced frequency for luminous and mimic runs...

qa/tests: reduced frequency for luminous and mimic runs
added all suites for nauitlus

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #26786 from aclamk/fix-rocksdb-compaction-ranges

common/kv/rocksdb: Fixed async compations.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27046 from smithfarm/wip-38812

rpm: refrain from building ceph-resource-agents on SLE

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Tim Serong <tserong@suse.com>

Merge pull request #27054 from cbodley/wip-37770

doc/rgw: update civetweb rgw_frontends config example

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27035 from cbodley/wip-test-rgw-reshard-wait-clock

test/rgw: test_rgw_reshard_wait uses same clock for timing

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26853 from nathan-weinberg/more-cluster-tests

mgr/dashboard: Added breadcrumb tests to Manager modules and Alerts menu

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #27052 from rzarzynski/wip-rgw-drop_rgw_decode_pki_token

rgw: drop unused rgw_decode_pki_token().

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27025 from yangdongsheng/poll_timeout

rbd: krbd: return -ETIMEDOUT in polling

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

doc/rgw: update civetweb rgw_frontends config example

all of these civetweb options have to be on the same line as rgw_frontends

Fixes: https://tracker.ceph.com/issues/37770
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27050 from theanalyst/build-doc-warns

admin/build-doc: keep-going when finding warnings

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26534 from majianpeng/skip-calc-crc-header-for-async-msg

msg/async: skip repeat calc crc header in Message::encode

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #27009 into master

* refs/pull/27009/head:
ceph_release: the next release will be octopus

Reviewed-by: Kai Wagner <kwagner@suse.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #26514 from dillaman/wip-38381

pybind/rados: fixed Python3 string conversion issue on get_fsid

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26985 from rhcs-dashboard/38768-fix-sparkline-component

mgr/dashboard: fix sparkline component

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

Merge pull request #27034 from jan--f/mgr-prometheus-remove-scrape-duration

pybind/mgr/prometheus: remove scrape_duration metric

Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #26701 from adamemerson/wip-trip-the-light-clangtastic

fix clang build failures, and clean up warnings

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26836 from p-na/wip-pna-fix-osd

 mgr/dashboard: "1 osds exist in the crush map but not in the osdmap" breaks OSD page

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #26717 from ifed01/wip-ifed-objectstore-tool-fix

tools: no-mon-config switch for ceph-objectstore-tool.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26926 from rouming/epoll-lost-epollet

msg/async/EventEpoll: set EPOLLET flag on del_event()

Reviewed-by: Kefu Chai <kchai@redhat.com>

rbd: krbd: return -ETIMEDOUT in polling

We don't want to wait on uevent forever, but the return value
of polling in timeout is 0 rather than a negative value.

Fixes: http://tracker.ceph.com/issues/38792
Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

Merge pull request #26934 from sebastian-philipp/doc-rados-mon_command

doc/rados/api/python: Add documentation for mon_command

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26562 from familyuu/log

mgr/dashboard: Add date range and log search functionality

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #27065 from tchaikov/wip-crimson-mgr-client

crimson/osd: report pg_stats to mgr

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: connect OSD to mgr

so OSD is able to report to mgr periodically using mgr::Client

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: include MonClient.h in .cc

mon::Client's definition is only necessary in .cc

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: add facilities for reporting pg stats

Signed-off-by: Kefu Chai <kchai@redhat.com>