common/WorkQueue: remove unused BatchWorkQueue

we are using ShardedWQ for handling peering events. and PeeringWQ
was the only user of BatchWorkQueue. so let's remove the latter.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36148 from athanatos/sjust/wip-lba-pin

crimson: pin LogicalExtentRef lba tree ancestors

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/os/seastore: link in LBAPin after read

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: initialize dirty extents after replay

lba_manager needs to preload BtreeLBAPin and parent nodes.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager/btree: improved debugging

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager/btree: wire in btree pin set

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager/btree: add lba pin structure

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #36458 from drunkard/master

doc: fix typo in output of osdmaptool example

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36145 from orozery/object_dispatch_read_extents_with_version

librbd: add read extents with version to object dispatch interface

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

crimson/os/seastore: actually use LBAManager::complete_transaction

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/: add a meta field to FixedKVNodeLayout and use in LBANode*

Also removes LBANode::depth.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager/btree: move BtreeLBAPin to btree_range_pin.h

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #36371 from votdev/issue_46775_alertmanagerspec

mgr/cephadm: Enhance AlertManagerSpec to allow adding additional webhook receiver URLs

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #36433 from votdev/issue_46814_alertmanager

cephadm: Deploying alertmanager image is broken

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

Merge pull request #36441 from Daniel-Pivonka/cephadm-46819

mgr/orchestrator: fix placement of '1' activating dry-run bug

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #36391 from jschmid1/followup_fix_36151

mgr/cephadm: correct to_json output from str->dict

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #36388 from sebastian-philipp/cephadm-inventory-get_daemons_by_service

mgr/cephadm: get_daemons_by_service should use dd.service_name

Merge pull request #36383 from mgfritch/orch-nfs-rados-url

python-common: don't assert when building rados url

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #36405 from tchaikov/wip-heartbeat-atomic

common/HeartbeatMap: use std::atomic<time_point> for timeout

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #36408 from tchaikov/wip-33725

tools/rados: Set locator key when exporting or importing a pool

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #36443 from tchaikov/wip-crimson-asok

crimson/admin: read non-string settings as well

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/admin: do not validate params of ConfigSetHook

validate_cmd() should have taken care of this

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/admin: read non-string settings as well

before this change, we were using

local_conf().get_val<std::string>(var);

for reading a setting when serving a "config get" command even if the
setting being queried is a non-string. so without this change, a failure
is returned complaining "unrecognized option..."

in this change:

* use get_val(string,string*) for querying the string representation
  of the queried setting
* drop the check for existence of "var" parameter, validate_cmd()
  always take care of this.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/admin: drop prefix from cmd desc

there is no need to repeat the prefix in the command's description,
validate_cmd() skips it anyway.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/admin: move config hooks into admin_socket.cc

they are not specific to OSD, let's move them into admin_socket.cc

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36438 from rhcs-dashboard/wip-pull_request_template_api-master

github: replace trigger phrase for api tests

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-By: Patrick Donelly <pdonell@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

doc: typo fix on output of osdmaptool example

Signed-off-by: Drunkard Zhang <gongfan193@gmail.com>

tools/rados: Set locator key when exporting or importing a pool

Fixes the following error when exporting a pool that contains objects
with a locator key set:

        error getting xattr set [object name]: (2) No such file or directory
        error from export: (2) No such file or directory

Fixes: https://tracker.ceph.com/issues/46824
Signed-off-by: Iain Buclaw <iain.buclaw@dunnhumby.com>

common/HeartbeatMap: use relaxed order

use memory_order_relaxed instead of memory_order_seq_cst, as the
ordering of storing of `timeout` and `suicide_timeout` does not matter,
and we don't need to synchorinize the cache for the consistent memory
view of local core. what we need is but the atomic read/write operation
of these individual variables.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/WorkQueue,librbd,rgw: use ceph::timespan for representing interval

better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36442 from rzarzynski/wip-crimson-op_truncate

crimson/osd: add support for CEPH_OSD_OP_TRUNCATE.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36398 from jdurgin/wip-pg-log-doc

osd/PGLog: improve documentation

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

common/options: update pg log bound descriptions to reflect current code

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

common/options: make pg log options dev-level

Changing these has performance and correctness implications, and is
rarely needed.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

common/options: describe osd_pg_log_trim_min

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

mgr/orchestrator: fix placement of '1' activating dry-run bug

when applying a placement of '1' for mon, mgr, rbd-mirror, crash, alertmanager, grafana, node-exporter, prometheus, or mds a dry-run was activated

Fixes: https://tracker.ceph.com/issues/46819
Signed-off-by: Daniel-Pivonka <dpivonka@redhat.com>

crimson/osd: add support for CEPH_OSD_OP_TRUNCATE.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

github: replace trigger phrase for api tests

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #36429 from tchaikov/wip-ceph-debug-docker-crimson

ceph-debug-docker: install ceph-crimson-osd for crimson flavor

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #35980 from tspmelo/wip-table-status

mgr/dashboard: Display table status in refresh button

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

cephadm: Deploying alertmanager image is broken

Fixes: https://tracker.ceph.com/issues/46814
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #36401 from tchaikov/wip-pg-stat-ack

messages/MPGStats: print seq instead of tid

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #36375 from ricardoasmarques/force-registry-auth-config-set

cephadm: Fix error bootstraping with '--registry-json'

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #36386 from bk201/wip-46045

mgr/cephadm: skip Dashboard configuration if Dashboard is disabled

Reviewed-by: Varsha Rao <varao@redhat.com>

Merge pull request #36400 from tchaikov/wip-rip-xio-msgr

cmake: do not check OFED_PREFIX anymore

Reviewed-by: Casey Bodley <cbodley@redhat.com>

ceph-debug-docker: install ceph-crimson-osd for crimson flavor

would be handy if we can have crimson packages when debugging crimson
flavored branch.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35060 from yison/rwl-image-writeback-cache-hook-seq1

rbd/cache: Replicated Write Log Hook part 11

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #35659 from agayev/track-per-zone-metadata

os/bluestore: track per zone metadata

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #30838 from ifed01/wip-ifed-single-alloc

os/bluestore: use single allocator for shared bluestore/bluefs device

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #36356 from mgfritch/orch-format-not-supported

mgr/orch: disallow unimplemented format types

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

librbd: enable image cache after getting exclusive lock

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

librbd: add InitRequest and ShutdownRequest state machine

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

librbd: split ReadRequest

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

librbd: add IMAGE_CACHE feature

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

Merge pull request #36417 from tchaikov/wip-crimson-injectargs

crimson/admin: implement "injectargs" command

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

mgr/cephadm: Enhance AlertManagerSpec to allow adding additional webhook receiver URLs

Fixes: https://tracker.ceph.com/issues/46775
Signed-off-by: Volker Theile <vtheile@suse.com>

crimson/osd: ignore SIGHUP

ignore SIGHUP, as crimson does not write log messages using the
facilities offered by ceph::logging. it uses seastar::logger instead.

and seastar::logger writes to stderr, stdout and/or syslog. there is no
need to reopen the log file when the settings change.

and we do send SIGHUP to OSD processes when performing thrash tests. to
avoid killing crimson-osd processes by accident. SIGHUP is ignored.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36407 from tchaikov/wip-mgr-autopep8

mgr/cephadm: track dependency in tox.ini

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

common/HeartbeatMap: use std::atomic<time_point> for timeout

since we now depends on GCC-7 and up for C++17 support, we can now use
std::atomic<time_point> for better readability.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36378 from orozery/librados_test_stub_assert_version

test/librados_test_stub: handle assert_version

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

crimson/admin: implement "injectargs" command

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/common: add ConfigProxy::inject_args()

prepare for the "injectargs" tell command.

Signed-off-by: Kefu Chai <kchai@redhat.com>

librbd: support reading multiple extents in the object dispatch interface

This commit extends the object dispatch read function to support multiple object extents (using a single librados ObjectOperation).
In addition, we add an option to return the version number for that read.
This new function will be used by the new crypto object dispatch layer to support read-modify-write operation required for encrypting and writing unaligned data.

Signed-off-by: Or Ozeri <oro@il.ibm.com>

Merge pull request #36406 from tchaikov/wip-pg-stat-seq

crimson/osd: send pg_stat with correct seq

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mgr/cephadm: track dependency in tox.ini

* track dependency for running tox tests with tox.ini. the fix
  test is not performed by "make check", so `install-deps.sh`
  is not responsible for preparing the autopep8 wheels. hence
  we can just put it in tox.ini
* remove "python_version >= '3'" as we do not support python2

this change addresses a regression introduced by
7d8a3b60321da10af6f4557a9d69eda09ca2c620, which fails the rpmbuild like:

RPM build errors:
    Installed (but unpackaged) file(s) found:
   /usr/share/ceph/mgr/requirements-fix.txt

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36348 from votdev/cephadm_tox_pylint

mgr/cephadm: Introduce tox and autopep8

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

crimson/osd: send pg_stat with correct seq

* set the osd_stat in pg_stat message sent to mgr, otherwise the seq id
  in the message would be always 0.
* return the seq id composed of up_epoch and osd_stat_seq when serving
  "flush_pg_stats" tell command.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: remove unused #include

these messages are handled by osd/PeeringState

Signed-off-by: Kefu Chai <kchai@redhat.com>

messages/MPGStats: print seq instead of tid

osd_stat.seq is a monotonic increasing sequence number which marks
the version of the of the report. since we don't use the tid to track
a MPGStats message and its MPGStatsAck message, it's more interesting to
us to print the sequence id. we use it to verify if a pg_stats is
flushed or not from a certain OSD.

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/OSD: do not include MPGStatsAck.h

pg_stat report and ack machinery was removed in
528736bed3a6adffa37d0bad6a0e1bedb55db931, and we don't generate or handle
MPGStatsAck anymore since mimic. so remove it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: proper BlueFS shared allocator setup

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #36395 from dillaman/wip-46679

test/librbd: unquiesce notifications are handled lazily

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #36309 from dillaman/wip-46694

librbd:  global and pool-level config overrides require image refresh to apply

Reviewed-by: Mykola Golub <mgolub@suse.com>

cmake: use add_compile_options for passing compile options

add_definitions() is for, well, adding definitions.

Signed-off-by: Kefu Chai <kchai@redhat.com>

kv: always enable rocksdb

RocksDB is included as a submodule of Ceph, there is no need to check
for its existence anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not check OFED_PREFIX anymore

OFED_PREFIX was added to compile with xio messenger for supporting RDMA
using the Accelio and/or MLNX_OFED package. but xio messenger was removed in
in cc9a9142fd41ee5c98792c6f6d2e5504

let remove the leftover in CMakeLists

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32410 from zdover23/wip-doc-pg-repair

doc/rados/operations: document pg repairing practices

osd/PGLog: add some documentation

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #36364 into master

* refs/pull/36364/head:
client: clean up the unnecessary client_lock for _conf->client_trace

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #36359 into master

* refs/pull/36359/head:
mds: place MDSGatherBuilder on the stack

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #36358 into master

* refs/pull/36358/head:
mds: resolve SIGSEGV in waiting for uncommitted fragments

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #36257 into master

* refs/pull/36257/head:
client: move client_lock to _unmount()
client: add timer_lock support

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #36155 into master

* refs/pull/36155/head:
qa: Fix traceback during fs cleanup between tests

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #35944 into master

* refs/pull/35944/head:
qa: defer cleaning the mountpoint's netnses and the bridge
qa/tasks/cephfs/mount.py: remove the stale netnses and bridge
qa/tasks/cephfs/mount.py: try to flush the stale ceph-brx dev info
qa/tasks/cephfs/mount.py: switch to run_shell_payload() helper
qa/tasks/cephfs/mount.py: clean up the none used code

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #35647 into master

* refs/pull/35647/head:
mgr/volumes: Add documentation regarding --retain-snapshots option
mgr/volumes: Avoid trashing retained subvolume on create errors
mgr/volumes: Add subvolume v2 test cases
mgr/volumes: Derive v2 from v1 to leverage common methods
mgr/volumes: Introduce v2 subvolumes
mgr/volumes: Use operation type during subvolume open

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>

Merge PR #35096 into master

* refs/pull/35096/head:
cephfs-journal-tool: fix wrong read_offset when get missing objects

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #32120 into master

* refs/pull/32120/head:
client: periodically forward metrics to session MDSs
client: track cap hits and misses globally
client: minor cleanup to MetaSession class

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

test/librbd: unquiesce notifications are handled lazily

The quiesce unit tests were expecting that the unquiesce notification
would be processed by the time the snapshot create command completed.
Since the RPC is immediately ACKed, there is a potential for a race
if the test doesn't wait for the unquiesce to process.

Fixes: https://tracker.ceph.com/issues/46679
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #32780 from petrutlucian94/windows.5

Windows support [part 5]

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: be more verbose in _open_super_meta by default.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: don't do init_rm_free when mounting bluefs after mkfs.

This is a duplicate op which might cause allocator (namely stupid one) to fail.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: simlify BlueFS::_stop_alloc

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: fix bluefs migrate/expand to match single allocator.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #36065 from trociny/wip-46445

monclient: schedule first tick using mon_client_hunt_interval

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/cephadm: Introduce tox and autopep8

This PR introduces tox and autopep8 to format the Python code according to PEP8. It does NOT include any lint related things.

Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #36002 from david-z/wip-fix-slow-req

mgr: don't update osd stat which is already out

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36101 from ifed01/wip-ifed-fix-store-test-with-db

test/objectstore_store_test: fix test cases broken due to additional …

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>

Merge pull request #36337 from tchaikov/wip-qa-ceph-waiting

qa/tasks/ceph: do not print out empty list of pg

Reviewed-by: Neha Ojha <nojha@redhat.com>

mgr/cephadm: correct to_json output from str->dict

Signed-off-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #36360 from matthewoliver/cephadm_add_ipv6_unwrap

cephadm: add unwrap_ipv6 helper method

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>