crimson/osd: support cls_cxx_write_zero()

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: support CEPH_OSD_OP_ZERO op

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/admin: drop stale comment

the UNIX domain socket file was removed in
71c08cb02eb6c0f8220c0cf4b82332398b25de2c

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36478 from zdover23/wip-doc-removing-auth-disable-warning

doc/rados: remove upgrade auth disable

This commit removes a recommendation to disable
authentication before performing an upgrade.

Reviewed-by: Joao Eduardo Luis
Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #35947 from orozery/crypto_object_dispatch_layer

librbd: add a crypto object dispatch layer

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #36474 from linuxbox2/wip-rgwlc-structv

rgwlc:  fix conditional decode of legacy lc op structures

doc/rados: remove upgrade auth disable

This commit removes a recommendation to disable
authentication before performing an upgrade.

Fixes: https://tracker.ceph.com/issues/30630
Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #36444 from sebastian-philipp/cephadm-upgrade-upstradestate-class

mgr/cephadm: Add UpgradeState class

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #36463 from SUSE/wip-fix-46833

cephadm: don't add `ceph-volume lvm activate` for adopted simple OSDs

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

rgwlc:  fix conditional decode of legacy lc op structures

Fixes the case where an upgraded radosgw operates on an unupgraded
OSD, so likely these cases do not execute.

Fixes: https://tracker.ceph.com/issues/46838
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #36349 from sebastian-philipp/cephadm-check_deamons_except

mgr/cephadm: guard _check_daemons

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #36246 from linuxbox2/wip-rgwlc-decode

rgwlc: fix error return from backward-compat decode

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #36345 from tspmelo/wip-action-disable

mgr/dashboard: Merge disable and disableDesc

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

mgr/caphadm/tests: Add `test_daemon_check_post`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #36396 from tspmelo/wip-doc-component

mgr/dashboard: Extract documentation link to a component

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #36372 from jan--f/c-v-fix-package-common-dep

ceph-volume: dependency on python-ceph-common

mgr/cephadm: Don't run `daemons_post` if _create_daemon fail

We can't run daemon_check_post for the type, if a single daemon failed.
Mainly cause `daemon_check_post` is run by service type.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: Add test_daemon_action_fail

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm/tests: Split `test_daemon_action`

Into:

* `test_list_daemons`
* `test_daemon_action` (now without listing daemons)

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: guard _check_daemons

Continue with other daemons, if one fails

Fixes: https://tracker.ceph.com/issues/46748
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #36446 from sebastian-philipp/mgr-cephadm-pep8-remotes.py

mgr/cephadm: PEP8tify a few files

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

librbd: add a crypto object dispatch layer

This commit adds a new object dispatch layer to enable encryption of RBD images.

Signed-off-by: Or Ozeri <oro@il.ibm.com>

Merge pull request #36148 from athanatos/sjust/wip-lba-pin

crimson: pin LogicalExtentRef lba tree ancestors

Reviewed-by: Kefu Chai <kchai@redhat.com>

cephadm: don't add `ceph-volume lvm activate` for adopted simple OSDs

This changes the logic in deploy_daemon_units() to add either `chown` calls
for simple (ceph-disk style) OSDs, or to add `ceph-volume lvm activate` calls
for LVM OSDs, rather than always adding both.  When I was working on
https://github.com/ceph/ceph/pull/34703, I'd originally added an "osd_simple"
flag to figure out what type of OSD was being adopted/deployed, but passing
that around was kinda ugly, so was removed from that PR.  This time around
I'm checking if /etc/ceph/osd/$OSD_ID-$OSD_FSID.json.adopted-by-cephadm
exists, which seems pretty safe IMO.  My only concern with this method is:
what happens if someone adopts a simple OSD, then later wants to migrate it
to LVM.  Presumably that's a destroy and recreate, keeping the same OSD ID?
If that's true, then the JSON file probably still exists, so the subsequent
create will do the wrong thing, i.e. will add `chown` calls, not `ceph-volume
lvm activate` calls.  Any/all feedback appreciated...

Fixes: https://tracker.ceph.com/issues/46833
Signed-off-by: Tim Serong <tserong@suse.com>

crimson/os/seastore: link in LBAPin after read

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: initialize dirty extents after replay

lba_manager needs to preload BtreeLBAPin and parent nodes.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager/btree: improved debugging

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager/btree: wire in btree pin set

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager/btree: add lba pin structure

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #36458 from drunkard/master

doc: fix typo in output of osdmaptool example

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36145 from orozery/object_dispatch_read_extents_with_version

librbd: add read extents with version to object dispatch interface

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

crimson/os/seastore: actually use LBAManager::complete_transaction

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/: add a meta field to FixedKVNodeLayout and use in LBANode*

Also removes LBANode::depth.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager/btree: move BtreeLBAPin to btree_range_pin.h

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #36371 from votdev/issue_46775_alertmanagerspec

mgr/cephadm: Enhance AlertManagerSpec to allow adding additional webhook receiver URLs

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #36433 from votdev/issue_46814_alertmanager

cephadm: Deploying alertmanager image is broken

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

Merge pull request #36441 from Daniel-Pivonka/cephadm-46819

mgr/orchestrator: fix placement of '1' activating dry-run bug

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #36391 from jschmid1/followup_fix_36151

mgr/cephadm: correct to_json output from str->dict

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #36388 from sebastian-philipp/cephadm-inventory-get_daemons_by_service

mgr/cephadm: get_daemons_by_service should use dd.service_name

Merge pull request #36383 from mgfritch/orch-nfs-rados-url

python-common: don't assert when building rados url

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #36405 from tchaikov/wip-heartbeat-atomic

common/HeartbeatMap: use std::atomic<time_point> for timeout

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: Extract documentation link to a component

Fixes: https://tracker.ceph.com/issues/36565
Fixes: https://tracker.ceph.com/issues/43375
Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/cephadm: PEP8tify

Those files should generate fewer conflicts for now.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: PEP8tify  upgrade.py

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: Add UpgradeState class

My resons are:

* This increase mypy coverage
* Makes the code more discoverable.
* Allowes us to have intelligent UpgradeState properties

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #36408 from tchaikov/wip-33725

tools/rados: Set locator key when exporting or importing a pool

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #36443 from tchaikov/wip-crimson-asok

crimson/admin: read non-string settings as well

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/admin: do not validate params of ConfigSetHook

validate_cmd() should have taken care of this

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/admin: read non-string settings as well

before this change, we were using

local_conf().get_val<std::string>(var);

for reading a setting when serving a "config get" command even if the
setting being queried is a non-string. so without this change, a failure
is returned complaining "unrecognized option..."

in this change:

* use get_val(string,string*) for querying the string representation
  of the queried setting
* drop the check for existence of "var" parameter, validate_cmd()
  always take care of this.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/admin: drop prefix from cmd desc

there is no need to repeat the prefix in the command's description,
validate_cmd() skips it anyway.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/admin: move config hooks into admin_socket.cc

they are not specific to OSD, let's move them into admin_socket.cc

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36438 from rhcs-dashboard/wip-pull_request_template_api-master

github: replace trigger phrase for api tests

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-By: Patrick Donelly <pdonell@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

doc: typo fix on output of osdmaptool example

Signed-off-by: Drunkard Zhang <gongfan193@gmail.com>

tools/rados: Set locator key when exporting or importing a pool

Fixes the following error when exporting a pool that contains objects
with a locator key set:

        error getting xattr set [object name]: (2) No such file or directory
        error from export: (2) No such file or directory

Fixes: https://tracker.ceph.com/issues/46824
Signed-off-by: Iain Buclaw <iain.buclaw@dunnhumby.com>

common/HeartbeatMap: use relaxed order

use memory_order_relaxed instead of memory_order_seq_cst, as the
ordering of storing of `timeout` and `suicide_timeout` does not matter,
and we don't need to synchorinize the cache for the consistent memory
view of local core. what we need is but the atomic read/write operation
of these individual variables.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/WorkQueue,librbd,rgw: use ceph::timespan for representing interval

better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36442 from rzarzynski/wip-crimson-op_truncate

crimson/osd: add support for CEPH_OSD_OP_TRUNCATE.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36398 from jdurgin/wip-pg-log-doc

osd/PGLog: improve documentation

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

common/options: update pg log bound descriptions to reflect current code

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

common/options: make pg log options dev-level

Changing these has performance and correctness implications, and is
rarely needed.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

common/options: describe osd_pg_log_trim_min

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

mgr/orchestrator: fix placement of '1' activating dry-run bug

when applying a placement of '1' for mon, mgr, rbd-mirror, crash, alertmanager, grafana, node-exporter, prometheus, or mds a dry-run was activated

Fixes: https://tracker.ceph.com/issues/46819
Signed-off-by: Daniel-Pivonka <dpivonka@redhat.com>

crimson/osd: add support for CEPH_OSD_OP_TRUNCATE.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

github: replace trigger phrase for api tests

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #36429 from tchaikov/wip-ceph-debug-docker-crimson

ceph-debug-docker: install ceph-crimson-osd for crimson flavor

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #35980 from tspmelo/wip-table-status

mgr/dashboard: Display table status in refresh button

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

ceph-volume: dependency on python-ceph-common

Since e5b585d15de8b07e0a179344d4187582a5c069f2 ceph-volume depends on
python-ceph-common. This commit introduces this dependency for the
ceph-osd rpm (which includes ceph-volume) and installs the dependency
for tox runs.

Fixes: https://tracker.ceph.com/issues/46772
Fixes: e5b585d15de8b07e0a179344d4187582a5c069f2
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

cephadm: Deploying alertmanager image is broken

Fixes: https://tracker.ceph.com/issues/46814
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #36401 from tchaikov/wip-pg-stat-ack

messages/MPGStats: print seq instead of tid

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #36375 from ricardoasmarques/force-registry-auth-config-set

cephadm: Fix error bootstraping with '--registry-json'

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #36386 from bk201/wip-46045

mgr/cephadm: skip Dashboard configuration if Dashboard is disabled

Reviewed-by: Varsha Rao <varao@redhat.com>

Merge pull request #36400 from tchaikov/wip-rip-xio-msgr

cmake: do not check OFED_PREFIX anymore

Reviewed-by: Casey Bodley <cbodley@redhat.com>

ceph-debug-docker: install ceph-crimson-osd for crimson flavor

would be handy if we can have crimson packages when debugging crimson
flavored branch.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35060 from yison/rwl-image-writeback-cache-hook-seq1

rbd/cache: Replicated Write Log Hook part 11

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #35659 from agayev/track-per-zone-metadata

os/bluestore: track per zone metadata

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #30838 from ifed01/wip-ifed-single-alloc

os/bluestore: use single allocator for shared bluestore/bluefs device

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #36356 from mgfritch/orch-format-not-supported

mgr/orch: disallow unimplemented format types

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

librbd: enable image cache after getting exclusive lock

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

librbd: add InitRequest and ShutdownRequest state machine

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

librbd: split ReadRequest

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

librbd: add IMAGE_CACHE feature

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

Merge pull request #36417 from tchaikov/wip-crimson-injectargs

crimson/admin: implement "injectargs" command

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

mgr/cephadm: Enhance AlertManagerSpec to allow adding additional webhook receiver URLs

Fixes: https://tracker.ceph.com/issues/46775
Signed-off-by: Volker Theile <vtheile@suse.com>

crimson/osd: ignore SIGHUP

ignore SIGHUP, as crimson does not write log messages using the
facilities offered by ceph::logging. it uses seastar::logger instead.

and seastar::logger writes to stderr, stdout and/or syslog. there is no
need to reopen the log file when the settings change.

and we do send SIGHUP to OSD processes when performing thrash tests. to
avoid killing crimson-osd processes by accident. SIGHUP is ignored.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36407 from tchaikov/wip-mgr-autopep8

mgr/cephadm: track dependency in tox.ini

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

common/HeartbeatMap: use std::atomic<time_point> for timeout

since we now depends on GCC-7 and up for C++17 support, we can now use
std::atomic<time_point> for better readability.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36378 from orozery/librados_test_stub_assert_version

test/librados_test_stub: handle assert_version

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

crimson/admin: implement "injectargs" command

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/common: add ConfigProxy::inject_args()

prepare for the "injectargs" tell command.

Signed-off-by: Kefu Chai <kchai@redhat.com>

librbd: support reading multiple extents in the object dispatch interface

This commit extends the object dispatch read function to support multiple object extents (using a single librados ObjectOperation).
In addition, we add an option to return the version number for that read.
This new function will be used by the new crypto object dispatch layer to support read-modify-write operation required for encrypting and writing unaligned data.

Signed-off-by: Or Ozeri <oro@il.ibm.com>

Merge pull request #36406 from tchaikov/wip-pg-stat-seq

crimson/osd: send pg_stat with correct seq

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mgr/cephadm: track dependency in tox.ini

* track dependency for running tox tests with tox.ini. the fix
  test is not performed by "make check", so `install-deps.sh`
  is not responsible for preparing the autopep8 wheels. hence
  we can just put it in tox.ini
* remove "python_version >= '3'" as we do not support python2

this change addresses a regression introduced by
7d8a3b60321da10af6f4557a9d69eda09ca2c620, which fails the rpmbuild like:

RPM build errors:
    Installed (but unpackaged) file(s) found:
   /usr/share/ceph/mgr/requirements-fix.txt

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36348 from votdev/cephadm_tox_pylint

mgr/cephadm: Introduce tox and autopep8

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

crimson/osd: send pg_stat with correct seq

* set the osd_stat in pg_stat message sent to mgr, otherwise the seq id
  in the message would be always 0.
* return the seq id composed of up_epoch and osd_stat_seq when serving
  "flush_pg_stats" tell command.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: remove unused #include

these messages are handled by osd/PeeringState

Signed-off-by: Kefu Chai <kchai@redhat.com>

messages/MPGStats: print seq instead of tid

osd_stat.seq is a monotonic increasing sequence number which marks
the version of the of the report. since we don't use the tid to track
a MPGStats message and its MPGStatsAck message, it's more interesting to
us to print the sequence id. we use it to verify if a pg_stats is
flushed or not from a certain OSD.

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/OSD: do not include MPGStatsAck.h

pg_stat report and ack machinery was removed in
528736bed3a6adffa37d0bad6a0e1bedb55db931, and we don't generate or handle
MPGStatsAck anymore since mimic. so remove it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: proper BlueFS shared allocator setup

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #36395 from dillaman/wip-46679

test/librbd: unquiesce notifications are handled lazily

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #36309 from dillaman/wip-46694

librbd:  global and pool-level config overrides require image refresh to apply

Reviewed-by: Mykola Golub <mgolub@suse.com>