cephadm: remove py2 from test_adoption

Since cephadm is py3 based, and py2 is EOL this patch
removes the py2 test iteration from test_adoption.sh

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

cephadm: remove py2 from test stream

Since py2 is EOL, and cephadm requires py3 anyway this
patch removes the py2 test iteration from the functional
testing suite.

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

cephadm: make devices lists more granular in gather-facts

The gather-facts commands returns a list of hdd and flash
devices. This patch changes the list content from being
simple strings representing the disks to a dict, making it
easier to extend in the future.

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

cephadm: add gather-facts to the qa test

Adds a gather-facts invocatio after check-host is run.

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

cephadm: enhance memory and cpu data

Added free/available memory state, and cpu loadavg info

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

cephadm: fixes to address PR requirements

Minor changes to address issues raised in the PR;
- formatting of ipv6 addresses
- missing docstrings
- NIC mtu and speed now int instead of string
- added NIC driver name
- removed discrete JSON method making gather-facts JSON only
- added upper/lower device lists to show NIC relationships
- added hostname to the JSON!
- added selinux/apparmor status
- added timestamp (epoch) for the gather-facts run
- added system uptime (secs)

Closes:

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

cephadm: add gather_facts command

The gather_facts command is intended to provide host
level metadata to the caller, which could then be used
in a number of places: orchestrator inventory, displayed
in the UI etc

Data is extracted mainly from sysfs and /proc

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

Merge pull request #36426 from sebastian-philipp/cephadm-redeploy-image

 mgr/cephadm, Add --image to `ceph orch redeploy <what>`

Reviewed-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #36445 from rhcs-dashboard/crushmap_fix

mgr/dashboard: Fix for CrushMap viewer items getting compressed vertically

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

mgr/cephadm: Refactor upgrade to call _daemon_action with new image

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm, Add --image to `ceph orch daemon redeploy <what>`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #36490 from tchaikov/wip-crimson-zero-op

crimson/osd: support CEPH_OSD_OP_ZERO op and cls_cxx_write_zero()

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: support cls_cxx_write_zero()

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: support CEPH_OSD_OP_ZERO op

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/admin: drop stale comment

the UNIX domain socket file was removed in
71c08cb02eb6c0f8220c0cf4b82332398b25de2c

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36455 from rzarzynski/wip-crimson-ops_executer-cleanups

crimson/osd: don't reset osd_op_params.clean_regions. Clean-up OpsExecuter

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: spawn osd_op_params in do_write_op(). Fix overriding.

This commit deduplicates the `OpsExexuter::do_osd_op()` by moving
`std::optional`-typed `osd_op_params` instantiation to `do_write_op()`.
It also fixes an issue where `clean_regions` of `osd_op_params` were
reset on every write, writefull or truncate.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: instantiate osd_op_params for every write.

This simplifies the logic of `OpsExecuter::flush_changes()` and
enables for moving the instantiation code to `do_write_op()` in
the future.

The performance impact seems negligible as the affected ops are
not the ones dominating in our expected workload (read/write, omap).

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: delegate the txn.empty() dispatch to OpsExecuter.

The benefit is no need to instantiate `osd_op_params` when only
non-modifying operation had been handled.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #36478 from zdover23/wip-doc-removing-auth-disable-warning

doc/rados: remove upgrade auth disable

This commit removes a recommendation to disable
authentication before performing an upgrade.

Reviewed-by: Joao Eduardo Luis
Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #35947 from orozery/crypto_object_dispatch_layer

librbd: add a crypto object dispatch layer

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #36474 from linuxbox2/wip-rgwlc-structv

rgwlc:  fix conditional decode of legacy lc op structures

doc/rados: remove upgrade auth disable

This commit removes a recommendation to disable
authentication before performing an upgrade.

Fixes: https://tracker.ceph.com/issues/30630
Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #36444 from sebastian-philipp/cephadm-upgrade-upstradestate-class

mgr/cephadm: Add UpgradeState class

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #36463 from SUSE/wip-fix-46833

cephadm: don't add `ceph-volume lvm activate` for adopted simple OSDs

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

rgwlc:  fix conditional decode of legacy lc op structures

Fixes the case where an upgraded radosgw operates on an unupgraded
OSD, so likely these cases do not execute.

Fixes: https://tracker.ceph.com/issues/46838
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #36349 from sebastian-philipp/cephadm-check_deamons_except

mgr/cephadm: guard _check_daemons

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #36246 from linuxbox2/wip-rgwlc-decode

rgwlc: fix error return from backward-compat decode

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #36345 from tspmelo/wip-action-disable

mgr/dashboard: Merge disable and disableDesc

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

mgr/caphadm/tests: Add `test_daemon_check_post`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #36396 from tspmelo/wip-doc-component

mgr/dashboard: Extract documentation link to a component

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #36372 from jan--f/c-v-fix-package-common-dep

ceph-volume: dependency on python-ceph-common

mgr/cephadm: Don't run `daemons_post` if _create_daemon fail

We can't run daemon_check_post for the type, if a single daemon failed.
Mainly cause `daemon_check_post` is run by service type.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: Add test_daemon_action_fail

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm/tests: Split `test_daemon_action`

Into:

* `test_list_daemons`
* `test_daemon_action` (now without listing daemons)

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: guard _check_daemons

Continue with other daemons, if one fails

Fixes: https://tracker.ceph.com/issues/46748
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #36446 from sebastian-philipp/mgr-cephadm-pep8-remotes.py

mgr/cephadm: PEP8tify a few files

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

librbd: add a crypto object dispatch layer

This commit adds a new object dispatch layer to enable encryption of RBD images.

Signed-off-by: Or Ozeri <oro@il.ibm.com>

Merge pull request #36148 from athanatos/sjust/wip-lba-pin

crimson: pin LogicalExtentRef lba tree ancestors

Reviewed-by: Kefu Chai <kchai@redhat.com>

cephadm: don't add `ceph-volume lvm activate` for adopted simple OSDs

This changes the logic in deploy_daemon_units() to add either `chown` calls
for simple (ceph-disk style) OSDs, or to add `ceph-volume lvm activate` calls
for LVM OSDs, rather than always adding both.  When I was working on
https://github.com/ceph/ceph/pull/34703, I'd originally added an "osd_simple"
flag to figure out what type of OSD was being adopted/deployed, but passing
that around was kinda ugly, so was removed from that PR.  This time around
I'm checking if /etc/ceph/osd/$OSD_ID-$OSD_FSID.json.adopted-by-cephadm
exists, which seems pretty safe IMO.  My only concern with this method is:
what happens if someone adopts a simple OSD, then later wants to migrate it
to LVM.  Presumably that's a destroy and recreate, keeping the same OSD ID?
If that's true, then the JSON file probably still exists, so the subsequent
create will do the wrong thing, i.e. will add `chown` calls, not `ceph-volume
lvm activate` calls.  Any/all feedback appreciated...

Fixes: https://tracker.ceph.com/issues/46833
Signed-off-by: Tim Serong <tserong@suse.com>

crimson/os/seastore: link in LBAPin after read

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: initialize dirty extents after replay

lba_manager needs to preload BtreeLBAPin and parent nodes.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager/btree: improved debugging

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager/btree: wire in btree pin set

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager/btree: add lba pin structure

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #36458 from drunkard/master

doc: fix typo in output of osdmaptool example

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36145 from orozery/object_dispatch_read_extents_with_version

librbd: add read extents with version to object dispatch interface

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

crimson/os/seastore: actually use LBAManager::complete_transaction

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/: add a meta field to FixedKVNodeLayout and use in LBANode*

Also removes LBANode::depth.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager/btree: move BtreeLBAPin to btree_range_pin.h

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #36371 from votdev/issue_46775_alertmanagerspec

mgr/cephadm: Enhance AlertManagerSpec to allow adding additional webhook receiver URLs

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #36433 from votdev/issue_46814_alertmanager

cephadm: Deploying alertmanager image is broken

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

Merge pull request #36441 from Daniel-Pivonka/cephadm-46819

mgr/orchestrator: fix placement of '1' activating dry-run bug

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #36391 from jschmid1/followup_fix_36151

mgr/cephadm: correct to_json output from str->dict

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #36388 from sebastian-philipp/cephadm-inventory-get_daemons_by_service

mgr/cephadm: get_daemons_by_service should use dd.service_name

Merge pull request #36383 from mgfritch/orch-nfs-rados-url

python-common: don't assert when building rados url

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #36405 from tchaikov/wip-heartbeat-atomic

common/HeartbeatMap: use std::atomic<time_point> for timeout

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: Extract documentation link to a component

Fixes: https://tracker.ceph.com/issues/36565
Fixes: https://tracker.ceph.com/issues/43375
Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/cephadm: PEP8tify

Those files should generate fewer conflicts for now.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: Fix for CrushMap viewer items getting compressed vertically

Intents to fix the crushmap viewer tree children getting compressed vertically when more osd's are added.

Fixes: https://tracker.ceph.com/issues/46826
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/cephadm: PEP8tify  upgrade.py

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: Add UpgradeState class

My resons are:

* This increase mypy coverage
* Makes the code more discoverable.
* Allowes us to have intelligent UpgradeState properties

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #36408 from tchaikov/wip-33725

tools/rados: Set locator key when exporting or importing a pool

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #36443 from tchaikov/wip-crimson-asok

crimson/admin: read non-string settings as well

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/admin: do not validate params of ConfigSetHook

validate_cmd() should have taken care of this

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/admin: read non-string settings as well

before this change, we were using

local_conf().get_val<std::string>(var);

for reading a setting when serving a "config get" command even if the
setting being queried is a non-string. so without this change, a failure
is returned complaining "unrecognized option..."

in this change:

* use get_val(string,string*) for querying the string representation
  of the queried setting
* drop the check for existence of "var" parameter, validate_cmd()
  always take care of this.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/admin: drop prefix from cmd desc

there is no need to repeat the prefix in the command's description,
validate_cmd() skips it anyway.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/admin: move config hooks into admin_socket.cc

they are not specific to OSD, let's move them into admin_socket.cc

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36438 from rhcs-dashboard/wip-pull_request_template_api-master

github: replace trigger phrase for api tests

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-By: Patrick Donelly <pdonell@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

doc: typo fix on output of osdmaptool example

Signed-off-by: Drunkard Zhang <gongfan193@gmail.com>

tools/rados: Set locator key when exporting or importing a pool

Fixes the following error when exporting a pool that contains objects
with a locator key set:

        error getting xattr set [object name]: (2) No such file or directory
        error from export: (2) No such file or directory

Fixes: https://tracker.ceph.com/issues/46824
Signed-off-by: Iain Buclaw <iain.buclaw@dunnhumby.com>

common/HeartbeatMap: use relaxed order

use memory_order_relaxed instead of memory_order_seq_cst, as the
ordering of storing of `timeout` and `suicide_timeout` does not matter,
and we don't need to synchorinize the cache for the consistent memory
view of local core. what we need is but the atomic read/write operation
of these individual variables.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/WorkQueue,librbd,rgw: use ceph::timespan for representing interval

better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36442 from rzarzynski/wip-crimson-op_truncate

crimson/osd: add support for CEPH_OSD_OP_TRUNCATE.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36398 from jdurgin/wip-pg-log-doc

osd/PGLog: improve documentation

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

common/options: update pg log bound descriptions to reflect current code

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

common/options: make pg log options dev-level

Changing these has performance and correctness implications, and is
rarely needed.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

common/options: describe osd_pg_log_trim_min

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

mgr/orchestrator: fix placement of '1' activating dry-run bug

when applying a placement of '1' for mon, mgr, rbd-mirror, crash, alertmanager, grafana, node-exporter, prometheus, or mds a dry-run was activated

Fixes: https://tracker.ceph.com/issues/46819
Signed-off-by: Daniel-Pivonka <dpivonka@redhat.com>

crimson/osd: add support for CEPH_OSD_OP_TRUNCATE.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

github: replace trigger phrase for api tests

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #36429 from tchaikov/wip-ceph-debug-docker-crimson

ceph-debug-docker: install ceph-crimson-osd for crimson flavor

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #35980 from tspmelo/wip-table-status

mgr/dashboard: Display table status in refresh button

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

ceph-volume: dependency on python-ceph-common

Since e5b585d15de8b07e0a179344d4187582a5c069f2 ceph-volume depends on
python-ceph-common. This commit introduces this dependency for the
ceph-osd rpm (which includes ceph-volume) and installs the dependency
for tox runs.

Fixes: https://tracker.ceph.com/issues/46772
Fixes: e5b585d15de8b07e0a179344d4187582a5c069f2
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

cephadm: Deploying alertmanager image is broken

Fixes: https://tracker.ceph.com/issues/46814
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #36401 from tchaikov/wip-pg-stat-ack

messages/MPGStats: print seq instead of tid

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #36375 from ricardoasmarques/force-registry-auth-config-set

cephadm: Fix error bootstraping with '--registry-json'

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #36386 from bk201/wip-46045

mgr/cephadm: skip Dashboard configuration if Dashboard is disabled

Reviewed-by: Varsha Rao <varao@redhat.com>

Merge pull request #36400 from tchaikov/wip-rip-xio-msgr

cmake: do not check OFED_PREFIX anymore

Reviewed-by: Casey Bodley <cbodley@redhat.com>

ceph-debug-docker: install ceph-crimson-osd for crimson flavor

would be handy if we can have crimson packages when debugging crimson
flavored branch.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35060 from yison/rwl-image-writeback-cache-hook-seq1

rbd/cache: Replicated Write Log Hook part 11

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #35659 from agayev/track-per-zone-metadata

os/bluestore: track per zone metadata

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #30838 from ifed01/wip-ifed-single-alloc

os/bluestore: use single allocator for shared bluestore/bluefs device

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #36356 from mgfritch/orch-format-not-supported

mgr/orch: disallow unimplemented format types

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

librbd: enable image cache after getting exclusive lock

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

librbd: add InitRequest and ShutdownRequest state machine

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

librbd: split ReadRequest

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

librbd: add IMAGE_CACHE feature

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

Merge pull request #36417 from tchaikov/wip-crimson-injectargs

crimson/admin: implement "injectargs" command

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

mgr/cephadm: Enhance AlertManagerSpec to allow adding additional webhook receiver URLs

Fixes: https://tracker.ceph.com/issues/46775
Signed-off-by: Volker Theile <vtheile@suse.com>