doc/mgr/orchestrator: Fix formatting and examples in create OSDs

Signed-off-by: Varsha Rao <varao@redhat.com>

Merge pull request #36335 from mgfritch/cephadm-vstart-mon-cidr

vstart: infer the mon public_network

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #36334 from mgfritch/cephadm-event-multiline

mgr/orch: allow for multiline OrchestratorEvent message

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #36162 from Daniel-Pivonka/cephadm-43681

mgr/cephadm: streamline rgw deployment

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #36362 from tchaikov/wip-crimson-read-write-file

crimson: extract read_file()

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: move read_file() into common/buffer_io

so it can be reused by other components in crimson

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/common: move write_file into crimson namespace

it simply does not belong to ceph::buffer

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36355 from batrick/ptl-tool-paging

script/ptl-tool: page through github response

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36093 from sebastian-philipp/build-integration-branch-stable-branch-name

build-integration-branch: Append stable branch name

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #36288 into master

* refs/pull/36288/head:
mds/CInode: Optimize only pinned by subtrees check

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

script/ptl-tool: page through github response

This fixes the script to go through the response pages from GitHub.
Previously it would only look at the first page and potentially miss
some reviews/comments.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #36249 into master

* refs/pull/36249/head:
client: expose ceph.quota.max_bytes xattr within snapshots

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #35977 into master

* refs/pull/35977/head:
doc/cephfs: Update about cephfs-shell custom exit codes
cephfs-shell: Define cephfs-shell exit code

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #36064 into master

* refs/pull/36064/head:
mgr/volumes: Fix traceback of ops when volume doesn't exist

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #36200 into master

* refs/pull/36200/head:
mds: fix mds peer request 'no_available_op_found'

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #36233 into master

* refs/pull/36233/head:
client: fix extra open ref decrease

Reviewed-by: Jeff Layton <jlayton@redhat.com>

mgr/cephadm: streamline rgw deployment

cephadm will create realm, zonegroup, and zone if needed before creating rgw service

fixes: https://tracker.ceph.com/issues/43681
Signed-off-by: Daniel-Pivonka <dpivonka@redhat.com>

Merge PR #24068 into master

* refs/pull/24068/head:
mds: rename {CDir,Migrator}::cache to mdcache
mds: make MDSCacheObject::is_ambiguous_auth() virtual
mds: make sure rename old inode's parent dirfrag is projected.
mds: track projected inode/fnode in Mutation
mds: use smart pointer to manager CDir::fnode
mds: use smart pointer to manage CInode::{inode,xattrs,old_inodes}
osdc/Filer: make layout pointer const

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/orch: allow for multiline OrchestratorEvent message

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge pull request #36053 from tchaikov/wip-mkdir

kv: replace compat_mkdir with fs::create_directory

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mds: rename {CDir,Migrator}::cache to mdcache

make it be consistant with CInode::mdcache

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: make MDSCacheObject::is_ambiguous_auth() virtual

CInode overrides is_ambiguous_auth(). Locker calls is_ambiguous_auth()
from base class.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: make sure rename old inode's parent dirfrag is projected.

if rename dest dentry is remote dentry, Server::_rename_prepare() only
pre dirty old inode, but does not project fnode for old inode's parent
dirfrag. This will trigger a assertion (introduced by previous commit)
in CDir::mark_dirty().

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: track projected inode/fnode in Mutation

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: use smart pointer to manager CDir::fnode

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: use smart pointer to manage CInode::{inode,xattrs,old_inodes}

this avoid copying whole inode_t and xattr map when journaling inodes.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #36307 from petrutlucian94/rocksdb_lz4

cmake: fix lz4 params when building rocksdb

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36295 from majianpeng/bluefs-reduce-unnecessary-flush

os/bluestore/BlueFS: Don't flush unused device.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #36232 from mgfritch/cephadm-ok-to-stop

mgr/cephadm: add `orch ok-to-stop` commands

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #34848 from changchengx/protocolv2

refine class member function implementation in ProtocolV2

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36007 from votdev/issue_46448_hosts_unit_tests

mgr/dashboard: Add hosts page unit tests

Reviewed-by: Sebastian Krah <skrah@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #36320 from s0nea/wip-dashboard-46573

mgr/dashboard: wait longer for health status to be cleared

Reviewed-by: Ni-Feng Chang <kiefer.chang@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

client: expose ceph.quota.max_bytes xattr within snapshots

For directories within snapshots, expose the ceph.quota.max_bytes
extended attribute information. This enables fetching quota
information when the snapshot was taken and is particularly useful
when cloning subvolume snapshots, to enforce the quota on the
clone subvolume as well.

Fixes: https://tracker.ceph.com/issues/46278
Signed-off-by: Shyamsundar Ranganathan <srangana@redhat.com>

Merge pull request #36323 from tchaikov/wip-crimson-msgr-v1-v2

crimson: picking peer addr of the compatible type

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #35728 from jan--f/c-v-add-subcommand-parse-drive-groups

ceph-volume: add drive-group subcommand

Merge pull request #36342 from tchaikov/wip-crimson-heartbeat-erase

crimson/osd: erase an element by iterator instead

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #36217 from sebastian-philipp/cephadm-common-mypy-ini

cephadm: use src/mypy.ini instead

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #36235 from matthewoliver/cephadm_iscsi_tcmu_runner

cephadm: Add tcmu-runner container when deploying ceph-iscsi

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

crimson/osd: implement cls_get_pool_stripe_width

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: erase an element by iterator instead

we should not remove an element while iterating it in a map, as erasing
the element invalidates the iterator, which causes segmfault when we are
advancing it after erasing the dereferenced element.

in this change, an iterator is used for walking through the map, in
comparision with creating a to-be-removed list, this one is more
efficient and more idiomatic.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36341 from tchaikov/wip-crimson-cls

crimson/osd: correct the function name of cls_cxx_map_get_vals_by_keys()

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: correct the function name of cls_cxx_map_get_vals_by_keys()

it was an oversight in 7a4c6359e483f8c71276ece5cde16eb0771ac5d2

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36079 from winndows/superfluous_break6

msg: Remove superfluous breaks

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #36297 from dvanders/dvanders_46443

osd: fix crash in _committed_osd_maps if incremental osdmap crc fails

Reviewed-by: Xiaoxi Chen <xiaoxchen@ebay.com>
Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

mgr/cephadm: add `orch host ok-to-stop` command

$ ceph orch host ok-to-stop host1
It is presumed safe to stop host host1

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: return HandleCommandResult from ok_to_stop

- return output from the result of the ok_to_stop command
- log ok-to-stop result during all invocations

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/orch: add errno to OrchestratorError

add errno to OrchestratorError and ServiceSpecValidationError exceptions

Signed-off-by: Michael Fritch <mfritch@suse.com>

qa/suites/rados/thrash/crc-failures: randomly inject bad incremental osdmap crc

Signed-off-by: Neha Ojha <nojha@redhat.com>

osd: don't write transaction when inc crc failed

80da5f9a987c6a48b93f25228fdac85890013520 exposed a flaw in how
handle_osd_map falls back to a full osdmap if the crc of an incremental
failed.

If the first message in a map message had a crc error, then the
loop would exit with last < start, which would then cause a null
dereference in _committed_osd_maps.

Fixes: https://tracker.ceph.com/issues/46443
Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>

qa/standalone/osd: add bad-inc-map.sh

Test that the osd doesn't crash when it gets a bad incremental osdmap.

Related-to: https://tracker.ceph.com/issues/46443
Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>

vstart: infer the mon public_network

set the mon public_network when deploying with the cephadm flag

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge pull request #36287 from dillaman/wip-librbd-close

librbd: use task finisher thread for image open/close callbacks

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #36253 from changchengx/exclusive

doc: specify RBD_LOCK_MODE_EXCLUSIVE for exclusive-lock

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #36328 from tchaikov/wip-crimson-cls_cxx_map_get_vals

crimson/osd: implement cls_cxx_map_get_vals()

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: implement cls_cxx_map_get_vals()

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: specify RBD_LOCK_MODE_EXCLUSIVE for exclusive-lock

The exclusive-lock could be transited transparently between clients
after finishing write operation. To disable "transparent" transition,
it needs to acquire the lock with RBD_LOCK_MODE_EXCLUSIVE.

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

Merge pull request #34928 from p-se/wip-pse-revise-monitoring-doc

mgr/dashboard: revise monitoring documentation

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

crimson: use pick_addr() for picking peer addr

in teuthology tests, there is good chance that we have ceph.conf
containing:

mon host = 172.21.15.122

which is translated to two monitors

- a: 172.21.15.122:3300
- a-legacy: 172.21.15.122:6789

both has protocol type of "any". so, to enable crimson to use settings
like this, we should let crimson to accept them, and drop the connection
if the peer claim to be using an incompatible protocol, when they are
exchanging banners.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: wait longer for health status to be cleared

Because of reasons the cluster needs more time to recover from
HEALTH_WARN while changes are made by `test_pool_update_metadata`.
Lets wait several times for the cluster status to be HEALTH_OK
again.

Fixes: https://tracker.ceph.com/issues/46573
Signed-off-by: Tatjana Dehler <tdehler@suse.com>

crimson/mon: use mon with only v2 address

crimson msgr supports v2 protocol now, so we can connect to monitor
which only provides v2 addresses.

Signed-off-by: Kefu Chai <kchai@redhat.com>

msg/msg_types.h: add pick_addr()

for picking an addr from an entity_addrvec_t by given protocol type.
so:
  - v2 => v2, any
  - v1 => v1, any
  - any => any, v1, v2

and add a helper of `addr_of_type()` to avoid repeatings.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36285 from sebastian-philipp/orch-completion-generic

mgr/orch: Add some more type annotations

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #36258 from rhcs-dashboard/fix-cpu-stats

mgr/dashboard: cpu stats incorrectly displayed

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/orch: Add some more type annotations

Made `orch.Completion` a generic type

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #36012 from adk3798/cephadm_44886

mgr/cephadm: allow use of authenticated registry

Merge pull request #36262 from sebastian-philipp/orch-readd-apply_dg

mgr/cephadm: re-add `apply_drivegroups()`

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>

Merge pull request #36306 from smithfarm/wip-add-octopus-to-release-table

doc/releases: add "octopus" column to Release Timeline

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #36301 from sebastian-philipp/doc-cephadm-status-no-progress

doc/cephadm: `status` doesn't show a progress

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>

cephadm: Add tcmu-runner container when deploying ceph-iscsi

Currently when we deploy ceph-iscsi via cephadm it doesn't include a
running tcmu-runner. Which means initiators will be able to login but
you wont see the LUNS on the initiator.

This patch deploys an additional tcmu-runner container along side the
ceph-iscsi container that just runs the tcmu-runner service.

Fixes: https://tracker.ceph.com/issues/46540
Signed-off-by: Matthew Oliver <moliver@suse.com>

mds/CInode: Optimize only pinned by subtrees check

Fixes: https://tracker.ceph.com/issues/46727
Signed-off-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #36090 from inspur-wyq/wip-37532

mon: fix the 'Error ERANGE' message when conf "osd_objectstore" is filestore

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36283 from rzarzynski/wip-bl-raw-privatization

common/bl: don't access raw::data nor raw::len directly. Use getters instead.

Reviewed-by: Neha Ojha <nojha@redhat.com>

cmake: fix lz4 params when building rocksdb

Recent RocksDB version use slightly different parameter names for
the LZ4 include/lib dirs, we'll have to pass the right ones.

We'll also have to fix the "CMAKE_TOOLCHAIN_FILE" parameter,
which isn't passed properly.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

doc/releases: add "octopus" column to Release Timeline

Octopus has been out for awhile. I suppose this should have been done
earlier, but "better late than never".

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #36245 from smithfarm/wip-mimic-is-eol

doc/releases: Mimic is EOL

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #36279 from tchaikov/wip-crimson-msgr-v2.1

crimson/net: enable on-wire-encryt and v2.1 support

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc/cephadm: `status` doesn't show a progress

Fixes: https://tracker.ceph.com/issues/45858
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #35852 from smithfarm/wip-opensuse-os-recommendations

doc/start/os-recommendations: current state of openSUSE

Reviewed-by: Tim Serong <tserong@suse.com>

Merge pull request #36269 from dang/wip-dang-46692

RGW - fix bulkupload, broken by zipper

Reviewed-by: Casey Bodley <cbodley@redhat.com>

cephadm: use src/mypy.ini instead

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

librbd: use task finisher thread for image open/close callbacks

There was a potential race condition with utilizing the AsioEngine
to deliver asynchronous image open and close callbacks. This left
the potential for the io_context thread to attempt to destroy itself.

This commit changes the behavior of the image open and close callbacks
to always delete the ImageCtx (now matches the synchronous API behavior)
and it always invokes the callback in Finisher thread whose lifetime is
tied to the CephContext.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #36219 from guits/guits-fix_zap_osdid_osdfsid

ceph-volume: filter by osd-id or osd-fsid when zapping

os/bluestore/BlueFS: Don't flush unused device.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

ceph-volume: filter by osd-id or osd-fsid when zapping

2f5c10c12c37e6865ce54bb4940d3779353cba4f introduced a bug:

`ceph-volume lvm zap` command fails under certain conditions.

when passing `--osd-id` or `--osd-fsid` to `ceph-volume lvm zap` command
it tries to zap additionnal devices that have nothing to do with the osd
being zapped.

When calling `api.get_lvs()` in `ensure_associated_lvs()` we have to
pass the osd-id/osd-fsid information so only related devices are
returned by `get_lvs()` method

Closes: https://tracker.ceph.com/issues/46627
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

mds: fix mds peer request 'no_available_op_found'

Fixes: https://tracker.ceph.com/issues/46583
Signed-off-by: Yanhu Cao <gmayyyha@gmail.com>

messages/MOSDBoot: pass OSDSuperblock by const ref

MOSDBoot's ctor does not change the parameter, so let's pass by const
reference.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os/alienstore: always use fsid in bluestore

alienstore should not be stateful in this perspective, it should proxy
all acccess of fsid to bluestore.

there are couple issues in existing implementation:

* when mkfs, bluestore tries to generate a new osd_fsid if the specified
  one is empty. but we explicitly pass the given uuid down to
  AlienStore::mkfs() so the bluestore can use it. so we should pass it
  down instad of storing it locally.
* when persisting superblock in OSD::mkfs(), superblock.osd_fsid() is
  read from store->get_fsid(), if user specifies an empty uuid, we
  should persist the generated uuid in the superblock.

in this change, all access to fsid is proxied to the underlying
bluestore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

stop.sh: stop osd before mon

osd sends a MOSDMarkMeDown message to monitor and waits for its ack
before timeout, so if we can stop osd before stopping mon, stop.sh can
return sooner without waiting until the timeout.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/mgr: only pick the addr of the same type

to avoid the attempts to connect an OSD which is bound to a v2
address to a v1 address of a mgr.

in general, osd is bound to both v1 and v2 addresses, but crimson
msgr does not support multiple bound address at the time of writing, so
to avoid the failures when trying to connect to incompatible addresses,
let's filter out them when connecting to monitor. this change
silence warnings like:

peer_addr_for_me v1:172.21.15.106:60008/0 type doesn't match myaddr
v2:0.0.0.0:6802/26710

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/MgrMap: let get_active_addrs() return a const ref

no need to create a temporary instance for referencing those addresses.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/mon: only pick the addr of the same type

to avoid the attempts to connect an OSD which is bound to a v2 address
to a v1 addrss of a monitor.

in general, osd is bound to both v1 and v2 addresses, but crimson msgr
does not support multiple bound address at the time of writing, so to
avoid the failures when trying to connect to incompatible addresses,
let's filter out them when connecting to monitor. this change silence
warnings like:

peer_addr_for_me v1:172.21.15.106:60008/0 type doesn't match myaddr
v2:0.0.0.0:6802/26710

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: print out client caps

Signed-off-by: Kefu Chai <kchai@redhat.com>

auth/cephx: implement random()->get_bytes() for crimson

instead of using CryptoRandom use the C++ standard library for
generating secret.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/admin: catch thrown exception

if the socket file exists, a std::system_error is thrown. and we should
catch it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/msgr: Revert "don't advertise the on-wire format v2.1."

This reverts commit a74948bc5095b32212189352c163030bfe10db71.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/net: enable msgr v2.1 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/net: enable on_wire encryption support

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/net: set is_rev1 for messenger v2.1 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/net: keep rx_preamble for msgr v2.1 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/net: drop stale TODO

Signed-off-by: Kefu Chai <kchai@redhat.com>