cephadm: Add tcmu-runner container when deploying ceph-iscsi

Currently when we deploy ceph-iscsi via cephadm it doesn't include a
running tcmu-runner. Which means initiators will be able to login but
you wont see the LUNS on the initiator.

This patch deploys an additional tcmu-runner container along side the
ceph-iscsi container that just runs the tcmu-runner service.

Fixes: https://tracker.ceph.com/issues/46540
Signed-off-by: Matthew Oliver <moliver@suse.com>

Merge pull request #35981 from trociny/wip-45268

librbd: don't resend async_complete if watcher is unregistered

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #36149 from adamemerson/wip-warn-more

Warning Cleanup

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jason Dillaman <jdillaman@redhat.com>

Merge pull request #36206 from idryomov/wip-read-from-replica-log-msgs

osd/PrimaryLogPG: fix up replica read log messages

Reviewed-by: Xie Xingguo <xie.xingguo@zte.com.cn>

Merge pull request #34059 from Songweibin/wip-objectmap-invalid

librbd: don't set objectmap invalid when load objectmap timed out

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #36212 from tspmelo/wip-cypress-gpu

mgr/dashboard: E2E: Fix random failures

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #36153 from votdev/increase_icon_border

mgr/dashboard: Display cluster status in favicon

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #35850 from sebastian-philipp/cephadm-set-paused-health

mgr/cephadm: Add CEPHADM_PAUSED to test_cli

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #36147 from mgfritch/cephadm-service-fixup-mypy

mgr/cephadm: clean-up service module type checking

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

librbd: propagate an error back when failed to invalidating

Signed-off-by: songweibin <song.weibin@zte.com.cn>

librbd: don't set objectmap invalid when load objectmap timed out

Add a new configuration option of `rbd_invalidate_object_map_on_timeout`,
if disabled and timed out to open/update object map, donot invalidating
object map and return -ETIMEDOUT to APP.

Signed-off-by: songweibin <song.weibin@zte.com.cn>

Merge pull request #35509 from zdover23/wip-doc-cephadm-adoption-step-9

doc/cephadm: remove confusing fragment from cephadm adoption step 9

Reviewed-by: Varsha Rao <varao@redhat.com>

mgr/dashboard: E2E: Increase defaultCommandTimeout

Fixes: https://tracker.ceph.com/issues/46623
Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: E2E: Disable GPU for Chrome Headless

Fixes: https://tracker.ceph.com/issues/46624
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #35916 from jschmid1/drivegroup_extension

python-common/drivegroups: implement option to change filter_logic

Merge pull request #35946 from jschmid1/extend_device_ls

mgr/cephadm: also list vendor/model/rotational in `device ls`

Merge pull request #35961 from p-se/cephadm-custom-image-fix

cephadm: fix usage of custom Prometheus image

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #36157 from rhcs-dashboard/fix-46601-master

mgr/dashboard/api: increase API health timeout

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #36164 from dsavineau/cephadm_remove_ktdreyer_copr_repo

cephadm: remove ktdreyer's copr repository

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

osd/PrimaryLogPG: fix up replica read log messages

Add missing spaces and don't repeat oid twice.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #36129 from matthewoliver/ipv6_cephadm_route_expiry

cephadm: Fix ipv6 network regex to support routes with expiry

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge PR #36196 into master

* refs/pull/36196/head:
mds/MDSRank: fix typo in "unrecognized"

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mds/MDSRank: fix typo in "unrecognized"

Fixes: beb12fa25315153e1a06a0104883de89776438a6
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #36081 from winndows/superfluous_break4

osd/OSDMap: Remove the superfluous break

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #36083 from winndows/superfluous_break8

key_value_store: Remove superfluous breaks

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #36080 from winndows/superfluous_break5

osd/PGBackend: Remove superfluous breaks

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #36067 from winndows/superfluous_break1

mon/OSDMonitor: Remove the superfluous break

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #36068 from winndows/superfluous_break2

common: Remove the superfluous break

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #36078 from winndows/superfluous_break7

mgr: Remove the superfluous break

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #35994 from jschmid1/osd_support_reweight

mgr/osd_support: add unittests

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #35836 from mgfritch/vstart-out-dir-logs

vstart: out dir can contain a large number of files

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #36152 from sebastian-philipp/ceph-tell-cephadm

ceph: Fix `tell` for cephadm

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35629 from dillaman/wip-librbd-asio-2

librbd: switch IO path to use new librados asio API

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #36134 into master

* refs/pull/36134/head:
mgr/nfs: Remove NParts and Cache_Size from MDCACHE block

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #36107 into master

* refs/pull/36107/head:
mds: fix mds forwarding request 'no_available_op_found'

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #36097 into master

* refs/pull/36097/head:
mds: fix nullptr dereference in MDCache::finish_rollback

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #35969 into master

* refs/pull/35969/head:
mds: optimize random threshold lookup for dentry load

Reviewed-by: Sidharth Anupkrishnan <sanupkri@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #35953 into master

* refs/pull/35953/head:
qa: Fix name collisions

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge PR #35902 into master

* refs/pull/35902/head:
ceph-dedup-tool: fix warning
mds: slave -> peer
mds: master -> leader

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #36139 into master

* refs/pull/36139/head:
mon: refactor ServiceMap::Service::get_task_summary()
mon: only dump non-empty "task status"
mgr/ServiceMap: move ServiceMap::Daemon implementation into .cc file

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #36140 into master

* refs/pull/36140/head:
mgr/volumes: fix "ceph nfs export" help messages

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #35851 from dang/wip-dang-zipper-6

Wip dang zipper 6

rbd: Parenthesize constant macro expressions

So something can't bind to the shifted argument tighter than the shift
operator.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

pybind: Silence distutils/setuptools warning

Setuptools would very much like to be imported BEFORE distutils

So move the conditional imports of setuptools above the import of
distutils.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

test/osdc: Silence implicit conversion warning

By converting explicitly.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

test/objectstore: Remove unused value

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

test/common: Fix pessimizing move

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

test/cls_queue: Remove pessimizing move

…and fix filetype in modeline.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

tools/cephfs: Explicitly convert to char

Whether char is signed or not is implementation defined, and if char
is signed converting 255 to a char will turn a positive value into a
negative.

Silence the warning by converting explicitly.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

test/random: Turn off spurious warning

Since we're testing deduction guides involving a type whose concrete
representation is unknown, this warning provides no value.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

osd: Remove unneeded copy in ranged loop initializer

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw/picojson: Pragma out limit comparison warning

intmax_t may or may not be identical to int64_t, depending on
platform. On our platform, it is.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Remove unneded copy in ranged initializer

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Fix dangling-gsl warning

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rbd: Remove unused lambda capture

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Remove unused lambda capture

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rbd_mirror: Remove unneded implicit conversions

Since we're comparing to uint64_ts and storing in a uint64_t, just
have the variable be a uint64_t.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Fix braced initializer warning

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rbd: Remove clashing definition of CephContext

Fallout from the common_fwd change

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

kv/RocksDBStore: Remove unused variable

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common: Clarify empty conditional body

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

client: Remove unneeded copies in ranged for

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

osd/OSDMap: Use reference to avoid copying in loop initializer

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

osd/OSDMap: Fix implicit conversion warning

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw/picojson: Fix assignment-used-as-value warning

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Fix sign comparison warning

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

{tools,rbd,rados,rgw}: Fix Boost.Bind placeholder warning

Use of the Boost placeholders in the default namespace is
deprecated. Switching to <boost/bind/bind.hpp> defines them in
boost::placeholders and silences the warning.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

cephadm: remove ktdreyer's copr repository

This was needed as a temporary solution until all missing EL8 packages
are built on EPEL 8.
Because we now have all dependencies on EPEL 8 then we can remove that
repository.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Clean up some leaks in rgw_admin

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

NULL out g_ceph_context when deleting it

This avoids a use-after-free when there is a crash on shutdown (or when
something like ASAN triggers an ABORT on leak).

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Zipper Object

This pass modifies RGW to use RGWObject and RGWRadosObject.
Also improves RGWBucket usage.  This converts many of the APIs to
unique_ptr.

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

mgr/cephadm: clean-up service module type checking

mypy cast to avoid:

cephadm/module.py: note: In member "_config_fn" of class "CephadmOrchestrator":
cephadm/module.py:1797: error: Incompatible return value type (got "Optional[function]", expected "Optional[Callable[[ServiceSpec], None]]")

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/dashboard/api: increase API health timeout

Fixes: https://tracker.ceph.com/issues/46601
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

cephadm: fix indentation

Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

cephadm: remove unused/commented out code

Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

cephadm: fix creating custom Prometheus image

Fix the code by improving a condition where a zero as uid or gid leads
to an additional attempt to extract the uid/gid instead of uid/gid
simply being used correctly.

Fixes: https://tracker.ceph.com/issues/46398
Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

Merge pull request #35553 from zdover23/wip-doc-iscsi-target-cli-daemon

doc/rbd: add rbd-target-gw enable and start

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Alexandra Settle <asettle@suse.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: Display cluster status in favicon

Use radius of 2px like on GitHub to increase visibility of the status circle.

Follow up of PR https://github.com/ceph/ceph/pull/36000.

Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #35846 from sebastian-philipp/podman-pull-retry

cephadm: Retry pull on transient error

Reviewed-by: Ricardo Marques <rimarques@suse.com>

ceph: Fix `tell` for cephadm

The tell target might contain dots now. Like
`mds.myfs.myhost.rnusmq` is now a valid
target.

Fixes: https://tracker.ceph.com/issues/46560
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34696 from tspmelo/wip-locl

mgr/dashboard: Use $localize and locl for TS translations

Reviewed-by: Ernesto Puertat <epuertat@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Sebastian Krah <skrah@suse.com>

mgr/cephadm: Add CEPHADM_PAUSED to test_cli

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #36000 from tspmelo/wip-favicon

mgr/dashboard: Display cluster status in favicon

Reviewed-by: Ernesto Puertat <epuertat@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #35456 from sebastian-philipp/cephadm-events

mgr/cephadm: Add k8s-style event system

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #35985 from bk201/wip-46413

mgr/dashboard: fix regression in delete OSD modal

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

mgr/nfs: Remove NParts and Cache_Size from MDCACHE block

As setting them to small value affects the performance and they are not related
to metadata caching. https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/495185

Fixes: https://tracker.ceph.com/issues/46579
Signed-off-by: Varsha Rao <varao@redhat.com>

Merge PR #36124 into master

* refs/pull/36124/head:
script/ptl-tool: add flag to turn off credits

Reviewed-by: Jos Collin <jcollin@redhat.com>

mgr/cephadm: assert the service_type during config/create

ensure the passed ServiceSpec type matches the CephadmService type

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: clean-up service module type checking

- convert to py3 style type checks
- other misc type checking fixups

Signed-off-by: Michael Fritch <mfritch@suse.com>

test/librbd: drop test_internal.ShrinkFlushCase test case

It's a racey test that is better exercised by the existing resize
state machine mock tests.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mgr/cephadm: clean-up service module type checking

- add return type to `config` func
- add missing `NFSServiceSpec` annotation to `config` func

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: clean-up service module type checking

- add missing return type to `create` func

Signed-off-by: Michael Fritch <mfritch@suse.com>

librbd: utilize neorados to issue async blacklist request

The librados API does not currently offer an async 'mon_command'
API method. Instead of adding one just to support this effort,
re-use the neorados API to issue an asynchronous 'mon_command'
for blacklisting a client.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: managed_lock::BreakRequest needs a reference to AsioEngine

The current usage of the asio::ContextWQ to similate an asynchronous blocklist
API call is resulting in deadlock in the rbd-mirror HA tests when multiple
blocklists are occurring concurrently. The next commit will switch to use the
neorados async MON command API (since librados doesn't offer one).

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: fix race condition with AIO completion callbacks

Now that librbd utilizes multiple threads for the IO path, it's
possible for a race condition to occur if a client app is waiting
on a completion to fire and the actual invokation of the
completion.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: directly post completions to the asio strand

Avoid the unnecessary wrapping of the strand completion in a
bind_executor by just directly posting the completion to the
strand.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: allocate the asio strands directly on the heap

This will assist with potential race condition debugging since the
stand pointer will be invalidated by the time the strand has been
destructed and shut down.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: ensure all asio completions are complete at ImageCtx destruction

With multiple threads of execution possible, we need to ensure that
all completions have fired prior to the destruction of the AsioEngine.
We also need to ensure that the AsioEngine is destoyed outside the
context of its owned stands.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: increased logging for asio::ContextWQ

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: switch the IO path to utilize the neorados API

IO operations to the cluster are now dispatched via the neorados
API which allows multiple threads to be utilized for processing
incoming and outgoing IO.

This also involves switching from a map for tracking sparse extents
to a vector of pairs since that matches the new API for sparse
read operations.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>