cephadm: enable logging to stderr in ceph-iscsi

All the other ceph servies when deployed are deployed with:

  --default-log-to-stderr=true --default-log-stderr-prefix=debug
  --default-log-to-file=false

Ceph-iscsi doesn't have these as commandline params but once
https://github.com/ceph/ceph-iscsi/pull/186 lands it'll have something
similar as config options. This patch sets these options

Signed-off-by: Matthew Oliver <moliver@suse.com>
(cherry picked from commit f65584637f8ee44e79e0976dcecc086d2ba7fe45)

doc/dev/cephadm: Note regarding network calls form CLI handlers

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
(cherry picked from commit 4985a9ddfca1510fc612a6d115dc00e8bdba229d)

Incorporating sebastian-philipp's suggestions.

Signed-off-by: Zac Dover <zac.dover@gmail.com>
(cherry picked from commit 845b05419ae870f87c66dcef4858d68e2ec8dd5c)

Add ceph orch zap command

Signed-off-by: Zac Dover <zac.dover@gmail.com>
(cherry picked from commit 0455c0e608a02266e8f98fef1643f03d9293ad0a)

mgr/cephadm: disable env_var passing until pr#34835 is merged

Signed-off-by: Joshua Schmid <jschmid@suse.de>
(cherry picked from commit 8f275834ff4a5adcda165ea39e263840e5dd139a)

mgr/cephadm: generate and pass env vars to cephadm

Signed-off-by: Joshua Schmid <jschmid@suse.de>
(cherry picked from commit ca9aafbccf70af466b841c7ae322410e8f3c0caf)

python-common: do not pass envs after entrypoint

Signed-off-by: Joshua Schmid <jschmid@suse.de>
(cherry picked from commit 726cabc75b30e4163f486ac014d193fa76ebf958)

bin/cephadm: move --env-var handling to global argparser

Signed-off-by: Joshua Schmid <jschmid@suse.de>
(cherry picked from commit 84fdccdbeec72f326270b11be0034dff42a44471)

mgr/cephadm: adapt env_var handling

Signed-off-by: Joshua Schmid <jschmid@suse.de>
(cherry picked from commit 6d7a65f2936c6e92330aeb466b28638c33a80e13)

cephadm: return absolute path from pathify

normalizes the user defined paths a bit better to avoid inconsistent
results with os.path.basename(), os.path.join(), etc.

For example:
$ cephadm shell --mount ~/

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 669840abc037fc8c49255a68bebae9a8279eac86)

qa/workunits/cephadm/test_cephadm.sh: add shell mount test

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 629a9eeeb54bd0bbbdc4e29dc4473ae08244c20a)

cephadm: fix shell mount for special files

block/char devs, named pipes, sockets etc

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 72ae06f117d50009baf1dbddacd0805d419f76bc)

mgr/cephadm: Remove AsyncCompletion

Simplyfy things a lot by not using multiprocessing.pool.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
(cherry picked from commit f7bf4d0c597080f45bb8466bd6bab3b9cecb03c4)

mgr/cephadm: Proper error for OSErrors

Previsously, an OSError resulted in

    OSError: cannot send (already closed?)

which is not helpful.

Fixes: https://tracker.ceph.com/issues/45032
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
(cherry picked from commit 7540ab7ba0921fd48071c31fad8f8d8e4a887be7)

cephadm: Fix error parsing <command> arguments

Optional arguments in the <command> parameter used in:
 <cephadm shell>
 <cephadm enter>
 <cephadm ceph-volume>
are not parsed properly. This fixes this problem.

Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
(cherry picked from commit ba07fa257cbdcb06037ccbab25f853cc3dfbb1e2)

mgr/cephadm: add some more type checks

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
(cherry picked from commit 85c812c4ed6a7eec25d0d7749dae376bea003fa8)

mgr/cephadm: cleanup async_map_completion etc.

This breaks the progress integration, but we haven't
used that anyway till now.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
(cherry picked from commit 24de3e21b911ca7dd3f2390104b10535e2eb2a6c)

mgr/cephadm: replace async_completion with trivial_completion

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
(cherry picked from commit 397d062759eeabad36a01ccee8bf0db3dfa1415b)

mgr/cephadm: replace async_map_completion with a simple wrapper

There is no need to wrap everything into completions.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
(cherry picked from commit a78f42e225bbc041324dc3e565deb207feddad26)
(cherry picked from commit 4bdbbeb6fd7db670fce4f39e6afac15bfa42ad54)

mgr/cephadm: ceph_default_alerts.yml can be utf-8 encoded

non-ascii chars introduced by 653c3f66823

Fixes: https://tracker.ceph.com/issues/45458
Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit bc26b1ab8cc71e36f6d3e3aa041a7b339d41a1f1)

cephadm: extend systemd stop timeout

daemon stop + unit.poststop might take longer to complete

Fixes: https://tracker.ceph.com/issues/45417
Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 3af9f0d9b99c8e9f9b9ec0b15f1932e15abf7f79)

pybind/mgr: properly mock the mon store for pytest

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
(cherry picked from commit fbc089755c3ae22854b284c1927a40204d0133fc)

mgr/cephadm: add extra function for the auth entity

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
(cherry picked from commit f18523fa219ef6e39902625ed2ae0da47695625a)

mgr/cephadm: node-exporter is not a ceph service

Fixes: https://tracker.ceph.com/issues/45427
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
(cherry picked from commit f9321ebbb42319aa47a0bac2d29790fa262ca4e9)

mgr/cephadm: move HostAssignment to new module

(speeds up my editor)

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
(cherry picked from commit 524efe899426b73dc8c665b226a89dfd8c7ae05b)

mgr/cephadm: move upgrade code to upgrade.py

Make it easier for everyone to read the upgrade algorithm.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
(cherry picked from commit cedf9dbb2b29a787c92e38f2214cc9881a442d61)

mgr/cephadm: refresh osd config when mons change

OSDs don't have a service spec, so the previous implementation
was not reconfiguring them when the list of mons changed.

Fixes: https://tracker.ceph.com/issues/45393
Signed-off-by: Tim Serong <tserong@suse.com>
(cherry picked from commit e4aa556fd0e2859b3992274b64d40e0f0561c76f)

qa/workunits/cephadm: test grafana adoption

now includes https://github.com/ceph/cephadm-adoption-corpus/commit/af85d49307e1652cd0007bf3074030bf0a63c9b7

Fixes: https://github.com/ceph/cephadm-adoption-corpus/pull/3
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
(cherry picked from commit b62dd11e3d92ec0770389aadd16b4fcb39832480)

mgr/orch: fix mypy errors

orchestrator/module.py: note: In member "_daemon_add_misc" of class "OrchestratorCli":
orchestrator/module.py:655: error: Incompatible types in assignment (expression has type "PlacementSpec", variable has type "Optional[str]")
orchestrator/module.py:656: error: Argument 1 to "ServiceSpec" has incompatible type "Optional[str]"; expected "str"
orchestrator/module.py:656: error: Argument "placement" to "ServiceSpec" has incompatible type "Optional[str]"; expected "Optional[PlacementSpec]"

orchestrator/module.py: note: In member "_apply_misc" of class "OrchestratorCli":
orchestrator/module.py:881: error: Incompatible types in assignment (expression has type "PlacementSpec", variable has type "Optional[str]")
orchestrator/module.py:882: error: Argument 1 to "ServiceSpec" has incompatible type "Optional[str]"; expected "str"
orchestrator/module.py:882: error: Argument "placement" to "ServiceSpec" has incompatible type "Optional[str]"; expected "Optional[PlacementSpec

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit e6d7fb67282dfd8d1f614980821f9cb7ff4fdf51)

mgr/orch: add missing `subcluster`, `port`, and `ssl`

subcluster, port, and ssl arguments are missing for `orch apply rgw`

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit d2066b6ecddc9a4d4dadb68cc2dcfadde38aa94a)

mgr/orch: add type checking for add/apply commands

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 02a3762e0e8b0db6727ff5533749f34c5c17763b)

mgr/orch: raise validation error for unexpected inbuf

ceph orch apply mds x -i nfs.yaml
Error ENOENT: unrecognized command -i; -h or --help for usage

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 393b1a422020094a661b6935c570fb40aa316fb5)

mgr/orch: remove inbuf logic during daemon add (rgw/iscsi)

Simplifies logic for handling the ServiceSpec

Allows for a ServiceSpec:
$ ceph orch daemon add -i iscsi.yaml

But not when combined with a service_type:
$ ceph orch daemon add iscsi <poolname> -i iscsi.yaml

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 601ca2710e871012dbbe3eba6328c9c71e62be3b)

mgr/orch: remove extraneous PlacementSpec.validate()

PlacementSpec is being validated during `__init__()`

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 83631fc3734ef51c874955fc4c86b187a33527c2)

mgr/orch: add mds, rgw, nfs daemon types

$ ceph orch add -i nfs.yaml
Error ENOENT: unknown daemon type `nfs`

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit e4c817eb474ef2f8a5798ae6cb72ec25771c444e)

mgr/orch: use `service_type` as the `daemon_type`

$ ceph orch add -i mon.yaml
Error ENOENT: unknown daemon type `None`

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 47e078133d546c68be471592f5431e02354c3102)

mgr/orch: pass `inbuf` to add/apply (mds, rgw, nfs)

$ ceph orch apply mds a -i mds.yaml
TypeError: _apply_mds() got an unexpected keyword argument 'inbuf'

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit f285bbf5428d1233917ec66e68b30517bd437a76)

mgr/orch: add misssing `raise_if_exception`

pickle exceptions during during `nfs apply`

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit e503b134c2bc10e78c5016a80efdfafb34d0be2d)

doc/cephadm: add section about setting config settings

Fixes: https://tracker.ceph.com/issues/44284
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
(cherry picked from commit 1256457a15eba7c6668f72a5f8743852e4f030ae)

mgr/cephadm: Add IscsiService

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
(cherry picked from commit 20337c9b2a8bd3385e03497dc6ff4a09d9069210)

mgr/cephadm: Add CrashService

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
(cherry picked from commit 95952995f05e27853cb761e9441047922b50196c)

mgr/cephadm: Add NodeExporterService

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
(cherry picked from commit 92c53f58f82fe2638150ac0fd72a2d74876969ad)

mgr/cephadm: Add PrometheusService

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
(cherry picked from commit 27dc31ac84f781c43c87c8ea6e887cb832cf78ba)

mgr/cephadm: Add AlertmanagerService

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
(cherry picked from commit d74f0e9949f72fd8c7793b4435d773d85b910f91)

mgr/cephadm: Add GrafanaService

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
(cherry picked from commit b197904dda8e0fa51f05a6b8b33ad09818f3d479)

mgr/cephadm: Add RbdMirrorService

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
(cherry picked from commit 3747badbff093a8e72e98ad4bf3f84bc7b13d76b)

mgr/cephadm: Add RgwService

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
(cherry picked from commit a67ab0ae963f45d6608f8166bfe0b537fe0b31d7)

mgr/cephadm: Add MdsService

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
(cherry picked from commit daf83a962d7652032d96caf24aa937172ef2a2b1)

mgr/cephadm: Add MgrService

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
(cherry picked from commit 5b67b81a33fe3066115369155ec1524c43e503d4)

mgr/cephadm: Add MonService

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
(cherry picked from commit e2ddfefaaa755abc58efde1b9e7756b04ee58154)

mgr/cephadm: Add NFSService

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
(cherry picked from commit 6c173b6db9226b961fd9f939146c4c140158b15a)

mgr/cephadm: move OSD related code to OSDService

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
(cherry picked from commit 1bd19fc967bcc7c2abfb6e24512c55663c030135)

mgr/cephadm: move nfs and osd to services/

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
(cherry picked from commit 011747c74e9308d9301613395020800f556e85c4)

debian: fix typo in cephadm user gecos

Signed-off-by: Alexey Miasoedov <alexey.miasoedov@gmail.com>
(cherry picked from commit 7577b6385dbc759896761522424f27f99872a383)

mgr/cephadm: move SpecStore and HostCache to inventoy.py

Make module.py a bit smaller

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
(cherry picked from commit c2c7829c9f9d544445706fa256802acc2672cf31)

cephadm: give ceph-iscsi permissions to configfs

The cephadm container mounts the configfs and then bind mounts (-v) it
into the container. Currently the container is not a priviliaged
container which leads to 2 problems:

 1. The container can't insert the iscsi_target_mod kernel module; and
 2. The container can't write to the configfs as that's only writeable
    by root.

We _can_ get around 1, by preloading the kernel module. I.E add it the
systemd unit file. But that doesn't help with 2.

I've tried mounting the configfs with uid and gid options, but configfs
doesn't use them.

If we make the container a priviliged container then magically both 1
and 2 are solved. We don't need to preload the module so that's one less
workaround. But more importantly, configfs can be written to so we can
create ISCSI targets etc.

So that's what this patch does, it makes iscsi containers privileged
containers by setting the CephContainer --priviliged while creating it.

Fixes: https://tracker.ceph.com/issues/45252
Signed-off-by: Matthew Oliver <moliver@suse.com>
(cherry picked from commit 9b7dcb894c370570bf1e16982508eadb8b0c6f32)

python-common: do not pass affinity to ceph_volume when None

Signed-off-by: Joshua Schmid <jschmid@suse.de>
(cherry picked from commit fbed226327ea67ea56513568213101f7a8ef51b7)

mgr/cephadm: add support for proper osd daemon tracking

Signed-off-by: Joshua Schmid <jschmid@suse.de>
(cherry picked from commit 1c0a0a6f6153dc555085f263f161666cd171964a)

cephadm: Add '--mount' option to mount host file or directory

Fixes: https://tracker.ceph.com/issues/45284
Signed-off-by: Ricardo Marques <rimarques@suse.com>
(cherry picked from commit 71c58f18b5b01e45e7ff1d0a0c086cd682d10307)

cephadm: minor osd_type cleanup in OSD adopt

This addresses a couple of the comments in
https://github.com/ceph/ceph/pull/34565

Signed-off-by: Tim Serong <tserong@suse.com>
(cherry picked from commit df201823b5263502b727c5a056de006b796bc49a)

cephadm: Add `chown` to unit.run for adoped simple OSDs

We're not using `ceph-volume simple activate` anymore, so need to
add `chown ceph:ceph` for each of block, block.db and block.wal
to ensure old-style simple OSDs are able to start.

Fixes: https://tracker.ceph.com/issues/45129
Signed-off-by: Tim Serong <tserong@suse.com>
(cherry picked from commit 6afa31d1f9ec62535a8a605dcf9c37de73ccf40a)

mgr/orch: match last occurance of hostname in service_id

The RGW realm/zone might have a substr that matches the hostname

Fixes: https://tracker.ceph.com/issues/45294
Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 3cbebf37a627154b9d4a7d67d53d5bf0d8639fc7)

mgr/cephadm: use keyword instead of positional args

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 7efa0a6fdadcafb4df0cd1068f7b8482ebae0ce5)

mgr/cephadm: get service_name from DaemonDescription

use the shared logic in the DaemonDescription class

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 01604b6e43b57e0f5ff93fe316da1eb75ed156f3)

mgr/orch: allow '.' chars in the service_id

service_id can contain a '.' char (mds, nfs, iscsi)

Fixes: https://tracker.ceph.com/issues/45293
Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 0bd6d46af68ad25d54ab7b3dfd3f55731fe810cf)

doc/mgr/orchestrator: update iscsi

`apply iscsi` and `add iscsi` are implemented in cephadm

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 586eb6ccb994197b00b64b8a3a164b4140902c5d)

mgr/orch: add iscsi apply cli

`$ ceph orch apply iscsi --pool <pool>`

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 884fc3d07fe86c004e188626eb4c8aeb384ee6a4)

mgr/cephadm: _apply_service: fix _create_iscsi invocation

Fixes: https://tracker.ceph.com/issues/45249
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
(cherry picked from commit 53da12e7eb5e58eaa7bb205f565cec6e8caa4f27)

cephadm: Make ceph-iscsi SSL aware

Ceph-iscsi's `rbd-target-api.py` supports listening over SSL if you
provide an SSL cert and key. Originally the script is opinionated and
requires these files to be named `/etc/ceph/iscsi-gateway.{crt,key}`.

When dealing with containers, having to place files inside a container to
enable SSL isn't very clean. To make things easier, like RGW, you can
now place the SSL cert and key data in the mon config-key store.

This will mean there are 2 ways to enable SSL in ceph-iscsi via orch/cephadm.

1. Push the SSL key and cert into the mon config-key under the keys, and
   then make sure api_secure is enabled (requires json):

  iscsi/{clientname}/iscsi-gateway.crt
  iscsi/{clientname}/iscsi-gateway.key

2. Provide the SSL key and cert in the json you pass the orchestrator and
   it'll push them up for you.

Also lockdown the caps so the container can only access iscsi ssl
key/certs.

Signed-off-by: Matthew Oliver <moliver@suse.com>
(cherry picked from commit 4c942d05949f0cb79963258b6f75093bb5d1d4d9)

cephadm: fix the hang-up of cryptsetup on creating encrypted OSD

Creating encrypted OSD fails due to the container's dedicated ipc
namespace. Sharing the ipc namespace with host resolve this problem.

Here is the log.

 ```
 host# cephadm shell
 ...
 # cat spec.yaml
 service_type: osd
 service_id: example_osd_spec
 placement:
   host_pattern: '*'
 data_devices:
   all: true
 encrypted: true
 # ceph orch apply osd -i spec.yaml
 NAME             HOST       DATA     DB WAL
 example_osd_spec ubuntu1804 /dev/sdb -  -
 # ceph osd tree
 ID  CLASS  WEIGHT  TYPE NAME     STATUS  REWEIGHT  PRI-AFF
 -1              0  root default
  0              0  osd.0           down         0  1.00000
 ```

It happens because cryptsetup wait some semaphore forever. This
semaphore is acquired in the host namespace.

 ```
 # exit              # log out from shell
 host# pstree -p
 ...
            |                  |-containerd-shim(11804)-+-ceph-volume(11821)---cryptsetup(12004)
 ...
 host# ps --pid 12004 -o pid,comm,args,wchan
   PID COMMAND         COMMAND                     WCHAN
 12004 cryptsetup      /usr/sbin/cryptsetup --key- semtimedop
 host#
 ```

Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
(cherry picked from commit bcdabfc96c15c83afc01c155eb6b424a609271ae)

Merge pull request #35120 from Vicente-Cheng/wip-45500-octopus

octopus: rgw: anonomous swift to obj that dont exist should 401

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #34993 from smithfarm/wip-45492-octopus

octopus: rgw: fix bug where bucket listing end marker not always set correctly

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #34989 from smithfarm/wip-45485-octopus

octopus: rgw: radosgw-admin: fix infinite loops in 'datalog list'

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #34988 from smithfarm/wip-45484-octopus

octopus: rgw: fix rgw tries to fetch anonymous user

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #35126 from smithfarm/wip-45598-octopus

octopus: librbd: avoid completing mirror:DisableRequest while holding its lock

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #34994 from smithfarm/wip-45364-octopus

octopus: qa/workunits/rbd: fix list-mapped filter in unmap_device

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #34985 from smithfarm/wip-45498-octopus

octopus: rgw: fix some list buckets handle leak

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #34940 from Vicente-Cheng/wip-45366-octopus

octopus: rgw/notifications: fix zero size in notifications

Reviewed-by: Yuval Lifshitz <yuvalif@yahoo.com>

Merge pull request #34992 from tspmelo/wip-45470-octopus

octopus: mgr/dashboard: Reduce requests in Mirroring page

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #34850 from s0nea/wip-45334-octopus

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #34947 from s0nea/wip-45429-octopus

octopus: doc/mgr/dashboard: Add troubleshooting guide

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #34987 from smithfarm/wip-45479-octopus

octopus: doc/rgw: document 'rgw gc max concurrent io'

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

librbd: avoid completing mirror:DisableRequest while holding its lock

Ensure that the lock is released before another thread has the chance
to complete the state machine and attempt to destruct the in-use lock.

Fixes: https://tracker.ceph.com/issues/45544
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit c45324a14f6bb02896549d5f79ad42f942afb5d1)

Merge pull request #34893 from sebastian-philipp/octopus-backport-34718-34695-34676-34698-34747-34590-34733-34707-34690-34824-34723-34728-34779

octopus: cephadm: Batch backport May (1)

Reviewed-by: Michael Fritch <mfritch@suse.com>

rgw: anonomous swift to obj that dont exist should 401

Currently, if you attempt to GET and object in the Swift API that
doesn't exist and you don't pass a `X-Auth-Token` it will 404 instead of
401.

This is actually a rather big problem as it means someone can leak data
out of the cluster, not object data itself, but if an object exists or
not.

This is caused by the SwiftAnonymousEngine's, frankly wide open
is_applicable acceptance. When we get to checking the bucket or object
for user acceptance we  deal with it properly, but if the object doesn't
exsit, because the user has been "authorised" rgw returns a 404.

Why? Because we always override the user with the Swift account.
Meaning as far as checks are concerned the auth user is the user, not
and anonymous user.

I assume this is because a swift container could have world readable
reads or writes and in slight s3 and swift api divergents can make these
interesting edge cases leak in.

This patch doesn't change the user to the swift account if they are
anonymous. So we can do some anonymous checks when it suits later in the
request processing path.

Fixes: https://tracker.ceph.com/issues/43617
Signed-off-by: Matthew Oliver <moliver@suse.com>
(cherry picked from commit b03d9754e113d24221f1ce0bac17556ab0017a8a)

15.2.2

Merge branch 'wip-joshd-fix-octopus' into octopus

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #35067 from cbodley/wip-qa-rgw-too-many-shards-octopus

octopus: qa/rgw: remove 1999-shard tests from rgw/verify

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

qa/rgw: remove 1999-shard tests from rgw/verify

causing test failures because the host runs out of disk space

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 01516061ece6ebe4d265a994e5f1f8d255334fcd)

PendingReleaseNotes: note about security fix

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
(cherry picked from commit 06f239fc35f35865d2cf92dda1ac8f4d5fe82bde)

mgr: require all caps for pre-octopus tell commands

This matches the requirements for admin socket commands
sent via tell elsewhere.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
(cherry picked from commit 347003e13167c428187a5450517850f4d85e09ad)

mon: enforce caps for pre-octopus client tell commands

This affects only the commands whitelisted here - in particular
injectargs requires write access to the monitors.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
(cherry picked from commit fc5e56b75a97c4652c87e9959aad1c4dec45010d)

qa/rgw: fix issue error in tests_ps.py

The Python interpreter on teuthology complains about the lack of
parens in a print statement.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 92180651095924baaaf33286998e32c687744a2b)

rgw/notifications: fix zero size in notifications

delete notifications and multipart upload complete notifications
send the correct object size

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>
Fixes: https://tracker.ceph.com/issues/45150
(cherry picked from commit f04f01ad429ff158f2da8346f884335a600c0e41)

Conflicts:
src/test/rgw/rgw_multi/tests_ps.py
  - NamedTemporaryFile is not introduced to octopus, use TemporaryFile

qa/workunits/rbd: fix list-mapped filter in unmap_device

In recent versions `rbd list-mapped` does not print the white space
at the end of the line.

Fixes: https://tracker.ceph.com/issues/45305
Signed-off-by: Mykola Golub <mgolub@suse.com>
(cherry picked from commit d667e385a12ea4125dbf74da97a9ba4911a3f7bc)

rgw: fix bug where bucket listing end marker not always set correctly

Due to using std::move a second time, often
RGWRados::cls_bucket_list_ordered would not set its end marker
correctly. This affected the behavior of
RGWRados::Bucket::List::list_objects_ordered, and might have caused it
return fewer elements than it may have otherwise.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 76043e00851be0b131448a7a2ab4fed2c22426fe)

mgr/dashboard: Reduce requests in Mirroring page

Fixes: https://tracker.ceph.com/issues/45385
Signed-off-by: Tiago Melo <tmelo@suse.com>
(cherry picked from commit f1a020eaf5644d9c7b43c40445a74f89e0e8b536)

radosgw-admin: fix infinite loop in 'datalog list'

affects datalog shards with more than 1000 entries

Fixes: https://tracker.ceph.com/issues/44751
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 166f38fe70e6afd2831bdb87a27c8fe757899b95)

rgw: fix rgw tries to fetch anonymous user

rgw tries to fetch the anonymous user although it does not exist.
Fixes: https://tracker.ceph.com/issues/44772
Signed-off-by: Or Friedmann <ofriedma@redhat.com>
(cherry picked from commit a221274ffce356d776010706deb1005ef1d68ee5)

doc/rgw: document 'rgw gc max concurrent io'

Fixes: https://tracker.ceph.com/issues/44958
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit b0563a093ada6d8d618a76f2df4617ee4190b999)