qa/cephadm: adjust host drain test to handle explicit placement warning

Since we're adding a warning if any host is listed explicitly
in the placement of any service when removing the host,
we need to adjust the host drain test that removes a host
without the --force flag to not have the explicit hostname
in the placement for the mon service.

Signed-off-by: Adam King <adking@redhat.com>

mgr/cephadm: warn when draining host explicitly listed in placement

In the case you apply a spec like

```
service_type: node-exporter
placement:
  hosts:
  - host3
```

and then you run `ceph orch host drain host3`, cephadm will remove
the daemon from that host and the placement would now match nothing.

This is definitely an issue that should be able to be bypassed as
it generally isn't serious, but it would be good to let users
know they have the host listed explicitly in placements like this
when they want to drain it.

Fixes: https://tracker.ceph.com/issues/63220
Signed-off-by: Adam King <adking@redhat.com>

Merge pull request #50212 from rishabh-d-dave/fs-swap-subcmd

cephfs: add command "ceph fs swap"

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #54417 from zdover23/wip-doc-2023-11-08-rados-troubleshooting-mon-common-issues-2-of-x

doc/rados: edit t-mon "common issues" (2 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #54397 from phlogistonjohn/jjm-buildpy-latin1-fix

cephadm: work around pip failure on some envs

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #54413 from zdover23/wip-doc-2023-11-08-rados-troubleshooting-mon-common-issues-1-of-x

doc/rados: edit t-mon "common issues" (1 of x)

doc/rados: edit t-mon "common issues" (2 of x)

Edit the second part of the section "Most Common Monitor Issues" in
doc/rados/troubleshooting/troubleshooting-mon.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

doc/rados: edit t-mon "common issues" (1 of x)

Edit the first part of the section "Most Common Monitor Issues" in
doc/rados/troubleshooting/troublehsooting-mon.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #54402 from athanatos/sjust/wip-fix-tmdriver

crimson/tools/store_nbd/tm_driver: use TransactionManager::remove

Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/tools/store_nbd/tm_driver: use TransactionManager::remove

231c7ac removed TransactionManager::dec_ref, adjust store_nbd/tm_driver
to use TransactionManager::remove instead.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #54158 from michaelalang/master

[cephadm](BUG): honor --skip-firewalld in section Open ports explicitly required for the daemon

Reviewed-by: Adam King <adking@redhat.com>

cephadm: work around pip failure on some envs

Work around an encoding/locale issue when the dashboard tests are run
(ubuntu 20.04).
The build.py changes brought in a9d1c62ca86 were validated for package
builds, teuthology, and other CI jobs but a different error was masking
this failure in the dashboard ci job.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #54178 from leonid-s-usov/balance-after-stop

mds/MDBalancer: ignore queued callbacks if MDS is not active

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #54149 from mihalicyn/ceph_mds_request_head_legacy_fallback_fix

MClientRequest: properly handle ceph_mds_request_head_legacy for ext_num_retry, ext_num_fwd, owner_uid, owner_gid

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #53912 from joscollin/wip-B63099-fix-prev-snapshot-none

client: avoid getvxattr op for ceph.mirror* attrs

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #53887 from lxbsz/wip-fix-check-access

client: two fix for checking mds auth access in client side

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #53899 from rishabh-d-dave/before-fs-rename-fail-fs

 cephfs,mon: fs rename must require FS to be offline and refuse_client_session to be set

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #54364 from rkachach/fix_issue_github_template

github: adding command for rook e2e jenkins job

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #54391 from rhcs-dashboard/fix-63469-main

mgr/dashboard: fix rgw multi-site import form helper

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #54101 from rhcs-dashboard/dashboard-v3-unit-test

mgr/dashboard: dashboard area chart unit test

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: fix rgw multi-site import form helper

Before : To obtain the token, generate it from your primary Ceph cluster. This token includes encoded information about the secondary cluster's endpoint, access key, and secret key.
Fix: To obtain the token, generate it from your primary Ceph cluster. This token includes encoded information about the primary cluster's endpoint, access key, and secret key.

Fixes: https://tracker.ceph.com/issues/63469
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #54372 from linuxbox2/wip-fix-deletemakers

rgwlc: fix delete marker regression in currentversion lc expiration

Merge pull request #54384 from cityofships/main

doc/cephadm: fix typo in set ssh key command

Reviewed-by: Zac Dover <zac.dover@proton.me>

doc/cephadm: fix typo in set ssh key command

Signed-off-by: Piotr Parczewski <piotr@stackhpc.com>

Merge pull request #54196 from ivancich/wip-log-bi-list-errors

rgw: log errors during "bi list"

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>

Merge pull request #54257 from ivancich/wip-fix-restore-bucket-index

rgw: handle instances w/ leading hyphens correctly

Reviewed-by: Michael Kidd <linuxkidd@gmail.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #53537 from cyx1231st/wip-crimson-osd-fix-complete-exit

crimson/osd: decouple cross-core pg submission out of the OrderedExclusivePhase

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #54197 from ivancich/wip-bucket-stats-add-gen

rgw: add layout generation info to "bucket stats"

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54365 from cbodley/wip-63455

common: admin socket catches exception by reference

Reviewed-by: Leonid Usov <leonid.usov@ibm.com>

rgwlc: fix delete marker regression in currentversion lc expiration

Fixes a regression introduced in https://github.com/ceph/ceph/pull/50680,
which attempted to re-use an already created sal object handle as
an optimiztion.  This change suppressed creation of delete markers in
current version expiration when buckets are versioned.

Fixes: https://tracker.ceph.com/issues/63458
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge PR #53568 into main

* refs/pull/53568/head:
common: resolve config proxy deadlock using refcounted pointers
common: add missing locks in config_proxy methods
common/ceph_mutex: note whether mutex debug methods are usable
qa: add reproducer for obs removal deadlock
qa: narrow search to debug_asok

Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #54134 from phlogistonjohn/jjm-cephadm-podman-props-2

cephadm: move more container engine specific logic

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #54264 from thotz/remove_realm_name_period_config

rgw/rgw_period : remove realm name from period config

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54342 from anthonyeleven/anthonyeleven/improvements

doc/radosgw: Improve dynamicresharding.rst

Reviewed-by: Zac Dover <zac.dover@proton.me>

Merge pull request #54147 from AliMasarweh/wip-alimasa-perf-counters

RGW: perf counters for persistent topic stats

reviewed-by: yuvalif, alimaredia, cbodley

common: admin socket catches exception by reference

probably not a real bug, but resolves a compiler warning:

src/common/admin_socket.cc: In member function ‘int RaiseHook::parse_signal(std::string&&, ceph::Formatter*, std::ostream&)’:
src/common/admin_socket.cc:869:21: warning: catching polymorphic type ‘class std::invalid_argument’ by value [-Wcatch-value=]
  869 |       } catch (std::invalid_argument) {
      |                     ^~~~~~~~~~~~~~~~

Fixes: https://tracker.ceph.com/issues/63455
Signed-off-by: Casey Bodley <cbodley@redhat.com>

cephadm: move logic to build pull command to container engines

Move the logic needed to build a pull command to a function in the
container_engines.py file. This continues the effort to improve the
locality of items with regards to specific options for container
engines. I had thought to move the whole pull function but the logging
and retry code made me change my mind. We can always move it later if it
makes sense.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: move option for setting unlimited pids into engine classes

Move the option for setting unlimited pids for the container engine
into the container engine classes. This continues the attempts to
improve the locality of items specific to certain classes to be
by making them part of the classes.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: move some podman specific mount logic to podman class

Signed-off-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #54285 from thmour/main

mgr/cephadm: discovery service (port 8765) fails on ipv6 only clusters

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #54126 from phlogistonjohn/jjm-cephadm-remove-get_daemon_args

cephadm: remove get_daemon_args function

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #53928 from adk3798/rm-cleanup-on-failure

cephadm: remove --cleanup-on-failure flag

Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #53737 from adk3798/rm-host-crushmap

mgr/cephadm: support for removing host entry from crush map during host removal

Reviewed-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #52492 from phlogistonjohn/jjm-cephadm-jinja

Add templating, via jinja2, to cephadm

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #53842 from BBoozmen/oozmen_sts_max_duration

RGW/STS: honor configured limits when updating max session duration

Reviewed-by: Pritha Srivastava <prsrivas@redhat.com>

Merge pull request #53997 from Inode1/rgw-subuser-user-policy-check

rgw: add subuser to user policy condition check

Reviewed-by: Pritha Srivastava <prsrivas@redhat.com>

Merge pull request #54050 from shreyanshjain7174/fix_62646

RadosGW: RGW Swift info not providing max_deletes_per_request in bulk_delete

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54174 from cbodley/wip-63306

rgw: fetch_remote_obj() uses uncompressed size for encrypted objects

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #54213 from cbodley/wip-63333

rgw-check-policy: remove check for nonempty tenant

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #50680 from linuxbox2/wip-notify-lctrans

rgwlc: dispatch S3 notifications on transition and mpu abort

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #54307 from rkachach/fix_issue_63326

mgr/dashboard: fix secure_monitoring_stack check

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

github: adding command for rook e2e jenkins job

Signed-off-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #54352 from zdover23/wip-doc-2023-11-06-rados-troubleshooting-mon-common-issues-formatting

doc/rados: format Q&A list in tshooting-mon.rst

Merge pull request #54222 from nbalacha/deep_copy_logs

librbd: additional logs for deep-copy debugging

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

librbd: additional logs for debugging

Added image and object information to the deep_copy class constructors
debugging easier.

Signed-off-by: N Balachandran <nibalach@redhat.com>

Merge pull request #54351 from cyx1231st/wip-seastore-cleanup-is-stable-cleanup-only

crimson/os/seastore: cleanup, be explicit about stable and stable_written

Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>
Reviewed-by: Myoungwon Oh <myoungwon.oh@samsung.com>

RGW: perf counters for topic stats

Signed-off-by: Ali Masarwa <ali.saed.masarwa@gmail.com>

Merge pull request #54347 from yuvalif/wip-yuval-coverity-y2k38

rgw: adding y2k38 annotations to supress coverity issues

reviewed-by: vedanshbhartia, idryomov

Merge pull request #54286 from jnsaff/patch-1

doc: Update mount-using-fuse.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #54336 from adk3798/cephadm-dashboard-e2e-fedora-link

mgr/dashboard: update fedora link for dashboard-cephadm-e2e test

Reviewed-by: Nizamudeen A <nia@redhat.com>

doc/rados: format Q&A list in tshooting-mon.rst

Bold the questions in a Q&A list in the "Most Common Monitor Issues"
section of doc/rados/troubleshooting/troubleshooting-mon.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

crimson/os/seastore: cleanup, be explicit about stable and stable_written

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #54325 from cyx1231st/wip-seastore-fix-seastore-pr-54036

crimson/os/seastore/transaction_manager: add the missing UNIT_TESTS_BUILT guard

Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #54344 from zdover23/wip-doc-2023-11-05-rados-troubleshooting-mon-clock-skews-edit

doc/rados: edit t-mon.rst text

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

rgw: adding y2k38 annotations to supress coverity issues

the supression comments must be directly above the line
where the issue is.

this is a followup on these commits:
edfb5b9049ee897779863d780bfbaa130a7fbc40
aca2668c88a048ab556c53411dcf2ccd43099d50

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

doc/rados: edit t-mon.rst text

Clarify the text in the "Clock Skew" section of
doc/rados/troubleshooting/troubleshooting-mon.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #54343 from zdover23/wip-doc-2023-11-05-rados-troubleshooting-troubleshooting-mon-clock-skews-formatting

doc/rados: format Q&A list in t-mon.rst

doc/rados: format Q&A list in t-mon.rst

Format a Q&A list in doc/rados/troubleshooting/troubleshooting.rst, in
the "Clock Skews" section.

Signed-off-by: Zac Dover <zac.dover@proton.me>

cephadm: eliminate the _get_daemon_args function

Move the last remaining bit of daemon args logic (ceph) out of
the _get_daemon_args function and remove the function.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: move monitoring daemon args out of _get_daemon_args

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: move jaeger tracing daemon args out of _get_daemon_args

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: move nfs daemon args out of _get_daemon_args

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: move ceph exporter daemon args out of _get_daemon_args

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: move haproxy daemon args out of _get_daemon_args

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: move custom container daemon args out of _get_daemon_args

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: move snmp daemon args out of _get_daemon_args

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: move daemon args assignment to a variable

Move the call to _get_daemon_args out of the function call.
This will aid with the next refactoring steps, were the content
of _get_daemon_args will be lifted into get_container.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: ensure entrypoint is assigned a string

Ensure that the entrypoint variable is assigned a string. The
custom container class can have None and that can confuse future
refactoring.
The mypy command we're using does not notice this.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: make get_daemon_args private

The get_daemon_args function now only has one caller. To double check
that and to prepare for this function's eventual removal we prefix the
name with an underscore to take it private.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: use ceph specific get_daemon_args calls

Use the new Ceph class specific get_daemon_args method instead of the
generic get_daemon_args calls that check all sorts of daemon types when
we're running specialized mon containers, which we know for certain are
ceph based.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: move tracing get_daemon_args logic to class

Move the logic used to get deamon args into a method of the
Tracing class.
This requires an update to save a reference to ctx.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: move monitoring get_daemon_args logic to class

Move the logic used to get deamon args into a method of the Monitoring
class.
This requires an update to save a reference to ctx.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: move ceph get_daemon_args logic to a method on the class

Move the logic used to get deamon args into a method of the Ceph class.
This requires an update to save a reference to ctx.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: consolidate if-blocks in get_container function

Instead of having a number of separate if-statement blocks try to
reduce the number of sections so that you can look at one block
and see what the function is doing for that daemon type.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: move all pids-limit logic to renamed function

Repurpose set_pids_limit_unlimited as _update_pids_limit, moving the
logic to decide if the daemon should have unlimited pids into the
function. This was the only caller of set_pids_limit_unlimited.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: remove get_deployment_container

Remove get_deployment_container replacing all calls to it with calls to
to_deployment_container. Now, callers can inject modifications to the
container object between calls or even have to_deployment_container
update container objects that were constructed in some other way.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: split get_deployment_container function

Split get_deployment_container into to_depoyment_container and
a call to get_container. A future change will remove
get_deployment_container entirely.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: enhance test case with env vars

Add some env vars to the custom container test to help assure
future changes don't overlook the ability to set them.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: fix mistaken call to get_container_mounts_for_type

When this function was refactored two variants were created:
get_container_mounts_for_type and _get_container_mounts_for_type.  The
latter doesn't append podman specific mounts so that it can be called
from other functions that do append podman specific mounts.
Unfortunately, I didn't call the correct function from
get_container_mounts and therefore it was redundantly doing the podman
mounts update. This corrects that error.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #54104 from phlogistonjohn/jjm-cephadm-move-extract

cephadm: move extract_uid_gid function

Reviewed-by: Adam King <adking@redhat.com>

doc/radosgw: Improve dynamicresharding.rst

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

Merge pull request #54056 from rkachach/fix_issue_ci_for_rook

mgr/rook: adding e2e testing for rook orchestrator

reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #54339 from aclamk/fix-aclamk-bs-reshard-doc-v2

doc/rados/configuration/bluestore-config-ref: Fix lowcase typo

Merge pull request #51784 from muahdib69/rsachere_cache_ratios

osd: make _set_cache_sizes ratio aware of cache_kv_onode_ratio

Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>

cephadm: switch to using template files

Switch off of the embedded template strings to using the recently
added template files.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: copy template strings into template files

Copy the current template strings into files under the `templates` dir.
Add a enum for holding the names of known template files.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: expand the templating module

Expand the templating module so that templates can be source from the
python package. Add (more) convenient to use methods.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: workaround jinja2 package loader issue

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: update agent unit file func to use jinja template

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: update get_unit_file to use jinja template

The somewhat complex string assembly of the main systemd unit file
for cephadm services can benefit from using a standard templating
approach.

Signed-off-by: John Mulligan <jmulligan@redhat.com>