mgr/cephadm: add deployment example to NFS docs

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: add utils.py

move `name_to_config_section` and `assert_valid_host` into utils.py

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: create/update keyring during nfs config

the keyring might already exist from a prior config/reconfig/redeploy.

attempt to get_or_create the keyring first and than update the keyring
caps afterward

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: pass NFSServiceSpec instead of pool/namespace args

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: add nfs apply test

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: fix nfs redeploy/reconfig

search for the `NFSServiceSpec` instead of passing empty
`cephadm_args` during `_create_daemon`

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: ensure nfs uses the correct entity

mon entity needs to start with `client.`

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: fix service name for nfs type

the `nfs` service was incorrectly reported as `mds`

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: apply nfs service spec

orch apply nfs <svc_id> [<placement>]

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: don't create a unique nfs daemon name

only one nfs daemon will be deployed per host

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: update docs

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: add minimal caps for nfs client keyring

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: add logging to NFSGanesha class

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: create empty nfs rados config obj

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: add nfs tests

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: add a minimal ganesha.conf

we only need to specify the rados pool and config object url

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: deploy nfs-ganesha service

orch daemon add nfs <svc_arg> <pool> [<namespace>] [<placement>]

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: split out `config-and-keyrings` json

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/orchestrator: rearrange interface to match impl. order

rgw impl. happens before rbd impl

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge PR #34146 into master

* refs/pull/34146/head:
mgr/cephadm: use docker.io in default container image name
mgr/cephadm: fix 'upgrade start' message when specifying a version

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #34166 into master

* refs/pull/34166/head:
mgr/cephadm: DriveGroupSpec needs to know about unmanaged attribute
mgr/cephadm: DriveGroupSpec needs to support/ignore _unmanaged_

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #34180 into master

* refs/pull/34180/head:
cephadm: Fix check_ip_port to work with IPv6

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #34179 from cbodley/wip-rgw-admin-user-axe

rgw: remove unused librgw_admin_user

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

rgw: remove unused librgw_admin_user

unused and not worth the effort to maintain

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cephadm: Fix check_ip_port to work with IPv6

Fixes: https://tracker.ceph.com/issues/43816
Signed-off-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #34173 from tchaikov/wip-test-rgw

test/rgw: should import parse not urlparse

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>

Merge pull request #34117 from smithfarm/wip-43746-follow-on

pybind/mgr/*: fix config_notify handling of default values

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34136 from tchaikov/test-msgr-stdout

test/msgr: print out informative messages to stdout

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #34015 from yuyuyu101/wip-pin-rocksdb

kv/RocksDBStore: using PinnableSlice to avoid extra value copy

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34140 from theanalyst/doc/releases/oct-distro-fix

doc: releases: add a note on debian packages & those built by the community

Reviewed-By: Kefu Chai <kchai@redhat.com>
Reviewed-By: Sage Weil <sweil@redhat.com>
Reviewed-By: Nathan Cutler <ncutler@suse.com>

test/rgw: should import parse not urlparse

urlparse is a function while parse is a submodule where parse_qsl() is
provided

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34138 from bk201/wip-44725

qa/tasks/mgr/dashboard/test_health: update mdsmap schema

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #34139 from tchaikov/wip-qa-test-py3

qa,test: py3 compatible changes and fix

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>

Merge pull request #34131 from tspmelo/wip-force-npm-resolutions

mgr/dashboard: Check for missing npm resolutions

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

mgr/cephadm: DriveGroupSpec needs to know about unmanaged attribute

Signed-off-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #34165 from tchaikov/wip-doc-release-octopus-indent

doc/releases/octopus: fix indents

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

mgr/cephadm: DriveGroupSpec needs to support/ignore _unmanaged_

Signed-off-by: Joshua Schmid <jschmid@suse.de>

doc/releases/octopus: fix indents

reduce the indent in subsection of "RGW object storage", some of the
nested items should be indented with two spaces instead of four.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/mgr/dashboard/test_health: ignore fsmap.standbys

Fixes: https://tracker.ceph.com/issues/44725
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

Merge pull request #34106 from adamemerson/wip-namespace-mon

Make ceph-mon compile without top-level using namespace in headers

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34161 from tchaikov/wip-perf-test-on-bionic

qa/suites/perf-basic: only test on bionic

qa/suites/perf-basic: only test on bionic

because centos8/rhel8 does not package collectl or pdsh anymore. but
these packages are required by CBT for collecting performance stats.
so instead of testing on all supported distros, let's run the perf tests
only on distros offering these packages.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #33851 into master

* refs/pull/33851/head:
mgr/orch: allow list daemons by service_name

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33957 into master

* refs/pull/33957/head:
Add alertmanager sample

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

Merge PR #33958 into master

* refs/pull/33958/head:
Add alertmanager adopt

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #34145 into master

* refs/pull/34145/head:
qa/workunits/cephadm: create lvs before calling cephadm

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #34130 from theanalyst/octopus-changelog

doc: add changelog for octopus release

Reviewed-By: Sage Weil <sage@redhat.com>
Reviewed-By: Nathan Cutler <ncutler@suse.com>

mgr/dashboard: Check for missing npm resolutions

Check if it's necessary to run "npm run fix:audit" before committing the changes.

This is important to resolve some issues with building and
to fix some npm vulnerabilities.

Fixes: https://tracker.ceph.com/issues/44723
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge PR #33967 into master

* refs/pull/33967/head:
mgr: ceph_pg_* metrics contains last value instead of sum all of them

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #34085 into master

* refs/pull/34085/head:
debian: add ceph-grafana-dashboards package
ceph.spec: put prometheus alerts in vendor-neutral location
mgr/cephadm: include prom alerts, if present in the container

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>
Reviewed-by: Paul Cuzner <pcuzner@redhat.com>

Merge PR #34089 into master

* refs/pull/34089/head:
mgr/cephadm: point alertmanager at all mgr/dashboard URLs

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Paul Cuzner <pcuzner@redhat.com>

Merge PR #34114 into master

* refs/pull/34114/head:
mgr/cephadm: Add example to run when debugging ssh failures

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #33917 from yuvalif/fix_bucket_notification_json_format

rgw/pubsub: invalid json format in bucket notification example

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc: releases: add a note on debian packages & those built by the community

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

mgr/cephadm: use docker.io in default container image name

Not all hosts have docker.io configured as a default registry.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #34076 from p-se/wip-pse-doc-preliminary-steps

mgr/dashboard: add preliminary steps to documentation

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/cephadm: fix 'upgrade start' message when specifying a version

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/cephadm: create lvs before calling cephadm

Fixes: https://tracker.ceph.com/issues/44682
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #33911 from tchaikov/wip-cephfs-doc

pybind/cephfs, doc: add python binding doc

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #31885 from jtlayton/wip-42447

doc: add a basic client configuration document

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Varsha Rao <varao@redhat.com>

Merge pull request #34124 from smithfarm/wip-fix-cbs-for-octopus

script/ceph-backport.sh: update for octopus

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Add alertmanager adopt

Signed-off-by: Eric Jackson <swiftgist@gmail.com>

Add alertmanager sample

Default contents of alertmanager.yml with additional peers key

Signed-off-by: Eric Jackson <swiftgist@gmail.com>

mgr/dashboard: add preliminary steps to documentation

to pick up newcomers with little to no Ceph experience.

Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

qa/tasks/cephfs: fix syntax error

it's a regression introduced by 9f6c764f10

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa: import py3 compatible modules using six

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa: import with full path

to be py3 compatible

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/*: fix config_notify handling of default values

After 5ace82e65c72847fb875fc01c419937a26a59d70 was merged, I found
three more instances of the code being patched.

The commit message of 5ace82e65c72847fb875fc01c419937a26a59d70 was/is:

"The default values are handled by mgr_module.py's _get_module_option();
the or here means that we break any non-true (0, false, none) value and
override it with the default."

Fixes: 5ace82e65c72847fb875fc01c419937a26a59d70
Fixes: https://tracker.ceph.com/issues/43746
Signed-off-by: Nathan Cutler <ncutler@suse.com>

qa/tasks/mgr/dashboard/test_health: update mdsmap schema

Fixes: https://tracker.ceph.com/issues/44725
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

script/ceph-backport.sh: bump version number

Signed-off-by: Nathan Cutler <ncutler@suse.com>

script/ceph-backport.sh: set target_branch in API case

When we falling back to the GitHub API to determine the milestone
number, we were not initializing target_branch, so the script was
broken for octopus backports.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

doc: add changelog for octopus release

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #34108 from mgfritch/cephadm-optional-config-keyring

cephadm: init config and keyring with None

Tested-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34101 from s1eke/master

doc: Fix some link errors about ceph-deploy

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/msgr: print out informative messages to stdout

they should not goto stderr

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/orch: allow list daemons by service_name

Services like rgw and mds are differentiated by service_name. For
example: mds.xyz vs. mds.abc. With current interface, we can't list all
daemons belonged to mds.xyz only. Add service_name as a new argument to
filter daemons by it.

Fixes: https://tracker.ceph.com/issues/44541
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

kv/RocksDBStore: using PinnableSlice to avoid extra value copy

Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge pull request #34103 from yanghonggang/perf-msgr-client

test/msgr/perf_msgr_client.cc: fix misleading total op num

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33753 from ukernel/wip-44448

mds: fix 'if there is lock cache on dir' check

debian: add ceph-grafana-dashboards package

Signed-off-by: Sage Weil <sage@redhat.com>

ceph.spec: put prometheus alerts in vendor-neutral location

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: include prom alerts, if present in the container

The prometheus config already looks in/etc/prometheus/alerting/*, so drop
this file into position.

Note that the file is currently only present in SUSE builds, so that needs
to be fixed still.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/octopus'

Merge PR #34100 into octopus

* refs/pull/34100/head:
doc/cephadm: notes on status/stability
doc/cephadm: some troubleshooting tips

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #34115 into octopus

* refs/pull/34115/head:
doc/releases/octopus: drop stray line
doc/releases/octopus: note about repository locations
doc/releases: include octopus in index
doc/install/get-packages: update package install instructions
doc/releases/octopus: final notes

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge PR #34126 into master

* refs/pull/34126/head:
qa/*/osd-backfill-recovery-log.sh: flush_pg_stats before checking log length

Reviewed-by: Sage Weil <sage@redhat.com>

doc/releases/octopus: drop stray line

Signed-off-by: Sage Weil <sage@redhat.com>

doc/releases/octopus: note about repository locations

Signed-off-by: Sage Weil <sage@redhat.com>

doc/releases: include octopus in index

Signed-off-by: Sage Weil <sage@redhat.com>

15.2.0

doc/install/get-packages: update package install instructions

- include cephadm method
- update to include new version URLs

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #34119 into octopus

* refs/pull/34119/head:
doc: update octopus release notes for rgw

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge PR #34122 into octopus

* refs/pull/34122/head:
cephadm: default to v15 tag for octopus

Reviewed-by: Michael Fritch <mfritch@suse.com>

script/ceph-backport.sh: update for octopus

Signed-off-by: Nathan Cutler <ncutler@suse.com>

qa/*/osd-backfill-recovery-log.sh: flush_pg_stats before checking log length

It is possible for the pg dump to not be the latest when we check for newprimary
in _common_test(). This is because mgr_stats_period is 5 seconds, and we may not
have fetched the latest stats just yet. This causes the test to look at the same
stats before and after wait_for_clean.

Fixes: https://tracker.ceph.com/issues/43807 (2)
Signed-off-by: Neha Ojha <nojha@redhat.com>

cephadm: default to v15 tag for octopus

Signed-off-by: Sage Weil <sage@redhat.com>

doc: update octopus release notes for rgw

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 984f3ff7e0fe10f1b54da4271a2a0879ffea0ddb)

Merge PR #34028 into master

* refs/pull/34028/head:
doc: update octopus release notes for rgw

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

mgr/cephadm: Add example to run when debugging ssh failures

```
$ ceph orch host add foobar
Error ENOENT: Failed to connect to foobar (foobar).  Check that the host is reachable and accepts connections using the cephadm SSH key
you may want to run:
> ssh -F =(ceph cephadm get-ssh-config) -i =(ceph config-key get mgr/cephadm/ssh_identity_key) rook@foobar
$  ssh -F =(ceph cephadm get-ssh-config) -i =(ceph config-key get mgr/cephadm/ssh_identity_key) rook@foobar
ssh: Could not resolve hostname foobar: Temporary failure in name resolution
```

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge PR #34105 into master

* refs/pull/34105/head:
Merge PR #34042 into octopus
Merge PR #33959 into octopus
Merge PR #34067 into octopus
mgr/DaemonServer: add explicit check that acting matches for merge
Merge pull request #34040 from dillaman/wip-44396-partial-fix
Merge PR #34098 into octopus
mgr/rook: list rgw services
mgr/rook: tolerate timestamps that are None
mgr/orch: add 'subcluster' property to RGWSpec
mgr/rook: do not create radosgw pools
mgr/rook: refactor apply/add for rgw
Merge PR #34082 into octopus
Merge PR #34068 into octopus
cephadm: relabel /etc/ganesha mount
Merge PR #34046 into octopus
Merge PR #34092 into octopus
Merge pull request #33719 from ukernel/wip-44416
rbd-mirror: leader watcher should not cancel get locker if locker is invalid
rbd-mirror: snapshot sync request needs to check for interruption
librbd: request exclusive lock when moving to trash
rbd-mirror: basic integration with sync throttling
rbd-mirror: don't prematurely finish snapshot replay loop
rbd-mirror: pass InstanceWatcher to snapshot Replayer
doc/releases/octopus.rst: add note about ec recovery below min_size
mgr/cephadm: configure rgw_frontends for rgw service
cephadm: switch grafana image to the ceph repo
Merge PR #34034 into octopus
qa/suites/rados/cephadm/upgrade: update starting version
Merge PR #33540 into octopus
Merge PR #34023 into octopus
Merge PR #34044 into octopus
Merge PR #34030 into octopus
doc/orchestrator: update rgw creation
mgr/cephadm: clean up client.crash.* container_image settings after upgrade
cephadm: make add-repo --release and --version independent
cephadm: env over last used
mgr/orch: accept port and ssl flags to 'apply rgw'
mgr/orch: 'ceph upgrade ...' -> 'ceph orch upgrade ...'
cephadm: fall back to default for infer_image
cephadm: remove outdated check
cephadm: consolidate default image logic
remove ceph_test_rados_watch_notify
python-common/ceph/deployment/service_spec: add ssl to RGWSpec
cephadm: only infer image for shell, run, inspect-image, pull, ceph-volume
mgr/test_orchestrator: fix service filtering when using dummy data
mgr/dashboard: fix adding/removing host errors
mgr/rook: fix 'orch ps' for osds
qa: fix all the fsx.sh-invoking yaml files to install dependencies
mds: pass proper MutationImpl::LockOp to Locker::wrlock_start()

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>