Client: make sure the Finisher's mutex lock not held during it being distructed

The objecter_finisher is already started in Client::Client(), but
in the failure path when initializing and starting the Client object,
we may not get a chance to call the Client::shutdown() to stop the
Finisher thread, which maybe still holding the mutex lock in it. Then
when destrucing the Finisher object the pthread_mutex_destroy() will
fail.

This fix will delay the objecter_finisher thread to start in ::init()
until we're ready to call Client::shutdown on any errors instead.

Fixes: https://tracker.ceph.com/issues/44389
Signed-off-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #34184 from cbodley/wip-qa-rgw-test-multi-fix

qa/rgw: multisite test fixes

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33581 into master

* refs/pull/33581/head:
mgr/cephadm: add watch_url to NFS bootstrap config
mgr/cephadm: check for an existing NFS config object
mgr/cephadm: combine `get_daemon_name` and `get_rados_user` funcs
mon: allow `auth caps` for `mgr` profile
mgr/cephadm: add deployment example to NFS docs
mgr/cephadm: add utils.py
mgr/cephadm: create/update keyring during nfs config
mgr/cephadm: pass NFSServiceSpec instead of pool/namespace args
mgr/cephadm: add nfs apply test
mgr/cephadm: fix nfs redeploy/reconfig
mgr/cephadm: ensure nfs uses the correct entity
mgr/cephadm: fix service name for nfs type
mgr/cephadm: apply nfs service spec
mgr/cephadm: don't create a unique nfs daemon name
mgr/cephadm: update docs
mgr/cephadm: add minimal caps for nfs client keyring
mgr/cephadm: add logging to NFSGanesha class
mgr/cephadm: create empty nfs rados config obj
mgr/cephadm: add nfs tests
mgr/cephadm: add a minimal ganesha.conf
mgr/cephadm: deploy nfs-ganesha service
mgr/cephadm: split out `config-and-keyrings` json
mgr/orchestrator: rearrange interface to match impl. order

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #34181 into master

* refs/pull/34181/head:
mgr/cephadm: bind grafana to all interfaces

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #34182 into master

* refs/pull/34182/head:
mgr/cephadm: fix 'orch ps --refresh'

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #33910 from xiexingguo/wip-44532-2

osd/PeeringState: do not trim pg log past last_update_ondisk

Reviewed-by: Yan Jun <yan.jun8@zte.com.cn>
Reviewed-by: Samuel Just <sjust@redhat.com>

mgr/cephadm: add watch_url to NFS bootstrap config

watch for changes to the per-daemon RADOS conf file

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: check for an existing NFS config object

assume an existing NFS config if the named rados object already exists

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: combine `get_daemon_name` and `get_rados_user` funcs

Signed-off-by: Michael Fritch <mfritch@suse.com>

mon: allow `auth caps` for `mgr` profile

cephadm orchestrator needs to update the caps during reconfig/redeploy

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: add deployment example to NFS docs

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: add utils.py

move `name_to_config_section` and `assert_valid_host` into utils.py

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: create/update keyring during nfs config

the keyring might already exist from a prior config/reconfig/redeploy.

attempt to get_or_create the keyring first and than update the keyring
caps afterward

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: pass NFSServiceSpec instead of pool/namespace args

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: add nfs apply test

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: fix nfs redeploy/reconfig

search for the `NFSServiceSpec` instead of passing empty
`cephadm_args` during `_create_daemon`

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: ensure nfs uses the correct entity

mon entity needs to start with `client.`

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: fix service name for nfs type

the `nfs` service was incorrectly reported as `mds`

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: apply nfs service spec

orch apply nfs <svc_id> [<placement>]

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: don't create a unique nfs daemon name

only one nfs daemon will be deployed per host

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: update docs

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: add minimal caps for nfs client keyring

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: add logging to NFSGanesha class

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: create empty nfs rados config obj

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: add nfs tests

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: add a minimal ganesha.conf

we only need to specify the rados pool and config object url

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: deploy nfs-ganesha service

orch daemon add nfs <svc_arg> <pool> [<namespace>] [<placement>]

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: split out `config-and-keyrings` json

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/orchestrator: rearrange interface to match impl. order

rgw impl. happens before rbd impl

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge PR #34146 into master

* refs/pull/34146/head:
mgr/cephadm: use docker.io in default container image name
mgr/cephadm: fix 'upgrade start' message when specifying a version

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #34166 into master

* refs/pull/34166/head:
mgr/cephadm: DriveGroupSpec needs to know about unmanaged attribute
mgr/cephadm: DriveGroupSpec needs to support/ignore _unmanaged_

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #34180 into master

* refs/pull/34180/head:
cephadm: Fix check_ip_port to work with IPv6

Reviewed-by: Sage Weil <sage@redhat.com>

qa/rgw: partial revert of 'import with full path'

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #34179 from cbodley/wip-rgw-admin-user-axe

rgw: remove unused librgw_admin_user

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

rgw: remove unused librgw_admin_user

unused and not worth the effort to maintain

Signed-off-by: Casey Bodley <cbodley@redhat.com>

mgr/cephadm: fix 'orch ps --refresh'

The use of host in the refresh loop was clobbering the argument value.

Fixes: https://tracker.ceph.com/issues/44513
Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: bind grafana to all interfaces

Fixes: https://tracker.ceph.com/issues/44608
Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: Fix check_ip_port to work with IPv6

Fixes: https://tracker.ceph.com/issues/43816
Signed-off-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #34173 from tchaikov/wip-test-rgw

test/rgw: should import parse not urlparse

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>

Merge pull request #34117 from smithfarm/wip-43746-follow-on

pybind/mgr/*: fix config_notify handling of default values

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34136 from tchaikov/test-msgr-stdout

test/msgr: print out informative messages to stdout

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #34015 from yuyuyu101/wip-pin-rocksdb

kv/RocksDBStore: using PinnableSlice to avoid extra value copy

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34140 from theanalyst/doc/releases/oct-distro-fix

doc: releases: add a note on debian packages & those built by the community

Reviewed-By: Kefu Chai <kchai@redhat.com>
Reviewed-By: Sage Weil <sweil@redhat.com>
Reviewed-By: Nathan Cutler <ncutler@suse.com>

test/rgw: should import parse not urlparse

urlparse is a function while parse is a submodule where parse_qsl() is
provided

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34138 from bk201/wip-44725

qa/tasks/mgr/dashboard/test_health: update mdsmap schema

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #34139 from tchaikov/wip-qa-test-py3

qa,test: py3 compatible changes and fix

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>

Merge pull request #34131 from tspmelo/wip-force-npm-resolutions

mgr/dashboard: Check for missing npm resolutions

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

mgr/cephadm: DriveGroupSpec needs to know about unmanaged attribute

Signed-off-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #34165 from tchaikov/wip-doc-release-octopus-indent

doc/releases/octopus: fix indents

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

mgr/cephadm: DriveGroupSpec needs to support/ignore _unmanaged_

Signed-off-by: Joshua Schmid <jschmid@suse.de>

doc/releases/octopus: fix indents

reduce the indent in subsection of "RGW object storage", some of the
nested items should be indented with two spaces instead of four.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/mgr/dashboard/test_health: ignore fsmap.standbys

Fixes: https://tracker.ceph.com/issues/44725
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

Merge pull request #34106 from adamemerson/wip-namespace-mon

Make ceph-mon compile without top-level using namespace in headers

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34161 from tchaikov/wip-perf-test-on-bionic

qa/suites/perf-basic: only test on bionic

qa/suites/perf-basic: only test on bionic

because centos8/rhel8 does not package collectl or pdsh anymore. but
these packages are required by CBT for collecting performance stats.
so instead of testing on all supported distros, let's run the perf tests
only on distros offering these packages.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #33851 into master

* refs/pull/33851/head:
mgr/orch: allow list daemons by service_name

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33957 into master

* refs/pull/33957/head:
Add alertmanager sample

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

Merge PR #33958 into master

* refs/pull/33958/head:
Add alertmanager adopt

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #34145 into master

* refs/pull/34145/head:
qa/workunits/cephadm: create lvs before calling cephadm

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #34130 from theanalyst/octopus-changelog

doc: add changelog for octopus release

Reviewed-By: Sage Weil <sage@redhat.com>
Reviewed-By: Nathan Cutler <ncutler@suse.com>

mgr/dashboard: Check for missing npm resolutions

Check if it's necessary to run "npm run fix:audit" before committing the changes.

This is important to resolve some issues with building and
to fix some npm vulnerabilities.

Fixes: https://tracker.ceph.com/issues/44723
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge PR #33967 into master

* refs/pull/33967/head:
mgr: ceph_pg_* metrics contains last value instead of sum all of them

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #34085 into master

* refs/pull/34085/head:
debian: add ceph-grafana-dashboards package
ceph.spec: put prometheus alerts in vendor-neutral location
mgr/cephadm: include prom alerts, if present in the container

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>
Reviewed-by: Paul Cuzner <pcuzner@redhat.com>

Merge PR #34089 into master

* refs/pull/34089/head:
mgr/cephadm: point alertmanager at all mgr/dashboard URLs

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Paul Cuzner <pcuzner@redhat.com>

Merge PR #34114 into master

* refs/pull/34114/head:
mgr/cephadm: Add example to run when debugging ssh failures

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #33917 from yuvalif/fix_bucket_notification_json_format

rgw/pubsub: invalid json format in bucket notification example

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc: releases: add a note on debian packages & those built by the community

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

mgr/cephadm: use docker.io in default container image name

Not all hosts have docker.io configured as a default registry.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #34076 from p-se/wip-pse-doc-preliminary-steps

mgr/dashboard: add preliminary steps to documentation

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/cephadm: fix 'upgrade start' message when specifying a version

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/cephadm: create lvs before calling cephadm

Fixes: https://tracker.ceph.com/issues/44682
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #33911 from tchaikov/wip-cephfs-doc

pybind/cephfs, doc: add python binding doc

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #31885 from jtlayton/wip-42447

doc: add a basic client configuration document

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Varsha Rao <varao@redhat.com>

Merge pull request #34124 from smithfarm/wip-fix-cbs-for-octopus

script/ceph-backport.sh: update for octopus

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Add alertmanager adopt

Signed-off-by: Eric Jackson <swiftgist@gmail.com>

Add alertmanager sample

Default contents of alertmanager.yml with additional peers key

Signed-off-by: Eric Jackson <swiftgist@gmail.com>

mgr/dashboard: add preliminary steps to documentation

to pick up newcomers with little to no Ceph experience.

Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

qa/tasks/cephfs: fix syntax error

it's a regression introduced by 9f6c764f10

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa: import py3 compatible modules using six

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa: import with full path

to be py3 compatible

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/*: fix config_notify handling of default values

After 5ace82e65c72847fb875fc01c419937a26a59d70 was merged, I found
three more instances of the code being patched.

The commit message of 5ace82e65c72847fb875fc01c419937a26a59d70 was/is:

"The default values are handled by mgr_module.py's _get_module_option();
the or here means that we break any non-true (0, false, none) value and
override it with the default."

Fixes: 5ace82e65c72847fb875fc01c419937a26a59d70
Fixes: https://tracker.ceph.com/issues/43746
Signed-off-by: Nathan Cutler <ncutler@suse.com>

qa/tasks/mgr/dashboard/test_health: update mdsmap schema

Fixes: https://tracker.ceph.com/issues/44725
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

script/ceph-backport.sh: bump version number

Signed-off-by: Nathan Cutler <ncutler@suse.com>

script/ceph-backport.sh: set target_branch in API case

When we falling back to the GitHub API to determine the milestone
number, we were not initializing target_branch, so the script was
broken for octopus backports.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

doc: add changelog for octopus release

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #34108 from mgfritch/cephadm-optional-config-keyring

cephadm: init config and keyring with None

Tested-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34101 from s1eke/master

doc: Fix some link errors about ceph-deploy

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/msgr: print out informative messages to stdout

they should not goto stderr

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/orch: allow list daemons by service_name

Services like rgw and mds are differentiated by service_name. For
example: mds.xyz vs. mds.abc. With current interface, we can't list all
daemons belonged to mds.xyz only. Add service_name as a new argument to
filter daemons by it.

Fixes: https://tracker.ceph.com/issues/44541
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

kv/RocksDBStore: using PinnableSlice to avoid extra value copy

Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge pull request #34103 from yanghonggang/perf-msgr-client

test/msgr/perf_msgr_client.cc: fix misleading total op num

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33753 from ukernel/wip-44448

mds: fix 'if there is lock cache on dir' check

debian: add ceph-grafana-dashboards package

Signed-off-by: Sage Weil <sage@redhat.com>

ceph.spec: put prometheus alerts in vendor-neutral location

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: include prom alerts, if present in the container

The prometheus config already looks in/etc/prometheus/alerting/*, so drop
this file into position.

Note that the file is currently only present in SUSE builds, so that needs
to be fixed still.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/octopus'

Merge PR #34100 into octopus

* refs/pull/34100/head:
doc/cephadm: notes on status/stability
doc/cephadm: some troubleshooting tips

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #34115 into octopus

* refs/pull/34115/head:
doc/releases/octopus: drop stray line
doc/releases/octopus: note about repository locations
doc/releases: include octopus in index
doc/install/get-packages: update package install instructions
doc/releases/octopus: final notes

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge PR #34126 into master

* refs/pull/34126/head:
qa/*/osd-backfill-recovery-log.sh: flush_pg_stats before checking log length

Reviewed-by: Sage Weil <sage@redhat.com>

doc/releases/octopus: drop stray line

Signed-off-by: Sage Weil <sage@redhat.com>