doc: broken link on troubleshooting-mon page
Fixes: https://tracker.ceph.com/issues/37451
This fix involves adding :ref: labels to the add-or-rm-mons.rst
page. I also added :ref: labels for other headings within
add-or-rm-mons.rst (for future reference).

Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>

Merge PR #25272 into master

* refs/pull/25272/head:
qa: add simple test-volumes.sh workunit and run it from fs/basic_functional
vstart.sh: create default fs via 'fs volume create'
mgr/volumes: fix oremote
mon/MgrMonitor: enable volumes module by default
mgr: create `volumes` module
mgr: cleaner constructor for CommandResult
mgr: block for latest osdmap after command execution
mgr: add MgrModule.mon_command helper
ceph_volume_client: enable using existing rados inst
mon: give ceph-mgr access to 'fs' commands

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #25160 from theanalyst/rgw-empty-zg-fix

rgw: handle cases around zone deletion

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24662 from theanalyst/reshard-stale-cleanup

rgw: reshard stale instance cleanup

qa: add simple test-volumes.sh workunit and run it from fs/basic_functional

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #25290 from noahdesu/dashboard-tox-workdir

mgr/dashboard: use dedicated tox working dir

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #25179 from tspmelo/wip-check-i18n

mgr/dashboard: Add i18n validation script

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #24900 from zmc/wip-minimal-health

mgr/dashboard: Replace dashboard service

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #25280 from tchaikov/wip-ceph-py3

ceph.in: write bytes to stdout in raw_write()

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #25261 from sebastian-philipp/mgr-handle-command-result

mgr: Add `HandleCommandResult` namedtuple

Merge pull request #25255 from tchaikov/wip-ceph-dencoder-without-cephfs

tools/ceph-dencoder: conditionally link against mds

Merge pull request #25244 from trociny/wip-crash-procname

mgr/crash: add process name to crash metadata

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24990 from dragonylffly/wip-fix-bench

tools/rados: always call rados.shutdown() before exit()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24939 from marcosps/simplify_examples

examples/librados: Remove not needed else clauses

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #24502 into master

* refs/pull/24502/head:
crushtool: implement --rebuild-class-roots command
crushtool: make --reweight re-sum choose_args weight-sets too
crushtool: --reweight should only reweight nonshadow roots
crush/CrushWrapper: reclassify: use default parent for created buckets
crush/CrushWrapper: reclassify: handle to-be-created buckets that we need twice
test/cli/crushtool/reclassify: add second gabe test case
crushtool: add --set-subtree-class; do not set class via --reclassify-root
test/cli/crushtool/reclassify: add reclassify test cases
doc/rados/operations/crush*: document reclassify
doc/rados/operations/crush: remove instructions for separate crush trees for ssd
crushtool: add --compare command
crushtool: implement --reclassify
crush/CrushCompiler: fix id scan to include class ids

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

crushtool: implement --rebuild-class-roots command

This simply rebuilds the class roots.  Normally this should create no
change in the map since whatever was making changes to the map before
should have rebuild the shadow roots at that point.

Signed-off-by: Sage Weil <sage@redhat.com>

crushtool: make --reweight re-sum choose_args weight-sets too

This ensures that the weights add us for each weight-set (and each
position).  Note that since we don't have anything that actually
creates positional weight-sets, the behavior here might not be what we
want in the end, but for the compat weight-sets (no position), we *do*
keep the weights as a properly summing tree.

Signed-off-by: Sage Weil <sage@redhat.com>

crushtool: --reweight should only reweight nonshadow roots

The shadow roots will be then be rebuilt accordingly.

Signed-off-by: Sage Weil <sage@redhat.com>

crush/CrushWrapper: reclassify: use default parent for created buckets

Note that these last 2 fixes also fix the 'h' test case.  \o/

Signed-off-by: Sage Weil <sage@redhat.com>

crush/CrushWrapper: reclassify: handle to-be-created buckets that we need twice

Signed-off-by: Sage Weil <sage@redhat.com>

test/cli/crushtool/reclassify: add second gabe test case

New adjusted test map from Dan.

Signed-off-by: Sage Weil <sage@redhat.com>

crushtool: add --set-subtree-class; do not set class via --reclassify-root

Sometimes we don't want the --reclassify-root to set the class of every
device because a small number of them are (correctly) a different class.
Allow both behaviors by adding a new, separate command to set the class
of all devices beneath a point in the hierarchy and do not implicitly do
that relabeling as part of --reclassify-root.

Signed-off-by: Sage Weil <sage@redhat.com>

test/cli/crushtool/reclassify: add reclassify test cases

Signed-off-by: Sage Weil <sage@redhat.com>

doc/rados/operations/crush*: document reclassify

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Replace dashboard service

This splits out the collection of health and log data from the
/api/dashboard/health controller into /api/health/{full,minimal} and
/api/logs/all.

/health/full contains all the data (minus logs) that /dashboard/health
did, whereas /health/minimal contains only what is needed for the health
component to function. /logs/all contains exactly what the logs portion
of /dashboard/health did.

By using /health/minimal, on a vstart cluster we pull ~1.4KB of data
every 5s, where we used to pull ~6KB; those numbers would get larger
with larger clusters. Once we split out log data, that will drop to
~0.4KB.

Fixes: http://tracker.ceph.com/issues/36675
Signed-off-by: Zack Cerza <zack@redhat.com>

Merge PR #24923 into master

* refs/pull/24923/head:
mds: properly update freeze_tree_state when fragmenting dirfrags

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/dashboard: use dedicated tox working dir

Avoids conflicting with other test environments.

Fixes: https://tracker.ceph.com/issues/37405
Signed-off-by: Noah Watkins <nwatkins@redhat.com>

Merge PR #24849 into master

* refs/pull/24849/head:
client: remove redundant abort logic during dentry invalidation test

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #24886 into master

* refs/pull/24886/head:
qa: fix delay type config name

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #24839 into master

* refs/pull/24839/head:
ceph-volume-client: allow setting mode of CephFS volumes

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Allow specifying a schema for unknown JObj keys

Signed-off-by: Zack Cerza <zack@redhat.com>

vstart.sh: create default fs via 'fs volume create'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/volumes: fix oremote

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MgrMonitor: enable volumes module by default

Signed-off-by: Sage Weil <sage@redhat.com>

mgr: create `volumes` module

This encapsulates and extends ceph_volume_client, providing
similar functionality as a service from ceph-mgr.

We used to call CephFS namespaces "filesystems", and the
ceph_volume_client-created directories "volumes".  That
terminology changes in this module: namespaces are now "volumes",
and the directory-based entity is a "subvolume".

External systems can use librados to access the command
interface provided by this module, instead of using
ceph_volume_client directly.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge PR #25020 into master

* refs/pull/25020/head:
script/ceph-backport.sh: adjust to change in github API

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #25235 from sebastian-philipp/dashboard-fix-type-annotation-error

qa/mgr/dashboard: Fix type annotation error.

Reviewed-by: Tatjana Dehler <tdehler@suse.com>

mgr: Add `HandleCommandResult` namedtuple

Mostly a documentation improvement to make it obvious
what `handle_command` is supposed to return .

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #25268 from smithfarm/wip-37392

qa/workunits/rados/test_librados_build.sh: grab files from explicit git branch

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25129 from rzarzynski/wip-common-annotate

common: avoid the overhead of ANNOTATE_HAPPENS_* in NDEBUG builds.

Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw: batch and process bucket instances

Sort through and batch bucket instances so that multiple calls to reading
current bucket info and locking can be avoided. For the most trivial case when
the bucket is already deleted we exit early with all the stale instances. When
the bucket reshard is in progress we only process the stale entries with status
done, if the bucket is available for locking then we lock down and mark the
other instances as well.

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

cli: use rgw-admin reshard stale-instances list/delete

Add a delete command as well that clears the resharded instances. We print out
the json status to indicate success or error state

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #25224 from sebastian-philipp/ceph-volume-fix-json

ceph-volume: fix JSON output in `inventory`

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #25256 from tchaikov/wip-kv-bench-cleanup

ceph_kvstorebench: include <errno.h> not asm-generic/errno.h

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #25073 from oshynsong/master

rgw: add list user admin OP API

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #25034 from smithfarm/wip-luminous-notes-12.2.10

doc: Luminous v12.2.10 release notes

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-By: Yuri Weinstein <yweinstein@redhat.com>

ceph.in: write bytes to stdout in raw_write()

in python3, sys.stdout.buffer is an io.BufferedWriter, while in python2
`sys.__stdout__` is a plain file. the former only accepts "bytes". so if
we send it a "str", it complains:

TypeError: a bytes-like object is required, not 'str'

it happens when quitting from the interactive mode of ceph CLI. in that
case, `new_style_command()` returns a tuple of `0, '\n', ''`, where the
second element is a str.

in this change, we always send `bytes` to raw_stdout.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25107 from familyuu/pool

 mgr/dashboard: Allow renaming an existing Pool

Reviewed-by: Volker Theile <vtheile@suse.com>

qa/mgr/dashboard: Fix type annotation error.

Also removed unused import

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: Allow renaming an existing Pool
fixes: http://tracker.ceph.com/issues/36560
Signed-off-by: guodan1 <guodan1@lenovo.com>

mgr/crash: group by process name in 'ceph crash json_report'

It is almost the same for daemons like osd or mon, but much better
for 'client' daemons like rgw or rbd-mirror.

Signed-off-by: Mykola Golub <mgolub@suse.com>

mgr/crash: add process name to 'ceph crash ls' output

Signed-off-by: Mykola Golub <mgolub@suse.com>

global: add process_name to crash meta

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #25251 from tchaikov/wip-bionic-dockerfile

tests: add ubuntu 18.04 dockerfile

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #25245 from wjwithagen/wjw-fix-install-flake8

install-deps.sh: Update python requirements for FreeBSD

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25146 from noahdesu/separate-tox-workdir

mgr/insights: use dedicated tox working dir

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25177 from iotcg/master

make-debs.sh: clean dir to allow building deb packages multiple times

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr: cleaner constructor for CommandResult

This is just to reduce the visual noise of all the
CommandResult("").  Nothing really uses the optional
tag part, but it's conceivably useful so let's not
rip it out just yet.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: block for latest osdmap after command execution

This is a hack, but as most of our python modules
currently use commands for osdmap manipulation,
it is a very useful one.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: add MgrModule.mon_command helper

The flexibilty of send_command and CommandResult
is needed sometimes, but lots of places just want
a simpler synchronous command to the monitor.

Signed-off-by: John Spray <john.spray@redhat.com>

tools/rados: always call rados.shutdown() before exit()

When connected to ceph cluster, if call exit(1) directly, will cause the finisher
thread segmentation fault as follows,

Caught signal (Segmentation fault) **in thread 7f45377fe700 thread_name:fn_anonymous

run 'rados mapext' without other arguments could easily get it. This patch fixes it
by calling rados.shutdown() before exit()

Fixes: http://tracker.ceph.com/issues/36732
Signed-off-by: Li Wang <laurence.liwang@gmail.com>

tools/rados: use unique_ptr for better memory management

Signed-off-by: Li Wang <laurence.liwang@gmail.com>

install-deps.sh: Update python requirements for FreeBSD

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #25223 from smithfarm/wip-rhel-8-py3

build/ops: rpm: RHEL 8 needs Python 3 build

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

qa/workunits/rados/test_librados_build.sh: grab files from explicit git branch

Fixes: https://tracker.ceph.com/issues/37392
Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph_volume_client: enable using existing rados inst

This is for mgr modules that need to use their own
self.rados instance rather than constructing
a fresh one.

Signed-off-by: John Spray <john.spray@redhat.com>

mon: give ceph-mgr access to 'fs' commands

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #25215 from majianpeng/test_shared_cache_memory_leak

test_shared_cache: fix memory leak.

mgr/dashboard: Add i18n validation script

This will check if there are new strings, validate them and ask the developer
to commit and fix those.

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge PR #24798 into master

* refs/pull/24798/head:
fix python collections module warning for v3.7 and above

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #25054 into master

* refs/pull/25054/head:
test/objectstore: enable fsck when needed.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #25247 from wjwithagen/wjw-fix-linker

do_freebsd.sh: FreeBSD building needs the llvm linker

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25194 from dillaman/wip-36650

librbd: extend API to include parent/child namespaces and image ids

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #24831 from p-na/wip-pna-cleanup-fe

mgr/dashboard: Cleanup

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge PR #24732 into master

* refs/pull/24732/head:
os/bluestore: fix missing assert->ceph_assert replacement.
os/bluestore: add bluefs perf counters to track DB/WAL/Slow volume

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #24499 into master

* refs/pull/24499/head:
common: reorder bl::iter to reduce its size from 40 to 32 on x86_64.

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #25248 into master

* refs/pull/25248/head:
mgr: fix Clang warning on superfluous brackets

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #24949 into master

* refs/pull/24949/head:
include/ceph_features: note kernel versions some features appear

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

ceph-volume: fix JSON output in `inventory`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #25225 from Aran85/sangfor-aff

mailmap,organization: Update sangfor affiliation

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25150 from majianpeng/shared_cache_race_bug

common/shared_cache: fix racing issues

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25165 from hsiang41/mgr/diskprediction_cloud_global_config

mgr: Diskprediction cloud activate when config changes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #25105 from tchaikov/wip-ceph-mutex-common

common: Mutex -> ceph::mutex

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Adam Emerson <aemerson@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

tools/ceph-dencoder: conditionally link against mds

ceph-dencoder should not link against libmds if WITH_CEPHFS is OFF.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mailmap,orgnization: Update sangfor affiliation

Signed-off-by: Zengran Zhang <zhangzengran@sangfor.com.cn>

Merge pull request #23834 from DikshaGodbole/wip-feature-24571

mgr/dashboard: Move Cluster/Audit logs from front page to dedicated Logs page

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

ceph_kvstorebench: include <errno.h> not asm-generic/errno.h

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25239 from cbodley/wip-rgw-rados-aio-cleanup

rgw: remove unused aio helper functions

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge PR #17526 into master

* refs/pull/17526/head:
qa/tasks/ceph_manager: avoid test_map_discontinuity stall with too few up osds

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

Merge PR #21444 into master

* refs/pull/21444/head:
qa: Replace 'ceph' with cluster name in restart()

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #25185 into master

* refs/pull/25185/head:
auth/AuthSessionHandler: no handler if no session key

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

tests: add ubuntu 18.04 dockerfile

ubuntu bionic is an LTS release. while 17.04 is not. so move 17.04 to
18.04

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr: fix Clang warning on superfluous brackets

```
/home/jenkins/workspace/ceph-master/src/mgr/OSDPerfMetricTypes.h:105:19: warning: braces around scalar initializer [-Wbraced-scalar-init]
        d.regex = {d.regex_str.c_str()};
                  ^~~~~~~~~~~~~~~~~~~~~
1 warning generated.
```

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

build: FreeBSD building needs the llvm linker

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

rbd: avoid usage of newly deprecated API methods

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: expose optional namespace in parent/child image output

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: expand API to include namespaces and image ids

The get parent and list children methods now return a common
struct that includes the full pool details (including the
optional namespace). Additionally, a new image list method has
been added which includes both the name and id (of v2 images).

Fixes: http://tracker.ceph.com/issues/36650
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #25206 from dillaman/wip-36622

pybind/rbd: add namespace helper API methods

Reviewed-by: Mykola Golub <mgolub@suse.com>

make-debs.sh: clean dir to avoid build deb packages error

Without this patch, it'll hit build error in the 2nd time.
Example:
make-deb.sh  #no error: first time
make-deb.sh  #hit error: second time

Signed-off-by: Changcheng Liu <changcheng.liu@intel.com>

mgr/dashboard: Added new logs component

Added new Logs component and created a cluster->logs menu item. Moved the logs from the dashboard home page to the dedicated logs page.Additional comments resolved.Dropped Logs from health component.

Fixes: https://tracker.ceph.com/issues/24571
Signed-off-by: Diksha Godbole <diksha.godbole@gmail.com>