rgw: data sync accepts ERR_PRECONDITION_FAILED on remove_object()

sync of deletes uses an If-UnModified-Since precondition, but does not
handle the corresponding ERR_PRECONDITION_FAILED error. treating this as
a failure means that we'll keep retrying the delete which will never
succeed. break this loop by treating ERR_PRECONDITION_FAILED as a
success

Fixes: http://tracker.ceph.com/issues/37448
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #25290 from noahdesu/dashboard-tox-workdir

mgr/dashboard: use dedicated tox working dir

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #25179 from tspmelo/wip-check-i18n

mgr/dashboard: Add i18n validation script

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #24900 from zmc/wip-minimal-health

mgr/dashboard: Replace dashboard service

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #25280 from tchaikov/wip-ceph-py3

ceph.in: write bytes to stdout in raw_write()

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #25261 from sebastian-philipp/mgr-handle-command-result

mgr: Add `HandleCommandResult` namedtuple

Merge pull request #25255 from tchaikov/wip-ceph-dencoder-without-cephfs

tools/ceph-dencoder: conditionally link against mds

Merge pull request #25244 from trociny/wip-crash-procname

mgr/crash: add process name to crash metadata

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24990 from dragonylffly/wip-fix-bench

tools/rados: always call rados.shutdown() before exit()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24939 from marcosps/simplify_examples

examples/librados: Remove not needed else clauses

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #24502 into master

* refs/pull/24502/head:
crushtool: implement --rebuild-class-roots command
crushtool: make --reweight re-sum choose_args weight-sets too
crushtool: --reweight should only reweight nonshadow roots
crush/CrushWrapper: reclassify: use default parent for created buckets
crush/CrushWrapper: reclassify: handle to-be-created buckets that we need twice
test/cli/crushtool/reclassify: add second gabe test case
crushtool: add --set-subtree-class; do not set class via --reclassify-root
test/cli/crushtool/reclassify: add reclassify test cases
doc/rados/operations/crush*: document reclassify
doc/rados/operations/crush: remove instructions for separate crush trees for ssd
crushtool: add --compare command
crushtool: implement --reclassify
crush/CrushCompiler: fix id scan to include class ids

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

crushtool: implement --rebuild-class-roots command

This simply rebuilds the class roots.  Normally this should create no
change in the map since whatever was making changes to the map before
should have rebuild the shadow roots at that point.

Signed-off-by: Sage Weil <sage@redhat.com>

crushtool: make --reweight re-sum choose_args weight-sets too

This ensures that the weights add us for each weight-set (and each
position).  Note that since we don't have anything that actually
creates positional weight-sets, the behavior here might not be what we
want in the end, but for the compat weight-sets (no position), we *do*
keep the weights as a properly summing tree.

Signed-off-by: Sage Weil <sage@redhat.com>

crushtool: --reweight should only reweight nonshadow roots

The shadow roots will be then be rebuilt accordingly.

Signed-off-by: Sage Weil <sage@redhat.com>

crush/CrushWrapper: reclassify: use default parent for created buckets

Note that these last 2 fixes also fix the 'h' test case.  \o/

Signed-off-by: Sage Weil <sage@redhat.com>

crush/CrushWrapper: reclassify: handle to-be-created buckets that we need twice

Signed-off-by: Sage Weil <sage@redhat.com>

test/cli/crushtool/reclassify: add second gabe test case

New adjusted test map from Dan.

Signed-off-by: Sage Weil <sage@redhat.com>

crushtool: add --set-subtree-class; do not set class via --reclassify-root

Sometimes we don't want the --reclassify-root to set the class of every
device because a small number of them are (correctly) a different class.
Allow both behaviors by adding a new, separate command to set the class
of all devices beneath a point in the hierarchy and do not implicitly do
that relabeling as part of --reclassify-root.

Signed-off-by: Sage Weil <sage@redhat.com>

test/cli/crushtool/reclassify: add reclassify test cases

Signed-off-by: Sage Weil <sage@redhat.com>

doc/rados/operations/crush*: document reclassify

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Replace dashboard service

This splits out the collection of health and log data from the
/api/dashboard/health controller into /api/health/{full,minimal} and
/api/logs/all.

/health/full contains all the data (minus logs) that /dashboard/health
did, whereas /health/minimal contains only what is needed for the health
component to function. /logs/all contains exactly what the logs portion
of /dashboard/health did.

By using /health/minimal, on a vstart cluster we pull ~1.4KB of data
every 5s, where we used to pull ~6KB; those numbers would get larger
with larger clusters. Once we split out log data, that will drop to
~0.4KB.

Fixes: http://tracker.ceph.com/issues/36675
Signed-off-by: Zack Cerza <zack@redhat.com>

Merge PR #24923 into master

* refs/pull/24923/head:
mds: properly update freeze_tree_state when fragmenting dirfrags

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/dashboard: use dedicated tox working dir

Avoids conflicting with other test environments.

Fixes: https://tracker.ceph.com/issues/37405
Signed-off-by: Noah Watkins <nwatkins@redhat.com>

Merge PR #24849 into master

* refs/pull/24849/head:
client: remove redundant abort logic during dentry invalidation test

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #24886 into master

* refs/pull/24886/head:
qa: fix delay type config name

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #24839 into master

* refs/pull/24839/head:
ceph-volume-client: allow setting mode of CephFS volumes

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Allow specifying a schema for unknown JObj keys

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge PR #25020 into master

* refs/pull/25020/head:
script/ceph-backport.sh: adjust to change in github API

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #25235 from sebastian-philipp/dashboard-fix-type-annotation-error

qa/mgr/dashboard: Fix type annotation error.

Reviewed-by: Tatjana Dehler <tdehler@suse.com>

mgr: Add `HandleCommandResult` namedtuple

Mostly a documentation improvement to make it obvious
what `handle_command` is supposed to return .

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #25268 from smithfarm/wip-37392

qa/workunits/rados/test_librados_build.sh: grab files from explicit git branch

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25129 from rzarzynski/wip-common-annotate

common: avoid the overhead of ANNOTATE_HAPPENS_* in NDEBUG builds.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25224 from sebastian-philipp/ceph-volume-fix-json

ceph-volume: fix JSON output in `inventory`

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #25256 from tchaikov/wip-kv-bench-cleanup

ceph_kvstorebench: include <errno.h> not asm-generic/errno.h

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #25073 from oshynsong/master

rgw: add list user admin OP API

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #25034 from smithfarm/wip-luminous-notes-12.2.10

doc: Luminous v12.2.10 release notes

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-By: Yuri Weinstein <yweinstein@redhat.com>

ceph.in: write bytes to stdout in raw_write()

in python3, sys.stdout.buffer is an io.BufferedWriter, while in python2
`sys.__stdout__` is a plain file. the former only accepts "bytes". so if
we send it a "str", it complains:

TypeError: a bytes-like object is required, not 'str'

it happens when quitting from the interactive mode of ceph CLI. in that
case, `new_style_command()` returns a tuple of `0, '\n', ''`, where the
second element is a str.

in this change, we always send `bytes` to raw_stdout.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25107 from familyuu/pool

 mgr/dashboard: Allow renaming an existing Pool

Reviewed-by: Volker Theile <vtheile@suse.com>

qa/mgr/dashboard: Fix type annotation error.

Also removed unused import

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: Allow renaming an existing Pool
fixes: http://tracker.ceph.com/issues/36560
Signed-off-by: guodan1 <guodan1@lenovo.com>

mgr/crash: group by process name in 'ceph crash json_report'

It is almost the same for daemons like osd or mon, but much better
for 'client' daemons like rgw or rbd-mirror.

Signed-off-by: Mykola Golub <mgolub@suse.com>

mgr/crash: add process name to 'ceph crash ls' output

Signed-off-by: Mykola Golub <mgolub@suse.com>

global: add process_name to crash meta

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #25251 from tchaikov/wip-bionic-dockerfile

tests: add ubuntu 18.04 dockerfile

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #25245 from wjwithagen/wjw-fix-install-flake8

install-deps.sh: Update python requirements for FreeBSD

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25146 from noahdesu/separate-tox-workdir

mgr/insights: use dedicated tox working dir

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25177 from iotcg/master

make-debs.sh: clean dir to allow building deb packages multiple times

Reviewed-by: Kefu Chai <kchai@redhat.com>

tools/rados: always call rados.shutdown() before exit()

When connected to ceph cluster, if call exit(1) directly, will cause the finisher
thread segmentation fault as follows,

Caught signal (Segmentation fault) **in thread 7f45377fe700 thread_name:fn_anonymous

run 'rados mapext' without other arguments could easily get it. This patch fixes it
by calling rados.shutdown() before exit()

Fixes: http://tracker.ceph.com/issues/36732
Signed-off-by: Li Wang <laurence.liwang@gmail.com>

tools/rados: use unique_ptr for better memory management

Signed-off-by: Li Wang <laurence.liwang@gmail.com>

install-deps.sh: Update python requirements for FreeBSD

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #25223 from smithfarm/wip-rhel-8-py3

build/ops: rpm: RHEL 8 needs Python 3 build

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

qa/workunits/rados/test_librados_build.sh: grab files from explicit git branch

Fixes: https://tracker.ceph.com/issues/37392
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #25215 from majianpeng/test_shared_cache_memory_leak

test_shared_cache: fix memory leak.

mgr/dashboard: Add i18n validation script

This will check if there are new strings, validate them and ask the developer
to commit and fix those.

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge PR #24798 into master

* refs/pull/24798/head:
fix python collections module warning for v3.7 and above

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #25054 into master

* refs/pull/25054/head:
test/objectstore: enable fsck when needed.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #25247 from wjwithagen/wjw-fix-linker

do_freebsd.sh: FreeBSD building needs the llvm linker

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25194 from dillaman/wip-36650

librbd: extend API to include parent/child namespaces and image ids

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #24831 from p-na/wip-pna-cleanup-fe

mgr/dashboard: Cleanup

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge PR #24732 into master

* refs/pull/24732/head:
os/bluestore: fix missing assert->ceph_assert replacement.
os/bluestore: add bluefs perf counters to track DB/WAL/Slow volume

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #24499 into master

* refs/pull/24499/head:
common: reorder bl::iter to reduce its size from 40 to 32 on x86_64.

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #25248 into master

* refs/pull/25248/head:
mgr: fix Clang warning on superfluous brackets

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #24949 into master

* refs/pull/24949/head:
include/ceph_features: note kernel versions some features appear

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

ceph-volume: fix JSON output in `inventory`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #25225 from Aran85/sangfor-aff

mailmap,organization: Update sangfor affiliation

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25150 from majianpeng/shared_cache_race_bug

common/shared_cache: fix racing issues

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25165 from hsiang41/mgr/diskprediction_cloud_global_config

mgr: Diskprediction cloud activate when config changes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #25105 from tchaikov/wip-ceph-mutex-common

common: Mutex -> ceph::mutex

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Adam Emerson <aemerson@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

tools/ceph-dencoder: conditionally link against mds

ceph-dencoder should not link against libmds if WITH_CEPHFS is OFF.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mailmap,orgnization: Update sangfor affiliation

Signed-off-by: Zengran Zhang <zhangzengran@sangfor.com.cn>

Merge pull request #23834 from DikshaGodbole/wip-feature-24571

mgr/dashboard: Move Cluster/Audit logs from front page to dedicated Logs page

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

ceph_kvstorebench: include <errno.h> not asm-generic/errno.h

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25239 from cbodley/wip-rgw-rados-aio-cleanup

rgw: remove unused aio helper functions

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge PR #17526 into master

* refs/pull/17526/head:
qa/tasks/ceph_manager: avoid test_map_discontinuity stall with too few up osds

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

Merge PR #21444 into master

* refs/pull/21444/head:
qa: Replace 'ceph' with cluster name in restart()

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #25185 into master

* refs/pull/25185/head:
auth/AuthSessionHandler: no handler if no session key

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

tests: add ubuntu 18.04 dockerfile

ubuntu bionic is an LTS release. while 17.04 is not. so move 17.04 to
18.04

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr: fix Clang warning on superfluous brackets

```
/home/jenkins/workspace/ceph-master/src/mgr/OSDPerfMetricTypes.h:105:19: warning: braces around scalar initializer [-Wbraced-scalar-init]
        d.regex = {d.regex_str.c_str()};
                  ^~~~~~~~~~~~~~~~~~~~~
1 warning generated.
```

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

build: FreeBSD building needs the llvm linker

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

rbd: avoid usage of newly deprecated API methods

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: expose optional namespace in parent/child image output

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: expand API to include namespaces and image ids

The get parent and list children methods now return a common
struct that includes the full pool details (including the
optional namespace). Additionally, a new image list method has
been added which includes both the name and id (of v2 images).

Fixes: http://tracker.ceph.com/issues/36650
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #25206 from dillaman/wip-36622

pybind/rbd: add namespace helper API methods

Reviewed-by: Mykola Golub <mgolub@suse.com>

make-debs.sh: clean dir to avoid build deb packages error

Without this patch, it'll hit build error in the 2nd time.
Example:
make-deb.sh  #no error: first time
make-deb.sh  #hit error: second time

Signed-off-by: Changcheng Liu <changcheng.liu@intel.com>

mgr/dashboard: Added new logs component

Added new Logs component and created a cluster->logs menu item. Moved the logs from the dashboard home page to the dedicated logs page.Additional comments resolved.Dropped Logs from health component.

Fixes: https://tracker.ceph.com/issues/24571
Signed-off-by: Diksha Godbole <diksha.godbole@gmail.com>

Merge pull request #25221 from tspmelo/wip-37354

mgr/dashboard: Fix test_remove_not_expired_trash qa test

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #24757 from votdev/issue_36480

mgr/dashboard: Can't handle user editing when tenants are specified

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr: Diskprediction_cloud action when config changes

Trigger diskprediction cloud action when receive notify configuration changed.

Signed-off-by: Rick Chen <rick.chen@prophetstor.com>

Merge pull request #25186 from tchaikov/wip-vive-le-make-install

cmake: do not pass unnecessary param to setup.py

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #25197 from iridescent-rsy/fix-typo

rbd-mirror,common: fix typos in logging messages and comments

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #25202 from trociny/wip-doc-rbd-qos

doc: describe RBD QoS settings

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Patrick Nawracay <pnawracay@suse.com>

pybind/rbd: fix python3 runtime compatibility issues

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

build/ops: rpm: RHEL 8 needs Python 3 build

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #25182 from jan--f/mgr-prometheus-doc-mention-instance-label

doc/mgr/prometheus: readd section about custom instance labels

Merge pull request #25119 from sebastian-philipp/doc-orchestrator-cli-spec

doc/orchestrator: Integrate CLI specification into the documentation

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

mgr/dashboard: Fix test_remove_not_expired_trash qa test

Fixes: http://tracker.ceph.com/issues/37354
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #25144 from rhcs-dashboard/wip-37295-master

mgr/dashboard: Show pool dropdown for block-mgr

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #25159 from tspmelo/wip-i18n-doc

mgr/dashboard: Update I18N documentation

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #25181 from smithfarm/wip-37301

build/ops: rpm: stop install-deps.sh clobbering spec file Python build setting

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Tim Serong <tserong@suse.com>

Merge pull request #25205 from smithfarm/wip-37330

build/ops: do_cmake.sh: automate py3 build options for certain distros

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>