ceph-volume: add db and wal support to raw mode

Using the raw mode, the OSD preparation can now be called with extra
arguments to specify a block device for either rocksdb db or wal.

The code expects the device to be available and no validation will be
performed. Also, the entire block device will be consumed.

The goal is run this on Kubernetes with Rook and OSD are running on PVC.
Users will request PVC from a certain size, the request will be
acknowledged and a block will be created, then attached to the machine.
Later on consumed by Rook's preparation job.

Signed-off-by: Sébastien Han <seb@redhat.com>

ceph-volume: fix raw list

There is no json anymore, just plain output.

Signed-off-by: Sébastien Han <seb@redhat.com>

ceph-volume: remove unnecessary comment

This comment is not relevant since we will never bring systemd support
to the raw mode.

Signed-off-by: Sébastien Han <seb@redhat.com>

ceph-volume: remove osd-id and osd-fsid flag

These are not used since we only rely on the device path so removing
them to avid confusion.

Signed-off-by: Sébastien Han <seb@redhat.com>

ceph-volume: remove "create" call from raw mode

This mode is intented to work when the cluster is running on Kubernetes
and OSDs are running on PVC.
So we always have a "prepare" pod and an "activate" pod.

Signed-off-by: Sébastien Han <seb@redhat.com>

Merge PR #28603 into master

* refs/pull/28603/head:
mgr: Add get_rates_from_data to mgr_util.py

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #32788 into master

* refs/pull/32788/head:
qa/tasks/mgr/dashboard: set pg_num to 32
mgr/pg_autoscaler: default to pg_num[_min] = 32

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/dashboard: check if user has config-opt permissions (#32638)

mgr/dashboard: check if user has config-opt permissions

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Improve the size of forms (#31863)

mgr/dashboard: Improve the size of forms

Reviewed-by: Sebastian Krah <skrah@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #32802 from uweigand/cmake-atomic-check

cmake: Improve test for 16-byte atomic support on IBM Z

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

mgr/dashboard: Unit test does not do the mentioned thing (#32799)

mgr/dashboard: Unit test does not do the mentioned thing

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #32798 from rhcs-dashboard/fix-make-check-deps-centos8

ceph.spec.in: fix 'make check' deps for centos8

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #32478 into master

* refs/pull/32478/head:
qa: update xfstests version
qa: install some dependencies for xfstests

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge PR #32754 into master

* refs/pull/32754/head:
mgr/cephadm: health alert for stray hosts and/or services

Reviewed-by: Michael Fritch <mfritch@suse.com>

qa: update xfstests version

Building xfstests on CentOS 8 has build errors:

    2020-01-23T21:15:14.939 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/xfstests/src/fssum.c:616: undefined reference to `major'
    2020-01-23T21:15:14.939 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/xfstests/src/fssum.c:617: undefined reference to `minor'
    2020-01-23T21:15:14.940 INFO:tasks.workunit.client.0.smithi155.stderr:collect2: error: ld returned 1 exit status

From: /ceph/teuthology-archive/pdonnell-2020-01-23_20:47:47-fs-wip-pdonnell-testing-20200122.210533-distro-basic-smithi/4697257/teuthology.log

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: install some dependencies for xfstests

These are needed for CentOS 8 / RHEL8.

Also organize the YAML to use the cleaner array syntax.

Fixes: https://tracker.ceph.com/issues/43459
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32350 into master

* refs/pull/32350/head:
mds: Reorganize class members in SnapServer header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32722 into master

* refs/pull/32722/head:
mds: fix null pointer dereference in Server::handle_client_link()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32616 into master

* refs/pull/32616/head:
cephadm: annotate call()
cephadm: podman inspect: image field was called `ImageID`:

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32795 into master

* refs/pull/32795/head:
cephadm: require fsid if name contains a daemon id
cephadm: infer the fsid by name

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #31155 from zhangsw/fix-rgw-reshard-process

rgw: one log shard fails shouldn't block other shards process when reshard buckets

Reviewed-by: Mark Kogan <mkogan@redhat.com>

Merge pull request #31103 from zhangsw/fix-rgw-reshard-cleanindex

rgw: clean index and remove bucket instance info when setting resharding status fails

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30920 from GaryHyg/wip-quota-bug

RGW: Set appropriate bucket quota value (when quota value is less than 0)

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30880 from jiahuizeng/radosgw

rgw: support radosgw-admin zone/zonegroup placement get command

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30861 from IlsooByun/rgw_lc_missing

rgw: do not miss the 1000th element of every iteration during lifecycle processing

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #32797 from tchaikov/wip-seastar-and-errorator

seastar,crimson: pickup change to pin socket to fixed core

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #29703 from baixueyu/wip-quota

rgw:  when you abort a multipart upload request, the quota may be not updated

Merge pull request #28622 from IlsooByun/master

rgw: incorrect return value when processing CORS headers

Merge pull request #29733 from baixueyu/wip-reload-coredump

rgw: maybe coredump when reload operator happened

Merge pull request #29795 from xxcs/xxcs03

rgw: fix one part of the bulk delete(RGWDeleteMultiObj_ObjStore_S3) fails but no error messages

Merge pull request #30913 from ofriedma/wip-ofriedma-lc-logs

rgw: Add more details to the LC delete and transit log

Merge PR #32787 into master

* refs/pull/32787/head:
qa/tasks/cephadm: work around .git suffix on ceph_repo
qa/tasks/cephadm: learn to pull cephadm from github

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

mgr/cephadm: health alert for stray hosts and/or services

Fixes: https://tracker.ceph.com/issues/43714
Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Add column filtering feature to cd-table (#32568)

mgr/dashboard: Add column filtering feature to cd-table

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

cephadm: require fsid if name contains a daemon id

$ cephadm shell --name mon.123
INFO:cephadm:Found fsids []
Traceback (most recent call last):
  File "./cephadm", line 2805, in <module>
    r = args.func()
  File "./cephadm", line 646, in _infer_fsid
    return func()
  File "./cephadm", line 1992, in command_shell
    no_config=True if args.config else False)
  File "./cephadm", line 1009, in get_container_mounts
    data_dir = get_data_dir(fsid, daemon_type, daemon_id)
  File "./cephadm", line 670, in get_data_dir
    return os.path.join(args.data_dir, fsid, '%s.%s' % (t, n))
  File "/usr/lib64/python3.7/posixpath.py", line 94, in join
    genericpath._check_arg_types('join', a, *p)
  File "/usr/lib64/python3.7/genericpath.py", line 149, in _check_arg_types
    (funcname, s.__class__.__name__)) from None
TypeError: join() argument must be str or bytes, not 'NoneType'

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: infer the fsid by name

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge pull request #32387 from athanatos/sjust/wip-glibcxx-assertions

Enable _GLIBCXX_ASSERTIONS and fix unittest problems

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/tasks/cephadm: work around .git suffix on ceph_repo

githubusercontent.com doesn't like the .git suffix

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32794 into master

* refs/pull/32794/head:
cephadm: normalize image_id in 'pull' output too
cephadm: fix image_id normalization
cephadm: pull: remove newline from ceph_version
mgr/cephadm: base upgrade check on return code, not stderr
mgr/cephadm: fix ok-to-stop
mgr/cephadm: some upgrade debug output
mgr/cephadm: remove dup check
mgr/cephadm: only need_upgrade_self if image_id doesn't match
mgr/cephadm: safe to restart mgr and other non-mon/osd/mds daemons

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #32791 from liewegas/fix-rgw-website

qa/suites/rgw/website: run test on ubuntu

Merge pull request #32276 from smithfarm/wip-drop-repo-access-doc

doc/cleanup: drop repo-access.rst

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #32269 from liuchang0812/issue-43199-lockdep-in-coroutines-manager

rgw: get elastic search info in start_sync, avoid creating new coroutines manager

Merge pull request #32290 from jiahuizeng/es-valid

rgw: relax es zone validity check

Merge pull request #32458 from ofriedma/wip-ofriedma-fix-trans-noncurr

cls/rgw: when object is versioned and lc transition it, the object is becoming non-current

Merge pull request #32180 from yehudasa/wip-43255

rgw: use bucket creation time from bucket instance info

cmake: Improve test for 16-byte atomic support on IBM Z

Commit d1b9d14324586d02dc6c8dd0a9bdf4b98ae16416 added a check for
16-byte atomics without library support on IBM Z.  Unfortunately
it turns out this test only works correctly when the test case
is built without optimization.  (This is normally true, but it
may not be the case when passing explicit CXXFLAGS to cmake.)

The underlying reason is that GCC may choose to use either an
inline implementation of the 16-byte atomics or library calls,
depending on whether or not it is able to prove the atomic
variable is properly aligned.  At -O0 it is never able to prove
that, but at higher optimization levels it depends on the complexity
of the expression (in particular, whether GCC can track down the
definition of the underlying object).

As the test case uses a very simple expression, it may happen that
this test can be built without requiring library support, but some
of the "real" uses of atomics in Ceph code cannot.   This defeats
the whole purpose of the test at configure time.

Fixed by making the access pattern in the test more complex, so
that the test fails even at high optimization levels.

Fixes: https://tracker.ceph.com/issues/43747
Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>

mgr/dashboard: Add required symbol directly in the text element

This prevents the symbol from breaking from the last word of the label,
if the width gets too small.

This will also remove the need to add an extra span just to show the symbol.

Fixes: https://tracker.ceph.com/issues/40751
Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Use custom classes for form label and inputs

With this change we are able to configura the col sizes of labels and inputs in
one single place.

Also increased the col size in smaller screens to be able to display longer
labels.

This was not applied in some situations where the col size was different than
usual.

Fixes: https://tracker.ceph.com/issues/40751
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #32736 from capri1989/credits-script

src/script/credits.sh - switch to bash

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

ceph.spec.in: fix 'make check' deps for centos8

When running 'FOR_MAKE_CHECK=1 ./install-deps.sh' in CentOS 8
these dependencies were not being installed.
Missing dependencies are provided by
https://copr.fedorainfracloud.org/coprs/ktdreyer/ceph-el8/

Signed-off-by: Alfonso Martínez <almartin@redhat.com>

mgr/dashboard: Unit test does not do the mentioned thing

The unit test AccessControlTest::test_load_v2() is testing a v1 database instead of a v2.

Fixes: https://tracker.ceph.com/issues/43760
Signed-off-by: Volker Theile <vtheile@suse.com>

mgr: Add get_rates_from_data to mgr_util.py

The function calculates the difference between the values
in a time series list.

The previous location of it was in the dashboard, that's why the
function has doc tests. In order to be able to test them as part of the
manager utilities a bit of tox magic was added.

Fixes: https://tracker.ceph.com/issues/40365
Signed-off-by: Stephan Müller <smueller@suse.com>

crimson/common/errorator: construct future<> with a future_state_base

this change is made to adapt
https://github.com/scylladb/seastar/commit/89e801b373423b068b3a70e50b656348f549ade0

* always use the helpers from `futurize` instead of using
  `::seastar::make_exception_future()` directly. as
  `make_exception_future(future_state_base&& state)` is located in
  `seastar::internal` namespace
* add `noexcept` specifier in both constructors. to match with the
  helpers of `make_exception_future2()` which dispatch the call to
  different constructor variants.

Signed-off-by: Kefu Chai <kchai@redhat.com>

seastar: pickup change to pin socket to fixed core

and clang / GCC-10 related fixes

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32373 from zdover23/wip-doc-pg-formats-update-dec-2019

doc: update list of formats for --format flag for "ceph pg dump"

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #32790 from neha-ojha/wip-pglog-defaults

doc/rados: update osd_min_pg_log_entries and add osd_max_pg_log_entries

Reviewed-by: Sage Weil <sage@redhat.com>

rgw_kms: fix concat_url for empty url case

Can't call back() on an empty string.  Causes unittest_rgw_kms
to fail when compiled with _GLIBCXX_ASSERTIONS.

Signed-off-by: Samuel Just <sjust@redhat.com>

fastbmap_allocator_impl: avoid std::vector[] for oob index

This was relatively harmless as the pointer won't actually be
dereferenced, but it runs afoul of std::vector's bounds checks with
_GLIBCXX_ASSERTIONS enabled.

Signed-off-by: Samuel Just <sjust@redhat.com>

unittest_random: min > max triggers a glibc assert

Signed-off-by: Samuel Just <sjust@redhat.com>

src/CMakeLists.txt: enable _GLIBCXX_ASSERTIONS in debug builds

We already enable this by default in our rpm builds.  Adding it here as
well increases the odds we'll catch things with make check and local
testing.

Signed-off-by: Samuel Just <sjust@redhat.com>

cephadm: normalize image_id in 'pull' output too

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: fix image_id normalization

Remove sha256: prefix from id (hash), not name.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32741 into master

* refs/pull/32741/head:
qa/cephfs: make filelock_interrupt.py work with python3

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

cephadm: pull: remove newline from ceph_version

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: base upgrade check on return code, not stderr

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: fix ok-to-stop

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: some upgrade debug output

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: remove dup check

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: only need_upgrade_self if image_id doesn't match

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: safe to restart mgr and other non-mon/osd/mds daemons

We don't have "ok-to-stop" checks for these.  And the mgr check for
self/active is already in the caller.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rgw/website: run test on ubuntu

The dnsmasq package on centos 8.0 is broken, see
https://tracker.ceph.com/issues/43744

For now, run this test on ubuntu.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #30152 from theanalyst/rgw/iam-policy-bitset-fn

rgw: iam: use a function to calculate the Action Bit string

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #29851 from linuxbox2/wip-rgw-linkvtrack

rgw: fix version tracking across bucket link steps

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30540 from mikulely/uri-beast-cleanup

rgw: fix REQUEST_URI setting in the rgw_asio_client.cc

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30539 from mikulely/fix-operation2

rgw: fix opslog operation field as per Amazon s3

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30471 from liuchang0812/delete-bucket-with-system-user

rgw: the http response code of delete bucket should not be 204-no-content

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc/rados/configuration: document osd_max_pg_log_entries

Signed-off-by: Neha Ojha <nojha@redhat.com>

doc/rados: update osd_min_pg_log_entries

update docs to match 0db140c15c16ce45de2e59f075de2b70c0a9cd95

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/tasks/mgr/dashboard: set pg_num to 32

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge PR #32644 into master

* refs/pull/32644/head:
qa: ignore trimmed cache items for dead cache drop
qa: use unit test comparisons

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #32772 from tchaikov/wip-43403

test/unittest_lockdep: do not start extra threads

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #31356 from xxcs/xxcs06

rgw: Incorrectly calling ceph::buffer::list::decode_base64 in bucket policy

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #31355 from zhangsw/fix-rgw-reshard-max-entries

rgw: reshard list may return more than specified max_entries.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #31344 from zhangsw/fix-rgw-reshard-lock-failed

rgw: return error if lock log shard fails

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #31239 from joke-lee/rgw-modify-zone-placement

rgw: radosgw-admin zone placement rm and radosgw-admin zonegroup placement rm support --storage-class

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #31591 from liuchang0812/current-version-in-rgw-user-info

rgw: fix RGWUserInfo decode current version

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

mgr/pg_autoscaler: default to pg_num[_min] = 32

78bf92448002eece7501da01b67f900a84207e70 increased the default to 16.
Increasing it further to 32 will provide enough parallelism to improve
out of the box performance for new users.

Fixes: https://tracker.ceph.com/issues/43757
Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/tasks/cephadm: learn to pull cephadm from github

Fixes: https://tracker.ceph.com/issues/43755
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #30636 into master

* refs/pull/30636/head:
common: Fix multiple logical errors in get_device_id.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #32732 into master

* refs/pull/32732/head:
msg/async: use dedicated method for clearing conn's priv up.
msg/async: mark down local_connection before draining the stack.

Reviewed-by: Kefu Chai <kchai@redhat.com>

googletest: pick up change to print GTEST_SKIP messages

to include the fix for
https://github.com/google/googletest/pull/2517

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/unittest_lockdep: do not start extra threads

also, stop linking against libglobal

Fixes: https://tracker.ceph.com/issues/43403
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #32726 into master

* refs/pull/32726/head:
mon/ConfigMonitor: clean out bad config records
mon/ConfigMonitor: make 'config get' on fsid work
common/options: remove NO_MON_UPDATE from mon_dns_srv_name
mon/ConfigMonitor: do not 'config get' on NO_MON_UPDATE options
mon/ConfigMonitor: do not set NO_MON_UPDATE values

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #32699 from tspmelo/wip-npm-audit-fix

mgr/dashboard: Fix npm vulnerabilities

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #32653 from votdev/issue_43603_clone_role

mgr/dashboard: Clone an existing user role

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #32648 from p-se/wip-pse-cd-api-err-msg

mgr/dashboard: monitoring: improve generic "Could not reach external API" message

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge PR #32765 into master

* refs/pull/32765/head:
unittest_lockdep: avoid any threads for death test

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #32728 from rishabh-d-dave/doc-teuth-broken-links

doc: fix links in developer_guide

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32703 from s0nea/wip-dashboard-pwd-expiration-field

mgr/dashboard: fix visibility of pwdExpirationDate field

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #32683 from liewegas/wip-variable-pg-log

osd: trim pg logs based on a per-osd budget

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>