rgw: make radosgw-admin user create and modify distinct

Currently if you run 'radosgw-admin user create ..' when the user
already exists and you happen to specify, at least,  '--uid' and
'--display-name' that match the existing user, radowgw-admin will
actaully go modify the existing user.

This behaviour is a little confusing, hence the bug this patch is
fixing. This patch instead simplifies the tool to make
'create' create and 'modify' modify.
Meaning when you go 'create' a user that already exists, you'll get an
error, as expected. If you want to modify a user, you actually have to
use 'modify'.

For exapmle, now:

  $ radosgw-admin user create --uid="test-user"  --display-name="test user"
  could not create user: unable to create user, user: test-user exists

Signed-off-by: Matthew Oliver <moliver@suse.com>
Fixes: https://tracker.ceph.com/issues/38619

Merge PR #30802 into master

* refs/pull/30802/head:
cephfs-shell: Modify Quota Test
cephfs-shell: Fix permission errors on quota tests
cephfs-shell: Add tests for quota
cephfs-shell: Print max_bytes and max_files as string on get Quota
cephfs-shell: Remove extra length argument passed to setxattr()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #32347 into master

* refs/pull/32347/head:
cephfs-shell: set shell.exit_code to 1 when reading conf fails
cephfs-shell: cephfs-shell breaks when conf_get() returns None
cephfs-shell: use libcephfs for exceptions
cephfs-shell: fix incorrect use of str.strip

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32459 into master

* refs/pull/32459/head:
qa/tasks: Nothing to clean up if the volume was not created
mgr/volumes: drop obsolete comment in _cmd_fs_volume_create
mgr/volumes: cleanup on fs create error

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32318 into master

* refs/pull/32318/head:
mds: don't add metadata to session close message

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31250 into master

* refs/pull/31250/head:
mds: Reorganize class members in MDSTableServer header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31481 into master

* refs/pull/31481/head:
mds: Reorganize struct members in Mutation header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32452 into master

* refs/pull/32452/head:
cephfs-journal-tool: fix the usage
cephfs-journal-tool: fix crash with empty argv

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32510 into master

* refs/pull/32510/head:
mon: use non-obsolete mon scrub cmd

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32247 from dzafman/wip-43307

Remove use of rules batching for upmap balancer and default for upmap_max_deviation to 5

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #32412 from tchaikov/wip-spec-more-deps-for-el8

ceph.spec.in: re-enable "make check" deps for el8

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #32483 from trociny/wip-pool_replayer-print_status

rbd-mirror: fix "pool replayer status" for case when init failed

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #32463 from trociny/wip-unlink-peer

librbd: when unlinking peer from mirror snaps do it in all namespaces

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #32456 from trociny/wip-43428

rbd-mirror: make logrotate work

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #32404 from kalebskeithley/master

rgw: build radosgw daemon as a shared lib + small executable

Merge pull request #32494 from ronen-fr/clang_3

crimson: fix lambda captures of non-variables

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32499 from sebastian-philipp/orchestrator-cli-mon-update-count-only

mgr/orchestrator_cli: _update_mons require host spec only

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson: fix lambda captures of non-variables

One cannot just capture a structured binding "non-variable".
(From the C++ standard, $8.4.5.2:
"If a lambda-expression [...] captures a structured binding (explicitly or implicitly),
the program is ill-formed.")

Co-authored-by: Kefu Chai <kchai@redhat.com>
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #32376 from idryomov/wip-krbd-fsx-object-map

qa/suites/krbd: fsx with object-map and fast-diff

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #32518 from tchaikov/wip-orchestrator-service-action-tests

qa/tasks/mgr/test_orchestrator_cli: fix service action tests

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

mgr/rook: Add some more type annotations

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator_cli: _update_mons require host spec only

* `mgr/cephadm` requres a host spec right now
* `mgr/rook` only supports `spec.count` right now

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #32496 from majianpeng/osd-remove-unused-func2

osd/OSD: remove unused func enqueue_peering_evt_front

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32484 from bzed/bluefs-use-uint64_t-for-len

os/bluestore/Blue(FS|Store): uint64_t alloc_size

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32473 from sseshasa/fix-mon-autotune-update

mon/OSDMonitor: Don't update mon cache settings if rocksdb is not used

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #28921 from rzarzynski/wip-msgr-drop-0c

msg/async: drop zero_copy_read() & co from ConnectedSocket.

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32112 from hjwsm1989/wip-42706

test/librados: don't release handler if set_pg_num failed

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32428 from tchaikov/wip-qa-mgr-mon-update

qa/tasks/mgr/test_orchestrator_cli: fix "mon update" test

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #31929 from majianpeng/msg-async-bugfix

msg/async: update refcount and perf counter properly

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/tasks/mgr/test_orchestrator_cli: fix service action tests

in b77f0c74a5213ee57de19d21c930b2f3c872a1c4, "reload"
service[-instance] action was dropped. so replace "reload" with
"restart" in the related tests.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test: Sort pool list because the order isn't guaranteed from "balancer pool ls"

Signed-off-by: David Zafman <dzafman@redhat.com>

mgr: Change default upmap_max_deviation to 5

Fixes: https://tracker.ceph.com/issues/43312
Signed-off-by: David Zafman <dzafman@redhat.com>

osdmaptool: Add --upmap-active to simulate active upmap balancing

Signed-off-by: David Zafman <dzafman@redhat.com>

doc: Add upmap options to osdmaptool man page and give example

Signed-off-by: David Zafman <dzafman@redhat.com>

tools: osdmaptool document non-upmap options that were missing

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Fix test case for pool based balancing instead of rule batched

Signed-off-by: David Zafman <dzafman@redhat.com>

Revert "tools: osdmaptool sync with balancer module behavior"

This was the rules batching version of osdmaptool

This reverts commit f165d4ca39b1edef4bf22a7597af79339c95026f.

Signed-off-by: David Zafman <dzafman@redhat.com>

Partially revert "mgr/balancer: balance pools with same crush_rule in batch"

This partially reverts commit 3a730d751deff892e7a0bddba87eba3dbb829c3e

Fixes: https://tracker.ceph.com/issues/43307
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #32358 from idryomov/wip-krbd-py3-fixup

qa: krbd_exclusive_option.sh: fixup for json.tool ordering change

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #32498 into master

* refs/pull/32498/head:
doc: mounting CephFS subdirectory and Persistent Mounts cleanup

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32480 into master

* refs/pull/32480/head:
qa: use -D_GNU_SOURCE when compiling fsync-tester.c
qa: add comment about fsync-tester distribution

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #29995 into master

* refs/pull/29995/head:
mds: add config to require forward to auth MDS

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31494 into master

* refs/pull/31494/head:
qa: force creation of fs with EC default data pool
qa: add tests for adding EC data pools
mon/MDSMonitor: warn when creating fs with default EC data pool

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32015 into master

* refs/pull/32015/head:
doc/cephfs/standby: document mds_join_fs
mgr/cephadm: map mds daemons to a particular fs
mon/MDSMonitor: respect mfs fscid preference
mon/MDSMonitor: assign standbys to their preferred fscid
mds/FSMap: track preferred fscid for standby daemons
mds: add mds_join_fs option; pass via MMDSBeacon

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32268 into master

* refs/pull/32268/head:
mds: only enable batch getattr/lookup for non-snap request.
mds: pin dentry when attaching batch lookup to it

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32307 into master

* refs/pull/32307/head:
mon: print FSMap if standbys exist

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32308 into master

* refs/pull/32308/head:
mon/MDSMonitor: cleanup check_subs

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #32467 into master

* refs/pull/32467/head:
qa: fix various py3 cephfs qa bugs

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #32164 into master

* refs/pull/32164/head:
mount.ceph: give a hint message when no mds is up or cluster is laggy

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #32363 into master

* refs/pull/32363/head:
qa: add .qa link

Reviewed-by: Sage Weil <sage@redhat.com>

mon: use non-obsolete mon scrub cmd

Fixes: e9a5ce0897efc6126caeebea9900bf05ec3d2174
Fixes: https://tracker.ceph.com/issues/43488
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #32426 from tchaikov/wip-qa-test_envlibrados_for_rocksdb.sh

qa: enable CRB repo for RHEL8

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

qa: enable CRB repo for RHEL8

instead of enabling PowerTools repo, we need to enable CodeReady Builder
repo for RHEL8

also, since we are moving to RHEL8, there is no need to install cmake3
specifically for CentOS. CentOS 8 comes with cmake3.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32242 from rishabh-d-dave/cv-methods-for-lvm-opt-S

ceph-volume: add methods to pass filters to pvs, vgs and lvs commands

qa: krbd_exclusive_option.sh: fixup for json.tool ordering change

In Python 3.5 json.tool was changed to produce unsorted output and
--sort-keys option was added to compensate.  This wasn't caught by
4fe245cc2f2d ("qa: update krbd tests for python3") because it raced
with 50933b863a1d ("qa: krbd_exclusive_option.sh: update for recent
kernel changes").

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

ceph-volume: add methods to pass filters to pvs, vgs and lvs commands

Filters can be passed to these commands by using option '-S'.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa/suites/krbd: fsx with object-map and fast-diff

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

test/librbd/fsx: don't disable features for krbd

rbd_default_features is set in krbd suite, no need to hard-code this in
fsx (especially now when object-map, fast-diff and deep-flatten are all
supported).

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

doc: mounting CephFS subdirectory and Persistent Mounts cleanup

Fixes: https://tracker.ceph.com/issues/37746
Signed-off-by: Jos Collin <jcollin@redhat.com>

osd/OSD: remove unused func enqueue_peering_evt_front

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #32485 from tchaikov/wip-clang

common,crimson: fixes to compile with clang and libc++

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #31728 from luis-henrix/copy-from-notrunc

osd: add a 'copy-from2' operation that includes truncate_{seq,size} parameters

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #32488 from ronen-fr/clang_3

crimson: CLANG-related fixes to errorator.h

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32490 from rzarzynski/wip-crimson-missing_err_include

crimson: add missing include in common/errorator.h

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson: add missing include in common/errorator.h

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: fix non-standard C++ constructs in errorator.h

Fixing use patterns accepted by gcc, but frowned upon by Clang:

Put noexcept specifier after attributes specifiers (as
per the C++ standard $11.3.5)

Limit the use of a not-completely-defined class names to
within function bodies, per the standard ("A class is considered
a completely-defined object type (or complete type) at the
closing } of the class-specifier.
Within the class member-specification, the class is regarded as
complete within function bodies, default arguments, noexcept-specifiers,
and default member initializers (including such things in nested classes).
Otherwise it is regarded as incomplete within its own class member-specification.")

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

crimson: fix FTBFS on Clang induced by errorator.

Reported-by Ronen Friedman <rfriedma@redhat.com>
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common: add missing #include

clang++-9.0.1 and libc++ require them to build

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: do not use auto in param list of function prototype

src/crimson/net/Socket.cc:114:57: error: 'auto' not allowed in function
prototype
static inline seastar::future<> close_and_handle_errors(auto& out) {
                                                        ^~~~

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: do not capture unused variables

to silence warnings like

src/crimson/net/Socket.cc:93:41: warning: lambda capture 'this' is not
used [-Wunused-lambda-capture]
    return in.read_exactly(bytes).then([this](auto buf) {
                                        ^~~~

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore/Blue(FS|Store): uint64_t alloc_size

Compiling on Debian's 32bit architectures failed with

<<PKGBUILDDIR>>/src/os/bluestore/BlueStore.cc:5539:37: error: no matching function for call to ‘p2align(size_t&, uint64_t&)’
 5539 |     total += p2align(len, alloc_size);
      |                                     ^
In file included from /<<PKGBUILDDIR>>/src/include/denc.h:42,
                 from /<<PKGBUILDDIR>>/src/include/encoding.h:40,
                 from /<<PKGBUILDDIR>>/src/include/compact_map.h:16,
                 from /<<PKGBUILDDIR>>/src/include/mempool.h:32,
                 from /<<PKGBUILDDIR>>/src/os/bluestore/BlueStore.h:36,
                 from /<<PKGBUILDDIR>>/src/os/bluestore/BlueStore.cc:25:
/<<PKGBUILDDIR>>/src/include/intarith.h:57:20: note: candidate: ‘template<class T> constexpr T p2align(T, T)’
   57 | constexpr inline T p2align(T x, T align) {
      |                    ^~~~~~~
/<<PKGBUILDDIR>>/src/include/intarith.h:57:20: note:   template argument deduction/substitution failed:
/<<PKGBUILDDIR>>/src/os/bluestore/BlueStore.cc:5539:37: note:   deduced conflicting types for parameter ‘T’ (‘unsigned int’ and ‘long long unsigned int’)
 5539 |     total += p2align(len, alloc_size);

As far as I understand it the available_freespace should at least be
able to return the same number as alloc_size, so we should use uint64_t
instead of size_t here, similar to
10a953afc8f803e50c96354470fb114b33e62599

Fixes: https://tracker.ceph.com/issues/43451
Signed-off-by: Bernd Zeimetz <bernd@bzed.de>

logrotate: also sighup rbd-mirror

Signed-off-by: Mykola Golub <mgolub@suse.com>

rbd-mirror: reopen all contexts logs on SIGHUP

Signed-off-by: Mykola Golub <mgolub@suse.com>

rbd-mirror: fix "pool replayer status" for case when init failed

Signed-off-by: Mykola Golub <mgolub@suse.com>

rbd-mirror: delay local/remote rados initialization until context created

We rely on that if a rados ref is initialized it contains the
valid context.

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #32474 from kalebskeithley/fedora32

do_cmake.sh: fedora-32 (rawhide) build with python-3.8

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32481 from tchaikov/wip=admin-build-doc-python3

admin/build-doc: always install python3-* for build deps

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

admin/build-doc: always install python3-* for build deps

since we've dropped the support of python36-*, i.e. python packages
provided by EPEL7 before RHEL7/CentOS7 included python3. as before the
inclusion of python3 as supported python3, python packages are named
python36-*. and they don't provide python3-*. so we had to install
python36-* explicitly. now that we are able to use the python3-*
packages, we can just install python3-*.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa: fix various py3 cephfs qa bugs

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: use -D_GNU_SOURCE when compiling fsync-tester.c

Otherwise loff_t is unavailable for some distributions.

Fixes: https://tracker.ceph.com/issues/43460
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: add comment about fsync-tester distribution

I was initially confused why this would not be included in the QA suite.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #32451 from rs-fabrica/doc_rados_operations_crush-map-edits

doc/rados/operations/crush-map-edits: recompile and set instructions

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #32442 into master

* refs/pull/32442/head:
doc/developer: fix dev mailing list address

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc/developer: fix dev mailing list address

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

mds: Reorganize struct members in Mutation header

Fixes: https://tracker.ceph.com/issues/42690
Signed-off-by: Varsha Rao <varao@redhat.com>

do_cmake.sh: fedora-32 (rawhide) build with python-3.8

Fedora-32 now has python-3.8.

Fedora-29 is EOL. All non-EOL versions of Fedora have python-3.7, except
Fedora-32

(Unclear to me why PYBUILD="3" isn't used, like Ubuntu and FreeBSD. That
would eliminate breakage when pyhon versions change in Rawhide.)

Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>

qa/tasks: Nothing to clean up if the volume was not created

There are only 2 cases which needs cleanup:
1. The volume is successfully created
2. The volume is successfully created but create_mds fails

In either case, we could do a 'volume rm'.

Signed-off-by: Jos Collin <jcollin@redhat.com>

mgr/volumes: drop obsolete comment in _cmd_fs_volume_create

This is fixed already.
Now the pool names are:
cephfs.<volume name>.meta
cephfs.<volume name>.data

Signed-off-by: Jos Collin <jcollin@redhat.com>

mgr/volumes: cleanup on fs create error

* clean up on fs create error
* drop unnecessary check in create_pool

Signed-off-by: Jos Collin <jcollin@redhat.com>

mon/OSDMonitor: Don't update mon cache settings if rocksdb is not used

Currently any update to mon cache config settings like 'mon_memory_target'
or 'rocksdb_cache_size' will cause a crash in case rocksdb and priority
cache manager is not being used.

If rocksdb (and therefore priority cache manager) is not being used, then
the update to the mon caches is ignored. Also, reduce the log levels of
some important log messages to help in debugging.

Fixes: https://tracker.ceph.com/issues/43454
Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>

Merge PR #32457 into master

* refs/pull/32457/head:
doc: fix Getting Started with CephFS

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #31309 from matthewoliver/rgw_swift_auth_note

doc: Add a RGW swift auth note

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #31432 from zhangsw/rgw-cleanup-cls-reshard-const

rgw:cleanup the "magic string" usage in cls_rgw_client.cc

Reviewed-by: Yuval Lifshitz <yuvalif@yahoo.com>

Merge pull request #31429 from zhangsw/rgw-cleanup-cls-guard-bucket-resharding

rgw: correct some error log about reshard in cls_rgw.cc.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #31404 from zhangsw/rgw-cleanup-deletelc

rgw:cleanup:remove un-used class member in RGWDeleteLC

Merge pull request #31345 from zhangsw/rgw-cleanup-rgw-admin-check-instance

rgw:cleanup:remove un-used create_new_bucket_instance in rgw_admin.cc

Reviewed-by: Casey Bodley <cbodley@redhat.com>

librbd: when unlinking peer from mirror snaps do it in all namespaces

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #32351 from mykaul/osd_fsid

ceph-volume/lvm/activate.py: clarify error message: fsid refers to osd_fsid

Merge pull request #27792 from smithfarm/wip-make-dist

make-dist: drop Python 2/3 autoselect

Reviewed-by: Kefu Chai <kchai@redhat.com>

lvm/activate.py: clarify error message: fsid refers to osd_fsid

Users complained[1] the error message isn't clear, and they thought
it referred to the cluster fsid instead of the osd_fsid.
Made it clearer.

[1] rook/rook#4547

Fixes: https://tracker.ceph.com/issues/43442
Signed-off-by: Yaniv Kaul <ykaul@redhat.com>