Merge pull request #39276 from smithfarm/wip-49130-nautilus

nautilus: rgw: fix trailing null in object names of multipart reuploads

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #39661 from pponnuvel/wip-49440-nautilus

nautilus: rgw: radosgw-admin: clarify error when email address already in use

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #39651 from smithfarm/wip-49055-nautilus

nautilus: common/mempool: Improve mempool shard selection

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #39698 from ifed01/wip-ifed-fix-huge-bluefs-nau

nautilus: os/bluestore: fix huge read/writes in BlueFS

Reviewed-by: Jianpeng Ma <jianpeng.ma@intel.com>
Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #39650 from smithfarm/wip-49182-nautilus

nautilus: test/rbd-mirror: fix broken ceph_test_rbd_mirror_random_write

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #39324 from rzarzynski/wip-librados-nul-objname-nautilus

nautilus: librados, tests: allow to list objects with the NUL character in names

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #39708 from smithfarm/wip-48282-nautilus

nautilus: osd: fix bluestore bitmap allocator calculate wrong last_pos with hint

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39509 from k0ste/wip-49314-nautilus

nautilus: mgr/dashboard: prometheus alerting: add some leeway for package drops and errors

Reviewed-by: Patrick Seidensal <pnawracay@suse.com>

Merge pull request #39329 from rzarzynski/wip-tool-nul-in-rados-nautilus

nautilus: tools/rados: add support for binary object names in the rados CLI

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #39441 from rhcs-dashboard/49278-py2-unicode-password

nautilus: mgr/dashboard: python 2: error when setting non-ASCII password

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #39129 from smithfarm/wip-48376-nautilus

nautilus: cephfs: client: check rdonly file handle on truncate

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #37906 from ideepika/backport-a7f18e4

nautilus: qa/tasks/{ceph,ceph_manager}: drop py2 support

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

osd: fix bluestore bitmap allocator calculated wrong last_pos with hint

Fixes: https://tracker.ceph.com/issues/48214
Signed-off-by: Xue Yantao <jhonxue@tencent.com>
(cherry picked from commit ad3c7cdf6fabbb4a8b542b793d632cae87ee3640)

common/mempool: include standard thread library

Attempt to address FTBFS:

/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/test_mempool.cc:399:11: error: request for member 'clear' in 'workers', which is of non-class type 'int'
  399 |   workers.clear();
      |           ^~~~~

seen in nautilus.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #39001 from Vicente-Cheng/wip-48520-nautilus

nautilus: client: add ceph.{cluster_fsid/client_id} vxattrs suppport

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

common/mempool: Add test for mempool in btree_map

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
(cherry picked from commit b426e52d0425f3ececf48c7936d3d6d0eeb69ebf)

Conflicts:
src/test/test_mempool.cc
- tests went into nautilus in a different order

Merge pull request #39544 from aaSharma14/wip-48733-nautilus

nautilus: mgr/dashboard: minimize console log traces of Ceph backend API tests

Reviewed-by: Avan Thakkar <athakkar@redhat.com>

Merge pull request #39518 from aaSharma14/wip-49323-nautilus

nautilus: mgr/dashboard: fix MTU Mismatch alert

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #38885 from yanghonggang/wip-48857-nautilus

nautilus: rgw: multisite: Verify if the synced object is identical to source

Reviewed-by: Casey Bodley <cbodley@redhat.com>

common/mempool: copies bufferlist assignment to mempool through rebuild

Fixes: https://tracker.ceph.com/issues/46027
Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
(cherry picked from commit b3a6d18ea64e0f70e0d3371cbfcd21280677000e)

Conflicts:
src/test/test_mempool.cc
- nautilus version of this file has less tests than master

nautilus: mgr/dashboard: python 2: error when setting user's non-ASCII password

Fixes: https://tracker.ceph.com/issues/49278
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #39658 from kotreshhr/ceph-vol-client-update-fix-volumes-key-nautilus

nautilus: pybind/ceph_volume_client: Update the 'volumes' key to 'subvolumes' in auth-metadata file

Reviewed-by: Ramana Raja <rraja@redhat.com>

os/bluestore: fix huge (>4GB) bluefs reads

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit fe558c4dd12b25b2d523987f22fde5b5898cfb83)

 Conflicts:
 (still present buf param in BlueFS::read, lack of )https://github.com/ceph/ceph/pull/34421)
src/os/bluestore/BlueFS.cc
src/os/bluestore/BlueFS.h
src/os/bluestore/BlueRocksEnv.cc
src/test/objectstore/test_bluefs.cc

os/bluestore: fix huge(>4GB) writes from RocksDB to BlueFS.

Fixes: https://tracker.ceph.com/issues/49168
Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 5f94883ec8d64c02b2bb499caad8eaf91dd715f7)

 Conflicts:
 (lack of bufferlist refactor from https://github.com/ceph/ceph/pull/36754)
 (lack of single allocator support from https://github.com/ceph/ceph/pull/30838)
src/os/bluestore/BlueFS.h
src/test/objectstore/test_bluefs.cc

test/unittest_bluefs: always remove temp bdev file

we leave files in build directory if the test fails. better off
removing them.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit a38122fdf77378d67eb97a5a2c55805181d921c2)

os/bluestore/BlueRocksEnv: Avoid flushing too much data at once.

Although, in _flush func we already check length. If length of dirty
is less then bluefs_min_flush_size, we will skip this flush.
But in fact, we found rocksdb can call many times Append() and then
call Flush(). This make flush_data is much larger than
bluefs_min_flush_size.

From my test, w/o this patch, it can reduce 99.99% latency(from
145.753ms to 20.474ms) for 4k randwrite with bluefs_buffered_io=true.

Because Bluefs::flush acquire lock. So we add new api try_flush
to avoid lock contention.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>
(cherry picked from commit 3a1d4ff6dcdcd897d31a85ef82c726599624b66d)

mgr/volumes: Bump up AuthMetadataManager's version

With ceph_volume_client and mgr-volumes co-existing
for sometime, the version of both needs to be same.
The ceph_volume_client version <=5 can't decode
'subvolumes' key in auth-metadata file. Hence to
handle version in-compatibility, the version of
ceph_volume_client is bumped up to 6 and the same
needs to be done in mgr-volume's AuthMetadataManager

Fixes: https://tracker.ceph.com/issues/49374
Signed-off-by: Kotresh HR <khiremat@redhat.com>
(cherry picked from commit 818c7781ff0467c6521bf2b3002b094fb8a71257)

pybind/ceph_volume_client: Bump up the version and compat_version to 6

With 'volumes' key updated to 'subvolumes', the version of
ceph_volume_client <= 5 can't decode auth-metadata file. Hence
bumping up ceph_volume_client version and compat_version to 6.

Fixes: https://tracker.ceph.com/issues/49294
Signed-off-by: Kotresh HR <khiremat@redhat.com>
(cherry picked from commit ce55a0bcdc5db139fc8f39665c372f1b7cc274a7)

pybind/ceph_volume_client: Fix auth-metadata file recovery

Fixes: https://tracker.ceph.com/issues/49294
Signed-off-by: Kotresh HR <khiremat@redhat.com>
(cherry picked from commit 99cdfdad59175a19bc06859e69301ca0478a2db4)

pybind/ceph_volume_client: Update the 'volumes' key to 'subvolumes' in auth metadata file

The older auth metadata files before nautilus release stores
the authorized subvolumes using the 'volumes' key. As the
notion of 'subvolumes' brought in by mgr/volumes, it makes
sense to use 'subvolumes' key. This patch would be tranparently
update 'volumes' key to 'subvolumes' and newer auth metadata
files would store them with 'subvolumes' key.

Also fails the deauthorize if the auth-id doesn't exist.

Fixes: https://tracker.ceph.com/issues/49294
Signed-off-by: Kotresh HR <khiremat@redhat.com>
(cherry picked from commit dee03c8d5c0b86cf51865090bec203419a3008a9)

pybind/ceph_volume_client: logger.warn() -> logger.warning()

logger.warn() is the same as logger.warning(), but was removed in py3.3

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit fe4c7fd7471bf724c4a1f027580d9bae3ddc24eb)

tools/rados: assign to optional<> without deref'ing it

this change addresses a regression introduced by
d333b35aa10bf03a8bc047994d5cf3fed019b49a

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 41d68594ae6df343782a1c382ac0bddc8cb7cb69)

rgw/radosgw-admin clarify error when email address already in use

The error message if you try and create an S3 user with an email
address that is already associated with another S3 account is very
confusing; this patch makes it much clearer

To reproduce:

radosgw-admin user create --uid=foo --display-name="Foo test" --email=bar@domain.invalid
radosgw-admin user create --uid=test --display-name="AN test" --email=bar@domain.invalid
could not create user: unable to parse parameters, user id mismatch, operation id: foo does not match: test

With this patch:

radosgw-admin user create --uid=test --display-name="AN test" --email=bar@domain.invalid
could not create user: unable to create user test because user id foo already exists with email bar@domain.invalid

Fixes: https://tracker.ceph.com/issues/49137
Fixes: https://tracker.ceph.com/issues/19411
Signed-off-by: Matthew Vernon <mv3@sanger.ac.uk>
(cherry picked from commit 05318d6f71e45a42a46518a0ef17047dfab83990)

common: move gen_rand_alphanumeric() helpers into common

and adds overloads that return std::string directly

the motivation is to remove cls_otp's dependency on rgw_common.h

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit bfa76731aa338ca2c8a9e6dc52a19ae71aeb0bbf)

Conflicts:
src/common/CMakeLists.txt
- no rabin.cc in nautilus

test: add unit tests for gen_rand_alphanumeric helpers

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit fb8975e0d4ce618c0b61aff3578e71706d23ae22)

rgw: fix trailing null in object names of multipart reuploads

the overload being called had a comment saying:
/* size should be the required string size + 1 */

this calls a different overload that returns std::string and handles
this null termination correctly

Fixes: https://tracker.ceph.com/issues/48874
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 2ede30782b00fe28ac714047f55b11b76ae1593f)

common/mempool: Modify shard selection function

Modify shard selection function, so no longer all threads use the same shard 0.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
(cherry picked from commit fccbdc0905e3868666fbb10803bac6b73f687cb1)

common/mempool: Add test for mempool shards

Add test that checks quality of mempool sharing.
It refuses to accept cases when variance is too large.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
(cherry picked from commit e4e07de8f36d0d44b341d5784a8bf56192201b95)

Conflicts:
src/test/test_mempool.cc
- nautilus version does not have as many tests as master version

test/rbd-mirror: fix broken ceph_test_rbd_mirror_random_write

It appears that commit 6eb8f30a238 broke the test utility and
its failure was masked by the test case that expected a failure
due to a timeout force-killing the app.

Fixes: https://tracker.ceph.com/issues/49117
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 8643b046fb4d5b05b4c75b83f16cd8ccc6a8b0a0)

Conflicts:
qa/workunits/rbd/rbd_mirror_helpers.sh
- no show_diff function in nautilus

Merge pull request #39626 from rhcs-dashboard/wip-49420-nautilus

nautilus: mgr/dashboard: set security headers

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

qa/tasks/ceph_manager: use s/ByteIO/StringIO in stdout for ceph-objectstore-tool

wrt master, we have moved to using run_ceph_objectstore_tool which uses
StringIO for stdout and stderr, to make the changes compatible with
nautilus, replacing use of ByteIO with StringIO.

see also: d8d44ed

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #39584 from liewegas/pr-39561-nautilus

nautilus: qa: use normal build for valgrind

qa/valgrind.supp: ignore valgrind leak

<error>
  <unique>0x12ee80e</unique>
  <tid>1</tid>
  <threadname>ceph-mon</threadname>
  <kind>Leak_StillReachable</kind>
  <xwhat>
    <text>8 bytes in 1 blocks are still reachable in loss record 2 of 31</text>
    <leakedbytes>8</leakedbytes>
    <leakedblocks>1</leakedblocks>
  </xwhat>
  <stack>
    <frame>
      <ip>0x4C2AC38</ip>
      <obj>/usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so</obj>
      <fn>operator new[](unsigned long)</fn>
      <dir>/builddir/build/BUILD/valgrind-3.15.0/coregrind/m_replacemalloc</dir>
      <file>vg_replace_malloc.c</file>
      <line>433</line>
    </frame>
    <frame>
      <ip>0xEB02D40</ip>
      <obj>/usr/lib64/libtcmalloc.so.4.4.5</obj>
    </frame>
    <frame>
      <ip>0x400F9B2</ip>
      <obj>/usr/lib64/ld-2.17.so</obj>
      <fn>_dl_init</fn>
    </frame>
    <frame>
      <ip>0x4001179</ip>
      <obj>/usr/lib64/ld-2.17.so</obj>
    </frame>
    <frame>
      <ip>0x5</ip>
    </frame>
    <frame>
      <ip>0x1FFF000BBE</ip>
    </frame>
    <frame>
      <ip>0x1FFF000BC7</ip>
    </frame>
    <frame>
      <ip>0x1FFF000BCA</ip>
    </frame>
    <frame>
      <ip>0x1FFF000BD4</ip>
    </frame>
    <frame>
      <ip>0x1FFF000BD9</ip>
    </frame>
    <frame>
      <ip>0x1FFF000BDC</ip>
    </frame>
  </stack>
</error>

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/dashboard: set security headers

Fixes: https://tracker.ceph.com/issues/49243
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
(cherry picked from commit 87bb3777877580f70018f4880488e264decc16e4)

 Conflicts:
qa/tasks/mgr/dashboard/test_requests.py
src/pybind/mgr/dashboard/services/auth.py
      - Adopting the master branch changes.

rgw multisite: fix single-part-MPU object etag misidentify problem

The single-part-MPU object is treated as a Atomic object.
Because its part_size is 0.

As MPU object's start_part_num is not 0 which can be used to
distinguish MPU object and Atomic object.

Fixes: https://tracker.ceph.com/issues/49357
Signed-off-by: Yang Honggang <yanghonggang@kuaishou.com>
(cherry picked from commit 63f989ca2bdc43142713b5da1eee2668ba238716)

qa/valgrind.supp: suppress free[] warnings

This is apparently not going to get fixed any time soon.

https://github.com/gperftools/gperftools/issues/792

Signed-off-by: Sage Weil <sage@newdream.net>
(cherry picked from commit 28fe149ea276693b47cc4d600780971f723672e4)

qa/suites: do not use notcmalloc flavor

teuthology now knows how to run valgrind against a tcmalloc binary

Signed-off-by: Sage Weil <sage@newdream.net>
(cherry picked from commit dc64ccf063f1ddf843fae4d2621ec25187f56135)

# Conflicts:
# qa/suites/rados/valgrind-leaks/1-start.yaml
# qa/suites/rgw/multisite/valgrind.yaml
# qa/suites/rgw/sts/tasks/0-install.yaml
# qa/suites/rgw/verify/validater/valgrind.yaml
# qa/suites/rgw/website/overrides.yaml

mgr/dashboard:minimize console log traces of Dashboard backend API tests

Changed log levels in some files to reduce console log traces

Fixes: https://tracker.ceph.com/issues/45822
Signed-off-by: Aashish Sharma <aasharma@redhat.com>
(cherry picked from commit f5dd0195cd45b6e00e43ef97e209e565e6452777)

tools/rados: replace omap_key_valid with std::optional.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
(cherry picked from commit eb87a0646a3635a7a35b6326e6d26e4b0b5b9d37)

tools/rados: replace omap_key_pretty with prettify().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
(cherry picked from commit cca92228f279d1334207b5e1f4bcfa225dca7f11)

tools/rados: add support for binary object names.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
(cherry picked from commit d333b35aa10bf03a8bc047994d5cf3fed019b49a)

Conflicts:
src/tools/rados/rados.cc
  * no input-file option,
  * no commands for cache tiering,
  * older API for omap retrival.

Merge pull request #39438 from rhcs-dashboard/wip-49271-nautilus

nautilus: mgr/dashboard: delete EOF when reading passwords from file

Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #39197 from ybwang0211/CrushWrapper

nautilus: crush/CrushWrapper: rebuild reverse maps after rebuilding crush map

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39160 from smithfarm/wip-48495-nautilus

nautilus: mon: paxos: Delete logger in destructor

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

mgr/dashboard:fix MTU Mismatch alert

This PR intends to fix the expression used for MTU Mismatch alert in prometheus

Signed-off-by: Aashish Sharma <aasharma@redhat.com>
(cherry picked from commit 8527489b9148a8845d6fccca6461b23ed7989172)

client: check rdonly file handle on truncate

Fixes: https://tracker.ceph.com/issues/48202
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 3bb209772eada4ba5010be521f7ecaae567598d0)

test/libcephfs: test truncate on rdonly fd

Fixes: https://tracker.ceph.com/issues/48202
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 2b41ddf343e8e314dc60fae0d5f4e007ce3857e4)

Conflicts:
        src/test/libcephfs/test.cc
- rewrite a trivial string + int concatenation to eliminate fmt::format

client: add ceph.{cluster_fsid/client_id} vxattrs suppport

These two vxattrs will only exist in local client side, with which
we can easily know which mountpoint the file belongs to and also
they can help locate the debugfs path quickly.

Fixes: https://tracker.ceph.com/issues/48337
Signed-off-by: Xiubo Li <xiubli@redhat.com>
(cherry picked from commit 5635f25dd4a134f2fb32d2cb9dd73c0e947295e6)

Conflicts:
src/client/Client.cc
  - add .hidden member because we still need it
            in nautilus
src/client/Client.h
  - drop the mirror.info xattr related code
            because nautilus does not introduce it

mgr/dashboard: prometheus alerting: add some leeway for package drops and errors (1%)

Fixes: https://tracker.ceph.com/issues/48201
Signed-off-by: Patrick Seidensal <pseidensal@suse.com>
(cherry picked from commit 9ac248b0c309b95d33bb0b1b64dc2e397bf8508e)

Merge pull request #39134 from smithfarm/wip-48879-nautilus

nautilus: mds: update defaults for recall configs

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #39133 from smithfarm/wip-48837-nautilus

nautilus: cephfs: mount.ceph: collect v2 addresses for non-legacy ms_mode options

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #39130 from smithfarm/wip-48814-nautilus

nautilus: mds: avoid spurious sleeps

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #39128 from smithfarm/wip-48371-nautilus

nautilus: mds: dir->mark_new() should together with dir->mark_dirty()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #39127 from smithfarm/wip-48130-nautilus

nautilus: cephfs: release client dentry_lease before send caps release to mds

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #38690 from batrick/4ca8aaafa32bb7397eec80a8eb366d2dd781a72f-nautilus

nautilus: qa: unmount volumes before removal

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #38552 from batrick/i48374

nautilus: cephfs: client: dump which fs is used by client for multiple-fs

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #39117 from rhcs-dashboard/wip-48961-nautilus

nautilus: mgr/dashboard: Fix for incorrect validation in rgw user form

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #39104 from k0ste/wip-49023-nautilus

nautilus: mgr/dashboard: trigger alert if some nodes have a MTU different than the median value

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #39184 from smithfarm/wip-49067-nautilus

nautilus: build/ops: install-deps.sh,deb,rpm: move python-saml deps into debian/control and ceph.spec.in

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #39136 from p-se/wip-48608-nautilus

nautilus: mgr/dashboard: enable different URL for users of browser to Grafana

Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #38839 from aaSharma14/wip-48738-nautilus

nautilus: mgr/dashboard: Use secure cookies to store JWT Token

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge PR #39270 into nautilus

* refs/pull/39270/head:
ceph-volume: mock VGs with string properties in tests
ceph-volume: add some flexibility to bytes_to_extents

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #39221 into nautilus

* refs/pull/39221/head:
ceph-volume: Fix usage of is_lv

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #39292 from kotreshhr/wip-mgr-backports-nautilus

nautilus: mgr/volume: subvolume auth_id management and few bug fixes

Reviewed-by: Ramana Raja <rraja@redhat.com>

mgr/dashboard: delete EOF when reading passwords from file

Signed-off-by: Alfonso Martínez <almartin@redhat.com>
(cherry picked from commit caeadf1397db00c6b7ba218b1910508099802e39)

Merge pull request #39419 from rhcs-dashboard/wip-49248-nautilus

nautilus: mgr/dashboard: customize CherryPy Server Header

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

qa: leave one standby available to avoid warning

Four file systems will use all MDS and generate this warning:

2020-11-02T03:48:33.407 INFO:teuthology.orchestra.run.smithi003.stdout:2020-11-02T03:24:21.817337+0000 mon.a (mon.0) 481 : cluster [WRN] Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY)

Fixes: https://tracker.ceph.com/issues/23718
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 59451923d31f5e4f707aa6d22ececc8edd395ca9)

qa: unmount volumes before removal

To avoid potential failures/hangs in umount.

Fixes: https://tracker.ceph.com/issues/23718
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 4ca8aaafa32bb7397eec80a8eb366d2dd781a72f)

qa: Fix a few mgr/volume test cases

Recovering dirty auth metadata file might not retain the order,
fixed the comparison in 'test_recover_auth_metadata_during_authorize'
and 'test_recover_auth_metadata_during_deauthorize'.

Fixes: https://tracker.ceph.com/issues/49192
Signed-off-by: Kotresh HR <khiremat@redhat.com>
(cherry picked from commit 4952d2132ffd92594e749a652970d5d9415c721d)

This commit is intended to hide CherryPy name and version, from HTTP header 'Server' response.

Fixes: <https://tracker.ceph.com/issues/44935>
Signed-off-by: anurag <abandhu@redhat.com>
(cherry picked from commit 6fea0f2ce861360643b86774fd86391b781c3789)

rgw: use static_ptr for etag verifiers

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 45a060612ec376110b84c2a2b7783c8a6aee191c)

rgw: add factory function create_etag_verifier()

move all of the etag verifier initialization into a helper function.
none of the errors there should be fatal and fail the download, they
should just turn etag verification off

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 8fa8974bbd63fbc8be9cdf929a875910e2147d65)

rgw: move etag verifiers to namespace rgw::putobj

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 6ad2e3eef3f9bbf41de471ef5bad9502023e113c)

rgw: simplify out SourceObjType

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 21cc9034410bb6e675b1b04888e1b85e3eb5d71f)

rgw: rgw_sync_obj_etag_verify accounts for compressed multipart uploads

the etag verifier for multipart uploads uses the manifest to get the
logical offsets for each part. but when compression is enabled, those
are offsets into the compressed data. use the source object's compression
info to translate those compressed part offsets back to their original
offsets

Fixes: https://tracker.ceph.com/issues/45992
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 51f79fad8398d35e07f01fa45704124e16fadeec)

rgw: ETagVerifier_MPU takes existing offset vector

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 2c1934bd3746697249b95b3f79f4c05425d7b40e)

rgw: add helper to decode compression info from single attr

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit a5520be135156c867a6502845603e2afdbb0a44a)

Conflicts:
src/rgw/rgw_compression.cc
src/rgw/rgw_compression.h

RGW:Multisite: Check rgw_sync_obj_etag_verify option only once

Signed-off-by: Prasad Krishnan <prasad.krishnan@flipkart.com>
(cherry picked from commit f92cfaf74f1eb8809653867b8c67a3ace37619f2)

RGW:Multisite: Convert is_mpu_obj into an enum SourceObjType

Signed-off-by: Prasad Krishnan <prasad.krishnan@flipkart.com>
(cherry picked from commit fa5422597837032d97f9afceff8b5a22fad0cda7)

RGW:Multisite: Rename rgw_sync_obj_integrity to rgw_sync_obj_etag_verify

Signed-off-by: Prasad Krishnan <prasad.krishnan@flipkart.com>
(cherry picked from commit 6c4262bfa7d54563ea8bf616154a6b3491d59347)

RGW:Multisite: Rename rgw_copy_verify_object to rgw_sync_obj_integrity

This patch renames the option rgw_copy_verify_object to
rgw_sync_obj_integrity and incorporates more changes suggested through
code-review comments.

Signed-off-by: Prasad Krishnan <prasad.krishnan@flipkart.com>
(cherry picked from commit 31e944fced60e47139973361cbb753aeaeb3c863)

RGW:Multisite: Create a new filter for ETag Verifier

This patch re-writes the ETag verifier into a filter that peeks into the
incoming stream of data and calculates MD5 checksum.

Signed-off-by: Prasad Krishnan <prasad.krishnan@flipkart.com>
(cherry picked from commit 2677c4b88806d4af6d525157e7006c1b0ca1b964)

Conflicts:
src/rgw/rgw_obj_manifest.h
src/rgw/rgw_rados.cc

[RGW][Multisite] Add multisite verifier support for MPU objects

The Etag for MPU objects is calculated using a method different from how
it is done for atomic objects. This patch makes use of the RGWObjManifest
to determine the parts in the source cluster and re-computes the ETag in
a similar fashion at the destination cluster during multisite sync for
verification.

Signed-off-by: Prasad Krishnan <prasad.krishnan@flipkart.com>
(cherry picked from commit 6f085bb9cb6cffddcf2eed18a7295a3dd59f755f)

Conflicts:
src/rgw/rgw_obj_manifest.h

RGW:Multisite: Verify if the synced object is identical to source

Introduce an option 'rgw_copy_verify_object' which allows the object
copied from remote cluster through multisite sync is identical to the
source object. This is done by generating the MD5 checksum of the data
being copied and compared to the ETAG stored as part of the object's
attribute.

Signed-off-by: Prasad Krishnan <prasad.krishnan@flipkart.com>
(cherry picked from commit 23fb5d7b786f1034a02c991c6a648e455b5b9c30)

Conflicts:
src/rgw/rgw_rados.cc

tools: switch scratch tool to use rados_nobjects_list_next2().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
(cherry picked from commit 05dca8aa338fecd701c5a681242322b50bd11e67)

librados: NObjectIteratorImpl takes care about NUL in obj name.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
(cherry picked from commit 22bb1498c93b1e1419920fe928e91885504f1f16)

test: add unit test for rados_nobjects_list_next2().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
(cherry picked from commit 8965506b8bd0b74ca5ec743e5eddd40dd4d37bac)

librados: introduce rados_nobjects_list_next2().

It's needed to properly list an object with the NUL
character in the name. Such names can appear mostly
as a consequence of bugs in clients.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
(cherry picked from commit 9d268b0dd72324347ce74408b9a7a2a95c37ad0f)

Conflicts:
src/librados/librados_c.cc (nautilus lacks
the LIBRADOS_C_API_DEFAULT_F macro which is
present in master; customized the changes to
to fit nautilus)

qa/cephfs: add methods to read/write on CephFS mounts

Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit 3f0284f272231c3b62b0f3f201cbaaecfa405bcd)

Conflicts:
    qa/tasks/cephfs/mount.py: get_file and IP module is not present in
        nautilus

ceph_volume_client: Fix failure of test_idempotency

With the test environment, 'args must be encodeable
 as a bytearray' error is seen for 'ceph_mds_command'.
Hence removed tuple and passed the JSON formatted string.

Fixes: https://tracker.ceph.com/issues/48830
Signed-off-by: Kotresh HR <khiremat@redhat.com>
(cherry picked from commit 1c6c172a9b665d6b769d67e301061dbd7b044472)