common: move gen_rand_alphanumeric() helpers into common

and adds overloads that return std::string directly

the motivation is to remove cls_otp's dependency on rgw_common.h

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit bfa76731aa338ca2c8a9e6dc52a19ae71aeb0bbf)

Conflicts:
src/common/CMakeLists.txt
- no rabin.cc in nautilus

test: add unit tests for gen_rand_alphanumeric helpers

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit fb8975e0d4ce618c0b61aff3578e71706d23ae22)

rgw: fix trailing null in object names of multipart reuploads

the overload being called had a comment saying:
/* size should be the required string size + 1 */

this calls a different overload that returns std::string and handles
this null termination correctly

Fixes: https://tracker.ceph.com/issues/48874
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 2ede30782b00fe28ac714047f55b11b76ae1593f)

Merge pull request #39626 from rhcs-dashboard/wip-49420-nautilus

nautilus: mgr/dashboard: set security headers

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #39584 from liewegas/pr-39561-nautilus

nautilus: qa: use normal build for valgrind

qa/valgrind.supp: ignore valgrind leak

<error>
  <unique>0x12ee80e</unique>
  <tid>1</tid>
  <threadname>ceph-mon</threadname>
  <kind>Leak_StillReachable</kind>
  <xwhat>
    <text>8 bytes in 1 blocks are still reachable in loss record 2 of 31</text>
    <leakedbytes>8</leakedbytes>
    <leakedblocks>1</leakedblocks>
  </xwhat>
  <stack>
    <frame>
      <ip>0x4C2AC38</ip>
      <obj>/usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so</obj>
      <fn>operator new[](unsigned long)</fn>
      <dir>/builddir/build/BUILD/valgrind-3.15.0/coregrind/m_replacemalloc</dir>
      <file>vg_replace_malloc.c</file>
      <line>433</line>
    </frame>
    <frame>
      <ip>0xEB02D40</ip>
      <obj>/usr/lib64/libtcmalloc.so.4.4.5</obj>
    </frame>
    <frame>
      <ip>0x400F9B2</ip>
      <obj>/usr/lib64/ld-2.17.so</obj>
      <fn>_dl_init</fn>
    </frame>
    <frame>
      <ip>0x4001179</ip>
      <obj>/usr/lib64/ld-2.17.so</obj>
    </frame>
    <frame>
      <ip>0x5</ip>
    </frame>
    <frame>
      <ip>0x1FFF000BBE</ip>
    </frame>
    <frame>
      <ip>0x1FFF000BC7</ip>
    </frame>
    <frame>
      <ip>0x1FFF000BCA</ip>
    </frame>
    <frame>
      <ip>0x1FFF000BD4</ip>
    </frame>
    <frame>
      <ip>0x1FFF000BD9</ip>
    </frame>
    <frame>
      <ip>0x1FFF000BDC</ip>
    </frame>
  </stack>
</error>

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/dashboard: set security headers

Fixes: https://tracker.ceph.com/issues/49243
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
(cherry picked from commit 87bb3777877580f70018f4880488e264decc16e4)

 Conflicts:
qa/tasks/mgr/dashboard/test_requests.py
src/pybind/mgr/dashboard/services/auth.py
      - Adopting the master branch changes.

qa/valgrind.supp: suppress free[] warnings

This is apparently not going to get fixed any time soon.

https://github.com/gperftools/gperftools/issues/792

Signed-off-by: Sage Weil <sage@newdream.net>
(cherry picked from commit 28fe149ea276693b47cc4d600780971f723672e4)

qa/suites: do not use notcmalloc flavor

teuthology now knows how to run valgrind against a tcmalloc binary

Signed-off-by: Sage Weil <sage@newdream.net>
(cherry picked from commit dc64ccf063f1ddf843fae4d2621ec25187f56135)

# Conflicts:
# qa/suites/rados/valgrind-leaks/1-start.yaml
# qa/suites/rgw/multisite/valgrind.yaml
# qa/suites/rgw/sts/tasks/0-install.yaml
# qa/suites/rgw/verify/validater/valgrind.yaml
# qa/suites/rgw/website/overrides.yaml

Merge pull request #39438 from rhcs-dashboard/wip-49271-nautilus

nautilus: mgr/dashboard: delete EOF when reading passwords from file

Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #39197 from ybwang0211/CrushWrapper

nautilus: crush/CrushWrapper: rebuild reverse maps after rebuilding crush map

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39160 from smithfarm/wip-48495-nautilus

nautilus: mon: paxos: Delete logger in destructor

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #39134 from smithfarm/wip-48879-nautilus

nautilus: mds: update defaults for recall configs

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #39133 from smithfarm/wip-48837-nautilus

nautilus: cephfs: mount.ceph: collect v2 addresses for non-legacy ms_mode options

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #39130 from smithfarm/wip-48814-nautilus

nautilus: mds: avoid spurious sleeps

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #39128 from smithfarm/wip-48371-nautilus

nautilus: mds: dir->mark_new() should together with dir->mark_dirty()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #39127 from smithfarm/wip-48130-nautilus

nautilus: cephfs: release client dentry_lease before send caps release to mds

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #38690 from batrick/4ca8aaafa32bb7397eec80a8eb366d2dd781a72f-nautilus

nautilus: qa: unmount volumes before removal

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #38552 from batrick/i48374

nautilus: cephfs: client: dump which fs is used by client for multiple-fs

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #39117 from rhcs-dashboard/wip-48961-nautilus

nautilus: mgr/dashboard: Fix for incorrect validation in rgw user form

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #39104 from k0ste/wip-49023-nautilus

nautilus: mgr/dashboard: trigger alert if some nodes have a MTU different than the median value

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #39184 from smithfarm/wip-49067-nautilus

nautilus: build/ops: install-deps.sh,deb,rpm: move python-saml deps into debian/control and ceph.spec.in

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #39136 from p-se/wip-48608-nautilus

nautilus: mgr/dashboard: enable different URL for users of browser to Grafana

Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #38839 from aaSharma14/wip-48738-nautilus

nautilus: mgr/dashboard: Use secure cookies to store JWT Token

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge PR #39270 into nautilus

* refs/pull/39270/head:
ceph-volume: mock VGs with string properties in tests
ceph-volume: add some flexibility to bytes_to_extents

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #39221 into nautilus

* refs/pull/39221/head:
ceph-volume: Fix usage of is_lv

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #39292 from kotreshhr/wip-mgr-backports-nautilus

nautilus: mgr/volume: subvolume auth_id management and few bug fixes

Reviewed-by: Ramana Raja <rraja@redhat.com>

mgr/dashboard: delete EOF when reading passwords from file

Signed-off-by: Alfonso Martínez <almartin@redhat.com>
(cherry picked from commit caeadf1397db00c6b7ba218b1910508099802e39)

Merge pull request #39419 from rhcs-dashboard/wip-49248-nautilus

nautilus: mgr/dashboard: customize CherryPy Server Header

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

qa: leave one standby available to avoid warning

Four file systems will use all MDS and generate this warning:

2020-11-02T03:48:33.407 INFO:teuthology.orchestra.run.smithi003.stdout:2020-11-02T03:24:21.817337+0000 mon.a (mon.0) 481 : cluster [WRN] Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY)

Fixes: https://tracker.ceph.com/issues/23718
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 59451923d31f5e4f707aa6d22ececc8edd395ca9)

qa: unmount volumes before removal

To avoid potential failures/hangs in umount.

Fixes: https://tracker.ceph.com/issues/23718
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 4ca8aaafa32bb7397eec80a8eb366d2dd781a72f)

qa: Fix a few mgr/volume test cases

Recovering dirty auth metadata file might not retain the order,
fixed the comparison in 'test_recover_auth_metadata_during_authorize'
and 'test_recover_auth_metadata_during_deauthorize'.

Fixes: https://tracker.ceph.com/issues/49192
Signed-off-by: Kotresh HR <khiremat@redhat.com>
(cherry picked from commit 4952d2132ffd92594e749a652970d5d9415c721d)

This commit is intended to hide CherryPy name and version, from HTTP header 'Server' response.

Fixes: <https://tracker.ceph.com/issues/44935>
Signed-off-by: anurag <abandhu@redhat.com>
(cherry picked from commit 6fea0f2ce861360643b86774fd86391b781c3789)

qa/cephfs: add methods to read/write on CephFS mounts

Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit 3f0284f272231c3b62b0f3f201cbaaecfa405bcd)

Conflicts:
    qa/tasks/cephfs/mount.py: get_file and IP module is not present in
        nautilus

ceph_volume_client: Fix failure of test_idempotency

With the test environment, 'args must be encodeable
 as a bytearray' error is seen for 'ceph_mds_command'.
Hence removed tuple and passed the JSON formatted string.

Fixes: https://tracker.ceph.com/issues/48830
Signed-off-by: Kotresh HR <khiremat@redhat.com>
(cherry picked from commit 1c6c172a9b665d6b769d67e301061dbd7b044472)

mgr/volumes: Evict clients based on auth-IDs and subvolume mounted

Add subvolume evict command which evicts the subvolume mounts
which are mounted using particular auth-ID.

Fixes: https://tracker.ceph.com/issues/44928
Signed-off-by: Kotresh HR <khiremat@redhat.com>
(cherry picked from commit 269adcc8b8ab0742ba741ed7c2b59ccfb17a63f9)

Conflicts:
    qa/tasks/cephfs/test_volumes.py: Few of the tests are re-organized,
      hence the conflicts. Resolved the same.

doc/mgr/volumes: Document 'fs subvolume authorized_list' cli

Fixes: https://tracker.ceph.com/issues/44931
Signed-off-by: Kotresh HR <khiremat@redhat.com>
(cherry picked from commit 9f9f8adc47486432c746565ea5a1f204736632c1)

qa: Add tests for list auth-ids of a subvolume

Fixes: https://tracker.ceph.com/issues/44931
Signed-off-by: Kotresh HR <khiremat@redhat.com>
(cherry picked from commit 1b98e63e75236ff1cd1c9cb1ead8eb965698d5b6)

Conflicts:
    qa/tasks/cephfs/test_volumes.py: Few of the tests are re-organized,
     hence the conflicts. Resolved the same.

mgr/volumes: Update the 'volumes' key to 'subvolumes' in auth metadata file

The older auth metadata files created by CephVolumeClient stores the
authorized subvolumes using the 'volumes' key as the notion of
'subvolumes' brought in by mgr/volumes. Hence, this would be tranparently
updated to 'subvolumes' and newer auth metadata files would store them
with 'subvolumes' key.

Also fails the deauthorize if the auth-id doesn't exist.

Fixes: https://tracker.ceph.com/issues/44931
Signed-off-by: Kotresh HR <khiremat@redhat.com>
(cherry picked from commit 5f32eb15918657bcf1f73025fbb3adf0cc06317e)

mgr/volumes: Optionally authorize existing auth-ids

Optionally allow authorizing auth-ids not created by mgr plugin
via the option 'allow_existing_id'. This can help existing deployers
of manila to disallow/allow authorization of pre-created auth IDs
via a manila driver config that sets 'allow_existing_id' to False/True.

Fixes: https://tracker.ceph.com/issues/44931
Signed-off-by: Kotresh HR <khiremat@redhat.com>
(cherry picked from commit 713270d1869e2370b674b1a4bc6f6a37023a5917)

mgr/volumes: Preserve existing caps while authorize/deauthorize auth-id

Authorize/Deauthorize used to overwrite the caps of auth-id which would
end up deleting existing caps. This patch fixes the same by retaining
the existing caps by appending or deleting the new caps as needed.

Fixes: https://tracker.ceph.com/issues/44931
Signed-off-by: Kotresh HR <khiremat@redhat.com>
(cherry picked from commit 2dece3be081fe572455d6b634e38a663d1643dc8)

mgr/volumes: Disallow authorize existing auth_id

This patch disallow the mgr plugin to authorize the auth_id
which is not created via mgr plugin. Those auth_ids could be
created by other means for other use cases which should not be modified
via mgr plugin.

Fixes: https://tracker.ceph.com/issues/44931
Signed-off-by: Kotresh HR <khiremat@redhat.com>
(cherry picked from commit c02890404e47a5a95c5cc16b699306045d586c7f)

mgr/volumes: Add subvolume authorized_list command

Fixes: https://tracker.ceph.com/issues/44931
Signed-off-by: Kotresh HR <khiremat@redhat.com>
(cherry picked from commit 1abec3d0ca8c4fa405cdbf56c55f44f37aca9ca8)

mgr/volumes: Add tenant_id option to subvolume authorize

Fixes: https://tracker.ceph.com/issues/44931
Signed-off-by: Kotresh HR <khiremat@redhat.com>
(cherry picked from commit 39acfcc91c1b6a85f6fa96a5e894d81a7225f9dc)

mgr/volumes: Persist auth and subvolume metadata

1. Subvolume create and delete operations create and delete subvolume
   metadata file respectively.
2. Subvolume authorize creates the auth meta file and persists the
   required metadata on subvolume metadata file and auth metdata file
   on disk. Subvolume deauthorize clears the required metadata on
   both metadata files.

Fixes: https://tracker.ceph.com/issues/44931
Signed-off-by: Kotresh HR <khiremat@redhat.com>
(cherry picked from commit 04d876ced756ca86580bdff4ac116333dbb102e5)

mgr/volumes: Filter inherited snapshots while listing snapshots

Filter inherited snapshots resulted as part of a snapshot
at ancestor level while listing snapshots of a subvolume
and subvolumegroup

Also, fail the snapshot info on inherited snapshot.

Fixes: https://tracker.ceph.com/issues/48501
Signed-off-by: Kotresh HR <khiremat@redhat.com>
(cherry picked from commit bd49b6409be79dba4a119c809983a05687242732)

Conflicts:
    qa/tasks/cephfs/test_volumes.py: Few of the tests are re-organized,
      hence the conflicts. Resolved the same.

doc/mgr/volumes: Document authorize/deauthorize cli commands

Fixes: https://tracker.ceph.com/issues/40401
Signed-off-by: Kotresh HR <khiremat@redhat.com>
(cherry picked from commit 1426c23ab50a4d64e1c4f1f61185117a270c8ec7)

qa/tasks/cephfs: test `fs subvolume authorize/deauthorize`

Fixes: https://tracker.ceph.com/issues/40401
Signed-off-by: Ramana Raja <rraja@redhat.com>
Signed-off-by: Kotresh HR <khiremat@redhat.com>
(cherry picked from commit 7c98dc1ad35a2244cbf949be9782a3412886b305)

Conflicts:
    qa/tasks/cephfs/test_volumes.py: Few of the tests are re-organized
        and CLIENTS_REQUIRED should be 2 for these tests which was 1
        for existing tests. Resolved the same.

mgr/volumes: Allow/deny auth IDs access to FS subvolumes

... via the `ceph fs subvolume authorize/deauthorize` command.

Fixes: https://tracker.ceph.com/issues/40401
Signed-off-by: Ramana Raja <rraja@redhat.com>
Signed-off-by: Kotresh HR <khiremat@redhat.com>
(cherry picked from commit 6c3b7547fbf3d987e715e9502359acd873374831)

Conflicts:
    src/pybind/mgr/volumes/fs/volume.py: subvolume pin is not available
in nautilus.

mon/MonCap: allow 'profile mgr' to create/update/del auth IDs

Signed-off-by: Ramana Raja <rraja@redhat.com>
(cherry picked from commit d3aea5579778a73dc2418e7b137dc21717944c3e)

Merge pull request #39202 from idryomov/wip-krbd-msgr2-nautilus

nautilus: krbd: add support for msgr2 (kernel 5.11)

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38894 from smithfarm/wip-48518-nautilus

nautilus: pybind/cephfs: fix missing terminating NULL char in readlink()'s C string

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #38627 from batrick/i48634

nautilus: qa: fix tox failures

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #38101 from kotreshhr/wip-48192-nautilus

nautilus: cephfs: mds: throttle cap acquisition via readdir

Reviewed-by: Ramana Raja <rraja@redhat.com>

ceph-volume: mock VGs with string properties in tests

The real things uses string properties as well.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit c4bbb1b999b08bd84d793537f2602fc8e4c93bae)

ceph-volume: add some flexibility to bytes_to_extents

For batch we want to report the projected sizes before creating any LVM
structures, we use the device size for that. This means we pass this
projected size to lvm/api.py::create_lv. This could result in situations
where we request more extents for the new LV than are actually
available, as LVM takes up some available space on the block device
(plus alignment).
We fix this by adjusting the extent count downwards if its less than 1% off the
requested value.

Fixes: https://tracker.ceph.com/issues/47758
Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit ece6e7eb9a92cb944b2f201a05a5acfb877c2dba)

ceph-volume: Fix usage of is_lv

In commit e09d3cd1a5bcfb42972b6958fb7e6c42a645549d is_lv method was removed from
api/lvm.py. This change fixes util/disk.py to not use this missing method.

Fixes: https://tracker.ceph.com/issues/48784
Signed-off-by: Michał Nasiadka <mnasiadka@gmail.com>
(cherry picked from commit 5050a88e7a85bb1aa6cd5473c9083a186e2264d5)

Merge pull request #39125 from smithfarm/wip-48595-nautilus

nautilus: tests: cancelling both noscrub *and* nodeep-scrub

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #39124 from smithfarm/wip-48379-nautilus

nautilus: mon: validate crush-failure-domain

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39076 from k0ste/wip-48957-nautilus

nautilus: mgr/prometheus: Fix 'pool filling up' with >50% usage

Reviewed-by: Patrick Seidensal <pnawracay@suse.com>

Merge pull request #39075 from k0ste/wip-49005-nautilus

nautilus: mgr: update mon metadata when monmap is updated

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37948 from aclamk/fix-46194-bluefs-log-growth-nau

nautilus: bluestore: Add protection against bluefs log file growth

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #39123 from smithfarm/wip-49026-nautilus

nautilus: rbd/bench: include used headers

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #39121 from smithfarm/wip-48691-nautilus

nautilus: librbd: clear implicitly enabled feature bits when creating images

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39047 from smithfarm/wip-48969-nautilus

nautilus: ocf: add support for mapping images within an RBD namespace

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #38830 from smithfarm/wip-48083-nautilus

nautilus: rbd-nbd: reexpand the conf meta in child process

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: enable different URL for users of browser to Grafana

Fixes: https://tracker.ceph.com/issues/47386
Signed-off-by: Patrick Seidensal <pseidensal@suse.com>
(cherry picked from commit 676f82923d875cfe9528a401963c09f29a6de7f4)

Conflicts:
        doc/mgr/dashboard.rst
        src/pybind/mgr/dashboard/controllers/grafana.py

Resolved some trivial conflicts and replaced the heading of the newly
added section in the documentation to another level, as the
documentation between master and octopus differs quite much.

(cherry picked from commit 56f304d31b154d87362620f08046720a3e0f3203)

Conflicts:

Removed cephadm documentation from cherry-picked commit as cephadm is
not part of Ceph Nautilus.

Removed paragraph that mentioned cephadm in dashboard documentation.

crush/CrushWrapper: rebuild reverse maps after rebuilding crush map

The Objecter will crash when localized reads are enabled and two threads
attempt to rebuild the (invalidated) reverse maps concurrently. This
should address the issue for the Objecter use-case without the need to
add additional locking.

Fixes: https://tracker.ceph.com/issues/44311
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 8b866794f5b3674c5e3ad9adceb5e3230d55a0e0)

qa/suites/krbd: add msgr2 modes to most subsuites

basic, rbd and rbd-nomount subsuites are expanded to run with each
of ms_mode=legacy, ms_mode=crc and ms_mode=secure.  This increases
the total number of jobs in the suite from 100 to 220.

fsx, singleton and thrash subsuites choose ms_mode at random (from
the above plus ms_mode=prefer-crc).

unmap and wac subsuites remain msgr1-only.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 65948736a41f424d8152b208d013419f8d6038a4)

Conflicts:
qa/suites/krbd/fsx/conf.yaml [ commit cff2e49ff09e
  ("qa/suites/krbd: fsx with object-map and fast-diff") and
  read_from_replica map option not in nautilus ]
qa/suites/krbd/fsx/ms_mode$/*.yaml [ read_from_replica map option
  not in nautilus ]

test/librbd/fsx: respect rbd_default_map_options in krbd_open()

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit b29dfc4624f3b8286d345ec62894fa70c5f6cf9b)

doc: deprecate [no]cephx_require_signatures map options

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit fd5f5722a29964bb33b305a381bcf9a48cdcbb47)

krbd: add support for msgr2

Recognize ms_mode map option and filter initial monitor addresses
accordingly: if ms_mode is not given or ms_mode=legacy, discard v2
addresses, otherwise discard v1 addresses.

Note that nothing was discarded (i.e. v2 addresses were passed to
the kernel) previously.  The intent was to preserve that behaviour
in case ms_mode is not given, allowing to change the kernel default
in the future.  However, it turns out that mount.ceph helper has
been misguidedly discarding v2 addresses since commit eae01275134e
("mount.ceph: fork a child to get info from local configuration"),
so that ship has sailed.

Fixes: https://tracker.ceph.com/issues/48976
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 08f714964b7fe5024504818f01328a41acc24965)

Conflicts:
doc/man/8/rbd.rst [ crush_location, read_from_replica and
  compression_hint map options not in nautilus ]
src/krbd.cc [ commit a1aecba6f44d ("krbd: make wait_for_udev_add()
  clearer and a bit more efficient") not in nautilus ]
src/tools/rbd/action/Kernel.cc [ commit 34f539d8af33 ("rbd:
  delay parsing of default kernel map options") not in nautilus ]

install-deps.sh,deb,rpm: move python-saml deps into debian/control and ceph.spec.in

these dependencies are only used for building python-saml which is in
turn used for the SAML support. this feature is tested using
`test_sso.py` while performing dashboard tests. we do not package or
ship python-saml along with other Ceph packages. so let's move these
dependencies to the "make check" sections in ceph.spec.in and
debian/control for simplifying install-deps.sh.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 5d6d770e2fbf06c9aca58246147d92aa9b221e1b)

Conflicts:
ceph.spec.in
debian/control
install-deps.sh

Merge pull request #39175 from rhcs-dashboard/49062-fix-iscsi-gw-add

nautilus mgr/dashboard: fix 'ceph dashboard iscsi-gateway-add'

Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

nautilus mgr/dashboard: fix 'ceph dashboard iscsi-gateway-add'

Fixes: https://tracker.ceph.com/issues/49062
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

mon: paxos: Delete logger in destructor

reset() can race with shutdown() leading to a use-after-free on the
'logger' object.

Fixes: https://tracker.ceph.com/issues/48386
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
(cherry picked from commit cc295d1c87552eb76b9188d88d7e6ab2f3108149)

Conflicts:
src/mon/Paxos.h
- stable branch has slightly different Paxos and get_name function declarations

mgr/dashboard: Use secure cookies to store JWT Token

This PR intends to store the jwt token in secure cookies instead of local storage

Fixes: https://tracker.ceph.com/issues/44591
Signed-off-by: Aashish Sharma <aasharma@redhat.com>
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
(cherry picked from commit 36703c63381e6723fff57266235f8230e6af1d92)
(cherry picked from commit 3c72dc309936b23e413dc1aee8ca49c795c48a0f)
(cherry picked from commit ea9876d6dfc7033a61cac68c985fd55fd52f6064)
(cherry picked from commit d3218fcb44d9202d751380f6b2702b14cf420e1f)
(cherry picked from commit 2cd7ce9e69b6a14778432abd17b34ba9e4020abc)

mds: dir->mark_new() should together with dir->mark_dirty()

Otherwise, the 'new' flag won't get cleared after dir is committed.
The leftover 'new' flag will confuse later journal and dir commit.

Fixes: https://tracker.ceph.com/issues/48249
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit be4710a1496c46f3307966ddfa416e932d878603)

Conflicts:
        src/mds/MDCache.cc
- the "mark_dirty" method requires an additional boilerplate argument in
  nautilus

mds: update defaults for recall configs

This is based on informal experience reportings from production
clusters.

Fixes: https://tracker.ceph.com/issues/48403
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 8df2388b9fb66e1606f47c095ecf0b5c71a1941e)

Conflicts:
src/common/options.cc
- nautilus does not have FLAG_RUNTIME

mount.ceph: collect v2 addresses for non-legacy ms_mode options

The kclient recently had support for msgr2 merged, and with that we have
a new ms_mode mount option. The mount helper's support for autodiscovery
of mons currently always ignores v2 addresses.

Change the helper to look for the ms_mode in the provided options and if
it's set to anything but "legacy", have it collect v2 addresses instead
of v1 addrs. The default is still "legacy" so if the option is not
provided, we will still collect v1 addrs.

Fixes: https://tracker.ceph.com/issues/48765
Signed-off-by: Jeff Layton <jlayton@redhat.com>
(cherry picked from commit 50da7ad7f0a5b75314c51ca4f4c1f2c013c0e8b8)

mds: avoid spurious sleeps

Like trim_interval, don't sleep for small amounts of time. This avoids
spurious sleeps like:

    2020-12-25T00:14:22.242+0000 7f6a95884700 20 mds.0.cache upkeep thread waiting interval 0.000000108s
    2020-12-25T00:14:22.242+0000 7f6a95884700 20 mds.0.cache upkeep thread waiting interval 0.000000108s
    2020-12-25T00:14:22.242+0000 7f6a95884700 20 mds.0.cache upkeep thread waiting interval 0.000000108s
    2020-12-25T00:14:22.242+0000 7f6a95884700 20 mds.0.cache upkeep thread waiting interval 0.000000108s
    2020-12-25T00:14:22.242+0000 7f6a95884700 20 mds.0.cache upkeep thread waiting interval 0.000000108s
    2020-12-25T00:14:22.242+0000 7f6a95884700 20 mds.0.cache upkeep thread waiting interval 0.000000108s
    2020-12-25T00:14:22.242+0000 7f6a95884700 20 mds.0.cache upkeep thread waiting interval 0.000000108s

Also, fix the same issue in the Client.

Fixes: https://tracker.ceph.com/issues/48753
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit eb47e990c33843b9baa366e2b2a187439210e680)

Conflicts:
src/client/Client.cc
- in the nautilus version of this file, the code being modified does not exist

cephfs: release client dentry_lease before send caps release to mds

Fixes: https://tracker.ceph.com/issues/47854
Signed-off-by: Wei Qiaomiao <wei.qiaomiao@zte.com.cn>
(cherry picked from commit 9b07e73f60feb6a7d5fd2db3cc22ca22871e8f1a)

test: cancelling both noscrub *and* nodeep-scrub

as part of osd-scrub-test.sh.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
(cherry picked from commit 43b1129030823817e0b7a21c85de5d3da841510a)

mon: validate crush-failure-domain

While creating erasure-coded profile make sure
that user is specifying valid crush-failure-domain.

Fixes: https://tracker.ceph.com/issues/47452
Signed-off-by: Prashant Dhange <pdhange@redhat.com>
(cherry picked from commit f8b7fddc4c4a787d58fb9efef6ecb13cf878f92b)

rbd/bench: include used headers

should help with fixing following failure:

src/tools/rbd/action/Bench.cc:20:26: error: aggregate 'std::atomic<bool>
terminating' has incomplete type and cannot be defined

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit f16ac13c13eceed7adb5a4a04a3372b921e63031)

librbd: drop explicit masking of implicit feature bits

Now that the create image state machine is handling the masking
of implicit features, all callers to the state machine can skip
the need to perform the masking themselves.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit e1dd69391900a236fbda5e4cee58e9bdc20824fc)

Conflicts:
src/librbd/api/Migration.cc
- in nautilus, this file does not mention RBD_FEATURES_IMPLICIT_ENABLE

librbd/image: mask out all implicit features when creating an image

This will ensure that all paths to the create image state machine
properly handle this condition. Previously, it was up to the callers
of the state machine to clear the implicit feature bits.

Fixes: https://tracker.ceph.com/issues/48647
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit f52f78caca6f9743e75c8289771375f5f582300a)

librbd/image: use internal feature flag mask when creating images

When validating the provided features for an image, utilize the
RBD_FEATURES_INTERNAL flag instead of just RBD_FEATURE_OPERATION.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 7dd77814402f0710f9283084b45960a054078897)

librbd/api: do not explicitly add the MIGRATION feature bit

This feature bit is automatically added to the source and destination
images via the cls method on the OSD.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 5380615037e1f78369e58eac9847b465e9e8bf19)

Conflicts:
src/librbd/api/Migration.cc
- no Migration<I>::prepare_import template in nautilus

mgr/dashboard: Fix for incorrect validation in rgw user form

The rgw users create form doesnt validate the username correctly if the username is a tenated one. For eg. Consider there is a user called tenate$sample. Now I am trying to create another user and I entered tenate$sample as username. But it doesn't async validate the username as existing. Instead it just shows the green tick and once the submit button is clicked it'll show the user as existing.

Fixes: https://tracker.ceph.com/issues/48907
Signed-off-by: Nizamudeen A <nia@redhat.com>
(cherry picked from commit 3c2e8e4f9a7645c20b5766b2c052063ce38c5650)

Merge pull request #39090 from smithfarm/wip-48284-nautilus

nautilus: rpm,deb: change sudoers file mode to 440

Reviewed-by: Kefu Chai <kchai@redhat.com>

common: do not dup the options when reexpanding

The old code will store all the options, which has `$pid` in them,
in may_reexpand_meta map. And when reexpanding later, the reexpand
code will dup them with a higher priority(CONF_OVERRIDE).

This will be a problem, if the default value has `$pid` and be
stored in the may_reexpand_meta map, and then the code set a new
different value, which may have no `$pid`, from CLI or config file.
The reexpand will override it with the default value always.

This will do not duplicate the options with CONF_OVERRIDE priority
when reexpanding, just refresh them and call the observers. And the
finalize_reexpand_meta() will always be called after the fork() is
done in child processes.

Fixes: https://tracker.ceph.com/issues/48240
Signed-off-by: Xiubo Li <xiubli@redhat.com>
(cherry picked from commit 56ca87ae3b3c341a78a8df8f95d3bf31828c9108)

common/config_proxy: clear the values.changed in _gather_changes()

Signed-off-by: Xiubo Li <xiubli@redhat.com>
(cherry picked from commit 7496be719c3419224ac7fc29d6db4e282b193554)

global: reexpand the conf meta in all the child processes

Especially for the tools or the daemons whose config options need
to expand the '$pid', they will be always expanded with the parent
processes. We need to reexpand them in child processes just after
the fork is done.

Fixes: https://tracker.ceph.com/issues/48240
Signed-off-by: Xiubo Li <xiubli@redhat.com>
(cherry picked from commit 73a48dfe8ed165cb6c8295a88f133acb30e7065e)

Conflicts:
PendingReleaseNotes

Merge pull request #38950 from Vicente-Cheng/wip-48643-nautilus

nautilus: cephfs: client: ensure we take Fs caps when fetching directory link count from cached inode

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #38948 from Vicente-Cheng/wip-48641-nautilus

nautilus: cephfs: client: set CEPH_STAT_RSTAT mask for dir in readdir_r_cb

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #37936 from kotreshhr/wip-47823-nautilus

nautilus: mgr/volumes: Make number of cloner threads configurable

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #38952 from smithfarm/wip-48887-nautilus

nautilus: cmake: detect and use sigdescr_np() if available

Reviewed-by: David Disseldorp <ddiss@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #38833 from smithfarm/wip-48342-nautilus

nautilus: pybind/mgr/rbd_support: delay creation of progress module events

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: trigger alert if some nodes have a MTU different than the median value

This PR intends to alert a user if a specific network is configured with a custom MTU

Fixes: https://tracker.ceph.com/issues/48748
Signed-off-by: Aashish Sharma <aasharma@redhat.com>
(cherry picked from commit 06cc0d8743f288e1ef77e96b817f0f3381ac53d3)