librbd: Endian fix for handling old image format resize requests

ResizeRequest<I>::send_update_header contains a comment
    // NOTE: format 1 image headers are not stored in fixed endian
format

This statement appears incorrect; Linux kernel code handling the old
format headers does assume they are in fixed little-endian format.

Fixed by performing a byte-swap as everywhere else; this actually
fixes test suite failures on big-endian systems.

Fixes (partially): https://tracker.ceph.com/issues/41605

Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>
(cherry picked from commit cfaa681249b70a96e31b425a81f28aca1dedfebf)
Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>

rgw,test: Add missing init_le calls

Add calls to init_le16/32/64 to properly byte-swap values on
big-endian systems when assigning to ceph_le16/32/64 variables.

Fixes (partially): https://tracker.ceph.com/issues/41605

Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>
(cherry picked from commit 34ea4311dd3653e909279cd865e0fdd0cae673d1)
Changes from cherry-picked commit:
- Omitted change to src/rgw/rgw_dencoder.cc (RGWBucketEntryPoint)
  (does not yet exist in nautilus).
Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>

librados,test: Fix incorrect use of __le16/32/64

Use ceph_le16/32/64 instead of __le16/32/64 (which are no-op outside
of kernel code).

Fixes (partially): https://tracker.ceph.com/issues/41605

Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>
(cherry picked from commit f798cc9d73d9fb83705cfe01d41a4c0ed7fa2fea)
Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>

transaction: Fix incorrect use of __le16/32/64

Use ceph_le16/32/64 instead of __le16/32/64 (which are no-op outside
of kernel code).

Note that this updates only those uses of __le16/32/64 which are
part of data structures that are serialized to disk/network
(i.e. Transaction::Op and Transaction::TransactionData).

Also note that code in this file performs combined operations on
little-endian values (in particular ++, +=, and |=) which are not
supported on the ceph_le16/32/64 class, and are therefore replaced
by more primitive operations.

Fixes (partially): https://tracker.ceph.com/issues/41605

Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>
(cherry picked from commit 7d3cd10edfbdf9de6344430b0e336c351a903bfc)
Changes from cherry-picked commit:
- Duplicated changes into src/crimson/os/Transaction.h and
  src/os/ObjectStore.h (are merged into src/os/Transaction.h
  in mainline).
Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>

bluestore: Fix incorrect use of __le16/32/64

Use ceph_le16/32/64 instead of __le16/32/64 (which are no-op outside
of kernel code).

Fixes (partially): https://tracker.ceph.com/issues/41605

Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>
(cherry picked from commit ec84342fd8a2667646f1fe689a23af70fc7ed32a)
Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>

async,crimson: Add missing init_le calls

Add calls to init_le16/32/64 to properly byte-swap values on
big-endian systems when assigning to ceph_le16/32/64 variables.

Fixes (partially): https://tracker.ceph.com/issues/41605

Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>
(cherry picked from commit 62def3ec70434694fcf918c00666ba0e5f9325c2)
Changes from cherry-picked commit:
- Omitted changes to src/crimson/net/ProtocolV2.cc
  (file does not yet exist in nautilus).
Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>

msg: Fix incorrect use of __le16/32/64

Use ceph_le16/32/64 instead of __le16/32/64 (which are no-op outside
of kernel code).

Fixes (partially): https://tracker.ceph.com/issues/41605

Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>
(cherry picked from commit ede1e851604bb55db4cc29f56ae9520c5d1ec727)
Changes from cherry-picked commit:
- Additional changes in src/msg/xio/XioMsg.h were necessary
  (the file was deleted in mainline).
Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>

messages: Fix incorrect use of __le16/32/64

Use ceph_le16/32/64 instead of __le16/32/64 (which are no-op outside
of kernel code).

Fixes (partially): https://tracker.ceph.com/issues/41605

Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>
(cherry picked from commit 4dd426483294a22f2383aedec14e24ab0bc408c8)
Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>

mds: Fix incorrect use of __le16/32/64

Use ceph_le16/32/64 instead of __le16/32/64 (which are no-op outside
of kernel code).

Fixes (partially): https://tracker.ceph.com/issues/41605

Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>
(cherry picked from commit ab0942491729edf950576d640fb6a6b099b556e6)
Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>

cephx: Fix incorrect use of __le16/32/64

Use ceph_le16/32/64 instead of __le16/32/64 (which are no-op outside
of kernel code).

Note that I've also changes cephx_calc_client_server_challenge to
use ceph_le64 instead of manually byte-swapping by mswab.  (This
is a no-op, but it seems more consistent to use the ceph_le types
throughout.)

Fixes (partially): https://tracker.ceph.com/issues/41605

Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>
(cherry picked from commit 2154be1a802a518aa80b5a29ad34eedfd4761aee)
Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>

checksum: Fix incorrect use of __le16/32/64

Use ceph_le16/32/64 instead of __le16/32/64 (which are no-op outside
of kernel code).

Note that I've changed the Alg::calc routines to return the
init_value_t type instead of value_t, to avoid having to
introduce explicit byte-swapping calls to init_le16/32/64 in
many places.  (This way, the byte-swapping happens implicitly
at the very end, wher the init_value_t value is assigned via
a value_t pointer.)

Fixes (partially): https://tracker.ceph.com/issues/41605

Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>
(cherry picked from commit 63609fd1aef9811d9d3c9bd27bb9d2c88f1c2fe8)
Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>

include: Fix new-style encoding routines on big-endian

The new-style encoding routines (denc.h) are broken on big-endian
systems.  While there is a lot of infrastucture in place to
recognize data types that need to be byte-swapped during encoding
and decoding on big-endian systems, nothing is actually ever swapped.

Fixed by using ceph_le16/32/64 instead of __le16/32/64 (which are
no-ops outside of kernel code).

Fixes (partially): https://tracker.ceph.com/issues/41605

Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>
(cherry picked from commit 754b82ee15e6d1b52bc83011c2d7680bdb1d3830)
Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>

include: Simplify usage of init_le16/32/64 routines

These routines currently just return plain __u16/32/64.  This patch
changes them to return ceph_le16/32/64 types instead.  This has a
number of benefits, in particular it allows the routines to now be
used to directly initialize variables of ceph_le16/32/64 type, as
one would expect from the names of those routines.

This doesn't make much of a difference in the current code base,
but it simplifies future patches to fix endian issues.

Fixes (partially): https://tracker.ceph.com/issues/41605

Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>
(cherry picked from commit c576bb28a7744fac6d0bb02cc67c7e6542218e95)
Changes from cherry-picked commit:
- Change in src/msg/msg_types.h (ceph_entity_name) not needed
Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>

include: Endian fix for shared kernel/user headers

Endian swapping is done differently in kernel space vs. user space,
but a few header files are shared between those two user cases.

Current code attempts to handle this by re-defining __le16/32/64
before pulling those headers into user space, but this is not
consistenly done: when ceph_fs.h is included via types.h, the
redefinition happens, but when ceph_fs.h is directly included,
the redefinition does not happen.

Fix this by performing the same redefinition directly *in* those
shared headers, when included by user space.

Note that the redefines where also in effect for rbd_types.h,
which is *not* shared with the kernel, so in the file I'm simply
replacing __le16/32/64 with ceph_le16/32/64 in-line.

Also note that it is now no longer possible to include any of
the three files using the redefined macros in C code (as they
make use of C++ features).  However, this currently happens in
exactly one file, src/mds/locks.c, which only uses a few CEPH_CAP_...
constants from the header. To fix this, I've simply duplicated
those definitions, which are unchangeable ABI constants anyway.

Fixes (partially): https://tracker.ceph.com/issues/41605

Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>
(cherry picked from commit 743ed43a785795412cdda16e3d25b12cb9228a32)
Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>

msg: fix addr2 encoding for sockaddrs

Currently, the sockaddr sa_family field is being sent across the wire
without any sort of endianness conversion.

Redefine the new-style entity_addr_t encoding to encode the sa_family
field as little-endian. This will allow LE machines in the field to
keep limping along, but note that BE machines may break until they are
all fixed.

To do this, use the code that was initially written for BSD/Apple
machines to encode the different fields separately.

Fixes: http://tracker.ceph.com/issues/40114
Signed-off-by: Jeff Layton <jlayton@redhat.com>
(cherry picked from commit 7b4b70b6853e036379a0aababec598d59edcecee)
Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>

Merge pull request #30579 from yuvalif/wip-yuval-backport-notif-nautilus

nautilus: rgw/pubsub: backport notifications and pubsub

Merge pull request #30601 from smithfarm/wip-42070-nautilus

nautilus: build/ops: install-deps.sh: add EPEL repo for non-x86_64 archs as well

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge branch 'nautilus' into wip-yuval-backport-notif-nautilus

Merge pull request #30686 from smithfarm/wip-40131-nautilus-follow-on

nautilus: doc: fix urls in posix.rst

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

install-deps.sh: add EPEL repo for non-x86_64 archs as well

Fixes: https://tracker.ceph.com/issues/42023
Signed-off-by: Kefu Chai <kchai@redhat.com>
Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit 49d740118f9a0ee6b6bb82954630c654946f1751)

Merge pull request #30322 from theanalyst/wip-41700-nautilus

nautilus: build/ops: install-deps.sh: install `python*-devel` for python*rpm-macros

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #30245 from smithfarm/wip-41711-nautilus

nautilus: doc/ceph-kvstore-tool: add description for 'stats' command

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc: fix urls in posix.rst

Signed-off-by: Jos Collin <jcollin@redhat.com>
(cherry picked from commit c5c5f031f77f778af73f8659b01695b88f158807)

Merge pull request #30664 from smithfarm/wip-42105-nautilus

nautilus: build/ops: admin/build-doc: use python3

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30004 from smithfarm/wip-41568-nautilus

nautilus: doc: pg_num should always be a power of two

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #30003 from smithfarm/wip-41529-nautilus

nautilus: doc: default values for mon_health_to_clog_* were flipped

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #29731 from pdvian/wip-41081-nautilus

nautilus: doc: Fix rbd namespace documentation

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

admin/build-doc: use python3

to address https://github.com/sphinx-doc/sphinx/issues/3620, we need to
use sphinx with its fix at
https://github.com/sphinx-doc/sphinx/commit/e049f86b2de1cfdf8a74c88dc9593d047c85d5cb
in other words, we need to use sphinx v2.0.0 and up. but sphinx 2.0
requires python >= 3.5, so we have to use python3 for building the
documents.

in this change:

* doc-requirements.txt: install python3 packages on debian derivatives
* build-doc: install python3.6 packages from EPEL7, and use python3
  venv for using sphinx2
* doc-requirements.txt: bump up all python packages to latest
  stable.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit ace8cb15a600daf5261c0ab349b43010a238a1d9)

Merge PR #30025 into nautilus

* refs/pull/30025/head:
doc: cephfs: add section on fsync error reporting to posix.rst

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #30649 from lordcirth/wip-doc-misplaced-ratio

doc: max_misplaced option was renamed in Nautilus

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

doc: max_misplaced was renamed in Nautilus

mgr/balancer/max_misplaced was changed to target_max_misplaced_ratio
in 14.2.0.

Signed-off-by: Nathan Fish <lordcirth@gmail.com>

Merge pull request #30554 from alfredodeza/nautilus-guits-41392

nautilus ceph-volume: do not fail when trying to remove crypt mapper

Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@redhat.com>

Merge pull request #30049 from smithfarm/wip-41333-nautilus

nautilus: rpm: always build ceph-test package

Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.de>

build-doc: allow building docs on fedora 30

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit e17e4bfcc048abd1757c3efd12c7443169d67477)

rgw/pubsub: backporting pubsub/notifications to nautilus

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>

rgw/pubsub: add notification filtering

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit 543f8f7a2729ab8bd2e1a14d98a3fd2c2e19f9bd)

rgw/pubsub: support deletion markers and multipart upload

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit 923799fd960f97b8ddbbe0e106139069d640588b)

rgw/pubsub: push notifications documentation

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit c345566831c72446c8250f630337825e526847a8)

rgw/pubsub: push notifications from ops

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit dc31b030ea30f3b7bfdeb9fa92e303892d17138a)

rgw/pubsub: allow pubsub REST API on master

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit a14353496d1f75a303495f538141a1a54ff28c60)

rgw/pubsub: service reordering issue

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit e91489f443b5b086ef96b51051d2726728412ff5)

rgw/pubsub: fix amqp topic bug. add disabled end2end push tests

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit abfd6f72db3b6e1a797a660a4a601a33605e5f25)

rgw/pubsub: fix duplicates due to multiple zone synching

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit 38c23833b35279cb6e1a9b575b5a463959fed348)

rgw/pubsub: run pubsub tests even if multisite fails

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit 1df809144c2ed96a715c8bd3df6ed5ef48399dd3)

rgw/pubsub: make new PSZone parameters optional in test

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit 0b6909c1e9832d7a6f375f38f32bed58d1e161c2)

rgw/pubsub: add conf parameter for full/incremental sync

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit af4c2d5dbb42170ade4ad78088b80b0b5a357c5a)

rgw: add tenant as parameter to User in multisite tests

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit 4def929395bce39e9a130d0e8740790815a6a503)

rgw/pubsub: revert the RGWSysObjectCtx change

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit 10b8b57d723971604cbabf984291232c194a55ec)

rgw/pubsub: cleanup tests for multiple notifications

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit b4005db82da1627a03d96baa45081fbb329070b1)

rgw/pubsub: fix doc on updates. attempt to fix multi-notifications

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit 3279a786ba0abc9d1955f03eaa5f94da0d12ed22)

rgw/pubsub: fix more test issues with teuthology failures

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit 51fc8a40e3484bcc16244723b091ba7f3157abfb)

rgw/pubsub: fix test issue with 3 zones

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit 83e5571c873a6f459aaf9a7c42a532339ed2a1b3)

rgw/pubsub: add another zone for pubsub

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit d8d817a6006e38b52f922b714870c32b9a76a169)

rgw/pubsub: make sure bucket is empty before deletion

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit 588e0c0e23f66eb35a056ad110404a97fa219a15)

rgw/pubsub: fix "no module named rgw_multi.zone_ps"

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit dfa501cfb77e72fe3e4cb2454aea99916705b812)

rgw/pubsub: fix "no module named rgw_multi.tests"

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit b03a56fe930be0ff7bad4e375e22a8e2ca0fed76)

rgw/pubsub: actually adding the ps tests

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit 8362ecc7dfd8056567b74fdaaa52809082f15421)

rgw/pubsub: fix popping the wrong key

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit 2d7c07bbeb9887d76b89eded7458ac7af8068026)

rgw/pubsub: fix topic arn. tenant support to multisite tests

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit be64adf916e0f318b22b57cedd02c2b0b2a503f2)

rgw/pubsub: handle subscription conf errors better

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit f4e3745bcb9f75814b4052e2b651c25a39c2f17b)

rgw/pubsub: more info on notification deletion compatibility

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>
(cherry picked from commit 2cf1872bdf63e9093884b6d3bbc20b9a42963d53)

rgw/pubsub: fix comments from PR #27493

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>
(cherry picked from commit cb256dd92ac39baddce252acb634b587653da4f3)

rgw/pubsub: clarify pubsub zone configuration

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>
(cherry picked from commit 89ddc407b48959831450638f14929f553511d240)

rgw/pubsub: wrong link in S3 doc

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit 98e6a0434609d7b29d61166276419eca1d552ad0)

rgw/pubsub: fix documentation link errors

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit dc904249a00aacecf9a06a26223b5b2d5c46ace2)

rgw/pubsub: fix comments from PR #27091

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit a0b43fa26c9ba1bebe6553c84fd98fad3a3d28ad)

rgw/pubsub: add more S3 compatibility documentation

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit e1e39d627ec9a044f9f7428400dd379350fc4bbb)

rgw/pubsub: test and doc bucket deletion impact on notifications

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit cb6e894937098999e4eb7d2c13c2f180329f2432)

rgw/pubsub: avoid static creation of amqp manager

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit 376a3df0d12b544a0378a1467fcf4b2c32fd8afa)

rgw/pubsub: add s3-compatible API documentation

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit 9fd0f873188532c7552f087f65c5b279eb1fc3ca)

rgw/pubsub: allow for endpoint definition via topics

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit 8ac119fdd50b6306e4b338e3f0b4731818c6e6c8)

rgw/pubsub: implement S3 compatible get/delete APIs

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit 0f8671b3a13af2713fed51f15c83cca8ef19dd19)

rgw: pubsub support s3 records. refactor ARN

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit 2bd353233112ead32181b94a2af1b04e3fa9e1de)

ceph-volume: do not fail when trying to remove crypt mapper

In a containerized context, at some point, need to run `simple scan` on a device
from a separate container (not the existing and running corresponding container
to that device), but this can't work because when it tries to remove the
mapper which is still in use by the corresponding running osd container,
it fails.
This can be a bit more permissive and simply throw a warning.

Closes: https://tracker.ceph.com/issues/41392
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 80d3e4ca054b4d1334c2fb99fede46ab54ca4305)

rgw/pubsub: initial version of S3 compliant API

(cherry picked from commit 2e94f4d1cecb5164bc646fb6b088dd197f878263)
Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>

Merge pull request #30304 from alfredodeza/nautilus-ceph-volume-zap-fix

nautilus ceph-volume lvm.zap fix cleanup for db partitions

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #30520 from smithfarm/wip-41956-nautilus

nautilus: ceph-volume: systemd fix typo in log message

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: systemd fix typo in log message

Fixes: https://tracker.ceph.com/issues/41942
Signed-off-by: Manu Zurmühl <m.zurmuehl@heinlein-support.de>
(cherry picked from commit 8d9780ce50ef5d986c4994ed6ba5629be0bbfb74)

Merge PR #29926 into nautilus

* refs/pull/29926/head:
mgr/volumes: return string type to ceph-manager
qa/tasks: test for prevent negative subvolume size
mgr/volumes: prevent negative subvolume size
mgr/volumes: drop unnecessary size
mgr/volumes: cleanup FS subvolume or subvolume group path
mgr/volumes: give useful error message

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/volumes: return string type to ceph-manager

The 'volumes' ceph-mgr module returns a byte type instead of a string
type as the stdout result of commands, `fs subvolume getpath` and
`fs subvolumegroup getpath` to the ceph-mgr. This causes the commands
to hang and the ceph-mgr to utilize 100% CPU. Fix the 'volumes'
ceph-mgr module to return string type to ceph-mgr as the stdout
result of the commands.

Introduced-by: 3d63cd947f55a8508f972ee249ea6a04836429ae
Fixes: https://tracker.ceph.com/issues/41903
Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 88aa0576a63c5b2e8d4478951b4f594fb5047b52)

qa/tasks: test for prevent negative subvolume size

Fixes: https://tracker.ceph.com/issues/41617
Signed-off-by: Jos Collin <jcollin@redhat.com>
(cherry picked from commit 9c6be9586166c3092aabefbfcd3f48e81e3d32af)

mgr/volumes: prevent negative subvolume size

Fixes: https://tracker.ceph.com/issues/41617
Signed-off-by: Jos Collin <jcollin@redhat.com>
(cherry picked from commit 8a1249626b56f345e20f1a7058aa70da9fb82576)

mgr/volumes: drop unnecessary size

Fixes: https://tracker.ceph.com/issues/41752
Signed-off-by: Jos Collin <jcollin@redhat.com>
(cherry picked from commit 6c4154b1bbdb9deeb0420a604def43e9f52aa4a0)

mgr/volumes: cleanup FS subvolume or subvolume group path

... on best effort basis when FS subvolume or subvolumegroup create
command fails.

Fixes: https://tracker.ceph.com/issues/41371
Signed-off-by: Ramana Raja <rraja@redhat.com>
(cherry picked from commit ecf53a3527ed7561ddceeed75ed4aef1daf8230d)

mgr/volumes: give useful error message

... when creating FS subvolume or subvolume group with invalid
data pool layout.

Fixes: https://tracker.ceph.com/issues/41337
Signed-off-by: Ramana Raja <rraja@redhat.com>
(cherry picked from commit cbe48308e9e6b7e1b673e11101fd7e1403f52019)

Merge pull request #29487 from rhcs-dashboard/nautilus-run-dashboard-api-tests

nautilus: mgr/dashboard: run-backend-api-tests.sh improvements

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge tag 'v14.2.4' into nautilus

v14.2.4

This was required because the release was done with ceph-volume changes
only, and omitting the other code changes that were untested.

mgr/dashboard: run-backend-api-tests.sh CI improvements

As there is now a jenkins job to run this script
(see https://github.com/ceph/ceph-build/pull/1351),
this refactoring adapt the script to be run in a jenkins job as well as locally.

Signed-off-by: Alfonso Martínez <almartin@redhat.com>
(cherry picked from commit 7563f8b6d661886b7b3c7a259b70433a8962f55a)

pybind/mgr: install setuptools >= 12

otherwise we have
```
  Running setup.py egg_info for package apache-libcloud

    libcloud/utils/py3.py:76: UserWarning: Missing backports.ssl_match_hostname package
      warnings.warn("Missing backports.ssl_match_hostname package")
    Traceback (most recent call last):
...
      File "build/bdist.linux-x86_64/egg/setuptools_scm/version.py", line 66, in _warn_if_setuptools_outdated

...
setuptools_scm.version.SetuptoolsOutdatedWarning: your setuptools is too old (<12)
...
Command python setup.py egg_info failed with error code 1 in /tmp/tmp.i95Fg82Ea4/venv/build/apache-libcloud
```

and more important thing is, setuptools >= 12 allows us to install the
python modules in venv, without uninstalling existing modules if they
are also installed in by system packages.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 2094c2f2fd6ac0f2028fcaf22f2b51a306e60172)

mgr/dashboard: set python binary for teuthology venv

* Added env. var. to set python binary when creating teuthology venv.
* venv/bin/activate: run after setting PATH (as it does the last PATH export).

Signed-off-by: Alfonso Martínez <almartin@redhat.com>
(cherry picked from commit 522cbe8a731cf01a48dd57b6169c66e4965ba272)

14.2.4

14.2.4

ceph-volume tests create a test file for checking unicode output

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests create a shell test for functional unicode

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests verify new logging fallback and encodings in terminal

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume create a logger for the terminal

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: instantiate the new terminal logger in main()

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume terminal remove unicode stream handler

This caused problems in environments where stderr was redirected, since
stderr sets the encoding to None. Getting it back again allows
everything to work correctly, and keeps all the current unit tests
passing

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #29769 from alfredodeza/nautilus-bz-1738379

nautilus ceph-volume: use the OSD identifier when reporting success

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #30294 from alfredodeza/nautilus-rm41378-2

nautilus ceph-volume: pre-install python-apt and its variants before test runs

Reviewed-by: Jan Fajerski <jfajerski@suse.com>