ceph-volume: consider /dev/root mounted

Fixes: https://tracker.ceph.com/issues/50604
Signed-off-by: David Caro <david@dcaro.es>
(cherry picked from commit c7b959c35ff463cfbffa2abf0f56a1c1a5f594c8)

Merge pull request #42493 from dsavineau/wip-51852-pacific

pacific: ceph-volume: lvm batch: fast_allocations(): avoid ZeroDivisionError

Merge pull request #42546 from dsavineau/wip-51958-pacific

pacific: ceph-volume/tests: retry when destroying osd

Merge pull request #42606 from s0nea/wip-dashboard-pacific-translations

mgr/dashboard: update translations for pacific

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #42298 from callithea/wip-51636-pacific

pacific: monitoring: fix Physical Device Latency unit

Reviewed-by: Waad Alkhoury <walkhour@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: clwluvw <NOT@FOUND>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #42642 from cfsnyder/wip-51696-pacific

pacific: rgw: fail as expected when set/delete-bucket-website attempted on a non-exis…

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42566 from cfsnyder/wip-51514-pacific

pacific: rgw/notifications: support metadata filter in CompleteMultipartUploa…

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #41367 from cbodley/wip-50845

pacific: rgw: deprecate the civetweb frontend

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #42637 from cfsnyder/wip-51779-pacific

pacific: rgw : add check for tenant provided in RGWCreateRole

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42635 from cfsnyder/wip-50645-pacific

pacific: rgw: allow rgw-orphan-list to process multiple data pools

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42634 from cfsnyder/wip-51014-pacific

pacific: rgw: remove quota soft threshold

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42633 from cfsnyder/wip-50679-pacific

pacific: rgw: fix segfault related to explicit object manifest handling

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42631 from cfsnyder/wip-50465-pacific

pacific: rgw/notifications: delete bucket notification object when empty

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42626 from cfsnyder/wip-52007-pacific

pacific: rgw: avoid occuring radosgw daemon crash when access a conditionally …

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42625 from cfsnyder/wip-51751-pacific

pacific: RGW Zipper - Make sure bucket list progresses

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42638 from cfsnyder/wip-50708-pacific

pacific: rgw: fix bucket object listing when marker matches prefix

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42649 from cfsnyder/wip-50379-pacific

pacific: rgw/amqp/test: fix mock prototype for librabbitmq-0.11.0

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42647 from cfsnyder/wip-49555-pacific

pacific: rgw/notification: add exception handling for persistent notification thread

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42646 from cfsnyder/wip-50463-pacific

pacific: rgw/multisite: return correct error code when op fails

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42645 from cfsnyder/wip-52051-pacific

pacific: rgw: when deleted obj removed in versioned bucket, extra del-marker added

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42644 from cfsnyder/wip-51804-pacific

pacific: rgw/http/notifications: support content type in HTTP POST messages

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42641 from cfsnyder/wip-50367-pacific

pacific: rgw: during reshard lock contention, adjust logging

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42640 from cfsnyder/wip-51801-pacific

pacific: rgw: radosgw-admin errors if marker not specified on data/mdlog trim

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #42639 from cfsnyder/wip-51781-pacific

pacific: rgw : modfiy error XML for deleterole

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42656 from cfsnyder/wip-51785-pacific

pacific: rgw multisite: metadata sync treats all errors as 'transient' for retry

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42655 from cfsnyder/wip-50803-pacific

pacific: radosgw-admin: skip GC init on read-only admin ops

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42654 from cfsnyder/wip-50731-pacific

pacific: rgw/rgw_file: Fix the return value of read() and readlink()

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42653 from cfsnyder/wip-50728-pacific

pacific: rgw : add check empty for sync url

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42652 from cfsnyder/wip-50711-pacific

pacific: rgw: fix for mfa resync crash when supplied with only one totp_pin.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42651 from cfsnyder/wip-51771-pacific

pacific: test/rgw: fix use of poll() with timers in unittest_rgw_dmclock_scheduler

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42650 from cfsnyder/wip-50642-pacific

pacific: rgw/sts: read_obj_policy() consults iam_user_policies on ENOENT

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42648 from cfsnyder/wip-50094-pacific

pacific: librgw/notifications: initialize kafka and amqp

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: run multisite tests with metadata sync error injection

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit cff6bf37e65964d2d987231598659f08a84b9410)

rgw: limit concurrency of metadata sync

limit the number of concurrent RGWMetaSyncSingleEntryCRs that each mdlog
shard is allowed to spawn. use META_SYNC_SPAWN_WINDOW=20 to match data-
and bucket sync

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit a07add0a5408f0480e93ea5920bc013119ba6401)

rgw: metadata sync treats all errors as 'transient'

collect_children() had a special case for EAGAIN that it treated as
a 'transient' error, which set the can_adjust_marker = false to bail out
of RGWMetaSyncShardCR and retry from the previous marker

but the http client doesn't return EAGAIN - rgw_http_error_to_errno()
defaults to EIO - so this retry logic based on can_adjust_marker never
runs. on any other error, RGWMetaSyncSingleEntryCR would not call
marker_tracker->finish() to advance the sync status marker, and
RGWMetaSyncShardCR would continue on with full- or incremental sync
without ever attempting to retry the failed entries

a detailed comment in collect_children() describes a different strategy
for handling 'permanent' errors, but that was never fully elaborated.
i also don't think there's a reasonable way to differentiate between
transient and permanent errors, so this treats all errors as transient
to be retried

if an error really is permanent for a given metadata key, metadata sync
will get stuck there and require manual intervention

Fixes: https://tracker.ceph.com/issues/39657
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 866d66b8749b28ec626a8d0adba3d14fdd8abead)

radosgw-admin: skip GC init on read-only admin ops

Fixes: https://tracker.ceph.com/issues/50520
Signed-off-by: Mark Kogan <mkogan@redhat.com>
(cherry picked from commit 9ac1991fc798af7e0ba0fac18209b71b5ae3f02b)

Conflicts:
src/rgw/rgw_admin.cc
src/rgw/rgw_rados.h
src/rgw/rgw_sal.cc
src/rgw/rgw_sal.h

Cherry-pick notes:
- Conflicts due to move from of RGWStoreManager in rgw_sal_rados.h to StoreManager in rgw_sal.h

rgw/rgw_file: Fix the return value of read() and readlink()

Fixes: https://tracker.ceph.com/issues/49189
Signed-off-by: Dai zhiwei <daizhiwei3@huawei.com>
Signed-off-by: luo rixin <luorixin@huawei.com>
(cherry picked from commit bfd83e8fa142873a0bdf09a4d1ad1b04127f5885)

rgw : add check empty for sync url

Fixes: https://tracker.ceph.com/issues/50103
Signed-off-by: caolei <halei15848934852@163.com>
(cherry picked from commit 3a4e0b79310b21eeee37043d5419887bb41c0cf6)

rgw: fix for mfa resync crash when supplied with only one totp_pin.

The fix returns an appropriate error message.

fixes: https://tracker.ceph.com/issues/50394

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>
(cherry picked from commit 1b943419c45a9b17be71668bd023aeb814171b2c)

test/rgw: use spawn library for test_rgw_dmclock_scheduler

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit a8e3589a2c875b6fadc853c75f20cb9256f294ca)

test/rgw: fix use of poll() with timers in unittest_rgw_dmclock_scheduler

the AsyncScheduler uses an asio timer to dispatch work to its executor
with an optional delay. when no delay is requested, it waits on the
timer with an expiration time in the past (crimson::dmclock::TimeZero)

tests are failing here because poll() is returning without executing the
handlers of those expired timers

asio implements these timers with timerfd and epoll. debugging with
strace, i see that these timers armed with timerfd_settime() are not
always immediately ready according to epoll_wait():

  eventfd2(0, EFD_CLOEXEC|EFD_NONBLOCK)   = 3
  epoll_create1(EPOLL_CLOEXEC)            = 4
  timerfd_create(CLOCK_MONOTONIC, TFD_CLOEXEC) = 5
  epoll_ctl(4, EPOLL_CTL_ADD, 3, {events=EPOLLIN|EPOLLERR|EPOLLET, data={u32=14164052, u64=14164052}}) = 0
  epoll_ctl(4, EPOLL_CTL_ADD, 5, {events=EPOLLIN|EPOLLERR, data={u32=14164064, u64=14164064}}) = 0
  timerfd_settime(5, TFD_TIMER_ABSTIME, {it_interval={tv_sec=0, tv_nsec=0}, it_value={tv_sec=0, tv_nsec=1}}, {it_interval={tv_sec=0, tv_nsec=0}, it_value={tv_sec=0, tv_nsec=0}}) = 0
  epoll_wait(4, [{events=EPOLLIN, data={u32=14164052, u64=14164052}}], 128, 0) = 1
  epoll_wait(4, [], 128, 0)               = 0
  epoll_wait(4, [], 128, 0)               = 0
  epoll_wait(4, [], 128, 0)               = 0
  epoll_wait(4, [], 128, 0)               = 0
  epoll_wait(4, [{events=EPOLLIN, data={u32=14164064, u64=14164064}}], 128, 0) = 1

in this example, it took 6 calls to context.poll() before it was ready
to execute the timer's handler

to work around this, replace calls to context.poll() with calls to
context.run_for() with a very short duration

Fixes: https://tracker.ceph.com/issues/42788
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 21baed999e31c5e69c75f0cbb8757ef91585d917)

rgw: read_obj_policy() consults iam_user_policies on ENOENT

when the head object doesn't exist, read_obj_policy() has to decide
whether to return ENOENT or EACCES

when there's a bucket policy, we check whether it has s3ListBucket
permissions. when there's an assumed role, we also need to check
against the role's policies in s->iam_user_policies

Fixes: https://tracker.ceph.com/issues/49780
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 5dc9375fa1888242f388f8b502f445f3ddc891f7)

rgw/amqp/test: fix mock prototype for librabbitmq-0.11.0

also use extern C for to get compilation errors when
function prototype change

Fixes: https://tracker.ceph.com/issues/50291
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>
(cherry picked from commit 2ba598ec4c294bd09d2df18ccd2096382e303d39)

librgw/notifications: initialize kafka and amqp

Fixes: https://tracker.ceph.com/issues/49738
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>
(cherry picked from commit 178f6bdac97b57300bbe0956633cf686a7e3ccee)

rgw/notification: add exception handling for persistent notification thread

Fixes: https://tracker.ceph.com/issues/49322
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>
(cherry picked from commit 915963ecb9effcb1f2d38f444c1bb9307f8ffbe1)

Conflicts:
src/rgw/rgw_notify.cc

rgw/multisite: return correct error code when op fails

when trying to disable/enbale sync on non-master zone

Fixes: https://tracker.ceph.com/issues/50201
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>
(cherry picked from commit 83e89dfa3358fe91597d6714483f96b21a234ae6)

rgw: when deleted obj removed in versioned bucket, extra del-marker added

After initial checks are complete, this will read the OLH earlier than
previously to check the delete-marker flag and under the bug's
conditions will return -ENOENT rather than create a spurious delete
marker.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 69d7589fb1305b7d202ffd126c3c835e7cd0dda3)

rgw/http/notifications: support content type in HTTP POST messages

Fixes: https://tracker.ceph.com/issues/51530
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>
(cherry picked from commit 6a1688b57d1e329cecd0c26c494cb08c9a4c3079)

Merge pull request #42629 from rhcs-dashboard/wip-52049-pacific

pacific: mgr/dashboard: show perf. counters for rgw svc. on Cluster > Hosts

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #42628 from rhcs-dashboard/wip-52048-pacific

pacific: mgr/dashboard: fix ssl cert validation for rgw service creation

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

rgw: fail as expected when set/delete-bucket-website attempted on a non-existent bucket, rgw should return HTTP 404 and NoSuchBucket.

Fixes: https://tracker.ceph.com/issues/51536
Signed-off-by: xiangrui meng <mengxr@chinatelecom.cn>
(cherry picked from commit c623aa45d35b269c6701a57e44ac05bb29a79dc8)

rgw: during reshard lock contention, adjust logging

When RGW fails to get a lock on a reshard log, we log it in such a way
that it looks like an error. Instead we'll make sure that the log
message is informational.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 6d3dee37791ad427a3435c493a1d7874ba075674)

rgw: remove quota soft threshold

Remove quota soft threshold, which causes expensive checks for sharded buckets

Fixes: 14eabd4aa7b8a2e2c0c43fe7f877ed2171277526
Signed-off-by: Zulai Wang <wangzl31@outlook.com>
(cherry picked from commit 32a39705765af0f87bec9101e5d337b797e05fea)

Conflicts:
src/common/options/rgw.yaml.in

Cherry-pick notes:
- Options defined in src/common/options.cc in Pacific vs src/common/options/rgw.yaml.in

rgw: radosgw-admin errors if marker not specified on data/mdlog trim

Check that a marker was specified and trim if we don't have one.

Also: In a world where we're parsing for generation, it doesn't really
make sense to have a 'no marker specified' as separate from a marker
that is just an empty string.

Also: Successful datalog trim returns zero, not
-ENODATA, and radosgw-admin should expect this.

Fixes: https://tracker.ceph.com/issues/51712
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit 4cb6fcf7f4e9548a8a0dd017b49a6ea23bedffd6)

Conflicts:
src/rgw/rgw_admin.cc

Cherry-pick notes:
- static_cast for RadosStore not needed in Pacific

rgw : modfiy error XML for deleterole

Fixes: https://tracker.ceph.com/issues/51157
Signed-off-by: caolei <halei15848934852@163.com>
(cherry picked from commit c7ab6579c7655352d08a4c12fc3a6951217dbe6f)

rgw: fix bucket object listing when marker matches prefix

When an iniitial marker that ends with a delimiter is provided, it
prevents listing of that "subdirectory" due to new logic at the cls
level to make listing more efficient. The fix catches that situation.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 220ef4b22d1d1667eb4f2c300a0b788e87b9067d)

rgw : add check for tenant provided in RGWCreateRole

Fixes: https://tracker.ceph.com/issues/51206
Signed-off-by: caolei <halei15848934852@163.com>
(cherry picked from commit 3c99ac14080c9f5b1611c9bbe4a223a9fd2927a0)

Conflicts:
src/rgw/rgw_rest_role.cc

Cherry-pick notes:
- Pacific constructs role explicitly vs. using store->get_role(), and does not wrap in a unique_ptr

rgw: allow rgw-orphan-list to process multiple data pools

We now allow multiple data pools to be specified either on the
command-line or at the prompt. The list should be space-separated.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit bde1278a5de3e9363c0f136b6ebc913bbb5e269d)

rgw: fix segfault related to explicit object manifest handling

Fixes: https://tracker.ceph.com/issues/50467
Signed-off-by: Mark Kogan <mkogan@redhat.com>
(cherry picked from commit ae00453515b17d25a5d42430e744855b020ccec6)

rgw/notifications: delete bucket notification object when empty

Fixes: https://tracker.ceph.com/issues/49650
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>
(cherry picked from commit 08e9846fd5a040887e2ddbfca36da97777fc76f6)

Conflicts:
src/rgw/rgw_pubsub.cc

Cherry-pick notes:
- RGWPubSub methods take a DoutPrefixProvider as first argument in Pacific

mgr/dashboard: show perf. counters for rgw svc. on Cluster > Hosts

- Use service_map_id to retrieve perf. counters for rgw services.
- Move logic from controller to CephService.

Fixes: https://tracker.ceph.com/issues/52022
Signed-off-by: Alfonso Martínez <almartin@redhat.com>
(cherry picked from commit 69a87a1dea6fe148f2a81b1d7775f5ade9626c09)

mgr/dashboard: Cluster > Hosts > Performance Counters: do not show context bar.

Do not show the context bar (Object Gateway selection).

Fixes: https://tracker.ceph.com/issues/52022
Signed-off-by: Alfonso Martínez <almartin@redhat.com>
(cherry picked from commit 73acb9b60d23e79bdcdd03437b0a813487420e33)

mgr/dashboard: fix ssl cert validation for rgw service creation

Fixes: https://tracker.ceph.com/issues/51965
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
(cherry picked from commit 5f52c00e5291b53b4fe9ab7f9fae5d499017f91b)

rgw: avoid occuring radosgw daemon crash when access a conditionally redirected static website

Fixes: https://tracker.ceph.com/issues/51715
Signed-off-by: xiangrui meng <mengxr@chinatelecom.cn>
Signed-off-by: yupeng chen <chenyupeng@chinatelecom.cn>
(cherry picked from commit 865d13e9fa043954dce5110f4f8ae693af9604ce)

RGW Zipper - Make sure bucket list progresses

The bucket list wrapper lost the automatic advancement of the marker for
each iteration.  Some places manually set that marker, and it worked
fine, but some depended on it being automatically set.  Fix it so that
the marker advances automatically each iteration.

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>
(cherry picked from commit 9cd74053b8bd95d3dee6cac6723cb051768865c3)

Merge pull request #42585 from rhcs-dashboard/wip-51994-pacific

pacific: mgr/dashboard: cephadm-e2e job script: improvements

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: update translations for pacific

Note: it's not a backport because the change is not intended to land in
master. It's pacific related only.

Signed-off-by: Tatjana Dehler <tdehler@suse.com>

mgr/dashboard: cephadm-e2e script: improvements

Improvements and some adaptations related to the jenkins job.

Fixes: https://tracker.ceph.com/issues/51612
Signed-off-by: Alfonso Martínez <almartin@redhat.com>
(cherry picked from commit 65b75000b7694cb3cbe617bbec28c513a2522be8)

 rgw/notifications: support metadata filter in CompleteMultipartUpload events

Fixes: https://tracker.ceph.com/issues/51261
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>
(cherry picked from commit 93b9f0fb77ca5ed0b5c89d45229732850e0a0c49)

Conflicts:
src/rgw/rgw_notify.h
src/rgw/rgw_op.cc
src/rgw/rgw_sal.h
src/rgw/rgw_sal_rados.cc
src/rgw/rgw_sal_rados.h
src/test/rgw/bucket_notification/test_bn.py

Cherry-pick notes:
- differences due to renaming of rgw::sal::RGWObject to rgw::sal::Object
- differences due to use of RadosNotification in master and reservation_t in Pacific

Merge pull request #42480 from smithfarm/wip-51836-pacific

pacific: rpm: drop use of $FIRST_ARG in ceph-immutable-object-cache

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #42482 from MrFreezeex/wip-51840-pacific

pacific: osd: log snaptrim message to dout

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-volume/tests: retry when destroying osd

Sometimes, it can happen that the osds being destroyed in those tests
are not yet marked as 'down' for some reason. Let's add some retries on
those tasks to avoid CI failures.

Fixes: https://tracker.ceph.com/issues/51903
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 38882161cc77c913e3f64c786734fd713a951f93)

Merge pull request #42426 from ifed01/wip-ifed-compact-after-upgrade-pac

pacific: os/bluestore: compact db after bulk omap naming upgrade.

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #42424 from ifed01/wip-ifed-bluefs-safer-flush-pac

pacific: os/bluestore: Remove possibility of replay log and file inconsistency

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #42477 from tchaikov/pacific-pr-40961

pacific: bind on loopback address if no other addresses are available

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #42472 from tchaikov/pacific-pr-42464

pacific: cmake, ceph.spec.in: build with header only fmt on RHEL

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #42458 from vshankar/wip-51819

pacific: cephfs-mirror: record directory path cancel in DirRegistry

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #42299 from callithea/wip-51374-pacific

pacific: monitoring: Clean up Grafana dashboards

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: p-se <NOT@FOUND>

Merge pull request #42213 from rhcs-dashboard/wip-51542-pacific

pacific: radosgw: include realm_{id,name} in service map

Reviewed-by: Waad Alkhoury <walkhour@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #42363 from k0ste/wip-51699-pacific

pacific: rgw: allow to set ssl options and ciphers for beast frontend

Reviewed-by: Casey Bodley <cbodley@redhat.com>

ceph-volume: lvm batch: fast_allocations(): avoid ZeroDivisionError

 * Fixes Bug #51526 ( https://tracker.ceph.com/issues/51526 )

Signed-off-by: Jonas Zeiger <jonas.zeiger@talpidae.net>
(cherry picked from commit d83d4f35863309637c178197d8a59e746678afba)

Merge pull request #42490 from dsavineau/wip-51851-pacific

pacific: ceph-volume/tests: update ansible environment variables in tox

Merge pull request #42398 from satoru-takeuchi/wip-50900-pacific

pacific: osd/scrub: separate between PG state flags and internal scrubber operation

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #41783 from pponnuvel/wip-51140-pacific

pacific: rgw: Improve error message on email id reuse

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

ceph-volume/tests: use ansible_facts

Since inject_facts_as_vars is set to false in the ansible.cfg file then we
have to update the references to use ansible_facts[<thing>] instead of
ansible_<thing>.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
(cherry picked from commit 842fc2b605a2321a031a240c4aa4348c1be24e14)

ceph-volume/tests: use pytest rerunfailures

We already install the dependency from ceph-ansible requirements.txt and to
avoid false positive (like after rebooting a node) we can retry failing test.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
(cherry picked from commit 95056a24e4fbc19307f5b32724bfdb459a42f7ab)

ceph-volume/tests: set ANSIBLE_CONFIG env var

Without loading the ansible.cfg file from ceph-ansible project, we don't
have the pipelining enabled which can result in significant performance
improvement.
This removes the ANSIBLE_ACTION_PLUGINS, ANSIBLE_RETRY_FILES_ENABLED and
ANSIBLE_SSH_RETRIES environment variables as it is already included in the
ansible.cfg file.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
(cherry picked from commit b758fdd829e3b4d7b790e6d35a02c97f3962d13e)

ceph-volume/tests: update ansible ssh_args env var

The ansible ssh_args parameter is usually defined in the ansible.cfg file.
Currently this variable is overrided in tox to manage the vagrant ssh file
but we lost all default values.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
(cherry picked from commit 0ad615bab555d9965aec36d025acb28708b07cf0)

osd: log snaptrim message to dout

This log message is not an error and is done on every tick of the
snaptrim process. Replace the derr logging to dout(10) to not log it
by default.

Fixes: https://tracker.ceph.com/issues/51799
Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@cern.ch>
(cherry picked from commit e2b2faef11c18df6c2f1f083d32d279e51b63e82)

rpm: drop use of $FIRST_ARG in ceph-immutable-object-cache

The use of $FIRST_ARG was probably required because the SUSE-specific
%service_* rpm macros were playing tricks on the shell positional parameters.
This is bad practice and error-prone, so let's assume that no macros should do
that anymore and hence it's safe to assume that positional parameters remain
unchanged after any rpm macro call.

Thanks to Franck Bui for providing the original patch
926433f5d45e557c42f050b43798ba29dc495e02 that this patch is modeled after.

NOTE: the use of FIRST_ARG had already been eliminated by
926433f5d45e557c42f050b43798ba29dc495e02 but was re-introduced later by
9466d7098573dafcfede5e9c852374fbbd99f9e7

Fixes: 9466d7098573dafcfede5e9c852374fbbd99f9e7
Fixes: https://tracker.ceph.com/issues/51797
Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit 1cb84a1160ed4108cae30100682b1e3ee7c7721d)

common/pick_address: define in_addr_t if it is not defined

neither mingw not not have in_addr_t defined, see
https://docs.microsoft.com/en-us/windows/win32/api/winsock2/ns-winsock2-in_addr,
so define it if it is not defined.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit a06f8edeca0a1d08ff181d18f913bcfe7570bdcd)

Conflicts:
cmake/modules/CephChecks.cmake: trivial resolution

common/pick_addr: use grading machinery to refactor pick_address()

as picking iface on the same NUMA node is not a hard requirement, the
grading machinery is a nice fit for this purpose.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 329d51c68ec6bf1864aa9430a62d65a93362a1b9)

common/pick_address: prefer non-loopback addresses

instead of filtering out loopback ifaces, check for loopback addresses,
and prefer non-loopback addresses over loopback addresses.

before this change, iface named "lo" is filtered out by default,
and "lo" is allowed if `ms_bind_exclude_lo_iface` is false.

after this change, iface with address out of 127/8 is prefered.
the iface marked down is not considered.

the option of "ms_bind_exclude_lo_iface" is removed. the tests are
updated accordingly.

Fixes: https://tracker.ceph.com/issues/50456
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit a9b9bcd53215a07608a28ac2c8e4a8c8b8e80e66)

Conflicts:
src/common/options/global.yaml.in: global.yaml.in was introduced
in master only, so, in this change src/common/options.cc is updated
instead.

common/pick_address: Allow binding on loopback iface

in 6147c0917157efd2d35610e759685656a4989abb, "lo" is also skipped when
daemon is trying to find an address to bind. but that change reverts the
fix of 201b59204374ebdab91bb554b986577a97b19c36, to address the problem.

an option named "ms_bind_exclude_lo_iface" is added, it defaults to
"true". but it can be changed to false to allow daemon to bind on "lo".

Fixes: https://tracker.ceph.com/issues/50012
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 7f01d36a2ca0576f1ff103ae3fa7c3662e93b722)

common/pick_address: document find_ip_in_subnet_list()

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit b106ec0bbf7fa726062989114f461f2d0a1f93a9)

common/pick_address: pass string by reference

to silence warnings from clang-tidy.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 6d0ed81f796209f27b96811f9140b7fff16a7940)

common/pick_addr: refactor pick_address.cc and ipaddr.cc

* do not replicate the same logic in IPv4 and IPv6 paths
* use helpers returning bool for filtering the candidate addresses
  for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 52785d5a3607b2f2ee6d41069d18a154b3eb5d45)

common/pick_address: use scope_guard for freeifaddrs()

for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit c3c110b5763ac420c4b88f8a545c1c87a71ce59a)

ceph.spec.in: build with header only fmt on RHEL

because fmt is packaged in EPEL, while librados is packaged
in RHEL, so we cannot have fmt as a runtime dependency of librados.
to address this issue, we should compile librados either with static library
or with header-only library of fmt. but because the fedora packaging
guideline does no encourage us to package static libraries, and it would
be complicated to package both static and dynamic library for fmt.

the simpler solution would be to compile Ceph with the header-only
version of fmt.

in this change, we compile ceph with the header-only version of fmt
on RHEL to address the runtime dependency issue.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 5419131499f4c4b5f6ba520342c39f9766e0cf4d)

Conflicts:
ceph.spec.in: trivial resolution

cmake: add an option "WITH_FMT_HEADER_ONLY"

in this change:

* an interface library named "fmt-header-only" is introduced. it brings
  the support to the header only fmt library.
* fmt::fmt is renamed to fmt
* an option named "WITH_FMT_HEADER_ONLY" is introduced
* fmt::fmt is an alias of "fmt-header-only" if "WITH_FMT_HEADER_ONLY"
  is "ON", and an alias of "fmt" otherwise.

because fmt is packaged in EPEL, while librados is packaged
in RHEL, so we cannot have fmt as a runtime dependency of librados.
to address this issue an option "WITH_FMT_HEADER_ONLY" is introduced, so
that we can enable it when building Ceph with the header version of fmt.
and the built packages won't have runtime dependency of fmt.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit d81478b5690f9e3d36a7738223c737fc96a34d28)

Conflicts:
src/CMakeLists.txt
src/common/CMakeLists.txt: trivial resolution