Merge pull request #27938 from pdvian/wip-39206-mimic

mimic: osd: shutdown recovery_request_timer earlier

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #27963 from xiexingguo/wip-mimic-upmap-fixes

mimic: crush: backport recent upmap fixes

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #27940 from pdvian/wip-39220-mimic

mimic: osd: FAILED ceph_assert(attrs || !pg_log.get_missing().is_missing(soid) || (it_objects != pg_log.get_log().objects.end() && it_objects->second->op == pg_log_entry_t::LOST_REVERT)) in PrimaryLogPG::get_object_context()

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #27055 from ifed01/wip-ifed-fix-storetest-mimic

mimic: test/store_test: fix/workaround for BlobReuseOnOverwriteUT and garbageCollection

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #27907 from smithfarm/wip-38443-mimic

mimic: tests: osd-markdown.sh can fail with CLI_DUP_COMMAND=1

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #27943 from smithfarm/wip-38879-mimic

mimic: core: ENOENT in collection_move_rename on EC backfill target

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #27972 from smithfarm/wip-37348-mimic

mimic: rgw: when using nfs-ganesha to upload file, rgw es sync module get failed

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #27948 from smithfarm/wip-37498-mimic

mimic: rgw: get or set realm zonegroup zone should check user's caps for security

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27356 from pdvian/wip-38922-mimic

mimic: rgw: Fix S3 compatibility bug when CORS is not found

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27367 from pdvian/wip-38924-mimic

mimic: rgw: Adding tcp_nodelay option to Beast

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27421 from pdvian/wip-38959-mimic

mimic: rgw-admin: fix data sync report for master zone

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27603 from pdvian/wip-39017-mimic

mimic: rgw admin: add tenant argument to reshard cancel

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27661 from pdvian/wip-39049-mimic

mimic: rgw: beast: set a default port for endpoints

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27666 from smithfarm/wip-39359-mimic

mimic: rgw: failed to pass test_bucket_create_naming_bad_punctuation in s3test

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27796 from cbodley/wip-38713

mimic: rgw: resolve bugs and clean up garbage collection code

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #27828 from ashishkumsingh/wip-39498-mimic

mimic: rgw: admin: handle delete_at attr in object stat output

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #26140 from ashishkumsingh/wip-37909-mimic

mimic: rbd_mirror: don't report error if image replay canceled

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #27391 from pdvian/wip-38955-mimic

mimic: tests: krbd discard qa fixes

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #27588 from pdvian/wip-38976-mimic

mimic: rbd: krbd: return -ETIMEDOUT in polling

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #27958 from dillaman/wip-39585-mimic

mimic: qa/tasks/rbd_fio: fixed missing delimiter between 'cd' and 'configure'

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #27959 from dillaman/wip-39583-mimic

mimic: qa/workunits/rbd: use more recent qemu-iotests that support Bionic

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #26655 from jan--f/c-v-simple-activate-all-mimic

mimic: ceph-volume: add --all flag to simple activate

Reviewed-by: Alfredo Deza <adeza@redhat.com>

rgw: ES sync: wrap all the decode bls in try block

since decode can throw, wrap all the try block

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit 12b12ccea23871688cc4101c72f00b0575f1c01a)

Conflicts:
src/rgw/rgw_sync_module_es.cc
- mimic uses val.begin() where master has val.cbegin()

osd/OSDMap: add log for better debugging

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit a89281ffbb50a4dfc700398e864138b5faaf00f5)

osd/OSDMap: calc_pg_upmaps - restrict optimization to origin pools only

The current implementation will try to cancel any pg_upmaps that
would otherwise re-map a PG out from an underfull osd, which is wrong,
e.g., because it could reliably fire the following assert:

src/osd/OSDMap.cc: 4405: FAILED assert(osd_weight.count(i.first))

Also it would not match the expectation if automatic balancing
has been strictly restricted to some specific pools by admin.

Fix by excluding any wild PG that does not belong to the origin pools
passed in when trying to do upmap/unmap.

Fixes: http://tracker.ceph.com/issues/38897
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit 01e8e9482ce7194d347e02ef41acfa6d8d14f614)

osd/OSDMap: drop local pool filter in calc_pg_upmaps

The local pre-loaded pool filter is completely redundant since
the below check:

if (!only_pools.empty() && !only_pools.count(i.first))

could reliably catch both cases - either optimization should be
restricted to specific pools feeded, or all existing pools.

Let's clean it up.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit 3e6bad9821b5fb3e780d970666fbdfbf217d905e)

crush: fix upmap overkill

It appears that OSDMap::maybe_remove_pg_upmaps's sanity checks
are overzealous. With some customized crush rules it is possible
for osdmaptool to generate valid upmaps, but maybe_remove_pg_upmaps
will cancel them.

Fixes: http://tracker.ceph.com/issues/37968
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit 5c4d241c7f796cb685e9944bf237028162122725)

Conflicts:
- maybe_remove_pg_upmaps input changing
        - slight c++11 auto conflicts

osd/OSDMap: using std::vector::reserve to reduce memory reallocation

In C++ vectors are dynamic arrays.
Vectors are assigned memory in blocks of contiguous locations.
When the memory allocated for the vector falls short of storing
new elements, a new memory block is allocated to vector and all
elements are copied from the old location to the new location.
This reallocation of elements helps vectors to grow when required.
However, it is a costly operation and time complexity is involved
in this step is linear.
Try to use std::vector::reserve whenever possible if performance
matters.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit 4a0eabb3a65107cbee5e692ade564102e2b2f8aa)

osd/OSDMap: more improvements to upmap

- add ability of appending a 2nd, 3rd, etc... pair to existing upmaps
  when possible, rather than just continuing to the next PG
- handle the underfull case: we can rm-pg-upmap-items if there exist
  any upmaps which remapped a PG out from an underfull OSD

See-also: http://tracker.ceph.com/issues/37940
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit a7d2adf4283cc11ed567b0ad07c4076a50d7d2a0)

Conflicts:
- various option changes since luminous
- s/assert/ceph_assert/
        - s/conf./conf->/

Merge pull request #26194 from pdvian/wip-38078-mimic

mimic: multisite: bucket full sync does not handle delete markers

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #26553 from pdvian/wip-38396-mimic

mimic: rgw: when exclusive lock fails due existing lock, log add'l info

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #26615 from pdvian/wip-38413-mimic

mimic: multisite: rgw_data_sync_status json decode failure breaks automated datalog trimming

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27122 from smithfarm/wip-38886-mimic

mimic: rgw: GetBucketCORS API returns Not Found error code when CORS configuration does not exist

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27304 from pdvian/wip-38907-mimic

mimic: rgw: fix read not exists null version return wrong

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27335 from liaoxin01/mimic

mimic: rgw: fix bug of apply default quota, for this create new a user may core using beast

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27354 from pdvian/wip-38919-mimic

mimic: rgw: don't crash on missing /etc/mime.types

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #26792 from cbodley/wip-38609

mimic: qa/rgw: reduce number of multisite log shards

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #26670 from tspmelo/wip-npm-update-mimic

mimic: mgr/dashboard: Update npm packages

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #27638 from tchaikov/mimic-39132

mimic: mgr: enable inter-module calls

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

qa/workunits/rbd: use more recent qemu-iotests that support Bionic

Fixes: http://tracker.ceph.com/issues/24668
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit b184acd218b36b85f5a25678d773c35407a2b745)

qa/tasks/rbd_fio: fixed missing delimiter between 'cd' and 'configure'

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 83c11efb98f28c19f70eac04a6a84fe820f968cb)

qa/rgw: add credentials to zone 'remove' in test_zonegroup_remove()

zone credentials are required to 'period update --commit' from
--rgw-zone remove

Signed-off-by: Casey Bodley <cbodley@redhat.com>
Conflicts:
src/test/rgw/rgw_multi/tests.py
(cherry picked from commit 56021eab9584f10fa43053621cce93b623c36265)

rgw: get or set realm zonegroup zone need check user's caps

fix: https://tracker.ceph.com/issues/37352

Signed-off-by: yuliyang <yuliyang@cmss.chinamobile.com>
(cherry picked from commit 6ecaec926fb81810f6be43744cd5c48d6ccfaf5a)

Conflicts:
src/rgw/rgw_rest_config.h
src/rgw/rgw_rest_realm.cc
- mimic lacks "override" in some places

osd/PG: !transaction_applied is true for async_recovery_targets as well

Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit 71996da6be171cd310f8cae48c95bedb02f000b8)

osd/PG: change error to dout in _scan_rollback_obs()

This message is now expected during scrubbing, since we do not delete
objects during a rollforward when !transaction_applied.

Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit b8e62e6ee235e43e9930faf486ca8bea985c25b9)

osd/PG: skip rollforward when !transaction_applied during append_log()

Earlier, we did pg_log.roll_forward(&handler), when
!transaction_applied, which advanced the crt and trimmed the entries
in rollforward(). Due to this, during _merge_object_divergent_entries(),
when we tried to rollback entries, those objects were not found in the
backend, and thus we hit this bug http://tracker.ceph.com/issues/36739.

With this change, we are advancing the crt value, without deleting the
objects, so that _merge_object_divergent_entries() does not fail
because of deleted objects.

Fixes: http://tracker.ceph.com/issues/36739
Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit 17419ee39342257f5a6f41c792e8e45a8e243720)

osd/PG: move down peers out from peer_purged

In purge_strays(), we'll aggressively clear stray_set and
add all related peers into peer_purged.

However, if the corrsponding peer is down and becomes
up again, (unconditionally) adding it to peer_purged
will prevent primary from re-purging it.
(See Active::react(const MNotifyRec& notevt))

On consuming a new osdmap, let's move any down peers out from
peer_purged simutaneously. This way we can lower the risk
of leaving any leftover PGs behind.

Related-to: http://tracker.ceph.com/issues/38931
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit f7c5b01e181630bb15e8b923b0334eb6adfdf50a)

osd/PG: introduce all_missing_unfound helper

We use pg_log.missing to track each peer's missing objects separately,
whereas missing_loc records the location of all (probably existing) good copies
for both primary and replicas' missing objects. Hence an item from
pg_log.missing or missing_loc is of different meaning and is not comparable.

During recovery, we can skip recovering primary only if
- primary is good, e.g., has no missing at all
- or all of the primary's missing objects do exist in missing_loc and are
  currently unfound

Obviously, the current "all missing objects are unfound" checker is broken.
Fix by introducing an independent all_missing_unfound helper to make the
count of missing objects that are currently unfound correct.

Fixes: http://tracker.ceph.com/issues/38784
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit d9497139a6f516ce015bb43a2e7f8958638cf8f8)

osd: shutdown recovery_request_timer earlier

recovery_request_timer may hold some QueuePeeringEvts which PGRef,
if we dont shutdown it earlier, it potentially cause the PGRef leak
when kicking pg.

Fixes: https://tracker.ceph.com/issues/38945
Signed-off-by: Zengran Zhang <zhangzengran@sangfor.com.cn>
(cherry picked from commit bb41bdfdb14768b5d46b7b1334bba1a3909e2618)

Conflicts:
src/osd/OSD.cc : Resolved in shutdown and start_shutdown

Merge pull request #27451 from badone/wip-mimic-spdk-rte_table_hash_cuckoo_create-build-error

mimic: spdk: update to latest spdk-18.05 branch

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26495 from pdvian/wip-38337-mimic

mimic: test: run-standalone.sh set local library location so mgr can find li…

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #26777 from pdvian/wip-38532-mimic

mimic: mgr/PyModule: put mgr_module_path first in sys.path

Reviewed-by: Tim Serong <tserong@suse.com>

Merge pull request #26811 from ashishkumsingh/wip-38565-mimic

mimic: run-standalone.sh: Need double-quotes to handle | in core_pattern on all distributions

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #27860 from ashishkumsingh/wip-39464-mimic

mimic: msg: output peer address when detecting bad CRCs

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #26709 from pdvian/wip-38507-mimic

mimic: osd/PrimaryLogPG: handle object !exists in handle_watch_timeout

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #26760 from pdvian/wip-38511-mimic

mimic: CLI: ability to change file ownership

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #26810 from ashishkumsingh/wip-38587-mimic

mimic: common/str_map: fix trim() on empty string

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #26833 from pdvian/wip-38561-mimic

mimic: mgr/BaseMgrModule: drop GIL for ceph_send_command

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #26862 from ashishkumsingh/wip-38646-mimic

mimic: osd: fixup OpTracker destruct assert, waiting_for_osdmap take ref      with OpRequest

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #27078 from yuriw/wip-yuriw-clients-mimic_2

qa/tests: initial checkin for `client-upgrade-mimic` suite

qa/tests: removed conflicting tests rbd_api_tests and rbd_import_export
and added devstack-tempest-gate.yaml instead

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #27447 from ifed01/wip-ifed-fix-39144

mimic: os/bluestore_tool: bluefs-bdev-expand: indicate bypassed for main dev

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #27558 from tspmelo/wip-26986-mimic

mimic: mgr/dashboard: Use human readable units on the OSD I/O graphs

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #27587 from pdvian/wip-38972-mimic

mimic: osd: process_copy_chunk remove obc ref before pg unlock

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #27366 from ifed01/wip-ifed-fix-alloc-overflow-mimic

mimic: os/bluestore: fix length overflow.

Reviewed-by: Jianpeng Ma <jianpeng.ma@intel.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #27570 from ifed01/wip-ifed-fix-21312-mimic

mimic: os/bluestore: call fault_range properly prior to looking for blob to …

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #27629 from pdvian/wip-39044-mimic

mimic: osd/PGLog: preserve original_crt to check rollbackability

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

qa/standalone/osd/osd-markdown: fix dup command disabling

The ceph cli tool checks for the presence of the variable, not its value.

Fixes: http://tracker.ceph.com/issues/38359
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit f7216d0b2c3a08d0f6e6106e838f84a7af29d396)

qa/standalone/osd/osd-markdown: disable CLI command dups

The markdown test is based on marking down a specific number of times, but
the duplicate commands from the CLI may not get absorbed/batched by the
mon, breaking the test.  Override the default qa/tasks/workunit.py
behavior of sending dups.

Fixes: http://tracker.ceph.com/issues/38359
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit c6a7b2cbd1430849ee92c627ff870f181697282d)

Merge pull request #27635 from smithfarm/wip-39342-mimic

mimic: ceph-objectstore-tool: rename dump-import to dump-export

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #27647 from smithfarm/wip-39353-mimic

mimic: mon/OSDMonitor: further improve prepare_command_pool_set E2BIG error message

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #27738 from ifed01/wip-ifed-fix-39445

mimic: os/bluestore: fix out-of-bound access in bmap allocator.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #27745 from ashishkumsingh/wip-39433-mimic

mimic: osd/PG: discover missing objects when an OSD peers and PG is degraded

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #27747 from s0nea/wip-39347-mimic

mimic: mgr/dashboard: Manager should complain about wrong dashboard certificate

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27845 from mykaul/patch-1

doc: Update bluestore config docs - fix typo (as -> has)

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/rados: Update bluestore config - fix typo

as -> has

Signed-off-by: Yaniv Kaul <ykaul@redhat.com>

mgr/dashboard: Add package-lock.json and use "npm ci"

This will make sure that in the future we always install the same version
of every npm package.

Since we don't use the same npm packages anymore, it's not possible to backport
this changes from master to mimic.

Fixes: http://tracker.ceph.com/issues/39080
Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Update npm packages

Fix some npm packages vulnerabilities.

Since we don't use the same npm packages anymore, it's not possible to backport
this changes from master to mimic.

Fixes: http://tracker.ceph.com/issues/39080
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #26643 from pdvian/wip-38448-mimic

mimic: MDSMonitor: do not assign standby-replay when degraded

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #26804 from pdvian/wip-38542-mimic

mimic: qa: unmount clients before deleting fs

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #26807 from pdvian/wip-38544-mimic

mimic: qa: remove libcephfs-java tests

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #26963 from ashishkumsingh/wip-38734-mimic

mimic: qa: extend MDS heartbeat grace for valgrind

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #27023 from pdvian/wip-38670-mimic

mimic: mds: wait for client to release shared cap when re-acquiring xlock

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #27058 from pdvian/wip-38689-mimic

mimic: mds: inode filtering on 'dump cache' asok

Reviewed-by: Venky Shankar <vshankar@redhat.com>

rgw: resolve bugs and clean up garbage collection code

Does a number of things to clean up rgw gc code:

  * adds additional logging to make future debugging easier.
  * resolves bug where the truncated flag was not always set correctly
    in gc_iterate_entries
  * resolves bug where marker in RGWGC::process was not advanced
  * resolves bug in which gc entries with a zero-length chain
    were not trimmed
  * resolves bug where same gc entry tag was added to list for
    deletion multiple times

Fixes: http://tracker.ceph.com/issues/38454
Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 73d7d369a2cc5edc8d51d70e02420e6efbfbe297)

Conflicts:
src/rgw/rgw_gc.cc dout() vs ldpp_dout()

msg: output peer address when detecting bad CRCs

Fixes: http://tracker.ceph.com/issues/39367
Signed-off-by: Greg Farnum <gfarnum@redhat.com>
(cherry picked from commit 19679231fc32daa405cc060847a02ac1cfd51300)

rgw: admin: handle delete_at attr in object stat output

Fixes: https://tracker.ceph.com/issues/39481
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit 90999a4de8e21551a98e0fe28acbadc158a7bc65)

cmake: update BuildSPDK for spdk-18.05

in spdk v18.05, libuuid is linked by libspdk_util.a, in which,
it is used by lib/util/uuid.c. and libspdk_vol.a uses the wrapper
function exposed by libspdk_util.a, so update the CMakefile script to
reflect the change.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit c2026b7473173e021c1f5ee072828538ea5c5d61)

os/tests: fix garbageCollection test case from store_test.

While running the test case using SSD as block device one could face
a failure caused by unexpectidly small blob size limit - compression
resulted in two blocks rather than single one which violated was test case
constraints.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit d7b45b6f38c8d24857b98803bf27177dce2a12c2)

mimic: test/store_test: workaround for failing BlobReuseOnOverwrite UT

Workarounds: https://tracker.ceph.com/issues/38795

Nautilus+ releases doesn't have the issue which is probably related to
modifications of BlueStore::MempoolThread::_trim_shards which introduces
different calculation for meta|data_alloc parameters

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

mgr/dashboard: .pylintrc: ignore too-many-statements check

Signed-off-by: Ricardo Dias <rdias@suse.com>
(cherry picked from commit c3465e03ad11d8c39b61a221aaee78762bc9465c)

osd/PG: discover missing objects when an OSD peers and PG is degraded

When a PG is remapped from OSD `a` to OSD `b`, the objects are
backfilled. When OSD `a` is restarted, objects become degraded
as `a` is no longer queried or considered as a backfill source.

As the PG is degraded, `PG::discover_all_missing` is not called
when a candidate OSD peers with the primary: The PG is already
active, thus `PG::activate` (and in turn missing object discovery)
is not called. Discovery is also not initiated from
`PG::RecoveryState::Active::react(const MNotifyRec& notevt)`
as there are no unfound objects.

This patch adds a call to `discover_all_missing` when
when an OSD sends its `MNotifyRec` message and the PG is degraded.

Fixes: https://tracker.ceph.com/issues/37439
Signed-off-by: Jonas Jelten <jj@stusta.net>
(cherry picked from commit e152d092f7b7839bb27ac7a5cf1c95f4d3752b32)

mgr/dashboard: Manager should complain about wrong dashboard certificate

Fixes: https://tracker.ceph.com/issues/24453
Signed-off-by: Volker Theile <vtheile@suse.com>
(cherry picked from commit a9ba5ba152cd54c3b4c94f5627cc85239906fe67)

os/bluestore: fix out-of-bound access in bmap allocator.

Fixes: https://tracker.ceph.com/issues/39334
Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit b0d2411c9c48b49fff827841c5ec6b66533d9c58)

 Conflicts:
src/os/bluestore/fastbmap_allocator_impl.h

os/bluestore_tool: bluefs-bdev-expand: indicate bypassed for main dev

Pre-nautilis releases are unable to expand main device but they do not
 report about that hence user thinks expansion succeeds. Which is confusing.

Fixes: https://tracker.ceph.com/issues/39144
Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #26829 from pdvian/wip-38546-mimic

mimic: doc: updated reference link for log based PG

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #27606 from ifed01/wip-ifed-fix-alloc-ut

mimic: os/bluestore: fix improper backport for p2 macros for bmap allocator

Reviewed-by: Xie Xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

os/bluestore: fix improper backport for p2 macros for bmap alocator

Fixes: 3244c871f36ee22ee826a96f30e82a8d05c9170c
This caused TestAllocatorLevel01.test_4G_alloc_bugN to fail in mimic.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

rgw: s3: awsv4 drop special handling for x-amz-credential

While s3 docs mention that every byte must be urlencoded, they are relaxed in
its implementation, when testing this behaviour on aws s3 itself, they seem to
be relaxed in handling aws credentials of the form

X-Amz-Credential=access1/20180817T123456Z/us-east-1...

wherein

X-Amz-Credential=access1%2F2018...

is expected. This allows for clients requesting the urls in both forms to
succeed as aws s3 does at the moment

Fixes: http://tracker.ceph.com/issues/26965
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit 25e1d2a32074254e9ebd780f3a41b2ff6469370f)

rgw: should recode canonical_uri when caculate s3v4 auth

fix: http://tracker.ceph.com/issues/23587

Signed-off-by: yuliyang <yuliyang@cmss.chinamobile.com>
(cherry picked from commit 919a8976cf40866e7212c5bb2ff38a9c5f8a1f75)