osd: unlock osd_lock when tweaking osd settings

unlock osd_lock when serving "debug kick_recovery_wq" command

we need to unlock osd_lock temporarily when updating the osd settings,
otherwise we will run into assert failure. because
OSD::handle_conf_change() acquires the osd_lock which is not a recursive
lock.

Fixes: http://tracker.ceph.com/issues/37762
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 8e94b081506fa9fbdbea09113f1549772bb6ec04)

 Conflicts:
src/osd/OSD.cc

osd: use unlock_guard for unlock osd temporarily

when OSD::do_command() gets called, osd_lock is acquired. but when
serving some of these commands, we need to call methods which also
acquire the osd_lock by themselves. for instance,
OSD::handle_conf_change() gets called by cct->_conf.apply_changes().
to allow them to do so, we unlock osd_lock before calling those methods,
and re-lock it after done with them.

unlock_guard is introduced to unlock and re-lock the lock in a RAII style.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 5c628a1cc9f703351ad3bd708e908df7c9a411bb)

 Conflicts:
src/osd/OSD.cc

wherever: guard handle_conf_change() from concurrent execution

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit aad318abc9a680d68aab96b051fb7457c8f7feac)

 Conflicts:
src/common/ceph_context.cc
src/mds/MDSDaemon.cc
src/mon/Monitor.cc
src/osd/OSD.cc
src/osdc/Objecter.cc

fix conflicts in the form of using `Mutex` in place of `ceph::mutex`
(w/ the appropriate locking/waiting/signalling semantics).

config: drop config_proxy::lock when invoking config observer

To prevent deadlocking when observer acquires its own locks
(lock order: config_proxy::lock -> foo::lock) and another
thread (say IO path) attempts to fetch an config value (lock
order: foo:lock -> config_proxy::lock).

Side effect of dropping locks when invoking config observer is
that remove_observer() can sneak in when the observer is still
in execution resulting in use-after-free. To mitigate this, any
in-progress observer call need to be completed before removing
the obserer. Also, remove_observer() needs to be invoked without
holding any observer locks so as to not run into deadlocks.

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 909c792457f9e8efaf2e53c968afbcf96cbd03e3)

 Conflicts:
src/common/config_obs_mgr.h
src/common/config_proxy.h
crimson/common/config_proxy.h

sources config_obs_mgr.h, config_proxy.h and crimson bits are not in
luminous, therfore the appropriate logic is moved to config.{h,cc}.

Merge pull request #25748 from alfredodeza/luminous-cryptsetup-allow-discards

luminous ceph-volume: enable device discards

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #24374 from smithfarm/wip-26919-luminous

luminous: common: (mon) command sanitization accepts floats when Int type is defined resulting in exception fault in ceph-mon

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #25387 from pdvian/wip-37482-luminous

luminous: rgw: Don't treat colons specially in resource part of ARN

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #25444 from theanalyst/wip-luminous-es-fixes

luminous: rgw: es fixes for working with nfs ganesha

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #25448 from smithfarm/wip-37519-luminous

luminous: rgw: fix max-size in radosgw-admin and REST Admin API

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #25484 from pdvian/wip-37549-luminous

luminous : rgw_file: user info never synced since librgw init

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #25506 from smithfarm/wip-37551-luminous

luminous: rgw: multisite: sync gets stuck retrying deletes that fail with ERR_PRECONDITION_FAILED

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #25512 from smithfarm/wip-36733-luminous

luminous: rgw: beast frontend fails to parse ipv6 endpoints

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #25644 from pdvian/wip-37563-luminous

luminous: rgw: fix version bucket stats

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24986 from joscollin/wip-36691-luminous

luminous: client: request next osdmap for blacklisted client

Reviewed-by: Yan, Zheng <zyan@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #24994 from jonsger/wip-36456-luminous

luminous: client: explicitly show blacklisted state via asok status command

Reviewed-by: Yan, Zheng <zyan@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #25043 from smithfarm/wip-36460-luminous

luminous: mds: rctime not set on system inode (root) at startup

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #25407 from brtknr/luminous

luminous: cephfs: ceph-volume-client: allow setting mode of CephFS volumes

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #25518 from smithfarm/wip-37423-luminous

luminous: qa: wrong setting for msgr failures

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #25520 from smithfarm/wip-37602-luminous

luminous: mds: severe internal fragment when decoding xattr_map from log event

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #25522 from smithfarm/wip-37606-luminous

luminous: mds: directories pinned keep being replicated back and forth between exporting mds and importing mds

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #25524 from smithfarm/wip-37604-luminous

luminous: mds: PurgeQueue write error handler does not handle EBLACKLISTED

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #25698 from xiexingguo/wip-37743

luminous: osd/OSDMap: cancel mapping if target osd is out

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #25021 from smithfarm/wip-36506-luminous

luminous: core: mon osdmap cash too small during upgrade to mimic

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #25289 from cbodley/wip-37091-luminous

luminous: rgw: catch exceptions from librados::NObjectIterator

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25458 from pdvian/wip-37535-luminous

luminous: librbd: reset snaps in rbd_snap_list()

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #25514 from smithfarm/wip-37343-luminous

luminous: osd: Prioritize user specified scrubs

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #25604 from smithfarm/wip-37697-luminous

luminous: osd: failed assert when osd_memory_target options mismatch

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #25772 from alfredodeza/luminous-ceph-volume-fix-join

luminous ceph-volume: Fix TypeError: join() takes exactly one argument (2 given)

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

ceph-volume: Fix TypeError: join() takes exactly one argument (2 given)

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
(cherry picked from commit 38f289b691ed1275bdf7342a4347434ed340c8fc)

Merge pull request #25352 from alfredodeza/luminous-rm37449

luminous: ceph-volume: zap: improve zapping to remove all partitions and all LVs, encrypted or not

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #24468 from vshankar/wip-36281

luminous: mds: add "drop cache" command

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #24728 from vshankar/wip-36217

luminous: cephfs-journal-tool: make --rank argument mandatory

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #24924 from vshankar/wip-24759

luminous: qa/suites/fs: use bigger cinder volumes

Reviewed-by:  Venky Shankar <vshankar@redhat.com>

Merge pull request #24324 from smithfarm/wip-32091-luminous

luminous: mds: migrate strays part by part when shutdown mds

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #24440 from pdvian/wip-36200-luminous

luminous: mds: fix mds damaged due to unexpected journal length

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #24465 from pdvian/wip-36206-luminous

luminous: libcephfs: expose CEPH_SETATTR_MTIME_NOW and CEPH_SETATTR_ATIME_NOW

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #24552 from pdvian/wip-36279-luminous

luminous : qa: tolerate failed rank while waiting for state

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #24872 from trociny/wip-36642-luminous

luminous: osdc: reduce ObjectCacher's memory fragments

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #25569 from batrick/i24929

luminous: qa: test_recovery_pool tries asok on wrong node

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #25752 from alfredodeza/luminous-better-devid

luminous common/blkdev, ceph-volume: improve get_device_id

Reviewed-by: Sage Weil <sage@redhat.com>

ceph-volume: Micron SSDs don't include vendor name in ID_SERIAL

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 3e4a638c931a5198aca74dc4037d0b03d5e563ba)

ceph-volume: update get_device_id to match in-tree implementation

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit c967dbf81f85602d2cd044ee5a91ba38cd76bdb9)

ceph-volume: enable device discards

When using SSDs as encrypted OSD device, discards do not pass the
encryption layer. This option activates discard requests.

Signed-off-by: Jonas Jelten <jj@stusta.net>
(cherry picked from commit 333330414730bf57c68efe6f1f13841b0a014dae)

osd/OSDMap: cancel mapping if target osd is out

Suppose we have a bad pg_upmap_item, say:
```
pg_upmap_items 2.0 [0,8,5,4,6,12]
```
which maps osd.6 to osd.12 that is currently marked as out.

It turns out **maybe_remove_pg_upmaps** can not handle the
above case well because **_apply_upmap** will silently discard
any bad mappings whenver they try to target at some current
__out__ OSDs. So if you call **pg_to_raw_up(2.0)**, you'll probably
get something like: ```up [8,4,6] ``` (e.g., the last mapping pair 6->12
is simply ignored by **_apply_upmap**).

Make **clean_pg_upmaps** do the tidy-up check instead, since it
already has __bare__ access to those pg_upmaps and pg_upmap_items.

Fixes: http://tracker.ceph.com/issues/37501
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
Signed-off-by: ningtao <ningtao@sangfor.com.cn>
(cherry picked from commit d06b65a0ab210e7a920761765531d929dd5798ec)

Conflicts:
        - *maybe_remove_pg_upmaps* input changed, in master
          we now have a passed in *nextmap* parameter

Merge pull request #25532 from smithfarm/wip-37643-luminous

luminous: ceph-create-keys: fix octal notation for Python 3 without losing compatibility with Python 2

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25609 from joscollin/wip-37685-luminous

luminous: doc/user-management: Remove obsolete reset caps command

Reviewed-by: Nathan Cutler <ncutler@suse.com>

rgw: fix version bucket stats

when link a null version delete_marker and the null version instance exists, the existing null version idx should not unlinked in rgw_bucket_link_olh. It will be removed in delete_obj.

Fixes: http://tracker.ceph.com/issues/21429
Signed-off-by: Shasha Lu <lu.shasha@eisoo.com>
(cherry picked from commit 52a843c598bc0553d8e75ea8290731e26a83aed1)

doc/user-management: Remove obsolete reset caps command

This command has not worked since hammer and makes no sense in an
environment where we do not allow empty caps

Fixes: http://tracker.ceph.com/issues/37663
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
(cherry picked from commit e4e7f3ea6af8316cd46bb74c3f6d6819b99cea53)

 Conflicts:
doc/rados/operations/user-management.rst
Dropped the allow empty caps paragraph, as done in master.

os/bluestore: add boundary for cache-autotune related settings

E.g., because user configures something wrong or we don't expected.

Fixes: http://tracker.ceph.com/issues/37507
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit c2c9245ee63cb135020061d90822745d91bb962d)

qa: add status field to method

From: ab4f33bd248bf192c1a81fea05a81c7f85ede52d
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #25552 from jan--f/25203-luminous

luminous ceph-volume: fix Batch object in py3 environments

Reviewed-by: Alfredo Deza <adeza@redhat.com>

qa: add status option

From: 333444e8aac065058c0a2353beff8005044659f7
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: remove dead code

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit a2ff87d4e25bc891fbfce4b0bdeb1a308ec22cb5)

qa: run asok command on correct machine

The MDS may not be on the same machine where the cluster command is run.

Fixes: http://tracker.ceph.com/issues/24858
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit fa25d6c8d1890bed2082b5c14f1c62a3db9ec27e)

ceph-volume: add Batch smoketest

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 936ee680f7633620c8a1127db9630a22326b9e2c)

ceph-volume: fix Batch object in py3 environments

Fixes: https://tracker.ceph.com/issues/37356
Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 851c7693daa315b7972702bedf09dd66a37667d8)

Merge pull request #25500 from joscollin/wip-37600-luminous

luminous: doc: broken link on troubleshooting-mon page

Reviewed-by: Nathan Cutler <ncutler@suse.com>

ceph-create-keys: Misc Python 3 fixes

Use octal notation for file permissions.

Switch file() calls to open().

Signed-off-by: James Page <james.page@ubuntu.com>
(cherry picked from commit 107579196eb4b52b1965fec6dcc53eba9d314b97)

Merge pull request #25418 from xiexingguo/wip-luminous-upmap-fixes

luminous: osd: backport recent upmap fixes

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #25286 from smithfarm/wip-37397-luminous

luminous: tests: /usr/bin/ld: cannot find -lradospp in rados mimic

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #25041 from smithfarm/wip-36750-luminous

luminous: mgr: [restful] deep_scrub is not a valid OSD command

Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #25047 from swinds24/fix_36764

luminous: mgr: hold lock while accessing the request list and submitting request

Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #25216 from k0ste/luminous_backports

luminous: mgr: prometheus: added bluestore db and wal devices to ceph_disk_occupation metric.//

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

mds: use uniform write handler for PQ

MDS would go damaged after blacklisting the MDS caused the PQ to see a write
error. Instead, just respawn and/or do usual write error handling (default
r/o).

Fixes: https://tracker.ceph.com/issues/37394
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 2fb665194f61914711454c2084eb1539bd3588b5)

Conflicts:
    src/msd/MDSRank.cc
- use Mutex::Locker instead of std::lock_guard

Merge pull request #25259 from k0ste/luminous_backports4

luminous: mgr/balancer: add cmd to list all plans

Reviewed-by: Sage Weil <sage@redhat.com>

mds: queue inodes into MDCache::export_pin_queue after setattr is journaled

This makes sure that, when doing inodes' migration, their "export_pin" is indeed the
mds it should be pinned to.

Fixes: http://tracker.ceph.com/issues/37368
Signed-off-by: Xuehan Xu <xuxuehan@360.cn>
(cherry picked from commit 93cbee3dd9c1446a027e7673645a1856ea68637a)

osdc/Journaler: make try_read_entry store data in contiguous memory

If bufferlist consists of discontiguous memory, decoding types whose
denc_traits needs contiguous memory is inefficient. The bufferlist may
get copied to temporary memory multiple times. (copy_shallow() in
src/include/denc.h actually does deep copy)

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
Fixes: http://tracker.ceph.com/issues/37399
(cherry picked from commit 39d044be3c8c29941443afd7daaf61603efec2c9)

qa: fix delay type config name

Setting as-is is a no-op.

Fixes: http://tracker.ceph.com/issues/36676
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 0e4ea92d191bee73402ab044d1a868e97cf56291)

osd: Prioritize user specified scrubs

Fixes: http://tracker.ceph.com/issues/37269
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 5f1a7ef029b0c4b59bc73fd709bdc096970a36ce)

osd: make scrub right now when pg stats_invalid is true

Signed-off-by: kungf <yang.wang@easystack.cn>
(cherry picked from commit a2ac3078ccc28656861718a248a3f4199fc17434)

test: Add test for requested scrub priority

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 1841928e28654722a5d98df89a40454bbd993314)

rgw: beast frontend parses ipv6 addrs

Fixes: http://tracker.ceph.com/issues/36662
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit a3b4124fd258887ff4cb3f65967f593658a41f4f)

rgw: beast frontend uses default port 443 for ssl_endpoint

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 9836ee2643c38afcd0221488d607dffa359e9bd4)

Conflicts:
src/rgw/rgw_asio_frontend.cc
- luminous uses different types for parse_endpoint arguments in function
  definition

Merge pull request #25326 from theanalyst/wip-luminous-reshard-fixes-1

luminous: bucket resharding fixes

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #24547 from linuxbox2/luminous-24595

luminous: rgw: apply quota config to users created via external auth

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24621 from cbodley/wip-24358

luminous: rgw: add ssl support to beast frontend

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24909 from smithfarm/wip-36414-luminous

luminous: rgw: librgw: crashes in multisite configuration

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24944 from jonsger/luminous-backport-pr#24700

luminous: rgw: SSE encryption does not detect ssl termination in proxy

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #25088 from jonsger/wip-36757-luminous

luminous: rgw: rgw-admin: reshard add can add a non-existent bucket

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #25195 from pdvian/wip-37284-luminous

luminous: rgw: radosgw-admin: translate reshard status codes (trivial)

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #25320 from theanalyst/wip-luminous-zone-deletion

luminous: rgw: fixes for zone deletion

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: data sync accepts ERR_PRECONDITION_FAILED on remove_object()

sync of deletes uses an If-UnModified-Since precondition, but does not
handle the corresponding ERR_PRECONDITION_FAILED error. treating this as
a failure means that we'll keep retrying the delete which will never
succeed. break this loop by treating ERR_PRECONDITION_FAILED as a
success

Fixes: http://tracker.ceph.com/issues/37448
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit f9d0ccf0069cb1a50621bdf8522038ab371f2a51)

doc: broken link on troubleshooting-mon page
Fixes: https://tracker.ceph.com/issues/37451
This fix involves adding :ref: labels to the add-or-rm-mons.rst
page. I also added :ref: labels for other headings within
add-or-rm-mons.rst (for future reference).

Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>
(cherry picked from commit ca05e456969a721631f5e9882c896cab76a4a3dc)

Conflicts:
doc/rados/troubleshooting/troubleshooting-mon.rst
Fixed the :ref: sentence and removed the _replace line.

rgw_file: user info never synced since librgw init

Fixed: https://tracker.ceph.com/issues/37527

Signed-off-by: Tao Chen <chentao@umcloud.com>
(cherry picked from commit e611be8dacc2afcfbe40fe5eb7847cf565c27d07)

Merge pull request #25284 from jdurgin/wip-osd-memory-target-docs-luminous

luminous: doc/rados: update bluestore provisioning and autotuning docs

Reviewed-by: Mark Nelson <mnelson@redhat.com>

librbd: reset snaps in rbd_snap_list()

in SnapIterator in rbd.pyx, rbd_snap_list_end() is called by
SnapIterator.__dealloc__(). and rbd_snap_list_end() frees
snaps->name if it is not nullptr.  but there is chance that
snaps->name is never initialized after snaps is allocated by
SnapIterator.__init__, in that case, we will free() a wild pointer.

Fixes: http://tracker.ceph.com/issues/37508
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 5431fba5c5c9122418b6a03df4ed89f86fbb35e3)

rgw: fix max-size-kb REST quota parameter and add max-size

Signed-off-by: Nick Erdmann <n@nirf.de>
(cherry picked from commit 99ce3aaf0b5b78e2c7452ee986d8dbfab08aa2d2)

Conflicts:
doc/radosgw/adminops.rst (trivial resolution)

rgw/radosgw-admin: allow negative values for --max-size

Fix regression in e664488e.

Signed-off-by: Nick Erdmann <n@nirf.de>
(cherry picked from commit 0ef92541d34cc897cec97854b358e7e702e44dcb)

rgw: ES sync: wrap all the decode bls in try block

since decode can throw, wrap all the try block

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit 12b12ccea23871688cc4101c72f00b0575f1c01a)

 Conflicts:
src/rgw/rgw_sync_module_es.cc:
using ::encode/::decode

rgw: ES sync: be more restrictive on object system attrs

This commit drops system attributes like url-keys, nfs-ganesha unix keys,
encryption keys and other system attributes when uploading metadata to ES.
This also brings in a tiny change where the attribute substring is no longer
compared and we compare the whole attribute name against the constants in
rgw_common

Fixes:http://tracker.ceph.com/issues/36233
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit 3e385c47c786ad152848b9460e1d96c6b2adf265)

src/rgw/rgw_sync_module_es.cc
conflict due to ::encode/::decode replacement in master, reverted to older version

rgw: es module: set compression type correctly

Parse compression type from the compression attr and only send this to ES.

Fixes: http://tracker.ceph.com/issues/22758
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit 5dc4655a4f14547410ee9c66ab32b0c943dc4753)

 Conflicts:
src/rgw/rgw_sync_module_es.cc

using global ::encode/::decode as the large changeset that changes all
occurences of encode/decode isn't going to be backported

rgw: fix typo

This fixes a typo in a log message. It's a separate commit so
downstream commits point to the right upstream commits via
cherry-pick.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 7d1768f2a332c5ac389cbb4bbeda28752e4f17bc)

rgw: perf -- remove bucket shards asynchronously rather than synchronously

We can now take advantage of the new asynchronous bucket shard removal
code and where we used to remove each shard synchronously now remove
them asynchronously. This would be a huge win when we have tens of
thousands of shards.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit cb0da45c5d6e4dff6514919b3b894df0eea7c1e4)

 Conflicts:
src/rgw/rgw_rados.cc
conflicts with placement set and rgw rados refactor

rgw: clean up unused bucket index shards

Clean up old bucket index shards when a resharding is complete. Also,
when a resharding fails, clean up unfinished bucket index shards. Do
both clean-ups asynchronously.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit f84f70d451036ee99011b8e2c7b974f15d3a005a)

 Conflicts:
src/rgw/rgw_rados.h
merge conflict as bucket_placement functions were moved after the rgw rados
refactor

rgw: a little code clean-up by adding const

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 48e22fb76feffb551074f274986bc4125a02003f)

rgw: remove never-called RGWRados member function

Because RGWRados::cls_rgw_init_index is never called, remove it.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 4593778c6d830a1a0d33528507f1795aee019ce4)

rgw: rename function for clarity

The function cls_rgw_bucket_init was renamed to
cls_rgw_bucket_init_index in order to better describe its
functionality.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 20868bd3b893ba2e17d2bfdc63d6544dbe677a9b)

rgw: fix bug in error codes returned

Fixes two bugs where positive error code was returned rather than
negative error code, which is the RGW norm.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit dc0732c524c2d8fb5b423f8cce8c6cc9aa7ca44a)

osd/OSDMap: fix upmap mis-killing for erasure-coded PGs

The up-set of erasure-coded PGs may include CRUSH_ITEM_NONE,
which as a result causes mis-killing of valid upmap items.

Fixes: https://tracker.ceph.com/issues/37493
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
Signed-off-by: ningtao <ningtao@sangfor.com.cn>
(cherry picked from commit f043dcc6d8663ed960a354b2441fd3f8dd66c62d)

Conflicts:
        - *maybe_remove_pg_upmaps* input changed, in master
          we now have a passed in *nextmap* parameter

mon: auto clean up stale upmap items

See https://www.spinics.net/lists/ceph-users/msg49518.html

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit 4181aa86906886423af80e478e45554606fc7cf6)

Conflicts:
        - *nextmap* has been renamed to *tmpmap*
        - add the *have_pg_upmaps* wrapper which is necessary for unittest