Merge pull request #22739 from majianpeng/osd-shardthread-do-bluestore-oncommits

os/bluestore: make osd shard-thread do oncommits

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

osd/OSD: choose a fixed thread do oncommits callback function

Now bluestore oncommit callback exec by osd op threads.
If there are multi threads of shard, it will cause out-of order.
For example, threads_per_shard=2
              Thread1                                 Thread2
    swap_oncommits(op1_oncommit)
                                            swap_oncommits(op2_oncommit)
    OpQueueItem.run(Op3)
                                            op2_oncommit.complete();
    op1_oncommit.complete()

This make oncommits out of order.
To avoiding this, we choose a fixed thread which has the smallest
thread_index of shard to do oncommit callback function.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

common/Finisher: only queue empty only wake up waiter.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

common/Finisher: only queue empty only wake up waiter.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #24004 from xiexingguo/wip-yet-more-async-fixes

osd/PG: async-recovery should respect historical missing objects

Reviewed-by: Yan Jun <yan.jun8@zte.com.cn>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #24130 from tchaikov/wip-gcc-7.3

rpm: bump up required GCC version to 7.3.1

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23779 from Devp00l/wip-table-actions-component

mgr/dashboard: Add table actions component

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #24160 from jdurgin/wip-rados-lead

doc/dev/index: update rados lead

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23662 from dillaman/wip-24412

librbd: support v2 cloning across namespaces

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: songweibin <song.weibin@zte.com.cn>

Merge pull request #24059 from cbodley/wip-rgw-opstate-rm

rgw, cls: remove cls_statelog and rgw opstate tracking

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge PR #24143 into master

* refs/pull/24143/head:
qa/workunits/cephtool/test_kvstore_tool.sh: run test in ., not /tmp

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mgr/dashboard: Use table actions component for roles

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Use table actions component for users

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Add HTML coverage reports

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Fixes ts warning in RBD snapshot tests

Typescript has thrown an warning because a private method is called in the
RBD snapshot unit tests.

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Use table actions component for RBD snapshots

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Use table actions component for OSDs

Signed-off-by: Stephan Müller <smueller@suse.com>

Merge pull request #24007 from tianshan/fix_35905

rgw: async sync_object and remove_object does not access coroutine me…

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23610 from XiangxiangMeng/fix-rgw-bucket-policy

rgw: policy: fix s3:x-amz-grant-read-acp keyword error

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #24128 from ricardoasmarques/wip-optional-user-password

mgr/dashboard: User password should be optional

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge PR #24145 into master

* refs/pull/24145/head:
qa/workunits/mon/caps.sh: wait longer to time out

Reviewed-by: João Eduardo Luís <joao@suse.de>

Merge pull request #23544 from joke-lee/wip-multiobjdelete-num-limit

rgw: delete multi object num limit

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #24001 from joke-lee/doc_life_cycle

rgw: update cls_rgw.cc and cls_rgw_const.h

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24114 from robbat2/wip-35990

rgw: fix chunked-encoding for chunks >1MiB

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #24110 from cbodley/wip-35986

rgw: abort_bucket_multiparts() ignores individual NoSuchUpload errors

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

cls: remove cls_statelog and tests

Signed-off-by: Casey Bodley <cbodley@redhat.com>

osd: remove statelog from osd_class_load_list config

Signed-off-by: Casey Bodley <cbodley@redhat.com>

common: remove config variables related to rgw opstate

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove dependencies on cls_statelog

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove RGWOpState and related classes

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: remove opstate commands

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: remove test for /admin/opstate rest apis

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove rest handlers for /admin/opstate

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove unused opstate params from copy/fetch/stat

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove opstate tracking from fetch_remote_obj()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18578 from gaosibei/wip-cleanup-duplicate-include

RGW: remove duplicate include header files in rgw_rados.cc

Reviewed-by: Amit Kumar <amitkuma@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #23927 from joke-lee/WIP-RGW-x-amz-version-id

rgw: return x-amz-version-id: null when delete obj in versioning suspended bucket

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: Use table actions component for RGW users

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Use table actions component for RGW buckets

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Use table actions component for RBDs

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Add table actions component

With the help of this component it is easy to create all actions for a
table with the right permission for each table action with a simple
configuration.

Per default action button and action drop down list are shown. All
actions will be filtered depending on the permissions on
initialization. Per default all actions that have the permission
'update' or 'delete' are only enabled if an item is selected which
holds no running task.

You can use change the default behavior, to show actions only in a
named drop down.
For each item you can change it's default behaviors by using custom
functions to enable/disable an action or show/hide it
as action button or drop down item.

Please look at the tests to understand how to use it in practice.

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: User password should be optional

Fixes: https://tracker.ceph.com/issues/36031
Signed-off-by: Ricardo Marques <rimarques@suse.com>

Merge PR #23092 into master

* refs/pull/23092/head:
common: drop RefCountObject.cc and improve inlineability.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #24109 into master

* refs/pull/24109/head:
doc: update docs for device management
mgr: make devicehealth always-on
mgr/devicehealth: do not create metrics pool on get-device-metrics
mgr/devicehealth: converge OPTIONS and DEFAULTS
mgr/devicehealth: squelch health warnings for unused devices
mgr/devicehealth: show-health-metrics -> get-health-metrics

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #24005 from p-na/refactor-deletion-modal-component

mgr/dashboard: Refactoring of `DeletionModalComponent`

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge PR #24149 into master

* refs/pull/24149/head:
unittest_bufferlist: add unit test for sha1 from multiple raw buffers

Reviewed-by: Piotr Dałek <piotr.dalek@corp.ovh.com>
Reviewed-by: Myoungwon Oh <omwmw@sk.com>

githubmap: update contributors

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/rbd: test clone v2 across namespaces

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: permit cross-namespace when using clone v2

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: use helper method to create librados::IoCtxs

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: replace librbd::ParentSpec with cls::rbd::ParentImageSpec

The newer struct includes support for pool namespaces.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: invoke new parent class API methods

If the OSDs don't support the new methods, revert to the legacy
class API methods.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: replace existing attach parent calls with new state machine

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: new attach parent state machine

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: replace existing detach parent calls with new state machine

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: new detach parent state machine

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: refactor flatten state machine to new code style

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: add new namespace compatible parent image cls methods

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: break cls_rbd_snap on-disk struct backwards compatability

The cls methods are now storing normalized parent overlaps within the snapshot
if the OSDs are required to be Nautilus or later. Therefore, we can safely
break the on-disk format compatibility for those newer OSDs.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: migrate legacy snapshot on-disk format to normalized parent

The parent reference should be written once under the "parent" key,
so all updates to a snapshot should ensure the new normalized format
is used if Nautilus or later OSDs are required.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: cls_rbd_snap on-disk struct supports parent overlap

This is the first step to fix the denormalization of the parent
image spec references between the HEAD revision and the snapshot
revisions.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: cls_rbd_parent on-disk struct supports namespaces

If the OSDs are required to be Nautilus or later, it will automatically break
backwards compatibility of the structure to support pool namespaces.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: ChildImageSpec now optionally includes child pool namespace

It will break version compatibility if used against older OSDs that
don't support RBD pool namespace support.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: new ParentImageSpec type to support parent namespaces

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: break composite cls rbd helpers into individual helpers

With the forthcoming need to use new methods to retrieve a clone's parent,
use per-method helpers to invoke the correct method within the image
refresh state machine.

Also used this time to merge newer cls rbd methods that should be supported
by all non-EOLed OSDs.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: corrected namespace indentation within cls_client

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test/librbd: not valid to have different parents between image snapshots

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

objclass: expose the required OSD release setting

This can be utilized to determine when it's safe to enable new
cls features.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #24076 from ifed01/wip-ifed-fix-repairer

os/bluestore: fix bloom filter num entry miscalculation in repairer

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #24085 from liewegas/wip-32731

osd/osd_types: fix pg_t::contains() to check pool id too

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23721 from batrick/log-entry-stack

log: avoid heap allocations for most log entries

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24111 from gregsfortytwo/wip-35987-connection-buffers

msg/async: clean up local buffers on dispatch

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #24124 from tchaikov/wip-35969

rpm: use updated gperftools

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

doc/dev/index: update rados lead

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #23754 from p-na/down-osd-raises-error

mgr/dashboard: Fix OSD down error display

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

osd/PG: explicitly clear num_objects_missing when approximate

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/PG: async-recovery should respect historical missing objects

Peers with async-recovery enabled are usually having a update-to-date
last-update iterator and hence might be moved out from the __async_recovery_targets__
set during the next peering circles.

7de35629f562436d2bdb85788bdf97b10db3f556 makes num_objects_missing
trace historical missing objects correctly, hence we could take
num_objects_missing into account when determing __async_recovery_targets__.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #24098 into master

* refs/pull/24098/head:
Rename "include/assert.h"

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

unittest_bufferlist: add unit test for sha1 from multiple raw buffers

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #24112 into master

* refs/pull/24112/head:
osd: avoid unncecessary ref-counting in OSD::enqueue_op.

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

qa/workunits/mon/caps.sh: wait longer to time out

5s -> 5m to gives us more leeway for when the mons are thrashing.

Also, *only* set this timeout when we expect a timeout.  If we don't,
wait forever.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #24090 from cbodley/wip-crimson-concurrent-dispatch

crimson/net: concurrent dispatch for SocketMessenger

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/workunits/cephtool/test_kvstore_tool.sh: run test in ., not /tmp

Notably, we can't make a bluestore store on a tmpfs.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #24092 into master

* refs/pull/24092/head:
mgr: Change signature of PyString_AsString to match return

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #24057 into master

* refs/pull/24057/head:
src/common: add a unit test (bufferlist.sha1())
osd, src/common: return sha1 value if zero-length buffer.
src/common/buffer.cc: remove unnecessary copy in sha1()

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #24072 into master

* refs/pull/24072/head:
osd/PG: kill __queue_query__
osd/OSD: deprecate __project_pg_history__

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #24104 into master

* refs/pull/24104/head:
mgr/diskprediction: add prototype diskprediction module

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #24103 from b-ranto/wip-dashboard-http

mgr/dashboard: Do not require cert for http

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #24015 from ricardoasmarques/wip-update-selection-on-refresh

mgr/dashboard: More configs for table `updateSelectionOnRefresh`

Reviewed-by: Kanika Murarka <kmurarka@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Rename SSLCherryPyConfig

The class does not always set up SSL, dropping the SSL bit.

Fixes: http://tracker.ceph.com/issues/36069
Signed-off-by: Boris Ranto <branto@redhat.com>

mgr/dashboard: Do not require cert for http

The ceph dashboard currently requires a SSL certificate even if it is
not running in the SSL mode since it is always querying for the
certificate file/key pair.

This patch fixes the behaviour by querying for the certificate file/key
only if it is running in the SSL mode.

Fixes: http://tracker.ceph.com/issues/36069
Signed-off-by: Boris Ranto <branto@redhat.com>

Merge pull request #24031 from rzarzynski/wip-common-bl-drop-0c

common: drop the unused zero-copy facilities in ceph::bufferlist

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #24120 into master

* refs/pull/24120/head:
doc: Fix Typos of Developer Guide

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #24105 from dillaman/wip-24516

librbd: object map improperly flagged as invalidated

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #23981 from cbodley/wip-35830

rgw: raise debug level on redundant data sync error messages

rpm: bump up required GCC version to 7.3.1

so we can include the fix of
https://gcc.gnu.org/ml/libstdc++/2017-07/msg00077.html and the fix at
https://github.com/gcc-mirror/gcc/commit/7c78f2e67270187dd755632eafe3a526f4e6df30
. otherwise, if a class offers implicit conversion operator to
std::string and std::string_view, seastar will fail to compile. like

/opt/rh/devtoolset-7/root/usr/include/c++/7/bits/basic_string.h:3946:7:
note: candidate: std::basic_string<_CharT, _Traits, _Alloc>&
std::basic_string<_CharT, _Traits, _Alloc>::operator+=(const
std::basic_string<_CharT, _Traits, _Alloc>&) [with _CharT = char;
_Traits = std::char_traits<char>; _Alloc = std::allocator<char>]
       operator+=(const basic_string& __str)
       ^~~~~~~~
/opt/rh/devtoolset-7/root/usr/include/c++/7/bits/basic_string.h:3988:7:
note: candidate: std::basic_string<_CharT, _Traits, _Alloc>&
std::basic_string<_CharT, _Traits,
_Alloc>::operator+=(std::basic_string<_CharT, _Traits,
_Alloc>::__sv_type) [with _CharT = char; _Traits =
std::char_traits<char>; _Alloc = std::allocator<char>;
std::basic_string<_CharT, _Traits, _Alloc>::__sv_type =
std::basic_string_view<char>]
       operator+=(__sv_type __sv)
       ^~~~~~~~

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24088 from dzafman/wip-35982

qa/standalone: Standalone test corrections

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

common: get_data() of buffer_raw isn't virtual anymore.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common: make the get_data() of buffer_raw interface final.

This is just to ensure the just dropped buffer::raw_pipe
was the solely user of this facility. After successful
validation, we can drop `virtual` on the method entirely.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common: drop can_zero_copy() from ceph::buffer interfaces.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>