rgw: RGWRadosGetOmapKeysCR uses omap_get_keys2

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWRadosGetOmapKeysCR uses completion return code

when this operates on a nonexistent object, the osd will reject the
request with ENOENT before trying to process the subops. so Objecter
will get back a subop return code of 0, try to decode an empty
bufferlist into the result and map that subop return code to EIO

by using the AioCompletion's return code, we get the correct result of
ENOENT instead

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18788 from amitkumar50/cov-mon-1

mon: NULL check of logger before use

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #19054 from cbodley/wip-librados-asio

librados: add async interfaces for use with Networking TS

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #19852 from yaozongyou/rgw-remove-unused-function

rgw: remove unused function get_system_obj_attrs
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #16383 from mikulely/1-ban-change-swift-obj-placement

rgw: drop the unnecessary handling of Swift's X-Storage-Policy on objects

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #17683 from shashalu/fix-lc-xml

rgw: RGWPutLC return ERR_MALFORMED_XML  when missing <Rule> tag in lifecycle.xml

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Amit Kumar <amitkuma@redhat.com>

Merge pull request #17873 from amitkumar50/cov-rgw-8

rgw: Returning when dst_ioctx.operate() returns error

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18861 from gaosibei/wip-fix-null-pointer

rgw: fix null pointer crush

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #19061 from Leeshine/wip-rgw-bug-21583

rgw: fix the bug of radowgw-admin zonegroup set requires realm

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #19854 from fangyuxiangGL/doc-sync-err

rgw: add radosgw-admin sync error trim to trim sync error log

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19809 from liu-chunmei/fix_dpdk_mzname_setting_issue

osd: fix dpdk memzon mz_name setting issue

Reviewed-By: Kefu Chai <kchai@redhat.com>

librados: add async interfaces for use with Networking TS

Defines asynchronous librados operations that satisfy all of the
"Requirements on asynchronous operations" imposed by the C++ Networking
TS [1] in section 13.2.7. These operations are implemented in terms of
boost::asio, but the interfaces themselves are free of boost types -
this makes the transition to std::net trivial when it's available.

These interfaces conform to the Extensible Asynchronous Model [2] that
originated in boost::asio. This model allows the last 'handler' argument
to either be a callback that gets the result, a coroutine yield_context
that will suspend until completion, or a 'use_future' tag to request the
result in a std::future (see the unit tests for examples of each). The
'Extensible' part also enables further integration with new frameworks.

For now, only async_read(), async_write(), and the read/write variants
of async_operate() are provided.

[1] Working Draft, C++ Extensions for Networking
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2017/n4711.pdf
[2] "Library Foundations for Asynchronous Operations"
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2014/n3896.pdf

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19616 from ifed01/wip-ifed-revert-extents

osd: get rid off extent map in object_info

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #19764 from ifed01/wip-ifed-storetest-cempty

objectstore/store_test: fix lack of flush prior to collection_empty()…

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #19841 from rzarzynski/wip-bs-dump-onode-by-ref

os/bluestore: _dump_onode() don't prolongate Onode anymore.

Reviewed-by: Sage Weil <sage@redhat.com>

rgw: add radosgw-admin sync error trim to trim sync error log

Signed-off-by: fang yuxiang fang.yuxiang@eisoo.com

Merge PR #19791 into master

* refs/pull/19791/head:
doc: misc fixes
doc: fix heading

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc: misc fixes

misc fixes for best-practices.

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #19662 from tchaikov/wip-denc-cxx-17

denc: use constexpr-if to replace some SFINAE impls

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

rgw: remove unused function get_system_obj_attrs

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

denc: use constexpr-if to replace some SFINAE impls

more readable this way IMO.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #19807 into master

* refs/pull/19807/head:
mds: warn for mds_cache_memory_limit bug in doc

Reviewed-by: Zheng Yan <zyan@redhat.com>

mds: warn for mds_cache_memory_limit bug in doc

Fixes: http://tracker.ceph.com/issues/22599
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #19657 from adamemerson/wip-hypermodern

The Day Has Come!

Update language standard version to C++17 and remove backported standard library features.

Reviewed-By: Casey Bodley <cbodley@redhat.com>
Reviewed-By: Kefu Chai <kchai@redhat.com>

Merge pull request #19823 from trociny/wip-ggate-list-xml

rbd-ggate: make list command produce valid xml format output

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19846 from fangyuxiangGL/compress-info

rgw: make radosgw object stat RGW_ATTR_COMPRESSION dump readable

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19732 from shinobu-x/using_find_primarylogpg

osd: Making use of find to reduce computational complexity

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #19765 from liewegas/wip-force-backfill-cleanups

qa: tolerate failure to force backfill

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19786 from dillaman/wip-osd-class-cap

osd: support class method whitelisting within caps

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

{common,osd,mds,rbd}: Remove unused captures

If we aren't referencing a variable, don't capture it. (This makes the
lambdas smaller, too, modulo Sufficiently Smart Compiler.)

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

dout: Remove VLA sanity check

This doesn't seem to be able to work for C++17 under both GCC and
Clang. Maybe there's a better way to do it. I might try routing
that argument through a constexpr function in the macro later.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common: Remove backport_std.h entirely

And use the experimental ostream joiner provided with the library.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common: Remove backported C++17 functions

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common: Remove backported typedefs

Since they're no longer needed.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

build: Bump language to C++17

Fix up all of the fallout from that.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

osd: fix dpdk memzon mz_name setting issue

Signed-off-by: chunmei Liu <chunmei.liu@intel.com>

objectstore/store_test: fix lack of flush prior to collection_empty() call

Fixes: http://tracker.ceph.com/issues/22409
Signed-off-by: Igor Fedotov <ifedotov@suse.com>

qa/standalone/scrub/osd-scrub-repair.sh: remove extents flag from object_info_t

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

osd: get rid off extent map in object_info

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #19475 from ZVampirEM77/wip-em-copyobj-ret

rgw: add xml output header in RGWCopyObj_ObjStore_S3 response msg

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: scienceluo <luo.kexue@zte.com.cn>

Merge pull request #19824 from qrGitHub/wip-rgw-useless-line3

rgw: drop useless type conversion

Merge pull request #19570 from yaozongyou/rgw-remove-unused-variable

cls/rgw: remove unused variable bl

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: make radosgw object stat RGW_ATTR_COMPRESSION dump readable

Signed-off-by: fang yuxiang fang.yuxiang@eisoo.com

Merge pull request #19575 from Liuchang0812/rgw-tier-type-check

rgw: proper error message when tier_type does not exist

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #19582 from qrGitHub/wip-rgw-redundant-readingAttrs

rgw: remove get_system_obj_attrs in function RGWDeleteLC::execute and RGWDeleteCORS::execute

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19676 from qrGitHub/wip-rgw-redundant-compare

rgw: remove redundant signature compare in LocalEngine::authenticate

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #19678 from yaozongyou/rgw-civetweb-status-code

rgw: log the right http status code in civetweb frontend's access log

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #19687 from qrGitHub/wip-rgw-bucketPolicy-invalidEffect

rgw: put bucket policy panics RGW process

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #19700 from qrGitHub/wip-rgw-optimize-function

rgw: optimize function RGWHandler::do_init_permissions

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #19710 from qrGitHub/wip-rgw-optimize-function2

rgw: optimize function abort_bucket_multiparts

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #19772 from fangyuxiangGL/quota

rgw: update quota is inconsistent at add/del object with compression

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19773 from chuang-he/fix_create_form_headers

rgw: fix the return type is wrong

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19431 from wido/mgr-pg-status

mgr/dashboard: Fix PG status coloring

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #19294 from myoungwon/wip-manifest-ref-flush

osd: flush operations for chunked objects

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: _dump_onode() don't prolongate Onode anymore.

Before the patch `BlueStore::_dump_onode()` takes `OnodeRef`
by value effectively extending the object's life time.

This is unnecessary and can be costly as `BlueStore::OnodeRef`
is ref-counted with `boost::intrusive_ptr`. Moreover, callers
are supposed to already hold a reference, so the optimization
for `count == 0` case stays without effect of inhibiting call
to `BlueStore::Onode::put()` and atomic integer increment.

    ```
    0000000000907d70 <BlueStore::_do_read(BlueStore::Collection*, boost::intrusive_ptr<BlueStore::Onode>, unsigned long, unsigned long, ceph::buffer::list&, unsigned int)>:
      907d70:       41 57                   push   %r15
      907d72:       41 56                   push   %r14

    ...

      907f1f:       48 8b 44 24 78          mov    0x78(%rsp),%rax
      907f24:       48 8b 00                mov    (%rax),%rax
      907f27:       48 85 c0                test   %rax,%rax
      907f2a:       48 89 84 24 a0 01 00    mov    %rax,0x1a0(%rsp)
      907f31:       00
      907f32:       74 04                   je     907f38 <BlueStore::_do_read(BlueStore::Collection*, boost::intrusive_ptr<BlueStore::Onode>, unsigned long, unsigned long, ceph::buffer::list&, unsigned int)+0x1c8>
      907f34:       f0 83 00 01             lock addl $0x1,(%rax)
      907f38:       48 8d 84 24 a0 01 00    lea    0x1a0(%rsp),%rax
      907f3f:       00
      907f40:       ba 1e 00 00 00          mov    $0x1e,%edx
      907f45:       4c 89 ff                mov    %r15,%rdi
      907f48:       48 89 c6                mov    %rax,%rsi
      907f4b:       48 89 44 24 20          mov    %rax,0x20(%rsp)
      907f50:       e8 9b bc fb ff          callq  8c3bf0 <BlueStore::_dump_onode(boost::intrusive_ptr<BlueStore::Onode>, int)>
      907f55:       48 8b bc 24 a0 01 00    mov    0x1a0(%rsp),%rdi
      907f5c:       00
      907f5d:       48 85 ff                test   %rdi,%rdi
      907f60:       74 05                   je     907f67 <BlueStore::_do_read(BlueStore::Collection*, boost::intrusive_ptr<BlueStore::Onode>, unsigned long, unsigned long, ceph::buffer::list&, unsigned int)+0x1f7>
      907f62:       e8 99 14 02 00          callq  929400 <BlueStore::Onode::put()>
      907f67:       48 8d 84 24 d0 00 00    lea    0xd0(%rsp),%rax
    ```

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #19428 from tchaikov/wip-erasure-cleanup

erasure-code: use std::count() instead

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #19777 from tchaikov/wip-rpm-fix-upgrade

rpm: adjust ceph-{osdomap,kvstore,monstore}-tool feature move

Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-By: Nathan Cutler <ncutler@suse.com>

rgw: drop useless type conversion

* variable 'max_buckets' is already 'uint64_t', so the type conversion is useless.

Signed-off-by: Bingyin Zhang <zhangbingyin@cloudin.cn>

Merge pull request #18827 from TaewoongKim/anticipation_timeout

osd: Make dmclock's anticipation timeout be configurable

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: myoungwon oh <omwmw@sk.com>

Merge pull request #19704 from trociny/wip-nbd-format

rbd-nbd: output format support for list-mapped command

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19722 from Songweibin/wip-rbd-import

rbd: replace positional_path parameter with arg_index in get_path()

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

osd: fix ObjectContextRef leak

To avoid ObjectContextRef leak, drop ObjectContextRef
before send a flush request to low tier

Signed-off-by: Myoungwon Oh <omwmw@sk.com>

osd: fix updating wrong object size

Signed-off-by: Myoungwon Oh <omwmw@sk.com>

osd: use stop_block() if the object is blocked

This commit prevents double free in finish_flush()
(stop_block() -> cancel_flush())

Signed-off-by: Myoungwon Oh <omwmw@sk.com>

src/test: remove version check and add data alignment for chunk_read test

Signed-off-by: Myoungwon Oh <omwmw@sk.com>

src/test: add chunked object unit test

Signed-off-by: Myoungwon Oh <omwmw@sk.com>

osd: add ordered flag if the object is flushing

Signed-off-by: Myoungwon Oh <omwmw@sk.com>

osd: add flush() for the chunked object.

If all chunks are dirty, the cheunked object will be flushed

Signed-off-by: Myoungwon Oh <omwmw@sk.com>

osd: set dirty flag if chunks are overwritten

Signed-off-by: Myoungwon Oh <omwmw@sk.com>

rbd-ggate: make list command produce valid xml format output

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #17010 from jgibson/bugfix-rgw-s3-policy-ip-address-condition

rgw: S3 Bucket Policy Conditions IpAddress and NotIpAddress do not work

Reviewed-By: Adam C. Emerson <aemerson@redhat.com>
Reviewed-By: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #19354 from shinobu-x/osd_atomic

osd: Use specializations, typedefs instead

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #19549 from bi-shun/brotli-compressor

compressor: Add Brotli Compressor

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19564 from kungf/list_bucket_osd

mon/OSDMonitor: list osd tree in named bucket

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #19816 from tchaikov/wip-22596

qa/standalone: remove osd-map-max-advance related tests

Reviewed-by: Sage Weil <sage@redhat.com>

rpm: adjust ceph-{osdomap,kvstore,monstore}-tool feature move

this is the rpm's counterpart of debian/control changes related to the
ceph-{osdomap,kvstore,monstore}-tool feature move. see #19328 and #19356.
the commit introducing this move is 6dba25e. and

$ git describe 6dba25e
v12.2.2-8-g6dba25e39d

so the first release that have this change is 12.2.2-8. in other words,
ceph-{base,osd,mon} >= 12.2.2.8 cannot co-exist with ceph-test < 12.2.2-8
in the same system. so we let ceph-test Requires ceph-common with the
same version. and since ceph-{osd,mon} Requires ceph-base with the same
version, and ceph-base Requires ceph-common with the same version, so by
tiering ceph-test with ceph-common with the same version, we enforce
this restriction.

Fixes: http://tracker.ceph.com/issues/22558
Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/standalone: remove osd-map-max-advance related tests

this setting was removed in 8967b73

Fixes: http://tracker.ceph.com/issues/22596
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19808 from cbodley/wip-boost-1.66

cmake: update minimum boost version to 1.66

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19358 from taoCH/wip-librgw-multitenancy

librgw: export multitenancy support

cmake: add WITH_BOOST_CONTEXT option

adds a more specific option for this boost::context dependency, which was
previously only used by the radosgw beast frontend. see
http://tracker.ceph.com/issues/20048 for more background

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cmake: remove Beast submodule and include path

the beast library is included in boost 1.66

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: update beast frontend for boost 1.66

Fixes: http://tracker.ceph.com/issues/22600
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19806 from cbodley/wip-common-escape

common: add streaming interfaces for json/xml escaping

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

cmake: update minimum boost version to 1.66

Signed-off-by: Casey Bodley <cbodley@redhat.com>

common: add streaming interfaces for json/xml escaping

adds stream output operators that escape json/xml strings without having
to allocate a separate output buffer

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test: EscapeJson.Utf8 uses json instead of xml

the xml version does not escape the \n, while json does

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test: use hex string literals

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19279 from taoCH/wip-fix-librgw-shutdown-err

librgw: fix shutdown err with resources uncleaned

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #19394 from zhangsw/cleanup-rgw-datasync-lease_cr

rgw: lease_cr->go_down is called twice, remove the needless one.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19725 from shinobu-x/osd_max_object_size

osd,os/bluestore: Display current size of osd_max_object_size

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #19718 from ifed01/wip-ifed-freelist-cleanup

os/bluestore: refactor FreeListManager to get clearer view on the number

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18324 from Jeegn-Chen/wip-bucket-quota-api

rgw: Admin API Support for bucket quota change

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19719 from shinobu-x/bluestore_extentmap_dup

bluestore: BlueStore::ExtentMap::dup impl

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #19698 from Liuchang0812/cleanup-ceph-tool

ceph: add the right bracket to watch-channel argument in the help message

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19660 from cfanz/tmp

include/encoding: fix compat version error message

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19654 from yaozongyou/test-fix-memory-leak

test: fix unittest memory leak to silence valgrind

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19598 from jcsp/wip-22457

packaging: explicit jinja2 dependency for dashboard

Reviewed-by: Wido den Hollander <wido@42on.com>

Merge pull request #19792 from joscollin/wip-doc-cephfs

doc: add mount.fuse.ceph to index

Reviewed-by: Yan, Zheng <zyan@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

doc: fix typo

Signed-off-by: Jos Collin <jcollin@redhat.com>