os/bluestore.cc: no need to fsync when failed to write label

Signed-off-by: tangwenjun <tang.wenjun3@zte.com.cn>

Merge PR #18424 into master

* refs/pull/18424/head:
mds: create snaprealm for quota enabled inode

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #19572 into master

* refs/pull/19572/head:
client: add log when we got unexpected flush ack

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #19672 into master

* refs/pull/19672/head:
ceph-fuse: ::rmdir() uses a deleted memory structure of dentry leads a core

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #19693 into master

* refs/pull/19693/head:
mds: use session.get_client()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #19696 into master

* refs/pull/19696/head:
client: avoid adding checked inode to the delayed_caps

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #19702 into master

* refs/pull/19702/head:
client: avoid remove same cap twice

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #19703 into master

* refs/pull/19703/head:
client: make mark_caps_clean and mark_caps_dirty as member function to Inode
client: fix dirty caps might never be flushed

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #19723 into master

* refs/pull/19723/head:
mds: clean up log's function name for CDir.cc and Beacon.cc
mds: clean up log's function name for CDentry.cc
mds: clean up log's function name for CInode.cc

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #19730 into master

* refs/pull/19730/head:
mds: Using stl min | max, MIN | MAX macros instead

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #19762 into master

* refs/pull/19762/head:
mds: fix dump last_sent

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #19767 into master

* refs/pull/19767/head:
client: enhance log for _closed_mds_session

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #18788 from amitkumar50/cov-mon-1

mon: NULL check of logger before use

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #19054 from cbodley/wip-librados-asio

librados: add async interfaces for use with Networking TS

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #19852 from yaozongyou/rgw-remove-unused-function

rgw: remove unused function get_system_obj_attrs
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #16383 from mikulely/1-ban-change-swift-obj-placement

rgw: drop the unnecessary handling of Swift's X-Storage-Policy on objects

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #17683 from shashalu/fix-lc-xml

rgw: RGWPutLC return ERR_MALFORMED_XML  when missing <Rule> tag in lifecycle.xml

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Amit Kumar <amitkuma@redhat.com>

Merge pull request #17873 from amitkumar50/cov-rgw-8

rgw: Returning when dst_ioctx.operate() returns error

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18861 from gaosibei/wip-fix-null-pointer

rgw: fix null pointer crush

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #19061 from Leeshine/wip-rgw-bug-21583

rgw: fix the bug of radowgw-admin zonegroup set requires realm

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #19854 from fangyuxiangGL/doc-sync-err

rgw: add radosgw-admin sync error trim to trim sync error log

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19809 from liu-chunmei/fix_dpdk_mzname_setting_issue

osd: fix dpdk memzon mz_name setting issue

Reviewed-By: Kefu Chai <kchai@redhat.com>

librados: add async interfaces for use with Networking TS

Defines asynchronous librados operations that satisfy all of the
"Requirements on asynchronous operations" imposed by the C++ Networking
TS [1] in section 13.2.7. These operations are implemented in terms of
boost::asio, but the interfaces themselves are free of boost types -
this makes the transition to std::net trivial when it's available.

These interfaces conform to the Extensible Asynchronous Model [2] that
originated in boost::asio. This model allows the last 'handler' argument
to either be a callback that gets the result, a coroutine yield_context
that will suspend until completion, or a 'use_future' tag to request the
result in a std::future (see the unit tests for examples of each). The
'Extensible' part also enables further integration with new frameworks.

For now, only async_read(), async_write(), and the read/write variants
of async_operate() are provided.

[1] Working Draft, C++ Extensions for Networking
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2017/n4711.pdf
[2] "Library Foundations for Asynchronous Operations"
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2014/n3896.pdf

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19616 from ifed01/wip-ifed-revert-extents

osd: get rid off extent map in object_info

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #19764 from ifed01/wip-ifed-storetest-cempty

objectstore/store_test: fix lack of flush prior to collection_empty()…

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #19841 from rzarzynski/wip-bs-dump-onode-by-ref

os/bluestore: _dump_onode() don't prolongate Onode anymore.

Reviewed-by: Sage Weil <sage@redhat.com>

rgw: add radosgw-admin sync error trim to trim sync error log

Signed-off-by: fang yuxiang fang.yuxiang@eisoo.com

Merge PR #19791 into master

* refs/pull/19791/head:
doc: misc fixes
doc: fix heading

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc: misc fixes

misc fixes for best-practices.

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #19662 from tchaikov/wip-denc-cxx-17

denc: use constexpr-if to replace some SFINAE impls

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

rgw: remove unused function get_system_obj_attrs

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

denc: use constexpr-if to replace some SFINAE impls

more readable this way IMO.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #19807 into master

* refs/pull/19807/head:
mds: warn for mds_cache_memory_limit bug in doc

Reviewed-by: Zheng Yan <zyan@redhat.com>

mds: warn for mds_cache_memory_limit bug in doc

Fixes: http://tracker.ceph.com/issues/22599
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #19657 from adamemerson/wip-hypermodern

The Day Has Come!

Update language standard version to C++17 and remove backported standard library features.

Reviewed-By: Casey Bodley <cbodley@redhat.com>
Reviewed-By: Kefu Chai <kchai@redhat.com>

Merge pull request #19823 from trociny/wip-ggate-list-xml

rbd-ggate: make list command produce valid xml format output

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19846 from fangyuxiangGL/compress-info

rgw: make radosgw object stat RGW_ATTR_COMPRESSION dump readable

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19732 from shinobu-x/using_find_primarylogpg

osd: Making use of find to reduce computational complexity

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #19765 from liewegas/wip-force-backfill-cleanups

qa: tolerate failure to force backfill

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19786 from dillaman/wip-osd-class-cap

osd: support class method whitelisting within caps

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

{common,osd,mds,rbd}: Remove unused captures

If we aren't referencing a variable, don't capture it. (This makes the
lambdas smaller, too, modulo Sufficiently Smart Compiler.)

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

dout: Remove VLA sanity check

This doesn't seem to be able to work for C++17 under both GCC and
Clang. Maybe there's a better way to do it. I might try routing
that argument through a constexpr function in the macro later.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common: Remove backport_std.h entirely

And use the experimental ostream joiner provided with the library.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common: Remove backported C++17 functions

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common: Remove backported typedefs

Since they're no longer needed.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

build: Bump language to C++17

Fix up all of the fallout from that.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

osd: fix dpdk memzon mz_name setting issue

Signed-off-by: chunmei Liu <chunmei.liu@intel.com>

objectstore/store_test: fix lack of flush prior to collection_empty() call

Fixes: http://tracker.ceph.com/issues/22409
Signed-off-by: Igor Fedotov <ifedotov@suse.com>

qa/standalone/scrub/osd-scrub-repair.sh: remove extents flag from object_info_t

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

osd: get rid off extent map in object_info

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #19475 from ZVampirEM77/wip-em-copyobj-ret

rgw: add xml output header in RGWCopyObj_ObjStore_S3 response msg

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: scienceluo <luo.kexue@zte.com.cn>

Merge pull request #19824 from qrGitHub/wip-rgw-useless-line3

rgw: drop useless type conversion

Merge pull request #19570 from yaozongyou/rgw-remove-unused-variable

cls/rgw: remove unused variable bl

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: make radosgw object stat RGW_ATTR_COMPRESSION dump readable

Signed-off-by: fang yuxiang fang.yuxiang@eisoo.com

Merge pull request #19575 from Liuchang0812/rgw-tier-type-check

rgw: proper error message when tier_type does not exist

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #19582 from qrGitHub/wip-rgw-redundant-readingAttrs

rgw: remove get_system_obj_attrs in function RGWDeleteLC::execute and RGWDeleteCORS::execute

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19676 from qrGitHub/wip-rgw-redundant-compare

rgw: remove redundant signature compare in LocalEngine::authenticate

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #19678 from yaozongyou/rgw-civetweb-status-code

rgw: log the right http status code in civetweb frontend's access log

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #19687 from qrGitHub/wip-rgw-bucketPolicy-invalidEffect

rgw: put bucket policy panics RGW process

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #19700 from qrGitHub/wip-rgw-optimize-function

rgw: optimize function RGWHandler::do_init_permissions

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #19710 from qrGitHub/wip-rgw-optimize-function2

rgw: optimize function abort_bucket_multiparts

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #19772 from fangyuxiangGL/quota

rgw: update quota is inconsistent at add/del object with compression

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19773 from chuang-he/fix_create_form_headers

rgw: fix the return type is wrong

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19431 from wido/mgr-pg-status

mgr/dashboard: Fix PG status coloring

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #19294 from myoungwon/wip-manifest-ref-flush

osd: flush operations for chunked objects

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: _dump_onode() don't prolongate Onode anymore.

Before the patch `BlueStore::_dump_onode()` takes `OnodeRef`
by value effectively extending the object's life time.

This is unnecessary and can be costly as `BlueStore::OnodeRef`
is ref-counted with `boost::intrusive_ptr`. Moreover, callers
are supposed to already hold a reference, so the optimization
for `count == 0` case stays without effect of inhibiting call
to `BlueStore::Onode::put()` and atomic integer increment.

    ```
    0000000000907d70 <BlueStore::_do_read(BlueStore::Collection*, boost::intrusive_ptr<BlueStore::Onode>, unsigned long, unsigned long, ceph::buffer::list&, unsigned int)>:
      907d70:       41 57                   push   %r15
      907d72:       41 56                   push   %r14

    ...

      907f1f:       48 8b 44 24 78          mov    0x78(%rsp),%rax
      907f24:       48 8b 00                mov    (%rax),%rax
      907f27:       48 85 c0                test   %rax,%rax
      907f2a:       48 89 84 24 a0 01 00    mov    %rax,0x1a0(%rsp)
      907f31:       00
      907f32:       74 04                   je     907f38 <BlueStore::_do_read(BlueStore::Collection*, boost::intrusive_ptr<BlueStore::Onode>, unsigned long, unsigned long, ceph::buffer::list&, unsigned int)+0x1c8>
      907f34:       f0 83 00 01             lock addl $0x1,(%rax)
      907f38:       48 8d 84 24 a0 01 00    lea    0x1a0(%rsp),%rax
      907f3f:       00
      907f40:       ba 1e 00 00 00          mov    $0x1e,%edx
      907f45:       4c 89 ff                mov    %r15,%rdi
      907f48:       48 89 c6                mov    %rax,%rsi
      907f4b:       48 89 44 24 20          mov    %rax,0x20(%rsp)
      907f50:       e8 9b bc fb ff          callq  8c3bf0 <BlueStore::_dump_onode(boost::intrusive_ptr<BlueStore::Onode>, int)>
      907f55:       48 8b bc 24 a0 01 00    mov    0x1a0(%rsp),%rdi
      907f5c:       00
      907f5d:       48 85 ff                test   %rdi,%rdi
      907f60:       74 05                   je     907f67 <BlueStore::_do_read(BlueStore::Collection*, boost::intrusive_ptr<BlueStore::Onode>, unsigned long, unsigned long, ceph::buffer::list&, unsigned int)+0x1f7>
      907f62:       e8 99 14 02 00          callq  929400 <BlueStore::Onode::put()>
      907f67:       48 8d 84 24 d0 00 00    lea    0xd0(%rsp),%rax
    ```

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #19428 from tchaikov/wip-erasure-cleanup

erasure-code: use std::count() instead

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #19777 from tchaikov/wip-rpm-fix-upgrade

rpm: adjust ceph-{osdomap,kvstore,monstore}-tool feature move

Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-By: Nathan Cutler <ncutler@suse.com>

rgw: drop useless type conversion

* variable 'max_buckets' is already 'uint64_t', so the type conversion is useless.

Signed-off-by: Bingyin Zhang <zhangbingyin@cloudin.cn>

Merge pull request #18827 from TaewoongKim/anticipation_timeout

osd: Make dmclock's anticipation timeout be configurable

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: myoungwon oh <omwmw@sk.com>

Merge pull request #19704 from trociny/wip-nbd-format

rbd-nbd: output format support for list-mapped command

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19722 from Songweibin/wip-rbd-import

rbd: replace positional_path parameter with arg_index in get_path()

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

osd: fix ObjectContextRef leak

To avoid ObjectContextRef leak, drop ObjectContextRef
before send a flush request to low tier

Signed-off-by: Myoungwon Oh <omwmw@sk.com>

osd: fix updating wrong object size

Signed-off-by: Myoungwon Oh <omwmw@sk.com>

osd: use stop_block() if the object is blocked

This commit prevents double free in finish_flush()
(stop_block() -> cancel_flush())

Signed-off-by: Myoungwon Oh <omwmw@sk.com>

src/test: remove version check and add data alignment for chunk_read test

Signed-off-by: Myoungwon Oh <omwmw@sk.com>

src/test: add chunked object unit test

Signed-off-by: Myoungwon Oh <omwmw@sk.com>

osd: add ordered flag if the object is flushing

Signed-off-by: Myoungwon Oh <omwmw@sk.com>

osd: add flush() for the chunked object.

If all chunks are dirty, the cheunked object will be flushed

Signed-off-by: Myoungwon Oh <omwmw@sk.com>

osd: set dirty flag if chunks are overwritten

Signed-off-by: Myoungwon Oh <omwmw@sk.com>

rbd-ggate: make list command produce valid xml format output

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #17010 from jgibson/bugfix-rgw-s3-policy-ip-address-condition

rgw: S3 Bucket Policy Conditions IpAddress and NotIpAddress do not work

Reviewed-By: Adam C. Emerson <aemerson@redhat.com>
Reviewed-By: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #19354 from shinobu-x/osd_atomic

osd: Use specializations, typedefs instead

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #19549 from bi-shun/brotli-compressor

compressor: Add Brotli Compressor

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19564 from kungf/list_bucket_osd

mon/OSDMonitor: list osd tree in named bucket

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #19816 from tchaikov/wip-22596

qa/standalone: remove osd-map-max-advance related tests

Reviewed-by: Sage Weil <sage@redhat.com>

rpm: adjust ceph-{osdomap,kvstore,monstore}-tool feature move

this is the rpm's counterpart of debian/control changes related to the
ceph-{osdomap,kvstore,monstore}-tool feature move. see #19328 and #19356.
the commit introducing this move is 6dba25e. and

$ git describe 6dba25e
v12.2.2-8-g6dba25e39d

so the first release that have this change is 12.2.2-8. in other words,
ceph-{base,osd,mon} >= 12.2.2.8 cannot co-exist with ceph-test < 12.2.2-8
in the same system. so we let ceph-test Requires ceph-common with the
same version. and since ceph-{osd,mon} Requires ceph-base with the same
version, and ceph-base Requires ceph-common with the same version, so by
tiering ceph-test with ceph-common with the same version, we enforce
this restriction.

Fixes: http://tracker.ceph.com/issues/22558
Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/standalone: remove osd-map-max-advance related tests

this setting was removed in 8967b73

Fixes: http://tracker.ceph.com/issues/22596
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19808 from cbodley/wip-boost-1.66

cmake: update minimum boost version to 1.66

Reviewed-by: Kefu Chai <kchai@redhat.com>

mds: clean up log's function name for CDir.cc and Beacon.cc

Signed-off-by: dongdong tao <tdd21151186@gmail.com>

mds: clean up log's function name for CDentry.cc

Signed-off-by: dongdong tao <tdd21151186@gmail.com>

mds: clean up log's function name for CInode.cc

Signed-off-by: dongdong tao <tdd21151186@gmail.com>

Merge pull request #19358 from taoCH/wip-librgw-multitenancy

librgw: export multitenancy support

cmake: add WITH_BOOST_CONTEXT option

adds a more specific option for this boost::context dependency, which was
previously only used by the radosgw beast frontend. see
http://tracker.ceph.com/issues/20048 for more background

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cmake: remove Beast submodule and include path

the beast library is included in boost 1.66

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: update beast frontend for boost 1.66

Fixes: http://tracker.ceph.com/issues/22600
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19806 from cbodley/wip-common-escape

common: add streaming interfaces for json/xml escaping

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

cmake: update minimum boost version to 1.66

Signed-off-by: Casey Bodley <cbodley@redhat.com>

common: add streaming interfaces for json/xml escaping

adds stream output operators that escape json/xml strings without having
to allocate a separate output buffer

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test: EscapeJson.Utf8 uses json instead of xml

the xml version does not escape the \n, while json does

Signed-off-by: Casey Bodley <cbodley@redhat.com>