Merge pull request #15045 from shashalu/fix-slo

rgw: fix X-Object-Meta-Static-Large-Object in SLO download

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14800 from theanalyst/doc/release-notes/sse-c-correction

doc: release-notes clarify about rgw encryption

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14922 from ukernel/wip-kcephfs-whitelist

qa: update log whitelists for kcephfs suite

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #14941 from wjwithagen/wip-wjw-freebsd-doc-manual-install

doc: add FreeBSD manual install

Merge pull request #14988 from ShiqiCooperation/master

test/unittest_bluefs: check whether mounted success

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14688 from cbodley/wip-rgw-multi-suite

qa/rgw: add multisite suite to configure and run multisite tests

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #15071 from cbodley/wip-qa-dnsmasq

qa: add task for dnsmasq configuration

Reviewed-by: Vasu Kulkarni <vasu@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #15090 from theanalyst/release/v12.0.3/notes

doc: release notes for v12.0.3 (dev)

Merge pull request #15133 from cbodley/wip-doc-rgw-remove-fcgi

doc/rgw: remove fastcgi page and sample configs

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15144 from drunkard/master

doc: typo fixes on hyperlink/words

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15161 from dzafman/wip-test-fix

test: Test fix for SnapSet change

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15147 from bassam/pr-speedup-ccache

cmake: improved build speed by 5x when using ccache

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15158 from liewegas/wip-mgr-dump

mon/MgrMonitor: add 'mgr dump [epoch]' command

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15159 from LiumxNL/fix-ping-monitor

pybind/rados: avoid call free() on invalid pointer

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15164 from tchaikov/wip-check-gperf-before-using-it

cmake: check the existence of gperf before using it

Reviewed-By: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #15089 from zhsj/add-mirror

doc: add new cn ceph mirror to doc and mirroring

Reviewed-by: Wido den Hollander <wido@42on.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15173 from joscollin/wip-test-crush-warning

test/crush: silence warnings from -Walloc-size-larger-than= and -Wstringop-overflow=

Merge pull request #15171 from ztczll/master

test/objectstore: Check apply_transaction() return values

Merge pull request #15034 from liewegas/wip-bluestore-sync-write

os/bluestore/KernelDevice: fix sync write vs flush

Reviewed-by: Varada Kari <varada.kari@gmail.com>

os/bluestore/KernelDevice: fix flush() vs sync write()

We need to set the atomic flag so that a flush will actually flush.

Signed-off-by: Sage Weil <sage@redhat.com>

test/crush: silence warnings from -Walloc-size-larger-than= and -Wstringop-overflow=

The following warnings appear during make:

/home/jcollin/workspace/ceph/src/test/crush/CrushWrapper.cc: In member function ‘virtual void CrushWrapper_choose_args_compat_Test::TestBody()’:
/home/jcollin/workspace/ceph/src/test/crush/CrushWrapper.cc:1057:20: warning: argument 1 value ‘18446744073709550912’ exceeds maximum object size 9223372036854775807 [-Walloc-size-larger-than=]
   crush_choose_arg choose_args[c.get_max_buckets()];
                    ^~~~~~~~~~~

In function ‘void* memset(void*, int, size_t)’,
    inlined from ‘virtual void CrushWrapper_choose_args_compat_Test::TestBody()’ at /home/jcollin/workspace/ceph/src/test/crush/CrushWrapper.cc:1058:9:
/usr/include/bits/string3.h:90:70: warning: ‘void* __builtin_memset(void*, int, long unsigned int)’: specified size 18446744073709550912 exceeds maximum object size 9223372036854775807 [-Wstringop-overflow=]
   return __builtin___memset_chk (__dest, __ch, __len, __bos0 (__dest));

Signed-off-by: Jos Collin <jcollin@redhat.com>

Check apply_transaction return values

Signed-off-by: zhanglei <243290414@qq.com>

doc: v12.0.3 release notes

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

doc: PendingReleaseNotes move the 12.0.2 section of release notes

So that we can track whatever diff goes in after

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #15148 from zealoussnow/wip-spec

build: move bash_completion.d/ceph to ceph-common

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: check the existence of gperf before using it

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15160 from bassam/pr-system-boost

cmake: fix boost components for WITH_SYSTEM_BOOST

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15107 from guihecheng/rgw_file-cleanup

rgw: use pre-defined calls to replace raw flag operation

doc: hyperlink fixes

Signed-off-by: Drunkard Zhang <gongfan193@gmail.com>

doc: typo fixes

Signed-off-by: Drunkard Zhang <gongfan193@gmail.com>

rgw_file: use pre-defined calls to replace raw flag operation

Signed-off-by: Gui Hecheng <guihecheng@cmss.chinamobile.com>

doc: remove duplicated section in doc/radosgw/adminops

Signed-off-by: Drunkard Zhang <gongfan193@gmail.com>

Merge pull request #15053 from dillaman/wip-librbd-blkin

blkin: librbd trace hooks

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

blkin: rbd object map update traces

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

blkin: initial trace hooks for IO path

To collect blkin traced for RBD, set the "rbd blkin trace all"
configuration parameter to true.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

blkin: add config variable rbd_blkin_trace_all

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

blkin: improved objecter trace events

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

blkin: librados should start new child trace

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

blkin: improve trace visibility into object cacher

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

blkin: pass trace information to object cacher

Signed-off-by: Victor Araujo <ve.ar91@gmail.com>
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

common/zipkin_trace: added missing include for encoding

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test/librados_test_stub: add new aio_operate overloads for tracing

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test: Test fix for SnapSet change

Caused by: a7b59221691954aedfac88695df8c89e36e10739

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #10736 from oliveiradan/SebastienHan_config_diff_get_fr

common/ceph_context: 'config diff get' option added

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #14914 from neha-ojha/wip-mon-crushtype-health-warn

mon: add crush type down health warnings

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15086 from joscollin/wip-cleanup-redundant-headers-1

common: Remove redundant includes

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #15109 from tchaikov/wip-mgrmon-send-digests-using-ticks

mon/MgrMonitor: send digests only if is_active()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15116 from liewegas/wip-bluestore-aio-warning

os/bluestore/KernelDevice: helpful warning when aio limit exhausted

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #15117 from liewegas/wip-luminous-legacy-check

qa/suites/rados/thrash: make sure osds have map before legacy scrub

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15120 from liewegas/wip-ec-buffer

osd: reduce buffer pinning from EC entries

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15121 from ivancich/wip-remove-an-unnecessary-loop

osd/pglog: remove loop through empty collection

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15124 from cbodley/wip-osd-swap

osd: fix argument-dependent lookup of swap()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15138 from liewegas/wip-list-objects-more

ceph_test_rados_api_list: more fix LibRadosListNP.ListObjectsError

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/rgw: add test config to rgw_multisite_tests task

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: add timing config options to test_multi.py

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: add timing configuration

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: add log_level argument to rgwadmin()

changes default level from info to debug

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: dont assume zone1 and zone2 are on different clusters

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: add 5min timeouts to sync checkpoints

if multisite tests are going to run in teuthology, they can't loop forever

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: add multisite suite

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cmake: fix boost components for WITH_SYSTEM_BOOST

when using WITH_SYSTEM_BOOST dont set header-only packages
for BOOST_COMPONENTS. On some distros these packages dont
exist.

Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>

pybind/rados: avoid call free() on invalid pointer

Signed-off-by: Mingxin Liu <mingxin@xsky.com>

Merge pull request #15126 from batrick/i19934

qa: silence upgrade test failure

Reviewed-by: Kefu Chai <kchai@redhat.com>

mon/MgrMonitor: add 'mgr dump [epoch]' command

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15037 from trociny/wip-19875

librbd: optimize copy-up to add hints only once to object op

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #15085 from runsisi/wip-fix-trashlist

librbd: do not raise an error if trash list returns -ENOENT

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #15134 from runsisi/wip-fix-trashpy

pybind/rbd: fix crash if more than 1024 images in trash bin

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge branch pull request #13551 from pritha-srivastava/wip-rgw-role-tenant

rgw: Replaced rgw userid with tenant in ARN for Roles.
rgw: Adding roles under the scope of tenants.
rgw: Added code for Role input validation.
rgw: Added code to fetch a role by its id.
rgw: Added code to allow tenant name as part of role name also.
rgw: Better reporting of missing input values in case of radosgw-admin
     commands for Roles.

Reviewed-By: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #15152 from guihecheng/rgw_file-extra-rele

rgw_file: removed extra rele() on fs in rgw_umount()

Merge pull request #15142 from bassam/pr-ccache-s-files

cmake: workaound ccache issue with .S assembly files

Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw_file: fix extra rele on fs in close

We got exactly a refcnt=0 after rele in close,
no need to rele twice.

Signed-off-by: Gui Hecheng <guihecheng@cmss.chinamobile.com>

Merge pull request #15137 from theanalyst/rgw/cleanup/header-drop

rgw: drop using std ns in header files and other cleanups

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15145 from zealoussnow/wip-0518

mailmap: Leo Zhang infomation and affiliation

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

build: move bash_completion.d/ceph from debian/ceph-base.install to debian/ceph-common.install

when only ceph-common is installed, can also use ceph completion

Signed-off-by: Leo Zhang <nguzcf@gmail.com>

Merge pull request #15143 from minshenglin/fix_broken_link

doc: fix broken link in erasure-code.rst

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: remove "-Wp" compile option to speedup ccache

"-Wp" in not needed and when building with ccache the "cache hits"
only happen afer preproccessing which is slower than a direct hit.
this change removes the "-Wp" option

Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>

build: move bash_completion.d/ceph to ceph-common

When only ceph-common is installed, can also use ceph completion

Signed-off-by: Leo Zhang <nguzcf@gmail.com>

mailmap: Leo Zhang infomation and affiliation

Signed-off-by: Leo Zhang <nguzcf@gmail.com>

doc: fix broken link in erasure-code.rst

Fixes: http://tracker.ceph.com/issues/19972
Signed-off-by: MinSheng Lin <minsheng.l@inwinstack.com>

cmake: workaound ccache issue with .S assembly files

when building with -DWITH_CCACHE=ON recent versions of ccache
(for example 3.3.3) seem to mangle object files from the cache
for source files that have a .S extension. This seems similar
but unrelated to PR #14633. The workaround here is to rename .S
files to .s (lower case).

Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>

Merge branch 'wip-sts-authorization'

Merge pull request #14307 from adamemerson/wip-sts-authorization

common: Add make_unique
submodule: Tencent's RapidJSON library
common: Add ISO-8601 Date Support
rgw: Add is_identity to AuthApplier class
rgw: Move globbing flags to header
buffer: Make the use of static areas more convenient
rgw: Add basic support for IAM policies
rgw: Build bucket permission and environment in req_state
rgw: Verify policies as WELL as ACLs
rgw: RESTful bucket policy ops
rgw: Write documentation for bucket policies

Reviewed-By: Casey Bodley <cbodley@redhat.com>
Reviewed-By: Radoslaw Zarzynski <rzarzynski@mirantis.com>
Reviewed-By: Kefu Chai <kchai@redhat.com>

qa/rgw: add rgw_multisite_tests task to run tests

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: add rgw_multisite task based on rgw_multi

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: add symlink to qa/tasks/rgw_multi

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: move startup polling logic to util/rgw.py

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: fixes for cluster name on cleanup

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: move compression type out of ceph.conf

this makes the 'compression type' setting global to all gateways, and
makes the setting visible to other tasks in ctx.rgw.compression_type

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: add import for StringIO

also removes unnecessary tuple parens

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: Write documentation for bucket policies

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: RESTful bucket policy ops

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Verify policies as WELL as ACLs

Only hook into object/bucket checks for now. Once we have STS (giving
us Roles) or User/Group policies it will make sense to hook into
those, too.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Build bucket permission and environment in req_state

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Add basic support for IAM policies

This support is currently incomplete but should provide a starting
point.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

buffer: Make the use of static areas more convenient

Make three wrapper functions to tidy up the process of making a
bufferlist holding a single static buffer.

The lack of any decent handling of const in buffer::list makes me wax
wroth, but it's a bit much to fix right now.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Move globbing flags to header

So they can be used when calling the function

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Add is_identity to AuthApplier class

Eventually this will allow us to match all authentication information
against all specified principals in a policy.

Right now it handles users and wildcards.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common: Add ISO-8601 Date Support

For parsing and unparsing from ceph::real_time.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

submodule: Tencent's RapidJSON library

This is a high performance, MIT licensed JSON parsing library. It
provides a SAX interface so that I can compile an S3 policy without
building up a JSONObject tree in the middle that gets thrown away.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common: Add make_unique

There are parts of C++14 that are both useful and easy to
implement. This is one of them.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

ceph_test_rados_api_list: more fix LibRadosListNP.ListObjectsError

Follow-on to d7e6e8d60309e4800389b36f786b633d0ca2ec07; we need to
make sure the client gets the updated osdmap.

Fixes: http://tracker.ceph.com/issues/19963
Signed-off-by: Sage Weil <sage@redhat.com>