rgw: Write documentation for bucket policies

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: RESTful bucket policy ops

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Verify policies as WELL as ACLs

Only hook into object/bucket checks for now. Once we have STS (giving
us Roles) or User/Group policies it will make sense to hook into
those, too.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Build bucket permission and environment in req_state

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Add basic support for IAM policies

This support is currently incomplete but should provide a starting
point.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

buffer: Make the use of static areas more convenient

Make three wrapper functions to tidy up the process of making a
bufferlist holding a single static buffer.

The lack of any decent handling of const in buffer::list makes me wax
wroth, but it's a bit much to fix right now.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Move globbing flags to header

So they can be used when calling the function

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Add is_identity to AuthApplier class

Eventually this will allow us to match all authentication information
against all specified principals in a policy.

Right now it handles users and wildcards.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common: Add ISO-8601 Date Support

For parsing and unparsing from ceph::real_time.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

submodule: Tencent's RapidJSON library

This is a high performance, MIT licensed JSON parsing library. It
provides a SAX interface so that I can compile an S3 policy without
building up a JSONObject tree in the middle that gets thrown away.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common: Add make_unique

There are parts of C++14 that are both useful and easy to
implement. This is one of them.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #13731 from liewegas/wip-19119

PendingReleaseNotes: warning about 'osd rm ...' and #19119

PendingReleaseNotes: warning about 'osd rm ...' and #19119

See http://tracker.ceph.com/issues/19119

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #14873 from liewegas/wip-bluefs-position

os/bluestore: put bluefs in the middle of the shared device

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #14902 from liewegas/wip-upmap

osd/OSDMap: improve upmap calculation

Reviewed-by: Loic Dachary <loic@dachary.org>

Merge pull request #14933 from liewegas/wip-asok-schema

common/perf_counters: make schema more friendly and update docs

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #14967 from tchaikov/wip-faster-cull

mgr: optimize DaemonStateIndex::cull() a little bit

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15002 from dachary/wip-19836-crush-pre-luminous

crush: encode can override weights with weight set

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15031 from tchaikov/wip-drop-unused-opts

config_opts: drop unused opts

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15043 from tchaikov/wip-19884

ceph-detect-init: detect init system by poking the system

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #15059 from liewegas/wip-mon-kvtype

mon: fix mon_keyvaluedb application

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15032 from tchaikov/wip-doc-opts

doc: add descriptions for mon/mgr options

Reviewed-by: John Wilkins <jowilkin@redhat.com>

Merge pull request #15127 from tchaikov/wip-vstart-mgr-debug-ms

vstart: "debug_ms=1" for mgr by default

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15103 from drunkard/master

doc: minor fixes in radosgw/

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15080 from hikycool/master

doc: update the usage of 'ceph-deploy purge'

Reviewed-by: Kefu Chai <kchai@redhat.com>

vstart: "debug_ms=1" for mgr by default

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15087 from liewegas/wip-thrash-ec

qa/suites/rados: fix ec thrashing

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15072 from liewegas/wip-crush-swap-bucket

mon,crush: add 'osd crush swap-bucket' command

Reviewed-by: Loic Dachary <loic@dachary.org>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15069 from tchaikov/wip-inttypes-cleanup

int_types.h: remove hacks to workaround old systems

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #15057 from liewegas/wip-osd-choose-cleanup

osd/PG: drop pre-firefly compat_mode for choose_*_acting

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #15042 from badone/wip-redundant-headers-3

common: Remove redundant includes

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14989 from liewegas/wip-19877

osd: fix past_intervals base case by adding epoch_pool_created to pg_history_t

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15113 from trociny/wip-unittest_librbd-cleanup

test/librbd: unit tests cleanup

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #15093 from dillaman/wip-19929

librbd: reacquire lock should update lock owner client id

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #10690 from ceph/wip-rgw-polymorphic-errors

rgw: polymorphic error codes

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15096 from jcsp/wip-journalrepair-test

qa: simplify TestJournalRepair

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #15110 from tchaikov/wip-build-using-cmake

doc: use do_cmake.sh instead of `cmake ..`

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15023 from tchaikov/wip-cmdparse-constness

cmdparse: more constness

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15112 from jcsp/wip-19940

ceph-create-keys: update client.admin if it already exists

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15097 from linuxbox2/wip-rgwfile-write3

rgw_file: v3: fix write-timer action

test/librbd: make unit tests pass when cache is disabled

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

test/librbd: TestMockImageRemoveRequest gmock warnings

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

doc: add hyperlink on ACL settings

The link listed at bottom but never used, I think it's missed by author.

Signed-off-by: Drunkard Zhang <gongfan193@gmail.com>

ceph-create-keys: update client.admin if it already exists

This means that anyone who de-privileges their client.admin
user will find that it gets resets with full privileges again.
This is necessary because we cannot distinguish between an old
key and a deliberately de-privileged key, and it's okay because
this is the admin key, it's meant to have access to everything.

Fixes: http://tracker.ceph.com/issues/19940
Signed-off-by: John Spray <john.spray@redhat.com>

doc: typo fixes in doc/radosgw/multisite

Signed-off-by: Drunkard Zhang <gongfan193@gmail.com>

doc: use do_cmake.sh instead of `cmake ..`

user could be confused at seeing errors like:

src/CMakeLists.txt:644 (add_subdirectory): The source directory /src/lua
does not contain a CMakeLists.txt file.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15094 from fullerdj/wip-djf-19893

qa/cephfs: Fix for test_data_scan

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #14938 from michaelsevilla/f19362

mds: add perf counters for file system operations

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15018 from ukernel/wip-19890

pybind: fix open flags calculation

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #14769 from ukernel/wip-19755

osdc/Filer: truncate large file party by party

Reviewed-by: John Spray <john.spray@redhat.com>

doc: change reference to internal link

Signed-off-by: Drunkard Zhang <gongfan193@gmail.com>

rgw: Searching for error codes and dumping output based on protocol flags.

The previous commit uses dynamic allocation to search for s3/ swift specific
error codes and dump the results. In order to avoid dynamic allocation,
we are making use of protocol flags to achieve the above.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

polymorphic error codes.

make set_req_state_err a method of req_state

get rid of params.perr; it's never used

rgw - polymorphic error object.

polymorphic error object: this way sts, which has additional error fields,
and its own xml format for how errors should be produced, can overload
the base rgw_err class.

STL fixup: use real mappings for error mappings, not a linear search.
insubstantiate error mappings exactly once in src/rgw/rgw_common.cc, instead
of multiple copies from every source file that includes src/rgw/rgw_http_errors.h.

remove dump_errno(s, err) - it was never used, and the code looked bogus.

move rgw_http_error_to_errno from header to rgw_rest_client.cc,
the only place it's used.
move rgw_http_status_code and http_codes to rgw_rest.cc,
the only place it's used.

base class (rgw_err) implements s3 errors; swift & sts errors use
the base class and overlay their own additional error codes to
the ones recognized by the base class.

For a while, I was trying to just have a "generic" abort_early() that
used dump_more.  Now, I have an op aware form that works the old way.
I would like the http routines to known less about "op" structures,
which is why I'm making this distinction.

Splitting up generic and s3 flavors of some things:
rgw_rest_init() - split into s3 and generic forms.
end_header: s3 vs generic forms: generic form uses dump_more.
abort_early: s3 vs generic forms: generic form uses dump_more, error_content.

And fixing some mistakes from the merge:
Fixup: should not have deleted dump_errno() local storage forms.
Fixup:RGWRESTFlusher  - take a dump_more to make it generic.
Fixup: need RGWRESTOp::send_response()

Move swift_err up; use it in bulkdelete_respond
I Think I don't need s->prot_flags when using swift_err.

Signed-off-by: Marcus Watts <mwatts@redhat.com>

rgw_file: v3: fix write-timer action

For now, unify with v4 write-on-close path, by calling
RGWFileHandle::close() on write-timer expire, since it will
call write_finish() as a side-effect.

Fixes: http://tracker.ceph.com/issues/19932
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #15052 from batrick/i19734

mds: use debug_mds for most subsys

Reviewed-by: John Spray <john.spray@redhat.com>

qa: simplify TestJournalRepair

This was sending lots of metadata ops to MDSs to persuade
them to migrate some subtrees, but that was flaky.  Use
the shiny new rank pinning functionality instead.

Signed-off-by: John Spray <john.spray@redhat.com>

qa/cephfs: Fix for test_data_scan

Don't assume that test_data_scan will be run on exactly 2 MDS nodes.

Fixes: http://tracker.ceph.com/issues/19893
Signed-off-by: Douglas Fuller <dfuller@redhat.com>

Merge pull request #15091 from ceph/luminous

merge v12.0.2 release tag

mds: add perf counters for file system operations

Shows the types of operations that clients are doing by adding counters to the
spot where the MDS responds to clients. This avoids counting operations that
were restarted while waiting for locks or for metadata to load.  Does not
support per-client activity. This is the first step towards comprehensive
performance tracking in the file system.

Signed-off-by: Michael Sevilla <mikesevilla3@gmail.com>

Merge pull request #15079 from ztczll/master

mailmap: ztczll affiliation

Reviewed-by: Kefu Chai <kchai@redhat.com>

librbd: reacquire lock should update lock owner client id

Fixes: http://tracker.ceph.com/issues/19929
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/suites/rados/threas-erasure-code-*: fix luminous facet

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/thrash-erasure-code-overwrites: fix

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15081 from tchaikov/wip-fio-readme

fio: update README.md so only the fio ceph engine is built

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc/dev/perf_counters: update docs for new friendly properties

Note that histograms are not yet documented.

Signed-off-by: Sage Weil <sage@redhat.com>

common/perf_counters: rename histogram adder method

Specifically, it's a u64 counter histogram.

Signed-off-by: Sage Weil <sage@redhat.com>

common/perf_counters: make 'perf schema' more friendly

The raw bits are pretty hard to interpret and the documentation isn't
super easy to find.  Expose a string description instead.

Signed-off-by: Sage Weil <sage@redhat.com>

common/perf_counters: histograms are counters

The histogram is a histogram of counters (not guages), so set the COUNTER
bit.  (This only matters because we expose and document the bits of the
type, for better for for worse.)

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: require force flag is swap-bucket alg types differ

Signed-off-by: Sage Weil <sage@redhat.com>

crush: add crush_alg_name

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: add 'osd crush swap-bucket' command

Swap contents of two crush buckets.  Do it safely by default so that
the origin bucket has to be an orphan (unless forced).

Signed-off-by: Sage Weil <sage@redhat.com>

crush: implement swap_bucket

Swap contents between buckets.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15048 from trociny/wip-19907

rbd-mirror: resolve admin socket path names collision

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #15026 from ukernel/wip-19891

qa/suites/fs: reserve more space for mds in full tests

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15039 from ukernel/wip-19903

client: fix UserPerm::gid_in_group()

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15035 from batrick/quiet-mds-grow-shrink

qa: silence spurious insufficient standby health warnings

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #14991 from ukernel/wip-19828

mds: fix use-after-free in Locker::file_update_finish()

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #14978 from cernceph/dvanders_quota

client: client_quota no longer optional

Reviewed-by: John Spray <john.spray@redhat.com>

doc: update the usage of 'ceph-deploy purge'

Now it refuses to purge data directly before remove ceph package.
So update the order between commands when using it.

Signed-off-by: Yu Shengzuo <yu.shengzuo@99cloud.net>

fio: update README.md so only the fio ceph engine is built

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15044 from lixiaoy1/master

test/fio_ceph_objectstore: fix fio plugin build failure by engine_data

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mailmap:ztczll affiliation

Signed-off-by: zhanglei <243290414@qq.com>

Merge pull request #15076 from atheism/atheism-mailmap

mailmap: Weibing Zhang mailmap affiliation

Merge pull request #15007 from ztczll/master

test/objectstore/: Check put_ref return value

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #15021 from runsisi/wip-fix-writefull

osd/PrimayLogPG: update modified range to include the whole object size for write_full op

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15046 from ukernel/wip-osdmap-encode

osd/OSDMap: bump encoding version for require_min_compat_client

Reviewed-by: Sage Weil <sage@redhat.com>

mailmap: Weibing Zhang mailmap affiliation

Signed-off-by: Weibing Zhang <atheism.zhang@gmail.com>

int_types.h: remove hacks to workaround old systems

Signed-off-by: Kefu Chai <kchai@redhat.com>

common: Remove redundant includes

Fixes: http://tracker.ceph.com/issues/19883 (Partially)
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

rbd-mirror: resolve admin socket path names collision

If global admin_socket parameter is set, for PoolReplayer admin
sockets generate and use unique paths in $run_dir. Provide the socket
paths in output of PoolReplayer's "rbd mirror status" admin socket
command.

A possible use case:

1) run "rbd mirror status $pool_replayer" command using global admin
   socket (known from configuration) and obtain socket paths for the
   PoolReplayer local and remote context sockets.
2) Use these paths to run commands provides by the contexts.

Fixes: http://tracker.ceph.com/issues/19907
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

doc: add descriptions for mon/mgr options

Signed-off-by: Kefu Chai <kchai@redhat.com>

crush: encode can override weights with weight set

Encode a "legacy" crushmap if (1) we're using weight sets, (2) the
client is lacking features for the weight set, but (3) the weight set
has a single position and no id remapping. Since these maps are only
used by clients (not humans), then there is no need to preserve the
original crush weights. We can just swap them for the real weights and
the legacy clients will behave as expected.

Fixes: http://tracker.ceph.com/issues/19836
Signed-off-by: Loic Dachary <loic@dachary.org>

crush,osd: s/chooseargs/choose_args/

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #13111 from cbodley/wip-rgw-mdlog-trim

rgw multisite: automated mdlog trimming

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

mon: fix mon_keyvaluedb application

In 42a6b0efe1a3269c8e6c10e89f92bc0f28923af2 I mistakenly thought that
create_or_open was the path taken for normal open (not during mkfs) and
made it assume a missing kv_type meant leveldb.  In reality, this is the
only path where mon_keyvaluedb is ever used (during mkfs), and
create_or_open is only called during mkfs.  The bug I was (probably?)
trying to fix was that regular open() did not write out a kv_type file
(with the assumption of leveldb) if it was missing.  As a result, the
previous fix was forcing all mons to be leveldb.

Fix this by reverting the create_or_open hunk (so that we express
mon_keyvaluedb on mkfs), and fixing the normal open path to write kv_type
if it is missing.  This effectively switches the mon back to rocksdb by
default (and allows teuthology to test both rocksdb and leveldb by setting
the option).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15004 from liewegas/wip-19880

os/bluestore: fix deferred write race

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #15029 from xiexingguo/wip-bluestore-20170510

os/bluestore: pre-calculate number of ghost buffers to evict

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15012 from tchaikov/wip-byteswap

byteorder: use gcc intrinsics for byteswap

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15019 from badone/wip-redundant-headers-2

common: Remove redundant includes

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15040 from tchaikov/wip-19902

osd/PrimaryLogPG: do not call on_shutdown() if (pg.deleting)

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>