test: add mgr/ and smoke test for mgr dashboard

Signed-off-by: Dan Mick <dan.mick@redhat.com>

ceph.spec.in, debian/control: build/runtime dependency for cherrypy

Signed-off-by: Dan Mick <dan.mick@redhat.com>

mgr: change config_prefix to be 'mgr/<id>/' and separate handle/key by '/'

dm_crypt config-key usage has established '/' as the separator; keep
that as a convention for now

Signed-off-by: Dan Mick <dan.mick@redhat.com>

mgr/PyModules: move constructor definition to .cc

Otherwise, the incomplete ServeThread type causes problems
for others who include PyModules.h

Signed-off-by: Dan Mick <dan.mick@redhat.com>

pybind/mgr/dashboard/module.py: allow custom server addr, port

Signed-off-by: Dan Mick <dan.mick@redhat.com>

mgr/PyModules.cc,PyState.cc: improve config access debug logging

Signed-off-by: Dan Mick <dan.mick@redhat.com>

client/SyntheticClient.cc: filer is now a std::unique_ptr

Signed-off-by: Dan Mick <dan.mick@redhat.com>

ceph_syn.cc, client/SyntheticClient.cc: Client -> StandaloneClient

Signed-off-by: Dan Mick <dan.mick@redhat.com>

ServeThread: add 'running' flag, don't notify non-running threads

Signed-off-by: Dan Mick <dan.mick@redhat.com>

pybind/mgr: web dashboard

A read only display of some of the cluster's state, plus
some cephfs/rbd/osd pages that demonstrate how to go
read state remotely from daemons.

Signed-off-by: John Spray <john.spray@redhat.com>

vstart: enable gui plugin

Signed-off-by: John Spray <john.spray@redhat.com>

osd: wire up "histogram dump" to `tell`

...so that mgr modules can get at it.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #15148 from zealoussnow/wip-spec

build: move bash_completion.d/ceph to ceph-common

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15160 from bassam/pr-system-boost

cmake: fix boost components for WITH_SYSTEM_BOOST

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15107 from guihecheng/rgw_file-cleanup

rgw: use pre-defined calls to replace raw flag operation

rgw_file: use pre-defined calls to replace raw flag operation

Signed-off-by: Gui Hecheng <guihecheng@cmss.chinamobile.com>

Merge pull request #15053 from dillaman/wip-librbd-blkin

blkin: librbd trace hooks

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

blkin: rbd object map update traces

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

blkin: initial trace hooks for IO path

To collect blkin traced for RBD, set the "rbd blkin trace all"
configuration parameter to true.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

blkin: add config variable rbd_blkin_trace_all

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

blkin: improved objecter trace events

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

blkin: librados should start new child trace

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

blkin: improve trace visibility into object cacher

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

blkin: pass trace information to object cacher

Signed-off-by: Victor Araujo <ve.ar91@gmail.com>
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

common/zipkin_trace: added missing include for encoding

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test/librados_test_stub: add new aio_operate overloads for tracing

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10736 from oliveiradan/SebastienHan_config_diff_get_fr

common/ceph_context: 'config diff get' option added

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #14914 from neha-ojha/wip-mon-crushtype-health-warn

mon: add crush type down health warnings

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15086 from joscollin/wip-cleanup-redundant-headers-1

common: Remove redundant includes

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #15109 from tchaikov/wip-mgrmon-send-digests-using-ticks

mon/MgrMonitor: send digests only if is_active()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15116 from liewegas/wip-bluestore-aio-warning

os/bluestore/KernelDevice: helpful warning when aio limit exhausted

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #15117 from liewegas/wip-luminous-legacy-check

qa/suites/rados/thrash: make sure osds have map before legacy scrub

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15120 from liewegas/wip-ec-buffer

osd: reduce buffer pinning from EC entries

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15121 from ivancich/wip-remove-an-unnecessary-loop

osd/pglog: remove loop through empty collection

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15124 from cbodley/wip-osd-swap

osd: fix argument-dependent lookup of swap()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15138 from liewegas/wip-list-objects-more

ceph_test_rados_api_list: more fix LibRadosListNP.ListObjectsError

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

cmake: fix boost components for WITH_SYSTEM_BOOST

when using WITH_SYSTEM_BOOST dont set header-only packages
for BOOST_COMPONENTS. On some distros these packages dont
exist.

Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>

Merge pull request #15126 from batrick/i19934

qa: silence upgrade test failure

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15037 from trociny/wip-19875

librbd: optimize copy-up to add hints only once to object op

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #15085 from runsisi/wip-fix-trashlist

librbd: do not raise an error if trash list returns -ENOENT

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #15134 from runsisi/wip-fix-trashpy

pybind/rbd: fix crash if more than 1024 images in trash bin

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge branch pull request #13551 from pritha-srivastava/wip-rgw-role-tenant

rgw: Replaced rgw userid with tenant in ARN for Roles.
rgw: Adding roles under the scope of tenants.
rgw: Added code for Role input validation.
rgw: Added code to fetch a role by its id.
rgw: Added code to allow tenant name as part of role name also.
rgw: Better reporting of missing input values in case of radosgw-admin
     commands for Roles.

Reviewed-By: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #15152 from guihecheng/rgw_file-extra-rele

rgw_file: removed extra rele() on fs in rgw_umount()

Merge pull request #15142 from bassam/pr-ccache-s-files

cmake: workaound ccache issue with .S assembly files

Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw_file: fix extra rele on fs in close

We got exactly a refcnt=0 after rele in close,
no need to rele twice.

Signed-off-by: Gui Hecheng <guihecheng@cmss.chinamobile.com>

Merge pull request #15137 from theanalyst/rgw/cleanup/header-drop

rgw: drop using std ns in header files and other cleanups

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15145 from zealoussnow/wip-0518

mailmap: Leo Zhang infomation and affiliation

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

build: move bash_completion.d/ceph from debian/ceph-base.install to debian/ceph-common.install

when only ceph-common is installed, can also use ceph completion

Signed-off-by: Leo Zhang <nguzcf@gmail.com>

Merge pull request #15143 from minshenglin/fix_broken_link

doc: fix broken link in erasure-code.rst

Reviewed-by: Kefu Chai <kchai@redhat.com>

build: move bash_completion.d/ceph to ceph-common

When only ceph-common is installed, can also use ceph completion

Signed-off-by: Leo Zhang <nguzcf@gmail.com>

mailmap: Leo Zhang infomation and affiliation

Signed-off-by: Leo Zhang <nguzcf@gmail.com>

doc: fix broken link in erasure-code.rst

Fixes: http://tracker.ceph.com/issues/19972
Signed-off-by: MinSheng Lin <minsheng.l@inwinstack.com>

cmake: workaound ccache issue with .S assembly files

when building with -DWITH_CCACHE=ON recent versions of ccache
(for example 3.3.3) seem to mangle object files from the cache
for source files that have a .S extension. This seems similar
but unrelated to PR #14633. The workaround here is to rename .S
files to .s (lower case).

Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>

Merge branch 'wip-sts-authorization'

Merge pull request #14307 from adamemerson/wip-sts-authorization

common: Add make_unique
submodule: Tencent's RapidJSON library
common: Add ISO-8601 Date Support
rgw: Add is_identity to AuthApplier class
rgw: Move globbing flags to header
buffer: Make the use of static areas more convenient
rgw: Add basic support for IAM policies
rgw: Build bucket permission and environment in req_state
rgw: Verify policies as WELL as ACLs
rgw: RESTful bucket policy ops
rgw: Write documentation for bucket policies

Reviewed-By: Casey Bodley <cbodley@redhat.com>
Reviewed-By: Radoslaw Zarzynski <rzarzynski@mirantis.com>
Reviewed-By: Kefu Chai <kchai@redhat.com>

rgw: Write documentation for bucket policies

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: RESTful bucket policy ops

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Verify policies as WELL as ACLs

Only hook into object/bucket checks for now. Once we have STS (giving
us Roles) or User/Group policies it will make sense to hook into
those, too.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Build bucket permission and environment in req_state

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Add basic support for IAM policies

This support is currently incomplete but should provide a starting
point.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

buffer: Make the use of static areas more convenient

Make three wrapper functions to tidy up the process of making a
bufferlist holding a single static buffer.

The lack of any decent handling of const in buffer::list makes me wax
wroth, but it's a bit much to fix right now.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Move globbing flags to header

So they can be used when calling the function

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Add is_identity to AuthApplier class

Eventually this will allow us to match all authentication information
against all specified principals in a policy.

Right now it handles users and wildcards.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common: Add ISO-8601 Date Support

For parsing and unparsing from ceph::real_time.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

submodule: Tencent's RapidJSON library

This is a high performance, MIT licensed JSON parsing library. It
provides a SAX interface so that I can compile an S3 policy without
building up a JSONObject tree in the middle that gets thrown away.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common: Add make_unique

There are parts of C++14 that are both useful and easy to
implement. This is one of them.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

ceph_test_rados_api_list: more fix LibRadosListNP.ListObjectsError

Follow-on to d7e6e8d60309e4800389b36f786b633d0ca2ec07; we need to
make sure the client gets the updated osdmap.

Fixes: http://tracker.ceph.com/issues/19963
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #13731 from liewegas/wip-19119

PendingReleaseNotes: warning about 'osd rm ...' and #19119

PendingReleaseNotes: warning about 'osd rm ...' and #19119

See http://tracker.ceph.com/issues/19119

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #14873 from liewegas/wip-bluefs-position

os/bluestore: put bluefs in the middle of the shared device

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #14902 from liewegas/wip-upmap

osd/OSDMap: improve upmap calculation

Reviewed-by: Loic Dachary <loic@dachary.org>

Merge pull request #14933 from liewegas/wip-asok-schema

common/perf_counters: make schema more friendly and update docs

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #14967 from tchaikov/wip-faster-cull

mgr: optimize DaemonStateIndex::cull() a little bit

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15002 from dachary/wip-19836-crush-pre-luminous

crush: encode can override weights with weight set

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15031 from tchaikov/wip-drop-unused-opts

config_opts: drop unused opts

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15043 from tchaikov/wip-19884

ceph-detect-init: detect init system by poking the system

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #15059 from liewegas/wip-mon-kvtype

mon: fix mon_keyvaluedb application

Reviewed-by: Kefu Chai <kchai@redhat.com>

rbd: remove unnecessary -ENOENT check for rbd cli

Signed-off-by: runsisi <runsisi@zte.com.cn>

librbd: translate error -ENOENT to 0 for list APIs

Signed-off-by: runsisi <runsisi@zte.com.cn>

Merge pull request #15032 from tchaikov/wip-doc-opts

doc: add descriptions for mon/mgr options

Reviewed-by: John Wilkins <jowilkin@redhat.com>

rgw: drop expat header in a couple of files

Since it is already a part of rgw_xml
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: drop using namespace std in header files

We're still having the include in include/type.h, so just sweeping
things under the rug for now

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

os/bluestore/KernelDevice: helpful warning when aio limit exhausted

Suggested-by: Dan Van Der Ster <daniel.vanderster@cern.ch>
Signed-off-by: Sage Weil <sage@redhat.com>

pybind/rbd: fix crash if more than 1024 images in trash bin

Signed-off-by: runsisi <runsisi@zte.com.cn>

Merge pull request #15127 from tchaikov/wip-vstart-mgr-debug-ms

vstart: "debug_ms=1" for mgr by default

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15103 from drunkard/master

doc: minor fixes in radosgw/

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15080 from hikycool/master

doc: update the usage of 'ceph-deploy purge'

Reviewed-by: Kefu Chai <kchai@redhat.com>

vstart: "debug_ms=1" for mgr by default

Signed-off-by: Kefu Chai <kchai@redhat.com>

common: Remove redundant includes

Fixes: http://tracker.ceph.com/issues/19883 (Partially)
Signed-off-by: Jos Collin <jcollin@redhat.com>

osd: fix argument-dependent lookup of swap()

because include/types.h has a 'using namespace std', the call to
::swap() had previously been selecting overloads from namespace std. but
once any other swap() functions are present in the global namespace,
argument-dependent lookup [1] will not consider those from std

for example, when common/sstring.hh has been included, its global swap()
function is the only overload considered, so calls to ::swap() result in
errors like this:

/home/cbodley/ceph/src/osd/osd_types.h: In member function ‘void ObjectModDesc::swap(ObjectModDesc&)’:
/home/cbodley/ceph/src/osd/osd_types.h:3135:56: error: no matching function for call to ‘swap(bool&, bool&)’
     ::swap(other.can_local_rollback, can_local_rollback);
                                                        ^
/home/cbodley/ceph/src/common/sstring.hh:589:6: note: candidate: template<class char_type, class size_type, size_type max_size> void swap(basic_sstring<char_type, size_type, Max>&, basic_sstring<char_type, size_type, Max>&)
 void swap(basic_sstring<char_type, size_type, max_size>& x,
      ^
/home/cbodley/ceph/src/common/sstring.hh:589:6: note:   template argument deduction/substitution failed:
/home/cbodley/ceph/src/osd/osd_types.h:3135:56: note:   mismatched types ‘basic_sstring<char_type, size_type, Max>’ and ‘bool’
     ::swap(other.can_local_rollback, can_local_rollback);

adding a `using std::swap;` to the calling scope and removing :: from
the call to `swap()` allows argument-dependent lookup to resolve the
overloads in both namespaces

[1] http://en.cppreference.com/w/cpp/language/adl

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa: silence upgrade test failure

The new fs setting standby_count_wanted is only avialable in luminous. Upgrade
tests were tripping on this.

Fixes: http://tracker.ceph.com/issues/19934
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

osd/PGLog: avoid pinning large buffers with ObjectModDesc

Accidentally removed by 5e0ec06376f832d32a6b1af390f925a59b03798f.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: encode snaps more efficiently

1- encode into a sized buffer.
2- do not needlessly copy the set<> to a vector<> before encoding.
set<> and vector<> encode identically.  Since we are converting from sorted
set<> to unsorted vector<>, the order doesn't change either.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: add test for crush type down health warnings

Signed-off-by: Neha Ojha <nojha@redhat.com>

osd/pglog: Remove loop through empty collection.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #15087 from liewegas/wip-thrash-ec

qa/suites/rados: fix ec thrashing

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15072 from liewegas/wip-crush-swap-bucket

mon,crush: add 'osd crush swap-bucket' command

Reviewed-by: Loic Dachary <loic@dachary.org>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15069 from tchaikov/wip-inttypes-cleanup

int_types.h: remove hacks to workaround old systems

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #15057 from liewegas/wip-osd-choose-cleanup

osd/PG: drop pre-firefly compat_mode for choose_*_acting

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #15042 from badone/wip-redundant-headers-3

common: Remove redundant includes

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14989 from liewegas/wip-19877

osd: fix past_intervals base case by adding epoch_pool_created to pg_history_t

Reviewed-by: Josh Durgin <jdurgin@redhat.com>