osd: shutdown our reserver_finisher earlier in the process

This finisher thread has a lot of callbacks which can hold PGRefs. Make
sure we drain them out before checking that all the PGs have finished
and have no outstanding references.

Moving this should be safe; we've already stopped the op thread et al
and the only things still running are the OSDService's objecter_finisher,
recovery_request_timer, and snap_sleep_timer (which has definitely been emptied
by the time we get here as it's synchronously cleared out on PG shutdown).

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

osd: Reset() the snaptrimmer on shutdown

We were failing to exit various wait states which held PGRefs. Error!

Fixes: http://tracker.ceph.com/issues/19931
Signed-off-by: Greg Farnum <gfarnum@redhat.com>

osd: make sure we drop Backoff PGRefs on shutdown

Fixes: http://tracker.ceph.com/issues/19931
Signed-off-by: Greg Farnum <gfarnum@redhat.com>

osd: print out pg debug ref counts on acquire/drop

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

osd: repair the PG_DEBUG_REFS build

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

osd: use PGRef instead of boost::intrusive_ptr<PG> everywhere

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #15148 from zealoussnow/wip-spec

build: move bash_completion.d/ceph to ceph-common

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15160 from bassam/pr-system-boost

cmake: fix boost components for WITH_SYSTEM_BOOST

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15107 from guihecheng/rgw_file-cleanup

rgw: use pre-defined calls to replace raw flag operation

rgw_file: use pre-defined calls to replace raw flag operation

Signed-off-by: Gui Hecheng <guihecheng@cmss.chinamobile.com>

Merge pull request #15053 from dillaman/wip-librbd-blkin

blkin: librbd trace hooks

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

blkin: rbd object map update traces

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

blkin: initial trace hooks for IO path

To collect blkin traced for RBD, set the "rbd blkin trace all"
configuration parameter to true.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

blkin: add config variable rbd_blkin_trace_all

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

blkin: improved objecter trace events

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

blkin: librados should start new child trace

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

blkin: improve trace visibility into object cacher

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

blkin: pass trace information to object cacher

Signed-off-by: Victor Araujo <ve.ar91@gmail.com>
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

common/zipkin_trace: added missing include for encoding

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test/librados_test_stub: add new aio_operate overloads for tracing

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10736 from oliveiradan/SebastienHan_config_diff_get_fr

common/ceph_context: 'config diff get' option added

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #14914 from neha-ojha/wip-mon-crushtype-health-warn

mon: add crush type down health warnings

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15086 from joscollin/wip-cleanup-redundant-headers-1

common: Remove redundant includes

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #15109 from tchaikov/wip-mgrmon-send-digests-using-ticks

mon/MgrMonitor: send digests only if is_active()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15116 from liewegas/wip-bluestore-aio-warning

os/bluestore/KernelDevice: helpful warning when aio limit exhausted

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #15117 from liewegas/wip-luminous-legacy-check

qa/suites/rados/thrash: make sure osds have map before legacy scrub

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15120 from liewegas/wip-ec-buffer

osd: reduce buffer pinning from EC entries

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15121 from ivancich/wip-remove-an-unnecessary-loop

osd/pglog: remove loop through empty collection

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15124 from cbodley/wip-osd-swap

osd: fix argument-dependent lookup of swap()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15138 from liewegas/wip-list-objects-more

ceph_test_rados_api_list: more fix LibRadosListNP.ListObjectsError

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

cmake: fix boost components for WITH_SYSTEM_BOOST

when using WITH_SYSTEM_BOOST dont set header-only packages
for BOOST_COMPONENTS. On some distros these packages dont
exist.

Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>

Merge pull request #15126 from batrick/i19934

qa: silence upgrade test failure

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15037 from trociny/wip-19875

librbd: optimize copy-up to add hints only once to object op

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #15085 from runsisi/wip-fix-trashlist

librbd: do not raise an error if trash list returns -ENOENT

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #15134 from runsisi/wip-fix-trashpy

pybind/rbd: fix crash if more than 1024 images in trash bin

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge branch pull request #13551 from pritha-srivastava/wip-rgw-role-tenant

rgw: Replaced rgw userid with tenant in ARN for Roles.
rgw: Adding roles under the scope of tenants.
rgw: Added code for Role input validation.
rgw: Added code to fetch a role by its id.
rgw: Added code to allow tenant name as part of role name also.
rgw: Better reporting of missing input values in case of radosgw-admin
     commands for Roles.

Reviewed-By: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #15152 from guihecheng/rgw_file-extra-rele

rgw_file: removed extra rele() on fs in rgw_umount()

Merge pull request #15142 from bassam/pr-ccache-s-files

cmake: workaound ccache issue with .S assembly files

Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw_file: fix extra rele on fs in close

We got exactly a refcnt=0 after rele in close,
no need to rele twice.

Signed-off-by: Gui Hecheng <guihecheng@cmss.chinamobile.com>

Merge pull request #15137 from theanalyst/rgw/cleanup/header-drop

rgw: drop using std ns in header files and other cleanups

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15145 from zealoussnow/wip-0518

mailmap: Leo Zhang infomation and affiliation

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

build: move bash_completion.d/ceph from debian/ceph-base.install to debian/ceph-common.install

when only ceph-common is installed, can also use ceph completion

Signed-off-by: Leo Zhang <nguzcf@gmail.com>

Merge pull request #15143 from minshenglin/fix_broken_link

doc: fix broken link in erasure-code.rst

Reviewed-by: Kefu Chai <kchai@redhat.com>

build: move bash_completion.d/ceph to ceph-common

When only ceph-common is installed, can also use ceph completion

Signed-off-by: Leo Zhang <nguzcf@gmail.com>

mailmap: Leo Zhang infomation and affiliation

Signed-off-by: Leo Zhang <nguzcf@gmail.com>

doc: fix broken link in erasure-code.rst

Fixes: http://tracker.ceph.com/issues/19972
Signed-off-by: MinSheng Lin <minsheng.l@inwinstack.com>

cmake: workaound ccache issue with .S assembly files

when building with -DWITH_CCACHE=ON recent versions of ccache
(for example 3.3.3) seem to mangle object files from the cache
for source files that have a .S extension. This seems similar
but unrelated to PR #14633. The workaround here is to rename .S
files to .s (lower case).

Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>

Merge branch 'wip-sts-authorization'

Merge pull request #14307 from adamemerson/wip-sts-authorization

common: Add make_unique
submodule: Tencent's RapidJSON library
common: Add ISO-8601 Date Support
rgw: Add is_identity to AuthApplier class
rgw: Move globbing flags to header
buffer: Make the use of static areas more convenient
rgw: Add basic support for IAM policies
rgw: Build bucket permission and environment in req_state
rgw: Verify policies as WELL as ACLs
rgw: RESTful bucket policy ops
rgw: Write documentation for bucket policies

Reviewed-By: Casey Bodley <cbodley@redhat.com>
Reviewed-By: Radoslaw Zarzynski <rzarzynski@mirantis.com>
Reviewed-By: Kefu Chai <kchai@redhat.com>

rgw: Write documentation for bucket policies

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: RESTful bucket policy ops

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Verify policies as WELL as ACLs

Only hook into object/bucket checks for now. Once we have STS (giving
us Roles) or User/Group policies it will make sense to hook into
those, too.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Build bucket permission and environment in req_state

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Add basic support for IAM policies

This support is currently incomplete but should provide a starting
point.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

buffer: Make the use of static areas more convenient

Make three wrapper functions to tidy up the process of making a
bufferlist holding a single static buffer.

The lack of any decent handling of const in buffer::list makes me wax
wroth, but it's a bit much to fix right now.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Move globbing flags to header

So they can be used when calling the function

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Add is_identity to AuthApplier class

Eventually this will allow us to match all authentication information
against all specified principals in a policy.

Right now it handles users and wildcards.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common: Add ISO-8601 Date Support

For parsing and unparsing from ceph::real_time.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

submodule: Tencent's RapidJSON library

This is a high performance, MIT licensed JSON parsing library. It
provides a SAX interface so that I can compile an S3 policy without
building up a JSONObject tree in the middle that gets thrown away.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common: Add make_unique

There are parts of C++14 that are both useful and easy to
implement. This is one of them.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

ceph_test_rados_api_list: more fix LibRadosListNP.ListObjectsError

Follow-on to d7e6e8d60309e4800389b36f786b633d0ca2ec07; we need to
make sure the client gets the updated osdmap.

Fixes: http://tracker.ceph.com/issues/19963
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #13731 from liewegas/wip-19119

PendingReleaseNotes: warning about 'osd rm ...' and #19119

PendingReleaseNotes: warning about 'osd rm ...' and #19119

See http://tracker.ceph.com/issues/19119

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #14873 from liewegas/wip-bluefs-position

os/bluestore: put bluefs in the middle of the shared device

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #14902 from liewegas/wip-upmap

osd/OSDMap: improve upmap calculation

Reviewed-by: Loic Dachary <loic@dachary.org>

Merge pull request #14933 from liewegas/wip-asok-schema

common/perf_counters: make schema more friendly and update docs

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #14967 from tchaikov/wip-faster-cull

mgr: optimize DaemonStateIndex::cull() a little bit

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15002 from dachary/wip-19836-crush-pre-luminous

crush: encode can override weights with weight set

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15031 from tchaikov/wip-drop-unused-opts

config_opts: drop unused opts

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15043 from tchaikov/wip-19884

ceph-detect-init: detect init system by poking the system

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #15059 from liewegas/wip-mon-kvtype

mon: fix mon_keyvaluedb application

Reviewed-by: Kefu Chai <kchai@redhat.com>

rbd: remove unnecessary -ENOENT check for rbd cli

Signed-off-by: runsisi <runsisi@zte.com.cn>

librbd: translate error -ENOENT to 0 for list APIs

Signed-off-by: runsisi <runsisi@zte.com.cn>

Merge pull request #15032 from tchaikov/wip-doc-opts

doc: add descriptions for mon/mgr options

Reviewed-by: John Wilkins <jowilkin@redhat.com>

rgw: drop expat header in a couple of files

Since it is already a part of rgw_xml
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: drop using namespace std in header files

We're still having the include in include/type.h, so just sweeping
things under the rug for now

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

os/bluestore/KernelDevice: helpful warning when aio limit exhausted

Suggested-by: Dan Van Der Ster <daniel.vanderster@cern.ch>
Signed-off-by: Sage Weil <sage@redhat.com>

pybind/rbd: fix crash if more than 1024 images in trash bin

Signed-off-by: runsisi <runsisi@zte.com.cn>

Merge pull request #15127 from tchaikov/wip-vstart-mgr-debug-ms

vstart: "debug_ms=1" for mgr by default

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15103 from drunkard/master

doc: minor fixes in radosgw/

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15080 from hikycool/master

doc: update the usage of 'ceph-deploy purge'

Reviewed-by: Kefu Chai <kchai@redhat.com>

vstart: "debug_ms=1" for mgr by default

Signed-off-by: Kefu Chai <kchai@redhat.com>

common: Remove redundant includes

Fixes: http://tracker.ceph.com/issues/19883 (Partially)
Signed-off-by: Jos Collin <jcollin@redhat.com>

osd: fix argument-dependent lookup of swap()

because include/types.h has a 'using namespace std', the call to
::swap() had previously been selecting overloads from namespace std. but
once any other swap() functions are present in the global namespace,
argument-dependent lookup [1] will not consider those from std

for example, when common/sstring.hh has been included, its global swap()
function is the only overload considered, so calls to ::swap() result in
errors like this:

/home/cbodley/ceph/src/osd/osd_types.h: In member function ‘void ObjectModDesc::swap(ObjectModDesc&)’:
/home/cbodley/ceph/src/osd/osd_types.h:3135:56: error: no matching function for call to ‘swap(bool&, bool&)’
     ::swap(other.can_local_rollback, can_local_rollback);
                                                        ^
/home/cbodley/ceph/src/common/sstring.hh:589:6: note: candidate: template<class char_type, class size_type, size_type max_size> void swap(basic_sstring<char_type, size_type, Max>&, basic_sstring<char_type, size_type, Max>&)
 void swap(basic_sstring<char_type, size_type, max_size>& x,
      ^
/home/cbodley/ceph/src/common/sstring.hh:589:6: note:   template argument deduction/substitution failed:
/home/cbodley/ceph/src/osd/osd_types.h:3135:56: note:   mismatched types ‘basic_sstring<char_type, size_type, Max>’ and ‘bool’
     ::swap(other.can_local_rollback, can_local_rollback);

adding a `using std::swap;` to the calling scope and removing :: from
the call to `swap()` allows argument-dependent lookup to resolve the
overloads in both namespaces

[1] http://en.cppreference.com/w/cpp/language/adl

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa: silence upgrade test failure

The new fs setting standby_count_wanted is only avialable in luminous. Upgrade
tests were tripping on this.

Fixes: http://tracker.ceph.com/issues/19934
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

osd/PGLog: avoid pinning large buffers with ObjectModDesc

Accidentally removed by 5e0ec06376f832d32a6b1af390f925a59b03798f.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: encode snaps more efficiently

1- encode into a sized buffer.
2- do not needlessly copy the set<> to a vector<> before encoding.
set<> and vector<> encode identically.  Since we are converting from sorted
set<> to unsorted vector<>, the order doesn't change either.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: add test for crush type down health warnings

Signed-off-by: Neha Ojha <nojha@redhat.com>

osd/pglog: Remove loop through empty collection.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #15087 from liewegas/wip-thrash-ec

qa/suites/rados: fix ec thrashing

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15072 from liewegas/wip-crush-swap-bucket

mon,crush: add 'osd crush swap-bucket' command

Reviewed-by: Loic Dachary <loic@dachary.org>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15069 from tchaikov/wip-inttypes-cleanup

int_types.h: remove hacks to workaround old systems

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #15057 from liewegas/wip-osd-choose-cleanup

osd/PG: drop pre-firefly compat_mode for choose_*_acting

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #15042 from badone/wip-redundant-headers-3

common: Remove redundant includes

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14989 from liewegas/wip-19877

osd: fix past_intervals base case by adding epoch_pool_created to pg_history_t

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mon/OSDMonitor: introduce debug option to allow filestore for ec overwrites

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15113 from trociny/wip-unittest_librbd-cleanup

test/librbd: unit tests cleanup

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #15093 from dillaman/wip-19929

librbd: reacquire lock should update lock owner client id

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

qa/suites/rados/thrash: make sure osds have map before legacy scrub

The OSDs must have a map reflecting the require_luminous flag in order
for the legacy conversion to happen.  A quick rados bench should ensure
that.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #10690 from ceph/wip-rgw-polymorphic-errors

rgw: polymorphic error codes

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15096 from jcsp/wip-journalrepair-test

qa: simplify TestJournalRepair

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>