Merge PR #16891 into master

* refs/remotes/upstream/pull/16891/head:
mds/MDSDaemon: add 'is_valid=false' when failed to parse caps

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #17022 from ceph/wip-doc-sans-gitbuilder

Update docs to remove gitbuilder and add shaman references

Reviewed-By: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

doc/rados/configuration comment out '...' on ini blocks

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/rados/api comment out '...' on ini blocks

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/install describe shaman urls and how to retrieve list and repo files

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/install update international for non-US

Use of "international" is ambiguous, for non-US users, international
could mean US.

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/install remove repetitive dev release information for rpm distros

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/install remove mention of autobuild key for dev packages

Dev packages are no longer signed, and the key is not used at all

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/install remove Apache and FastCGI instructions for RGW

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc: update repo-access to point to ceph-ci.git

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #17025 from ceph/wip-doc-warnings

fix sphinx build warnings and errors

Reviewed-by: Dan Mick <dmick@redhat.com>
Reviewed-by: Andrew Schoen <aschoen@redhat.com>

doc: fix short underline on release-notes

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #15597 from ceph/wip-19200

selinux: Allow nvme devices

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc exclude man_index when not building man pages

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc: create an index/toctree only for man pages

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc: exclude mgr from man pages, define a man index.rst

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/release-notes use new ref label for erasure code profiles

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/rados/operations use new ref labels in health-checks

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/rados/operations add a ref label to erasure code profiles

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/rados/operations add a ref label to pool application

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/rados/operations use new ref label in health-checks

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/rados/operations add ref label for placement group number

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/rados/api/python document required_state manually

Signed-off-by: Alfredo Deza <adeza@redhat.com>

admin: bump sphinx to 1.6

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/rados/operations use the new ref label for crush map tunables

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/rados/operations add label for crush map tunables for ref linking

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/cephfs add label to health messages for use in refs linking

Signed-off-by: Alfredo Deza <adeza@redhat.com>

admin: do not allow errors or warnings from doc build

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/rados/operations include pg-repair document in the toctree

Signed-off-by: Alfredo Deza <adeza@redhat.com>

pybind fix rbd trash_move docstring, needed a newline

Signed-off-by: Alfredo Deza <adeza@redhat.com>

pybind: fix invalid doctring in Rados.require_state

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/rados/operations add missing link for crush pdf

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/rados/operations add newline after definition

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/radosgw add newline after field list

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/radosgw fix missing newline on definition

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/rados/api remove '*' in automethod call

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/logging fix unexpected indent in paragraph

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc: bump the max depth of the toctree, so that it can find nested indexes

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc escape and remove invalid chars in release notes, causing sphinx errors

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/rados use func reference for  rados_set_alloc_hint linking

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #16967 from liewegas/wip-upgrade-health

mon: fix legacy health checks in 'ceph status' during upgrade; fix jewel-x upgrade combo

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17012 from joscollin/wip-rgw-todo-warning

rgw: Drop #warning TODO

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #17037 from theanalyst/release-notes-12-1-4

doc: release notes for v12.1.4 Luminous

Reviewed-By: Gregory Farnum <gfarnum@redhat.com>
Reviewed-By: Ken Dreyer <ktdreyer@redhat.com>

doc: release notes for v12.1.4 Luminous

Adding the bits of changelog that went in post 12.1.3, one of the PRs
was directly cherry-picked onto Luminous and hence does not have a PR
reference.

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #16808 from liupan1111/wip-add-report

test/fio: generate db histogram to help debug rocksdb performance

Merge pull request #16987 from dzafman/wip-pidfile

test: Fix and enable test_pidfile.sh

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #17016 from chuanhong-wang/dev_rdma_dead_lock

Fix a dead lock when doing rdma performance test by fio

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #16649 from ztczll/zlrgw

rgw: remove unused variables

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #16971 from scienceluo/wip-luo-fix-deadcode-branch

rbd: fix logically dead code in function list_process_image

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

Merge pull request #17000 from gregsfortytwo/wip-20985-divergent-handling

osd: be more precise about our asserts and cases when rebuilding missing sets

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16753 from ceph/wip-fix-ppc64

Fix ppc64 support for ceph

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16934 from amitkumar50/cov-librbd-2

librbd: Initializing members image,operation,journal

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

selinux: Allow nvme devices

This commit allows nvme devices which use a different label than
standard block devices.

Fixes: http://tracker.ceph.com/issues/19200
Signed-off-by: Boris Ranto <branto@redhat.com>

Fix a dead lock when running fio: "RDMADispatcher::polling_stop" hold the lock and wait
"RDMADispatcher::polling" ending, then RDMADispatcher::polling can't hold the lock and
can't end its work.

Signed-off-by: Wang Chuanhong <chuanhong.wang@163.com>

rgw: Drop #warning TODO

Drop "#warning TODO" from rgw_crypt.cc,  as the tracker is created for this requirement and the
corresponding PR is in progress for a long time.

Tracker: http://tracker.ceph.com/issues/19851
PR: https://github.com/ceph/ceph/pull/14498

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #16976 from liewegas/wip-osd-empty

mgr: implement 'osd safe-to-destroy' and 'osd ok-to-stop' commands

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

Merge pull request #16625 from Yan-waller/wip-walle-0727testlibrados

test/librados: reorder ASSERT_EQ() arguments

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #16883 from liewegas/wip-20939

crush,mon: fix weight-set vs crush device classes

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

qa/suites/rados/objectstore/objectstore: less debug

Saw an ENOSPC.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #17007 from alex-mikheev/wip_rdma_notify_fix

src/msg/rdma: fixes failure on assert in notify()

Reviewed-by: Haomai Wang <haomai@xsky.com>

src/msg/rdma: fixes failure on assert in notify()

The commit fixes incorrect eventfd handling introduced in
2e75b876d1e8e9c2ac556808f958fcbfeaad7d52

Signed-off-by: Alex Mikheev <alexm@mellanox.com>

mds/MDSDaemon: add 'is_valid=false' when failed to parse caps

Fixes: http://tracker.ceph.com/issues/20990
Signed-off-by: Yanhu Cao <gmayyyha@gmail.com>

Merge pull request #16978 from gmayyyha/mgr-ms-verify-auth

mgr/DaemonServer.cc: add 'is_valid=false' when decode caps error

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #16906 from liewegas/wip-bluestore-stupid

os/bluestore: switch default allocator to stupid; test both bitmap and stupid in qa

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

crush/CrushWrapper: fixing timing of removal in remove_item_under

Do it after we reweight, not before

Signed-off-by: Sage Weil <sage@redhat.com>

crush/CrushWrapper: fix iterator invalidation in cleanup_dead_classes

Signed-off-by: Sage Weil <sage@redhat.com>

crush/CrushWrapper: keep weights and/or ids null if empty

Signed-off-by: Sage Weil <sage@redhat.com>

crush/builder: fix ENOENT when removing last bucket item

We were decrementing size and then breaking out ENOENT condition check.
Fix by decrementing size only after we break out of the loop and verify
we found the item.

Fix a follow-on bug by avoiding realloc when we have 0 items left.  This case
was never exercised before due to the ENOENT issue; now we return explicitly.
It's really not necessary to realloc at all, probably, since these are very
small arrays, but in any case leaving a single item allocation there in place of
a 0-length allocation is fine.  (And the 0-length allocation behvaior on realloc
is undefined.. may either return a pointer or NULL.)

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #17005 from liewegas/wip-verify-slowness

qa/suites/rados/verify/validater/valgrind: whitelist PG_

qa/suites/rados/verify/validater/valgrind: whitelist PG_

Peering might be slow due to valgrind.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #17004 from liewegas/wip-clock-skew-whitelist

qa/suites/rados/multimon/tasks/mon_lock_with_skew: whitelist PG_

qa/suites/rados/multimon/tasks/mon_lock_with_skew: whitelist PG_

Default pool pgs not up because mons too broken for OSDs to peer.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15921 from joscollin/wip-CID1411830-Uninitialized-pointer-field

test/osd: Non-static class members not initialized in UnsetRedirectOp

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

test/osd: Non-static class member not initialized in UnsetRedirectOp

Fixes the coverity scan report:
CID 1411830: Uninitialized pointer field (UNINIT_CTOR)
2. uninit_member: Non-static class member completion is not initialized in this constructor nor in any functions that it calls.
4. uninit_member: Non-static class member comp is not initialized in this constructor nor in any functions that it calls.

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #15922 from joscollin/wip-CID1411829-Uninitialized-scalar-field

test/osd: initialize Non-static class members in WeightedTestGenerator

Reviewed-by: Xie Xingguo <xie.xingguo@zte.com.cn>

Merge pull request #16924 from liewegas/wip-20923

os/bluestore: fail early on very large objects

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

test/osd/safe-to-destroy.sh: test 'osd safe-to-destroy'

This is hard with workunits/cephtool/test.sh because we don't
control the whole cluster.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr: implement 'osd safe-to-destroy' and 'ok-to-stop' commands

An osd is safe to destroy if

- we have osd_stat for it
- osd_stat indicates no pgs stored
- all pgs are known
- no pgs map to it

An osd is ok ot stop if

- we have pg stats
- no pgs will drop below min_size

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ClusterState: record osd_stat for out osds too

I'm not quite sure why we were doing this. :/

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: include number of locally stored PGs in osd_stat_t

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMap: add parse_osd_id_list helper

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16992 from liewegas/wip-20977

os/bluestore: do not segv on kraken upgrade debug print

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #16993 from liewegas/wip-mimic

common: kick off mimic

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #16994 from liewegas/wip-20983

os/bluestore: fix clone dirty_range again

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #16995 from liewegas/wip-20738

os/memstore: memstore_page_set=false

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #16996 from liewegas/wip-mgr-vs-clock-skew

qa/suites/rados/multimon: whitelist mgr down vs clock skew test

mon: correctly print out mds versions (instead of mon ones)

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

osd: be more precise about our asserts and cases when rebuilding missing sets and handling divergent priors

Fixes: http://tracker.ceph.com/issues/20985
Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #16898 from amitkumar50/cov-messages-6

messages: Initialization of member variables

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

os/bluestore: do not segv on kraken upgrade debug print

When loading an onode from kraken we have a compat path that calls
get_ref before the SharedBlob pointer is initialized.  This is fine except
that if debugging is enabled the operator<< on the Blob will segv on
printing *b.shared_blob (which is NULL).

Fix operator<< to print something else if it is NULL.  shared_blob does
get set up right after the call to decode() so having it be NULL at this
point is otherwise harmless.

Fixes: http://tracker.ceph.com/issues/20977
Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/multimon: whitelist mgr down vs clock skew test

Clock skew might make us fail the mgr.

Signed-off-by: Sage Weil <sage@redhat.com>

os/memstore: memstore_page_set=false

This regularly returns bad results, see http://tracker.ceph.com/issues/20738

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix clone dirty_range again

If we are cloning a blob for a 1 byte logical extent then dirty_range_begin
will equal _end and we won't dirty the source onode (with possibly newly
shared blobs).

Fix by using a separate flag to indicate whether we are dirtying instead
of overloading the begin/end markers for this.  Note that even if they
are equal dirty_range will still dirty the shard in question.

This is a result of 0ae5d92d42500e5ab08253a00bda47b957767ebc.

Fixes: http://tracker.ceph.com/issues/20983
Signed-off-by: Sage Weil <sage@redhat.com>

include/rados: nautilus is next

Signed-off-by: Sage Weil <sage@redhat.com>

include/ceph_features: define SERVER_NAUTILUS

Nautilus will be the next major release after luminous.

Signed-off-by: Sage Weil <sage@redhat.com>

include/ceph_features: incarnation 3 can begin!

With all upgrades passing through luminous, we can now start
reusing bits retired in luminous.  Our sentinel bitmask will be the
combination of SERVER_MIMIC and SERVER_JEWEL (i.e.,
CEPH_FEATUREMASK_SERVER_MIMIC).

Signed-off-by: Sage Weil <sage@redhat.com>

include/ceph_features: SERVER_M -> SERVER_MIMIC

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_release: kick off mimic 13.y.z

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16628 from theanalyst/rgw-policy-list-buckets

rgw: policy: support for s3 conditionals in ListBucket Op

Merge pull request #16968 from liewegas/wip-store-test-expect-death

ceph_test_objectstore: drop expect regex