rbd-mirror: load/update image map state machine

Signed-off-by: Venky Shankar <vshankar@redhat.com>

rbd-mirror: track on-disk image to instance map

Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #17070 from ktdreyer/rpm-macros-in-comments

rpm: rm macros in comments

Reviewed-By: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #17043 from renhwztetecs/renhw-wip-pgmap-pool

mon/pgmap: update pool nearfull display

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17002 from mychoxin/yuanxin_adjust_blockdevice

os/bluestore: move aio_callback{,_priv} to base class BlockDevice

Reviewed-by: Pan Liu <liupan1111@gmail.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16940 from jcsp/wip-20955

mgr: add missing call to pick_addresses

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16407 from dzafman/wip-20243

osd: Improve size scrub error handling and ignore system attrs in xattr checking

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17062 from trociny/wip-qa-rbd-mirror-data-pool

qa/workunits/rbd: test data pool is mirrored correctly

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #16517 from gohighsec/master

rbd: check if an image is already mapped before rbd map

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #16591 from PCzhangPC/fix7

rbd: introduce commands of "image-meta ls/rm"

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17073 from dillaman/wip-20567

rbd-mirror: Set the data pool correctly when creating images

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc: Update mirroring docs to mention data-pool configuration

Signed-off-by: Adam Wolfe Gordon <awg@digitalocean.com>
(cherry picked from commit 1e001c19e98ac1cd2428dd66f047ea4d6b2b01b1)

rbd-mirror: Set the data pool correctly when creating images

Previously the data-pool option was ignored by rbd-mirror, so when using
erasure coding data for the image on the secondary site would end up in
the metadata pool.

Configure the data-pool using the data-pool's name from the primary site
when creating images on the secondary site.

Fixes: http://tracker.ceph.com/issues/20567
Signed-off-by: Adam Wolfe Gordon <awg@digitalocean.com>
(cherry picked from commit dffaac83d19de544e279a530393b3998f1e1fbb4)

Merge pull request #17069 from yuriw/wip_add_luminous

Decreased amount of jobs on master, kraken, luminous runs

Merge pull request #17040 from mdw-at-linuxbox/rgw-master-fix-plus

Rgw master fix plus

rpm: rm macros in comments

rpm expands all macros in a .spec file, even those in comments. Drop the
percent signs so rpm will not expand these.

This change silences rpmlint's warning about macros in comments.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #17066 from amitkumar50/cov-os-1

os: Initializing uninitialized members aio_info

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

os: Initializing uninitialized members aio_info

Fixes the coverity issue:

** 717332 Uninitialized pointer field
2. uninit_member: Non-static class member field iocb.data is not
initialized in this constructor nor in any functions that it calls.
4. uninit_member: Non-static class member field iocb.key is not
initialized in this constructor nor in any functions that it calls.
6. uninit_member: Non-static class member field iocb.__pad2 is not
initialized in this constructor nor in any functions that it calls.
8. uninit_member: Non-static class member field iocb.aio_lio_opcode
is not initialized in this constructor nor in any functions that it calls.
10. uninit_member: Non-static class member field iocb.aio_reqprio
is not initialized in this constructor nor in any functions that it calls.

CID 717332 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR)
12. uninit_member: Non-static class member field iocb.aio_fildes is not
initialized in this constructor nor in any functions that it calls.

Signed-off-by: Amit Kumar amitkuma@redhat.com

Decreased amount of jobs on master, kraken, luminous runs

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #16896 from amitkumar50/cov-librado-2

rados: Initializing members of librados/TestCase.h

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16939 from trociny/wip-nonlinux-fsx

test/librbd: make fsx build on non-linux platform

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17027 from joscollin/wip-cleanup-Wsign-compare

test/osd: silence warnings from -Wsign-compare

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

qa/workunits/rbd: test data pool is mirrored correctly

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge PR #16891 into master

* refs/remotes/upstream/pull/16891/head:
mds/MDSDaemon: add 'is_valid=false' when failed to parse caps

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #17022 from ceph/wip-doc-sans-gitbuilder

Update docs to remove gitbuilder and add shaman references

Reviewed-By: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

doc/rados/configuration comment out '...' on ini blocks

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/rados/api comment out '...' on ini blocks

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/install describe shaman urls and how to retrieve list and repo files

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/install update international for non-US

Use of "international" is ambiguous, for non-US users, international
could mean US.

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/install remove repetitive dev release information for rpm distros

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/install remove mention of autobuild key for dev packages

Dev packages are no longer signed, and the key is not used at all

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/install remove Apache and FastCGI instructions for RGW

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc: update repo-access to point to ceph-ci.git

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #17025 from ceph/wip-doc-warnings

fix sphinx build warnings and errors

Reviewed-by: Dan Mick <dmick@redhat.com>
Reviewed-by: Andrew Schoen <aschoen@redhat.com>

doc: fix short underline on release-notes

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #15597 from ceph/wip-19200

selinux: Allow nvme devices

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc exclude man_index when not building man pages

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc: create an index/toctree only for man pages

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc: exclude mgr from man pages, define a man index.rst

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/release-notes use new ref label for erasure code profiles

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/rados/operations use new ref labels in health-checks

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/rados/operations add a ref label to erasure code profiles

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/rados/operations add a ref label to pool application

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/rados/operations use new ref label in health-checks

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/rados/operations add ref label for placement group number

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/rados/api/python document required_state manually

Signed-off-by: Alfredo Deza <adeza@redhat.com>

admin: bump sphinx to 1.6

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/rados/operations use the new ref label for crush map tunables

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/rados/operations add label for crush map tunables for ref linking

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/cephfs add label to health messages for use in refs linking

Signed-off-by: Alfredo Deza <adeza@redhat.com>

admin: do not allow errors or warnings from doc build

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/rados/operations include pg-repair document in the toctree

Signed-off-by: Alfredo Deza <adeza@redhat.com>

pybind fix rbd trash_move docstring, needed a newline

Signed-off-by: Alfredo Deza <adeza@redhat.com>

pybind: fix invalid doctring in Rados.require_state

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/rados/operations add missing link for crush pdf

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/rados/operations add newline after definition

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/radosgw add newline after field list

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/radosgw fix missing newline on definition

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/rados/api remove '*' in automethod call

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/logging fix unexpected indent in paragraph

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc: bump the max depth of the toctree, so that it can find nested indexes

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc escape and remove invalid chars in release notes, causing sphinx errors

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/rados use func reference for  rados_set_alloc_hint linking

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #16967 from liewegas/wip-upgrade-health

mon: fix legacy health checks in 'ceph status' during upgrade; fix jewel-x upgrade combo

Reviewed-by: Kefu Chai <kchai@redhat.com>

mon/pgmap: update pool nearfull display

if "pool full" and "pool nearfull" show the same,
we will not be able to distinguish between these two scenes in health check.

Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

radosgw: fix small sized chunked uploads with s3

The logic to compute the number of bytes to copy with a chunked file
upload was failing to take into account the amount of data that had been
consumed, but was still pending in the parse buffer.  This would cause
strange behavior ranging from "just works" to "fails".  There was also a
strange "works, one byte at a time" mode.  Using the correct stream_pos
offset eliminates the bad behavior.

This fix also adds debug statements to make the correct behavior easier to see.

Fixes: http://tracker.ceph.com/issues/21003
Signed-off-by: Marcus Watts <mwatts@redhat.com>

Merge pull request #17012 from joscollin/wip-rgw-todo-warning

rgw: Drop #warning TODO

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #17037 from theanalyst/release-notes-12-1-4

doc: release notes for v12.1.4 Luminous

Reviewed-By: Gregory Farnum <gfarnum@redhat.com>
Reviewed-By: Ken Dreyer <ktdreyer@redhat.com>

doc: release notes for v12.1.4 Luminous

Adding the bits of changelog that went in post 12.1.3, one of the PRs
was directly cherry-picked onto Luminous and hence does not have a PR
reference.

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #16808 from liupan1111/wip-add-report

test/fio: generate db histogram to help debug rocksdb performance

test/osd: silence warnings from -Wsign-compare

The following warning appears during build:

In file included from ceph/src/test/osd/TestOSDMap.cc:2:0:
ceph/src/googletest/googletest/include/gtest/gtest.h: In instantiation of 'testing::AssertionResult testing::internal::CmpHelperEQ(const char*, const char*, const T1&, const T2&) [with T1 = int; T2 = long unsigned int]':
ceph/src/googletest/googletest/include/gtest/gtest.h:1421:23:   required from 'static testing::AssertionResult testing::internal::EqHelper<lhs_is_null_literal>::Compare(const char*, const char*, const T1&, const T2&) [with T1 = int; T2 = long unsigned int; bool lhs_is_null_literal = false]'
ceph/src/test/osd/TestOSDMap.cc:456:3:   required from here
ceph/src/googletest/googletest/include/gtest/gtest.h:1392:11: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
   if (lhs == rhs) {
           ^
Signed-off-by: Jos Collin <jcollin@redhat.com>

radosgw: fix zero sized chunked uploads with s3

s3 allows for zero sized objects, which can be
created using chunked uploads.  It is important to
call "recv_body (and ChunkMeta::create_next()) even
on zero sized objects, so that the proper chunk signature
can be presented at signature verification time.

Fixes: http://tracker.ceph.com/issues/21000
Signed-off-by: Marcus Watts <mwatts@redhat.com>

rgw: replace '+' with "%20" in canonical query string for s3 v4 auth.

( note: this patch modified: now replaces "+" with " ".  mwatts@redhat.com )

Fixes: http://tracker.ceph.com/issues/20501
Signed-off-by: Zhang Shaowen <zhangshaowen@cmss.chinamobile.com>

Merge pull request #16987 from dzafman/wip-pidfile

test: Fix and enable test_pidfile.sh

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #17016 from chuanhong-wang/dev_rdma_dead_lock

Fix a dead lock when doing rdma performance test by fio

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #16649 from ztczll/zlrgw

rgw: remove unused variables

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #16971 from scienceluo/wip-luo-fix-deadcode-branch

rbd: fix logically dead code in function list_process_image

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

Merge pull request #17000 from gregsfortytwo/wip-20985-divergent-handling

osd: be more precise about our asserts and cases when rebuilding missing sets

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16753 from ceph/wip-fix-ppc64

Fix ppc64 support for ceph

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16934 from amitkumar50/cov-librbd-2

librbd: Initializing members image,operation,journal

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

selinux: Allow nvme devices

This commit allows nvme devices which use a different label than
standard block devices.

Fixes: http://tracker.ceph.com/issues/19200
Signed-off-by: Boris Ranto <branto@redhat.com>

Fix a dead lock when running fio: "RDMADispatcher::polling_stop" hold the lock and wait
"RDMADispatcher::polling" ending, then RDMADispatcher::polling can't hold the lock and
can't end its work.

Signed-off-by: Wang Chuanhong <chuanhong.wang@163.com>

rgw: Drop #warning TODO

Drop "#warning TODO" from rgw_crypt.cc,  as the tracker is created for this requirement and the
corresponding PR is in progress for a long time.

Tracker: http://tracker.ceph.com/issues/19851
PR: https://github.com/ceph/ceph/pull/14498

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #16976 from liewegas/wip-osd-empty

mgr: implement 'osd safe-to-destroy' and 'osd ok-to-stop' commands

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

Merge pull request #16625 from Yan-waller/wip-walle-0727testlibrados

test/librados: reorder ASSERT_EQ() arguments

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #16883 from liewegas/wip-20939

crush,mon: fix weight-set vs crush device classes

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

qa/suites/rados/objectstore/objectstore: less debug

Saw an ENOSPC.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #17007 from alex-mikheev/wip_rdma_notify_fix

src/msg/rdma: fixes failure on assert in notify()

Reviewed-by: Haomai Wang <haomai@xsky.com>

src/msg/rdma: fixes failure on assert in notify()

The commit fixes incorrect eventfd handling introduced in
2e75b876d1e8e9c2ac556808f958fcbfeaad7d52

Signed-off-by: Alex Mikheev <alexm@mellanox.com>

mds/MDSDaemon: add 'is_valid=false' when failed to parse caps

Fixes: http://tracker.ceph.com/issues/20990
Signed-off-by: Yanhu Cao <gmayyyha@gmail.com>

Merge pull request #16978 from gmayyyha/mgr-ms-verify-auth

mgr/DaemonServer.cc: add 'is_valid=false' when decode caps error

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #16906 from liewegas/wip-bluestore-stupid

os/bluestore: switch default allocator to stupid; test both bitmap and stupid in qa

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

crush/CrushWrapper: fixing timing of removal in remove_item_under

Do it after we reweight, not before

Signed-off-by: Sage Weil <sage@redhat.com>

crush/CrushWrapper: fix iterator invalidation in cleanup_dead_classes

Signed-off-by: Sage Weil <sage@redhat.com>

crush/CrushWrapper: keep weights and/or ids null if empty

Signed-off-by: Sage Weil <sage@redhat.com>

crush/builder: fix ENOENT when removing last bucket item

We were decrementing size and then breaking out ENOENT condition check.
Fix by decrementing size only after we break out of the loop and verify
we found the item.

Fix a follow-on bug by avoiding realloc when we have 0 items left.  This case
was never exercised before due to the ENOENT issue; now we return explicitly.
It's really not necessary to realloc at all, probably, since these are very
small arrays, but in any case leaving a single item allocation there in place of
a 0-length allocation is fine.  (And the 0-length allocation behvaior on realloc
is undefined.. may either return a pointer or NULL.)

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #17005 from liewegas/wip-verify-slowness

qa/suites/rados/verify/validater/valgrind: whitelist PG_

qa/suites/rados/verify/validater/valgrind: whitelist PG_

Peering might be slow due to valgrind.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #17004 from liewegas/wip-clock-skew-whitelist

qa/suites/rados/multimon/tasks/mon_lock_with_skew: whitelist PG_

qa/suites/rados/multimon/tasks/mon_lock_with_skew: whitelist PG_

Default pool pgs not up because mons too broken for OSDs to peer.

Signed-off-by: Sage Weil <sage@redhat.com>