ceph-volume tests centos7 use the new ansible syntax for lvm

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit fbce7ad593ac5342ee6d1769e04d7f3adadab78a)

ceph-volume tests use the new ansible syntax for lvm

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 8f0f202ec4b4f39d748b5aec013b0bc541439c2b)

ceph-volume tests create tests for the get_lv helper method

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 7584d64c0fd585fd0e02991ccb9cad2d147fb130)

ceph-volume tests create tests for the new arg validator

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit d5eb9640aa4b20eabc9b5fdd07559082346478c6)

ceph-volume util create a validator module for argparse

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 0ce77806bd04d69cd9d1cdcd71e886f7763a1eb2)

ceph-volume lvm.prepare enforce usage of vg/lv when preparing lvm devices

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 192fe4e1dd24713f33131583de4ca81b625bad98)

ceph-volume lvm.common update help values for vg/lv usage

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit a0286da13bc1edef96489d7d525a1cd8e7066acf)

ceph-volume lvm.create update docstring for vg/lv usage

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit f907f7a91dce2a5748dd4c07d5919fa56aae4a83)

Merge pull request #17197 from theanalyst/wip-20965-luminous

luminous: src/common/LogClient.cc: 310: FAILED assert(num_unsent <= log_queue.size())

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17202 from jdurgin/wip-lumionus-pg-log

osd: adjust osd_min_pg_log_entries

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

osd: adjust osd_min_pg_log_entries

Return osd_min_pg_log_entries to its original values, which matches
osd_pg_log_dups_tracked, so the extended dup log is not used in the
general case.

This helps address: http://tracker.ceph.com/issues/21026

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 1c4df03394312fe67f36448613d8b54cb1a0e2c9)

Merge pull request #17192 from idryomov/wip-krbd-unmap-tests-pool-luminous

luminous: qa: fix POOL_APP_NOT_ENABLED warning in krbd:unmap suite

common/LogClient: make last_log non-atomic

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit c36a98be97e15294f57c6a640fe6a1c277dce8a3)

common/LogClient: fix indentation

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit bf92a8a2693e0b6a01a861977a5376eb584c9e42)

common/LogClient: assign seq and queue atomically

The _get_mon_log_message() assumes that log_last and log_queue
are in sync, but it was previously possible to increment log_last
setting e.seq in do_log(), and only later queue it.  If a racing
thread ran get_mon_log_message() in the meantime it would fail
an assertion.

Fix by assigning the seq and queueing it atomically.  If the
cluster log is not enabled, use the get_next_seq() helper so that
graylog or syslog messages still have a seq assigned.

Fixes: http://tracker.ceph.com/issues/18209
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 1f8f58becf1ec53c729b0befeb1f19c601588c4a)

qa: fix POOL_APP_NOT_ENABLED warning in krbd:unmap suite

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit ad715b3368522a3913ccaa233abc9729f728e328)

Merge pull request #17117 from theanalyst/wip-21056-luminous

luminous: RGW: Get Bucket ACL does not honor the s3:GetBucketACL action

Reviewed-By: Casey Bodley <cbodley@redhat.com>

Merge pull request #17118 from theanalyst/wip-21054-luminous

luminous: rgw: GetObject Tagging needs to exit earlier if the object has no attributes

Reviewed-By: Casey Bodley <cbodley@redhat.com>

Merge pull request #17170 from ceph/luminous-volume-man-fix

luminous: doc add ceph-volume and ceph-volume-systemd man pages to CMakeLists file

Reviewed-by: Dan Mick <dmick@redhat.com>

doc add ceph-volume and ceph-volume-systemd man pages to CMakeLists file

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #17158 from ceph/wip-rm21030-backport

luminous: create the ceph-volume and ceph-volume-systemd man pages

doc: include ceph-disk and ceph-disk-volume man pages in index

Signed-off-by: Alfredo Deza <adeza@redhat.com>

debian: include ceph-volume and ceph-volume-systemd man pages

Signed-off-by: Alfredo Deza <adeza@redhat.com>

rpm: include ceph-volume and ceph-volume-systemd man pages

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/man create the ceph-volume man page

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/man create the ceph-volume-systemd man page

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #17124 from ceph/backport-ceph-volume-docs

ceph-volume docs

Merge pull request #17123 from ceph/backport-bz1478599

ceph-volume: Use a delimited CLI output parser instead of JSON

doc ignore ceph-volume when building man pages

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc: add LVM tag term to the glossary

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/dev include ceph-volume devel docs

Signed-off-by: Alfredo Deza <adeza@redhat.com>

docs: include ceph-volume's index file in the main toctree

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume update the systemd docstring to use the new API

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/glossary add terminology used by ceph-volume

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume create the systemd lvm reference

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume create the intro to ceph-volume

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume create the main index

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume add the lvm scan reference

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume create the lvm prepare reference

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume create the lvm activate reference

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/install use the new mgr-administrator-guide label

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/mgr add a label for the administrator guide

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/dev/ceph-volume add a toctree for the dev docs

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/dev/ceph-volume add systemd reference

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/dev/ceph-volume add plugin interface

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/dev/ceph-volume add the internal lvm api

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: lvm add a parser for the delimeted output in the CLI

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.api uses the new parses for vgs

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.api uses the new parser for lvs

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests adapt tests for the new LVM api parser

Signed-off-by: Alfredo Deza <adeza@redhat.com>

debian: include ceph-volume-systemd CLI tool

Signed-off-by: Alfredo Deza <adeza@redhat.com>

rgw: rgw_op: exit early if object has no attrs in GetObjectTagging

Check for the return value of get_obj_attrs and exit earlier in case of
errors

Fixes: http://tracker.ceph.com/issues/21010
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit b2246226cc11fcd87b4fd499c34434f82ca18654)

rgw: evaluate the correct bucket action for GetACL bucket operation

We currently evaluate s3:GetObjectAcl instead of s3:GetBucketAcl for
getting BucketACL

Fixes: http://tracker.ceph.com/issues/21013
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit 0e6b4f7862aacb2745e11cd7dbd16bcbec079bab)

Merge pull request #17074 from yuriw/wip_add_luminous

Decreased amount of jobs on master, kraken, luminous runs

Merge pull request #17077 from dillaman/wip-qa-rbd-mirror-data-pool-luminous

qa/workunits/rbd: test data pool is mirrored correctly

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits/rbd: test data pool is mirrored correctly

Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit c4c7075029f20f238585145c464a8c972d9ad038)

Decreased amount of jobs on master, kraken, luminous runs

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #17023 from adamwg/awg/rbd-mirror-data-pool

rbd-mirror: Set the data pool correctly when creating images

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc: Update mirroring docs to mention data-pool configuration

Signed-off-by: Adam Wolfe Gordon <awg@digitalocean.com>

12.1.4

rbd-mirror: Set the data pool correctly when creating images

Previously the data-pool option was ignored by rbd-mirror, so when using
erasure coding data for the image on the secondary site would end up in
the metadata pool.

Configure the data-pool using the data-pool's name from the primary site
when creating images on the secondary site.

Fixes: http://tracker.ceph.com/issues/20567
Signed-off-by: Adam Wolfe Gordon <awg@digitalocean.com>

Merge pull request #17001 from gregsfortytwo/wip-20985-divergent-handling-luminous

Wip 20985 divergent handling luminous

mgr: implement 'osd safe-to-destroy' and 'ok-to-stop' commands

An osd is safe to destroy if

- we have osd_stat for it
- osd_stat indicates no pgs stored
- all pgs are known
- no pgs map to it

An osd is ok ot stop if

- we have pg stats
- no pgs will drop below min_size

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit bf9380457bba5a834ffa2927c73165e0f1960332)

osd/osd_types: include number of locally stored PGs in osd_stat_t

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit c2945479f4364ba599262e00cbf200dff66617bb)

osd/OSDMap: add parse_osd_id_list helper

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 6fc33a046ba880e42302c3ff54b69747df47b9a2)

crush/CrushWrapper: fixing timing of removal in remove_item_under

Do it after we reweight, not before

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 46ded6eb5b099db783e6ff4a1baa44eb1300e9c3)

crush/CrushWrapper: fix iterator invalidation in cleanup_dead_classes

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 061c21786adc92a304467d1160f26ca09718b03f)

crush/CrushWrapper: keep weights and/or ids null if empty

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit b3838c83d501bf2f40bb799300cde049123b0eb0)

crush/builder: fix ENOENT when removing last bucket item

We were decrementing size and then breaking out ENOENT condition check.
Fix by decrementing size only after we break out of the loop and verify
we found the item.

Fix a follow-on bug by avoiding realloc when we have 0 items left.  This case
was never exercised before due to the ENOENT issue; now we return explicitly.
It's really not necessary to realloc at all, probably, since these are very
small arrays, but in any case leaving a single item allocation there in place of
a 0-length allocation is fine.  (And the 0-length allocation behvaior on realloc
is undefined.. may either return a pointer or NULL.)

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 5a982050c1daddee33bb2b5d63a5936723283765)

qa/workunits/mon/crush_ops.sh: test weight sets vs device classes

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 9c42597f096a3a08a552a7a14c6e5da415cf0037)

mon/OSDMonitor: remove choose_args when pool is removed

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 49c468b78849b09ad3db2f6028ebf588c6fd2ac3)

crush/CrushWrapper: fill in weight-sets when we build shadow trees

When we build the shadow buckets for the class hierarchies, we need
to fill in the weight-sets for each shadow bucket too.

Skip the ids vector for now since it's not yet used by anything.

Fixes: http://tracker.ceph.com/issues/20939
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 0ed55e6150e3d82b2955cf3e9fa0f01b36a6474a)

crush/CrushWrapper: remove unused 'unused' arg for trim_roots_with_classes

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 2649b27d13d364c6533d514b0d0df53fd6519551)

crush: do add/remove before updating weight-sets

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 1f62ee8408d8cff91df5fd23f2ddfd58197a5b10)

osd: Fix Paxos shutdown handling for commit_finish race

Fixes: http://tracker.ceph.com/issues/20921
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit c1404574895bc5a091f36b8119c7500fe1e287f1)

qa/suites/rados/verify/validater/valgrind: whitelist PG_

Peering might be slow due to valgrind.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 41e5a85308ba71caf7de0e516811ee3b1228b310)

qa/suites/rados/multimon/tasks/mon_lock_with_skew: whitelist PG_

Default pool pgs not up because mons too broken for OSDs to peer.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 12007044b1a039bd7d22e5a6c0253263bdaa0419)

Merge pull request #17003 from liewegas/wip-luminous-squelch-mon-down

qa/tasks/thrashosds-health.yaml: ignore MON_DOWN

qa/tasks/thrashosds-health.yaml: ignore MON_DOWN

See http://tracker.ceph.com/issues/20910

It's not clear why this is happening (it could just be load on the
test machines) but it's very noisy, so silencing it for now on
the luminous branch.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fail early on very large objects

We have a hard 4GB object size limit (although in practice we want
to be *well* below that!).

See http://tracker.ceph.com/issues/20923
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 70f6760d3793d5a06e3c583608d220b354e65f84)

os/bluestore: do not segv on kraken upgrade debug print

When loading an onode from kraken we have a compat path that calls
get_ref before the SharedBlob pointer is initialized.  This is fine except
that if debugging is enabled the operator<< on the Blob will segv on
printing *b.shared_blob (which is NULL).

Fix operator<< to print something else if it is NULL.  shared_blob does
get set up right after the call to decode() so having it be NULL at this
point is otherwise harmless.

Fixes: http://tracker.ceph.com/issues/20977
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 50523a225b2f415b8c3d82faeae59c0eede54663)

os/bluestore: fix clone dirty_range again

If we are cloning a blob for a 1 byte logical extent then dirty_range_begin
will equal _end and we won't dirty the source onode (with possibly newly
shared blobs).

Fix by using a separate flag to indicate whether we are dirtying instead
of overloading the begin/end markers for this.  Note that even if they
are equal dirty_range will still dirty the shard in question.

This is a result of 0ae5d92d42500e5ab08253a00bda47b957767ebc.

Fixes: http://tracker.ceph.com/issues/20983
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit d5ba7061ee588c232138af1d880faf09be4adeed)

os/memstore: memstore_page_set=false

This regularly returns bad results, see http://tracker.ceph.com/issues/20738

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 7c96868797702eb3567a7cf7aa0e0a7ce422612c)

qa/suites/rados/multimon: whitelist mgr down vs clock skew test

Clock skew might make us fail the mgr.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit ad23d7dc1fb21cbb330ad639d3a41613dc2a9c43)

mon: correctly print out mds versions (instead of mon ones)

Signed-off-by: Greg Farnum <gfarnum@redhat.com>
(cherry picked from commit 6f7545e0b58c3e25aff386f179c0fb77bb96adc2)

osd: be more precise about our asserts and cases when rebuilding missing sets and handling divergent priors

Fixes: http://tracker.ceph.com/issues/20985
Signed-off-by: Greg Farnum <gfarnum@redhat.com>
(cherry picked from commit a9921a2fdaab823bf348ae7fc773647c9664952d)

Merge commit 'c56d9c07b342c08419bbc18dcf2a4c5fae62b9cf' into luminous

12.1.3

rados/tool: fixup rados stat command hint

Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>
(cherry picked from commit 85f3feb912464dcedc888396b4d1f9b23d19cb82)

crush: "osd crush class rename" support

In 076a6abd80cc90ebcb901f908f880ef030721b2a I killed the 'class rename' command
and thought it was totally useless but I was wrong.

Consider the following user case:
(1) randomly choose some OSDs(e.g., from different hosts) and try to make them for private use only,
    say, by grouping them into 'pool1'
(2) ceph osd crush set-device-class pool1 'OSDs from (1)'
(3) ceph osd crush rule create-replicated rule_for_pool1 default host pool1
(4) ceph osd pool rename pool1 pool2
(5) ceph osd crush class rename pool1 pool2

From the above user case, we need to safely change a pool name without worrying
any risk of data migration. That is why the 'osd crush class rename' command
is still needed here.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit d792e8d528768eeccc6ad71af5aa3fd81780eb2e)

ceph_test_objectstore: drop expect regex

If logging is enabled (as it now is in teuthology) this won't match the
forked output.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 863468803de3f92f9bada6bffd010a204ee996d7)

qa: test that "fs new" correctly set the application_metadata

Signed-off-by: Greg Farnum <gfarnum@redhat.com>
(cherry picked from commit c85af7b14681be916cea1119f3f9cdf556e2707a)

mdsmon: treat the osdmon correctly when doing plugged updates

Make sure it's writeable before invoking changes, and propose_pending()
on it when we're done.
Make the PaxosService::C_RetryMessage public so we can do this from FSCommands.

Maybe-
Fixes: http://tracker.ceph.com/issues/20959
Signed-off-by: Greg Farnum <gfarnum@redhat.com>
(cherry picked from commit 435717791ec499f71c9d1485b1e4e63239a343e2)

mdsmon: don't add pool application metadata until running fully-luminous

Signed-off-by: Greg Farnum <gfarnum@redhat.com>
(cherry picked from commit bcd3554bc5a3965077af655498fd434910f13040)

12.1.3

Merge pull request #16969 from linuxbox2/wip-disable-dynreshard-L

rgw: disable dynamic reshading for 1st L point release

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

rgw: disable dynamic reshading for 1st L point release

Temporarily default RGW dynamic bucket resharding to OFF (usability
standup).

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

os/bluestore:  clearer comments, not slower code.

Signed-off-by: Mark Nelson <mnelson@redhat.com>
(cherry picked from commit da92220d9d6936ef1aa19333038b7a1d982485a4)

mon/Elector: force election epoch bump on start

We are generally careful when bumping the epoch so that we can join
existing rounds.  However, if we restart in the middle of an election,
and change versions, we need to be certain that our previous ACK (as
$version - 1) isn't accepted as truth for the restarted daemon (running
$version) keeping the same epoch.

The conservatism with bumping is to avoid spurious election cycles, but
mon restarts are more rare, and we need them here.

Fixes: http://tracker.ceph.com/issues/20949
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit ef425374250014393c1d432a3eda95179bb70537)

qa/suites/upgrade/kraken-x/stress-split: more whitelisting

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit b61be07d45e7cc1b1f4b606408bf4ed04853fb9b)