fuse: use c++ allocations for group list

Valgrind is unhappy about our turning on supplimentary group handling
with fuse by default. The problem is that we end up calling delete to
free the supplimentary gids list, but fuse uses malloc to allocate it.

Note that I was initially concerned that I needed to use malloc and
free there to handle the case of userland calling ceph_userperm_new,
but we leave freeing the pointer up to the caller in that case.

Convert fuse to use new/delete to allocate and free the group lists
instead.

Tracker: http://tracker.ceph.com/issues/21065
Signed-off-by: Jeff Layton <jlayton@redhat.com>
(cherry picked from commit f4fe5e2d524f8cca74f80a8a80fcd3e82b9effcb)

qa/suites/upgrade/jewel-x/parallel: tolerate laggy mgr

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit bf296018ff5b0a610869b8cbc8e385038aa6e2b7)

qa/suites/upgrade/jewel-x/stress-split: tolerate sloppy past_intervals

This is harmless in general, esp during upgrade.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit d5d5d7d1d22fc8da7b7df21cf527aa4c9ed54c9f)

Merge pull request #17208 from ceph/backport-bz1484002

luminous ceph-volume: use unique logical volumes

Merge pull request #16985 from dzafman/wip-standalone-luminous

luminous: tests: qa/standalone: misc fixes

Merge pull request #17113 from theanalyst/wip-luminous-20962

luminous: rgw: Fix rgw not responding occasionally when receiving SIGHUP signal.

Merge pull request #17114 from theanalyst/wip-21043-luminous

luminous: rgw: S3 v4 auth fails when query string contains

Merge pull request #17172 from tchaikov/wip-luminous-pr-16887

luminous: mon: fix wrong mon-num counting logic of 'ceph features' command

Merge pull request #17173 from tchaikov/wip-luminous-pr-16940

luminous: mgr: add missing call to pick_addresses

Merge pull request #17174 from tchaikov/wip-luminous-pr-17059

luminous: compressor: conditionalize on HAVE_LZ4

Merge pull request #17176 from tchaikov/wip-luminous-pr-16967

luminous: mon: fix legacy health checks in 'ceph status' during upgrade; fix jewel-x upgrade combo

Merge pull request #17191 from tchaikov/wip-luminous-pr-17065

luminous: mon/OSDMonitor: do not send_pg_creates with stale info

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17193 from theanalyst/wip-21048-luminous

luminous: Include front/back interface names in OSD metadata

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17195 from theanalyst/wip-21077-luminous

luminous: osd: osd_scrub_during_recovery only considers primary, not replicas

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #17198 from theanalyst/wip-21079-luminous

luminous: mon: bug in functon reweight_by_utilization

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/suites/upgrade/jewel-x/parallel: tolerate OBJECT_MISPLACED

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 5455f599b3a156c0f85f42d7faef8c5a59359982)

qa/suites/upgarde/jewel-x/parallel: tolerate mgr warning

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 2504ab167518ee48e5e1fd4d3efccb072a186442)

ceph-volume tests centos7 use the new ansible syntax for lvm

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit fbce7ad593ac5342ee6d1769e04d7f3adadab78a)

ceph-volume tests use the new ansible syntax for lvm

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 8f0f202ec4b4f39d748b5aec013b0bc541439c2b)

ceph-volume tests create tests for the get_lv helper method

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 7584d64c0fd585fd0e02991ccb9cad2d147fb130)

ceph-volume tests create tests for the new arg validator

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit d5eb9640aa4b20eabc9b5fdd07559082346478c6)

ceph-volume util create a validator module for argparse

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 0ce77806bd04d69cd9d1cdcd71e886f7763a1eb2)

ceph-volume lvm.prepare enforce usage of vg/lv when preparing lvm devices

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 192fe4e1dd24713f33131583de4ca81b625bad98)

ceph-volume lvm.common update help values for vg/lv usage

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit a0286da13bc1edef96489d7d525a1cd8e7066acf)

ceph-volume lvm.create update docstring for vg/lv usage

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit f907f7a91dce2a5748dd4c07d5919fa56aae4a83)

Merge pull request #17197 from theanalyst/wip-20965-luminous

luminous: src/common/LogClient.cc: 310: FAILED assert(num_unsent <= log_queue.size())

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17202 from jdurgin/wip-lumionus-pg-log

osd: adjust osd_min_pg_log_entries

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

osd: adjust osd_min_pg_log_entries

Return osd_min_pg_log_entries to its original values, which matches
osd_pg_log_dups_tracked, so the extended dup log is not used in the
general case.

This helps address: http://tracker.ceph.com/issues/21026

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 1c4df03394312fe67f36448613d8b54cb1a0e2c9)

Merge pull request #17192 from idryomov/wip-krbd-unmap-tests-pool-luminous

luminous: qa: fix POOL_APP_NOT_ENABLED warning in krbd:unmap suite

mon/PGMap: reweight::by_utilization - skip DNE osds

EC could set one or more members of acting set to CRUSH_ITEM_NONE,
which as a result can cause pgs_by_osd.resize() attempt to apply
for a large amount of memory which we can not afford.

Fix the above problem by always excluding a current DNE osd from
acting set.

Fixes: http://tracker.ceph.com/issues/20970
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit 6ae96d33a84404a144e06c423796e4a3886d8e61)

common/LogClient: make last_log non-atomic

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit c36a98be97e15294f57c6a640fe6a1c277dce8a3)

common/LogClient: fix indentation

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit bf92a8a2693e0b6a01a861977a5376eb584c9e42)

common/LogClient: assign seq and queue atomically

The _get_mon_log_message() assumes that log_last and log_queue
are in sync, but it was previously possible to increment log_last
setting e.seq in do_log(), and only later queue it.  If a racing
thread ran get_mon_log_message() in the meantime it would fail
an assertion.

Fix by assigning the seq and queueing it atomically.  If the
cluster log is not enabled, use the get_next_seq() helper so that
graylog or syslog messages still have a seq assigned.

Fixes: http://tracker.ceph.com/issues/18209
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 1f8f58becf1ec53c729b0befeb1f19c601588c4a)

osd: Fixes for osd_scrub_during_recovery handling

Fixes: http://tracker.ceph.com/issues/18206
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 367c32c69a512d2bea85a9b3860ec28bb4433750)

tests: osd-scrub-snaps.sh minor cleanup

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 9f3d970a0dbb754ef34ffafa1c3e8d0d3c5982c7)

osd: include front_iface+back_iface in metadata

Fixes: http://tracker.ceph.com/issues/20956
Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 79491c15473310b2ac315f8698b11bdde588b20d)

common: return iface instead of addr from ipaddr.cc helpers

So that we can use the same helper functions to look
up interface names that we use to look up addresses.

Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 687ea102697b2ec64b503f16c9aaeb46bee5da99)

mon/OSDMonitor: do not send_pg_creates with stale info

we reset the "creating_pgs" with the newly accepted paxos proposal, but
the creating_pgs_by_osd_epoch is out-of-sync with the new creating_pgs.
so we are at risk of using stale creating_pgs_by_osd_epoch along with
the new creating_pgs.pgs. to avoid this racing, we need to check the
creating_pgs_epoch before sending pg-creates using
creating_pgs_by_osd_epoch.

Fixes: http://tracker.ceph.com/issues/20785
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 689cf03f5515bb116ddf3966ef57550baf97eee9)

qa: fix POOL_APP_NOT_ENABLED warning in krbd:unmap suite

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit ad715b3368522a3913ccaa233abc9729f728e328)

Merge pull request #17117 from theanalyst/wip-21056-luminous

luminous: RGW: Get Bucket ACL does not honor the s3:GetBucketACL action

Reviewed-By: Casey Bodley <cbodley@redhat.com>

Merge pull request #17118 from theanalyst/wip-21054-luminous

luminous: rgw: GetObject Tagging needs to exit earlier if the object has no attributes

Reviewed-By: Casey Bodley <cbodley@redhat.com>

mon/Monitor: use legacy health checks in 'status' output before upgrade completion

This matches what the 'health' command does.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit ced39aeeef441fc588c5f2570710ab107b55828c)

qa/suites/upgrade/jewel-x/parallel: no loadgenbig

When we do the thrashing this leads to ENOSPC on smithi.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit a0b9f37dbc80e863b1c4df102bacf049d0aac3fa)

compressor: conditionalize on HAVE_LZ4

do not enable lz4 compressor plugin unless HAVE_LZ4 is ON. so, for
example, when the compressor type is "random", the compressor factory
won't complain with:

load failed dlopen(): "/usr/lib64/ceph/compressor/libceph_lz4.so: cannot
open shared object file: No such file or directory"

anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit b5ea349c0dfd31c5126e229d5455a2c111ce47ff)

cmake: require lz4 v1.7 for building lz4 compressor

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit b567261208d937d1937497d3e4f6eb97f7a8ed21)

mgr: add missing call to pick_addresses

Previously this was presumably simply not
applying the public_network setting at all.

Fixes: http://tracker.ceph.com/issues/20955
Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 33fe08dd39342ab7d6f6bc60d7cefbe5af1a0a3d)

mon: hex feature bits of 'ceph features'

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit b68f31aaca30a83f77ccf81191bf978a15dcbdbe)

mon: fix inaccurate mon-num counting of 'ceph features' command

Monitors need to talk to each other, which causes the inaccurate
mon-num counting logic of 'ceph features' command.
E.g.:
(1) By specifying 2 monitors, you get 4 from 'ceph features':
    "mon": {
        "group": {
            "features": 2305244844532236283,
            "release": "luminous",
            "num": 4
        }
    },
(2) By specifying 3 monitors, you get 9 from 'ceph features':
   "mon": {
        "group": {
            "features": 2305244844532236283,
            "release": "luminous",
            "num": 9
        }
    },

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit dda5480d433053feec245a02c1fa419d7a1f4cff)

Merge pull request #17170 from ceph/luminous-volume-man-fix

luminous: doc add ceph-volume and ceph-volume-systemd man pages to CMakeLists file

Reviewed-by: Dan Mick <dmick@redhat.com>

doc add ceph-volume and ceph-volume-systemd man pages to CMakeLists file

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #17158 from ceph/wip-rm21030-backport

luminous: create the ceph-volume and ceph-volume-systemd man pages

doc: include ceph-disk and ceph-disk-volume man pages in index

Signed-off-by: Alfredo Deza <adeza@redhat.com>

debian: include ceph-volume and ceph-volume-systemd man pages

Signed-off-by: Alfredo Deza <adeza@redhat.com>

rpm: include ceph-volume and ceph-volume-systemd man pages

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/man create the ceph-volume man page

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/man create the ceph-volume-systemd man page

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #17124 from ceph/backport-ceph-volume-docs

ceph-volume docs

Merge pull request #17123 from ceph/backport-bz1478599

ceph-volume: Use a delimited CLI output parser instead of JSON

doc ignore ceph-volume when building man pages

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc: add LVM tag term to the glossary

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/dev include ceph-volume devel docs

Signed-off-by: Alfredo Deza <adeza@redhat.com>

docs: include ceph-volume's index file in the main toctree

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume update the systemd docstring to use the new API

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/glossary add terminology used by ceph-volume

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume create the systemd lvm reference

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume create the intro to ceph-volume

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume create the main index

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume add the lvm scan reference

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume create the lvm prepare reference

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume create the lvm activate reference

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/install use the new mgr-administrator-guide label

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/mgr add a label for the administrator guide

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/dev/ceph-volume add a toctree for the dev docs

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/dev/ceph-volume add systemd reference

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/dev/ceph-volume add plugin interface

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/dev/ceph-volume add the internal lvm api

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: lvm add a parser for the delimeted output in the CLI

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.api uses the new parses for vgs

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.api uses the new parser for lvs

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests adapt tests for the new LVM api parser

Signed-off-by: Alfredo Deza <adeza@redhat.com>

debian: include ceph-volume-systemd CLI tool

Signed-off-by: Alfredo Deza <adeza@redhat.com>

rgw: rgw_op: exit early if object has no attrs in GetObjectTagging

Check for the return value of get_obj_attrs and exit earlier in case of
errors

Fixes: http://tracker.ceph.com/issues/21010
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit b2246226cc11fcd87b4fd499c34434f82ca18654)

rgw: evaluate the correct bucket action for GetACL bucket operation

We currently evaluate s3:GetObjectAcl instead of s3:GetBucketAcl for
getting BucketACL

Fixes: http://tracker.ceph.com/issues/21013
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit 0e6b4f7862aacb2745e11cd7dbd16bcbec079bab)

tests: run-standalone.sh skip core_pattern if already set

If sudo requires a password it is nice to be able to avoid
sudo sysctl by setting core_pattern ahead of time.

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 599d7622dd0bd7f73145cc229b576d5de7f04fda)

radosgw: fix small sized chunked uploads with s3

The logic to compute the number of bytes to copy with a chunked file
upload was failing to take into account the amount of data that had been
consumed, but was still pending in the parse buffer.  This would cause
strange behavior ranging from "just works" to "fails".  There was also a
strange "works, one byte at a time" mode.  Using the correct stream_pos
offset eliminates the bad behavior.

This fix also adds debug statements to make the correct behavior easier to see.

Fixes: http://tracker.ceph.com/issues/21003
Signed-off-by: Marcus Watts <mwatts@redhat.com>
(cherry picked from commit 96c4a26b46343f2850adae4f49e5b8be5104e0b5)

radosgw: fix zero sized chunked uploads with s3

s3 allows for zero sized objects, which can be
created using chunked uploads.  It is important to
call "recv_body (and ChunkMeta::create_next()) even
on zero sized objects, so that the proper chunk signature
can be presented at signature verification time.

Fixes: http://tracker.ceph.com/issues/21000
Signed-off-by: Marcus Watts <mwatts@redhat.com>
(cherry picked from commit 4d3d72fd1c5e0e3913ee9f9b7f8ce2bfe6d4b6fa)

rgw: replace '+' with "%20" in canonical query string for s3 v4 auth.

( note: this patch modified: now replaces "+" with " ".  mwatts@redhat.com )

Fixes: http://tracker.ceph.com/issues/20501
Signed-off-by: Zhang Shaowen <zhangshaowen@cmss.chinamobile.com>
(cherry picked from commit 8ef21a05e6b2a8563ea37345746ade4a9cd189d7)

Fix rgw not responding occasionally when receiving SIGHUP signal.

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>
(cherry picked from commit 18c97e568ef7df6d9ce7d68f55b353a1a9d498ab)

tests: run-standalone.sh improve error message

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 175b10795d12922e833b3a33591918d8033fb407)

tests: qa/standalone: Add trap for signals to resetore the kernel core pattern

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 982aef784c7f8157a4fc162b38413f7516b3f76d)

Merge pull request #17074 from yuriw/wip_add_luminous

Decreased amount of jobs on master, kraken, luminous runs

Merge pull request #17077 from dillaman/wip-qa-rbd-mirror-data-pool-luminous

qa/workunits/rbd: test data pool is mirrored correctly

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits/rbd: test data pool is mirrored correctly

Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit c4c7075029f20f238585145c464a8c972d9ad038)

Decreased amount of jobs on master, kraken, luminous runs

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #17023 from adamwg/awg/rbd-mirror-data-pool

rbd-mirror: Set the data pool correctly when creating images

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc: Update mirroring docs to mention data-pool configuration

Signed-off-by: Adam Wolfe Gordon <awg@digitalocean.com>

12.1.4

rbd-mirror: Set the data pool correctly when creating images

Previously the data-pool option was ignored by rbd-mirror, so when using
erasure coding data for the image on the secondary site would end up in
the metadata pool.

Configure the data-pool using the data-pool's name from the primary site
when creating images on the secondary site.

Fixes: http://tracker.ceph.com/issues/20567
Signed-off-by: Adam Wolfe Gordon <awg@digitalocean.com>

Merge pull request #17001 from gregsfortytwo/wip-20985-divergent-handling-luminous

Wip 20985 divergent handling luminous

mgr: implement 'osd safe-to-destroy' and 'ok-to-stop' commands

An osd is safe to destroy if

- we have osd_stat for it
- osd_stat indicates no pgs stored
- all pgs are known
- no pgs map to it

An osd is ok ot stop if

- we have pg stats
- no pgs will drop below min_size

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit bf9380457bba5a834ffa2927c73165e0f1960332)