Merge pull request #16829 from ivancich/wip-dup-ops-cleanup-1

osd: clean up dup index logic; maintain index flag logic in fewer places

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #16887 from xiexingguo/wip-mon-count

mon: fix wrong mon-num counting  logic of 'ceph features' command

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16892 from xiexingguo/wip-clean-pg-temp

mon/OSDMonitor: fix 'osd pg temp' unable to cleanup pg-temp

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16897 from amitkumar50/cov-MDentryLink-1

messages: Initialization of is_primary

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #16919 from ceph/wip-volume-tests

ceph-volume: adds functional CI testing

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: lvm activate should check if the device is mounted to prevent errors from mount

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util add a helper to check if a device is mounted

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #16834 from mdw-at-linuxbox/policy

radosgw: usage: fix bytes_sent bug.

Merge pull request #16828 from liewegas/wip-18209

common/LogClient: assign seq and queue atomically

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

Merge pull request #16920 from liewegas/wip-osd-crush-ls

mon/OSDMonitor: implement 'osd crush ls <node>'

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #16923 from liewegas/wip-20738

qa/suites/rados/objectstore: logs

Merge pull request #16737 from dillaman/wip-rbd-config

rbd: switched from legacy to new-style configuration options

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #16796 from rzarzynski/wip-rgw-20897

rgw: honor the tenant part of rgw_bucket during comparisons.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #16874 from rzarzynski/wip-rgw-drop-init_anon_user

rgw: drop the unused function init_anon_user()

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #16938 from javacruft/32bit-compat-fixes

tools: Align use of uint64_t in service_daemon::AttributeType

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #16840 from mychoxin/yuanxin_writing_mistake

os/ObjectStore.h: fix mistake in comment TRANSACTION ISOLATION

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #16947 from liewegas/wip-jewel-x

qa/suites/upgarde/jewel-x/point-to-point-x: disable app warnings

Merge pull request #16917 from trociny/wip-qa-admin-sock

qa/workunits/rbd: use command line option to specify watcher asok

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd: switched from legacy to new-style configuration options

Fixes: http://tracker.ceph.com/issues/20737
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/suites/upgarde/jewel-x/point-to-point-x: disable app warnings

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16889 from oritwas/wip-rgw-20177

rgw: Use namespace for lc_pool and roles_pool

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

ceph-volume: lvm activate should not ignore exit status codes

Signed-off-by: Alfredo Deza <adeza@redhat.com>

os/ObjectStore: correct mistake in writing

Signed-off-by: mychoxin <mychoxin@gmail.com>

Align use of uint64_t in service_daemon::AttributeType

size_t on a 32-bit architecture is a 32 bit unsigned int which
created ambiguity when casting to bool, uint64_t or std::string
(which are boost::variants for service_daemon::AttributeType).

Align to use of uint64_t to resolve compilation failures in
all 32-bit architectures.

Signed-off-by: James Page <james.page@ubuntu.com>

qa/workunits/rbd: use command line option to specify watcher asok

The previous method to get the watcher admin socket was fragile
and had started to fail after the recent changes to vstart ceph.conf.

Fixes: http://tracker.ceph.com/issues/20954
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #16927 from optimistyzy/89_ext

bluestore/NVMeDevice: update NVMeDevice code due to SPDK upgrade

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Pan Liu <liupan1111@gmail.com>

bluestore,NVMeDevice: update NVMeDevice code due to SPDK upgrade

Signed-off-by: Ziye Yang <optimistyzy@gmail.com>

Merge pull request #16900 from liewegas/wip-20925-b

os/bluestore: Revert "os/bluestore: allow multiple DeferredBatches in flight at once"

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #16902 from liewegas/wip-health-deprecation

mon: add mon_health_preluminous_compat_warning

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #16916 from jcsp/wip-scrub-logging

mon, osd: turn down non-error scrub message severity

Reviewed-by: Sage Weil <sage@redhat.com>

Test bytes_sent bugs.

Rearrange logic to make it easier to measure accumulation.
Instrument the boto request/response loop to count bytes in and out.
Accumulate byte counts in usage like structure.
Compare actual usage reported by ceph against local usage measured.
Report and assert if there are any short-comings.
Remove zone placement rule that was newly added at end: tests should be rerunable.

Nit: the logic to wait for "delete_obj" is not quite right.

Fixes: http://tracker.ceph.com/issues/19870
Signed-off-by: Marcus Watts <mwatts@redhat.com>

Fix bytes_sent bugs.

log bytes sent/received.
add cct to bufferingfilter
add cct to RGWRestfulIO
AccountingFilter - save cct for debugging output
implement AccountingFilter::complete_request() - account for bytes reported here.
BufferingFilter<T>::complete_request() - ignore counts from send_content_length() complete_header();

Code quality note:
this patch makes "cct" available for a lot of newly added debug
statements.  The debug statements are mostly not very useful (and should
go away in the future) - *But* the "cct" logic should be redone and
incorporated into some base class (such RestfulClient) so that it is
possible to easily add in debug statements such as these in the future.

Fixes: http://tracker.ceph.com/issues/19870
Signed-off-by: Marcus Watts <mwatts@redhat.com>

qa/suites/rados/objectstore: logs

Hunting http://tracker.ceph.com/issues/20738

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16755 from ivancich/wip-pull-new-dmclock

osd: bring in dmclock library changes

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #16546 from asomers/openstack_shebang2

qa: Fix shebangs on openstack scripts

Reviewed-by: Amik Kumar <amitkuma@redhat.com>

Merge pull request #15579 from ovh/bp-parallel-rbd-lsl

rbd: parallelize "rbd ls -l"

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

common/LogClient: make last_log non-atomic

Signed-off-by: Sage Weil <sage@redhat.com>

common/LogClient: fix indentation

Signed-off-by: Sage Weil <sage@redhat.com>

common/LogClient: assign seq and queue atomically

The _get_mon_log_message() assumes that log_last and log_queue
are in sync, but it was previously possible to increment log_last
setting e.seq in do_log(), and only later queue it.  If a racing
thread ran get_mon_log_message() in the meantime it would fail
an assertion.

Fix by assigning the seq and queueing it atomically.  If the
cluster log is not enabled, use the get_next_seq() helper so that
graylog or syslog messages still have a seq assigned.

Fixes: http://tracker.ceph.com/issues/18209
Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: implement 'osd crush ls <node>'

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16861 from wido/docs-mgr-zabbix-enable

doc: Update mgr doc on how to enable Zabbix module

Reviewed-By: John Spray <jcsp@redhat.com>

osd: downgrade (ok) PG scrub messages to debug

Otherwise someone watching the log at INFO level gets
pelted with potentially millions of log messages
while the system is scrubbing.

Fixes: http://tracker.ceph.com/issues/20947
Signed-off-by: John Spray <john.spray@redhat.com>

mon: downgrade "scrub ok" message to debug.

This hides lines like:
[INF]  scrub ok on 0,1,2: ScrubResult(keys {pgmap_pg=13} crc {pgmap_pg=2458062599})

from the normal cluster log views.

Fixes: http://tracker.ceph.com/issues/20947
Signed-off-by: John Spray <john.spray@redhat.com>

messages: Initialization of is_primary

Fixes the coverity issue:

** 717269 Uninitialized scalar field
CID 717269 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR)
2. uninit_member: Non-static class member is_primary is not initialized
in this constructor nor in any functions that it calls.

Signed-off-by: Amit Kumar amitkuma@redhat.com

Merge pull request #16908 from dillaman/wip-20701

doc: update rbd-mirroring documentation

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16910 from liewegas/wip-iana

doc/dev/iana: document our official IANA numbers

Merge pull request #16911 from liewegas/wip-lead

doc/dev/index: update rados lead

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

ceph-volume: remove unused config from vagrant_variables.yml files

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #16879 from smithfarm/wip-release-notes-11-2-1

doc: release-notes.rst: add Kraken v11.2.1 and update releases.rst

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #16894 from liupan1111/wip-remove-friendly

bluestore: drop unused friend class in SharedDriverQueueData

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #16907 from theanalyst/doc/rgw/s3-api-tagging

doc: rgw mention about tagging & bucket policies in s3api

Reviewed-By: Adam Emerson <aemerson@redhat.com>
Reviewed-By: Matt Benjamin <mbenjami@redhat.com>

doc/dev/index: update rados lead

Signed-off-by: Sage Weil <sage@redhat.com>

doc/dev/iana: document our official IANA numbers

PEN 50495
port 3300

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-volume: adds CEPH_VOLUME_DEBUG=1 to functional tests

This will show us tracebacks if ceph-volume fails

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: add placeholders for prepare_activate testing in tox.ini

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

doc/release-notes: indicate that rbd-mirror should use unique IDs

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #16782 from wjwithagen/wip-wjw-FreeBSD-Release

PendingReleaseNotes:Announce FreeBSD availability

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-By: Jason Dillaman <jdillaman@redhat.com>

ceph-volume: adds the xenial distro factor

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #16734 from theanalyst/rgw-lc-awsv4

rgw_lc: support for AWSv4 authentication

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc: updated rbd-mirror daemon instructions

Fixes: http://tracker.ceph.com/issues/20701
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #16821 from theanalyst/rgw-lc-log-drop-oldlc

rgw_op: Drop the Old LifecycleConfiguration from logs

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

ceph-volume: create a centos7 factor for functional testing

We want to run these tests on multiple distros so this change sets the
foundation for that.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

doc: re-ordered rbd table of contents

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #16854 from yaozongyou/fix-rgw-not-responding

rgw: fix not responding when receiving SIGHUP signal.

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

doc: rgw mention about tagging & bucket policies in s3api

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge PR #16378 into master

* refs/remotes/upstream/pull/16378/head:
doc: remove accidental additions to release notes
qa/cephfs: Fix race in test_volume_client
qa/cephfs: Test filtered df
PendingReleaseNotes: add note about df filtering
client: Support new, filtered MStatfs
objecter: Support new, filtered MStatfs
mon/PGMap stats: Support new, filtered MStatfs
messages: Add optional data pool to MStatfs

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

doc: remove accidental additions to release notes

Presumably this was caused by a bad rebase.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

rgw: honor the tenant part of rgw_bucket during comparisons.

Fixes: http://tracker.ceph.com/issues/20897
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

ceph-volume: setup nodes for testinfra testing

This uses the playbook that exists in ceph-ansible to prepare the nodes for
testing by installing net-tools.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: vagrantfile runs storagectl once

It assumes that if there is a disk left it has already run. This avoids
issues when reloading/restarting machines with vagrant.

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: tox define vagrant cwd

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: adds a functional testing scenario for lvm create

This setups up the basic test harness and adds a test for the create
subcommand. The test uses ceph-ansible to deploy a cluster using
``ceph-volume lvm create``, tests the cluster state using the
ceph-ansible test suite, reboots the nodes and then tests again.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #16877 from dillaman/wip-20918

test/librbd: fix race condition with OSD map refresh

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

mon: add mon_health_preluminous_compat_warning

Signed-off-by: Sage Weil <sage@redhat.com>

PendingReleaseNotes:Announce FreeBSD availability

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Revert "os/bluestore: allow multiple DeferredBatches in flight at once"

This reverts commit ca32d575eb2673737198a63643d5d1923151eba3.

If we have multiple batches in flight then we have to worry about writes
to the same blocks reordering.

Also, 3c6a6c46d5808d6c42ed4dcfb441bad64366686b is sufficient to avoid the
stall/deadlock in http://tracker.ceph.com/issues/20295.

# Conflicts:
# src/os/bluestore/BlueStore.cc

Fixes: http://tracker.ceph.com/issues/20925
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16882 from dillaman/wip-20941

librbd: default localize parent reads to false

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #15339 from trociny/wip-rbd-ggate

rbd-ggate: tool to map images on FreeBSD via GEOM Gate

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

bluestore/NVMEDevice: remove unused friendly class declaration.

Signed-off-by: Pan Liu <wanjun.lp@alibaba-inc.com>

Merge pull request #16846 from amitkumar50/cov-messages-4

messages: Initialize member variables

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #16793 from amitkumar50/cov-717286

messages: initialize variable tid in MMDSFindIno

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #16890 from Songweibin/wip-librados-Wreorder

libradosstriper: silence warning from -Wreorder

Reviewed-by: Jos Collin <jcollin@redhat.com>

mon/OSDMonitor: fix 'osd pg temp' unable to cleanup pg-temp

This is not very elegant way but should work..

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

rgw:  use namespace for roles pool

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

test: add wrapper to run rbd-ggate test on FreeBSD

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

rbd-ggate: tool to map images on FreeBSD via GEOM Gate

rbd-ggate spawns a process responsible for the creation of ggate
device and forwarding I/O requests between the GEOM Gate kernel
subsystem and RADOS.

On FreeBSD it provides functionality similar to rbd-nbd on Linux.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

libradosstriper: silence warning from -Wreorder

Signed-off-by: songweibin <song.weibin@zte.com.cn>

rgw: initialize lc pool as namespace

Signed-off-by: Orit Wasserman <owasserm@redhat.com>
Fixes: http://tracker.ceph.com/issues/20177

mon: hex feature bits of 'ceph features'

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon: fix inaccurate mon-num counting of 'ceph features' command

Monitors need to talk to each other, which causes the inaccurate
mon-num counting logic of 'ceph features' command.
E.g.:
(1) By specifying 2 monitors, you get 4 from 'ceph features':
    "mon": {
        "group": {
            "features": 2305244844532236283,
            "release": "luminous",
            "num": 4
        }
    },
(2) By specifying 3 monitors, you get 9 from 'ceph features':
   "mon": {
        "group": {
            "features": 2305244844532236283,
            "release": "luminous",
            "num": 9
        }
    },

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #16881 from liewegas/wip-kraken-x

qa/suites/upgrade/kraken-x: fixes

qa/suites/upgrade/kraken-x/stress-split*: whitelist

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/upgrade/kraken-x/parallel: whitelist

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16823 from adamemerson/wip-use-after-free

rgw: Fix use after free in IAM policy parser

Merge pull request #16866 from amitkumar50/cov-librbd

librbd: initialization of state member variables

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #16867 from amitkumar50/cov-librbd-1

librbd: Initializing member variables

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rgw: Fix the last policy use-after-free

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Fix another use after free

This one was caused by iterator invalidation in set operations. In
this case just replace the set entirely with a bitfield.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Fix use after free in IAM policy parser

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

librbd: default localize parent reads to false

Fixes: http://tracker.ceph.com/issues/20941
Signed-off-by: Jason Dillaman <dillaman@redhat.com>