Merge pull request #16850 from liewegas/wip-bluestore-dirty-range

os/bluestore: fix dirty_shard off-by-one

Reviewed-by: Liu-Chunmei <chunmei.liu@intel.com>

Merge pull request #16872 from markhpc/wip-bluestore-faster_gncb

os/bluestore:  clearer comments, not slower code.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16941 from jcsp/wip-20956

osd: include front_iface+back_iface in metadata

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16944 from liewegas/wip-kraken-x

mon/Elector: force election epoch bump on start

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #16953 from neha-ojha/wip-20798

test/librados: create unique lock names

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16962 from ceph/wip-bz1479797

ceph-volume: tests add tests for the is_mounted utility

Reviewed-by: Andrew Schoen aschoen@redhat.com

Merge pull request #16618 from adamemerson/wip-throttle

common/Throttle: Clean up

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16702 from hjwsm1989/wip-recover-timeout-expired

 osd: do not forget pg_stat acks which failed to send

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

Merge pull request #16928 from amitkumar50/cov-messages-7

messages: Initializing members in MOSDPGUpdateLogMissing

Reviewed-by: Jos Collin <jcollin@redhat.com>

doc/release-notes: fix link typo

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16966 from idryomov/wip-krbd-unmap-tests-pool

qa: fix POOL_APP_NOT_ENABLED warning in krbd:unmap suite

Reviewed-by: Sage Weil <sage@redhat.com>

Revert "qa/suites/upgrade/jewel-x/parallel: thrash layout"

This reverts commit 435777dbffc77c93d06476caf83be141359a5778.

This test combination is not yet stable.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16950 from liewegas/wip-20959

osd/PG: force rebuild of missing set on jewel upgrade

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

ceph-volume: is_mounted should use a bytes->string util to compare strings

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: create a utf-8 string decoder for py3 compat

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #16945 from liupan1111/wip-fix-spdk-compile

bluestore: fixed compilation error when enable spdk with gcc 4.8.5

Reviewed-by: Haomai Wang <haomai@xsky.com>

ceph-volume: tests add tests for the is_mounted utility

Signed-off-by: Alfredo Deza <adeza@redhat.com>

qa: fix POOL_APP_NOT_ENABLED warning in krbd:unmap suite

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #16956 from theanalyst/doc-changelog-v11.2.1

doc: add changelog for v11.2.1

Reviewed-By: Nathan Cutler <ncutler@suse.com>

doc: add changelog for v11.2.1

Generated as::

git log v11.2.0..v11.2.1 --after="Thu Jan 19 13:08:40 2017 +0000" > doc/changelog/v11.2.1.txt

In order to avoid qe suite merges before this date

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #16893 from yuyuyu101/wip-fix-multicct-ib

msg/async/rdma: fix multi cephcontext confllicting

Merge pull request #16905 from liewegas/wip-upgrade-sloppy-flush

qa/tasks/ceph.py: tolerate flush pg stats exception

qa/suites/upgrade/jewel-x/parallel: thrash layout

We can't kill and restart osds because that will interfere with
the upgrade process.  We can, however, thrash the layout by
tweaking osd weights and so on.  This will exercise osd recovery
paths during the upgrade that aren't normally exercised (outside
of stress-split..which doesn't upgrade individual osds while they
are non-clean).

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: force rebuild of missing set on jewel upgrade

Previously we were detecting the need to rebuild missing based on
whether the "divergent_priors" omap key was present.  Unfortunately,
jewel does not always set this, so it is not a reliable indicator.
(It only gets set if you actually have a divergent prior at some
point in the PG's life time on that OSD.)

Fix by using the info_struct_v on the PG to detect whether we need
to do the conversion.  We didn't bump the value when we adding
the missing persistence, but the fastinfo was also added during
the same period between jewel and kraken, so it will work just as
well.

Fixes: http://tracker.ceph.com/issues/20958
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16006 from shangfufei/wip_modify_connect_logic

Optimization for connection establishment.

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #16382 from PCzhangPC/bug2

common: bench test fall into dead loop when <seconds>=0

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #16396 from amitkumar50/branch20604

core: remove startsync

Reviewed-by: Sage Weil <sweil@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #16886 from liupan1111/wip-add-size

os/bluestore: move size and block_size to the base class BlockDevice

Reviewed-by: Sage Weil <sweil@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Liu-Chunmei <chunmei.liu@intel.com>

Merge pull request #16888 from gmayyyha/osd-ms-verify-auth

osd/OSD.cc: add 'isvalid=false' when failed to parse caps

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #16829 from ivancich/wip-dup-ops-cleanup-1

osd: clean up dup index logic; maintain index flag logic in fewer places

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #16887 from xiexingguo/wip-mon-count

mon: fix wrong mon-num counting  logic of 'ceph features' command

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16892 from xiexingguo/wip-clean-pg-temp

mon/OSDMonitor: fix 'osd pg temp' unable to cleanup pg-temp

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16897 from amitkumar50/cov-MDentryLink-1

messages: Initialization of is_primary

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #16919 from ceph/wip-volume-tests

ceph-volume: adds functional CI testing

Reviewed-by: Alfredo Deza <adeza@redhat.com>

test/librados: create unique lock names

Signed-off-by: Neha Ojha <nojha@redhat.com>

ceph-volume: lvm activate should check if the device is mounted to prevent errors from mount

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util add a helper to check if a device is mounted

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #16834 from mdw-at-linuxbox/policy

radosgw: usage: fix bytes_sent bug.

mon/Elector: force election epoch bump on start

We are generally careful when bumping the epoch so that we can join
existing rounds.  However, if we restart in the middle of an election,
and change versions, we need to be certain that our previous ACK (as
$version - 1) isn't accepted as truth for the restarted daemon (running
$version) keeping the same epoch.

The conservatism with bumping is to avoid spurious election cycles, but
mon restarts are more rare, and we need them here.

Fixes: http://tracker.ceph.com/issues/20949
Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/upgrade/kraken-x/stress-split: more whitelisting

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16828 from liewegas/wip-18209

common/LogClient: assign seq and queue atomically

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

Merge pull request #16920 from liewegas/wip-osd-crush-ls

mon/OSDMonitor: implement 'osd crush ls <node>'

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #16923 from liewegas/wip-20738

qa/suites/rados/objectstore: logs

Merge pull request #16737 from dillaman/wip-rbd-config

rbd: switched from legacy to new-style configuration options

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #16796 from rzarzynski/wip-rgw-20897

rgw: honor the tenant part of rgw_bucket during comparisons.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #16874 from rzarzynski/wip-rgw-drop-init_anon_user

rgw: drop the unused function init_anon_user()

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #16938 from javacruft/32bit-compat-fixes

tools: Align use of uint64_t in service_daemon::AttributeType

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #16840 from mychoxin/yuanxin_writing_mistake

os/ObjectStore.h: fix mistake in comment TRANSACTION ISOLATION

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #16947 from liewegas/wip-jewel-x

qa/suites/upgarde/jewel-x/point-to-point-x: disable app warnings

Merge pull request #16917 from trociny/wip-qa-admin-sock

qa/workunits/rbd: use command line option to specify watcher asok

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd: switched from legacy to new-style configuration options

Fixes: http://tracker.ceph.com/issues/20737
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/suites/upgarde/jewel-x/point-to-point-x: disable app warnings

Signed-off-by: Sage Weil <sage@redhat.com>

bluestore: fixed compilation error when enable spdk with gcc 4.8.5

Signed-off-by: Pan Liu <wanjun.lp@alibaba-inc.com>
Signed-off-by: Ziye Yang <optimistyzy@gmail.com>

Merge pull request #16889 from oritwas/wip-rgw-20177

rgw: Use namespace for lc_pool and roles_pool

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

ceph-volume: lvm activate should not ignore exit status codes

Signed-off-by: Alfredo Deza <adeza@redhat.com>

osd: include front_iface+back_iface in metadata

Fixes: http://tracker.ceph.com/issues/20956
Signed-off-by: John Spray <john.spray@redhat.com>

common: return iface instead of addr from ipaddr.cc helpers

So that we can use the same helper functions to look
up interface names that we use to look up addresses.

Signed-off-by: John Spray <john.spray@redhat.com>

os/ObjectStore: correct mistake in writing

Signed-off-by: mychoxin <mychoxin@gmail.com>

messages: Initializing members in MOSDPGUpdateLogMissing

Fixes the coverity issues:

** 1355242 Uninitialized scalar field
2. uninit_member: Non-static class member map_epoch is not initialized
in this constructor nor in any functions that it calls.
4. uninit_member: Non-static class member min_epoch is not initialized
in this constructor nor in any functions that it calls.
CID 1355242 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR)
6. uninit_member: Non-static class member rep_tid is not initialized
in this constructor nor in any functions that it calls.

** 1355243 Uninitialized scalar field
2. uninit_member: Non-static class member map_epoch is not initialized
in this constructor nor in any functions that it calls.
4. uninit_member: Non-static class member min_epoch is not initialized
in this constructor nor in any functions that it calls.
CID 1355243 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR)
6. uninit_member: Non-static class member rep_tid is not initialized
in this constructor nor in any functions that it calls.

Signed-off-by: Amit Kumar amitkuma@redhat.com

Align use of uint64_t in service_daemon::AttributeType

size_t on a 32-bit architecture is a 32 bit unsigned int which
created ambiguity when casting to bool, uint64_t or std::string
(which are boost::variants for service_daemon::AttributeType).

Align to use of uint64_t to resolve compilation failures in
all 32-bit architectures.

Signed-off-by: James Page <james.page@ubuntu.com>

qa/workunits/rbd: use command line option to specify watcher asok

The previous method to get the watcher admin socket was fragile
and had started to fail after the recent changes to vstart ceph.conf.

Fixes: http://tracker.ceph.com/issues/20954
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #16927 from optimistyzy/89_ext

bluestore/NVMeDevice: update NVMeDevice code due to SPDK upgrade

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Pan Liu <liupan1111@gmail.com>

osd/OSD: add 'isvalid=false' when failed to parse caps

Signed-off-by: Yanhu Cao <gmayyyha@gmail.com>

bluestore,NVMeDevice: update NVMeDevice code due to SPDK upgrade

Signed-off-by: Ziye Yang <optimistyzy@gmail.com>

throttle: Do not destroy condition variables with waiters

Destroying a condition variable on which someone is waiting is Undefined
Behavior. it's bad and terrible and awful. On some machines it makes
the destructor just outright hang.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common: Automatic reference to PerfCounters

PerfCountersRef is a unique_ptr using a deleter that unregisters the
managed PerfCounters on deletion.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

scope_guard: Support in-place construction and one-shots

Kefu Chai just pointed out this exists so I just rolled the in-place variants of the
'ward' function I'd written into it.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common: Add in_place_t/in_place_type_t from C++17

To better signal in-place construction among other overloads.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

{os,mds,common}: Disambiguate std::max and ceph::max

There will be a more thorough reckoning regarding namespaces later.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common: Add locking template functions and macros

These infer the type of the unique/shared/shunique lock from the type of
the mutex.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #16900 from liewegas/wip-20925-b

os/bluestore: Revert "os/bluestore: allow multiple DeferredBatches in flight at once"

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #16902 from liewegas/wip-health-deprecation

mon: add mon_health_preluminous_compat_warning

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #16916 from jcsp/wip-scrub-logging

mon, osd: turn down non-error scrub message severity

Reviewed-by: Sage Weil <sage@redhat.com>

Test bytes_sent bugs.

Rearrange logic to make it easier to measure accumulation.
Instrument the boto request/response loop to count bytes in and out.
Accumulate byte counts in usage like structure.
Compare actual usage reported by ceph against local usage measured.
Report and assert if there are any short-comings.
Remove zone placement rule that was newly added at end: tests should be rerunable.

Nit: the logic to wait for "delete_obj" is not quite right.

Fixes: http://tracker.ceph.com/issues/19870
Signed-off-by: Marcus Watts <mwatts@redhat.com>

Fix bytes_sent bugs.

log bytes sent/received.
add cct to bufferingfilter
add cct to RGWRestfulIO
AccountingFilter - save cct for debugging output
implement AccountingFilter::complete_request() - account for bytes reported here.
BufferingFilter<T>::complete_request() - ignore counts from send_content_length() complete_header();

Code quality note:
this patch makes "cct" available for a lot of newly added debug
statements.  The debug statements are mostly not very useful (and should
go away in the future) - *But* the "cct" logic should be redone and
incorporated into some base class (such RestfulClient) so that it is
possible to easily add in debug statements such as these in the future.

Fixes: http://tracker.ceph.com/issues/19870
Signed-off-by: Marcus Watts <mwatts@redhat.com>

qa/suites/rados/objectstore: logs

Hunting http://tracker.ceph.com/issues/20738

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16755 from ivancich/wip-pull-new-dmclock

osd: bring in dmclock library changes

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #16546 from asomers/openstack_shebang2

qa: Fix shebangs on openstack scripts

Reviewed-by: Amik Kumar <amitkuma@redhat.com>

Merge pull request #15579 from ovh/bp-parallel-rbd-lsl

rbd: parallelize "rbd ls -l"

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

common/LogClient: make last_log non-atomic

Signed-off-by: Sage Weil <sage@redhat.com>

common/LogClient: fix indentation

Signed-off-by: Sage Weil <sage@redhat.com>

common/LogClient: assign seq and queue atomically

The _get_mon_log_message() assumes that log_last and log_queue
are in sync, but it was previously possible to increment log_last
setting e.seq in do_log(), and only later queue it.  If a racing
thread ran get_mon_log_message() in the meantime it would fail
an assertion.

Fix by assigning the seq and queueing it atomically.  If the
cluster log is not enabled, use the get_next_seq() helper so that
graylog or syslog messages still have a seq assigned.

Fixes: http://tracker.ceph.com/issues/18209
Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: implement 'osd crush ls <node>'

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16861 from wido/docs-mgr-zabbix-enable

doc: Update mgr doc on how to enable Zabbix module

Reviewed-By: John Spray <jcsp@redhat.com>

osd: downgrade (ok) PG scrub messages to debug

Otherwise someone watching the log at INFO level gets
pelted with potentially millions of log messages
while the system is scrubbing.

Fixes: http://tracker.ceph.com/issues/20947
Signed-off-by: John Spray <john.spray@redhat.com>

mon: downgrade "scrub ok" message to debug.

This hides lines like:
[INF]  scrub ok on 0,1,2: ScrubResult(keys {pgmap_pg=13} crc {pgmap_pg=2458062599})

from the normal cluster log views.

Fixes: http://tracker.ceph.com/issues/20947
Signed-off-by: John Spray <john.spray@redhat.com>

messages: Initialization of is_primary

Fixes the coverity issue:

** 717269 Uninitialized scalar field
CID 717269 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR)
2. uninit_member: Non-static class member is_primary is not initialized
in this constructor nor in any functions that it calls.

Signed-off-by: Amit Kumar amitkuma@redhat.com

Merge pull request #16908 from dillaman/wip-20701

doc: update rbd-mirroring documentation

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16910 from liewegas/wip-iana

doc/dev/iana: document our official IANA numbers

Merge pull request #16911 from liewegas/wip-lead

doc/dev/index: update rados lead

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

ceph-volume: remove unused config from vagrant_variables.yml files

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #16879 from smithfarm/wip-release-notes-11-2-1

doc: release-notes.rst: add Kraken v11.2.1 and update releases.rst

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #16894 from liupan1111/wip-remove-friendly

bluestore: drop unused friend class in SharedDriverQueueData

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #16907 from theanalyst/doc/rgw/s3-api-tagging

doc: rgw mention about tagging & bucket policies in s3api

Reviewed-By: Adam Emerson <aemerson@redhat.com>
Reviewed-By: Matt Benjamin <mbenjami@redhat.com>

doc/dev/index: update rados lead

Signed-off-by: Sage Weil <sage@redhat.com>

doc/dev/iana: document our official IANA numbers

PEN 50495
port 3300

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-volume: adds CEPH_VOLUME_DEBUG=1 to functional tests

This will show us tracebacks if ceph-volume fails

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: add placeholders for prepare_activate testing in tox.ini

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

doc/release-notes: indicate that rbd-mirror should use unique IDs

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #16782 from wjwithagen/wip-wjw-FreeBSD-Release

PendingReleaseNotes:Announce FreeBSD availability

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-By: Jason Dillaman <jdillaman@redhat.com>