test: Fix and enable test_pidfile.sh

Increase delay so daemonized process can make progress and write/lock pidfile
Add --log_flush_on_exit to ensure log gets flushed for TEST_without_pidfile test

Fixes: http://tracker.ceph.com/issues/20770
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #16709 from dzafman/wip-standalone

qa/standalone: misc fixes

Merge pull request #16954 from gregsfortytwo/wip-20891-pool-metadata

mon: fix "fs new" pool metadata update, tests

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16981 from yuyuyu101/wip_rdma_mt_fix

msg/async/rdma: fixes crash for multi rados client within one process

Merge pull request #16664 from amitkumar50/coverity-1414508-3

rdma: Assign instead of compare

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #16965 from TsaiJin/wip-use-rdtsc-to-get-time

src/msg/async/rdma: decrease cpu usage by rdtsc instruction

Reviewed-by: Haomai Wang <haomai@xsky.com>

msg/async/rdma: add atomic and lock protection

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async/rdma: fixes crash in fio

fio creates multiple CephContext in a single process.
Crash(es) happen because rdma stack has a global resources that
are still used from one ceph context while have already been destroyed
by another context.

The commit removes global instances of RDMA dispatcher and infiniband
and makes them context (rdma stack) specific.

Signed-off-by: Adir Lev <adirl@mellanox.com>
Signed-off-by: Alex Mikheev <alexm@mellanox.com>

Merge pull request #16980 from ceph/revert-16893-wip-fix-multicct-ib

Revert "msg/async/rdma: fix multi cephcontext confllicting"

Revert "msg/async/rdma: fix multi cephcontext confllicting"

Merge pull request #16977 from liewegas/wip-no-pidfile

test/CMakeLists: disable test_pidfile.sh

Reviewed-by: Xie Xingguo <xie.xingguo@zte.com.cn>

test/CMakeLists: disable test_pidfile.sh

Too flaky, see http://tracker.ceph.com/issues/20975

Signed-off-by: Sage Weil <sage@redhat.com>

qa: Fix broken test_activate_osd() due to missing space

Signed-off-by: David Zafman <dzafman@redhat.com>

qa: Fix races with waiting for scrubs

The trigger_scrub sets the last_scrub_stamp backwards to
force a scheduled scrub.  In a small window this stamp could get propagated
to the mgr.  A test failure occurred because wait_for_scrub() was confused
by seeing a backward moving date.

The most critical change is having wait_for_scrub() make sure that the
date advances past the previous in value.

A test failed because the random backoff kept delayed triggered scrub, so
set osd_scrub_backoff throughout.

Signed-off-by: David Zafman <dzafman@redhat.com>

qa: Testing of ceph-helpers.sh, teardown on fail to dump logs, save cores

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Avoid confusion over legacy snaps when head_exists corrupt

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Fix Paxos shutdown handling for commit_finish race

Fixes: http://tracker.ceph.com/issues/20921
Signed-off-by: David Zafman <dzafman@redhat.com>

qa: Add support for core dumps

Save core dumps when running tests locally
Dump logs to output whenever cores seen

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #16918 from liewegas/wip-doc-bluestore-migration

doc/rados/operations/bluestore-migration: document bluestore migration process

Reviewed-by: Vasu Kulkarni <vasu@redhat.com>

doc/rados/operations/bluestore-migration: document bluestore migration process

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16975 from theanalyst/doc-12-1-3-release-notes

doc: 12.1.3 release notes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16974 from liewegas/wip-release-notes

doc/release-notes: remove mention of crush weight optimization

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

qa: test that "fs new" correctly set the application_metadata

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

doc: 12.1.3 release notes

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

doc/release-notes: remove mention of crush weight optimization

It's not there yet.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15330 from guihecheng/rgw_file-fsid-mount

rgw_file: introduce new fsid and rgw_mount

mdsmon: treat the osdmon correctly when doing plugged updates

Make sure it's writeable before invoking changes, and propose_pending()
on it when we're done.
Make the PaxosService::C_RetryMessage public so we can do this from FSCommands.

Maybe-
Fixes: http://tracker.ceph.com/issues/20959
Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #16648 from ovh/bp-kill-str-conv

osdc, class_api: kill implicit string conversions

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

Merge pull request #16850 from liewegas/wip-bluestore-dirty-range

os/bluestore: fix dirty_shard off-by-one

Reviewed-by: Liu-Chunmei <chunmei.liu@intel.com>

Merge pull request #16872 from markhpc/wip-bluestore-faster_gncb

os/bluestore:  clearer comments, not slower code.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16941 from jcsp/wip-20956

osd: include front_iface+back_iface in metadata

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16944 from liewegas/wip-kraken-x

mon/Elector: force election epoch bump on start

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #16953 from neha-ojha/wip-20798

test/librados: create unique lock names

Reviewed-by: Sage Weil <sage@redhat.com>

qa: For FreeBSD skip osd-dup.sh because there is no bluestore

Signed-off-by: David Zafman <dzafman@redhat.com>

qa: Raise mon-data-avail-warn to pass tests with less space

Signed-off-by: David Zafman <dzafman@redhat.com>

qa: Add common generalized inject_eio() to ceph-helpers.sh

Retry for a while to allow pool to appear

Signed-off-by: David Zafman <dzafman@redhat.com>

qa: osd-scrub-repair.sh handle older versions of jq

Signed-off-by: David Zafman <dzafman@redhat.com>

qa: Add support for specifying sub-tests with run-standalone.sh

Fix test-ceph-helpers.sh to pass additional arguments on

Signed-off-by: David Zafman <dzafman@redhat.com>

qa: ceph-helpers.sh fixes

Add missing teardown to cleanup test directory
Fix pgid due to elimination of initial default pool
Testing could never fail because run_tests return ignored

Signed-off-by: David Zafman <dzafman@redhat.com>

qa: Skip ceph-helpers.sh which is tested through test-ceph-helpers.sh

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #16962 from ceph/wip-bz1479797

ceph-volume: tests add tests for the is_mounted utility

Reviewed-by: Andrew Schoen aschoen@redhat.com

Merge pull request #16618 from adamemerson/wip-throttle

common/Throttle: Clean up

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16702 from hjwsm1989/wip-recover-timeout-expired

 osd: do not forget pg_stat acks which failed to send

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

Merge pull request #16928 from amitkumar50/cov-messages-7

messages: Initializing members in MOSDPGUpdateLogMissing

Reviewed-by: Jos Collin <jcollin@redhat.com>

doc/release-notes: fix link typo

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16966 from idryomov/wip-krbd-unmap-tests-pool

qa: fix POOL_APP_NOT_ENABLED warning in krbd:unmap suite

Reviewed-by: Sage Weil <sage@redhat.com>

Revert "qa/suites/upgrade/jewel-x/parallel: thrash layout"

This reverts commit 435777dbffc77c93d06476caf83be141359a5778.

This test combination is not yet stable.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16950 from liewegas/wip-20959

osd/PG: force rebuild of missing set on jewel upgrade

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

ceph-volume: is_mounted should use a bytes->string util to compare strings

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: create a utf-8 string decoder for py3 compat

Signed-off-by: Alfredo Deza <adeza@redhat.com>

msg: get the time increment by rdstc inscturtion to decrease the cpu usage

     in RDMADispatcher polling thread

Signed-off-by: Jin Cai <caijin.caij@alibaba-inc.com>

Merge pull request #16945 from liupan1111/wip-fix-spdk-compile

bluestore: fixed compilation error when enable spdk with gcc 4.8.5

Reviewed-by: Haomai Wang <haomai@xsky.com>

ceph-volume: tests add tests for the is_mounted utility

Signed-off-by: Alfredo Deza <adeza@redhat.com>

qa: fix POOL_APP_NOT_ENABLED warning in krbd:unmap suite

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #16956 from theanalyst/doc-changelog-v11.2.1

doc: add changelog for v11.2.1

Reviewed-By: Nathan Cutler <ncutler@suse.com>

doc: add changelog for v11.2.1

Generated as::

git log v11.2.0..v11.2.1 --after="Thu Jan 19 13:08:40 2017 +0000" > doc/changelog/v11.2.1.txt

In order to avoid qe suite merges before this date

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #16893 from yuyuyu101/wip-fix-multicct-ib

msg/async/rdma: fix multi cephcontext confllicting

Merge pull request #16905 from liewegas/wip-upgrade-sloppy-flush

qa/tasks/ceph.py: tolerate flush pg stats exception

qa/suites/upgrade/jewel-x/parallel: thrash layout

We can't kill and restart osds because that will interfere with
the upgrade process.  We can, however, thrash the layout by
tweaking osd weights and so on.  This will exercise osd recovery
paths during the upgrade that aren't normally exercised (outside
of stress-split..which doesn't upgrade individual osds while they
are non-clean).

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: force rebuild of missing set on jewel upgrade

Previously we were detecting the need to rebuild missing based on
whether the "divergent_priors" omap key was present.  Unfortunately,
jewel does not always set this, so it is not a reliable indicator.
(It only gets set if you actually have a divergent prior at some
point in the PG's life time on that OSD.)

Fix by using the info_struct_v on the PG to detect whether we need
to do the conversion.  We didn't bump the value when we adding
the missing persistence, but the fastinfo was also added during
the same period between jewel and kraken, so it will work just as
well.

Fixes: http://tracker.ceph.com/issues/20958
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16006 from shangfufei/wip_modify_connect_logic

Optimization for connection establishment.

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #16382 from PCzhangPC/bug2

common: bench test fall into dead loop when <seconds>=0

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #16396 from amitkumar50/branch20604

core: remove startsync

Reviewed-by: Sage Weil <sweil@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #16886 from liupan1111/wip-add-size

os/bluestore: move size and block_size to the base class BlockDevice

Reviewed-by: Sage Weil <sweil@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Liu-Chunmei <chunmei.liu@intel.com>

Merge pull request #16888 from gmayyyha/osd-ms-verify-auth

osd/OSD.cc: add 'isvalid=false' when failed to parse caps

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #16829 from ivancich/wip-dup-ops-cleanup-1

osd: clean up dup index logic; maintain index flag logic in fewer places

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #16887 from xiexingguo/wip-mon-count

mon: fix wrong mon-num counting  logic of 'ceph features' command

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16892 from xiexingguo/wip-clean-pg-temp

mon/OSDMonitor: fix 'osd pg temp' unable to cleanup pg-temp

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16897 from amitkumar50/cov-MDentryLink-1

messages: Initialization of is_primary

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #16919 from ceph/wip-volume-tests

ceph-volume: adds functional CI testing

Reviewed-by: Alfredo Deza <adeza@redhat.com>

mdsmon: don't add pool application metadata until running fully-luminous

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

test/librados: create unique lock names

Signed-off-by: Neha Ojha <nojha@redhat.com>

ceph-volume: lvm activate should check if the device is mounted to prevent errors from mount

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util add a helper to check if a device is mounted

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #16834 from mdw-at-linuxbox/policy

radosgw: usage: fix bytes_sent bug.

mon/Elector: force election epoch bump on start

We are generally careful when bumping the epoch so that we can join
existing rounds.  However, if we restart in the middle of an election,
and change versions, we need to be certain that our previous ACK (as
$version - 1) isn't accepted as truth for the restarted daemon (running
$version) keeping the same epoch.

The conservatism with bumping is to avoid spurious election cycles, but
mon restarts are more rare, and we need them here.

Fixes: http://tracker.ceph.com/issues/20949
Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/upgrade/kraken-x/stress-split: more whitelisting

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16828 from liewegas/wip-18209

common/LogClient: assign seq and queue atomically

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

Merge pull request #16920 from liewegas/wip-osd-crush-ls

mon/OSDMonitor: implement 'osd crush ls <node>'

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #16923 from liewegas/wip-20738

qa/suites/rados/objectstore: logs

Merge pull request #16737 from dillaman/wip-rbd-config

rbd: switched from legacy to new-style configuration options

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #16796 from rzarzynski/wip-rgw-20897

rgw: honor the tenant part of rgw_bucket during comparisons.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #16874 from rzarzynski/wip-rgw-drop-init_anon_user

rgw: drop the unused function init_anon_user()

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #16938 from javacruft/32bit-compat-fixes

tools: Align use of uint64_t in service_daemon::AttributeType

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #16840 from mychoxin/yuanxin_writing_mistake

os/ObjectStore.h: fix mistake in comment TRANSACTION ISOLATION

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #16947 from liewegas/wip-jewel-x

qa/suites/upgarde/jewel-x/point-to-point-x: disable app warnings

Merge pull request #16917 from trociny/wip-qa-admin-sock

qa/workunits/rbd: use command line option to specify watcher asok

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd: switched from legacy to new-style configuration options

Fixes: http://tracker.ceph.com/issues/20737
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/suites/upgarde/jewel-x/point-to-point-x: disable app warnings

Signed-off-by: Sage Weil <sage@redhat.com>

bluestore: fixed compilation error when enable spdk with gcc 4.8.5

Signed-off-by: Pan Liu <wanjun.lp@alibaba-inc.com>
Signed-off-by: Ziye Yang <optimistyzy@gmail.com>

Merge pull request #16889 from oritwas/wip-rgw-20177

rgw: Use namespace for lc_pool and roles_pool

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

ceph-volume: lvm activate should not ignore exit status codes

Signed-off-by: Alfredo Deza <adeza@redhat.com>

osd: include front_iface+back_iface in metadata

Fixes: http://tracker.ceph.com/issues/20956
Signed-off-by: John Spray <john.spray@redhat.com>

common: return iface instead of addr from ipaddr.cc helpers

So that we can use the same helper functions to look
up interface names that we use to look up addresses.

Signed-off-by: John Spray <john.spray@redhat.com>

os/ObjectStore: correct mistake in writing

Signed-off-by: mychoxin <mychoxin@gmail.com>

messages: Initializing members in MOSDPGUpdateLogMissing

Fixes the coverity issues:

** 1355242 Uninitialized scalar field
2. uninit_member: Non-static class member map_epoch is not initialized
in this constructor nor in any functions that it calls.
4. uninit_member: Non-static class member min_epoch is not initialized
in this constructor nor in any functions that it calls.
CID 1355242 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR)
6. uninit_member: Non-static class member rep_tid is not initialized
in this constructor nor in any functions that it calls.

** 1355243 Uninitialized scalar field
2. uninit_member: Non-static class member map_epoch is not initialized
in this constructor nor in any functions that it calls.
4. uninit_member: Non-static class member min_epoch is not initialized
in this constructor nor in any functions that it calls.
CID 1355243 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR)
6. uninit_member: Non-static class member rep_tid is not initialized
in this constructor nor in any functions that it calls.

Signed-off-by: Amit Kumar amitkuma@redhat.com

Align use of uint64_t in service_daemon::AttributeType

size_t on a 32-bit architecture is a 32 bit unsigned int which
created ambiguity when casting to bool, uint64_t or std::string
(which are boost::variants for service_daemon::AttributeType).

Align to use of uint64_t to resolve compilation failures in
all 32-bit architectures.

Signed-off-by: James Page <james.page@ubuntu.com>

qa/workunits/rbd: use command line option to specify watcher asok

The previous method to get the watcher admin socket was fragile
and had started to fail after the recent changes to vstart ceph.conf.

Fixes: http://tracker.ceph.com/issues/20954
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #16927 from optimistyzy/89_ext

bluestore/NVMeDevice: update NVMeDevice code due to SPDK upgrade

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Pan Liu <liupan1111@gmail.com>

osd/OSD: add 'isvalid=false' when failed to parse caps

Signed-off-by: Yanhu Cao <gmayyyha@gmail.com>