mgr: fix crashable DaemonStateIndex::get calls

This function was recently fixed to return null
on missing entries: handle that properly.

Fixes: http://tracker.ceph.com/issues/17737
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #17850 from batrick/vstart-confg-path

qa: revert "qa: use config_path property instead of literal"

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #17881 from joscollin/wip-mgr-signed-unsigned-warning

mgr: silence warning from -Wsign-compare

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17756 from jcsp/wip-19291

mgr: set explicit thread name

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #17808 from xiexingguo/wip-eqduot-vs-enospc

osd/PrimaryLogPG: prepare_transaction - fix EDQUOT vs ENOSPC

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17819 from neha-ojha/wip-cbt-teuthology-fix-pg-num

qa/suites/rados/perf: create pool with lower pg_num

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #17827 from tchaikov/wip-mono-heartbeat

common: use mono clock for HeartbeatMap

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17844 from liewegas/wip-21259

os/bluestore: debug bluestore cache shutdown

Merge pull request #17845 from liewegas/wip-21408

os/bluestore: ignore 0x2000~2000 extent oddity from luminous upgrade

Reviewed-by: Varada Kari <varada.kari@gmail.com>

Merge PR #17853 into master

* refs/remotes/upstream/pull/17853/head:
mds: check ongoing catter-gather process before capping log

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #17851 into master

* refs/remotes/upstream/pull/17851/head:
client: use unique_ptr for locks
client: use unique_ptr for Formatter

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge PR #17843 into master

* refs/remotes/upstream/pull/17843/head:
ceph_volume_client: perform snapshot operations in

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #17757 from ceph/wip-rgw-ansible-var

qa/tests: Add additional required ceph-ansible vars due to upstream changes

Merge pull request #17910 from smithfarm/wip-21499

tests: point-to-point-x: upgrade client.1 to -x along with cluster nodes

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #17904 from carltonf/wip-fix-21498

ceph-disk: fix '--runtime' omission for ceph-osd service

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17770 from aclamk/fix-bluestore-fio-hash

bluestore/fio: Fixed problem with all objects having the same hash

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17901 from tchaikov/wip-cmake-gperftools

cmake: update the error message for gperftools bug

qa: point-to-point-x: upgrade client.1 to -x along with cluster nodes

The client.1 rgw in workload_x had not been upgraded to -x.

Fixes: http://tracker.ceph.com/issues/21499
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #17806 from tchaikov/wip-osdc-new-options

osdc/Journaler: use new style options

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #17823 from xiexingguo/wip-clone-bytes

osd/osd_type: get_clone_bytes - inline size() for overlapping size

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17794 from hjwsm1989/wip-21425

qa/suites/rest/basic/tasks/rest_test: more whitelisting

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17744 from xiexingguo/wip-blue-eio

os/bluestore: propagate read-EIO to high level callers

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17460 from dengquan/master

java/native: fix milliseconds to mtime/atime conversion

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17422 from aclamk/fix-cmake-unsupported-allocator

cmake: in case of bad "ALLOCATOR" selected issue warning

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17374 from jcsp/wip-mgr-config-tidy

mgr: use new style config opts + add metadata

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17309 from jcsp/wip-daemonserver-locks

mgr: locking fixes

Reviewed-by: Sage Weil <sage@redhat.com>

osdc/Journaler: use new style options

also use uint64_t for these options, as they are casted to uint64_t when
reading from conf.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17830 from Yan-waller/wip-walle-0920primarypg

osd/PrimaryLogPG: misc cleanups

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-disk: fix '--runtime' omission for ceph-osd service

f425a127b introduces a regression that ceph-disk omits "--runtime" when
enabling ceph-osd@$ID.service units for device-backed OSDs.

Fixes: http://tracker.ceph.com/issues/21498
Signed-off-by: Carl Xiong <cxiong@suse.com>

cmake: update the error message for gperftools bug

it's a follow-up of #17788

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17891 from b-ranto/wip-selinux-lnk-getattr

selinux: Allow getattr on lnk sysfs files

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #17739 from liewegas/wip-filestore-configs

common/options: document filestore and filejournal options

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17740 from liewegas/wip-objecter-configs

common/options: document objecter, filer, and journal options

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17579 from liewegas/wip-kill-snapdirs

osd: kill snapdirs

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17895 from ivancich/wip-fix-mclock-coverity

tests: fix uninitialized value found by coverity scan

Reviewed-by: Kefu Chai <kchai@redhat.com>

use ovh instead of vps

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

tests: fix uninitialized value found by coverity scan

Unit test for mclock priority queue had a default constructor, which
didn't initialize a value.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

osd: make shutdown debug conditional (and off by default)

Signed-off-by: Sage Weil <sage@redhat.com>

common/options: document filestore and filejournal options

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: can_create false for cache ops

Some cache ops do writes, but none of them *create* objects, so we
should not set can_create.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: drop odd maintain_ssc hack

This was added two years ago by b61f3e43f1c9d43daa0dedd7cbd0fe5787cbdfbb,
but I can see why it is needed here.  The test case for that
original bug (from a3442cf07841a7269713e30ca270f2f9c356a598) passes
fine.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-objectstore-tool: drop pre-snapset support for setting up snapmapper

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-objectstore-tool: drop head_exists corruption

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_rados_api_pool: make LibRadosPools.PoolList tolerate long pool names

There was a baked-in assumption that the first pool was <20 chars.
No more!

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: object_info_t: remove legacy_snaps

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: drop last legacy_snaps users (just debug output!)

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: SnapSet: remove head_exists

Signed-off-by: Sage Weil <sage@redhat.com>

osd: drop last head_exists = true calls

No more users!

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: find_object_context: drop !head_exists path

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: do_osd_ops: head_exists is always true for LIST_SNAPS

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: SnapSet: drop is_legacy

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: start_flush: drop legacy snapset handling

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: default head_exists = true

This will be helpful as we start removing code that would otherwise
set this to true in some convoluted way.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: finish_promote: drop legacy snapset handling

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: do_copy_get: drop legacy snapset path

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: _delete_oid: drop legacy snapset handling

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: do_osd_ops: drop legacy snapset handling from LIST_SNAPS

This is so much simpler, yay!

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: on_local_recover: drop legacy snapset handling

This was there to deal with jewel peers, which are no more.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: _scan_snaps: drop legacy snapset handling

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: find_object_context: drop legacy snapset handling

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: issue_repop: drop obsolete condition on head_obc

The only user is trim_object, which sets obc = clone, head_obc =
head.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: snapset_obc -> head_obc

This is only used for trim_object, now!

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: do_op: drop snapset_obc lookup on snapdir

We'll never find it.  This also means we can drop the goofy hack
to clear it for cache tiers.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: trim_object: ctx->snapset_obc -> head_obc

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: trim_object: simplify a bit more for post-snapdir world

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: trim_object: drop legacy SnapSet handling

Signed-off-by: Sage Weil <sage@redhat.com>

common: use mono clock for HeartbeatMap

to avoid problems when admin sets system clock.

see https://gcc.gnu.org/bugzilla/show_bug.cgi?id=53901 for the reason
why we cannot use atomic<time_point>.

Signed-off-by: Kefu Chai <kchai@redhat.com>
Signed-off-by: Xinze Chi <xinze@xsky.com>

selinux: Allow getattr on lnk sysfs files

This showed up during downstream testing for luminous. We are doing
getattr on the sysfs lnk files and the current policy does not allow
this.

Signed-off-by: Boris Ranto <branto@redhat.com>

Merge pull request #17872 from joscollin/wip-osdmap-signed-unsigned-warning

osd: silence warnings from -Wsign-compare

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

mgr: silence warning from -Wsign-compare

Fixed the warning:

ceph/src/mgr/Mgr.cc: In lambda function:
ceph/src/mgr/Mgr.cc:413:42: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
     for (unsigned int osd_id = 0; osd_id < osd_map.get_max_osd(); ++osd_id) {
                                   ~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~

Signed-off-by: Jos Collin <jcollin@redhat.com>

osd: silence warnings from -Wsign-compare

Fixed the warning :

ceph/src/osd/OSDMap.cc: In member function ‘int OSDMap::validate_crush_rules(CrushWrapper*, std::ostream*) const’:
ceph/src/osd/OSDMap.cc:3278:25: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
     if (pool.get_size() < (int)newcrush->get_rule_mask_min_size(ruleno) ||
         ~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ceph/src/osd/OSDMap.cc:3279:18: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
  pool.get_size() > (int)newcrush->get_rule_mask_max_size(ruleno)) {
  ~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #17833 from mogeb/cache-tiering-doc

doc: update link to placing-different-pools

Reviewed-by: Kefu Chai <kchai@redhat.com>

mds: check ongoing catter-gather process before capping log

When deactivating mds, MDLog::trim() may start scatter-gather
process on mdsdir inode. Locker::scatter_writebehind() submits
log entry. So mds should make sure there is no scatter-gather
before capping log.

Fixes: http://tracker.ceph.com/issues/21467
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge PR #17849 into master

* refs/remotes/upstream/pull/17849/head:
qa: get config only on running MDS

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

client: use unique_ptr for locks

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

client: use unique_ptr for Formatter

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: revert "qa: use config_path property instead of literal"

This reverts commit f95798b3adb30c866725ef26aee75c4811655752.

The config_path method wasn't available through inheritance as I thought. Oops.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: get config only on running MDS

Fixes: http://tracker.ceph.com/issues/21466
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #17791 into master

* refs/remotes/upstream/pull/17791/head:
client: set client_try_dentry_invalidate to false by default

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #17784 into master

* refs/remotes/upstream/pull/17784/head:
vstart: create MDSs before filesystems

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #17783 into master

* refs/remotes/upstream/pull/17783/head:
mon/MDSMonitor: fix clog msgs on MDS rank add/rm

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #17778 into master

* refs/remotes/upstream/pull/17778/head:
client: move Fh init to ctor

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge PR #17768 into master

* refs/remotes/upstream/pull/17768/head:
ceph: do link/rename semantic checks after srcdn is readable

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

os/bluestore: ignore 0x2000~2000 extent oddity from luminous upgrade

Luminous does a block_size granularity freelist, and assumes that
0~ROUND_UP_TO(SUPER_RESERVED,block_size) is used.  Current master uses
min_alloc_size granularity and changes that assumption to
0~ROUND_UP_TO(SUPER_RESERVED,min_alloc_size).  That means if master
fsck's a luminous-created bluestore, it will think 0x2000~2000 is used
(current baked-in min_alloc_size-based assumption) but the old freelist
says it is free (old mkfs assumption).  The disparity is harmless since
the extent is below min_alloc_size, so ignore it.

Fixes: http://tracker.ceph.com/issues/21408
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #17807 from xiexingguo/wip-kill-add-interval-usage

osd/PrimaryLogPG: kill add_interval_usage

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17723 from amitkumar50/cov-librados-3

rados: Initialization of alignment

Reviewed-by: Jos Collin <jcollin@redhat.com>

doc: update 'ceph osd pool set' example

Signed-off-by: Mohamad Gebai <mgebai@suse.com>

osd: debug_bluestore on shutdown

...just like we do with filestore etc.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: dump stray cache content on shutdown

Tracking down http://tracker.ceph.com/issues/21259.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #17747 from tchaikov/wip-qa

qa/standalone: respect $TEMPDIR

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #17828 from jdurgin/wip-21428

osd/PrimaryLogPG: request osdmap update in the right block

Reviewed-by: Sage Weil <sage@redhat.com>

ceph_volume_client: perform snapshot operations in

... client configured snap directory name, instead of in hard-coded
'.snap' directory.

Fixes: http://tracker.ceph.com/issues/21476
Signed-off-by: Ramana Raja <rraja@redhat.com>

doc: update link to placing-different-pools

Signed-off-by: Mohamad Gebai <mgebai@suse.com>

osd/PrimaryLogPG: drop unused variables

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

osd/PrimaryLogPG: avoid accessing value of map.end()

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

osd/PrimaryLogPG: request osdmap update in the right block

Fixes: http://tracker.ceph.com/issues/21428
Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #17785 from dzafman/wip-add-repair

test: Fix ceph-objectstore-tool usage check

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd/osd_type: get_clone_bytes - inline size() for overlapping size

This is faster and simpler. A similar issue can be found at:
https://github.com/ceph/ceph/pull/17807

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #17138 from Liuchang0812/wip-updata-osd-hostname-in-metaupdater

mgr: preventing blank hostname in DaemonState

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #17738 from jtlayton/wip-jlayton-20988

common: lockdep fixes

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Adam Emerson <aemerson@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>