osd/PG: make num_objects_missing can trace missing objects correctly

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/osd_types: kill overlaps_with() of pg_info_t

Which has no consumers.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/PG: clear calc_replicated_acting a bit

- kill usable, use want->size instead
- introduce a (separate) lambda function for sorting

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/PG: move comments to the proper place

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/PG: find_best_info - add completeness as the preferred option

Async recovery peers usually have a relative complete
log history but may exist a lot of missing objects. Choosing them
as auth_log_shard and further as primary if current up_primary is
unrecoverable, say, could have a bigger chance to block client I/Os.

Among peers with identical new log history, we now consider those
who are now complete (having no missing objects) as the preferred
ones when determining auth_log_shard.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #23795 from tchaikov/wip-cmake-definitions-cleanup

cmake: no need to add "-D" before definitions

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #23774 from tchaikov/wip-include-cleanup

auth,common: include cleanups

Reviewed-by: Noah Watkins <nwatkins@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23733 from dillaman/wip-migration-parent

librbd: always open first parent image if it exists for a snapshot

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #23788 from alfredodeza/wip-rm34311

ceph-volume: tests.functional inherit SSH_ARGS from ansible

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume tests.functional use Debian ansible os family in deploy.yml

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge PR #23742 into master

* refs/pull/23742/head:
mon/OSDMonitor: don't change in-memory state on prune

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

ceph-volume lvm.batch use the abspath attribute for Device objects in bluestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.batch remove usage of non-existent method

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #23801 from alfredodeza/wip-rm34317

doc/rados update invalid bash on bluestore migration

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-volume tests.functional inherit SSH_ARGS from ansible with synchronize module

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #23787 from alfredodeza/wip-rm34310

ceph-volume lvm.batch remove non-existent sys_api property

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #23741 from liewegas/wip-protect-monmap

mon: only share monmap after authenticating

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23738 from dmick/wip-fix-crashdump

global/signal_handler.cc: report assert_file as correct name

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23734 from hjwsm1989/wip-speedup-upmap

osd/OSDMap: map pgs with smaller batchs in calc_pg_upmaps

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #23681 from tchaikov/wip-26994

qa/mgr/selftest: handle always-on module fall out

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/rados update invalid bash on bluestore migration

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #23706 from tspmelo/wip-update-angular

mgr/dashboard: Update Angular packages

Reviewed-by: Alfonso Martínez <almartin@redhat.com>

cmake: no need to add "-D" before definitions

and there is no need to make the definition public for the executable
target.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon,auth,common: include necessary headers

we cannot assume the included header alway indirectly include the use types

in this case, CephContext is defined by ceph_context.h, and
g_ceph_context is declared by global_context.h. so we need to included
them respectively.

and remove unused headers.

Signed-off-by: Kefu Chai <kchai@redhat.com>

auth/AuthMethodList: add std:: before type names

we cannot assume 'using namespace std`

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-volume lvm.batch remove non-existent sys_api property

Signed-off-by: Alfredo Deza <adeza@redhat.com>

qa/mgr/selftest: handle always-on module fall out

need a non-always-on module. hello doesn't work because it isn't
installed. so switch to selftest.

Signed-off-by: Noah Watkins <nwatkins@redhat.com>

Merge pull request #23765 from chenzhua/master

doc: fix spelling errors in rbd doc

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #23596 from cbodley/wip-rgw-no-datalog-marker

rgw multisite: incremental data sync uses truncated flag to detect end of listing

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #23768 from wjwithagen/wjw-fix-shared_mutex

core: Clang misses <optional> include

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: Update Angular packages

Signed-off-by: Tiago Melo <tmelo@suse.com>

core: Clang misses <optional> include

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #23764 from ceph/docs

doc: fixed typo in cephfs snapshots

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #23699 from tchaikov/wip-monc-drop-rwlock

auth: drop the RWLock in AuthClientHandler

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc: fixed typo in cephfs snapshots

Signed-off-by: Kai Wagner <kwagner@suse.com>

doc: fix spelling errors in rbd doc

Signed-off-by: Chen Zhenghua <chen.zhenghua@zte.com.cn>

Merge pull request #23763 from chenzhua/master

doc: fix cephfs spelling errors

Reviewed-by: Jos Collin <jcollin@redhat.com>

doc: fix cephfs spelling errors

Signed-off-by: Chen Zhenghua <chen.zhenghua@zte.com.cn>

mon/OSDMonitor: don't change in-memory state on prune

We were modifying the in-memory state when running through several
osdmap pruning functions.

Should the transaction we're encoding not be committed, we may end up in
a state where we have a stale in-memory state that does not match what
is on disk; including having in-memory state while not having on-disk
state.

We prevent this sort of inconsistency by working on temporary states
instead.

Fixes: http://tracker.ceph.com/issues/24612
Signed-off-by: Joao Eduardo Luis <joao@suse.de>

librbd: always open first parent image if it exists for a snapshot

The deep-copy and migration features required force-opening the
parent image just in case the deep-flatten feature wasn't enabled
on an image. This change simplies the code by always opening the
direct parent image, which really only matters if a cloned image
has snapshots w/o the deep-flatten feature.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #23718 into master

* refs/pull/23718/head:
cleanup: Replace always-false assertions with abort
include/assert: Smarten up ceph_assert
common: Call ceph_abort(), not abort()
include,common: Remove assert and clobber-foo
rbd_replay: Use ceph_assert for asserts.
rbd_fuse: Use ceph_assert for asserts.
osdc: Use ceph_assert for asserts.
osd: Use ceph_assert for asserts.
msg: Use ceph_assert for asserts.
mgr: Use ceph_assert for asserts.
messages: Use ceph_assert for asserts.
mds: Use ceph_assert for asserts.
log: Use ceph_assert for asserts.
libradosstriper: Use ceph_assert for asserts.
librados: Use ceph_assert for asserts.
kv: Use ceph_assert for asserts.
test: Use ceph_assert for asserts.
key_value_store: Use ceph_assert for asserts.
json_spirit: Use ceph_assert for asserts.
journal: Use ceph_assert for asserts.
global: Use ceph_assert for asserts.
erasure-code: Use ceph_assert for asserts.
rbd: Use ceph_assert for asserts.
tools: Use ceph_assert for asserts.
rgw: Use ceph_assert for asserts.
mon: Use ceph_assert for asserts.
crush: Use ceph_assert for asserts in C++ code.
crimson: Use ceph_assert for asserts.
compressor: Use ceph_assert for asserts.
cls: Use ceph_assert for asserts.
client: Use ceph_assert for asserts.
auth: Use ceph_assert for asserts
os: Use ceph_assert for asserts
common: Use ceph_assert for asserts
include: Use ceph_assert for asserts
include/assert: Deprecate assert macro

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

mon: only share monmap after authenticated

It is no longer necessary to fetch a monmap pre-authentication, something
we previous did for get_monmap_privately().  New code has replaced this
with get_monmap_and_config(), and it authenticates in order to get that
same information (plus configs).

That change was made in mimic, but we must support upgrades from N-2,
which means that luminous daemons still need to function.  The only caller
for get_monmap_privately() in luminous is from ceph-osd during mkfs.
Disabling this here means that new OSDs cannot be created using nautilus
mons and a luminous ceph-osd.  Include a note for the (future) nautilus
upgrade notes.

Reported-by: Christopher Ryan Harrell <harrellcr@email.arizona.edu>
Signed-off-by: Sage Weil <sage@redhat.com>

mon/AuthMonitor: do not share monmap until authenticated

Reported-by: Christopher Ryan Harrell <harrellcr@email.arizona.edu>
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #23750 from tchaikov/wip-doc-cephx

doc/dev/cephx_protocol: fix couple errors

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23747 from tchaikov/wip-monclient-no-magic-nr

mon/MonClient: avoid using magic number for the `MAuth::protocol`

Reviewed-by: Sage Weil <sage@redhat.com>

cleanup: Replace always-false assertions with abort

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

include/assert: Smarten up ceph_assert

Change ceph_assert_msg to not require a CephContext. Add
ceph_assert_msgf that accepts a format string.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common: Call ceph_abort(), not abort()

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

include,common: Remove assert and clobber-foo

Since everything is using the ceph_assert macro, get rid of the assert
macro in include/assert.h .

As that is no longer a concern, get of macros to break the compile if
our macro gets replaced with the system assert.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rbd_replay: Use ceph_assert for asserts.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rbd_fuse: Use ceph_assert for asserts.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

osdc: Use ceph_assert for asserts.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

osd: Use ceph_assert for asserts.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

msg: Use ceph_assert for asserts.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

mgr: Use ceph_assert for asserts.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

messages: Use ceph_assert for asserts.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

mds: Use ceph_assert for asserts.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

log: Use ceph_assert for asserts.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

libradosstriper: Use ceph_assert for asserts.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

librados: Use ceph_assert for asserts.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

kv: Use ceph_assert for asserts.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

test: Use ceph_assert for asserts.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

key_value_store: Use ceph_assert for asserts.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

json_spirit: Use ceph_assert for asserts.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

journal: Use ceph_assert for asserts.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

global: Use ceph_assert for asserts.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

erasure-code: Use ceph_assert for asserts.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rbd: Use ceph_assert for asserts.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

tools: Use ceph_assert for asserts.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Use ceph_assert for asserts.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

mon: Use ceph_assert for asserts.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #20526 from k0ste/newfeature

docs: radosgw: ldap-auth: fixed option name 'rgw_ldap_searchfilter'

Reviewed-by: Casey Bodley <cbodley@redhat.com>

crush: Use ceph_assert for asserts in C++ code.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

crimson: Use ceph_assert for asserts.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

compressor: Use ceph_assert for asserts.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

cls: Use ceph_assert for asserts.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

client: Use ceph_assert for asserts.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

auth: Use ceph_assert for asserts

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

os: Use ceph_assert for asserts

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common: Use ceph_assert for asserts

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

include: Use ceph_assert for asserts

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

include/assert: Deprecate assert macro

Prior to switching callers to ceph_assert.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

doc/dev/cephx_protocol: fix couple errors

there are 3 phases of authentication and authorization in cephx:

0. proto = CEPH_AUTH_UNKNOWN, for start the session,
   create auth handlers on both sides.
1. GET_AUTH_SESSION_KEY, to get authenticated, the client will
   get the session ticket at the end of this phase. the session ticket,
   is actually also a service ticket for CEPH_ENTITY_TYPE_AUTH.
2. CEPHX_GET_PRINCIPAL_SESSION_KEY, to get authorized, the client
   will get the requested, service tickets at the end of this phase.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23391 from votdev/cleanup_25161

mgr/dashboard: Prevent API call on every keystroke

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mon/MonClient: avoid using magic number for the `MAuth::protocol`

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23746 from libingyang-zte/master

doc: Fix Spelling Error of Rados Deployment/Operations

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

doc: Fix Spelling Error of Rados Deployment/Operations
Signed-off-by: Li Bingyang <li.bingyang1@zte.com.cn>

Merge pull request #23745 from chenzhua/master

doc: fix spelling errors of cephfs

Reviewed-by: Jos Collin <jcollin@redhat.com>

doc: fix spelling errors of cephfs

Signed-off-by: Chen Zhenghua <chen.zhenghua@zte.com.cn>

Merge pull request #23688 from tchaikov/wip-monclient-sub

mon/MonClient: extract MonSub out

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23736 from dillaman/wip-librbd-gmock

test/librbd: fix gmock warning in snapshot rollback test

Reviewed-by: songweibin <song.weibin@zte.com.cn>
Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #23088 into master

* refs/pull/23088/head:
mds: adjust export size after renaming directory into freezing subtree
mds: clear scatterlock's flushed when removing dirty
mds: export subtree part by part
mds: change default mds_max_export_size to 20M
mds: limit total size of exporting subtrees
mds: optimize the way how max export size is enforced
mds: make CInode::get_dirfrags container agnostic
mds: cleanup MutationImpl::More initialization

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23548 into master

* refs/pull/23548/head:
mds: migrate strays part by part when shutdown mds

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23439 into master

* refs/pull/23439/head:
qa: whitelist cap revoke warning
doc: document cap revoke non-responders client eviction
test: validate client eviction for cap revoke non-responders
mds: add counter for tracking cap non-responding clients
mds: evict clients that do not respond to cap revoke by MDS
mds: pass timeout argument for fetching late clients

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #23413 into master

* refs/pull/23413/head:
mds: fix premature stop of MDBalancer::try_rebalance()
mds: don't queue straydir for split

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23469 into master

* refs/pull/23469/head:
mds: access last std::vector element with `back()`
mds: disallow certain file operations to "." and ".." dirents

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #23656 into master

* refs/pull/23656/head:
build/ops: rpm: package cephfs-shell for SUSE as well as Fedora

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #23640 into master

* refs/pull/23640/head:
mds: use monotonic waits in Beacon

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

githubmap: update contributors

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: whitelist cap revoke warning

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>