osdc/Objecter: fix split vs reconnect race

If we are reconnecting to an OSD, we may not have a connection established
so this feature check is completely unreliable.  If there happens to be a
split, that could mean we don't resend an op that should have been resent.

Instead, we can infer the OSD has the new behavior if require_osd_release
is luminous or later.  If not, we can look at the per-osd features in the
OSDMap.

Fixes: http://tracker.ceph.com/issues/22544
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #23840 from idryomov/wip-krbd-untar-build-packages

qa: rbd_workunit_kernel_untar_build: install build dependencies

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #22983 from trociny/wip-journal-limit-inflight-appends

librbd: optionally limit journal in-flight appends

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #23770 from Songweibin/wip-rbd-namespace

rbd: correct parameter of namespace and verify it before set_namespace

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa: rbd_workunit_kernel_untar_build: install build dependencies

Commit f0fe0936e64d ("qa: use recent kernel to kernel build testing")
bumped the kernel to 4.17.

Fixes: http://tracker.ceph.com/issues/35074
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #23838 from wjwithagen/wjw-fix-ConfUtil

common: Reverse deleted include

Reviewed-by: Kefu Chai <kchai@redhat.com>

common: Reverse deleted include

Reversing 1 delete from
https://github.com/ceph/ceph/commit/09d3f546b32cdf126a1246899a0b0a3eb25282d8

Clang trips over this:
home/jenkins/workspace/ceph-master/src/common/ConfUtils.cc:94:19: error: implicit instantiation of undefined template 'std::__1::basic_ostringstream<char, std::
__1::char_traits<char>, std::__1::allocator<char> >'
    ostringstream oss;
                  ^
/usr/include/c++/v1/iosfwd:123:32: note: template is declared here
    class _LIBCPP_TEMPLATE_VIS basic_ostringstream;
                               ^

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #23802 from votdev/unique_validator

mgr/dashboard: Add unique validator

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #23629 from wangxiaoguang/fix_deep_scrub

os/bluestore: fix deep-scrub operation againest disk silent errors

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #23555 from tspmelo/wip-task-refactor

mgr/dashboard: Extract/Refactor Task merge

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

os/bluestore: fix deep-scrub operation againest disk silent errors

Say a object who has data caches, but in a while later, caches' underlying
physical device has silent disk erros accidentally, then caches and physical
data are not same. In such case, deep-scrub operation still tries to read
caches firstly and won't do crc checksum, then deep-scrub won't find such
data corruptions timely.

Here introduce a new flag 'CEPH_OSD_OP_FLAG_BYPASS_CLEAN_CACHE' which tells
deep-scrub to bypass object caches. Note that we only bypass cache who is in
STATE_CLEAN state. For STATE_WRITING caches, currently they are not written
to physical device, so deep-scrub operation can not read physical device and
can read these dirty caches safely. Once they are in STATE_CLEAN state(or not
added to bluestore cache), next round deep-scurb can check them correctly.

As to above discussions, I refactor BlueStore::BufferSpace::read sightly,
adding a new 'flags' argument, whose value will be 0 or:
     enum {
       BYPASS_CLEAN_CACHE = 0x1,     // bypass clean cache
     };

flags 0: normal read, do not bypass clean or dirty cache
flags BYPASS_CLEAN_CACHE: bypass clean cache, currently only for deep-scrube
                        operation

Test:
   I deliberately corrupt a object with cache, with this patch, deep-scrub
   can find data error very timely.

Signed-off-by: Xiaoguang Wang <xiaoguang.wang@easystack.cn>

Merge pull request #23546 from xiexingguo/wip-complete-to

osd/PGLog: trim - avoid dereferencing invalid iter

Reviewed-by: Yan Jun <yan.jun8@zte.com.cn>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #23800 from rjfd/wip-fix-crash-mgr-module

mgr/crash: fix python3 invalid syntax problems

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Noah Watkins <nwatkins@redhat.com>

Merge pull request #23803 from tchaikov/wip-ceph_assert

include/buffer.h: do not use ceph_assert() unless __CEPH__ is defined

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #23820 from LenzGr/nautilus-404-page

mgr/dashboard: Updated image on 404 page

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Extract/Refactor Task merge

Signed-off-by: Stephan Müller <smueller@suse.com>
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #23785 from alfredodeza/wip-rm34309

ceph-volume:  update batch documentation to explain filestore strategies

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

librbd: option to limit in-flight appends

Signed-off-by: Mykola Golub <mgolub@suse.com>

journal: limit in-flight appends

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge PR #23509 into master

* refs/pull/23509/head:
OSD: add recovery bytes counter to osd logger

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #23572 into master

* refs/pull/23572/head:
qa/standalone/osd/osd-force-create-pg: add force-create-pg test
mon/MonCommands: fix 'osd force-create-pg'

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #23611 into master

* refs/pull/23611/head:
mgr/devicehealth: fix exceptions
doc/mgr/devicehealth: update docs
mgr/devicehealth: leave health monitoring off by default
mgr/devicehealth: add 'device monitoring {on,off}' commands
mgr/devicehealth: 10m wakeup interval is sufficient
mgr/devicehealth: scrape at scrape_interval intervals
mgr/devicehealth: UTC everywhere
mgr/devicehealth: fix sleep_interval configurable

Merge PR #23633 into master

* refs/pull/23633/head:
osd/PrimaryLogPG.cc: limit trimming at can_rollback_to

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mgr/dashboard: Add unique validator

Relocate an already existing async validator into a separate validator that can be reused by every other form. This validator is useful to check immediately after typing if an entered value, e.g. username, already exists.

The API request will be triggered after a delay of 500ms (can be modified). During this delay, every keystroke will reset the timer, so the REST API is not flooded with request.

Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #23712 from tspmelo/wip-browser-support

mgr/dashboard: Add documentation about supported browsers

Reviewed-by: Volker Theile <vtheile@suse.com>

rbd: minor cleanup for rbd namespace

Signed-off-by: songweibin <song.weibin@zte.com.cn>

rbd: verify the specified namespace before set

Signed-off-by: songweibin <song.weibin@zte.com.cn>

rbd: add new librbd method to assert namespace exists

Signed-off-by: songweibin <song.weibin@zte.com.cn>

Merge pull request #23796 from votdev/fix_form_button_layout

mgr/dashboard: Add gap between panel footer buttons

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Updated image on 404 page

Replaced the image of a Mimic octopus with a Nautilus octopus in
preparation for the new major release code name.

The image was taken from Flickr (https://www.flickr.com/photos/146401137@N06/40335060661) and is
licensed under the Creative Commons "Attribution 2.0 Generic"
(CC BY 2.0) license.

Fixes: https://tracker.ceph.com/issues/24489
Signed-off-by: Lenz Grimmer <lgrimmer@suse.com>

mgr/dashboard: Add documentation about supported browsers

Fixes: http://tracker.ceph.com/issues/27207
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #23814 from joke-lee/doc_rgw_period_root_pool_long_description

doc: fix long description error for rgw_period_root_pool

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #23767 from tchaikov/wip-operator-insertion-ADL

include/types: move operator<< into the proper namespace

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23795 from tchaikov/wip-cmake-definitions-cleanup

cmake: no need to add "-D" before definitions

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #23774 from tchaikov/wip-include-cleanup

auth,common: include cleanups

Reviewed-by: Noah Watkins <nwatkins@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

core: add missing flag name for CEPH_OSD_OP_FLAG_WITH_REFERENCE

Signed-off-by: Xiaoguang Wang <xiaoguang.wang@easystack.cn>

doc: fix long description error for rgw_period_root_pool

Signed-off-by: yuliyang <yuliyang@cmss.chinamobile.com>

Merge pull request #23733 from dillaman/wip-migration-parent

librbd: always open first parent image if it exists for a snapshot

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #23788 from alfredodeza/wip-rm34311

ceph-volume: tests.functional inherit SSH_ARGS from ansible

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume tests.functional use Debian ansible os family in deploy.yml

Signed-off-by: Alfredo Deza <adeza@redhat.com>

mgr/devicehealth: fix exceptions

Signed-off-by: Sage Weil <sage@redhat.com>

doc/mgr/devicehealth: update docs

- several new commands weren't documented

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/devicehealth: leave health monitoring off by default

For now at least.. unless all of the tests have been adjusted accordingly
to expect a device_health_metrics pool.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #23742 into master

* refs/pull/23742/head:
mon/OSDMonitor: don't change in-memory state on prune

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

ceph-volume lvm.batch use the abspath attribute for Device objects in bluestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.batch remove usage of non-existent method

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #23801 from alfredodeza/wip-rm34317

doc/rados update invalid bash on bluestore migration

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-volume tests.functional inherit SSH_ARGS from ansible with synchronize module

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #23787 from alfredodeza/wip-rm34310

ceph-volume lvm.batch remove non-existent sys_api property

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #23741 from liewegas/wip-protect-monmap

mon: only share monmap after authenticating

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23738 from dmick/wip-fix-crashdump

global/signal_handler.cc: report assert_file as correct name

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23734 from hjwsm1989/wip-speedup-upmap

osd/OSDMap: map pgs with smaller batchs in calc_pg_upmaps

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #23681 from tchaikov/wip-26994

qa/mgr/selftest: handle always-on module fall out

Reviewed-by: Kefu Chai <kchai@redhat.com>

include/buffer.h: do not use ceph_assert() unless __CEPH__ is defined

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/rados update invalid bash on bluestore migration

Signed-off-by: Alfredo Deza <adeza@redhat.com>

mgr/crash: fix python3 invalid syntax problems

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: Add gap between panel footer buttons

In forms there was no gap between the panel footer buttons.

Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #23706 from tspmelo/wip-update-angular

mgr/dashboard: Update Angular packages

Reviewed-by: Alfonso Martínez <almartin@redhat.com>

cmake: no need to add "-D" before definitions

and there is no need to make the definition public for the executable
target.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon,auth,common: include necessary headers

we cannot assume the included header alway indirectly include the use types

in this case, CephContext is defined by ceph_context.h, and
g_ceph_context is declared by global_context.h. so we need to included
them respectively.

and remove unused headers.

Signed-off-by: Kefu Chai <kchai@redhat.com>

auth/AuthMethodList: add std:: before type names

we cannot assume 'using namespace std`

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-volume lvm.batch remove non-existent sys_api property

Signed-off-by: Alfredo Deza <adeza@redhat.com>

qa/mgr/selftest: handle always-on module fall out

need a non-always-on module. hello doesn't work because it isn't
installed. so switch to selftest.

Signed-off-by: Noah Watkins <nwatkins@redhat.com>

doc/man/ceph-volume uncomment flags that are now fully supported in batch

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume add filestore docs for batch

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume uncomment section of newly supported batch operations

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #23765 from chenzhua/master

doc: fix spelling errors in rbd doc

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #23596 from cbodley/wip-rgw-no-datalog-marker

rgw multisite: incremental data sync uses truncated flag to detect end of listing

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #23768 from wjwithagen/wjw-fix-shared_mutex

core: Clang misses <optional> include

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: Update Angular packages

Signed-off-by: Tiago Melo <tmelo@suse.com>

core: Clang misses <optional> include

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

rbd: correct parameter of namespace which cannot be recognized

Signed-off-by: songweibin <song.weibin@zte.com.cn>

Merge pull request #23764 from ceph/docs

doc: fixed typo in cephfs snapshots

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #23699 from tchaikov/wip-monc-drop-rwlock

auth: drop the RWLock in AuthClientHandler

Reviewed-by: Casey Bodley <cbodley@redhat.com>

include/types: move operator<< into the proper namespace

we should define the operator<< in the namespace of its 2nd parameter.

normally, the operator<< defined in the global namespace works. but
there is a subtle difference between the operator<< defined in std
namespace and the global one because of Koenig lookup.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: fixed typo in cephfs snapshots

Signed-off-by: Kai Wagner <kwagner@suse.com>

doc: fix spelling errors in rbd doc

Signed-off-by: Chen Zhenghua <chen.zhenghua@zte.com.cn>

Merge pull request #23763 from chenzhua/master

doc: fix cephfs spelling errors

Reviewed-by: Jos Collin <jcollin@redhat.com>

doc: fix cephfs spelling errors

Signed-off-by: Chen Zhenghua <chen.zhenghua@zte.com.cn>

mon/OSDMonitor: don't change in-memory state on prune

We were modifying the in-memory state when running through several
osdmap pruning functions.

Should the transaction we're encoding not be committed, we may end up in
a state where we have a stale in-memory state that does not match what
is on disk; including having in-memory state while not having on-disk
state.

We prevent this sort of inconsistency by working on temporary states
instead.

Fixes: http://tracker.ceph.com/issues/24612
Signed-off-by: Joao Eduardo Luis <joao@suse.de>

librbd: always open first parent image if it exists for a snapshot

The deep-copy and migration features required force-opening the
parent image just in case the deep-flatten feature wasn't enabled
on an image. This change simplies the code by always opening the
direct parent image, which really only matters if a cloned image
has snapshots w/o the deep-flatten feature.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #23718 into master

* refs/pull/23718/head:
cleanup: Replace always-false assertions with abort
include/assert: Smarten up ceph_assert
common: Call ceph_abort(), not abort()
include,common: Remove assert and clobber-foo
rbd_replay: Use ceph_assert for asserts.
rbd_fuse: Use ceph_assert for asserts.
osdc: Use ceph_assert for asserts.
osd: Use ceph_assert for asserts.
msg: Use ceph_assert for asserts.
mgr: Use ceph_assert for asserts.
messages: Use ceph_assert for asserts.
mds: Use ceph_assert for asserts.
log: Use ceph_assert for asserts.
libradosstriper: Use ceph_assert for asserts.
librados: Use ceph_assert for asserts.
kv: Use ceph_assert for asserts.
test: Use ceph_assert for asserts.
key_value_store: Use ceph_assert for asserts.
json_spirit: Use ceph_assert for asserts.
journal: Use ceph_assert for asserts.
global: Use ceph_assert for asserts.
erasure-code: Use ceph_assert for asserts.
rbd: Use ceph_assert for asserts.
tools: Use ceph_assert for asserts.
rgw: Use ceph_assert for asserts.
mon: Use ceph_assert for asserts.
crush: Use ceph_assert for asserts in C++ code.
crimson: Use ceph_assert for asserts.
compressor: Use ceph_assert for asserts.
cls: Use ceph_assert for asserts.
client: Use ceph_assert for asserts.
auth: Use ceph_assert for asserts
os: Use ceph_assert for asserts
common: Use ceph_assert for asserts
include: Use ceph_assert for asserts
include/assert: Deprecate assert macro

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

mon: only share monmap after authenticated

It is no longer necessary to fetch a monmap pre-authentication, something
we previous did for get_monmap_privately().  New code has replaced this
with get_monmap_and_config(), and it authenticates in order to get that
same information (plus configs).

That change was made in mimic, but we must support upgrades from N-2,
which means that luminous daemons still need to function.  The only caller
for get_monmap_privately() in luminous is from ceph-osd during mkfs.
Disabling this here means that new OSDs cannot be created using nautilus
mons and a luminous ceph-osd.  Include a note for the (future) nautilus
upgrade notes.

Reported-by: Christopher Ryan Harrell <harrellcr@email.arizona.edu>
Signed-off-by: Sage Weil <sage@redhat.com>

mon/AuthMonitor: do not share monmap until authenticated

Reported-by: Christopher Ryan Harrell <harrellcr@email.arizona.edu>
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #23750 from tchaikov/wip-doc-cephx

doc/dev/cephx_protocol: fix couple errors

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23747 from tchaikov/wip-monclient-no-magic-nr

mon/MonClient: avoid using magic number for the `MAuth::protocol`

Reviewed-by: Sage Weil <sage@redhat.com>

cleanup: Replace always-false assertions with abort

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

include/assert: Smarten up ceph_assert

Change ceph_assert_msg to not require a CephContext. Add
ceph_assert_msgf that accepts a format string.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common: Call ceph_abort(), not abort()

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

include,common: Remove assert and clobber-foo

Since everything is using the ceph_assert macro, get rid of the assert
macro in include/assert.h .

As that is no longer a concern, get of macros to break the compile if
our macro gets replaced with the system assert.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rbd_replay: Use ceph_assert for asserts.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rbd_fuse: Use ceph_assert for asserts.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

osdc: Use ceph_assert for asserts.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

osd: Use ceph_assert for asserts.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

msg: Use ceph_assert for asserts.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

mgr: Use ceph_assert for asserts.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

messages: Use ceph_assert for asserts.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

mds: Use ceph_assert for asserts.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

log: Use ceph_assert for asserts.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

libradosstriper: Use ceph_assert for asserts.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>