osd/OSDMap: add osd status to utilization dumper

It's mildly irritating that 'osd df' doesn't contain the
information whether an OSD is up/down/destroyed.

Fixes: http://tracker.ceph.com/issues/35544
Signed-off-by: Paul Emmerich <paul.emmerich@croit.io>

Merge pull request #23827 from trociny/wip-mock-valgrind

 test/librbd: fix valgrind warnings

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #23898 from tchaikov/wip-crimson-net-cleanup

crimson/net: s/repeat/keep_doing/

Reviewed-by: Casey Bodley <cbodley@redhat.com>

crimson/net: s/repeat/keep_doing/

repeat ~= while() {}
keep_doing ~= for (;;) {}

so it's simpler this way.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23842 from votdev/issue_34315

mgr/dashboard: Display RGW user/bucket quota max size in human readable form

Reviewed-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #23905 from idryomov/wip-cram-suite-repo

qa/tasks/cram: use suite_repo repository for all cram jobs

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #23491 from p-na/per-osd-settings

mgr/dashboard: Add support for managing individual OSD settings in the backend

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #23735 from votdev/suggest_snapshot_name

mgr/dashboard: Auto-create a name for RBD image snapshots

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #23627 from Devp00l/wip-hints-for-jest-failures

mgr/dashboard: Add hints to resolve unit test failures

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #23903 from libingyang-zte/master

doc: Fix Spelling Error of Rados Operations

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

doc: Fix Spelling Error of Rados Operations

Signed-off-by: Li Bingyang <li.bingyang1@zte.com.cn>

Merge pull request #23900 from libingyang-zte/master

doc: Fix Spelling Error of Rados Operations

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #23899 from luomuyao/cleanup

rgw: drop unused tmp in main()

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

doc: Fix Spelling Error of Rados Operations

Signed-off-by: Li Bingyang <li.bingyang1@zte.com.cn>

rgw: drop unused tmp in main()

Signed-off-by: luomuyao <luo.muyao@zte.com.cn>

qa/tasks/cram: use suite_repo repository for all cram jobs

Currently git.ceph.com is hardcoded for all cram jobs.  Testing
modifications is a pain: one needs to push to either ceph/ceph.git or
ceph/ceph-ci.git (depending on where the ceph branch is at, triggering
unnecessary builds in the latter case) and wait for the mirror to sync.
Runs scheduled against branches in developer's forks fail.

Move away from git.ceph.com to allow mixing branches and repositories,
similar to workunits.

Fixes: https://tracker.ceph.com/issues/27211
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #23798 from liewegas/wip-pod-hostname

common/util: pass real hostname when running in kubernetes/rook container

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23322 from ricardoasmarques/wip-role-management-api

mgr/dashboard: Add REST API for role management

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Add REST API for role management

Fixes: https://tracker.ceph.com/issues/25138
Signed-off-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: Add support for managing individual OSD settings (backend)

Add options to mark OSDs in/out/down/reweight/lost/remove/destroy/create

Fixes: http://tracker.ceph.com/issues/24270
Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

Auto-create a name for RBD image snapshots.

Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #23891 from libingyang-zte/master

doc: Fix Spelling Error of Rados Operations

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

mgr/dashboard: Display RGW user/bucket quota max size in human readable form

Fixes: https://tracker.ceph.com/issues/34315
Signed-off-by: Volker Theile <vtheile@suse.com>

doc: Fix Spelling Error of Rados Operations

Signed-off-by: Li Bingyang <li.bingyang1@zte.com.cn>

Merge pull request #23879 from dwahler/master

doc: add note about LVM volumes to ceph-deploy quick start

Reviewed-by: Alfredo Deza <adeza@redhat.com>

doc: add note about LVM volumes to ceph-deploy quick start

This commit also fixes some incorrect indentation that was causing
previous steps in the same section to be incorrectly numbered.

Signed-off-by: David Wahler <dwahler@gmail.com>

unittest_config: $host expands to ceph_get_short_hostname(), not hostname -s

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #23540 into master

* refs/pull/23540/head:
include/ceph_fs: rename old auid field
PendingReleaseNotes: note about auid support removal
radosgw-admin: remove -a --auth-uid arg
rgw: remove auid member from RGWUserInfo
auth: remove auid member from EntityAuth
osd: remove auid session member
mon: remove auid session member
doc/dev/cephx_protocol: drop auid reference
auth: remove auid args from handle_request and verify_authorizer
mon/OSDMonitor: remove 'osd pool {get,set} <name> auid ...'
mon/OSDMonitor: remove auid arg for 'osd lspools' and deprecate
osd/OSDCap: remove auid from grammar
osd/OSDCap: remove auid from is_capable() etc args
auth: clean up cap parse error messages
mon/AuthMonitor: raise health warning on invalid caps
mon/AuthMonitor: drop ancient auth inc encoding compat
messages/MPoolOp: drop auid member
osdc/Objecter: drop change_pool_auid
pybind/rados: drop auid arg to pool_create
pybind/rados: drop change_auid
rados: drop mkpool, rmpool commands
rados: remove 'chown' command
librados: deprecate calls that take auid
librados: mark all auid calls deprecated
mon/OSDMonitor: drop variable pool auid for prepare_new_pool
mon/OSDMonitor: remove pool auid change support
osdc/Objecter: do not pass auid to create_pool
ceph-authtool: remove auid options
qa/workunits/cephtool: remove auid tests

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

Merge PR #23739 into master

* refs/pull/23739/head:
mgr/telemetry: add crashdump info to report

Reviewed-by: Wido den Hollander <wido@42on.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #23766 into master

* refs/pull/23766/head:
BlueStore: Fix concurrency conflicts

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23841 from idryomov/wip-krbd-namespaces

krbd: support for images within namespaces

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #23663 from xiexingguo/wip-incompat-async-fixes

osd: some recovery improvements and cleanups

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23837 from rjfd/wip-update-dmclock

dmclock: update dmclock submodule sha1 to tip of ceph/dmclock.git master

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

include/ceph_fs: rename old auid field

Signed-off-by: Sage Weil <sage@redhat.com>

PendingReleaseNotes: note about auid support removal

Signed-off-by: Sage Weil <sage@redhat.com>

radosgw-admin: remove -a --auth-uid arg

It was already a no-op.

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: remove auid member from RGWUserInfo

Unused.

Signed-off-by: Sage Weil <sage@redhat.com>

auth: remove auid member from EntityAuth

Note that for the keyring we still accept (but ignore) auid lines so that
we can parse old keyrings that have them.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: remove auid session member

Signed-off-by: Sage Weil <sage@redhat.com>

mon: remove auid session member

Signed-off-by: Sage Weil <sage@redhat.com>

doc/dev/cephx_protocol: drop auid reference

Signed-off-by: Sage Weil <sage@redhat.com>

auth: remove auid args from handle_request and verify_authorizer

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: remove 'osd pool {get,set} <name> auid ...'

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: remove auid arg for 'osd lspools' and deprecate

Use 'osd pool ls' instead.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDCap: remove auid from grammar

This is an incompatible change: old caps that include auid will no longer
parse.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDCap: remove auid from is_capable() etc args

Signed-off-by: Sage Weil <sage@redhat.com>

auth: clean up cap parse error messages

Consistent formatted.  Drop newline.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/AuthMonitor: raise health warning on invalid caps

Raise a health warning if we have invalid (unparsable) caps in the auth
database.  Include a simple test.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #23840 from idryomov/wip-krbd-untar-build-packages

qa: rbd_workunit_kernel_untar_build: install build dependencies

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #22983 from trociny/wip-journal-limit-inflight-appends

librbd: optionally limit journal in-flight appends

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #23770 from Songweibin/wip-rbd-namespace

rbd: correct parameter of namespace and verify it before set_namespace

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

krbd: support for images within namespaces

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

krbd: create udev_enumerate in enumerate_devices()

Make it easier to run more than one scan in a row.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

krbd: introduce krbd_spec

Don't substitute "@-" for HEAD when printing the spec.  Instead, omit
the snapshot part.  The same would be done for the namespace part.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

krbd: change krbd_unmap_by_spec() to take "" instead of NULL

krbd_map() and krbd_is_mapped() take "", krbd_unmap_by_spec() is the
odd one out.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

krbd: remove unused include

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

mon/AuthMonitor: drop ancient auth inc encoding compat

This is for *ancient* mons (I'm guessing bobtail?).

Signed-off-by: Sage Weil <sage@redhat.com>

messages/MPoolOp: drop auid member

Signed-off-by: Sage Weil <sage@redhat.com>

osdc/Objecter: drop change_pool_auid

Signed-off-by: Sage Weil <sage@redhat.com>

pybind/rados: drop auid arg to pool_create

Signed-off-by: Sage Weil <sage@redhat.com>

pybind/rados: drop change_auid

Signed-off-by: Sage Weil <sage@redhat.com>

rados: drop mkpool, rmpool commands

- mkpool and rmpool users should use the normal cli/mon commands

Signed-off-by: Sage Weil <sage@redhat.com>

rados: remove 'chown' command

Signed-off-by: Sage Weil <sage@redhat.com>

librados: deprecate calls that take auid

Return EOPNOTSUPP for auid-specific calls, or EINVAL for ones that get a non-default
auid value.

Signed-off-by: Sage Weil <sage@redhat.com>

librados: mark all auid calls deprecated

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: drop variable pool auid for prepare_new_pool

Always set to 0 for now.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: remove pool auid change support

EOPNOTSUPP

Signed-off-by: Sage Weil <sage@redhat.com>

osdc/Objecter: do not pass auid to create_pool

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-authtool: remove auid options

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/cephtool: remove auid tests

Signed-off-by: Sage Weil <sage@redhat.com>

qa: rbd_workunit_kernel_untar_build: install build dependencies

Commit f0fe0936e64d ("qa: use recent kernel to kernel build testing")
bumped the kernel to 4.17.

Fixes: http://tracker.ceph.com/issues/35074
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #23838 from wjwithagen/wjw-fix-ConfUtil

common: Reverse deleted include

Reviewed-by: Kefu Chai <kchai@redhat.com>

common: Reverse deleted include

Reversing 1 delete from
https://github.com/ceph/ceph/commit/09d3f546b32cdf126a1246899a0b0a3eb25282d8

Clang trips over this:
home/jenkins/workspace/ceph-master/src/common/ConfUtils.cc:94:19: error: implicit instantiation of undefined template 'std::__1::basic_ostringstream<char, std::
__1::char_traits<char>, std::__1::allocator<char> >'
    ostringstream oss;
                  ^
/usr/include/c++/v1/iosfwd:123:32: note: template is declared here
    class _LIBCPP_TEMPLATE_VIS basic_ostringstream;
                               ^

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

dmclock: update dmclock submodule sha1 to tip of ceph/dmclock.git master

This update will fix the build of Ceph when -DWITH_TESTS=OFF is used.

Signed-off-by: Ricardo Dias <rdias@suse.com>

Merge pull request #23802 from votdev/unique_validator

mgr/dashboard: Add unique validator

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Add hints to resolve unit test failures

Signed-off-by: Stephan Müller <smueller@suse.com>

Merge pull request #23629 from wangxiaoguang/fix_deep_scrub

os/bluestore: fix deep-scrub operation againest disk silent errors

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #23555 from tspmelo/wip-task-refactor

mgr/dashboard: Extract/Refactor Task merge

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

osd/PG: force auth_log_shard to be primary when appropriate

So if there are a lot fo missing objects on primary, we can
make use of auth_log_shard to restore client I/O quickly.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: fix deep-scrub operation againest disk silent errors

Say a object who has data caches, but in a while later, caches' underlying
physical device has silent disk erros accidentally, then caches and physical
data are not same. In such case, deep-scrub operation still tries to read
caches firstly and won't do crc checksum, then deep-scrub won't find such
data corruptions timely.

Here introduce a new flag 'CEPH_OSD_OP_FLAG_BYPASS_CLEAN_CACHE' which tells
deep-scrub to bypass object caches. Note that we only bypass cache who is in
STATE_CLEAN state. For STATE_WRITING caches, currently they are not written
to physical device, so deep-scrub operation can not read physical device and
can read these dirty caches safely. Once they are in STATE_CLEAN state(or not
added to bluestore cache), next round deep-scurb can check them correctly.

As to above discussions, I refactor BlueStore::BufferSpace::read sightly,
adding a new 'flags' argument, whose value will be 0 or:
     enum {
       BYPASS_CLEAN_CACHE = 0x1,     // bypass clean cache
     };

flags 0: normal read, do not bypass clean or dirty cache
flags BYPASS_CLEAN_CACHE: bypass clean cache, currently only for deep-scrube
                        operation

Test:
   I deliberately corrupt a object with cache, with this patch, deep-scrub
   can find data error very timely.

Signed-off-by: Xiaoguang Wang <xiaoguang.wang@easystack.cn>

osd/PG: make num_objects_missing can trace missing objects correctly

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/osd_types: kill overlaps_with() of pg_info_t

Which has no consumers.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/PG: clear calc_replicated_acting a bit

- kill usable, use want->size instead
- introduce a (separate) lambda function for sorting

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/PG: move comments to the proper place

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/PG: find_best_info - add completeness as the preferred option

Async recovery peers usually have a relative complete
log history but may exist a lot of missing objects. Choosing them
as auth_log_shard and further as primary if current up_primary is
unrecoverable, say, could have a bigger chance to block client I/Os.

Among peers with identical new log history, we now consider those
who are now complete (having no missing objects) as the preferred
ones when determining auth_log_shard.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

test/librbd: fix ioctx reference leak in DetachChildRequest mock tests

Signed-off-by: Mykola Golub <mgolub@suse.com>

test/librbd: fix "uninitialised value" valgrind warnings

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #23546 from xiexingguo/wip-complete-to

osd/PGLog: trim - avoid dereferencing invalid iter

Reviewed-by: Yan Jun <yan.jun8@zte.com.cn>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #23800 from rjfd/wip-fix-crash-mgr-module

mgr/crash: fix python3 invalid syntax problems

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Noah Watkins <nwatkins@redhat.com>

Merge pull request #23803 from tchaikov/wip-ceph_assert

include/buffer.h: do not use ceph_assert() unless __CEPH__ is defined

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #23820 from LenzGr/nautilus-404-page

mgr/dashboard: Updated image on 404 page

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Extract/Refactor Task merge

Signed-off-by: Stephan Müller <smueller@suse.com>
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #23785 from alfredodeza/wip-rm34309

ceph-volume:  update batch documentation to explain filestore strategies

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

librbd: option to limit in-flight appends

Signed-off-by: Mykola Golub <mgolub@suse.com>

journal: limit in-flight appends

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge PR #23509 into master

* refs/pull/23509/head:
OSD: add recovery bytes counter to osd logger

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #23572 into master

* refs/pull/23572/head:
qa/standalone/osd/osd-force-create-pg: add force-create-pg test
mon/MonCommands: fix 'osd force-create-pg'

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #23611 into master

* refs/pull/23611/head:
mgr/devicehealth: fix exceptions
doc/mgr/devicehealth: update docs
mgr/devicehealth: leave health monitoring off by default
mgr/devicehealth: add 'device monitoring {on,off}' commands
mgr/devicehealth: 10m wakeup interval is sufficient
mgr/devicehealth: scrape at scrape_interval intervals
mgr/devicehealth: UTC everywhere
mgr/devicehealth: fix sleep_interval configurable

Merge PR #23633 into master

* refs/pull/23633/head:
osd/PrimaryLogPG.cc: limit trimming at can_rollback_to

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mon/MonClient: set hostname in MMonSubscribe message

Signed-off-by: Sage Weil <sage@redhat.com>