mgr/dashboard: Add support for managing individual OSD settings (backend)

Add options to mark OSDs in/out/down/reweight/lost/remove/destroy/create

Fixes: http://tracker.ceph.com/issues/24270
Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

Merge pull request #23654 from cbodley/wip-crypto-assert-always

crypto: use ceph_assert_always for assertions

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #22668 into master

* refs/pull/22668/head:
mds: reorganize and curate perf log counters
mds: show session load average in dump session
mds: allow request load average decay rate configurable via admin socket
mds: decay counter for tracking request load average
mds: add perf counters to track current open/stale sessions

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23438 into master

* refs/pull/23438/head:
client: fix choose_target_mds for requests that do name lookup

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23467 into master

* refs/pull/23467/head:
cephfs-journal-tool: enable purge_queue journal's event commands

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #23529 into master

* refs/pull/23529/head:
mon: fix idempotent FS creation

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: João Eduardo Luís <joao@suse.de>

Merge PR #23543 into master

* refs/pull/23543/head:
tools/cephfs: make cephfs-data-scan prints the max used ino

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23584 into master

* refs/pull/23584/head:
mds: don't modify filepath when printing
mds: hold slave request refernce when dumping MDRequestImpl

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23589 into master

* refs/pull/23589/head:
mds: more description for failed authpin
mds: cleanup CDir freezing/frozen tree check

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23315 from Devp00l/wip-clean-notification-messages

mgr/dashboard: Cleaner notifications

Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #23637 from noahdesu/qa-mgr-mod-path

qa: fix manager module paths

crypto: use ceph_assert_always for assertions

use ceph_assert_always() to enforce this comment in ceph_crypto.h:

This assert MUST NOT be compiled out, even on non-debug builds.

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23639 from trociny/wip-24910

librbd: potential race on image create request complete

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #23110 from Songweibin/wip-rbd-snap-rollback

librbd: use object map when doing snap rollback

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #23609 from ErwanAliasr1/evelu-ccache

run-make-check: Showing configuration before the build

Merge pull request #23601 from tchaikov/wip-pybind-freebsd-clang

pybind: do not check MFLAGS

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #23593 from jtlayton/wip-jlayton-26948

librados: add a rados_omap_iter_size function

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23588 from gmayyyha/ceph-df-json

mon/PGMap: command 'ceph df -f json' output add total_percent_used

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23563 from yaozongyou/mon-ping-timeout

mon/MonClient: use mon_client_ping_timeout during ping_monitor

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10305 from majianpeng/memdb-perfcounter

kv/MemDB: add perfcounter.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23628 from tchaikov/wip-cmake-gtest-cleanup

cmake: link against gtest in a better way

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

librbd: potential race on image create request complete

Fixes: http://tracker.ceph.com/issues/24910
Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge PR #23616 into master

* refs/pull/23616/head:
doc/releases/schedule: note Jewel EOL date

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #23618 into master

* refs/pull/23618/head:
doc: updated infernalis release reference link
doc: added ref label for infernalis release notes
doc: used reST label for ceph-deploy reference
doc: added reference label for ceph-deploy page
doc: updated cross referencing locations link

Reviewed-by: Sage Weil <sage@redhat.com>

test: Fix test to use -gt instead of creating an empty file "0"

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge PR #23636 into master

* refs/pull/23636/head:
messages: fix get_opname return type

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23620 from tchaikov/wip-dpdk-cmake

cmake: add dpdk::dpdk if dpdk is built or found

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/ceph_objectstore_tool.py: set mgr module path

Signed-off-by: Noah Watkins <nwatkins@redhat.com>

qa/standalone/ceph-helpers.sh: fix mgr module path

callers of get_python_path were not passing in a $1 parameter, so
ceph_lib was an empty string resulting in an invalid path to the built
cython modules. assume this is called from the `lib` parent directory.

pass path to the manager modules when starting ceph-mgr.

Signed-off-by: Noah Watkins <nwatkins@redhat.com>

messages: fix get_opname return type

It cannot be a constexpr.

Introduce-by: 00b867fae2e8e912119fc47348dee78631c6f66b / #23623
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23574 from cbodley/wip-26895

rgw multisite: bucket sync transitions back to StateInit on OP_SYNCSTOP

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge PR #23575 into master

* refs/pull/23575/head:
tools/cephfs-shell:pep8 code clean
tools/cephfs-shell:resolved removing files using pattern inside sub-directories
tools/cephfs-shell:changed output representation for the commands cwd and ls
tools/cephfs-shell:resolved removing directories using pattern inside sub-directories

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #18096 into master

* refs/pull/18096/head:
os/bluestore: factor VDO into bluestore's 'allocated' value
os/filestore: estimate omap_allocated
os/ObjectMap: add get_db() accessor
osd/OSDMap: include data, omap in 'osd df' output
os/bluestore: report omap and internal metadata usage
os/bluestore/BlueFS: drop get_fs_usage(); add get_used()
kv: add per-prefix space estimation
osd: decouple statfs update from hb peers, pg count
osd: drop useless OSDService::update_osd_stat()
osd/osd_types: separate accounting for data, omap, and metadata
osd: re-add get_num_pgs()

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

tools/cephfs-shell:pep8 code clean

Signed-off-by: Pavani Rajula <rpavani1998@gmail.com>

tools/cephfs-shell:resolved removing files using pattern inside sub-directories

Signed-off-by: Pavani Rajula <rpavani1998@gmail.com>

tools/cephfs-shell:changed output representation for the commands cwd and ls

Signed-off-by: Pavani Rajula <rpavani1998@gmail.com>

tools/cephfs-shell:resolved removing directories using pattern inside sub-directories

Signed-off-by: Pavani Rajula <rpavani1998@gmail.com>

Merge pull request #23623 from tchaikov/wip-inline-constants

messages: define HEAD_VERSION and COMPAT_VERSION inlined

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

messages: define HEAD_VERSION and COMPAT_VERSION inlined

otherwise we could have ../lib/libceph-common.so.0: undefined reference
to `MOSDPGNotify::HEAD_VERSION

when the compiler tries to be smart and pass them by reference.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23419 from tspmelo/fix-26856

mgr/dashboard: Add decorator to skip parameter encoding

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #23446 from tspmelo/wip-25075

mgr/dashboard: Use human readable units on the sparkline graphs

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #23388 from ricardoasmarques/wip-fix-auto-logout

mgr/dashboard: Fix redirect to login page on session lost

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

run-make-check: Showing configuration before the build

The actual code is resetting the statistics before doing the actual compilation and prints them after the build.

That is nice to understand how much the cache was used but doesn't help understanding how much it _could_ have been used.

This patch is adding a reporting (-s) when cleaning the statistics so we can estimate :
- the actual number of files in cache
- the actual size of the cache

With this two missing information, its now possible estimate if there is some miss-usage of the cache.

Signed-off-by: Erwan Velu <erwan@redhat.com>

cmake: link against gtest in a better way

* add FindGMock.cmake which allows user to use the libgtest-dev
  shipped by distro
* add GMock::{GMock,Main}, GTest::{GTest,Main} targets to be
  compatible with FindGTest.cmake and FindGMock.cmake, which
  expose the built libraries with properties adhered to
  them. so the consumer of them can import them in a better way.
* update tests to drop the commands like
  set_target_properties(foo PROPERTIES COMPILE_FLAGS ${UNITTEST_CXX_FLAGS}),
  as they are already linked against gmock and gtest.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: fix Finddpdk.cmake

* add dpdk::dpdk if dpdk is built or found, as seastar checks for
  it before adding its component libraries. if user installs libdpdk
  and builds WITH_SEASTAR=ON or WITH_DPDK=ON, cmake fails to configure
  the building system without this fix.
* add dpdk::cflags target for populating the -march=<arch> compile
  option.
* also use pkg-config for finding dpdk.
* link common_async_dpdk against dpdk::dpdk instead

please note, the reason why we can remove the "-march=native" compile
option from Finddpdk.cmake, is that the distro shipped header files
are "generic" in the sense of "-march=<ARCH>", they do not enable
the arch specific intrinsic by default, hence the source files
including them do not need specific compiler options.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not build DPDK with -march=native

-march=native causes gcc to use opcodes according to whichever CPU happens to
be installed in the build host, which can be different for every build. This
makes it impossible to achieve a reproducible build.

Also, if the build host has a very new CPU, running the resulting binaries on
older CPUs (of the same family, i.e. x86_64) could result in segmentation
fault.

References: http://tracker.ceph.com/issues/24948
Signed-off-by: Nathan Cutler <ncutler@suse.com>

cmake: honour CMAKE_REQUIRED_LIBRARIES in CheckIncludeFiles

to silence cmake warning regarding to CMP0075

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: updated infernalis release reference link

Updated Infernalis reference link with reST label.

Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>

doc: added ref label for infernalis release notes

Added reference label for Infernalis release notes.

Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>

doc: used reST label for ceph-deploy reference

Updated ceph-deploy man page link with ref label. Fixes
404 error.

Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>

doc: added reference label for ceph-deploy page

Added reST label for ceph-deploy page.

Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>

doc: updated cross referencing locations link

Updated cross referencing locations link for doc/start/documenting-ceph.rst

Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>

mds: don't modify filepath when printing

filepath::depth() may call parse_bits(), which modifies mutable member
'bits'. dump_ops_in_flight asok command prints filepath without holding
mds_lock. So multiple threads may call parse_bits() at the same time.

Fixes: http://tracker.ceph.com/issues/26894
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: hold slave request refernce when dumping MDRequestImpl

dump_ops_in_flight asok command dumps MDRequestImpl without holding
mds_lock. MDS may free MDRequestImpl::slave_request in the middle of
dumping MDRequestImpl.

Fixes: http://tracker.ceph.com/issues/26894
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge PR #23468 into master

* refs/pull/23468/head:
Allow to unittest shell using transcript

Merge PR #22555 into master

* refs/pull/22555/head:
msg: define MFoo::create helper
msg: add msgref cast method
msg: cleanup factory/ref definition in messages
mds: use message factory to avoid leaks
msg: add factory method to correctly construct smart_ptr
mds: remove dead MDS-MDS forwarding code
mds: manage Message lifetime with intrusive_ptr
common: add templated Context factory
msg: add const version of get_payload
msg: use queue of messages for dispatch
msg: dispatch intrusive_ptr Messages
msg: use deque for dispatcher pointers

Reviewed-by: Zheng Yan <zyan@redhat.com>

doc/releases/schedule: note Jewel EOL date

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge PR #23551 into master

* refs/pull/23551/head:
qa: use recent kernel to kernel build testing

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23608 from tchaikov/wip-librbd-test

test/librbd: fix -Wsign-compare warnings

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa: use recent kernel to kernel build testing

Fixes: http://tracker.ceph.com/issues/24679
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23597 into master

* refs/pull/23597/head:
qa: add cephfs-shell skeleton test case

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa: add cephfs-shell skeleton test case

Right now just tests that "help" works.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #21286 from joke-lee/wip-auth-uri-encode-decode

rgw:  should recode  canonical_uri when caculate s3 v4 auth

Merge pull request #23533 from wjwithagen/wjw-fix-random_shuffle

osd: make random shuffle comply with C++17

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #23547 from yaozongyou/mon-get-rank

mon/MonMap: remove duplicate code in get_rank

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.de>

test/librbd: fix -Wsign-compare warnings

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Rename to taskMessageService

Renames taskManagerMessageService to taskMessageService.

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Add decorator to skip parameter encoding

By enconding all parameters of api services we were also encoding parameters
that were being sent in the body of the request.
Those parameters don't need to be enconded and the server never decodes them.

With this new decorator you can specify if you don't want a parameter to be
enconded.

This is a regression introduced in f21d0da5a3e19e3f9bbde084e71b4a09f8dcb0a1.

Fixes: http://tracker.ceph.com/issues/26856
Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Cleaner notifications

Now background tasks and recent notifications won't differ in their
wording anymore and all notifications have the same style.

Fixes: https://tracker.ceph.com/issues/24460
Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Use human readable units on the sparkline graphs

Fixes: http://tracker.ceph.com/issues/25075
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #23445 from ricardoasmarques/wip-rbd-actions-disable

mgr/dashboard: Disable RBD actions during task execution

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #23485 from javacruft/misc-32bit-fixes

build: 32 bit architecture fixes

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23581 from JiYou/fix.examples.Makefile

examples/Makefile: add -Wno-unused-parameter to avoid compile error

Reviewed-by: Kefu Chai <kchai@redhat.com>

Allow to unittest shell using transcript

Signed-off-by: Pavani Rajula <rpavani1998@gmail.com>

Merge pull request #23600 from wjwithagen/wjw-fix-freebsd-manual

doc: fix/cleanup freebsd osd disk creation

Reviewed-by: Kefu Chai <kchai@redhat.com>

pybind: do not check MFLAGS

clang does not check for directories passed by -iquote/path/to/foo.h, if
a header is included using `#include <foo.h>`

it complains:

/home/kefu/ceph/src/pybind/rados/tmpwzjOsS/rados_dummy.c:2:10: error:
'rados/librados.h' file not found with <angled> include; use "quotes"
instead
         ^~~~~~~~~~~~~~~~~~
         "rados/librados.h"
1 error generated.

and MFLAGS does not exist in the env variables of setup.py launched by
the top-level make.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: fix/cleanup osd disk creation

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

msg: define MFoo::create helper

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: reorganize and curate perf log counters

Categorize perf counters into high (PRIO_{CRITICAL/INTERESTING}),
medium (PRIO_USEFUL) and low priority (PRIO_DEBUGONLY). High and
medium priority stats are transmitted to ceph mgr and included in
prometheus output.

Signed-off-by: Venky Shankar <vshankar@redhat.com>
Fixes: http://tracker.ceph.com/issues/24004

mds: show session load average in dump session

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mds: allow request load average decay rate configurable via admin socket

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mds: decay counter for tracking request load average

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mds: add perf counters to track current open/stale sessions

Signed-off-by: Venky Shankar <vshankar@redhat.com>

examples/Makefile: add -Wno-unused-parameter to avoid compile error

Signed-off-by: You Ji <youji@ebay.com>

os/bluestore: factor VDO into bluestore's 'allocated' value

If we are on VDO, report the *raw* storage we've allocated, as reported
by VDO.

Signed-off-by: Sage Weil <sage@redhat.com>

os/filestore: estimate omap_allocated

Assume all of leveldb/rocksdb is omap.  This is an overestimate, but
better than nothing.

We don't populate the metadata overhead (no easy way to calculate this
that comes to mind).  And we don't populate the compression-related
fields.  It's possible we could make something up here in the VDO
case...

Signed-off-by: Sage Weil <sage@redhat.com>

os/ObjectMap: add get_db() accessor

This is just to let us get at the underlying KeyValueDB for DBObjectMap.
It is not really better or worse than adding accessors for things like
GetEstimatedSize() to ObjectMap.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMap: include data, omap in 'osd df' output

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: report omap and internal metadata usage

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: drop get_fs_usage(); add get_used()

Signed-off-by: Sage Weil <sage@redhat.com>

kv: add per-prefix space estimation

Signed-off-by: Sage Weil <sage@redhat.com>

osd: decouple statfs update from hb peers, pg count

These don't need to be updated in synchrony.  (In fact, the statfs update
could be much more infrequent.)

Signed-off-by: Sage Weil <sage@redhat.com>

osd: drop useless OSDService::update_osd_stat()

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: separate accounting for data, omap, and metadata

Signed-off-by: Sage Weil <sage@redhat.com>

osd: re-add get_num_pgs()

We removed this but it's a useful accessor.

Signed-off-by: Sage Weil <sage@redhat.com>

msg: add msgref cast method

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

msg: cleanup factory/ref definition in messages

This eliminates duplicate code definitions.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: use message factory to avoid leaks

Fixes: http://tracker.ceph.com/issues/24306
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

msg: add factory method to correctly construct smart_ptr

This is to avoid easy leaks of the form:

    boost::intrusive_ptr(new M*(...));

where a ref is leaked because constructing the intrusive_ptr adds a ref.

It is expected that eventually all constructors move to protected/private so
that the message's factory is the only way to build a new message (safely).

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: remove dead MDS-MDS forwarding code

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>