Merge pull request #21571 from tchaikov/wip-23517

cmake: hide symbols import from other libraries in libcls_*

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #21573 from tchaikov/wip-cmake

cmake: s/boost_256/boost_sha256/

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #21570 from alfredodeza/wip-cv-doc-updates

ceph-volume update man page for prepare/activate flags

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

cmake: s/boost_256/boost_sha256/

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21484 from tchaikov/wip-cmake-common-cleanup

cmake, common: silence cmake and gcc warnings

Reviewed-by: Casey Bodley <cbodley@redhat.com>

cmake: disable gflags support in rocksdb

to silence the warnings like

CMake Warning at CMakeLists.txt:73 (find_package):
  By not providing "Findgflags.cmake" in CMAKE_MODULE_PATH this project
has
  asked CMake to find a package configuration file provided by "gflags",
but
  CMake did not find one.

  Could not find a package configuration file provided by "gflags" with
any
  of the following names:

    gflagsConfig.cmake
    gflags-config.cmake

  Add the installation prefix of "gflags" to CMAKE_PREFIX_PATH or set
  "gflags_DIR" to a directory containing one of the above files.  If
"gflags"
  provides a separate development package or SDK, be sure it has been
  installed.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake,make-dist: use sha256 instead of md5 for checksum

the boost's download page offers the SHA256 hash, so it'd be easier to
verify the hash this way.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: hide symbols import from other libraries in libcls_*

so they will not be involved when resolving symbols. ld tries to
keep a shared library around even if it fails to load it if it offers
some unique symbols. in that case, the library will not be properly
unloaded, and even worse it will interfere with following dlopen()
calls, because it is marked with NODELETE by dlopen(). if it has some
unresolved symbol and does offer some "unique" symbols required by
the library to be loaded, the library will fail to load, despite the
fact that the "unique" symbol is also offered by the executable.

for more details, see
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=60731 and
https://sourceware.org/bugzilla/show_bug.cgi?id=14577

Fixes: http://tracker.ceph.com/issues/23517
Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/man/ceph-volume update flags for activate

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/man/ceph-volume update flags for prepare

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #21516 from rjfd/wip-dashboard-rest-improvements

mgr/dashboard: RESTController improvements

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #21353 from Devp00l/pool-list

mgr/dashboard: Pool listing

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #21563 from idryomov/wip-krbd-lock-timeout-fixup

doc: lock_timeout is a per mapping option

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc: lock_timeout is a per mapping option

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #21522 from idryomov/wip-krbd-lock-timeout-test

qa: krbd_exclusive_option.sh: add lock_timeout test case

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #21513 from idryomov/wip-krbd-notrim-test

 qa: krbd_fallocate.sh: add notrim test case

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: Add pool listing

Adds a new top level menu entry, called "Pool", which lists all pools
and contains a minimalistic detail view.

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: K-v-table uses autoReload input

The "autoReload" input from the data table can now be used in the key
value table as well.

Signed-off-by: Stephan Müller <smueller@suse.com>

Merge pull request #21517 from rjfd/wip-dashboard-fix-rbd-auth

mgr/dashboard: rbd: add @AuthRequired to snapshots controller

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #20869 from votdev/dashboard_rgw_ro

mgr/dashboard: Add RGW user and bucket lists (read-only)

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #21386 from votdev/api_interceptor

mgr/dashboard: Rename and refactor ApiInterceptorService class

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #21531 from LenzGr/mgr-dashboard-documentation

doc: updated the dashboard feature list

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

doc: updated the dashboard feature list

Updated and improved the Mgr Dashboard feature list: added RBD management
features.

Signed-off-by: Lenz Grimmer <lgrimmer@suse.com>

Merge PR #21499 into master

* refs/pull/21499/head:
doc/cephfs: update kernel client quotas support info

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

githubmap: update contributors

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #21472 into master

* refs/pull/21472/head:
mds: optimize purge queue expire pos update

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #21434 into master

* refs/pull/21434/head:
client: add client option descriptions

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #21498 into master

* refs/pull/21498/head:
client: flush the mdlog in _fsync before waiting on unstable reqs

Reviewed-by: dongdong tao <tdd21151186@gmail.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #21530 from jan--f/mgr-prometheus-fix-osd-metadata

pybind/mgr/prometheus: fix creation of osd_metadata metric

Reviewed-by: Boris Ranto <branto@redhat.com>

pybind/mgr/prometheus: fix creation of osd_metadata metric

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #21446 from smithfarm/wip-ctest-py3-fallback

cmake/mgr: use Python 3 virtualenv if mgr subinterpreter is Python 3

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21477 from liewegas/wip-kvstore-tool

ceph-kvstore-tool: make it a bit more friendly

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21488 from tchaikov/wip-i-choose-to-die-instead-of-leak

msg/async/EventKqueue: assert on OOM

Reviewed-by: Haomai Wang <haomai@xsky.com>

mgr/dashboard: Add ability to cancel notifications.

Signed-off-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Do not show notifications for HTTP 409 errors.

Signed-off-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Rename auth-interceptor and refactor it to display notifications for more errors than 401 and 500.

Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #21496 from ceph/wip-rm23678

ceph-volume: allow skipping systemd interactions on activate/create

Reviewed-by: Andrew Schoen <aschoen@redhat.com

mgr/dashboard: Modify RGW proxy response of errors to display a detailed message in the error notification.

Signed-off-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Force datatable to hide the loading indicator in case of an error.

Signed-off-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Add RGW user and bucket lists (read-only)

Signed-off-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: rbd: add @AuthRequired to snapshots controller

Signed-off-by: Ricardo Dias <rdias@suse.com>

Merge pull request #21466 from jcsp/wip-dashboard-cmake-all

cmake: include frontend build in 'all' target

Reviewed-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: restcontroller: removed @args_from_json

Signed-off-by: Ricardo Dias <rdias@suse.com>

Merge pull request #21497 from votdev/forceidentifier_datatable

mgr/dashboard: Add 'forceIdentifier' attribute to datatable

Reviewed-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #21469 from liewegas/wip-log-nomsgs

mon/LogMonitor: do not crash on log sub w/ no messages

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #21467 from tchaikov/wip-perf-stats-ns-incarnation

include/ceph_features: fix OS_PERF_STAT_NS's incarnation

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #21459 from runsisi/wip-mgrc-register

mgr/MgrClient: service registration filtered by service name instead of daemon name

Reviewed-by: John Spray <john.spray@redhat.com>

qa: krbd_exclusive_option.sh: add lock_timeout test case

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa: krbd_exclusive_option.sh: be more lax about ps states

Allow running this script by hand -- foreground process group vs
background process groups, etc.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #21344 from yangdongsheng/lock_timeout

rbd: support lock_timeout in rbd mapping

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

mgr/dashboard: .gitignore: ignore vscode .venv dir

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: restcontroller: support explicit resource ID

In the current implementation, RESTController tries to infer the
resource ID by looking at the declared parameters of the resource
methods (get, delete, set, and patch). This inference may not always
be correct, and therefore with this commit we can now explicitely
specify the structure of the resource ID. If the resource ID is
not explicitly specified, then we fallback to the inference algorithm.

Signed-off-by: Ricardo Dias <rdias@suse.com>

cmake/mgr: use Python 3 virtualenv if mgr subinterpreter is Python 3

Signed-off-by: Nathan Cutler <ncutler@suse.com>

mgr/dashboard: Force identifier for RBD datatable.

Signed-off-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Add 'forceIdentified' attribute to datatable

Signed-off-by: Volker Theile <vtheile@suse.com>

qa: krbd_fallocate.sh: add notrim test case

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa: krbd_fallocate.sh: s/deallocated/zeroes/

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa: krbd_fallocate.sh: parameterize assert_deallocated()

Take num_objects_expected, similar to assert_deallocated_unaligned().

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

rbd: support lock_timeout in rbd mapping

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

Merge pull request #21056 from hitoshikamei/rbdmap-notrim

rbd: add notrim option to rbd map

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #21490 from tchaikov/wip-tox

mgr/dashboard: create venv for running tox

Reviewed-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: create venv for running tox

the tox shipped with centos is 1.4.2, and is way too old. it does not
allow us to define the dependency using generative-envlist.

Signed-off-by: Kefu Chai <kchai@redhat.com>

tools/setup-virtualenv.sh: bump up the version of tox to 2.9.1

2.8.1 brings an important fix, see
https://github.com/tox-dev/tox/issues/515.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21489 from thmour/thmour_parallel_create

ceph-volume: allow parallel creates

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge PR #16608 into master

* refs/pull/16608/head:
qa: whitelist mds down wrn during cephfs testing
mds: add config to disable fragmentation
qa: add max_mds thrash test
qa: mds_thrash updates for new max_mds behavior
doc: update upgrade procedure and release notes
qa: add test for cluster resizing
qa: remove use of mds deactivate
cephfs: add new down/joinable fs flags
mds: evict all clients if last mds shutting down
cephfs: deprecate ceph mds deactivate
cephfs: kill allow_dirfrags
cephfs: Kill allow_multimds
cephfs: Change behavior of cluster_down flag
mon/FSCommands: Set extra MDS to standby
cephfs: Health check changes
mon/MDSMonitor: Remove command support for legacy syntax

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #21456 into master

* refs/pull/21456/head:
MDSMonitor: initialize new Filesystem epoch from pending

Reviewed-by: John Spray <john.spray@redhat.com>

Merge PR #21457 into master

* refs/pull/21457/head:
mds: unset deleted vars in shutdown_pass

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #21458 into master

* refs/pull/21458/head:
MDSMonitor: cleanup and protect fsmap access

Reviewed-by: João Eduardo Luís <joao@suse.de>

Merge PR #21473 into master

* refs/pull/21473/head:
client: fix error operator precedence

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

client: add client option descriptions

Fixes: http://tracker.ceph.com/issues/22933
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: unset deleted vars in shutdown_pass

So future passes do not try to delete again.

Fixes: http://tracker.ceph.com/issues/23766
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #21480 from smithfarm/wip-backport-create-issue

scripts: new backport-create-issue script

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

script/backport-create-issue: enable script to be imported

Signed-off-by: Nathan Cutler <ncutler@suse.com>

doc/cephfs: update kernel client quotas support info

Signed-off-by: Luis Henriques <lhenriques@suse.com>

ceph-volume tests functional python tests for --no-systemd in activate

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.activate add --no-systemd flag to skip starting/enabling services

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.common add --no-systemd flag to skip enabling/starting services

Signed-off-by: Alfredo Deza <adeza@redhat.com>

client: flush the mdlog in _fsync before waiting on unstable reqs

Recently, we changed ganesha to call to ceph_ll_sync_inode after
issuing a setattr call, to ensure that the attribute changes makes it
to the backing store before returning.

While testing with this, I noticed that we'd occasionally see setattr
calls stall for a long time. Analysis showed that the client's _fsync
request was stalled waiting on an mdlog flush.

This patch adds a call to flush_mdlog_sync prior to waiting on unsafe
requests in _fsync and seems to resolve the issue.

Tracker: http://tracker.ceph.com/issues/23714
Signed-off-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #21436 from trociny/wip-test-out_of_order

test/librbd: ensure OutOfOrder test has enough concurrent management ops

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #21493 from trociny/wip-deep-copy-missing-return

librbd: missing 'return' in deep_copy::ObjectCopyRequest::send_read_object

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #21385 from ricardoasmarques/wip-list-all-rbds

mgr/dashboard: RBD management (frontend)

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #21451 from ceph/wip-rm23581

ceph-volume: automatic VDO detection

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

mgr/dashboard: RBD management

Signed-off-by: Ricardo Marques <rimarques@suse.com>

librbd: missing 'return' in deep_copy::ObjectCopyRequest::send_read_object

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #21383 from b-ranto/wip-mgr-rgw-metadata

prometheus: Implement rgw_metadata metric

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>

mgr/dashboard: Add 'executing' cell template

Signed-off-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: Configurable type for 'cd-submit-button'

Signed-off-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: Add 'delete-confirmation-modal' component

Signed-off-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: Force focus on submit

Forcing focus on submit button will guarantee that all inputs will lose focus
before submission.

Signed-off-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: Add get method to summary service

Signed-off-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #21462 from ricardoasmarques/wip-backgroud-tasks-style

mgr/dashboard: Improve background tasks style

Reviewed-by: Laura Paduano <lpaduano@suse.com>

prometheus: Implement rgw_metadata metric

This maps rgw id to its hostname and ceph version running on the node.

Signed-off-by: Boris Ranto <branto@redhat.com>

Merge pull request #21443 from LenzGr/mgr-dashboard-documentation

doc: Improved dashboard documentation

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

ceph-volume: allow parallel creates

Currently the create code decides the vg_name "ceph-$cluster_fsid" as
the primary vg_name and creates a new name if this already exists.

If this code is run N times in parallel, the script will try to
create N times the vg with the name "ceph-$cluster_fsid" and it
will fail to create the N osds successfully.

Creating vgs with names like "ceph-$uuid4" lets our scripts to run
without any problems.

Signed-off-by: Theofilos Mouratidis <t.mour@cern.ch>

cmake: use CMP0028 NEW

it's intended behavior to use imported or alias target for dependencies
with name with "::" in it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common: silence GCC warnings

Preforker.h:111:8: warning: ignoring return value of ‘ssize_t
safe_write(int, const void*, size_t)’, declared with attribute
warn_unused_result [-Wunused-result]
        (void)safe_write(fd[1], &r, sizeof(r));
        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

and

compat.cc:36:28: warning: comparison between signed and unsigned integer
expressions [-Wsign-compare]
     if (off + sizeof(data) > len)
         ~~~~~~~~~~~~~~~~~~~^~~~~

Fixes: http://tracker.ceph.com/issues/23774
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21485 from tchaikov/wip-tox

test/dashboard: hardcode .coverage path to workaround tox bugs

Reviewed-by: Ricardo Dias <rdias@suse.com>

msg/async/EventKqueue: assert on OOM

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/dashboard: hardcode .coverage path to workaround tox bugs

rhel/centos 7.3 comes with tox 1.4.2, which has a bunch of issues related
to environment variable substitution. for instance, see
- https://github.com/tox-dev/tox/issues/515
so, instead of having different coverage files for py3 and py2.7, it
would be simpler to just use the same path for coverage. this approach
works because pytest-cov plugin erases the coverage data file at the
beginning of test[1].

---
[1] https://pypi.org/project/pytest-cov/#coverage-data-file .

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21474 from jecluis/mimic-mon-pending-releasenotes

PendingReleaseNotes: mention some monitor changes

Reviewed-by: Kefu Chai <kchai@redhat.com>