Merge pull request #23489 from alfredodeza/wip-cv-mgr

ceph-volume tests/functional add mgrs daemons to lvm tests

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge PR #23475 into master

* refs/pull/23475/head:
qa/standalone/crush/crush-choose-args: run mgr

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #23478 into master

* refs/pull/23478/head:
mgr/localpool: pg_num is an int arg to 'osd pool create'

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23267 from ifed01/wip-ifed-rm-fio-dir

test/fio:  pass config params to object store in a different manner

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-volume tests/functional add mgrs daemons to lvm tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>

mgr/localpool: pg_num is an int arg to 'osd pool create'

In bd565bca3f42d7946c392b2833bb3d9982a8ac5d we start returning EINVAL if
the pg_num argument is not valid, whereas before it was silently ignored.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #23472 from jcsp/wip-orchestrator-rook-doc

doc: Rook/orchestrator doc fixes

Reviewed-by: Alfredo Deza <adeza@redhat.com>

qa/standalone/crush/crush-choose-args: run mgr

The osd purge command needs a running mgr.

Fixes: d2b41d4095074447df247907934626f4287cd91b
Signed-off-by: Sage Weil <sage@redhat.com>

test/fio: pass config params to object store in a different manner

This avoids an existing bug with config params substituion.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

doc: Rook/orchestrator doc fixes

Signed-off-by: John Spray <john.spray@redhat.com>

Merge PR #23432 into master

* refs/pull/23432/head:
Revert "rocksdb: pickup change to link against libsnappy.a"
cmake,make-dist: revert "build gperftools if WITH_STATIC_LIBSTDCXX"
cmake: cleanup snappy related script
debian: strip "-Wl,-Bsymbolic-functions" from LDFLAGS

Merge PR #23243 into master

* refs/pull/23243/head:
mon/OSDMonitor: remove ambiguous type case for 'osd pool set' value
mon: use throwing variant of cmd_getval
common/cmdparse: add cmd_getval_throws variants that throw bad_cmd_get
mon: catch bad_cmd_exception and reply EINVAL
common/cmdparse: add bad_cmd_get exception

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23338 into master

* refs/pull/23338/head:
msg/Messenger: send_message -> send_to
msg/msg_types: add entity_addr_t::legacy_or_front_addr()
mon/MonMap: kill get_inst()
mon: kill MonMap::get_inst() users
mon/MonClient: replace get_mon_{addr,inst} with get_mon_addrs
mon/MonClient: get_connection -> connect_to_mon
src/test: get_connection -> connect_to_*
mon/MonMap: add get_addrs() (temp wrappers)
msg/Messenger: make connect_to() key method, get_connection() becomes wrapper

Merge PR #23412 into master

* refs/pull/23412/head:
mgr: add Orchestrator.available method + CLI
doc: add general documentation on orchestrator
doc: add page for Rook module
doc: add development guide for Ceph+Rook in k8s
script: add kubejacker script
mgr/rook: Rook orchestrator module
mgr/orchestrator_cli: module for calling into orchestators
mgr: introduce `orchestrator` python interface

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23430 from tchaikov/wip-silence-clang-warnings

tools, mgr: silence clang warnings

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #23420 from tspmelo/fix-24902

mgr/dashboard: Escape regex pattern in DeletionModalComponent

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #23414 from tspmelo/wip-breadcrumbs

mgr/dashboard: Add breadcrumbs component

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #23465 from ukernel/wip-revert-22850

Revert "cephfs-journal-tool: enable purge_queue journal's event comma…

Revert "cephfs-journal-tool: enable purge_queue journal's event commands"

This reverts commit 8ebfa93982a95d6ccc66faa2a828b9386af45238. That
commit changes PurgeItem's encode/decode function, breaks backward
compatibility

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #22562 from wjwithagen/wjw-fix-dashboard-build

mgr/dashboard: Fix building under FreeBSD

Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #23450 from neha-ojha/wip-fix-complete-to

osd/PGLog.cc: check if complete_to points to log.end()

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

FreeBSD: update build script to current state

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #23456 from yuriw/wip-yuriw-crontab-master

qa/tests: changed ceph qa email address to bypass dreamhost's spam filter

qa/tests: changed ceph qa email address to bypass dreamhost's spam filter

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #23448 from alfredodeza/wip-rm26862

ceph-volume: `lvm batch` allow extra flags (like dmcrypt) for bluestore

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge PR #23433 into master

* refs/pull/23433/head:
client: fix compile warning

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #22850 into master

* refs/pull/22850/head:
cephfs-journal-tool: enable purge_queue journal's event commands

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22999 into master

* refs/pull/22999/head:
mds: consider max age of dispatch queue when finding stale client
mds: reset heartbeat map at potential time-consuming places
mds: change MDSRank::finished_queue to deque

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23341 into master

* refs/pull/23341/head:
mds: calculate load by checking self CPU usage

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23370 into master

* refs/pull/23370/head:
client: check for unmounted condition before printing debug output

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #23387 from jcsp/wip-mgr-rw

mgr: fix permissions on `balancer execute`

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23205 from jcsp/wip-devicehealth-cleanup

mgr/devicehealth: clean up error handling

Reviewed-by: Sage Weil <sage@redhat.com>

osd/PGLog.cc: check if complete_to points to log.end()

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #23443 from alfredodeza/wip-rm24970

ceph-volume: `lvm batch` documentation and man page updates

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume lvm.batch move import statement to the top

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.batch allow extra flags for bluestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #23347 from votdev/bug_25190

mgr/dashboard: RestClient can't handle ProtocolError exceptions

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

mgr: add Orchestrator.available method + CLI

The CLI `status` method tells you what backend
is selected, and also gives the backend
an opportunity to complain if something
seems wrong with its configuration.

Signed-off-by: John Spray <john.spray@redhat.com>

doc: add general documentation on orchestrator

Signed-off-by: John Spray <john.spray@redhat.com>

doc: add page for Rook module

This is pretty sparse because users don't really
interact with the module, they interact with other
modules that use it.

Signed-off-by: John Spray <john.spray@redhat.com>

doc: add development guide for Ceph+Rook in k8s

Signed-off-by: John Spray <john.spray@redhat.com>

script: add kubejacker script

For those with running k8s/Rook environments, that want
to inject newly built binaries from their development
tree into their kubernetes environment.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr/rook: Rook orchestrator module

Signed-off-by: John Spray <john.spray@redhat.com>

mgr/orchestrator_cli: module for calling into orchestators

This is going to be pretty useful for development: calls
into the same places that the dashboard would call into.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: introduce `orchestrator` python interface

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #23410 from ifed01/wip-ifed-fio-new-param

test/fio: new option to control file preallocation.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23404 from badone/wip-daemon_health_metrics-race

mgr/MgrClient: Protect daemon_health_metrics

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #23377 from dzafman/wip-24875

osd: do_sparse_read(): Verify checksum earlier so we will try to repair

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #23361 from smithfarm/wip-install-deps-with-seastar

build/ops: install-deps.sh: refrain from installing/using lsb_release, and other cleanup

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23285 from tspmelo/wip-package-lock

mgr/dashboard: Add package-lock.json

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge PR #22433 into master

* refs/pull/22433/head:
common/config: Add description to (near)full ratio settings

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

doc/man/ceph-volume add batch information

Signed-off-by: Alfredo Deza <adeza@redhat.com>

mgr/dashboard: Add breadcrumbs component

Now we don't need to add a breadcrumb in each new page we create,
just add the necessary extra data in the route module and the
breadcrumb will be automatically created.

I used a modified version of ngx-breadcrumbs from McNull:
https://github.com/McNull/ngx-breadcrumbs

Fixes: http://tracker.ceph.com/issues/24781
Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/devicehealth: improve error handling

Avoid catch-all exception handlers, especially
ones that just `pass`

When catching RADOS object not found type errors,
catch specifically those, and make sure we're still
logging any actual unexpected IO errors from RADOS.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr/devicehealth: style/whitespace cleanup

We don't usually do gratuitious whitespace commits,
but for the python code it's worthwhile as it allows
smart editors to show us real problems without them
getting lost in the style noise.

Signed-off-by: John Spray <john.spray@redhat.com>

doc/ceph-volume include batch doc in the toc tree

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume decument batch reporting and bluestore support

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #23316 from votdev/bug_25090

mgr/dashboard: Datatable error panel blinking on page loading

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

Revert "rocksdb: pickup change to link against libsnappy.a"

This reverts commit 125b5da096989e4bd476acdb99ceb04f44e00758.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake,make-dist: revert "build gperftools if WITH_STATIC_LIBSTDCXX"

as the higher version of libstdc++ is backward compatible with the lower
ones. so there is no need to statically link against C++ libraries. they
can always use the libstdc++ ships with the distro.

This reverts commit a6c73b6ac1d81cc9f467aa38e5afb146d5dea6c2

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22735 from votdev/bug_24677

mgr/dashboard: RGW proxy can't handle self-signed SSL certificates

Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

mgr/dashboard: Escape regex pattern in DeletionModalComponent

Fixes: http://tracker.ceph.com/issues/24902
Signed-off-by: Tiago Melo <tmelo@suse.com>

cmake: cleanup snappy related script

Signed-off-by: Kefu Chai <kchai@redhat.com>

debian: strip "-Wl,-Bsymbolic-functions" from LDFLAGS

Fixes: http://tracker.ceph.com/issues/25209
Signed-off-by: Kefu Chai <kchai@redhat.com>

mds: consider max age of dispatch queue when finding stale client

session renew messages from clients can be in the dispatch queue,
waiting for getting dispatched.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: reset heartbeat map at potential time-consuming places

Signed-off-by: Yan, Zheng <zyan@redhat.com>
Fixes: http://tracker.ceph.com/issues/26858

Merge pull request #23310 from joscollin/wip-fix-rbd-doc-urls

doc: fix the broken urls

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

client: fix compile warning

/home/pdonnell/ceph/src/client/fuse_ll.cc: In member function ‘int CephFuse::Handle::init(int, const char**)’:
/home/pdonnell/ceph/src/client/fuse_ll.cc:1126:59: warning: format ‘%lu’ expects argument of type ‘long unsigned int’, but argument 3 has type ‘Option::size_t’ [-Wformat=]
      sprintf(strsplice, "max_write=%" PRIu64, fuse_max_write);

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23424 from tchaikov/wip-fix-FTBFS-clang

osd: make OSD::HEARTBEAT_MAX_CONN inline

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

mon/OSDMonitor: remove ambiguous type case for 'osd pool set' value

This was there for compatibility with very old monitors.  Modern mons are
always taking a JSON string (which may in turn contain a string, int, or
float).

Signed-off-by: Sage Weil <sage@redhat.com>

mon: use throwing variant of cmd_getval

This ensure we uniformly return EINVAL when the input JSON has a bad
value type.

Signed-off-by: Sage Weil <sage@redhat.com>

common/cmdparse: add cmd_getval_throws variants that throw bad_cmd_get

If input is bad (not missing).

Signed-off-by: Sage Weil <sage@redhat.com>

mon: catch bad_cmd_exception and reply EINVAL

If we get bad input we should always reply EINVAL.  Note that bad_cmd_get
will be reserved for invalid input, not missing/optional input, like
passing a float when an int is expected.

Signed-off-by: Sage Weil <sage@redhat.com>

common/cmdparse: add bad_cmd_get exception

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #23423 from ctriple/master

doc/start: fix kube-helm.rst typo: docuiment -> document

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd: make OSD::HEARTBEAT_MAX_CONN inline

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/start: fix kube-helm.rst typo: docuiment -> document

Signed-off-by: Zhou Peng <p@ctriple.cn>

cephfs-journal-tool: enable purge_queue journal's event commands

Resolves: http://tracker.ceph.com/issues/24604
Signed-off-by: Xuehan Xu <xuxuehan@360.cn>

Merge pull request #23411 from tchaikov/wip-install-py3-for-qa-cephfs

qa: install python3-cephfs for fs suite

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23185 into master

* refs/pull/23185/head:
doc: Create pull_request_template.md

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #23417 into master

* refs/pull/23417/head:
pybind: remove execution permission bits of cephfs.pyx

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

pybind: remove execution permission bits of cephfs.pyx

they were added in c2a671a4b6

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #23158 into master

This is a milestone in the CephFS shell project. QA testing is still in the
works and will be merged separately. Installation code via CMake will also be
written in another PR.

* refs/pull/23158/head:
doc: add cephfs-shell to cephfs doc index
cephfs-shell: add new CephFS shell

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc: add cephfs-shell to cephfs doc index

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

cephfs-shell: add new CephFS shell

The CephFS shell is an alternative client to access CephFS without mounting in
the usual way. The shell provides commands to interactively access the file
system, like put, ls, chmod, etc.

Fixes: http://tracker.ceph.com/issues/24286
Signed-off-by: Pavani Rajula <rpavani1998@gmail.com>

Merge PR #23307 into master

* refs/pull/23307/head:
osd/OSD: fix HeartbeatInfo.is_healthy() check
osd/OSD: use first_tx to calculate failed_for
osd: refactor heartbeat health check

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #22717 into master

* refs/pull/22717/head:
mgr/DaemonServer: osd purge: accept --yes-i-really-mean-it too
mon,mgr: guard 'osd purge' with safe-to-destroy check
mon/MonCommands: hide 'osd destroy-actual'
mon/MonCommands: add HIDDEN flag
mon,mgr: guard 'osd destroy' with 'osd safe-to-destroy' check
mon/MonCommand: deprecate 'osd rm'
mon/OSDMonitor: point users at 'ceph osd safe-to-destroy' for purge|destroy

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #23339 into master

* refs/pull/23339/head:
Revert "mgr/pybind: expose erasure code profiles"

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Datatable error panel blinking on page loading

Fixes https://tracker.ceph.com/issues/25090

Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #23390 from tchaikov/wip-cmake-fmt-4.x

cmake: do not use Findfmt.cmake for checking libfmt-dev

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #23352 from jcsp/wip-25197

mgrc: enable disabling stats via mgr_stats_threshold

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23326 from tchaikov/wip-gnu-cold-attr

common/assert: mark assert helpers with [[gnu::cold]]

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #23320 from tchaikov/wip-cmake-viva-les-local-symbols

cmake: always prefer local symbols

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23330 from jcsp/wip-cli-eperm

ceph.in: friendlier message on EPERM

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr: silence -Wunused-lambda-capture warning

ceph/src/mgr/DaemonServer.cc:1772:17: warning: lambda capture 'host' is
not used [-Wunused-lambda-capture]
          devid, [&f, &host] (const DeviceState& dev) {
                       ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

tools/rados: silence clang++ warning

ceph/src/tools/rados/rados.cc:1951:47: warning: moving a temporary
object prevents copy elision [-Wpessimizing-move]
  boost::optional<pg_t> pgid(i != opts.end(), std::move(pg_t()));
                                              ^
ceph/src/tools/rados/rados.cc:1951:47: note: remove std::move
call here
  boost::optional<pg_t> pgid(i != opts.end(), std::move(pg_t()));

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/suites/fs: add python3-cephfs to packages

the default set of packages to install is in
$suite/qa/packages/packages.yaml . see get_package_list() in
teuthology/teuthology/task/install/__init__.py for how we prepare a
package list for install task.

for running python3 tests in
fs/basic_functional/tasks/volume-client, we need to install
python3-cephfs. please note that,
_package_override() in teuthology/teutholoy/task/install/rpm.py will
take care of the different naming on centos/rhel, where the python3
packages are named python34-*.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/fio: new option to control file preallocation.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #23261 from dillaman/wip-vstart

vstart: do not attempt to re-initialize dashboard for existing cluster

Reviewed-by: John Spray <john.spray@redhat.com>

qa: do not install python3 packages in task.install

This reverts commit c1efd59f618e24cf060d564ac0f21d5b0b57fd4a

task.install.rpm installs packages listed in
$suites/qa/packages/packages.yaml, the packge list applies to the
upgrade tests also. but we don't have python3 bindings packages in jewel
-- they were introduced in kraken.

Signed-off-by: Kefu Chai <kchai@redhat.com>