qa/rgw: vary omap osd-side limit

Signed-off-by: Sage Weil <sage@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: use get_omap_keys2

Making looping less fragile.

Signed-off-by: Sage Weil <sage@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23511 from excellentkf/WIP-test

common/options: correct the description of rgw_enable_lc_threads option

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23496 from bstillwell/crush_doc_fixes

doc: Fix a couple typos and improve diagram formatting

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23477 from liewegas/wip-kv-merge-name

kv/KeyValueDB: return const char* from MergeOperator::name()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23474 from tchaikov/wip-config-expand-meta

common: expand meta in parse_argv()

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23481 from Yan-waller/wip-walle-180808-smooth-intervals

mgr,mon: fix to apply changed mon_stat_smooth_intervals

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #23482 from Yan-waller/wip-walle-180808-mgr-deadlock

mgr: fix beacon interruption caused by deadlock

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #23487 from xiexingguo/wip-hb-peers-v2

osd/OSD: choose heartbeat peers more carefully

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Yan Jun <yan.jun8@zte.com.cn>

Merge pull request #23514 from bstillwell/operations_doc_fixes

doc: Multiple spelling fixes

Reviewed-by: Alfredo Deza <adeza@redhat.com>

doc: Multiple spelling fixes

I ran a lot of the docs through aspell and found a number of spelling problems.

Signed-off-by: Bryan Stillwell <bstillwell@godaddy.com>

Merge PR #23471 into master

* refs/pull/23471/head:
mon/PGMap: fix spacing around pretty-printed SI units
include/types: render SI units adjacent to number

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: João Eduardo Luís <joao@suse.de>

Merge PR #23473 into master

* refs/pull/23473/head:
OSD: translate the error result

Reviewed-by: Sage Weil <sage@redhat.com>

doc: Fix a couple typos and improve diagram formatting

I found a couple misspelled words in the crush-map documentation, and also
tweaked the formatting of the CRUSH hierarchy diagram to center some of the
entries.

Signed-off-by: Bryan Stillwell <bstillwell@godaddy.com>

Merge PR #23422 into master

* refs/pull/23422/head:
log/Log: fix buffer overflow

Reviewed-by: Kefu Chai <kchai@redhat.com>

common/options: correct the description of rgw_enable_lc_threads option

Signed-off-by: excellentkf <likefei@cmss.chinamobile.com>

Merge pull request #23466 from tspmelo/wip-update-packages

mgr/dashboard: Update frontend packages

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #23431 from falcon78921/wip-doc-rados-pool

doc: fixed links in Pools section

Reviewed-by: John Wilkins <jowilkin@redhat.com>

osd/OSD.cc: force updating heartbeat peers periodically

Because the cluster topology may change (e.g., because we add some
new racks, hosts and disks) and we want the existing osds are then
aware of new incoming osds, guaranteeing osds are always trying to
do heartbeat as wide as possible(e.g., across all racks, hosts etc.).

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

crush/CrushWrapper: check against dead crush rule

We currently remove a rule without adjusting the **rules** array
because we don't recycle the rule_no and hence there can be
holes in the "rules" array.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/OSD.cc: choose heartbeat peers by failure domain

By default, monitor requires at least two valid failure votes/reports from
different hosts to mark an OSD down, which turns out to be impossible sometimes
for a replicated-pool of size of 2 in those clusters made up of hosts
with contiguous labeled OSDs.

This patch instead does a breadth-first search based on the highest level of
failure domain at cluster-wide, to try to make heartbeat peers can cover all failure domains
whenever possible, which can hopefully help accelerating osd failure detection
in the above case..

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #23441 from tchaikov/wip-cmake-rocksdb-cleanup

cmake: rocksdb related cleanup

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #23462 into master

* refs/pull/23462/head:
doc: cleanup in cephfs/capabilities

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mon/PGMap: fix spacing around pretty-printed SI units

"1k foo" not "1kfoo"

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #23489 from alfredodeza/wip-cv-mgr

ceph-volume tests/functional add mgrs daemons to lvm tests

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge PR #23475 into master

* refs/pull/23475/head:
qa/standalone/crush/crush-choose-args: run mgr

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #23478 into master

* refs/pull/23478/head:
mgr/localpool: pg_num is an int arg to 'osd pool create'

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23267 from ifed01/wip-ifed-rm-fio-dir

test/fio:  pass config params to object store in a different manner

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-volume tests/functional add mgrs daemons to lvm tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>

crush/CrushWrapper: get_children_of_type - switch outcome to vector

Because we don't really care about the order.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mgr: fix beacon interruption caused by deadlock

There is potential deadlock between this code and DaemonServer::send_report()

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

mgr: fix to apply changed mon_stat_smooth_intervals

While using config set or injectargs command to decrease
mon_stat_smooth_intervals, it didn't apply the new value.
this change fix it.

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

mgr/localpool: pg_num is an int arg to 'osd pool create'

In bd565bca3f42d7946c392b2833bb3d9982a8ac5d we start returning EINVAL if
the pg_num argument is not valid, whereas before it was silently ignored.

Signed-off-by: Sage Weil <sage@redhat.com>

kv/KeyValueDB: return const char* from MergeOperator::name()

This avoids a c_str() on a temporary string later down the line, which
avoids a use-after-free.

Fixes: http://tracker.ceph.com/issues/26875
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #23472 from jcsp/wip-orchestrator-rook-doc

doc: Rook/orchestrator doc fixes

Reviewed-by: Alfredo Deza <adeza@redhat.com>

qa/standalone/crush/crush-choose-args: run mgr

The osd purge command needs a running mgr.

Fixes: d2b41d4095074447df247907934626f4287cd91b
Signed-off-by: Sage Weil <sage@redhat.com>

common: expand meta in parse_argv()

it fixes the issue where the conf->foo fails to expand if it is set
as "$bar" in a .conf file, and "bar" is another config var, and bar
is set using config_proxy.parse_argv().

because before 773ef117, apply_changes() calls update_legacy_vals(),
which expand meta vars for legacy config vals, but after 773ef117,
we cannot change config_values in apply_changes() anymore. so
update_legacy_vals() calls is moved to apply_changes()'s callers
accordingly. but the apply_changes() is also called by global_init()
directly instead of by the routines in config.cc. so in 773ef117,
i missed it.

in this change, it's put back into parse_argv() instead of into
global_init(), as update_legacy_vals() is a low level function, and
parse_argv() could change the config_values, it's better to call
update_legacy_vals() out there, just like set_mon_vals() and
parse_config_files().

Signed-off-by: Kefu Chai <kchai@redhat.com>

OSD: translate the error result

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

test/fio: pass config params to object store in a different manner

This avoids an existing bug with config params substituion.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

doc: Rook/orchestrator doc fixes

Signed-off-by: John Spray <john.spray@redhat.com>

Merge PR #23432 into master

* refs/pull/23432/head:
Revert "rocksdb: pickup change to link against libsnappy.a"
cmake,make-dist: revert "build gperftools if WITH_STATIC_LIBSTDCXX"
cmake: cleanup snappy related script
debian: strip "-Wl,-Bsymbolic-functions" from LDFLAGS

Merge PR #23243 into master

* refs/pull/23243/head:
mon/OSDMonitor: remove ambiguous type case for 'osd pool set' value
mon: use throwing variant of cmd_getval
common/cmdparse: add cmd_getval_throws variants that throw bad_cmd_get
mon: catch bad_cmd_exception and reply EINVAL
common/cmdparse: add bad_cmd_get exception

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23338 into master

* refs/pull/23338/head:
msg/Messenger: send_message -> send_to
msg/msg_types: add entity_addr_t::legacy_or_front_addr()
mon/MonMap: kill get_inst()
mon: kill MonMap::get_inst() users
mon/MonClient: replace get_mon_{addr,inst} with get_mon_addrs
mon/MonClient: get_connection -> connect_to_mon
src/test: get_connection -> connect_to_*
mon/MonMap: add get_addrs() (temp wrappers)
msg/Messenger: make connect_to() key method, get_connection() becomes wrapper

Merge PR #23412 into master

* refs/pull/23412/head:
mgr: add Orchestrator.available method + CLI
doc: add general documentation on orchestrator
doc: add page for Rook module
doc: add development guide for Ceph+Rook in k8s
script: add kubejacker script
mgr/rook: Rook orchestrator module
mgr/orchestrator_cli: module for calling into orchestators
mgr: introduce `orchestrator` python interface

Reviewed-by: Sage Weil <sage@redhat.com>

include/types: render SI units adjacent to number

Make

 out << si_u_t(foo) << " objects"

result in strings like

 "0 objects"
 "123k objects"
 "3M objects"

instead of

 "0  objects"
 "0 k objects"
 "3 M objects"

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #23430 from tchaikov/wip-silence-clang-warnings

tools, mgr: silence clang warnings

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #23420 from tspmelo/fix-24902

mgr/dashboard: Escape regex pattern in DeletionModalComponent

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #23414 from tspmelo/wip-breadcrumbs

mgr/dashboard: Add breadcrumbs component

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #23465 from ukernel/wip-revert-22850

Revert "cephfs-journal-tool: enable purge_queue journal's event comma…

mgr/dashboard: Update prettier package

Prettier updated the following rule, that implied we update our code:
"Never inline decorators unless they’re lone parameter decorators"

Signed-off-by: Tiago Melo <tmelo@suse.com>

Revert "cephfs-journal-tool: enable purge_queue journal's event commands"

This reverts commit 8ebfa93982a95d6ccc66faa2a828b9386af45238. That
commit changes PurgeItem's encode/decode function, breaks backward
compatibility

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mgr/dashboard: Update Codelyzer package

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Update all @angular packages

no-unused-variable is deprecated because typescript now covers most of its
functionality.

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Update frontend packages

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #22562 from wjwithagen/wjw-fix-dashboard-build

mgr/dashboard: Fix building under FreeBSD

Reviewed-by: Tiago Melo <tmelo@suse.com>

doc: cleanup in cephfs/capabilities

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #23450 from neha-ojha/wip-fix-complete-to

osd/PGLog.cc: check if complete_to points to log.end()

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

FreeBSD: update build script to current state

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #23456 from yuriw/wip-yuriw-crontab-master

qa/tests: changed ceph qa email address to bypass dreamhost's spam filter

qa/tests: changed ceph qa email address to bypass dreamhost's spam filter

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #23448 from alfredodeza/wip-rm26862

ceph-volume: `lvm batch` allow extra flags (like dmcrypt) for bluestore

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge PR #23433 into master

* refs/pull/23433/head:
client: fix compile warning

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #22850 into master

* refs/pull/22850/head:
cephfs-journal-tool: enable purge_queue journal's event commands

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22999 into master

* refs/pull/22999/head:
mds: consider max age of dispatch queue when finding stale client
mds: reset heartbeat map at potential time-consuming places
mds: change MDSRank::finished_queue to deque

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23341 into master

* refs/pull/23341/head:
mds: calculate load by checking self CPU usage

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23370 into master

* refs/pull/23370/head:
client: check for unmounted condition before printing debug output

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #23387 from jcsp/wip-mgr-rw

mgr: fix permissions on `balancer execute`

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23205 from jcsp/wip-devicehealth-cleanup

mgr/devicehealth: clean up error handling

Reviewed-by: Sage Weil <sage@redhat.com>

osd/PGLog.cc: check if complete_to points to log.end()

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #23443 from alfredodeza/wip-rm24970

ceph-volume: `lvm batch` documentation and man page updates

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume lvm.batch move import statement to the top

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.batch allow extra flags for bluestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #23347 from votdev/bug_25190

mgr/dashboard: RestClient can't handle ProtocolError exceptions

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

mgr: add Orchestrator.available method + CLI

The CLI `status` method tells you what backend
is selected, and also gives the backend
an opportunity to complain if something
seems wrong with its configuration.

Signed-off-by: John Spray <john.spray@redhat.com>

doc: add general documentation on orchestrator

Signed-off-by: John Spray <john.spray@redhat.com>

doc: add page for Rook module

This is pretty sparse because users don't really
interact with the module, they interact with other
modules that use it.

Signed-off-by: John Spray <john.spray@redhat.com>

doc: add development guide for Ceph+Rook in k8s

Signed-off-by: John Spray <john.spray@redhat.com>

script: add kubejacker script

For those with running k8s/Rook environments, that want
to inject newly built binaries from their development
tree into their kubernetes environment.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr/rook: Rook orchestrator module

Signed-off-by: John Spray <john.spray@redhat.com>

mgr/orchestrator_cli: module for calling into orchestators

This is going to be pretty useful for development: calls
into the same places that the dashboard would call into.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: introduce `orchestrator` python interface

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #23410 from ifed01/wip-ifed-fio-new-param

test/fio: new option to control file preallocation.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23404 from badone/wip-daemon_health_metrics-race

mgr/MgrClient: Protect daemon_health_metrics

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #23377 from dzafman/wip-24875

osd: do_sparse_read(): Verify checksum earlier so we will try to repair

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #23361 from smithfarm/wip-install-deps-with-seastar

build/ops: install-deps.sh: refrain from installing/using lsb_release, and other cleanup

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23285 from tspmelo/wip-package-lock

mgr/dashboard: Add package-lock.json

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge PR #22433 into master

* refs/pull/22433/head:
common/config: Add description to (near)full ratio settings

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

log/Log: fix buffer overflow

If we have a long line that has to be dynamically allocated, we need to
adjust m_log_buf_pos accordingly afterward.  In particular,
_write_and_copy does this adjustment, which means the later += line_used
is not only unnecessary but very bad, since line_used is > MAX_LOG_BUF.

Simplify all of this by getting rid of _write_and_copy: if we are writing
a big chunk let's just send the whole thing to m_fd and not bother keeping
the tail of it around for later.

Fix _log_safe_write to write the passed buffer, not m_log_buf.

Replace the paranoid check with a simple assert.

Fixes: 65da5ba216cafb8a91893d0e7fc092209f0fb286
Fixes: https://github.com/ceph/ceph/pull/23422
Signed-off-by: Sage Weil <sage@redhat.com>

doc/man/ceph-volume add batch information

Signed-off-by: Alfredo Deza <adeza@redhat.com>

mgr/dashboard: Add breadcrumbs component

Now we don't need to add a breadcrumb in each new page we create,
just add the necessary extra data in the route module and the
breadcrumb will be automatically created.

I used a modified version of ngx-breadcrumbs from McNull:
https://github.com/McNull/ngx-breadcrumbs

Fixes: http://tracker.ceph.com/issues/24781
Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/devicehealth: improve error handling

Avoid catch-all exception handlers, especially
ones that just `pass`

When catching RADOS object not found type errors,
catch specifically those, and make sure we're still
logging any actual unexpected IO errors from RADOS.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr/devicehealth: style/whitespace cleanup

We don't usually do gratuitious whitespace commits,
but for the python code it's worthwhile as it allows
smart editors to show us real problems without them
getting lost in the style noise.

Signed-off-by: John Spray <john.spray@redhat.com>

doc/ceph-volume include batch doc in the toc tree

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume decument batch reporting and bluestore support

Signed-off-by: Alfredo Deza <adeza@redhat.com>

cmake: add ALLOCATOR option

ALLOCATOR is in fact an option..

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: refactor RocksDB related script

* rename Findrocksdb.cmake to FindRocksDB.cmake to match its name
* add RocksDB::RocksDB target to BuildRocksDB.cmake and
  FindRocksDB.cmake
* use RocksDB::RocksDB target instead of accessing its property
  directly, and do not link against its dependencies explicitly.
  let its INTERFACE_LINK_LIBRARIES do the job.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23316 from votdev/bug_25090

mgr/dashboard: Datatable error panel blinking on page loading

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

Revert "rocksdb: pickup change to link against libsnappy.a"

This reverts commit 125b5da096989e4bd476acdb99ceb04f44e00758.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake,make-dist: revert "build gperftools if WITH_STATIC_LIBSTDCXX"

as the higher version of libstdc++ is backward compatible with the lower
ones. so there is no need to statically link against C++ libraries. they
can always use the libstdc++ ships with the distro.

This reverts commit a6c73b6ac1d81cc9f467aa38e5afb146d5dea6c2

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add LZ4::LZ4 target for LZ4

Signed-off-by: Kefu Chai <kchai@redhat.com>