Merge pull request #23294 from noahdesu/timely-mgr-health-updates

mgr: timely health updates between monitor and manager

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #23257 from ifed01/wip-ifed-fix-rcoll-race

os/bluestore: fix race between remove_collection and object removals

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23492 from liewegas/wip-mgrc-warning

mgr/MgrClient: make some noise for a user if no mgr daemon is running

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge PR #23500 into master

* refs/pull/23500/head:
debian: mark python-ceph-argparse Arch = all
rpm: package cephfs-shell for fedora
tools/cephfs,deb: package cephfs-shell
cmake: install script and egg-info files of cephfs-shell
tools/cephfs: add setup.py for cephfs-shell

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23480 into master

* refs/pull/23480/head:
tools/cephfs: fix mkdirs(exist_ok) call
tools/cephfs: cephfs-shell cleanup
tools/cephfs: run cephfs-shell thru autopep8

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #22868 from iliul/modify-log-level

rgw: Fix log level of gc_iterate_entries

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: Fix log level of gc_iterate_entries

Fixes: http://tracker.ceph.com/issues/23801
Signed-off-by: iliul <liul.stone@gmail.com>

Merge PR #22838 into master

* refs/pull/22838/head:
kv/RocksDBStore: Handle nullptr if clock cache is chosen.
kv/rocksdb_cache/BinnedLRUCache: Don't promote data to the high pri pool.
src/kv: Initial import of a custom RocksDB cache.
src/rocksdb: switch back to master branch.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #22970 into master

* refs/pull/22970/head:
os/bluestore: limit OSD memory usage by tuning the cache size.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #22967 from linuxbox2/wip-slo-nosize

rgw: Swift SLO size_bytes member is optional

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23056 from stiopaa1/fixRgwAdminUser

rgw: fix dependencies/target_link_libraries

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #23282 from zhangsw/fix-rgw-server-header

rgw: add Http header 'Server' in response headers.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23405 from joke-lee/WIP-RGW-BUCKET-NOTIFY

rgw: add s3 notification sub resources

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23434 from joke-lee/wip-cors-rule-num-limit

rgw: cors rules num limit

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #23210 from alfredodeza/wip-bluestore-sizing

doc: document sizing for `block.db`

Reviewed-by: Ben England <bengland@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23511 from excellentkf/WIP-test

common/options: correct the description of rgw_enable_lc_threads option

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23496 from bstillwell/crush_doc_fixes

doc: Fix a couple typos and improve diagram formatting

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23477 from liewegas/wip-kv-merge-name

kv/KeyValueDB: return const char* from MergeOperator::name()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23474 from tchaikov/wip-config-expand-meta

common: expand meta in parse_argv()

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23481 from Yan-waller/wip-walle-180808-smooth-intervals

mgr,mon: fix to apply changed mon_stat_smooth_intervals

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #23482 from Yan-waller/wip-walle-180808-mgr-deadlock

mgr: fix beacon interruption caused by deadlock

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #23487 from xiexingguo/wip-hb-peers-v2

osd/OSD: choose heartbeat peers more carefully

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Yan Jun <yan.jun8@zte.com.cn>

kv/RocksDBStore: Handle nullptr if clock cache is chosen.

Signed-off-by: Mark Nelson <mnelson@redhat.com>

kv/rocksdb_cache/BinnedLRUCache: Don't promote data to the high pri pool.

Signed-off-by: Mark Nelson <mnelson@redhat.com>

src/kv: Initial import of a custom RocksDB cache.

Signed-off-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #23514 from bstillwell/operations_doc_fixes

doc: Multiple spelling fixes

Reviewed-by: Alfredo Deza <adeza@redhat.com>

doc: Multiple spelling fixes

I ran a lot of the docs through aspell and found a number of spelling problems.

Signed-off-by: Bryan Stillwell <bstillwell@godaddy.com>

os/bluestore: limit OSD memory usage by tuning the cache size.

Signed-off-by: Mark Nelson <mnelson@redhat.com>

Merge PR #23471 into master

* refs/pull/23471/head:
mon/PGMap: fix spacing around pretty-printed SI units
include/types: render SI units adjacent to number

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: João Eduardo Luís <joao@suse.de>

Merge PR #23473 into master

* refs/pull/23473/head:
OSD: translate the error result

Reviewed-by: Sage Weil <sage@redhat.com>

doc: Fix a couple typos and improve diagram formatting

I found a couple misspelled words in the crush-map documentation, and also
tweaked the formatting of the CRUSH hierarchy diagram to center some of the
entries.

Signed-off-by: Bryan Stillwell <bstillwell@godaddy.com>

tools/cephfs: fix mkdirs(exist_ok) call

we need to use pathlib.Path.mkdirs(parents=True, exist_ok=True) or
os.mkdirs(path), but not both.

Signed-off-by: Kefu Chai <kchai@redhat.com>

tools/cephfs: cephfs-shell cleanup

Signed-off-by: Kefu Chai <kchai@redhat.com>

tools/cephfs: run cephfs-shell thru autopep8

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #23422 into master

* refs/pull/23422/head:
log/Log: fix buffer overflow

Reviewed-by: Kefu Chai <kchai@redhat.com>

common/options: correct the description of rgw_enable_lc_threads option

Signed-off-by: excellentkf <likefei@cmss.chinamobile.com>

Merge pull request #23466 from tspmelo/wip-update-packages

mgr/dashboard: Update frontend packages

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

debian: mark python-ceph-argparse Arch = all

as the python{3,}-ceph-argparse is pure python

Signed-off-by: Kefu Chai <kchai@redhat.com>

rpm: package cephfs-shell for fedora

Signed-off-by: Kefu Chai <kchai@redhat.com>

tools/cephfs,deb: package cephfs-shell

change `#!/usr/bin/env python3` to `#!/usr/bin/python3` as per
https://www.debian.org/doc/packaging-manuals/python-policy/programs.html#interpreter-directive

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: install script and egg-info files of cephfs-shell

egg-info offers requires.txt, which is parsed by dh_python3 to prepare
the dependencies for the cephfs-shell packaging. also, the meta-info
in the .egg allows user to use eggs if they wish. see
https://wiki.debian.org/Python/FAQ#How_should_we_package_Python_eggs.3F
.

Fixes: http://tracker.ceph.com/issues/26852
Signed-off-by: Kefu Chai <kchai@redhat.com>

tools/cephfs: add setup.py for cephfs-shell

easier to manage its dependency this way, also pave its road to pypi.
please note, cephfs is not hosted by pypi yet.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23431 from falcon78921/wip-doc-rados-pool

doc: fixed links in Pools section

Reviewed-by: John Wilkins <jowilkin@redhat.com>

osd/OSD.cc: force updating heartbeat peers periodically

Because the cluster topology may change (e.g., because we add some
new racks, hosts and disks) and we want the existing osds are then
aware of new incoming osds, guaranteeing osds are always trying to
do heartbeat as wide as possible(e.g., across all racks, hosts etc.).

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

crush/CrushWrapper: check against dead crush rule

We currently remove a rule without adjusting the **rules** array
because we don't recycle the rule_no and hence there can be
holes in the "rules" array.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/OSD.cc: choose heartbeat peers by failure domain

By default, monitor requires at least two valid failure votes/reports from
different hosts to mark an OSD down, which turns out to be impossible sometimes
for a replicated-pool of size of 2 in those clusters made up of hosts
with contiguous labeled OSDs.

This patch instead does a breadth-first search based on the highest level of
failure domain at cluster-wide, to try to make heartbeat peers can cover all failure domains
whenever possible, which can hopefully help accelerating osd failure detection
in the above case..

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #23441 from tchaikov/wip-cmake-rocksdb-cleanup

cmake: rocksdb related cleanup

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #23462 into master

* refs/pull/23462/head:
doc: cleanup in cephfs/capabilities

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mon/PGMap: fix spacing around pretty-printed SI units

"1k foo" not "1kfoo"

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #23489 from alfredodeza/wip-cv-mgr

ceph-volume tests/functional add mgrs daemons to lvm tests

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

mgr/MgrClient: make some noise for a user if no mgr daemon is running

Otherwise a cli command may unexpectedly hang forever with not output
because no ceph-mgr is running, which can be confusing for an admin.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #23475 into master

* refs/pull/23475/head:
qa/standalone/crush/crush-choose-args: run mgr

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #23478 into master

* refs/pull/23478/head:
mgr/localpool: pg_num is an int arg to 'osd pool create'

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23267 from ifed01/wip-ifed-rm-fio-dir

test/fio:  pass config params to object store in a different manner

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-volume tests/functional add mgrs daemons to lvm tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>

crush/CrushWrapper: get_children_of_type - switch outcome to vector

Because we don't really care about the order.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mgr: fix beacon interruption caused by deadlock

There is potential deadlock between this code and DaemonServer::send_report()

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

mgr: fix to apply changed mon_stat_smooth_intervals

While using config set or injectargs command to decrease
mon_stat_smooth_intervals, it didn't apply the new value.
this change fix it.

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

mgr/localpool: pg_num is an int arg to 'osd pool create'

In bd565bca3f42d7946c392b2833bb3d9982a8ac5d we start returning EINVAL if
the pg_num argument is not valid, whereas before it was silently ignored.

Signed-off-by: Sage Weil <sage@redhat.com>

kv/KeyValueDB: return const char* from MergeOperator::name()

This avoids a c_str() on a temporary string later down the line, which
avoids a use-after-free.

Fixes: http://tracker.ceph.com/issues/26875
Signed-off-by: Sage Weil <sage@redhat.com>

src/rocksdb: switch back to master branch.

Signed-off-by: Mark Nelson <mnelson@redhat.com>

mgr: add locked assertion

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

Merge pull request #23472 from jcsp/wip-orchestrator-rook-doc

doc: Rook/orchestrator doc fixes

Reviewed-by: Alfredo Deza <adeza@redhat.com>

qa/standalone/crush/crush-choose-args: run mgr

The osd purge command needs a running mgr.

Fixes: d2b41d4095074447df247907934626f4287cd91b
Signed-off-by: Sage Weil <sage@redhat.com>

common: expand meta in parse_argv()

it fixes the issue where the conf->foo fails to expand if it is set
as "$bar" in a .conf file, and "bar" is another config var, and bar
is set using config_proxy.parse_argv().

because before 773ef117, apply_changes() calls update_legacy_vals(),
which expand meta vars for legacy config vals, but after 773ef117,
we cannot change config_values in apply_changes() anymore. so
update_legacy_vals() calls is moved to apply_changes()'s callers
accordingly. but the apply_changes() is also called by global_init()
directly instead of by the routines in config.cc. so in 773ef117,
i missed it.

in this change, it's put back into parse_argv() instead of into
global_init(), as update_legacy_vals() is a low level function, and
parse_argv() could change the config_values, it's better to call
update_legacy_vals() out there, just like set_mon_vals() and
parse_config_files().

Signed-off-by: Kefu Chai <kchai@redhat.com>

OSD: translate the error result

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

test/fio: pass config params to object store in a different manner

This avoids an existing bug with config params substituion.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

doc: Rook/orchestrator doc fixes

Signed-off-by: John Spray <john.spray@redhat.com>

Merge PR #23432 into master

* refs/pull/23432/head:
Revert "rocksdb: pickup change to link against libsnappy.a"
cmake,make-dist: revert "build gperftools if WITH_STATIC_LIBSTDCXX"
cmake: cleanup snappy related script
debian: strip "-Wl,-Bsymbolic-functions" from LDFLAGS

Merge PR #23243 into master

* refs/pull/23243/head:
mon/OSDMonitor: remove ambiguous type case for 'osd pool set' value
mon: use throwing variant of cmd_getval
common/cmdparse: add cmd_getval_throws variants that throw bad_cmd_get
mon: catch bad_cmd_exception and reply EINVAL
common/cmdparse: add bad_cmd_get exception

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23338 into master

* refs/pull/23338/head:
msg/Messenger: send_message -> send_to
msg/msg_types: add entity_addr_t::legacy_or_front_addr()
mon/MonMap: kill get_inst()
mon: kill MonMap::get_inst() users
mon/MonClient: replace get_mon_{addr,inst} with get_mon_addrs
mon/MonClient: get_connection -> connect_to_mon
src/test: get_connection -> connect_to_*
mon/MonMap: add get_addrs() (temp wrappers)
msg/Messenger: make connect_to() key method, get_connection() becomes wrapper

Merge PR #23412 into master

* refs/pull/23412/head:
mgr: add Orchestrator.available method + CLI
doc: add general documentation on orchestrator
doc: add page for Rook module
doc: add development guide for Ceph+Rook in k8s
script: add kubejacker script
mgr/rook: Rook orchestrator module
mgr/orchestrator_cli: module for calling into orchestators
mgr: introduce `orchestrator` python interface

Reviewed-by: Sage Weil <sage@redhat.com>

include/types: render SI units adjacent to number

Make

 out << si_u_t(foo) << " objects"

result in strings like

 "0 objects"
 "123k objects"
 "3M objects"

instead of

 "0  objects"
 "0 k objects"
 "3 M objects"

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #23430 from tchaikov/wip-silence-clang-warnings

tools, mgr: silence clang warnings

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #23420 from tspmelo/fix-24902

mgr/dashboard: Escape regex pattern in DeletionModalComponent

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #23414 from tspmelo/wip-breadcrumbs

mgr/dashboard: Add breadcrumbs component

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #23465 from ukernel/wip-revert-22850

Revert "cephfs-journal-tool: enable purge_queue journal's event comma…

mgr/dashboard: Update prettier package

Prettier updated the following rule, that implied we update our code:
"Never inline decorators unless they’re lone parameter decorators"

Signed-off-by: Tiago Melo <tmelo@suse.com>

Revert "cephfs-journal-tool: enable purge_queue journal's event commands"

This reverts commit 8ebfa93982a95d6ccc66faa2a828b9386af45238. That
commit changes PurgeItem's encode/decode function, breaks backward
compatibility

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mgr/dashboard: Update Codelyzer package

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Update all @angular packages

no-unused-variable is deprecated because typescript now covers most of its
functionality.

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Update frontend packages

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #22562 from wjwithagen/wjw-fix-dashboard-build

mgr/dashboard: Fix building under FreeBSD

Reviewed-by: Tiago Melo <tmelo@suse.com>

doc: cleanup in cephfs/capabilities

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #23450 from neha-ojha/wip-fix-complete-to

osd/PGLog.cc: check if complete_to points to log.end()

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

FreeBSD: update build script to current state

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #23456 from yuriw/wip-yuriw-crontab-master

qa/tests: changed ceph qa email address to bypass dreamhost's spam filter

qa/tests: changed ceph qa email address to bypass dreamhost's spam filter

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #23448 from alfredodeza/wip-rm26862

ceph-volume: `lvm batch` allow extra flags (like dmcrypt) for bluestore

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge PR #23433 into master

* refs/pull/23433/head:
client: fix compile warning

Reviewed-by: Zheng Yan <zyan@redhat.com>

doc/bluestore detail the different sizing strategies for block.db based on cluster workload

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/bluestore expand on configuration with fast and slow devices

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge PR #22850 into master

* refs/pull/22850/head:
cephfs-journal-tool: enable purge_queue journal's event commands

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22999 into master

* refs/pull/22999/head:
mds: consider max age of dispatch queue when finding stale client
mds: reset heartbeat map at potential time-consuming places
mds: change MDSRank::finished_queue to deque

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23341 into master

* refs/pull/23341/head:
mds: calculate load by checking self CPU usage

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23370 into master

* refs/pull/23370/head:
client: check for unmounted condition before printing debug output

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #23387 from jcsp/wip-mgr-rw

mgr: fix permissions on `balancer execute`

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23205 from jcsp/wip-devicehealth-cleanup

mgr/devicehealth: clean up error handling

Reviewed-by: Sage Weil <sage@redhat.com>

osd/PGLog.cc: check if complete_to points to log.end()

Signed-off-by: Neha Ojha <nojha@redhat.com>

os/fusestore: del redundant and sometimes improper open_collection call

Signed-off-by: Igor Fedotov <ifedotov@suse.com>