osd: vary tick interval +/- 5% to avoid scrub livelocks

If you have two pgs that need to scrub on two OSDs, each the primary
for one pg and the replica for the other, you can end up in a livelock:

- both osds locally reserve a scrub slot
- both osds send a scrub schedule request
- both scrub requests are rejected
- both osds wait exactly 1 second
- repeat

Seems a bit unlikely, but I've seen test cases where it goes on more an
hour.

Fixes: http://tracker.ceph.com/issues/26890
Signed-off-by: Sage Weil <sage@redhat.com>

osd: tick at OSD_TICK_INTERVAL, not heartbeat interval

Heartbeat inveral is not related!

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #23422 into master

* refs/pull/23422/head:
log/Log: fix buffer overflow

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23466 from tspmelo/wip-update-packages

mgr/dashboard: Update frontend packages

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #23431 from falcon78921/wip-doc-rados-pool

doc: fixed links in Pools section

Reviewed-by: John Wilkins <jowilkin@redhat.com>

Merge pull request #23441 from tchaikov/wip-cmake-rocksdb-cleanup

cmake: rocksdb related cleanup

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #23462 into master

* refs/pull/23462/head:
doc: cleanup in cephfs/capabilities

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23489 from alfredodeza/wip-cv-mgr

ceph-volume tests/functional add mgrs daemons to lvm tests

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge PR #23475 into master

* refs/pull/23475/head:
qa/standalone/crush/crush-choose-args: run mgr

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #23478 into master

* refs/pull/23478/head:
mgr/localpool: pg_num is an int arg to 'osd pool create'

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23267 from ifed01/wip-ifed-rm-fio-dir

test/fio:  pass config params to object store in a different manner

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-volume tests/functional add mgrs daemons to lvm tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>

mgr/localpool: pg_num is an int arg to 'osd pool create'

In bd565bca3f42d7946c392b2833bb3d9982a8ac5d we start returning EINVAL if
the pg_num argument is not valid, whereas before it was silently ignored.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #23472 from jcsp/wip-orchestrator-rook-doc

doc: Rook/orchestrator doc fixes

Reviewed-by: Alfredo Deza <adeza@redhat.com>

qa/standalone/crush/crush-choose-args: run mgr

The osd purge command needs a running mgr.

Fixes: d2b41d4095074447df247907934626f4287cd91b
Signed-off-by: Sage Weil <sage@redhat.com>

test/fio: pass config params to object store in a different manner

This avoids an existing bug with config params substituion.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

doc: Rook/orchestrator doc fixes

Signed-off-by: John Spray <john.spray@redhat.com>

Merge PR #23432 into master

* refs/pull/23432/head:
Revert "rocksdb: pickup change to link against libsnappy.a"
cmake,make-dist: revert "build gperftools if WITH_STATIC_LIBSTDCXX"
cmake: cleanup snappy related script
debian: strip "-Wl,-Bsymbolic-functions" from LDFLAGS

Merge PR #23243 into master

* refs/pull/23243/head:
mon/OSDMonitor: remove ambiguous type case for 'osd pool set' value
mon: use throwing variant of cmd_getval
common/cmdparse: add cmd_getval_throws variants that throw bad_cmd_get
mon: catch bad_cmd_exception and reply EINVAL
common/cmdparse: add bad_cmd_get exception

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23338 into master

* refs/pull/23338/head:
msg/Messenger: send_message -> send_to
msg/msg_types: add entity_addr_t::legacy_or_front_addr()
mon/MonMap: kill get_inst()
mon: kill MonMap::get_inst() users
mon/MonClient: replace get_mon_{addr,inst} with get_mon_addrs
mon/MonClient: get_connection -> connect_to_mon
src/test: get_connection -> connect_to_*
mon/MonMap: add get_addrs() (temp wrappers)
msg/Messenger: make connect_to() key method, get_connection() becomes wrapper

Merge PR #23412 into master

* refs/pull/23412/head:
mgr: add Orchestrator.available method + CLI
doc: add general documentation on orchestrator
doc: add page for Rook module
doc: add development guide for Ceph+Rook in k8s
script: add kubejacker script
mgr/rook: Rook orchestrator module
mgr/orchestrator_cli: module for calling into orchestators
mgr: introduce `orchestrator` python interface

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23430 from tchaikov/wip-silence-clang-warnings

tools, mgr: silence clang warnings

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #23420 from tspmelo/fix-24902

mgr/dashboard: Escape regex pattern in DeletionModalComponent

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #23414 from tspmelo/wip-breadcrumbs

mgr/dashboard: Add breadcrumbs component

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #23465 from ukernel/wip-revert-22850

Revert "cephfs-journal-tool: enable purge_queue journal's event comma…

mgr/dashboard: Update prettier package

Prettier updated the following rule, that implied we update our code:
"Never inline decorators unless they’re lone parameter decorators"

Signed-off-by: Tiago Melo <tmelo@suse.com>

Revert "cephfs-journal-tool: enable purge_queue journal's event commands"

This reverts commit 8ebfa93982a95d6ccc66faa2a828b9386af45238. That
commit changes PurgeItem's encode/decode function, breaks backward
compatibility

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mgr/dashboard: Update Codelyzer package

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Update all @angular packages

no-unused-variable is deprecated because typescript now covers most of its
functionality.

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Update frontend packages

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #22562 from wjwithagen/wjw-fix-dashboard-build

mgr/dashboard: Fix building under FreeBSD

Reviewed-by: Tiago Melo <tmelo@suse.com>

doc: cleanup in cephfs/capabilities

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #23450 from neha-ojha/wip-fix-complete-to

osd/PGLog.cc: check if complete_to points to log.end()

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

FreeBSD: update build script to current state

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #23456 from yuriw/wip-yuriw-crontab-master

qa/tests: changed ceph qa email address to bypass dreamhost's spam filter

qa/tests: changed ceph qa email address to bypass dreamhost's spam filter

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #23448 from alfredodeza/wip-rm26862

ceph-volume: `lvm batch` allow extra flags (like dmcrypt) for bluestore

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge PR #23433 into master

* refs/pull/23433/head:
client: fix compile warning

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #22850 into master

* refs/pull/22850/head:
cephfs-journal-tool: enable purge_queue journal's event commands

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22999 into master

* refs/pull/22999/head:
mds: consider max age of dispatch queue when finding stale client
mds: reset heartbeat map at potential time-consuming places
mds: change MDSRank::finished_queue to deque

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23341 into master

* refs/pull/23341/head:
mds: calculate load by checking self CPU usage

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23370 into master

* refs/pull/23370/head:
client: check for unmounted condition before printing debug output

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #23387 from jcsp/wip-mgr-rw

mgr: fix permissions on `balancer execute`

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23205 from jcsp/wip-devicehealth-cleanup

mgr/devicehealth: clean up error handling

Reviewed-by: Sage Weil <sage@redhat.com>

osd/PGLog.cc: check if complete_to points to log.end()

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #23443 from alfredodeza/wip-rm24970

ceph-volume: `lvm batch` documentation and man page updates

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume lvm.batch move import statement to the top

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.batch allow extra flags for bluestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #23347 from votdev/bug_25190

mgr/dashboard: RestClient can't handle ProtocolError exceptions

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

mgr: add Orchestrator.available method + CLI

The CLI `status` method tells you what backend
is selected, and also gives the backend
an opportunity to complain if something
seems wrong with its configuration.

Signed-off-by: John Spray <john.spray@redhat.com>

doc: add general documentation on orchestrator

Signed-off-by: John Spray <john.spray@redhat.com>

doc: add page for Rook module

This is pretty sparse because users don't really
interact with the module, they interact with other
modules that use it.

Signed-off-by: John Spray <john.spray@redhat.com>

doc: add development guide for Ceph+Rook in k8s

Signed-off-by: John Spray <john.spray@redhat.com>

script: add kubejacker script

For those with running k8s/Rook environments, that want
to inject newly built binaries from their development
tree into their kubernetes environment.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr/rook: Rook orchestrator module

Signed-off-by: John Spray <john.spray@redhat.com>

mgr/orchestrator_cli: module for calling into orchestators

This is going to be pretty useful for development: calls
into the same places that the dashboard would call into.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: introduce `orchestrator` python interface

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #23410 from ifed01/wip-ifed-fio-new-param

test/fio: new option to control file preallocation.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23404 from badone/wip-daemon_health_metrics-race

mgr/MgrClient: Protect daemon_health_metrics

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #23377 from dzafman/wip-24875

osd: do_sparse_read(): Verify checksum earlier so we will try to repair

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #23361 from smithfarm/wip-install-deps-with-seastar

build/ops: install-deps.sh: refrain from installing/using lsb_release, and other cleanup

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23285 from tspmelo/wip-package-lock

mgr/dashboard: Add package-lock.json

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge PR #22433 into master

* refs/pull/22433/head:
common/config: Add description to (near)full ratio settings

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

log/Log: fix buffer overflow

If we have a long line that has to be dynamically allocated, we need to
adjust m_log_buf_pos accordingly afterward.  In particular,
_write_and_copy does this adjustment, which means the later += line_used
is not only unnecessary but very bad, since line_used is > MAX_LOG_BUF.

Simplify all of this by getting rid of _write_and_copy: if we are writing
a big chunk let's just send the whole thing to m_fd and not bother keeping
the tail of it around for later.

Fix _log_safe_write to write the passed buffer, not m_log_buf.

Replace the paranoid check with a simple assert.

Fixes: 65da5ba216cafb8a91893d0e7fc092209f0fb286
Fixes: https://github.com/ceph/ceph/pull/23422
Signed-off-by: Sage Weil <sage@redhat.com>

doc/man/ceph-volume add batch information

Signed-off-by: Alfredo Deza <adeza@redhat.com>

mgr/dashboard: Add breadcrumbs component

Now we don't need to add a breadcrumb in each new page we create,
just add the necessary extra data in the route module and the
breadcrumb will be automatically created.

I used a modified version of ngx-breadcrumbs from McNull:
https://github.com/McNull/ngx-breadcrumbs

Fixes: http://tracker.ceph.com/issues/24781
Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/devicehealth: improve error handling

Avoid catch-all exception handlers, especially
ones that just `pass`

When catching RADOS object not found type errors,
catch specifically those, and make sure we're still
logging any actual unexpected IO errors from RADOS.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr/devicehealth: style/whitespace cleanup

We don't usually do gratuitious whitespace commits,
but for the python code it's worthwhile as it allows
smart editors to show us real problems without them
getting lost in the style noise.

Signed-off-by: John Spray <john.spray@redhat.com>

doc/ceph-volume include batch doc in the toc tree

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume decument batch reporting and bluestore support

Signed-off-by: Alfredo Deza <adeza@redhat.com>

cmake: add ALLOCATOR option

ALLOCATOR is in fact an option..

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: refactor RocksDB related script

* rename Findrocksdb.cmake to FindRocksDB.cmake to match its name
* add RocksDB::RocksDB target to BuildRocksDB.cmake and
  FindRocksDB.cmake
* use RocksDB::RocksDB target instead of accessing its property
  directly, and do not link against its dependencies explicitly.
  let its INTERFACE_LINK_LIBRARIES do the job.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23316 from votdev/bug_25090

mgr/dashboard: Datatable error panel blinking on page loading

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

Revert "rocksdb: pickup change to link against libsnappy.a"

This reverts commit 125b5da096989e4bd476acdb99ceb04f44e00758.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake,make-dist: revert "build gperftools if WITH_STATIC_LIBSTDCXX"

as the higher version of libstdc++ is backward compatible with the lower
ones. so there is no need to statically link against C++ libraries. they
can always use the libstdc++ ships with the distro.

This reverts commit a6c73b6ac1d81cc9f467aa38e5afb146d5dea6c2

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add LZ4::LZ4 target for LZ4

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: cleanup snappy related script

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22735 from votdev/bug_24677

mgr/dashboard: RGW proxy can't handle self-signed SSL certificates

Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

mgr/dashboard: Escape regex pattern in DeletionModalComponent

Fixes: http://tracker.ceph.com/issues/24902
Signed-off-by: Tiago Melo <tmelo@suse.com>

cmake: cleanup snappy related script

Signed-off-by: Kefu Chai <kchai@redhat.com>

debian: strip "-Wl,-Bsymbolic-functions" from LDFLAGS

Fixes: http://tracker.ceph.com/issues/25209
Signed-off-by: Kefu Chai <kchai@redhat.com>

mds: consider max age of dispatch queue when finding stale client

session renew messages from clients can be in the dispatch queue,
waiting for getting dispatched.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: reset heartbeat map at potential time-consuming places

Signed-off-by: Yan, Zheng <zyan@redhat.com>
Fixes: http://tracker.ceph.com/issues/26858

Merge pull request #23310 from joscollin/wip-fix-rbd-doc-urls

doc: fix the broken urls

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

client: fix compile warning

/home/pdonnell/ceph/src/client/fuse_ll.cc: In member function ‘int CephFuse::Handle::init(int, const char**)’:
/home/pdonnell/ceph/src/client/fuse_ll.cc:1126:59: warning: format ‘%lu’ expects argument of type ‘long unsigned int’, but argument 3 has type ‘Option::size_t’ [-Wformat=]
      sprintf(strsplice, "max_write=%" PRIu64, fuse_max_write);

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc: fixed links in Pools section

Fixed broken links (404 error) in the Pools section of Ceph Storage Cluster.

Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>

Merge pull request #23424 from tchaikov/wip-fix-FTBFS-clang

osd: make OSD::HEARTBEAT_MAX_CONN inline

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

mon/OSDMonitor: remove ambiguous type case for 'osd pool set' value

This was there for compatibility with very old monitors.  Modern mons are
always taking a JSON string (which may in turn contain a string, int, or
float).

Signed-off-by: Sage Weil <sage@redhat.com>

mon: use throwing variant of cmd_getval

This ensure we uniformly return EINVAL when the input JSON has a bad
value type.

Signed-off-by: Sage Weil <sage@redhat.com>

common/cmdparse: add cmd_getval_throws variants that throw bad_cmd_get

If input is bad (not missing).

Signed-off-by: Sage Weil <sage@redhat.com>

mon: catch bad_cmd_exception and reply EINVAL

If we get bad input we should always reply EINVAL.  Note that bad_cmd_get
will be reserved for invalid input, not missing/optional input, like
passing a float when an int is expected.

Signed-off-by: Sage Weil <sage@redhat.com>

common/cmdparse: add bad_cmd_get exception

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #23423 from ctriple/master

doc/start: fix kube-helm.rst typo: docuiment -> document

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd: make OSD::HEARTBEAT_MAX_CONN inline

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/start: fix kube-helm.rst typo: docuiment -> document

Signed-off-by: Zhou Peng <p@ctriple.cn>

cephfs-journal-tool: enable purge_queue journal's event commands

Resolves: http://tracker.ceph.com/issues/24604
Signed-off-by: Xuehan Xu <xuxuehan@360.cn>

Merge pull request #23411 from tchaikov/wip-install-py3-for-qa-cephfs

qa: install python3-cephfs for fs suite

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23185 into master

* refs/pull/23185/head:
doc: Create pull_request_template.md

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #23417 into master

* refs/pull/23417/head:
pybind: remove execution permission bits of cephfs.pyx

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

pybind: remove execution permission bits of cephfs.pyx

they were added in c2a671a4b6

Signed-off-by: Kefu Chai <kchai@redhat.com>