The autoscaler by default will start out each pool with minimal
pgs and `scale-up` the pgs when there is more usage in each pool.
Users can now use the commands:
`osd pool set autoscale-profile scale-down` to make the pools
start out with a full complement of pgs and only `scale-down`
when usage ratio across the pools are not even.
`osd pool set autoscale-profile scale-up` (by default) to make the pools
start out with minimal pgs and `scale-up` the pgs when there
is more usage in each pool.
Edited KVMonitor.cc file to make the `autoscale_profile` variable
persistent.
Edited tests/test_cal_final_pg_target.py so that it takes into account
the new `profile` argument when calling cal_final_pg_target(). Also,
added some new test cases for when profile is `scale-up`
Renamed tests/test_autoscaler.py to a more appropriate name:
tests/test_cal_ratio.py
Sage Weil [Wed, 19 May 2021 11:55:30 +0000 (07:55 -0400)]
Merge PR #41286 into master
* refs/pull/41286/head:
qa/suites/orch/rook: disable centos for now
qa/suites/orch/rook/smoke: initial smoke suite
qa/tasks/rook: ROOK_HOSTPATH_REQUIRES_PRIVILEGED=true on centos
qa/tasks/rook: simplify shutdown
qa/tasks/rook: archive logs
qa/tasks/rook: more orderly cluster teardown
qa/tasks/rook: deploy ceph via rook on top of kubernetes
qa/tasks/kubeadm: install kubernetes with kubeadm
qa/suites: move rados/cephadm -> orch/cephadm; symlink
qa/tasks/cephadm: add whitespace between functions
qa/tasks/cephadm: clean up ctx.manager setup
luo rixin [Wed, 19 May 2021 02:27:18 +0000 (10:27 +0800)]
common/crc32c_aarch64: fix crc32c unittest failed on aarch64
On centos 8.2 for aarch64 with gcc 8.3, the complier will use
register v0 conflicting with the register v0 be usded in inline
asm code. Adding the related registers into clobber list to inform
complier avoiding the confict.
Fixes: https://tracker.ceph.com/issues/50835 Signed-off-by: luo rixin <luorixin@huawei.com>
Patrick Donnelly [Tue, 18 May 2021 20:36:05 +0000 (13:36 -0700)]
Merge PR #40885 into master
* refs/pull/40885/head:
doc: document cephfs-mirror configuration options
cephfs-mirror: use sensible mount timeout when mounting local/remote fs
test: add tests for settting mount timeout
pybind/cephfs: add interface to set mount timeout
libcephfs: add interface to set mount timeout
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Sage Weil [Tue, 18 May 2021 14:54:15 +0000 (09:54 -0500)]
qa/tasks/rook: simplify shutdown
For some reason deleting common.yaml sometimes fails. Not really
sure why, but since we will tear down kubernetes anyway this
cleanup isn't really needed.
Sage Weil [Fri, 7 May 2021 18:24:31 +0000 (13:24 -0500)]
qa/tasks/kubeadm: install kubernetes with kubeadm
- install k8s with kubeadm
- initial support for flannel only
- remove taint from bootstrap/master node
- create PVs for all scratch_devs + a 'scratch' SC
- kubeadm.kubectl task
Kefu Chai [Sat, 15 May 2021 04:18:07 +0000 (12:18 +0800)]
crimson/os: define SeastoreCollection as a class
to be consistent with the forward declaration. C++ standard does not
differentiate class from struct in this perspective. but Clang warngs at
seeing it. so silence the warning.
Patrick Donnelly [Tue, 18 May 2021 02:46:56 +0000 (19:46 -0700)]
Merge PR #41239 into master
* refs/pull/41239/head:
librbd: use uint64_t instead of size_t for SparseExtent::length
mgr/PyModule: use Py_ssize_t for the PyList index
os/bluestore: print size_t using %xz
client: print int64_t using PRId64
Reviewed-by: Neha Ojha <nojha@redhat.com> Reviewed-by: Igor Fedotov <ifedotov@suse.com> Reviewed-by: Patrick Donnelly <pdonnell@redhat.com> Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Kefu Chai [Mon, 17 May 2021 17:13:03 +0000 (01:13 +0800)]
crimson/os: use compile-time validation
libfmt does compile-time format argument validation of the format string
and the argument when the the user-defined literal is used. but the
downside is that the formatter materialize the whole formatted string
into a std::string, before printing them argument into seastar's log buffer
inserter. presumably, the inserter would be more efficient in
comparision to the pre-format approach. so this validation is only
enabled for non NDEBUG build. so it is able to help us to identify
errors like
Patrick Donnelly [Mon, 17 May 2021 15:38:41 +0000 (08:38 -0700)]
Merge PR #41314 into master
* refs/pull/41314/head:
qa/tasks/nfs: add test to check if cmds fail on not passing required arguments
mgr/nfs: fix flake8 missing whitespace around parameter equals error
mgr/nfs: annotate _cmd_nfs_* methods return value
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com> Reviewed-by: Sage Weil <sage@redhat.com>
crimson/monc: fix send_message() racing with reopen_session().
The `send_message()` method is a high-level facility for
communicating with a monitor. If there is an active conn
available, it sends the message immediately; otherwise
the message is queued. This method assumes the queue is
already drained if the connection is available.
`active_con` is managed by `reopen_session()` where it's
first cleared and then reset after finding new alive mon.
This is followed by draining the `pending_messages` queue
which happens in `on_session_opened()` after the `MAuth`
exchange is finished.
Unfortunately, the path from the `active_con` clearing
to draining the queue is long and divided into multiple
continuations which results in lack of atomicity. When
e.g. `run_command()` interleaves the stages, following
crash happens:
```
INFO 2021-05-07 08:13:43,914 [shard 0] monc - do_auth_single: mon v2:172.21.15.82:6805/34166 => v2:172.21.15.82:3300/0 returns auth_reply(proto 2 0 (0) Success) v1: 0
ceph-osd: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-3910-g1b18e076/rpm/el8/BUILD/ceph-17.0.0-3910-g1b18e076/src/crimson/mon/MonClient.cc:1034: seastar::future<> crimson::mon::Client::send_message(MessageRef): Assertion `pending_messages.empty()' failed.
Aborting on shard 0.
Backtrace:
0# 0x000055CDE6DB532F in ceph-osd
1# FatalSignal::signaled(int, siginfo_t const*) in ceph-osd
2# FatalSignal::install_oneshot_signal_handler<6>()::{lambda(int, siginfo_t*, void*)#1}::_FUN(int, siginfo_t*, void*) in ceph-osd
3# 0x00007FC1BF20BB20 in /lib64/libpthread.so.0
4# gsignal in /lib64/libc.so.6
5# abort in /lib64/libc.so.6
6# 0x00007FC1BD806B09 in /lib64/libc.so.6
7# 0x00007FC1BD814DE6 in /lib64/libc.so.6
8# crimson::mon::Client::send_message(boost::intrusive_ptr<Message>) in ceph-osd
9# crimson::mon::Client::renew_subs() in ceph-osd
10# 0x000055CDE764FB0B in ceph-osd
11# 0x000055CDE10457F0 in ceph-osd
12# 0x000055CDEA0AB88F in ceph-osd
13# 0x000055CDEA0B0DD0 in ceph-osd
14# 0x000055CDEA2689FB in ceph-osd
15# 0x000055CDE9DC0EDA in ceph-osd
16# main in ceph-osd
17# __libc_start_main in /lib64/libc.so.6
18# _start in ceph-osd
```
The problem caused following failure at Sepia:
http://pulpito.front.sepia.ceph.com/rzarzynski-2021-05-07_07:41:02-rados-master-distro-basic-smithi/6104549
Kefu Chai [Mon, 17 May 2021 04:58:11 +0000 (12:58 +0800)]
cmake: use RELATIVE in file(GLOB_RECURSE..)
less repeating this way. because the consumer of the globbed file list
is add_npm_command(), which in turn calls into add_custom_command(),
and the DEPENDS specified by add_custom_command() checks for the files
using relative paths, it's safe to have this change.
Xuehan Xu [Thu, 11 Mar 2021 05:21:31 +0000 (13:21 +0800)]
crimson/osd: optimize crimson-osd's client requests process parallelism
Make client requests go to the concurrent pipeline stage "wait_repop" once they
are "submitted" to the underlying objectstore, which means their on-disk order
is guaranteed, so that successive client requests can go into the "process"
pipeline stage.
projects / ceph.git / log