script/run-cbt.sh: add support for ceph-osd testing.
The patch brings `--classical` switch to the `run-cbt.sh`
script. Its purpose is to automate the apple-to-apple
comparison between OSD implementations. In both cases
memstore is selected and the same benchmarks are used.
test/crimson: cbt test does rand-reads instead of seq-reads.
The rationale behind the change is not only that random
read performance tends to be more important and meaningful.
Actually, the current procedure is broken because of how
`rados bench` internally operates: the number of seq read
operations is limited by the number of write ops used to
create the data set (`write` with the `--no-cleanup` option):
```cpp
int ObjBencher::write_bench(/* ... */) {
// ...
//write object size/number data for read benchmarks
encode(data.object_size, b_write);
encode(data.finished, b_write);
This makes significant problem as the cbt test uses short (3 sec.)
period of prefill. In the consequence, the sequential read testing
takes around half-second.
* refs/pull/30476/head:
doc: list FS subvolumes, subvolume groups and their snapshots
qa/tasks: tests for ls
mgr/volumes: list FS subvolumes, subvolume groups and their snapshots
Reviewed-by: Ramana Raja <rraja@redhat.com> Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
It turns out this change isn't actually needed to make ceph-daemon do its thing, since
ceph-volume alrady skips doing anything with tmpfs if the data dir already exists.
Jos Collin [Fri, 27 Sep 2019 11:42:34 +0000 (17:12 +0530)]
qa/tasks: tests for ls
tests for:
* fs subvolume ls
* 'fs subvolume ls' for non-existent default group
* fs subvolumegroup ls
* 'fs subvolumegroup ls' for not-existent /volume
* fs subvolume snapshot ls
* fs subvolumegroup snapshot ls
Fixes: https://tracker.ceph.com/issues/41842 Signed-off-by: Jos Collin <jcollin@redhat.com>
Sage Weil [Mon, 7 Oct 2019 20:31:14 +0000 (15:31 -0500)]
Merge PR #30603 into master
* refs/pull/30603/head:
ceph-daemon: -n type.id instead of -i id
ceph-daemon: drop unused VERSION
ceph-daemon: clean up dir helpers, tighten up permissions
ceph-daemon: fchmod before writing to keyring file
test_ceph_daemon.sh: skip ssh until container image has remoto
ceph-daemon: decode utf-8 in run() helper
mgr/ssh: clean up debug cruft
mgr/ssh: clean up bare except: block
ceph-daemon: clean up bare except: blocks
ceph-daemon: all imports to top
ceph-volume: no_tmpfs -> tmpfs
doc/bootstrap: add new bootstrap documentation
ceph-daemon: add --output-pub-ssh-key for bootstrap
ceph-daemon: make 'shell' easier to use
ceph-daemon: support docker; prefer podman
qa: add ceph-daemon
debian: ceph-daemon package, required by ceph-mgr-ssh
ceph.spec.in: ceph-daemon package, required by ceph-mgr
common/options: cleanup whitespace
mgr/ssh: simplify getting the cluster fsid
mgr/ssh: pipe ceph-daemon script to stdin of python3
ceph-daemon: add support for args and/or stdin from top of script
ceph-daemon: make ceph-volume use get_config_and_keyring
ceph-daemon: ls: behave if /var/log/ceph doesn't exist
ceph-daemon: implement 'adopt' for legacy style daemons
ceph-daemon: fix fsid detection for legacy osds
ceph-daemon: make rm-cluster clean up system-ceph*.slice too
ceph-daemon: configure ssh orchestrator
ceph-daemon: be more restrictive with file permissions
mgr/ssh: create osd with ceph-daemon
mgr/ssh: pass daemon id separately to _create_daemon
ceph-daemon: add --config-and-keyring to ceph-volume command
ceph-daemon: create log path for shell (if needed)
mgr/ssh: use _run_ceph_daemon for _create_daemon
mgr/ssh: factor _run_ceph_daemon out of _get_device_inventory
mon/ConfigMonitor: allow entity type only for 'config get'
ceph-daemon: add ceph-volume subcommand
ceph-daemon: remove unused CephContainer dname property
ceph-daemon: drop useless uid/gid checks
mgr/ssh: deploy new mgrs with ceph-daemon
mgr/ssh: factor _create_daemon out of create_mon
mon/MonCap: allow mgr to create new auth keys
mgr/ssh: run c-v with podman when getting inventory
mgr/ssh: simplify ssh connection management
mgr/ssh: use ceph-daemon for deploying mon
ceph-daemon: allow --mon-network for deploying new mon (vs specifying IP)
ceph-daemon: --config-and-keyring (not key)
common/options: add 'image' config option
test_ceph_daemon: specify image name
vstart.sh: add --ssh to enable+configure ssh orchestrator
mgr/ssh: use ssh identity from config-key, if present
mgr/ssh: hardcode default ssh_config
ceph-daemon: store ssh identity in mon config-key store
ceph-daemon: --privileged arg for 'exec'
ceph-daemon: make deploy work for osd (do a c-v prepare)
ceph-daemon: make shell privileged
ceph-daemon: move get_container_mounts to a helper
ceph-daemon: pass full path for entrypoint
ceph-daemon: make id portion of 'shell' optional
ceph-volume: accept --no-tmpfs argument for bluestore
ceph-daemon: 'unit' command
ceph-daemon: fix run command to use call(), not check_output()
src/ceph-daemon: whitespace
ceph-daemon: add 'enter', 'exec' commands
ceph-daemon: bind config to default location
test_ceph_daemon.sh: test deploy mds too
ceph-daemon: generate ssh keys
ceph-daemon: --config, not --conf
ceph-daemon: long lines
ceph-daemon: add --config to bootstrap
ceph-daemon: add 'shell' command
ceph-daemon: do not import subprocess symbols directly
ceph-daemon: add mons with 'deploy mon.x ...'
ceph-daemon: add 'ls'
ceph-daemon: simplify uid/gid a bit
ceph-daemon: fix libudev
ceph-daemon: autodetect uid/gid from container image
ceph-daemon: default to empty log files, log to stderr (systemd journal)
ceph-daemon: rm-{daemon,cluster}
ceph-daemon: fix bootstrap config
ceph-daemon: fix args.fsid usage
ceph-daemon: be careful overwriting live files
ceph-daemon: slurp some options over from the standard systemd unit
ceph-daemon: add ceph.target and ceph-$fsid.target units
test_ceph_daemon.sh: stupid test script
ceph-daemon: bootstrap and deploy (mgr) work
ceph-daemon: initial checkin
ceph-mon: fix debug print of public_addr
Sage Weil [Sun, 6 Oct 2019 14:08:28 +0000 (09:08 -0500)]
Merge PR #30217 into master
* refs/pull/30217/head:
crimson: common/admin_socket kludge so that it builds
mon/MonClient: fix sending mon command to a specific rank
src/.gitignore: ignore .tox
mon/MonClient: interpret numeric mon target name as rank
mgr,mgr/MgrClient: use fsid to signal mon-mgr vs cli MCommands
qa/workunits/cephtool: fix errpr checks for 'ceph daemon' commands
common/ceph_context: make 'config unset' idempotent
qa/tasks/dump_stuck: mon.a, not mon.0
qa/suites/rados/singleton/all/admin-socket: fix test
common/config: EPERM setting config option after startup
qa/workunits/cephtool/test.sh: fix tell output error check
common/admin_socket: pass Formatter from generic infrastructure
common/admin_socket: pass ostream to call() for error output
os/bluestore: fix asok hook return value
rgw: fix asok return value
common/ceph_context: return error code from asok commands
test/pybind/test_rados: fix accidental mon tell test
mon: print entity_name along with caps to debug log
PendingReleaseNotes: notes about asok changes
mgr/MgrClient: empty target string for 'tell' means active mgr
common/admin_socket: report error code as part of output string
osd: change trigger_[deep_]scrub tommands to a pg tell command
osd: remove old command workqueue, threadpool
osd: drop MMonCommand handling
osdc/Objecter: resend OSD tell commands on EAGAIN
osd: route tell commands to asok; migrate commands
osd: use unique_ptr<Formatter> for asok_command
common/ceph_context: add generic asok 'injectargs'
common/admin_socket: allow dup prefixes
common/admin_socket: refactor with sync and async execute_command variants
common/admin_socket: pass input bufferlist
osd: transition to call_async() for asok
common/admin_socket: support alternative call_async()
mon/MonClient: send tell commands out of band via MCommand
mon: accept tell commands via MCommand and send them to asok handler
common/admin_socket: return int from hook call()
mgr/DaemonServer: route MCommand (for octopus+) to asok commands
do not use 'ceph tell mgr'
pybind/ceph_argparse: disambiguate mgr tell and CLI commands
ceph: make 'ceph tell mgr.*' send to the active mgr
ceph: send 'ceph tell mgr.X' to the right mgr
librados: add rados_mgr_command_target
mgr/MgrClient: add start_command variant that takes a target
common/admin_socket: drop unregister_command(); use per-hook variant
common/admin_socket: drop explicit prefix arg to register_command
common/admin_socket: simplify command routing
common/admin_socket: add ability to process MCommand via asok queue
common/admin_socket: pass cmdvec to execute_command
common/admin_socket: use pipe for general wakeup
include/compat: add flags arg to pipe_cloexec
common/admin_socket: drop unused args
Sage Weil [Fri, 4 Oct 2019 21:59:24 +0000 (16:59 -0500)]
ceph-daemon: clean up dir helpers, tighten up permissions
- don't pass args.{data,log}_dir to get_{data,log}_dir
- pass uid, gid, and mode to makedirs
- add make_data_dir and make_log_dir helpers, that optionally take uid/gid
- add make_data_dir_base that requires uid/gid
- use standard data and log dir modes, defined at top of file (0o700 for
data, 0o770 for logs). The data dir mode applies both to the fsid
directory for the whole cluster and to each daemon's subdirectory; the
log mode applies only to the fsid dir for the whole cluster, where
all daemon logs are combined together in one directory.
Sage Weil [Sat, 5 Oct 2019 01:42:07 +0000 (20:42 -0500)]
Merge PR #29035 into master
* refs/pull/29035/head:
mgr/pg_autoscaler: changes made reflect jdurgin's request
mgr/pg_autoscaler: current pg_num compared to distance between the last initial_pg_num
mgr/progress & mgr/pg_autoscaler: changes reflect liewegas' comment
mgr/pg_autoscaler: get rid of white space
mgr/progress: change threshold value to origin
mgr/progress: cleaning up for pg_autoscaler
mgr/progress: Added Pg Autoscaler Event
Sage Weil [Thu, 3 Oct 2019 21:13:47 +0000 (16:13 -0500)]
mgr/ssh: pipe ceph-daemon script to stdin of python3
This avoids any need for the script to be present on the remote host.
We introduce a config option to indicate where the script should be
read from, since the location varies between a vstart environment (source
dir) and a real install (/usr/sbin).
Sage Weil [Wed, 2 Oct 2019 21:19:14 +0000 (16:19 -0500)]
ceph-daemon: implement 'adopt' for legacy style daemons
Caveats:
- this doesn't clean out /etc/ceph/*, since we don't know which is the
last daemon to go, and/or whether the user wants to keep it around
for using the ceph CLI on this host
- leaves behind /var/lib/ceph/bootstrap-* keys, even after all daemons
have been converted.
Sage Weil [Tue, 1 Oct 2019 16:34:22 +0000 (11:34 -0500)]
mon/ConfigMonitor: allow entity type only for 'config get'
e.g., 'ceph config get osd debug_osd' to return the config value that
would apply to a generic OSD (either from the osd or global section of
the config).
Sage Weil [Tue, 1 Oct 2019 15:29:53 +0000 (10:29 -0500)]
mgr/ssh: simplify ssh connection management
- Use a single instance of the config and identity files for the whole
module. There's no need to create these for *every* connection--it just
pollutes /tmp.
- Drop the SSHConnection wrapper, since the temp files are tied to the
daemon lifecycle now.
- Prefix the tmp files so I can tell wtf is going on.
- Always connect to root@host, to avoid remoto's localhost detection
feature. This ensures we have a consistent connection model and user.
(The daemon might be running as user ceph and try to connect to localhost,
but end up running the command as the wrong user and/or inside the
ceph-mgr container.)
projects / ceph.git / log