Jason Dillaman [Wed, 6 Apr 2016 20:08:15 +0000 (16:08 -0400)]
rbd-mirror: workaround for intermingled lockdep singletons
If librados constructs the CephContext, lockdep will be initialized
in both the librados shared address space and also within the app
address space. This breaks lockdep's functionality and results in
a crash upon shutdown.
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
Jason Dillaman [Wed, 6 Apr 2016 21:18:33 +0000 (17:18 -0400)]
journal: Future does not require metadata shared pointer
Avoid keeping the metadata object alive through the lifespan of the
future. Callers can expect to receive a re-entrant callback if the
future is already safe and flush/wait is invoked.
Fixes: http://tracker.ceph.com/issues/15364 Signed-off-by: Jason Dillaman <dillaman@redhat.com>
Sage Weil [Wed, 6 Apr 2016 01:26:06 +0000 (21:26 -0400)]
os/bluestore: use short, relative paths with bluefs
If we're using bluefs, only pass in the short relative
path (db, db.wal, db.slow). The leading components
are ignored and only lead to errors if the configuration
provides relative paths that do not match (e.g., if one
if using ceph-objectstore-tool).
Fixes: http://tracker.ceph.com/issues/15376 Signed-off-by: Sage Weil <sage@redhat.com>
Gentoo's normal build process uses a sandbox to catch writes outside the
build environment; this includes providing a value other than /tmp for
TMPDIR. Use TMPDIR by default for CEPH_BUILD_VIRTUALENV.
Signed-off-by: Robin H. Johnson <robin.johnson@dreamhost.com>
Sage Weil [Tue, 5 Apr 2016 20:37:25 +0000 (16:37 -0400)]
mds: fix legacy layout decode with pool 0
If you data pool was pool 0, this was transforming
that to -1 unconditionally, which broke upgrades. We
only want do that for a fully zeroed ceph_file_layout,
so that it still maps to a file_layout_t. If any fields
are set, though, we trust the fl_pgpool to be a valid
pool.
Sage Weil [Mon, 4 Apr 2016 18:39:18 +0000 (14:39 -0400)]
os/bluestore/BlueFS: revamp bdev ids
You cannot tell from the old bdev vector which device
was which.
- use a fixed id for each type/slot
- go from fast(small) to slow(big)
- normalize the allocation fallback to try any slower
device.
- clean up the BlueStore instantiation/setup accordingly
Erwan Velu [Thu, 24 Mar 2016 14:26:55 +0000 (15:26 +0100)]
tests: Removing one ceph-dencoder call in check-generated.sh
The first ceph-dencoder call is very unlikely to fail and represent a
bottleneck as the parallel computation are only done one this test is
completed.
The idea of this patch is to run immediately the 4 dencoders process in
parallel and check the resulting error code. If one fail then we report the
failure.
As the failure is very unlikely, that saves time and makes the code
simplier too.
Erwan Velu [Wed, 23 Mar 2016 10:13:48 +0000 (11:13 +0100)]
tests: Avoiding a fixed 10sec sleep in test_mon_cephdf_commands()
The current code was waiting 10s to expect the file being put.
If the file was put in a shorter time than 10s, the test just waits for
nothing reducing the execution speed of that test.
This patch simply check if the file is actually available every second
during 10sec to exit prematurely.
This patch saves exactly 10 sec on a local system, surely a little bit
less on an infra but still saves time. Signed-off-by: Erwan Velu <erwan@redhat.com>
Erwan Velu [Wed, 23 Mar 2016 09:45:33 +0000 (10:45 +0100)]
tests: Optmizing sleep sequence in cephtool/test.sh
The actual code double the wait time between two calls leading to a
possible 511s of waiting time which sounds a little bit excessive.
This patch offer to reduce the global wait time to 300s and test more
often the rados status to exit the loop earlier. In a local test, that
saves 6 secs per run. Signed-off-by: Erwan Velu <erwan@redhat.com>
Erwan Velu [Tue, 22 Mar 2016 10:20:24 +0000 (11:20 +0100)]
tests: Reducing sleep loops in ceph_objectstore_tool
This python script is making excessive sleep calls while running some
ceph commands.
Waiting up to 5 seconds to get the proper health status can be shorten
to avoid the worst case of waiting almost 5 seconds for nothing.
This patch removes also two sleeps calls after a wait_for_health call
which is already supposed to provides a clean state. Waiting
respectively 20 & 15 seconds after that call is just loosing time which
is precious at make check time.
Erwan Velu [Mon, 21 Mar 2016 15:20:25 +0000 (16:20 +0100)]
tests: Reducing sleep time for osd to be up
OSDs are taking some time to be up but waiting 10 secs seems execessive
here between two loops. In the worst case, we can be in a situation of
waiting 10secs for nothing as we are just a few microsecs after the osd
is up.
This patch simply reduce the sleep from 10 to 1 seconds. Signed-off-by: Erwan Velu <erwan@redhat.com>
Erwan Velu [Mon, 21 Mar 2016 10:47:51 +0000 (11:47 +0100)]
tests: Making "objectstore" calls parallel in osd-scrub-repair.sh
osd-scrub-repair is making several similar objectore calls in a
sequential way while they could be easily parallelized.
Each single objectore call can spent up to dozen of seconds so making
the call parallel is saving a lot of time while keeping the code pretty
simple.
This particular patch saves approx. 2 minutes on the actual code on a recent
laptop. The global running time of osd-scrub-repair drops from 9m33 to
7m37 ! Signed-off-by: Erwan Velu <erwan@redhat.com>
Erwan Velu [Mon, 21 Mar 2016 10:30:13 +0000 (11:30 +0100)]
tests: Optimizing wait_for_clean()
wait_for_clean() is a very common call when running the make check.
It does wait the cluster to be stable before continuing.
This script was doing the same calls twice and could be optimized by
making the useful calls only once.
is_clean() function was checking num_pgs & get_num_active_clean()
The main loop itself was also calling get_num_active_clean()
This patch is inlining the is_clean() inside this loop to benefit from a
single get_num_active_clean() call. This avoid a useless call of (ceph +
xmlstarlet).
This patch does move all the 'timer reset' conditions into an else
avoiding spawning other ceph+xmlstarlet call while we already know we
should reset the timer.
The last modification is to reduce the sleeping time as the state of the
cluster is changing very fast.
This whole patch could looks like almost not a big win but for a test
like test/osd/osd-scrub-repair.sh, we drop from 9m56 to 9m30 while
reducing the number system calls.
At the scale of make check, that's a lot of saving.
Erwan Velu [Mon, 21 Mar 2016 10:16:12 +0000 (11:16 +0100)]
tests: Reducing commands in get_num_active_clean()
get_num_active_clean() is called very often but spawn 1 useless process.
The current "grep -v | wc -l" can be easily replaced by "grep -cv" which
do the same while spawning one process less.
Erwan Velu [Wed, 16 Mar 2016 13:24:04 +0000 (14:24 +0100)]
tests: Adding parallelism for sequential ceph-dencoder calls
The current code was running sequentially two ceph-dencoder calls.
This process is executed pretty fast but adding sequentiality and by the number
of loops to execute, it have a cost.
This patch is just making this two calls being run in parallel.
As a result, the test/encoding/readable.sh test is running in 4m50 instead of 6.
The associate loadavg isn't impacted as it stays at 6 while being run with
nproc=8.
This patch save 1/6th of building time without impact the loadavg.
Erwan Velu [Tue, 15 Mar 2016 15:00:17 +0000 (16:00 +0100)]
tests: Adding parallelism to encoding/readable.sh
When running make -j x check, we face a weird situation where the makefile
targets are spawn in parallel up to "x" but one of those target is very very
long and sequential.
The "readable.sh" test is trying to run ~7.9K tests where 5.3K are actually
executed.
The current code is taking 23mn on a recent laptop (Intel(R) Core(TM)
i7-4810MQ CPU @ 2.80GHz, 32GB of RAM & SSD).
This patch implements parallelism to speed up this process which is not really CPU and
neither IO bound.
By default, readable.sh is now using the number of logical processors to determine
the level of parallelism (by using nproc). If needed, defining the MAX_PARALLEL_JOBS
variable will override this default value.
On the same system, where nproc=8, the resulting execution time is 5m55 seconds :
4x faster than the original code.
The global 'make check' is therefore getting faster too and dropped from 30 to
16 minutes : 2x faster than the original code.
Sage Weil [Mon, 4 Apr 2016 21:14:33 +0000 (17:14 -0400)]
global/global_init: expand metavariables in setuser_match_path
Back in 8290536d7d373ea6dddd8cdb55ac71e42b5278e2 we moved the
apply_changse (and, indirectly, config var expansion) to happen
after set do the drop privileges, but we need the metavar
expansion for setuser_match_path (which docs suggest setting to
/var/lib/ceph/$type/$cluster-$id).
Fixes: http://tracker.ceph.com/issues/15365 Signed-off-by: Sage Weil <sage@redhat.com>
Sage Weil [Wed, 9 Mar 2016 15:04:14 +0000 (10:04 -0500)]
osdc/Objecter: fix narrow race with tid assignment
The operation completion could finish and be freed before we
do the info->register_tid assignment. Avoid this by doing the
assignment in _op_submit itself.
Fixes: #14364 Signed-off-by: Sage Weil <sage@redhat.com>
Sage Weil [Mon, 4 Apr 2016 12:39:32 +0000 (08:39 -0400)]
ceph-rest-api: do not include single-option CephChoices in prefix
This breaks commands like
COMMAND("fs flag set name=flag_name,type=CephChoices,strings=enable_multiple "
"name=val,type=CephString", \
"Set a global CephFS flag", \
"fs", "rw", "cli,rest")
with only one option:
PUT fs/flag/set?flag_name=enable_multiple&val=true: 400
FAILURE: url http://localhost:5000/api/v0.1/fs/flag/set?flag_name=enable_multiple&val=true
expected 200, got 400
Response content: <html><body><table border=1><th>Possible commands:</th><th>Method</th><th>Description</th><tr><td>fs/flag/set?flag_name=enable_multiple&va
l=val(<string>)
</td><td>PUT</td><td>Set a global CephFS flag</td></tr>
</table></body></html>
projects / ceph.git / log