from Elasticsearch 5.0+ the type string is deprecated, while it is still allowed
in 5.0, it returns a 400 error on 6.0 clusters as this type is fully removed. We
now determine the es version while initializing the cluster from elasticsearch's
default endpoint and use that to determine what string type to use. This way we
support both 2.x and 5.x,6.x es versions as we default to string type for
clusters < 5.0
rgw: es: introduce new classes for storing ES version info
Getting the basic information from elasticsearch, while we're only currently
interested in elasic search version, other data such as the ES cluster name etc
may be useful in the future as well.
This commit introduces an enum mapping various ES types. Also encode_json now
has a new member for string type so that this can be changed easily when
upgrading ES versions for eg. wherein new ES engines do not support the string
type anymore.
Tiago Melo [Fri, 11 Jan 2019 12:08:01 +0000 (11:08 -0100)]
mgr/dashboard: Fix summary refresh call stack
Currently each time we created a task it would call the summary refresh method.
Since that method was a recursive method, overtime we would get N calls to the
refresh each 5 seconds.
alfonsomthd [Tue, 22 Jan 2019 12:26:38 +0000 (13:26 +0100)]
mgr/dashboard: added command to tox.ini
* Added 'run' command to run arbitrary commands so
you don't have to run the whole test/lint suite.
Use case examples:
- Run only 1 test:
tox -e py3-run pytest tests/test_rgw_client.py::RgwClientTest::test_ssl_verify
- Run pycodestyle for only 1 file:
tox -e py3-run pycodestyle tests/test_rgw_client.py
Signed-off-by: Alfonso Martínez <almartin@redhat.com>
Kefu Chai [Sat, 19 Jan 2019 16:01:55 +0000 (00:01 +0800)]
tools/ceph_kvstore_tool: extract StoreTool into kvstore_tool.cc
to workaround n bug in GCC 7.3, which leads to ICE like
build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.0.1-2751-gb268822/rpm/el7/BUILD/ceph-14.0.1-2751-gb268822/src/tools/ceph_kvstore_tool.cc:625:1:
internal compiler error: Segmentation fault
}
^
Please submit a full bug report,
with preprocessed source if appropriate.
See <http://bugzilla.redhat.com/bugzilla> for instructions.
xie xingguo [Fri, 18 Jan 2019 10:14:52 +0000 (18:14 +0800)]
osd/OSDMap: be more aggressive when trying to balance
Previously we'd require the absolute deviation >= 1 before
an osd can fill in the overfull or underfull set, which as
a result can get some osds stuck severely underfull forever.
This patch tries to get us out of those corner cases mentioned
above and can be scrutinized from two aspects:
- a standard deviation is introduced to evaluate the efficiency
of balancing, therefore making the distribution of pgs always
converge to the perfect status (standard deviation == 0).
- populate overfull or underfull osds more aggressively by
gradually allowing the absolute deviations converging towards
to 0 instead of 1.
It turns out the balancer module works even better now after
applying this patch. E.g.:
```
OSD=5 MON=1 MGR=1 MDS=0 ../src/vstart.sh -x -l -b -n -d
bin/ceph osd set-require-min-compat-client luminous
bin/ceph balancer mode upmap
bin/ceph balancer on
bin/ceph osd pool create rbd 117
// wait until automatic balancing is done
bin/ceph osd pool create aaa 133
```
__before__:
```
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS
0 hdd 0.00980 1.00000 10 GiB 1.1 GiB 3.9 MiB 0 B 1 GiB 9.0 GiB 10.60 1.00 151 up
1 hdd 0.00980 1.00000 10 GiB 1.1 GiB 3.9 MiB 0 B 1 GiB 9.0 GiB 10.60 1.00 153 up
2 hdd 0.00980 1.00000 10 GiB 1.1 GiB 3.9 MiB 0 B 1 GiB 9.0 GiB 10.60 1.00 147 up
3 hdd 0.00980 1.00000 10 GiB 1.1 GiB 3.9 MiB 0 B 1 GiB 9.0 GiB 10.60 1.00 149 up
4 hdd 0.00980 1.00000 10 GiB 1.1 GiB 3.9 MiB 0 B 1 GiB 9.0 GiB 10.60 1.00 150 up
TOTAL 50 GiB 5.3 GiB 19 MiB 0 B 5 GiB 45 GiB 10.60
```
__after__:
```
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS
0 hdd 0.00980 1.00000 10 GiB 1.1 GiB 6.2 MiB 0 B 1 GiB 9.0 GiB 10.62 1.00 150 up
1 hdd 0.00980 1.00000 10 GiB 1.1 GiB 6.2 MiB 0 B 1 GiB 9.0 GiB 10.62 1.00 151 up
2 hdd 0.00980 1.00000 10 GiB 1.1 GiB 6.2 MiB 0 B 1 GiB 9.0 GiB 10.62 1.00 149 up
3 hdd 0.00980 1.00000 10 GiB 1.1 GiB 6.2 MiB 0 B 1 GiB 9.0 GiB 10.62 1.00 150 up
4 hdd 0.00980 1.00000 10 GiB 1.1 GiB 6.2 MiB 0 B 1 GiB 9.0 GiB 10.62 1.00 150 up
TOTAL 50 GiB 5.3 GiB 31 MiB 0 B 5 GiB 45 GiB 10.62
```
Sage Weil [Fri, 18 Jan 2019 15:38:59 +0000 (09:38 -0600)]
Merge PR #25132 into master
* refs/pull/25132/head:
os/bluestore: implement kv update on bluefs extents info change.
os/bluestore: use original algorithm to calculate BlueFS gift size for explicit allocation.
os/bluestore: get rid of bluefs space gifting performed in background
os/bluestore: do KV/FM/allocator opening in two stages.
os/bluestore: detach FreelistManager from DB instance.
test/osd/store_test: add test case for explicit bluefs expanding.
os/bluestore: implement explicit bluefs space allocation at slow device
os/bluestore: unify allocator dump on bluefs space failures
os/bluestore: stop keeping bluefs_extents at KV
Kefu Chai [Fri, 11 Jan 2019 10:47:39 +0000 (18:47 +0800)]
crimson/osd: enable crimson-osd to boot
* add state.h to encapsulate the state represeting different stages
related to booting an OSD. the boot process of an OSD can be blocked
by
- waiting for PG consuming updated osdmaps
- waiting for osdmaps marking osd.{whoami} up
- waiting for new osdmaps to bring this osd up to speed.
- waiting for current OSD to be healthy
we could chain these "waits" in a more seastarized way, and let
OSD::start() wait on the future returned by this chain. but that'd
requires adding some seastar::shard_future<> as member variables of
`OSD` class, which is a little bit more convoluted than the state
machine approach used in this change. we could switch over to the
`future<>` chain approach, if we found that these futures could have
more consumers than merely `OSD::start()`.
* all osdmaps are now stored in an `std::map` in `OSD`, we can
improve it by
- caching it using an LRU cache
- trimming the stale ones
- persisting the evicted maps into the meta collection in ObjectStore
* superblock is not persited to store, neither is it read from the
store.
Kefu Chai [Thu, 10 Jan 2019 13:15:18 +0000 (21:15 +0800)]
crimson: set src for message
monitor will panic at seeing a MOSDBoot message which is not from an
OSD. see OSDMonitor::preprocess_boot()
```
ceph_assert(m->get_orig_source_inst().name.is_osd());
```
/home/jenkins-build/build/workspace/ceph-pull-requests-arm64/src/seastar/fmt/include/fmt/format.h:2120:14:
error: comparison of integer expressions of different signedness: 'const
wchar_t' and 'char' [-Werror=sign-compare]
if (*out == value)
~~~~~^~~~~~~~
cc1plus: all warnings being treated as errors
where libfmt compares a wchar_t with a literal '}', which is char.
because the former is unsigned, and the latter is of a signed type,
GCC is annoyed. but since both of them are ASCII, and when performing
comparison, the signed one is converted to unsigned, then result of
comparison is correct per-se. hence, it's safe to silence this very
warning.
and warning like:
/home/jenkins-build/build/workspace/ceph-pull-requests-arm64/src/seastar/src/core/future-util.cc:61:5:
required from 'seastar::future<> seastar::sleep_abortable(typename
Clock::duration, seastar::abort_source&) [with Clock =
std::chrono::_V2::steady_clock; typename Clock::duration =
std::chrono::duration<long int, std::ratio<1, 1000000000> >]'
/home/jenkins-build/build/workspace/ceph-pull-requests-arm64/src/seastar/src/core/future-util.cc:68:105:
required from here
/home/jenkins-build/build/workspace/ceph-pull-requests-arm64/src/seastar/src/core/future-util.cc:48:28:
error: 'seastar::sleep_abortable(typename Clock::duration,
seastar::abort_source&)::sleeper::sleeper(typename Clock::duration,
seastar::abort_source&) [with Clock = std::chrono::_V2::steady_clock;
typename Clock::duration = std::chrono::duration<long int, std::ratio<1, 1000000000> >]::<lambda()>' declared with greater visibility than the
type of its field 'seastar::sleep_abortable(typename Clock::duration,
seastar::abort_source&)::sleeper::sleeper(typename Clock::duration,
seastar::abort_source&) [with Clock = std::chrono::_V2::steady_clock;
typename Clock::duration = std::chrono::duration<long int, std::ratio<1, 1000000000> >]::<lambda()>::<this capture>' [-Werror=attributes]
: tmr([this] { done.set_value(); }) {
^ Signed-off-by: Kefu Chai <kchai@redhat.com>
Kefu Chai [Fri, 21 Dec 2018 10:34:22 +0000 (18:34 +0800)]
test/crimson/monc: start/stop perf counter
in CephContext::CephContext(), we assume that
ceph::common::local_perf_coll() is ready when a CephContext is to be
contructed. so we need to prepare start it before creating CephContext.
projects / ceph.git / log