]> git.apps.os.sepia.ceph.com Git - ceph.git/log
ceph.git
4 years agocmake: add unittest_seastar_config as a ceph test 37073/head
Kefu Chai [Thu, 10 Sep 2020 03:53:08 +0000 (11:53 +0800)]
cmake: add unittest_seastar_config as a ceph test

Signed-off-by: Kefu Chai <kchai@redhat.com>
4 years agotest/crimson: fix racing in testing
Kefu Chai [Thu, 10 Sep 2020 03:41:27 +0000 (11:41 +0800)]
test/crimson: fix racing in testing

before this change, unittest_seastar_config set the specified option to
the shard id in parallel and expects that the values of the option are
consistent across all shards after setting the value on all shards. but
sharded::invoke_on_all() is executed in parallel, and so does
ConfigProxy::do_change(), so there are actually "n x n" continuations
racing each other. the order of the continuations storing the setting
cannot be determined. so we cannot expected that the last batch of
continuations hitting shareds store the same value.

in this change, only a single `conf.set_val()` call is performed with
a known value. and this value is checked with the values stored on all
shards.

Signed-off-by: Kefu Chai <kchai@redhat.com>
4 years agotest/crimson: use seastar::make_ready_future()
Kefu Chai [Thu, 10 Sep 2020 02:58:40 +0000 (10:58 +0800)]
test/crimson: use seastar::make_ready_future()

instead of seastar::now(). as seastar::now() was moved into
seastar/util/laster.hh.

this change is a leftover of aa28b51f3c5434253c1788927100fb0b6e41bdbb

Signed-off-by: Kefu Chai <kchai@redhat.com>
4 years agoMerge pull request #36962 from tchaikov/wip-qa-py3-cleanup
Kefu Chai [Thu, 10 Sep 2020 01:39:20 +0000 (09:39 +0800)]
Merge pull request #36962 from tchaikov/wip-qa-py3-cleanup

qa: py3 cleanups

Reviewed-by: Neha Ojha <nojha@redhat.com>
4 years agoMerge pull request #37054 from cbodley/wip-47306
Casey Bodley [Wed, 9 Sep 2020 21:11:43 +0000 (17:11 -0400)]
Merge pull request #37054 from cbodley/wip-47306

rgw: ObjectCache::put() clears stale objv

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Mark Kogan <mkogan@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
4 years agoMerge pull request #36619 from cbodley/wip-46734
Matt Benjamin [Wed, 9 Sep 2020 19:57:24 +0000 (15:57 -0400)]
Merge pull request #36619 from cbodley/wip-46734

dmclock: update submodule to fix hang on thread shutdown

4 years agoMerge pull request #35548 from adamemerson/wip-datalog-fifo
Matt Benjamin [Wed, 9 Sep 2020 19:56:26 +0000 (15:56 -0400)]
Merge pull request #35548 from adamemerson/wip-datalog-fifo

FIFO the Datalog!

4 years agoMerge pull request #36964 from trociny/wip-47275
Jason Dillaman [Wed, 9 Sep 2020 15:32:06 +0000 (11:32 -0400)]
Merge pull request #36964 from trociny/wip-47275

mgr: don't update pending service map epoch on receiving map from mon

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
4 years agoMerge pull request #34777 from nSedrickm/grafana-iframe
Lenz Grimmer [Wed, 9 Sep 2020 09:10:20 +0000 (11:10 +0200)]
Merge pull request #34777 from nSedrickm/grafana-iframe

mgr/dashboard: increase Grafana iframe height to avoid scroll bar

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
4 years agoMerge pull request #37057 from tchaikov/wip-bump-seastar
Kefu Chai [Wed, 9 Sep 2020 07:36:04 +0000 (15:36 +0800)]
Merge pull request #37057 from tchaikov/wip-bump-seastar

seastar,crimson: bump seastar to use upstream master

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
4 years agoMerge pull request #37042 from xxhdx1985126/wip-47357
Kefu Chai [Wed, 9 Sep 2020 05:48:46 +0000 (13:48 +0800)]
Merge pull request #37042 from xxhdx1985126/wip-47357

crimson/osd: avoid accessing WaitForObjectRecovery::pi when no pulling occurred

Reviewed-by: Kefu Chai <kchai@redhat.com>
4 years agoseastar: bump seastar to use upstream master 37057/head
Kefu Chai [Wed, 9 Sep 2020 04:00:41 +0000 (12:00 +0800)]
seastar: bump seastar to use upstream master

before this change we are using a fork which includes a fix to address
the FTBFS with GnuTLS < 3.5.6

after this change, we are using the lastest upstream which already
contains this fix.

Signed-off-by: Kefu Chai <kchai@redhat.com>
4 years agocrimson/osd: print notify_id instead of its address
Kefu Chai [Wed, 9 Sep 2020 03:46:01 +0000 (11:46 +0800)]
crimson/osd: print notify_id instead of its address

this addresses the FTBFS like

    /usr/include/fmt/core.h:902:19: error: static assertion failed: formatting of non-void pointers is disallowed
      902 |     static_assert(!sizeof(T), "formatting of non-void pointers is disallowed");
          |                   ^~~~~~~~~~

also, i think it is more helpful in this context to have the
notify id instead of the address, as we can use the id to reference
the notification printed by other logging messages.

Signed-off-by: Kefu Chai <kchai@redhat.com>
4 years agocrimson/os: cast pointers to void* before printing them
Kefu Chai [Wed, 9 Sep 2020 03:26:58 +0000 (11:26 +0800)]
crimson/os: cast pointers to void* before printing them

to address the FTBFS like:

../src/crimson/os/seastore/lba_manager/btree/btree_range_pin.cc:110:14:   required from here
/usr/include/fmt/core.h:902:19: error: static assertion failed: formatting of non-void pointers is disallowed
  902 |     static_assert(!sizeof(T), "formatting of non-void pointers is disallowed");
      |                   ^~~~~~~~~~

Signed-off-by: Kefu Chai <kchai@redhat.com>
4 years agocrimson/os: instantiate future_type<> with future_stored_type
Kefu Chai [Wed, 9 Sep 2020 03:25:03 +0000 (11:25 +0800)]
crimson/os: instantiate future_type<> with future_stored_type

to accommodate the change on seastar's side, which started to use
monostate for representing `void` return type, and hide this convention
using future_stored_type_t.

Signed-off-by: Kefu Chai <kchai@redhat.com>
4 years agocrimson/common: move continuation_base_from_future out of internal namespace
Kefu Chai [Wed, 9 Sep 2020 02:45:18 +0000 (10:45 +0800)]
crimson/common: move continuation_base_from_future out of internal namespace

otherwise we have

../src/crimson/common/errorator.h:1090:8: error: specialization of ‘template<class Future> struct seastar::continuation_base_from_future’ in different namespace [-fpermissive]
 1090 | struct continuation_base_from_future<Container<::crimson::errorated_future_marker<Values...>>> {
      |        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

since seastar has moved its continuation_base_from_future() out
of seastar::internal into seastar namespace

Signed-off-by: Kefu Chai <kchai@redhat.com>
4 years agocrimson/net: print entity_addr_t not the in4_addr in it
Kefu Chai [Wed, 9 Sep 2020 02:24:09 +0000 (10:24 +0800)]
crimson/net: print entity_addr_t not the in4_addr in it

for two reasons:

* we don't have operator<<(ostream&, in4_addr&) defined.
* we don't have any fmt::format() facility specialized for in4_addr
* to print entity_addr_t instead of in4_addr is more correct in
  the sense that we should support IPv6

Signed-off-by: Kefu Chai <kchai@redhat.com>
4 years agotest/crimson: include seastar/util/later.hh for seastar::now()
Kefu Chai [Wed, 9 Sep 2020 05:24:11 +0000 (13:24 +0800)]
test/crimson: include seastar/util/later.hh for seastar::now()

seastar::now() is extracted into seastar/util/later.hh in a recent
change in seastar, so update accordingly

Signed-off-by: Kefu Chai <kchai@redhat.com>
4 years agocrimson: include seastar/core/when_all.hh for when_all()
Kefu Chai [Wed, 9 Sep 2020 02:23:48 +0000 (10:23 +0800)]
crimson: include seastar/core/when_all.hh for when_all()

Signed-off-by: Kefu Chai <kchai@redhat.com>
4 years agocrimson/osd: use temporary RecoveryInfo for recovering object delete ops 37042/head
Xuehan Xu [Wed, 9 Sep 2020 04:37:25 +0000 (12:37 +0800)]
crimson/osd: use temporary RecoveryInfo for recovering object delete ops

Right now, recover_delete use WaitForObjectRecovery::pi, which may not be available
if there were no pulling, to pass recovery_info, so use temporary recovery_info instead

Fixes: https://tracker.ceph.com/issues/47357
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
4 years agocrimson/osd: set push future an exception when there are errors during pushing
Xuehan Xu [Wed, 9 Sep 2020 04:36:18 +0000 (12:36 +0800)]
crimson/osd: set push future an exception when there are errors during pushing

Fixes: https://tracker.ceph.com/issues/47357
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
4 years agocrimson/net: do not print non-void* pointer
Kefu Chai [Wed, 9 Sep 2020 02:17:28 +0000 (10:17 +0800)]
crimson/net: do not print non-void* pointer

to address the failure like

fmt/core.h:902:19: error: static assertion failed: formatting of non-void pointers is disallowed
  902 |     static_assert(!sizeof(T), "formatting of non-void pointers is disallowed");
      |                   ^~~~~~~~~~

and i searched around, couldn't find any other
statement printing the address of message, so
i think it's not of much use to print their
addresses in a single place, unless we can cross
check them in different places.

Signed-off-by: Kefu Chai <kchai@redhat.com>
4 years agocrimson: use seastar::make_ready_future()
Kefu Chai [Wed, 9 Sep 2020 02:08:20 +0000 (10:08 +0800)]
crimson: use seastar::make_ready_future()

instead of seastar::now(). as seastar::now() was moved into
seastar/util/laster.hh.

Signed-off-by: Kefu Chai <kchai@redhat.com>
4 years agocls/fifo: Add 'exclusive' option to Trim 35548/head
Adam C. Emerson [Fri, 4 Sep 2020 00:43:47 +0000 (20:43 -0400)]
cls/fifo: Add 'exclusive' option to Trim

To support RGW's MetadataLog, add a flag to FIFO::trim that when true
trims up to but not including the given entry.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
4 years agorgw: Datalog with selectable backends, FIFO and Omap
Adam C. Emerson [Wed, 10 Jun 2020 16:14:36 +0000 (12:14 -0400)]
rgw: Datalog with selectable backends, FIFO and Omap

By default, use whatever is present (and use FIFO if nothing is
present.)

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
4 years agorgw: Fix async caller into Datalog
Adam C. Emerson [Fri, 5 Jun 2020 23:28:29 +0000 (19:28 -0400)]
rgw: Fix async caller into Datalog

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
4 years agorgw: Remove superfluous wrapper around datalog
Adam C. Emerson [Tue, 12 May 2020 23:59:17 +0000 (19:59 -0400)]
rgw: Remove superfluous wrapper around datalog

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
4 years agorgw: Normalize DataLog function signatures
Adam C. Emerson [Tue, 12 May 2020 22:50:14 +0000 (18:50 -0400)]
rgw: Normalize DataLog function signatures

Pull out the stuff we aren't using so FIFO and Timelog Datalog can use
the same interface.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
4 years agocommon: Make error conversion functions `nodiscard`
Adam C. Emerson [Wed, 2 Sep 2020 16:18:22 +0000 (12:18 -0400)]
common: Make error conversion functions `nodiscard`

As suggested by Casey, so the compiler can catch failures to return
errors.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
4 years agorgw: Get rid of BucketFilter class, use a std::function
Adam C. Emerson [Mon, 4 May 2020 19:23:21 +0000 (15:23 -0400)]
rgw: Get rid of BucketFilter class, use a std::function

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
4 years agorgw: Remove date from REST log trim/list
Adam C. Emerson [Thu, 30 Apr 2020 20:38:41 +0000 (16:38 -0400)]
rgw: Remove date from REST log trim/list

Also only accept one marker.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
4 years agorgw: Remove some TimeLog options from radosgw-admin
Adam C. Emerson [Thu, 30 Apr 2020 19:06:13 +0000 (15:06 -0400)]
rgw: Remove some TimeLog options from radosgw-admin

Remove date-oriented trim/list functionality from MDLOG, DataLog, and
Sync-Error-Log.

Don't provide start/end markers. Support only a start marker for list
and an end marker for trim.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
4 years agogitignore: Add vstart_environment.sh
Adam C. Emerson [Tue, 4 Aug 2020 02:49:36 +0000 (22:49 -0400)]
gitignore: Add vstart_environment.sh

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
4 years agorgw: FIFO for legacy RADOS client
Adam C. Emerson [Thu, 14 May 2020 15:11:12 +0000 (11:11 -0400)]
rgw: FIFO for legacy RADOS client

Use optional_yield for most operations and provide an AioCompletion*
overload for trim.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
4 years agocls/fifo: Use neorados interface
Adam C. Emerson [Tue, 4 Feb 2020 16:33:48 +0000 (11:33 -0500)]
cls/fifo: Use neorados interface

Rewrite around the asynchronous ASIO-based RADOS interface, and
include support for calls from multiple threads.

`allocate_unique` pattern contributed by
Casey Bodley <cbodley@redhat.com>

Co-authored-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
4 years agocls/fifo: FIFO over RADOS
Yehuda Sadeh [Thu, 5 Sep 2019 11:15:15 +0000 (04:15 -0700)]
cls/fifo: FIFO over RADOS

This is an implementation of fifo queue over rados. Data is appended
to rados object until it's full. At that point data will be written to
a new object. Data is read from the tail sequentially (can be iterated
using marker). Data can be trimmed (up to and including marker).

Queue has a header object (meta), and zero or more data objects (parts).

The software has two layers: the higher level client operations side
that deals with the application apis, and manages the meta and parts,
and there’s the objclass level that deals with the rados layout of
meta and part objects. There are different objclass methods that deal
with reading and modifying each of these entities.

A single part has max possible size, however, it may become full once
a certain smaller size is reached (full_size_threshold). It is
imperative that once a part has reached its capacity, it will not
allow any more writes into it. For this reason, it is important that
data being written to the queue does not exceed max_entry_size . This
is enforced, by the higher level client api.

Written entries go to the current head object, and when it’s full, a
new head part is created. When listing entries, data is iterated from
tail to the current head. Trim can either change the pointer within
the current tail object, and if needed it removes tail objects.

A unitest has been created to test functionality.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
Squashed-by: Adam C. Emerson <aemerson@redhat.com>
4 years agostrtol: Add parse/consume for string_view friendly interface
Adam C. Emerson [Fri, 6 Mar 2020 04:13:47 +0000 (23:13 -0500)]
strtol: Add parse/consume for string_view friendly interface

Also these don't have the stringstream overhead.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
4 years agocommon/Thread: Don't store pointer to thread_name
Adam C. Emerson [Tue, 7 Jul 2020 20:39:57 +0000 (16:39 -0400)]
common/Thread: Don't store pointer to thread_name

Having Thread::create store a pointer to a string that is passed to
ceph_pthread_setname in Thread::entry_wrapper can lead to using a
pointer in the calling thread's stack that gets freed before use.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
4 years agocommon/ceph_json: Support en/decoding Boost's flat containers
Adam C. Emerson [Tue, 10 Sep 2019 22:53:58 +0000 (18:53 -0400)]
common/ceph_json: Support en/decoding Boost's flat containers

Specifically boost::container::flat_map and boost::container::flat_set

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
4 years agocommon/async: Add handler for librados::AioCompletion
Adam C. Emerson [Tue, 5 May 2020 22:32:59 +0000 (18:32 -0400)]
common/async: Add handler for librados::AioCompletion

Support using a librados::AioCompletion as a completion handler in
Boost.Asio when the arguments are void or only an error code.

(To support more arguments we'd have to wrap up the AioCompletion with
pointers to hold them.)

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
4 years agocommon: add ceph::allocate_unique()
Casey Bodley [Fri, 3 Apr 2020 14:41:10 +0000 (10:41 -0400)]
common: add ceph::allocate_unique()

adds an allocator-aware version of std::make_unique(). this is similar to
std::allocate_shared(), though it's slightly less convenient because,
unlike std::shared_ptr<T>, the Deleter has to be specified as a template
parameter in std::unique_ptr<T, Deleter>

Signed-off-by: Casey Bodley <cbodley@redhat.com>
4 years agocommon: bind_like
Adam C. Emerson [Fri, 21 Feb 2020 17:46:47 +0000 (12:46 -0500)]
common: bind_like

A template to ensure that one completion has the same executor and
allocator as another.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
4 years agocommon: Add bind_allocator template
Adam C. Emerson [Fri, 21 Feb 2020 17:45:01 +0000 (12:45 -0500)]
common: Add bind_allocator template

Bind an allocator to a completion, analogously to
boost::async::bind_executor.

Based on bind_executor.hpp which is copyright 2003-2019
Christopher M. Kohlhoff <chris@kohlhoff.com> and under
the Boost Software License, Version 1.0

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
4 years agoinclude: Import (modified) uses_allocator.h
Adam C. Emerson [Fri, 21 Feb 2020 17:40:55 +0000 (12:40 -0500)]
include: Import (modified) uses_allocator.h

From https://github.com/phalpern/uses-allocator

To handle uses_allocator construction

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
4 years agoMerge pull request #37049 from yuriw/wip-yuriw-crontab-master
Yuri Weinstein [Tue, 8 Sep 2020 22:57:20 +0000 (15:57 -0700)]
Merge pull request #37049 from yuriw/wip-yuriw-crontab-master

qa/tests: removed extra line

Reviewed-by: Neha Ojha <nojha@redhat.com>
4 years agorgw: ObjectCache::put() clears stale objv 37054/head
Casey Bodley [Tue, 8 Sep 2020 19:27:55 +0000 (15:27 -0400)]
rgw: ObjectCache::put() clears stale objv

if an existing object is cached with an object version, but it's
mutated without updating that version number, clear the OBJV flag so
that later cache reads asking for an object version result in a miss and
re-read the version from the osd

Fixes: https://tracker.ceph.com/issues/47306
Signed-off-by: Casey Bodley <cbodley@redhat.com>
4 years agoqa/tests: removed extra line 37049/head
Yuri Weinstein [Tue, 8 Sep 2020 17:03:51 +0000 (10:03 -0700)]
qa/tests: removed extra line

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
4 years agoMerge pull request #36230 from trociny/wip-43174-2
Kefu Chai [Tue, 8 Sep 2020 13:34:11 +0000 (21:34 +0800)]
Merge pull request #36230 from trociny/wip-43174-2

os/bluestore: fix collection_list ordering

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
4 years agoMerge pull request #36955 from agayev/zbc-to-zbd
Kefu Chai [Tue, 8 Sep 2020 13:24:11 +0000 (21:24 +0800)]
Merge pull request #36955 from agayev/zbc-to-zbd

os/bluestore: Switch from libzbc library to libzbd library

Reviewed-by: Kefu Chai <kchai@redhat.com>
4 years agoMerge pull request #36950 from ceph/wip-bz1872983
Jan Fajerski [Tue, 8 Sep 2020 12:04:00 +0000 (14:04 +0200)]
Merge pull request #36950 from ceph/wip-bz1872983

ceph-volume: simple scan should ignore tmpfs

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
4 years agoMerge pull request #36976 from rhcs-dashboard/fix-47285-master
Lenz Grimmer [Tue, 8 Sep 2020 10:34:30 +0000 (12:34 +0200)]
Merge pull request #36976 from rhcs-dashboard/fix-47285-master

mgr/dashboard: fix duplicate logs

Reviewed-by: Laura Paduano <lpaduano@suse.com>
4 years agoMerge pull request #37003 from xxhdx1985126/wip-47310
Kefu Chai [Tue, 8 Sep 2020 05:38:51 +0000 (13:38 +0800)]
Merge pull request #37003 from xxhdx1985126/wip-47310

crimson/osd: drop repop if osdmap has increased

Reviewed-by: Kefu Chai <kchai@redhat.com>
4 years agoMerge pull request #36974 from jan--f/snap-schedule-keep-with-empty-retention
Venky Shankar [Tue, 8 Sep 2020 04:45:28 +0000 (10:15 +0530)]
Merge pull request #36974 from jan--f/snap-schedule-keep-with-empty-retention

snap-schedule: don't prune all on empty retention

4 years agocrimson/osd: add can_discard_op method for detecting "discardable" client requests 37003/head
Xuehan Xu [Tue, 8 Sep 2020 04:36:41 +0000 (12:36 +0800)]
crimson/osd: add can_discard_op method for detecting "discardable" client requests

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
4 years agocrimson/osd: drop repop if it can be discarded
Xuehan Xu [Tue, 8 Sep 2020 04:36:06 +0000 (12:36 +0800)]
crimson/osd: drop repop if it can be discarded

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
4 years agocrimson/osd: rename is_valid_rep_op_reply to can_discard_replica_op
Xuehan Xu [Tue, 8 Sep 2020 04:35:14 +0000 (12:35 +0800)]
crimson/osd: rename is_valid_rep_op_reply to can_discard_replica_op

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
4 years agoMerge pull request #36903 from sebastian-philipp/python-common-simplify-hsp-to-json
Sebastian Wagner [Mon, 7 Sep 2020 15:58:29 +0000 (17:58 +0200)]
Merge pull request #36903 from sebastian-philipp/python-common-simplify-hsp-to-json

python-common: Make HostPlacementSpec.to_json() more friendly

Reviewed-by: Adam King <adking@redhat.com>
4 years agoMerge pull request #36994 from ktdreyer/boost-context-s390x
Nathan Cutler [Mon, 7 Sep 2020 10:39:12 +0000 (12:39 +0200)]
Merge pull request #36994 from ktdreyer/boost-context-s390x

rpm: enable WITH_BOOST_CONTEXT for s390x

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
4 years agoMerge pull request #36432 from sebastian-philipp/cephadm-repo_digest
Sebastian Wagner [Mon, 7 Sep 2020 09:57:51 +0000 (11:57 +0200)]
Merge pull request #36432 from sebastian-philipp/cephadm-repo_digest

mgr/cephadm: convert tags to repo_digest

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Varsha Rao <varao@redhat.com>
4 years agoMerge pull request #37015 from tchaikov/wip-seastar-packet
Kefu Chai [Mon, 7 Sep 2020 09:44:05 +0000 (17:44 +0800)]
Merge pull request #37015 from tchaikov/wip-seastar-packet

common/buffer_seastar: reserve the packet slots in ctor

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
4 years agoMerge pull request #36946 from zdover23/wip-docs-36796-fix
Sebastian Wagner [Mon, 7 Sep 2020 08:48:27 +0000 (10:48 +0200)]
Merge pull request #36946 from zdover23/wip-docs-36796-fix

doc/cephadm: Document mgr/cephadm/manage_etc_ceph_ceph_conf

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
4 years agoMerge pull request #36995 from smithfarm/wip-47305
Sebastian Wagner [Mon, 7 Sep 2020 08:39:25 +0000 (10:39 +0200)]
Merge pull request #36995 from smithfarm/wip-47305

rpm: on SUSE, podman is required for cephadm to work

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
4 years agoMerge pull request #36992 from uweigand/fix-encode-float
Kefu Chai [Mon, 7 Sep 2020 06:50:37 +0000 (14:50 +0800)]
Merge pull request #36992 from uweigand/fix-encode-float

include/encoding: Fix encode/decode of float types on big-endian systems

Reviewed-by: Kefu Chai <kchai@redhat.com>
4 years agoMerge pull request #36829 from changchengx/osdmap
Kefu Chai [Mon, 7 Sep 2020 06:49:08 +0000 (14:49 +0800)]
Merge pull request #36829 from changchengx/osdmap

crush,osd: refine crushmap comment and implementation

Reviewed-by: Kefu Chai <kchai@redhat.com>
4 years agoMerge pull request #36947 from pdvian/wip-mgr-get-rule-root
Kefu Chai [Mon, 7 Sep 2020 06:46:52 +0000 (14:46 +0800)]
Merge pull request #36947 from pdvian/wip-mgr-get-rule-root

mgr/mgr_module : Fix keyerror for op if rule steps first element is opcode

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
4 years agoMerge pull request #36972 from smithfarm/wip-47112
Kefu Chai [Mon, 7 Sep 2020 06:44:44 +0000 (14:44 +0800)]
Merge pull request #36972 from smithfarm/wip-47112

rpm,deb: drop /etc/sudoers.d/cephadm

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: David Disseldorp <ddiss@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>
4 years agoMerge pull request #36937 from tchaikov/wip-mgr-cleanup
Kefu Chai [Mon, 7 Sep 2020 06:42:01 +0000 (14:42 +0800)]
Merge pull request #36937 from tchaikov/wip-mgr-cleanup

mgr: use range-based loop and cleanups

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
4 years agoMerge pull request #36999 from neha-ojha/wip-47309
Kefu Chai [Mon, 7 Sep 2020 06:40:51 +0000 (14:40 +0800)]
Merge pull request #36999 from neha-ojha/wip-47309

qa/*/mon/mon-last-epoch-clean.sh: mark osd out instead of down

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
4 years agoMerge pull request #37008 from tchaikov/wip-crimson-pg-tell
Kefu Chai [Mon, 7 Sep 2020 06:39:44 +0000 (14:39 +0800)]
Merge pull request #37008 from tchaikov/wip-crimson-pg-tell

crimson/osd: add "ceph tell <pgid> <command>" support

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
4 years agocommon/buffer_seastar: reserve the packet slots in ctor 37015/head
Kefu Chai [Mon, 7 Sep 2020 06:05:20 +0000 (14:05 +0800)]
common/buffer_seastar: reserve the packet slots in ctor

for better performance, and simpler this way

Signed-off-by: Kefu Chai <kchai@redhat.com>
4 years agoMerge pull request #37010 from tchaikov/wip-crimson-osd-net
Kefu Chai [Mon, 7 Sep 2020 05:00:14 +0000 (13:00 +0800)]
Merge pull request #37010 from tchaikov/wip-crimson-osd-net

crimson/net: move implementation into .cc file

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
4 years agoMerge pull request #37001 from tchaikov/wip-crimson-basic
Kefu Chai [Mon, 7 Sep 2020 01:52:45 +0000 (09:52 +0800)]
Merge pull request #37001 from tchaikov/wip-crimson-basic

qa/suites: add crimson-rados/basic tests

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
4 years agocmake: add "WITH_ZBD" option 36955/head
Kefu Chai [Sat, 5 Sep 2020 05:00:17 +0000 (13:00 +0800)]
cmake: add "WITH_ZBD" option

for enabling Zoned Block Device support using libzbd

Signed-off-by: Kefu Chai <kchai@redhat.com>
4 years agoos/bluestore: Switch from libzbc library to libzbd library
Abutalib Aghayev [Wed, 2 Sep 2020 20:12:50 +0000 (16:12 -0400)]
os/bluestore: Switch from libzbc library to libzbd library

Libzbc maintainers recommend switching to libzbd, which is lighter and supports
both ZNS SSDs and HM-SMR HDDs.

Signed-off-by: Abutalib Aghayev <agayev@cs.cmu.edu>
4 years agoMerge PR #36600 into master
Patrick Donnelly [Sun, 6 Sep 2020 15:54:38 +0000 (08:54 -0700)]
Merge PR #36600 into master

* refs/pull/36600/head:
mds: make caps dump more readable

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>
4 years agocrimson/osd: add "ceph tell <pgid> <command>" support 37008/head
Kefu Chai [Sun, 6 Sep 2020 06:53:22 +0000 (14:53 +0800)]
crimson/osd: add "ceph tell <pgid> <command>" support

* add an abstract class of `PGCommand` for `ceph tell <pgid> <command>`
* add two sample implementations for the pg tell commands.
  - "query"
  - "mark_unfound_lost"

Signed-off-by: Kefu Chai <kchai@redhat.com>
4 years agocrush/CrushWrapper: refine type_map/name_map comment 36829/head
Changcheng Liu [Wed, 26 Aug 2020 01:41:42 +0000 (09:41 +0800)]
crush/CrushWrapper: refine type_map/name_map comment

type_map is the item(bucket & device) type id and its type name.
name_map is the item(bucket & device) id and its name.

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>
4 years agocrush/CrushWrapper: use "new_bucket_id" as var name of the added bucket
Changcheng Liu [Wed, 26 Aug 2020 06:22:10 +0000 (14:22 +0800)]
crush/CrushWrapper: use "new_bucket_id" as var name of the added bucket

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>
4 years agocrush/CrushWrapper: align variable name with comment
Changcheng Liu [Wed, 26 Aug 2020 06:18:34 +0000 (14:18 +0800)]
crush/CrushWrapper: align variable name with comment

In the prolog of "for loop" statement, it writes "add child in the
location with 0 weight"

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>
4 years agoosd/OSDMap: no need to give same locations per time to create bucket
Changcheng Liu [Wed, 26 Aug 2020 01:52:57 +0000 (09:52 +0800)]
osd/OSDMap: no need to give same locations per time to create bucket

Suggested-by: Kefu Chai <kefu@redhat.com>
Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>
4 years agocrush/CrushWrapper: refine comment to improve readability
Changcheng Liu [Wed, 26 Aug 2020 06:16:13 +0000 (14:16 +0800)]
crush/CrushWrapper: refine comment to improve readability

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>
4 years agoosd/OSDMap: assert osd_weight default initialized value is CEPH_OSD_OUT
Changcheng Liu [Thu, 3 Sep 2020 15:47:52 +0000 (23:47 +0800)]
osd/OSDMap: assert osd_weight default initialized value is CEPH_OSD_OUT

refer: http://www.cplusplus.com/reference/vector/vector/resize/
Explicitly specify initialization value to vector::resize operation.

Suggested-by: Kefu Chai <kefu@redhat.com>
Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>
4 years agoosd/OSDMap: use class member instead of parameter to reset container size
Changcheng Liu [Wed, 26 Aug 2020 02:21:48 +0000 (10:21 +0800)]
osd/OSDMap: use class member instead of parameter to reset container size

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>
4 years agoosd/OSDMap: avoid re-initialize osd state and weight done in set_max_osd
Changcheng Liu [Wed, 26 Aug 2020 07:47:04 +0000 (15:47 +0800)]
osd/OSDMap: avoid re-initialize osd state and weight done in set_max_osd

In set_max_osd function, container's osd_weight and osd_vector have been
assigned with the right state/weight value.

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>
4 years agocrush: correct typo
Changcheng Liu [Wed, 26 Aug 2020 01:48:51 +0000 (09:48 +0800)]
crush: correct typo

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>
4 years agocrush: no need to double check root bucket id
Changcheng Liu [Mon, 24 Aug 2020 02:53:39 +0000 (10:53 +0800)]
crush: no need to double check root bucket id

root bucket id should be definitely less than 0. assert(r < 0) here.
There's no possible to find device items in find_nonshadow_roots.

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>
4 years agoinclude/encoding: Fix encode/decode of float types on big-endian systems 36992/head
Ulrich Weigand [Fri, 4 Sep 2020 13:42:41 +0000 (15:42 +0200)]
include/encoding: Fix encode/decode of float types on big-endian systems

Currently, floating-point types use "raw" encoding, which means they're
simply copied as byte stream.

This means that if the decoding happens on a machine that differs in
byte order from the source machine, the returned value will be
incorrect. As one effect of this problem, a big-endian OSD node cannot
join a cluster where the MON node is little-endian (or vice versa),
because the OSDMap (incremental) structure contains floating-point
values, and as a result of this conversion problem, the OSD node will
crash with an assertion failure as soon as it receives any OSDMap update
from the MON.

This should be fixed by always encoding floating-point values in
little-endian byte order just as is done for integers. (Note that this
still assumes source and target machines used the same floating-point
format except for byte order. But given that nearly all platforms these
days use IEEE binary32/binary64 for float/double, that seems a
reasonable assumption.)

Fixes: https://tracker.ceph.com/issues/47302
Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>
4 years agoMerge pull request #36924 from ddiss/sigabbrev_np
Kefu Chai [Sun, 6 Sep 2020 10:45:58 +0000 (18:45 +0800)]
Merge pull request #36924 from ddiss/sigabbrev_np

cmake: detect and use sigdescr_np() if available

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
4 years agoMerge pull request #36929 from tchaikov/wip-python-common-no-unicode
Kefu Chai [Sun, 6 Sep 2020 10:44:37 +0000 (18:44 +0800)]
Merge pull request #36929 from tchaikov/wip-python-common-no-unicode

python-common: s/unicode/str/

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
4 years agocrimson/net: move implementation into .cc file 37010/head
Kefu Chai [Sun, 6 Sep 2020 10:32:44 +0000 (18:32 +0800)]
crimson/net: move implementation into .cc file

for faster compilation.

Signed-off-by: Kefu Chai <kchai@redhat.com>
4 years agoMerge pull request #37007 from xxhdx1985126/wip-47313-47311
Kefu Chai [Sun, 6 Sep 2020 08:26:07 +0000 (16:26 +0800)]
Merge pull request #37007 from xxhdx1985126/wip-47313-47311

crimson/osd: accurately judge whether to do drop_recovery_read on recovering objs

Reviewed-by: Kefu Chai <kchai@redhat.com>
4 years agoMerge pull request #37004 from tchaikov/wip-discard-repop
Kefu Chai [Sun, 6 Sep 2020 08:00:36 +0000 (16:00 +0800)]
Merge pull request #37004 from tchaikov/wip-discard-repop

crimson/osd: discard op if pg changes or sender is down

Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>
4 years agocrimson/osd: add PG::mark_unfound_lost()
Kefu Chai [Sun, 6 Sep 2020 06:50:26 +0000 (14:50 +0800)]
crimson/osd: add PG::mark_unfound_lost()

it is a stub implementation for supporing asock command

Signed-off-by: Kefu Chai <kchai@redhat.com>
4 years agocrimson/osd: add PG::dump_primary()
Kefu Chai [Sun, 6 Sep 2020 06:49:30 +0000 (14:49 +0800)]
crimson/osd: add PG::dump_primary()

for implementing asock command "ceph tell 1.0 query"

Signed-off-by: Kefu Chai <kchai@redhat.com>
4 years agocrimson/osd: drop unused header
Kefu Chai [Sun, 6 Sep 2020 06:48:45 +0000 (14:48 +0800)]
crimson/osd: drop unused header

osd_admin.h is referenced in the .cc file, so drop it in the header

Signed-off-by: Kefu Chai <kchai@redhat.com>
4 years agocrimson/osd: add OSD::get_pg()
Kefu Chai [Sun, 6 Sep 2020 06:47:01 +0000 (14:47 +0800)]
crimson/osd: add OSD::get_pg()

for implementing asock commands.

and add `PGMap::get_pg()` as a helper

Signed-off-by: Kefu Chai <kchai@redhat.com>
4 years agoosd: refactor PrimaryLogPG::do_command("query")
Kefu Chai [Sun, 6 Sep 2020 06:44:12 +0000 (14:44 +0800)]
osd: refactor PrimaryLogPG::do_command("query")

we should let recovery_state.dump_peering_state() close the formatter
section, because it's recovery_state.dump_peering_state(), who opens it.

Signed-off-by: Kefu Chai <kchai@redhat.com>
4 years agoosd/PGStateUtils: do not forward declare unused class
Kefu Chai [Sun, 6 Sep 2020 06:42:38 +0000 (14:42 +0800)]
osd/PGStateUtils: do not forward declare unused class

PG is not referenced in this header

Signed-off-by: Kefu Chai <kchai@redhat.com>
4 years agocrimson/osd: print pgid in OSD::load_pg()
Kefu Chai [Sat, 5 Sep 2020 13:51:37 +0000 (21:51 +0800)]
crimson/osd: print pgid in OSD::load_pg()

Signed-off-by: Kefu Chai <kchai@redhat.com>
4 years agocrimson/os: fix indent
Kefu Chai [Sat, 5 Sep 2020 13:51:16 +0000 (21:51 +0800)]
crimson/os: fix indent

Signed-off-by: Kefu Chai <kchai@redhat.com>