Jason Dillaman [Mon, 5 Oct 2020 18:04:14 +0000 (14:04 -0400)]
librbd: support preprocessing source object data prior to deep-copy
Let object dispatch layers potentially mutate the data read from the
source image prior to issuing the actual deep-copy operations against
the destination image.
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
The write-ops now only stores write vs zero ops and the type of
zero operation is delayed until the actual op is sent. This will
make the state machine compatible with the copyup process hook.
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
Jason Dillaman [Thu, 24 Sep 2020 19:15:23 +0000 (15:15 -0400)]
librbd: support preprocessing parent data prior to copyup
Let object dispatch layers potentially mutate the copyup data read
from the parent prior to issuing the actual copyup operation. This
can allow for a layer like the crypto layer to re-encrypt the parent
image data using the child image's encryption keys, for example.
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
Jason Dillaman [Wed, 23 Sep 2020 19:57:20 +0000 (15:57 -0400)]
librbd: new hook for pre-processing copyup data
This will permit the crypto layer to properly encrypt and potentially
align the sparse copyup data prior to it being written. It passes
potentially multiple sets of data in one pass to permit the deep-copy
state machine to utilize the same API and allow the crypto layer to
potentially handle layered alignment issues.
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
Jason Dillaman [Wed, 30 Sep 2020 01:24:23 +0000 (21:24 -0400)]
librbd: rename SnapshotExtent to SparseExtent
The processing of copyup needs to be able to denote data extents that
are potentially zeroed or included in the associated bufferlist. By
renaming the type, it can be re-used for this second purpose.
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
Jason Dillaman [Tue, 29 Sep 2020 00:35:38 +0000 (20:35 -0400)]
librbd: copyup state machine should always issue a sparse-read
When reading from the parent, always keep the data in a sparse
extent-map format. The forthcoming copyup preprocessing hook will
want to pass a set of sparse image-extent data.
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
Jason Dillaman [Tue, 29 Sep 2020 00:04:48 +0000 (20:04 -0400)]
librbd: switch remaining uses of ExtentMap to Extents
The neorados API already requires the vector-based approach vs
the map-based approach. Now the remaining sparse-read functionality
has been switched to use the consistent approach.
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
Jason Dillaman [Fri, 25 Sep 2020 14:40:32 +0000 (10:40 -0400)]
librbd: deep-copy should update object-map before writing to object
For the original use-case of RBD mirroring it was (maybe) more
acceptable to write to the object before updating the object map
because an interrupted sync will be retried. However, when using
the deep-copy object copy state machine as part of copyup, it's
more likely that the object-map has the potential to become
out-of-sync with reality if it's updated after the object is
written.
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
mds: support sending empty perf metrics to ceph-manager
Right now, there are no per-mds metrics that are tracked and
sent by mds. However, such metrics will get added soon. So,
send empty performance metrics to ceph-manager for now.
Changcheng Liu [Wed, 23 Sep 2020 07:39:47 +0000 (15:39 +0800)]
mailmap: update Intel employee mail/org
1. "changcheng.liu@aliyun.com" need be classified into intel until now.
This reverts part of commit: df07e9f3
2. add "Yuan Lu <yuan.y.lu@intel.com>" in mailmap
Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>
luo rixin [Tue, 1 Sep 2020 09:06:40 +0000 (17:06 +0800)]
rgw/rgw_file: Fix the incorrect lru object eviction
In func lookup_fh, when RGWFileHandle not be found in fh_cache, it
need to recycle an object and create an new RGWFileHandle. When there
are multi threads use lookup_fh to find and create RGWFileHandle concurrently,
it must to make sure evict lru object from the partiton of fh_cache which new
RGWFileHandle will be inserted to.
Fixes: https://tracker.ceph.com/issues/47235 Signed-off-by: luo rixin <luorixin@huawei.com>
Jan Fajerski [Fri, 9 Oct 2020 09:22:01 +0000 (11:22 +0200)]
Merge PR #32778 into master
* refs/pull/32778/head:
msg: Use sockets instead of pipes for wakeup events on win32
common: Convert remote errors on win32
common,msg: Initialize Windows WSA and TLS
common,os,kv: Define aligned_free
common: avoid CLOCK_*_COARSE warnings on win32
global: Windows support
Kefu Chai [Fri, 9 Oct 2020 07:54:35 +0000 (15:54 +0800)]
cls/fifo: cast ceph_le64 to uint64_t before printing it
silences warning like:
../src/cls/fifo/cls_fifo.cc: In member function ‘int rados::cls::fifo::{anonymous}::EntryReader::get_next_entry(ceph::buffer::v15_2_0::list*, uint64_t*, ceph::real_time*)’:
../src/include/rados/objclass.h:33:18: warning: format ‘%llu’ expects argument of type ‘long long unsigned int’, but argument 7 has type ‘ceph_le64’ {aka ‘ceph_le<long long unsigned int>’} [-Wformat=]
33 | cls_log(level, "<cls> %s:%d: " fmt, __FILE__, __LINE__, ##__VA_ARGS__)
../src/include/rados/objclass.h:33:18: note: in definition of macro ‘CLS_LOG’
33 | cls_log(level, "<cls> %s:%d: " fmt, __FILE__, __LINE__, ##__VA_ARGS__)
| ^~~~~~~~~~~~~~~
../src/cls/fifo/cls_fifo.cc:703:48: note: format string is defined here
703 | CLS_LOG(10, "%s():%d: pre_header.pre_size=%llu", __func__, __LINE__,
| ~~~^
| |
| long long unsigned int
Jan Fajerski [Thu, 8 Oct 2020 06:45:26 +0000 (08:45 +0200)]
ceph-volume: don't exit before empty report can be printed
get_plan() called exit in case of an empty plan. This prevented a report
being printed under these circumstances. Avoid exit in this case. Also
adds tests to ensure an empty report is printed.
Fixes: https://tracker.ceph.com/issues/47760 Signed-off-by: Jan Fajerski <jfajerski@suse.com>
Nizamudeen A [Wed, 7 Oct 2020 17:21:10 +0000 (22:51 +0530)]
mgr/dashboard: Add short descriptions to the telemetry report preview
A small info icon with a popover on the telemetry report preview form step2 for
Report ID and Report Preview.
Report ID: "A randomized UUID to identify a particular cluster over the course of several telemetry reports."
Report preview: "The actual telemetry data that will be submitted."
Fixes: https://tracker.ceph.com/issues/47610 Signed-off-by: Nizamudeen A <nia@redhat.com>
Jason Dillaman [Wed, 7 Oct 2020 14:58:57 +0000 (10:58 -0400)]
librbd: avoid failing IO with -ESHUTDOWN when disabling exclusive-lock
When dynamically disabling the exclusive-lock feature with in-flight IO,
it was previously possible for IO to fail with -ESHUTDOWN when it
attempts to acquire the lock due to the PreReleaseRequest setting the
lock required flag. There is also the possibility for a race with the
first IO racing with exclusive-lock shutdown when the lock was not
already acquired.
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
rgw: fix setting of namespace in ordered and unordered bucket listing
The namespace is not always set correctly during bucket listing. This
can, for example, cause the listing of incomplete multipart uploads,
which are in the _multipart_ namespace, to not paginate correctly, and
cause entries to be re-listed.
Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
rgw: allow rgw-orphan-list to note when rados objects are in namespace
Currently namespaces and locators are ignored when `rados ls` is run
by rgw-orphan-list to record RADOS's known objects.
However there have been cases where RADOS objects have a locator, and
when one is included in the listing, the script does not handle it
correctly. Now when objects have locators, we will prevent their
output from entering the .intermediate file.
Additionally we do not expect RGW data objects to be in RADOS
namespaces, so when a namespaced object is detected, we'll error out
with a message.
Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
include: explicitly define all types needed for libc++ iterator
There is a serious difference between iterator in
libstdc++ (gcc) versus libc++ (clang)
This generates a rather big and vague set of warnings and notes.
The crux is the warnings about `requirement '!__is_forward_iterator....`
Extra information can also be found on:
https://www.fluentcpp.com/2018/05/08/std-iterator-deprecated/
```
/home/jenkins/workspace/ceph-master-compile/src/librbd/deep_copy/ObjectCopyRequest.cc:142:15: error: no matching constructor for initialization of 'io::Extents' (aka 'vector<pair<unsigned long, unsigned long> >')
io::Extents image_extents{read_op.image_interval.begin(),
^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/jenkins/workspace/ceph-master-compile/src/librbd/deep_copy/ObjectCopyRequest.cc:769:35: note: in instantiation of member function 'librbd::deep_copy::ObjectCopyRequest<librbd::ImageCtx>::send_read' requested here
template class librbd::deep_copy::ObjectCopyRequest<librbd::ImageCtx>;
^
/usr/include/c++/v1/vector:516:14: note: candidate constructor not viable: no known conversion from 'interval_set<unsigned long, std::map>::iterator' to 'std::__1::vector<std::__1::pair<unsigned long, unsigned long>, std::__1::allocator<std::__1::pair<unsigned long, unsigned long> > >::size_type' (aka 'unsigned long') for 1st argument
explicit vector(size_type __n, const allocator_type& __a);
^
/usr/include/c++/v1/vector:518:5: note: candidate constructor not viable: no known conversion from 'interval_set<unsigned long, std::map>::iterator' to 'std::__1::vector<std::__1::pair<unsigned long, unsigned long>, std::__1::allocator<std::__1::pair<unsigned long, unsigned long> > >::size_type' (aka 'unsigned long') for 1st argument
vector(size_type __n, const value_type& __x);
^
/usr/include/c++/v1/vector:559:5: note: candidate constructor not viable: no known conversion from 'interval_set<unsigned long, std::map>::iterator' to 'const std::__1::vector<std::__1::pair<unsigned long, unsigned long>, std::__1::allocator<std::__1::pair<unsigned long, unsigned long> > >' for 1st argument
vector(const vector& __x, const allocator_type& __a);
^
/usr/include/c++/v1/vector:568:5: note: candidate constructor not viable: no known conversion from 'interval_set<unsigned long, std::map>::iterator' to 'initializer_list<std::__1::vector<std::__1::pair<unsigned long, unsigned long>, std::__1::allocator<std::__1::pair<unsigned long, unsigned long> > >::value_type>' (aka 'initializer_list<std::__1::pair<unsigned long, unsigned long> >') for 1st argument
vector(initializer_list<value_type> __il, const allocator_type& __a);
^
/usr/include/c++/v1/vector:579:5: note: candidate constructor not viable: no known conversion from 'interval_set<unsigned long, std::map>::iterator' to 'std::__1::vector<std::__1::pair<unsigned long, unsigned long>, std::__1::allocator<std::__1::pair<unsigned long, unsigned long> > >' for 1st argument
vector(vector&& __x, const allocator_type& __a);
^
/usr/include/c++/v1/vector:521:9: note: candidate template ignored: requirement '!__is_forward_iterator<interval_set<unsigned long, std::map>::iterator>::value' was not satisfied [with _InputIterator = interval_set<unsigned long, std::map>::iterator]
vector(_InputIterator __first,
^
/usr/include/c++/v1/vector:536:9: note: candidate template ignored: requirement 'is_constructible<std::__1::pair<unsigned long, unsigned long>, unsigned long &>::value' was not satisfied [with _ForwardIterator = interval_set<unsigned long, std::map>::iterator]
vector(_ForwardIterator __first,
^
/usr/include/c++/v1/vector:502:40: note: candidate constructor not viable: requires single argument '__a', but 2 arguments were provided
_LIBCPP_INLINE_VISIBILITY explicit vector(const allocator_type& __a)
^
/usr/include/c++/v1/vector:514:14: note: candidate constructor not viable: requires single argument '__n', but 2 arguments were provided
explicit vector(size_type __n);
^
/usr/include/c++/v1/vector:558:5: note: candidate constructor not viable: requires single argument '__x', but 2 arguments were provided
vector(const vector& __x);
^
/usr/include/c++/v1/vector:565:5: note: candidate constructor not viable: requires single argument '__il', but 2 arguments were provided
vector(initializer_list<value_type> __il);
^
/usr/include/c++/v1/vector:571:5: note: candidate constructor not viable: requires single argument '__x', but 2 arguments were provided
vector(vector&& __x)
^
/usr/include/c++/v1/vector:519:5: note: candidate constructor not viable: requires 3 arguments, but 2 were provided
vector(size_type __n, const value_type& __x, const allocator_type& __a);
^
/usr/include/c++/v1/vector:496:5: note: candidate constructor not viable: requires 0 arguments, but 2 were provided
vector() _NOEXCEPT_(is_nothrow_default_constructible<allocator_type>::value)
^
/usr/include/c++/v1/vector:529:9: note: candidate constructor template not viable: requires at least 3 arguments, but 2 were provided
vector(_InputIterator __first, _InputIterator __last, const allocator_type& __a,
^
/usr/include/c++/v1/vector:543:9: note: candidate constructor template not viable: requires at least 3 arguments, but 2 were provided
vector(_ForwardIterator __first, _ForwardIterator __last, const allocator_type& __a,
^
1 error generated.
```
Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>
Adam Kupczyk [Wed, 7 Oct 2020 10:34:04 +0000 (06:34 -0400)]
os/bluestore: fix memory accounting in TwoQBufferCacheShard
Variable buffer_bytes has been redefined in TwoQBufferCacheShard,
causing PriCache to see always 0 usage when 2q cache was selected,
as it looks at it through BufferCacheShard::_get_bytes().
Kefu Chai [Tue, 6 Oct 2020 11:09:18 +0000 (19:09 +0800)]
crimson/os: mark class with "final" dtor "final"
to silence warning like
scheduler.cc:145:30: warning: class with destructor marked 'final' cannot be inherited from [-Wfinal-dtor-non-final-class]
~ClassedOpQueueScheduler() final {};
^
/home/jenkins-build/build/workspace/ceph-perf-crimson/ceph-pr/src/crimson/osd/scheduler/scheduler.cc:52:7: note: mark 'ClassedOpQueueScheduler<T>' as 'final' to silence this warning
class ClassedOpQueueScheduler : public Scheduler {
^
Kefu Chai [Tue, 6 Oct 2020 11:03:38 +0000 (19:03 +0800)]
crimson/os: mark dtor of interface class "virtual"
we use a smart_ptr in Seastore to manage the life cycle of
`TransactionManager` which is in turn a subclass
`ExtentCallbackInterface`, so should declare the dtor of latter as
virtual.
projects / ceph.git / log