os/ObjectStore: add noexcept to ensure move ctor is used

otherwise vector::push_back() will use the copy ctor if it resizes,
to enforce its strong exception guarantee.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8193 from liewegas/wip-bluestore

os/bluestore: a few fixes

Merge pull request #8404 from cbodley/wip-rgw-realm-pull-fix

radosgw-admin: fix for 'realm pull'
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #8417 from dillaman/wip-librbd-mirroring-unsupported

librbd: avoid throwing error if mirroring is unsupported

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #8325 from trociny/wip-rbd-mirror-test-misc

test: rbd-mirror: script improvements for manual testing

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #8413 from runsisi/wip-fix-eexist

cls_journal: fix -EEXIST checking

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: avoid throwing error if mirroring is unsupported

Attempting to remove an image will remove the image from the mirroring
directory.  However, if the OSD is older and doesn't support this
new feature, avoid throwing an error.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rgw: add exclusive flag to set_as_default()

this dodges the race in RGWRealm::create() and RGWZoneParams::create()
that decides whether to set the new object as a default. by calling
set_as_default() with exclusive=true, it will fail with EEXIST if a
default is already set

it also fixes an issue with 'realm pull' on a secondary zone, where a
'default' zone may be created but never actually set_as_default()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add exclusive flag to RGWRealm::create_control

this was causing failures in 'radosgw-admin realm pull':

  ERROR creating control for new realm realm0: (17) File exists

'realm pull' calls RGWRealm::create() with exclusive=false, but
RGWRealm::create_control() was using exclusive=true

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #8324 from tchaikov/wip-mon-cleanup

mon: remove remove_legacy_versions()

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

cls_journal: fix -EEXIST checking

Signed-off-by: runsisi <runsisi@zte.com.cn>

Merge pull request #8371 from runsisi/wip-fix-eexist

cls_rbd: fix -EEXIST checking in cls::rbd::image_set

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8400 from qhjindev/fix_mon

mon: remove unnecessary comment for update_from_paxos

Merge pull request #8399 from oritwas/wip-cmake-mirrorwatcher

cmake: add missing librbd/MirrorWatcher.cc and librd/ObjectWatcher.cc

mon: remove unnecessary comment for update_from_paxos

The return value comment for update_from_paxos is unnecessary.

Signed-off-by: Qinghua Jin <qhjin_dev@163.com>

cmake: add missing librbd/MirrorWatcher.cc and librd/ObjectWatcher.cc

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #8394 from xiexingguo/xxg-wip-compile

rgw: fix compiling error

Merge pull request #8358 from dachary/wip-releases

releases: what is merged where and when ?

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.cz>
Reviewed-by: Sage Weil <sage@redhat.com>

releases: what is merged where and when ?

Signed-off-by: Loic Dachary <loic@dachary.org>

releases: update understanding the release cycle

Signed-off-by: Loic Dachary <loic@dachary.org>

release-notes: v10.0.5 release notes

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #7875 from badone/wip-coverity-rgw_file-unused

[rgw] Check return code in RGWFileHandle::write

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

rgw: fix compiling error

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #8200 from H3C/wip-rgw-bugfixb

rgw/rgw_admin:fix bug about list and stats command
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

qa/workunits/rbd: use cluster1/2 instead local/remote as cluster names

After adding tests that start daemons in both clusters, local/remote
names look confusing.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

qa/workunits/rbd: add helper to determine rbd-mirror local cluster

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #8373 from wjwithagen/patch-4

ceph-detect-init/run-tox.sh: FreeBSD: No init detect

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #8387 from dreamhost/wip-mp-listpartresult-etag

rgw: Multipart ListPartsResult ETag quotes

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

qa/workunits/rbd: improvements for manual testing

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

rgw: Multipart ListPartsResult ETag quotes

ListPartsResult output has always missed quotes on the ETag since it was
first committed.

Fixes: #15334
Backports: hammer, infernalis
Signed-off-by: Robin H. Johnson <robin.johnson@dreamhost.com>

Merge pull request #8372 from jcsp/wip-15309

qa: update rest test cephfs calls

Merge pull request #8258 from asheplyakov/bug-12065

monclient: avoid key renew storm on clock skew

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8348 from xiexingguo/xxg-wip-failurepending

osd: fix failure report handling during ms_handle_connect()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8369 from xiexingguo/xxg-wip-tick

osd: fix tick relevant issues

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8354 from tchaikov/wip-missing-cls

osd/ClassHandler: only dlclose() the classes not missing

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8355 from dillaman/wip-15029

librbd: send notifications for mirroring status updates

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

cls_rbd: fix -EEXIST checking in cls::rbd::image_set

Signed-off-by: runsisi <runsisi@hust.edu.cn>

Merge pull request #8382 from athanatos/wip-15241

osd: fix epoch check in handle_pg_create

Reviewed-by: Sage Weil <sage@redhat.com>

OSD::handle_pg_create: check same_primary_since

Rather than add a flag to handle_pg_peering_evt, check
same_primary_since here and pass the create event to
handle_pg_peering_evt as if it originated at the current epoch (the
project_pg_history checks in handle_pg_peering_evt will be noops).

Fixes: tracker.ceph.com/issues/15241
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #8147 from xiaoxichen/fix-get-map

osd/OSD.cc: finish full_map_request every MOSDMap message.

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

librbd: integrate listener for new mirroring notification payloads

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: send notifications on mirroring updates

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8084 from sunfch/wip-rgw-init-max-response

rgw: accept data only at the first time in response to a request

librbd: helper methods for mirroring notifications

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: mirroring object notification types

Initial payloads for updated mirror pool mode and image state.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: generic object watcher and mirroring watcher

The ObjectWatcher can handle generic watch and recovery of
RADOS objects.  The MirroringWatcher will handle updates
to the RBD mirroring object.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8375 from rjfd/wip-15265

librbd: disable image mirroring when image is removed

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8331 from sileht/sileht/rados-cython

pybind/rados: python3 fix

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rbd: rbd-mirroring: Automatically disable image mirroring when image is removed

Signed-off-by: Ricardo Dias <rdias@suse.com>
Fixes: #15265

os/bluestore: _do_write: fix _do_zero_tail_extent to handle shared extents

Also add a test case to reproduce this case.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: _do_zero: simply truncate up if past eof

Avoid the _do_zero_tail_extent here; we're zeroing so this is no different
than a truncate up

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: prevent rename src from getting trimmed from lru

We were putting it at the end of the LRU *and* not holding a reference.

Fix by letting oldo carry a ref to the Onode.  This avoids a subsequent
lookup and also prevents it from getting trimmed.

Note that we could also put it at the front of the LRU, but given that it
is the rename source, I'm not sure that's appropriate.  For example,
temp recovery objects get renamed into place and we don't want to keep
negative onodes for those around.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: _do_zero: handle unzeroed tail blocks

- _do_write_zero when handling tail extent if it is
  shared.
- handle tail extent we are also zeroing later (not in
  the tail handler).  THis mirrors how _do_write is
  structured.
- handle eof to offset zeroing if our offset starts in
  the tail extent.  just like _do_write's wal path.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: _do_write_zero: preserve onode size

Instead of handling this in each caller, make the helper
keep onode.size intact.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: _do_write: handle gap from eof to offset in wal path

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: _do_allocate: fix whitespace

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: _do_truncate: no need to zero trailing block on truncate down

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: _do_write: only use append case for aligned eof

The append case here only works if the EOF was aligned
and there is thus no partial-block zeroing we need to
do.  If the condition fails, we fall through to the
generic WAL path below.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: _do_zero_tail_extent: do not assume tail block is zeroed

We will change this shortly.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: _do_zero_tail_extent helper

We do this from both _do_truncate and _do_write.  No
functional change here.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: break _do_zero out of _zero

Seems useless now, but it maintains symmetry and we'll
need it later.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: do not do gift beyond the max ratio

Signed-off-by: Sage Weil <sage@redhat.com>

global/signal_handler: fix warning

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: release wal_cleaning extents in order

We need to order the freelist updates so that they match the
commit order of the actual transactions.  Otherwise we might, say,
set a key here, delete it in the _txc_update_fm, but commit in
the wrong order and end up with the key surviving.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: avoid temp released interval_map for wal cleanup

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/FreelistManager: detect overlapping extents on init

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: trim onode_map when adding new items

This ensures we maintain the cache size even for read workloads.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: keep onode_map max_size

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: wait for wal op that wrote cached tail before using

If the cached tail is written by a WAL op, we need to wait for it to
apply before we use the cached tail.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: note txc seq for tail cache

We'll need this soon...

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: assign per-Sequencer seq to each TransContext

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: add AppendWalVsTailCache test

This specifically exercises a WAL overwrite racing with a subsequent
append that uses the previous writes cached tail.  We need to make sure
the use of the cached_tail waits for the previous WAL write or else our
update of tail block will be overwritten with the prior (WAL) value.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix fsck vs enodes

- fix used_blocks accounting for shared extents
- make sure enodes only exist when objects follow

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: bluestore_inject_wal_apply_delay

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: reorder onode_map vs enode_set ctor/dtor

The onode_map should get declared after the enode_set so that
the enode_set is empty when it is destroyed.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: only cache tail in _pad_tail if we have the whole block

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: only use tail if tail_offset matches

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix off-by-one on caching tail block

We can cache the tail as long as we overwrite all prior content;
we don't have to extent the size.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/KernelDevice: print buffered flag in debug line

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-detect-init/run-tox.sh: FreeBSD: No init detect

 - Freebsd has /etc/rc.d with scripts to start daemons at boottime.
   Test for that are currently not integrated with Ceph tests

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #8368 from tchaikov/wip-cmake-fix-rbd

cmake: add StandardPolicy.cc to librbd

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8337 from jmunhoz/fix-15292

mon: remove unused variable

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #8352 from runsisi/wip-fix-missing-iter

rbd-mirror: fix missing increment for iterators

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8374 from trociny/fix-cls-rbd-read_peers

cls_rbd: read_peers: update last_read on next cls_cxx_map_get_vals

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7062 from wonzhq/scrub-prio-pool

osd: add the support of per pool scrub priority

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8210 from liewegas/wip-12814

osd: add 'proxy' cache mode

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #8315 from mslovy/wip-clean-pg

osd: duplicated clear for peer_missing

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8330 from xiexingguo/xxg-wip-race

common: fix race during optracker switches between enabled/disabled mode

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #8335 from liewegas/wip-13992

mon/MonClient: fix shutdown race

Reviewed-by: Kefu Chai <kchai@redhat.com>

cls::rbd: read_peers: update last_read on next cls_cxx_map_get_vals

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

qa: update rest test cephfs calls

Since the ceph task was already creating filesystems
during setup, I presume that these calls only ever
worked when they were using the same names as
the existing filesystem.

Fixes: http://tracker.ceph.com/issues/15309
Signed-off-by: John Spray <john.spray@redhat.com>

osd: skip heartbeat_check for osd which we haven't sent ping to

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd: initialize last_pg_stats_ack correctly on first pg_stats message sending

So we don't potentially make an incorrect judgement in the tick_without_osd_lock()
procedure if OSD's first pg_stats message was sent, which as a result shall cause
an unnecessary hunting of new monitor.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd: update osd_stat_updated field if we force to report to monitor

On tick_without_osd_lock() waking up, we will force an immediate
report to monitor if a long enough period has elapsed since our
last report was sent.

However, the send_pg_stats() procedure is a noop unless either
osd_stat_updated field is set or pg_stat_queue is empty.
Since the latter condition is non-deterministic, we shall
manually set osd_stat_updated here to kick a pg relevant report
off.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #7857 from tchaikov/wip-14844

ceph.spec.in: enable lttng and babeltrace explicitly

thanks to ktdreyer, we have lttng and babel in EPEL-7 now.

Reviewed-by: Nathan Cutler <ncutler@suse.com>

cmake: add StandardPolicy.cc to librbd

fixes the librbd build.

Signed-off-by: Kefu Chai <kchai@redhat.com>

rbd-mirror: fix missing increment of iterators

Signed-off-by: runsisi <runsisi@hust.edu.cn>

Merge pull request #8287 from dillaman/wip-15232

rbd-mirror: initial failover / failback support

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

test/pybind/test_rados: force setting readonly cache mode

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #7712 from vitek/listbucket_encoding_type_fix

rgw: S3: set EncodingType in ListBucketResult

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>