osd: Handle recovery read errors

Fixes: http://tracker.ceph.com/issues/13937
Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Fix log messages

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: CLEANUP: Remove unused pending_read member

Remove unused struct

Signed-off-by: David Zafman <dzafman@redhat.com>

Revert "test: Disable tests due to recovery race"

This reverts commit 5bc55338f5a1645bc651811fae2f89ad855ff86e.

Conflicts:
src/test/Makefile.am (no longer exists)
src/test/erasure-code/Makefile.am (no longer exists)

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #11449 from dzafman/wip-scrub-prep

test/osd-scrub-repair.sh: Use test case specific object names to help…

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/osd-scrub-repair.sh: Use test case specific object names to help with diagnostics

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #11422 from xiexingguo/xxg-wip-bluestore-1011

os/bluestore: fix race condtion during blob spliting

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11394 from liewegas/wip-bluestore-shared-blob-intrusive

os/bluestore: use std::unordered_map for SharedBlob lookup

Merge pull request #11233 from zealoussnow/wip-0927

doc: fix start development cluster operation in index.rst

Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: use std::unordered_map for SharedBlob lookup

Many blobs aren't shared.  Save 8 bytes per SharedBlob by using a normal
unordered_map instead of instrusive::set.

More importantly, perhaps, it avoids us having to tune the intrusive
unordered_set size manually.  std::unordered_map does this automatically
for you, but the intrusive one does not.  And it's unclear how to
statically size it given that it's a per-collection structure and we have
no idea how many objects we'll have, how many blobs per object, and how
many objects will be cloned.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11183 from liewegas/wip-msgr-features

msg: make loopback Connection feature accurate all the time

Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: improve ExtentMap::reshard() readability a little

As ep, sp and esp can be a little confusing.

Signed-off-by: xiexingguo <xie.xingguo@zte.com.cn>

Merge pull request #11427 from ifed01/wip-bluestore-fix-test

test/store_test: fix errors on the whole test suite run caused by the…

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: add sanity check to shard offset when updating

Signed-off-by: xiexingguo <xie.xingguo@zte.com.cn>

os/bluestore: fix race conditon during blob spliting

Signed-off-by: xiexingguo <xie.xingguo@zte.com.cn>

test/store_test: fix errors on the whole test suite run caused by the lack of properites restore in some test cases

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: don't scan omap kvpair during fsck()

Because it does not check anything, and removing it can
speed up the process of fsck()

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: kiill dead member from bluestore_shared_blob_t

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #11150 from renhwztetecs/renhw-wip-mds-server-open

mds/server: merge the snapshot request judgment

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11209 from david-z/wip-clear-dup-logic-mdsmonitor

mon: clear duplicated logic in MDSMonitor

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11358 from stiopaa1/mds_sessionmaph_unneededCount

mds/SessionMap.h: remove unneeded use of count

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11362 from batrick/i17531

mds: respawn using /proc/self/exe

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11359 from batrick/i17276

client: add pid to metadata

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11395 from dillaman/wip-17549

librbd: ignore notify errors on missing image header

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #11404 from bassamtabbara/wip-fix-yasm-system-check

cmake: Fix for cross compiling

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11356 from dillaman/wip-17528

librbd: update internals to use optional separate data pool

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #11297 from stiopaa1/mds_sessionmap_avoidCopyingPlusConst

mds/SessionMap.cc: avoid copying and add const

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11283 from jcsp/wip-print-legacy-client-fscid

mds: include legacy client fsid in FSMap print

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

cmake: Fix for cross compiling

The check for yasm tool was calling uname -m which will not
work when cross compiling. Use CMAKE_SYSTEM_PROCESSOR instead.

Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>

Merge pull request #11206 from dzafman/wip-16474

osd: Remove extra call to reg_next_scrub() during splits

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #11399 from theanalyst/wip-release-notes

scripts/release-notes: allow title guesses from gh tags & description update

Reviewed-by: Loic Dachary <ldachary@redhat.com>

build/ops: ceph-release-notes guess pr title based on gh tags

We add a switch --use-tags which helps guess the component of the pr
based on gh tags, still needs work, but provides better estimates when
looking at really large release notes where guessing each pr would be
too much work

Signed-off-by: Abhishek Lekshmanan <alekshmanan@suse.com>

scripts: add description to ceph-release-notes

Signed-off-by: Abhishek Lekshmanan <alekshmanan@suse.com>

librbd: include pool id in data block prefix if data pool enabled

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: initialize the data pool IO context during image open state machine

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: store data pool id with image header during creation

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11355 from dillaman/wip-17424

rbd: add support for separate image data pool

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #10731 from yehudasa/wip-rgw-sync-plugins

rgw: sync modules, metadata search

Reviewed-by: Casey Bodley <cbodley@redhat.com>

librbd: ignore notify errors on missing image header

The rename op on v1 images fails since the header no longer exists. In
the general case, the removal of the header object will also fail the
watcher which has its own recovery path.

Fixes: http://tracker.ceph.com/issues/17549
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11363 from xiexingguo/xxg-wip-fix-rbd-warn

cls_rbd: silence compiler warnings

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11371 from stiopaa1/mds_cinode_removeCount

mds/CInode.h: remove unneeded use of count

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11383 from stiopaa1/mds_fsmap_passSharedPtrByConstRef

mds/FSMap: pass shared_ptr by const ref

rbd: include data pool name within info command

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: support overriding image data pool when creating images

Fixes: http://tracker.ceph.com/issues/17424
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11290 from trociny/wip-17017

rbd-mirror HA: move librbd::image_watcher::Notifier to librbd::object_watcher

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11260 from runsisi/wip-fix-mirror-image-disable

librbd: fix rollback if failed to disable mirroring for image

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11301 from liewegas/wip-bluestore-buffered-write

os/bluestore: allow default to buffered write

Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #11370 from ceph/wip-krbd-unmap-options

rbd: expose rbd unmap options

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11255 from dzafman/wip-scrub-boundary

osd: fix scrub boundary to not include a SnapSet

Reviewed-by: Sage Weil <sage@redhat.com>

msg: make loopback Connection feature accurate all the time

In 626360aab05545ddacb0ac28e54a70e31fd5695d we made the
OSD cluster loopback connection CEPH_FEATURES_ALL, but
all other loopback connections got features == 0.  I
can't come up with any reason we wouldn't want those
connections to have accurate feature bits, so let's just
use CEPH_FEATURES_ALL for all of them.

While we're here, make the cflags argument required.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11377 from xiexingguo/xxg-wip-bluestore-1008

os/bluestore: drop inline_dirty from struct ExtentMap

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: narrow scope of 'pos'

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #11281 from jcsp/wip-17466

mds: handle bad standby_for_fscids in fsmap

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

os/bluestore: get rid of multiple call to dirty_blob()

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #10722 from stiopaa1/mon_moncap_addMoveToStrings

mon/MonCap.h: add std::move for std::string

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10916 from jordan41177/fix-17169

mon: OSDMonitor: fix the check error of pg creating

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10720 from wjwithagen/wip-wjw-freebsd-socket-shutdown

msg/simple/Accepter.cc: replace shutdown() with selfpipe event in poll() (FreeBSD)

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10601 from stiopaa1/mon_monclient_subwantRemoveCount

mon/MonClient.h: remove repeated searching of map

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11145 from atheism/msg-async-fix-specifier

msg: fix format specifier for unsigned value id

Reviewed-by: Kefu Chai <kchai@redhat.com>

mds/FSMap: pass shared_ptr by const ref

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

os/bluestore: fix raw blob encoded size

The maximum length of fixed fields of bluestore_blob_t shall be:

uint64_t sbid                       ///< 8 bytes
uint32_t compressed_length_orig     ///< 4 bytes
uint32_t compressed_length          ///< 4 bytes
uint32_t flags                      ///< 4 bytes
uint8_t csum_type                   ///< 1 byte
uint8_t csum_chunk_order            ///< 1 byte
unused_t unused;                    ///< 2 bytes

8 + 4 + 4 + 4 + 1 + 1 + 2 = 24

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mds/SessionMap.h: remove unneeded use of count

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

os/bluestore: add counter to trace number of garbage collection reads

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: kill dead lines to keep code clean

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #11379 from zealoussnow/fix-dead-link

doc: fix dead link "Hardware Recommendations"

Reviewed-by: Kefu Chai <kchai@redhat.com>

mon: clear duplicated logic in MDSMonitor

Signed-off-by: Zhi Zhang <zhangz.david@outlook.com>

doc: fix dead link "Hardware Recommendations"

Signed-off-by: Leo Zhang <nguzcf@gmail.com>

os/bluestore: drop inline_dirty from struct ExtentMap

We don't need this member as we can simply clear inline_bl
to mark it as dirty.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

rbd/cls_rbd: silence compiler warnings

Signed-off-by: xiexingguo <xie.xingguo@zte.com.cn>

mds/CInode.h: remove unneeded use of count

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #10057 from Yan-waller/yj-wip-rgwradosccremover-0701

rgw: minor cleanup

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #10674 from ceph/wip-rgw-no-empty-amzrqid

rgw/rest: don't print empty x-amz-request-id

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10849 from zhangsw/master

rgw: remove a redundant judgement when listng objects.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

os/bluestore: allow default to buffered write

Add config option to buffered writes (in the absense of
a client WILLNEED hint).  Set the config option to false,
though, so that the default behavior is unchanged.

2Q should do a half-decent job of preventing this from
pushing actually-hot data from the cache when it's
enabled, but we'll need to do more testing first.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11353 from dillaman/wip-17423

librbd: API changes to support separate data pool

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

rgw: index metadata in elasticsearch using realm name for path

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw_admin: sync status command shows if not syncing from zone

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: setting sync-from zone by name not by id

Using the zone name is easier and clearer.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw_admin: update usage

add refrence to --sync-from*

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw_admin: config options to set sync_from and sync_from_all

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: support partial mesh for zone sync

zone configuration now includes two new fields: sync_from_all
which is boolean, and sync_from, which is a least of zones to
sync from. By default sync_from_all is set to true. Sync will
only happen from all the zones, or from the specified zones if
sync_from all is false. We also check to see whether zone can
export data (depending on tier_type).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: es module, remove entry on delete

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add cr to send DELETE to remove endpoint

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rest conn functions cleanup, only append zonegroup if not empty

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: es sync module, store custom metadata

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: es sync module, keep object mtime

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: utility function to dump iso8601

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: es sync module, store acl information

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: es sync module, store object attrs

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: es sync module, send object info to elasticsearch

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cmake: fix linkage of ceph_test_librgw_file_nfsns

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: initial implementation of elasticsearch sync module

sync module that will handle rgw metadata indexing.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: a new cr to send http PUT requests

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: allow null store in RGWRESTConn

We're not necessarily going to connect to rgw/s3 endpoints,
we only need store param to handle s3 signing.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: REST client, don't sign requests if empty key

If key is not passed in, don't try to sign the request.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move the rgw sync code module around

No real code change

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: some abstraction around log sync module

Moving code that fetches remote object meta to its own classes.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: log sync module gets source object's meta

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: propagate attrs, mtime, size of remote object

Use new rgwx-stat http param that allows getting only object's
meta. Use that when calling stat_remote_object().

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add cr to stat remote obj

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>