os/LevelDBStore:fix bug when compact_on_mount

Initialize the performance counter(l_rocksdb_compact)
before use it in compact().

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

os/RocksDBStore: fix bug when compact_on_mount

Initialize the performance counter(l_rocksdb_compact)
before use it in compact().

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

Merge pull request #4366 from BCLibCoop/bclibcoop/rgw-hostnames

rgw: Document & improve region hostnames

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

doc: Changes in region hostname do not seem to take effect until radosgw is restarted.

Signed-off-by: Robin H. Johnson <robbat2@gentoo.org>

Merge pull request #4335 from liewegas/wip-osd-df

mon: fix min/max variance for 'cpeh osd df' when osds are marked out

Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw: Document a lurking problem I see with subdomain/domain splitting

Signed-off-by: Robin H. Johnson <robbat2@gentoo.org>

rgw: Speed up the building of valid de-duplicated hostnames for regions.

Signed-off-by: Robin H. Johnson <robbat2@gentoo.org>

doc: Document region hostnames settings.

The hostnames key in region settings was undocumented. Also includes how
the setting interacts with the ``rgw dns name`` configuration setting.

Signed-off-by: Robin H. Johnson <robbat2@gentoo.org>

Merge pull request #3481 from ktdreyer/wip-rpm-rm-old-fedoras

ceph.spec.in: remove reference to EOL Fedoras

Merge pull request #4316 from Ved-vampir/librdb

librdb: add perf counters descriptions

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4361 from oritwas/wip-11091

rgw: shouldn't return content-type: application/xml if content length is 0

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #4351 from ceph/wip-11256

Wip 11256

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4363 from nilamdyuti/wip-doc-ceph-object-gateway

doc: Corrected the steps of Apache configuration for Debian-based
distros in object gateway installation guide.

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

doc: Corrected the steps of Apache configuration for Debian-based distros
in object gateway installation guide.

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

mon: make note about osd df calc bug

Signed-off-by: Sage Weil <sage@redhat.com>

mon: fix min variance calc in 'osd df'

If the reweight is 0, ignore the osd for the purposes of calculating the
min.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #4354 from zhouyuan/fix_ct_doc

doc: Fix misleading overlay settting in Cache Tier

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4359 from ceph/wip-fix-ceph-dencoder-build-master

Move ceph-dencoder build to client
Rework mds/Makefile.am to support a dencoder client build
rgw/Makefile.am: Populate DENCODER_SOURCES properly
Dencoder should never be built with tcmalloc

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

rgw: shouldn't return content-type: application/xml if content length is 0

Fixed: #11091

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

Move ceph-dencoder build to client

The patch simply moves the ceph-dencoder build from server part of the
Makefiles to client part of the Makefiles.

Refs: #10691

Signed-off-by: Boris Ranto <branto@redhat.com>

Rework mds/Makefile.am to support a dencoder client build

The patch adds all the mds sources to DENCODER_SOURCES to allow a
dencoder client build. The patch also splits the Makefile.am file to
better accomodate the change.

Refs: #10691

Signed-off-by: Boris Ranto <branto@redhat.com>

rgw/Makefile.am: Populate DENCODER_SOURCES properly

Dencoder is built if ENABLE_CLIENT is set. However, the rgw/Makefile.am
populated DENCODER_SOURCES only if WITH_RADOSGW was set. The patch fixes
this and populates DENCODER_SOURES if ENABLE_CLIENT is set.

Signed-off-by: Boris Ranto <branto@redhat.com>

Dencoder should never be built with tcmalloc

The patch adds disabled perfglue stubs to DENCODER sources in order to
avoid tcmalloc-enabled ceph-dencoder builds.

Refs: #10691

Signed-off-by: Boris Ranto <branto@redhat.com>

Merge pull request #4348 from ceph/wip-mon-clean

mon: remove dead MonitorStore code

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

librdb: add perf counters descriptions

Sign-off-by: Alyona Kiseleva <akiselyova@mirantis.com>

doc: Fix misleading overlay settting in Cache Tier

set-overlay is needed in either writeback or readonly mode of Cache
Tier setup.

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

ceph_json: add decode / encoder for multimap

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls_rgw: use multimap to keep pending operations in bucket index

Fixes: #11256
Multiple concurrent requests might be sent using the same tag, need the
entry map to be able to hold multiple entries.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: generate new tag for object when setting object attrs

Fixes: #11256
Backport: firefly, hammer

Beforehand we were reusing the object's tag, which is problematic as
this tag is used for bucket index updates, and we might be clobbering a
racing update (like object removal).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #4350 from almightybeeij/master

Check that delta_sum.stats.sum.num_object_copies and delta_sum.stats.sum...

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4349 from ceph/wip-dist-systemd-tmpfiles-d

#11383: packaging: ship systemd/ceph.tmpfiles.d in tarballs

packaging: ship systemd/ceph.tmpfiles.d in tarballs

Prior to this commit, the tarballs did not contain
any files under the top-level "systemd" directory. This caused problems
with RPM builds on Fedora and RHEL 7, because as of commit
aa88364f30e2d2f254ade185a83ba263b48e2a73, those RPMs depend on the
systemd/ceph.tmpfiles.d file.

(Longer-term we might want to improve the tarball generation code to be
less complex/manual.)

http://tracker.ceph.com/issues/11383 Refe: #11383

Reported-by: Greg Farnum <gfarnum@redhat.com>
Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Check that delta_sum.stats.sum.num_object_copies and delta_sum.stats.sum.num_object are greater than zero

This fixes division by zero.

Signed-off-by BJ Lougee <almightybeeij@gmail.com>

mon: remove dead MonitorStore code

Signed-off-by: Sage Weil <sage@redhat.com>

doc/release-notes: v0.94.1

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #4317 from ceph/wip-rpm-with-systemd-comparison

ceph.spec.in: set _with_systemd on RHEL 7 and Fedora

Merge pull request #4327 from dachary/wip-wheezy

install-deps.sh: Debian GNU/Linux wheezy needs backports

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4341 from dachary/wip-test-timeout

tests: display ceph report when stuck

Reviewed-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Merge pull request #4334 from smithfarm/master

do_autogen: add missing '--without-lttng' option to usage()

Reviewed-by: Loic Dachary <ldachary@redhat.com>

tests: display ceph report when stuck

When the cluster is stuck (wait_for_clean times out), display ceph
report to stderr for debugging purposes.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4239 from yuyuyu101/wip-librbd-conf

Wip librbd conf

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4323 from ceph/wip-11363

librbd: ImageWatcher should cancel in-flight ops on watch error

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4140 from ceph/wip-rbd-rebuild-object-map

rbd: add ability to rebuild corrupt/missing object maps

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4325 from wonzhq/async-resize-2

librbd/AsyncResize: avoid dup incrementing refresh seq

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4336 from ceph/wip-11369

librbd: failure to update the object map should always return success

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

install-deps.sh: Debian GNU/Linux wheezy needs backports

It is not enough for the backports to be available, they also need to be
explicitly allowed to take precedence whenever a package is installed
indirectly. This is causing problems with libp11-kit-dev pulled by
libcurl4-gnutls-dev.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4319 from ceph/wip-11348

librbd: update default image format to version 2

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

do_autogen.sh: add missing '--without-lttng' option to usage

Signed-off-by: Nathan Cutler <ncutler@suse.cz>

librbd: notify of header update after rebuilding object map

This will allow clients to detect that the object map is no longer
invalid and to reload the object map from disk.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: rebuilding object map shouldn't update piecemeal

The object map is now rebuilt in-memory and written back to the
OSDs as a single operation.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: add test case for cls_rbd object_map_save

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls_rbd_client: add object_map_save helper method

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls_rbd: add object_map_save method

Allow the object map rebuild process to build the full object map
in memory and save it to disk with one operation.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: add ImageWatcher test for new rebuild object map request

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: added rebuild_object_map test to test_librbd

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: librados_test_stub reads should deep-copy

If a client of librados_test_stub modified a bufferlist
retrieved via a read call, the client will actually be
changing the contents of the file.  Therefore, read calls
should deep-copy the contents of the buffer::ptrs.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: allow snapshot object maps to be updated

Rebuild will need to be able to update/resize the object maps
for image snapshots.  This was previously not permitted.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: update in-memory object map before on-disk update completes

When multiple IO updates are occurring within the same object, this
will allow only a single object map update request to be sent to the
OSD.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls_rbd: treat zero-byte object maps as missing

Acquiring the lock on a missing object map will create an
empty object. Treat the empty object as a non-existant
object map to support rebuilding corrupt object maps while
holding the exclusive lock.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: add object map rebuild command

An invalid object map can now be rebuilt via the rbd CLI.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: add rebuild_object_map to public API

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librados_test_stub: add another overload of aio_operate

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: connect async rebuild object map to state machine

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: require callers to ObjectMap::aio_update to acquire lock

This is needed to allow an atomic compare and update operation
from the rebuild object map utility.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: added RebuildObjectMapRequest state machine

It will verify the object map is properly sized, verify
the existence of each object within the image (snapshot),
and clear the invalid object map flag once complete.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: correct basic object map errors during refresh

If the object map is corrupt on-disk or too small for the image,
correct these basic issues as soon as possible.  The object map
is still flagged as invalid, but there will be less required repair
work if future IO is able to properly update the object map.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: use generic helper for issuing async requests

resize, flatten, and rebuild object map now use the same
bootstrap code for sending the request to the remote lock owner
or executing the request locally.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: connect ImageWatch rebuild request to async rebuild method

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: add preliminary methods for rebuilding object map

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: add hooks for rebuild object map to ImageWatcher

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: add new notify payload for rebuild object map

The request will be sent to the client which owns the exclusive
lock to rebuild the object map for the current image HEAD.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: failure to update the object map should always return success

If an object map update fails, the object map will be flagged as
invalid.  However, if a subsequent update failure occurs, the error
code will propagate back to the caller.

Fixes: #11369
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: librados_test_stub reads should deep-copy

If a client of librados_test_stub modified a bufferlist
retrieved via a read call, the client will actually be
changing the contents of the file.  Therefore, read calls
should deep-copy the contents of the buffer::ptrs.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #4330 from dachary/wip-hammer-release-notes

release-notes: draft v0.94.1 release notes

librbd: ImageWatcher should cancel in-flight ops on watch error

Upon an watch error notification from librados, pending requests
should be canceled and in-flight IO should be flushed prior to
unlocking the image.

Fixes: #11363
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #4333 from dachary/wip-fedora

tests: add Dockerfile for fedora 21

Reviewed-by: Kefu Chai <kchai@redhat.com>

tests: add Dockerfile for fedora 21

Signed-off-by: Loic Dachary <ldachary@redhat.com>

release-notes: draft v0.94.1 release notes

Signed-off-by: Loic Dachary <ldachary@redhat.com>

librbd: Fix image_watcher test

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Librbd: Don't affect global md_config_t

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Librbd: Don't apply to global configserver

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

librbd/AsyncResize: avoid dup incrementing refresh seq

The refresh_seq is incremented in notify_change when calling
notify_async_complete after the locker owner completes the resize
request.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

Merge pull request #4306 from dachary/wip-ec-troubleshooting

doc: an erasure code crush ruleset can be dynamically modified

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4320 from liewegas/wip-crush-straw2-div0

crush/mapper: fix divide-by-0 in straw2

Reviewed-by: Yann Dupont <yann@objoo.org>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

crush/mapper: fix divide-by-0 in straw2

If the item weight is 0 we don't want to divide; instead draw a minimal
value.

Fixes: #11357
Reported-by: Yann Dupont <yd@objoo.org>
Tested-by: Yann Dupont <yd@objoo.org>
Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/rbd/copy.sh: removed deprecated --new-format option

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: ensure old-format RBD tests still work

Override the RBD default image format back to version 1
to ensure tests properly cover the old format.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: deprecate --new-format command-line option

Image formats can be explicitly chosen via the --image-format X
command-line option.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

common/config_opts: updated rbd_default_format

New images will now default to format 2 RBD images.

Fixes: #11348
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librados_test_stub: added rados_ioctx_cct implementation

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #4174 from trociny/wip-10976.master

osd: fix PG::all_unfound_are_queried_or_lost for non-existent osds

Reviewed-by: Kefu Chai <tchaikov@gmail.com
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #4111 from wonzhq/hitsets-check

osd/ReplicatedPG: correct the checking if the promoting object is in other hit sets

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #4136 from XinzeChi/wip-pg-delete-sleep

osd: return fast if PG::deleting is true in snap_trimmer and PG::scrub

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4178 from XinzeChi/wip-tracker-message

osd: more useful message to find out potential unhealth osd

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4155 from wonzhq/prdop

osd/ReplicatedPG: don't check order in finish_proxy_read

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4139 from Ved-vampir/oscnt

Perf counters descriptions

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3928 from dmitryya/rados_bench_with_iops_metric

rados bench: Add IOPS metric to `rados bench'

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4253 from majianpeng/scrub-fix2

Scrub bug fix

Reviewed-by: Samuel Just <sjust@redhat.com>

ceph.spec.in: set _with_systemd on RHEL 7 and Fedora

Commit 71a5090bca049a43e30a7f0cf99141950ef9c5dd added a "_with_systemd"
conditional to the RPMs, but I erred with the version comparison
operator, so this only applied to RHEL 8+, not RHEL 7+.

Adjust the conditional so that it will really apply to RHEL 7+. While
we're here, add Fedora as well.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
Reported-by: Boris Ranto <branto@redhat.com>

Merge pull request #4224 from rzarzynski/wip-10682_bucketlisting

rgw: add support for end_marker parameter for GET on Swift container.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>