librbd: resize should only update image size within header

Previously, the whole RBD image format 1 header struct was
re-written to disk on a resize operation.

Fixes: #13674
Backport: infernalis, hammer, firefly
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit d5be20b6d4646284571568ab28cbf45b0729390b)

9.2.0

Merge pull request #6444 from liewegas/wip-pg-key

osd/PG: tolerate missing epoch key

Reviewed-by: Samuel Just <sjust@redhat.com>

osd/PG: tolerate missing epoch key

An orphan PG may have an info attr but no epoch key.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #6344 from dillaman/wip-13559-infernalis

librbd: potential assertion failure during cache read

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #6363 from ceph/wip-init-rbdmap

Drop redhat-lsb-core dependency

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6366 from liewegas/wip-osd-fixboot-infernalis

osd: fix OSDService vs Objecter init order

Reviewed-by: Samuel Just <sjust@redhat.com>

osd: fix OSDService vs Objecter init order

This reverts c7d96a5ed1d2cb844622af29b13705b8f7be6be7, but still keeps
the Objecter init *after* we have authenticated.  This way we don't
crash when we get mon messages like MOSDPGCreate, and we also don't
request maps we aren't prepared to handle.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph.spec.in: We no longer need redhat-lsb-core

Drop the redhat-lsb-core dependency as it is no longer necessary on
fedora/rhel.

The other two init scripts do not use redhat-lsb-core either. The
init-ceph.in conditionally requires /lib/lsb/init-functions and does not
use any of the functions defined in that file (at least not directly).
The init-radosgw file includes /etc/rc.d/init.d/functions on non-debian
platforms instead of /lib/lsb/init-functions file so it does not require
redhat-lsb-core either.

Signed-off-by: Boris Ranto <branto@redhat.com>

init-rbdmap: Rewrite to use logger + clean-up

This patch rewrites the init-rbdmap init script so that it uses logger
instead of the log_* functions. The patch also fixes various smaller
bugs like:
* MAP_RV was undefined if mapping already existed
* UMNT_RV and UMAP_RV were almost always empty (if they succeeded) ->
  removed them
* use of continue instead RET_OP in various places (RET_OP was not being
  checked after the switch to logger messages)
* removed use of DESC (used only twice and only one occurrence actually
  made sense)

Signed-off-by: Boris Ranto <branto@redhat.com>

librbd: potential assertion failure during cache read

It's possible for a cache read from a clone to trigger a writeback if a
previous read op determined the object doesn't exist in the clone,
followed by a cached write to the non-existent clone object, followed
by another read request to the same object.  This causes the cache to
flush the pending writeback ops while not holding the owner lock.

Fixes: #13559
Backport: hammer
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: reproduce crash during read-induced CoW

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6356 from ceph/wip-fix-release-notes

doc/release-notes.rst: recovery isn't in the unified queue yet

doc/release-notes.rst: recovery isn't in the unified queue yet

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #6313 from ceph/wip-osd-compat

osd: allow firefly -> infernalis+ upgrades

Reviewed-by: David Zafman <dzafman@redhat.com>

osd: init objecter after we authenticate

Move objecter init until after we authenticate.  We ignore
osdmap messages while initializing anyway so it's pointless to
ask for them.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #6248 from rzarzynski/wip-rgw-swift-acctmd-doc-infernalis

rgw, doc: remove remark for lack of custom account metadata of Swift API

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #6212 from rzarzynski/wip-rgw-objexp-doc-infernalis

rgw, doc: object expiration of Swift API is supported now.

Reviewed-by: Loic Dachary <ldachary@redhat.com>

osd/PG: make upgrade() use sequencer

Signed-off-by: Sage Weil <sage@redhat.com>

Revert "os/FileStore: require upgrade to hammer before moving beyond"

This reverts commit 10cb507d9d52464b2fd28b3cc84a92328944e462.

Revert "osd: require an upgrade to hammer first"

This reverts commit c0f83df412c962a6b3a2f3cdd1d7c8c795ef1146.

Revert "ceph-objectstore-tool: require hammer upgrade"

This reverts commit b474991df74ab4d31b011bb6c5973fcd3c2e0179.

Revert "osd: drop support for pre-hammer pg metadata"

This reverts commit cd4e676e6d45c8166290ef834d73c2a0bda98fa2.

Revert "ceph-objectstore-tool: drop support for pre-pgmeta PGs"

This reverts commit b297e6d2b206c0da733ae9d4ac5fb321f113ae5d.

Revert "os: drop deprecated collection_* attr methods"

This reverts part of commit
893e00bc0fd793090a35d275919ca0348e9f05b0.

[we leave KeyValueStore untouched]

Signed-off-by: Sage Weil <sage@redhat.com>

Revert "os/FileStore: fix version check"

This reverts commit 14d3a2b82bea668ee2e9898d0b612812d8f7e8cc.

Merge pull request #6249 from dillaman/wip-13274-infernalis

LTTng-UST tracing should be dynamically enabled

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #6284 from dillaman/wip-13483-infernalis

qa: remove legacy OS support from rbd/qemu-iotests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa: remove legacy OS support from rbd/qemu-iotests

RHEL7 derivatives were failing test 002 since they were using
legacy test cases for now unsupported OSes.

Fixes: #13483
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6273 from jdurgin/wip-13496

librbd: fix rebuild_object_map() when no object map exists

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: fix rebuild_object_map() when no object map exists

Enabling the object map feature and then attempting to rebuild it
results in an assert failure, since the number of objects was
accidentally passed to ObjectMap::aio_resize() instead of the size of
the image.

Fixes: #13496
Signed-off-by: Josh Durgin <jdurgin@redhat.com>

ceph_context: remove unsafe cast for singletons

It was previously assumed that a CephContext singleton would
inherit from CephContext::AssociatedSingletonObject, but it was
not enforced.  This could result in unknown behavior when the
singleton is destroyed due to the implied virtual destructor.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6267 from dillaman/wip-13305-infernalis

ceph.spec.in: only run systemd-tmpfiles on ceph run directory

Reviewed-by: Sage Weil <sage@redhat.com>

ceph.spec.in: only run systemd-tmpfiles on ceph run directory

There is a kmod bug that affects RHEL7 derivatives which will
result in device permissions being reset back to defaults.

Fixes: #13305
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6265 from dillaman/wip-cmake-rbd-replay-infernalis

CMake: fix rbd_replay error

CMake: fix rbd_replay error

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

osd: conditionally initialize the tracepoint provider

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librados: conditionally initialize the tracepoint provider

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: conditionally initialize the tracepoint provider

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

common: new tracing config options

Config options to enable LTTng-UST tracepoint providers for
the OSD, OSD objectstore, librados, and librbd.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tracing: merge tracepoint provider makefiles into main makefile

This results in the tracepoint provider shared libraries being
placed in the library path for unittests.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tracing: dynamic tracepoint provider helper

The TracepointProvider class is a configuration observer.  When
tracing is enabled, it will dynamically load the associated
tracepoint provider.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

packaging: add new tracepoint probe shared libraries

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

ceph.spec.in: add new tracepoint probe shared libraries

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

lttng: move tracepoint probes to dynamic libraries

LTTng-UST initializes itself at program load, which means it is
currently always enabled.  This can lead to issues with SElinux
and AppArmor which might restrict access to the necessary device
files.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6261 from tchaikov/wip-12748-infernalis

osd: fix the snapshot reads of evicted tiering pool

Reviewed-by: Sage Weil <sage@redhat.com>

osd: fix the snapshot reads of evicted tiering pool

reset ssc->exsits in finish_ctx() if the ctx->cache_evict is true, and
the head is removed.

Fixes: #12748
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6215 from dillaman/wip-13429-infernalis

restore pre-infernalis API compatibility

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6221 from athanatos/wip-13434

Wip 13434

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6217 from dillaman/wip-13380-infernalis

cls: new force-promotion flag for class methods

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #6230 from liewegas/wip-crushtool-err

mon/OSDMonitor: put crushtool error in log

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #6235 from dzafman/wip-13462

osd: Correct the object_info_t::decode() version

Reviewed-by: Samuel Just <sjust@redhat.com>

doc/infernalis: hate hate

Signed-off-by: Sage Weil <sage@redhat.com>

doc/release-notes: i hate rst

Signed-off-by: Sage Weil <sage@redhat.com>

doc/release-notes: final infernalis notes

Signed-off-by: Sage Weil <sage@redhat.com>

Merge branch 'infernalis' of jenkins.front.sepia.ceph.com:ceph/ceph into infernalis

doc/release-notes: fix some attributions

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

doc/release-notes: infernalis notable changes

Signed-off-by: Sage Weil <sage@redhat.com>

Revert "common, global: use lttng ust functions for handling fork-like calls"

This reverts commit 811898912f3e43d7312efe5d06c496e1e7e22fa4.

Merge pull request #6237 from ceph/wip-13458

rgw: location constraints should return api name

Reviewed-by: Sage Weil <sage@redhat.com>

rgw, doc: remove remark for lack of custom account metadata of Swift.

Backport: infernalis
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

9.1.0

Merge pull request #6245 from liewegas/wip-setuptools

debian/control: python-setuptools is a build dependency

Reviewed-by: Alfredo Deza <adeza@redhat.com>

debian/control: python-setuptools is a build dependency

cd ./ceph-detect-init ; python setup.py build
Traceback (most recent call last):
  File "setup.py", line 23, in <module>
    from setuptools import setup
ImportError: No module named setuptools

Signed-off-by: Sage Weil <sage@redhat.com>

doc/release-notes: 9.1.0

Signed-off-by: Sage Weil <sage@redhat.com>

ReplicatedPG::maybe_handle_cache_detail: always populate missing_oid

Signed-off-by: Samuel Just <sjust@redhat.com>

ReplicatedPG::_rollback_to: handle block on full correctly

If we fail to promote the clone due to a full cache tier, we need
to block writes on the head just as if we were waiting on the
promote.  Generally therefore, we block writes on head if we block
an op on a full cache tier.  Add a new blocked set to support this.

Fixes: 13434
Signed-off-by: Samuel Just <sjust@redhat.com>

osd: Correct the object_info_t::decode() version

Caused by: 13b9dc70

Fixes: #13462
Backport: Hammer, Firefly

Signed-off-by: David Zafman <dzafman@redhat.com>

rgw: location constraints should return api name

Fixes: #13458
The S3 location constraints request should return the api name,
not the region name.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

mon/OSDMonitor: put crushtool error in log

Putting it in *ss is useless for mkpool callers who don't return
an error string.  Log it to the mon log, too.

Signed-off-by: Sage Weil <sage@redhat.com>

ReplicatedPG: allow maybe_handle_cache to return status detailing what happened

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #6220 from dreamhost/wip-pybind-rados-omap-binary

pybind/rados: Fix binary omap values.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

pybind/rados, get_omap_vals: Fix pydoc type.

pydoc type for filter_prefix did not match code.

Signed-off-by: Robin H. Johnson <robin.johnson@dreamhost.com>

test: pybind/test_rados: add binary data.

Signed-off-by: Robin H. Johnson <robin.johnson@dreamhost.com>

pybind/rados: Fix binary omap values.

The prior code caused binary omap values to be discarded. This fixes
them to use the same model as the xattr iterator, and correctly return
binary data as python strings, eg:
'object_prefix': '\x15\x00\x00\x00rbd_data.449d2ae8944a'

Signed-off-by: Robin H. Johnson <robin.johnson@dreamhost.com>

Merge pull request #6193 from athanatos/wip-13240

ReplicatedPG::maybe_handle_cache: do not promote before checking full

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6190 from SUSE/wip-drop-my-conf-opt

ceph.spec.in: drop MY_CONF_OPTS

Reviewed-by: Ken Dreyer, <kdreyer@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

rgw: fix wrong etag calculation during POST on S3 bucket.

Fixes: #11241
Backport: infernalis, hammer, firefly
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
(cherry picked from commit 742906ab5ee644d5932570bf95342697b673a498)

Merge pull request #6183 from ceph/wip-fix-selinux-man-page

Fix selinux man page

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #6191 from SUSE/wip-concentrate-buildrequires

ceph.spec.in: move BuildRequires out of subpackages

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #6184 from dillaman/wip-13372-infernalis

librbd: invalidate object map on error even w/o holding lock

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

LibRBD: Adjust correct op latency scope

Op latency need to cover from queuing to finishing op.

Signed-off-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Jason Dillaman <jdillaman@redhat.com>

librados_test_stub: prevent interleaving of operations

It was possible for unrelated ops to interleave within a different
transaction for a given object.

Fixes: #13313
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

cls: new force-promotion flag for class methods

Class methods that use the cls_get_request_origin should not be
proxied to a base tier -- otherwise the origin will reflect the
cache tier instead of the client.

Fixes: #13380
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librados: restored pre-infernalis API compatibility

Fixes: #13429
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

buffer: restored pre-infernalis API compatibility

Fixes: #13429
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6044 from ceph/wip-12958

rgw: orphan tool shouldn't clean up head objects

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: orphan tool shouldn't clean up head objects

Fixes: #12958
Head objects are mutable, so removing them can race with object removal
and a later recreation, so we might end up cleaning them up when we don't
need to.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw, doc: mention that Swift objexp is supported now.

Backport: infernalis
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #6187 from SUSE/wip-13317-infernalis

doc: remove mention of --lazy-remove from radosgw-admin manpage

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6211 from SUSE/wip-dead-bool

CephxServiceHandler.cc: fix get_auth conditional

Reviewed-by: Kefu Chai <kchai@redhat.com>

CephxServiceHandler.cc: fix get_auth conditional

The KeyServer class has a public method get_auth() that returns a boolean
value. This value is being checked here - fix the conditional so it triggers
when get_auth() returns false.

http://tracker.ceph.com/issues/9756 References: #9756

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #6194 from dillaman/wip-13220-infernalis

rbd-replay-prep and rbd-replay improvements

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

ReplicatedPG::maybe_handle_cache: do not promote before checking full

Otherwise, ops with need_promote() can order past ops without it
causing an rgw write sequence like:

writefull 0~0
writefull 0~512kb
write 512kb~512kb

to reorder to

writefull 0~0 -- delayed: full
writefull 0~512kb -- delayed: full
write 512kb~512kb -- promote and execute
writefull 0~0 -- execute
writefull 0~512kb -- execute

Fixes: 13240
Signed-off-by: Samuel Just <sjust@redhat.com>

tests: removed obsolete rbd_replay test cases

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

ceph-dencoder: new rbd_replay trace file types

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-replay: added version control to trace output file

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6116 from jcsp/wip-cephfs-direntry

pybind/cephfs: fix DirEntry helpers

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6188 from liewegas/wip-13395

os/FileStore: kludge sloppy hammer temp objects into temp collection

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

rbd-replay-prep: added --verbose command line option

This new command-line will dump all the processed events as they
are extracted from the trace file.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>