9.1.0

Merge pull request #6245 from liewegas/wip-setuptools

debian/control: python-setuptools is a build dependency

Reviewed-by: Alfredo Deza <adeza@redhat.com>

debian/control: python-setuptools is a build dependency

cd ./ceph-detect-init ; python setup.py build
Traceback (most recent call last):
  File "setup.py", line 23, in <module>
    from setuptools import setup
ImportError: No module named setuptools

Signed-off-by: Sage Weil <sage@redhat.com>

doc/release-notes: 9.1.0

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #6220 from dreamhost/wip-pybind-rados-omap-binary

pybind/rados: Fix binary omap values.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

pybind/rados, get_omap_vals: Fix pydoc type.

pydoc type for filter_prefix did not match code.

Signed-off-by: Robin H. Johnson <robin.johnson@dreamhost.com>

test: pybind/test_rados: add binary data.

Signed-off-by: Robin H. Johnson <robin.johnson@dreamhost.com>

pybind/rados: Fix binary omap values.

The prior code caused binary omap values to be discarded. This fixes
them to use the same model as the xattr iterator, and correctly return
binary data as python strings, eg:
'object_prefix': '\x15\x00\x00\x00rbd_data.449d2ae8944a'

Signed-off-by: Robin H. Johnson <robin.johnson@dreamhost.com>

Merge pull request #6193 from athanatos/wip-13240

ReplicatedPG::maybe_handle_cache: do not promote before checking full

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6190 from SUSE/wip-drop-my-conf-opt

ceph.spec.in: drop MY_CONF_OPTS

Reviewed-by: Ken Dreyer, <kdreyer@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #6183 from ceph/wip-fix-selinux-man-page

Fix selinux man page

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #6191 from SUSE/wip-concentrate-buildrequires

ceph.spec.in: move BuildRequires out of subpackages

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #6184 from dillaman/wip-13372-infernalis

librbd: invalidate object map on error even w/o holding lock

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rgw: fix wrong etag calculation during POST on S3 bucket.

Fixes: #11241
Backport: infernalis, hammer, firefly
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
(cherry picked from commit 742906ab5ee644d5932570bf95342697b673a498)

LibRBD: Adjust correct op latency scope

Op latency need to cover from queuing to finishing op.

Signed-off-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Jason Dillaman <jdillaman@redhat.com>

librados_test_stub: prevent interleaving of operations

It was possible for unrelated ops to interleave within a different
transaction for a given object.

Fixes: #13313
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #6044 from ceph/wip-12958

rgw: orphan tool shouldn't clean up head objects

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: orphan tool shouldn't clean up head objects

Fixes: #12958
Head objects are mutable, so removing them can race with object removal
and a later recreation, so we might end up cleaning them up when we don't
need to.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #6187 from SUSE/wip-13317-infernalis

doc: remove mention of --lazy-remove from radosgw-admin manpage

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6211 from SUSE/wip-dead-bool

CephxServiceHandler.cc: fix get_auth conditional

Reviewed-by: Kefu Chai <kchai@redhat.com>

CephxServiceHandler.cc: fix get_auth conditional

The KeyServer class has a public method get_auth() that returns a boolean
value. This value is being checked here - fix the conditional so it triggers
when get_auth() returns false.

http://tracker.ceph.com/issues/9756 References: #9756

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #6194 from dillaman/wip-13220-infernalis

rbd-replay-prep and rbd-replay improvements

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

ReplicatedPG::maybe_handle_cache: do not promote before checking full

Otherwise, ops with need_promote() can order past ops without it
causing an rgw write sequence like:

writefull 0~0
writefull 0~512kb
write 512kb~512kb

to reorder to

writefull 0~0 -- delayed: full
writefull 0~512kb -- delayed: full
write 512kb~512kb -- promote and execute
writefull 0~0 -- execute
writefull 0~512kb -- execute

Fixes: 13240
Signed-off-by: Samuel Just <sjust@redhat.com>

tests: removed obsolete rbd_replay test cases

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

ceph-dencoder: new rbd_replay trace file types

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-replay: added version control to trace output file

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6116 from jcsp/wip-cephfs-direntry

pybind/cephfs: fix DirEntry helpers

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6188 from liewegas/wip-13395

os/FileStore: kludge sloppy hammer temp objects into temp collection

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

rbd-replay-prep: added --verbose command line option

This new command-line will dump all the processed events as they
are extracted from the trace file.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-replay-prep: stream events to the prep file

Avoid building the entire prep event history in memory before
attempting to write the prepped trace file.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-replay-prep: simplify IO dependency calculation

Only track read-after-write and write-after-write IO dependencies
via the associated write completions.  All IO events after a write
completion are considered to be dependent and can be pruned down
to at most the number of concurrent IOs.  This reduces the prep
time from a simple 'rbd bench-write' from over 4 hrs down to seconds.

Fixes: #13378, #13384
Backport: hammer

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-replay: improve error messages

Fixes: #13221
Backport: hammer
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-replay-prep: support new read/write APIs

Added support for read2, write2, aio_read2, and aio_write2 APIs.

Fixes: 13325
Backport: hammer
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-replay-prep: avoid using assert statements on user inputs

Fixes: #13220
Backport: hammer
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6170 from jcsp/wip-spurious-warning

mds: avoid emitting cap warnings before evicting session

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #6128 from jcsp/wip-snapserver-crash

mds: fix SnapServer crash on deleted pool

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge branch 'wip-yrf-fix1' of git://github.com/H3C/ceph into infernalis

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge branch 'wip-13256' of git://github.com/ukernel/ceph into infernalis

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

qa: avoid using sudo in fsstress

This test required root in order to copy its built
binary into /usr (presumably to avoid rebuilding it).

That's not really a good thing anyway because there's
no guarantee that a binary in that path is the binary
we wanted, so just run the thing straight out of /tmp.  The
build is really quick anyway.

Signed-off-by: John Spray <john.spray@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #6186 from ceph/wip-12913-2

cls_rgw: fix bucket listing when dealing with invisible entries

os/FileStore: kludge sloppy hammer temp objects into temp collection

When we are running with a mixed hammer cluster, hammer primaries
will generate temp object names that are sloppy.  Make sure we still
put them into the temp collection.

Note that this isn't a problem on write because the primary (hammer)
OSD generated the transaction and explicitly specified a temp
collection; it's only transactions we do on our own with the sloppy
temp ghobject_t that trip over this.

Fixes: #13395
Signed-off-by: Sage Weil <sage@redhat.com>

ceph.spec.in: move python-sphinx BuildRequires to the right section

These belong in the section for distro-conditional dependencies.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph.spec.in: move BuildRequires out of subpackages

Since the main package and all the subpackages are built in the
same environment, concentrate all the BuildRequires in the main
package.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph.spec.in: drop MY_CONF_OPTS

http://tracker.ceph.com/issues/13402 Fixes: #13402

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #6185 from liewegas/wip-13382

osd/PG: fix generate_past_intervals

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #6182 from liewegas/wip-13379

mon: do not remove proxied sessions

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

doc: remove mention of --lazy-remove from radosgw-admin manpage

"The --lazy-remove flag has been removed over 3 years ago. Docs need to be
updated."

http://tracker.ceph.com/issues/13317 Fixes: #13317

Signed-off-by: Nathan Cutler <ncutler@suse.com>

osd/PG: fix generate_past_intervals

We may be only calculating older past intervals and have a valid
history.same_interval_since value, in which case the local
same_interval_since value will end at the newest old interval we had to
generate.

This was introduced by 70316541bbb115d9a35954bfba373cf1dc084b7e.

Signed-off-by: Sage Weil <sage@redhat.com>

librbd: invalidate object map on error even w/o holding lock

If there is a deep bug that prevents the object map from owning
the lock before objects are updated, flag the object map as invalid
regardless.

Fixes: #13372
Backport: hammer
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

selinux: Fix man page location

The SELinux man page was previously located in two places and the man
page that was supposed to be updated when rgw selinux changes were
proposed did not get updated properly. Fixing this by moving
selinux/ceph_selinux.8 to man/ceph_selinux.8. Also, populate EXTRA_DIST
with ceph_selinux.8.

Signed-off-by: Boris Ranto <branto@redhat.com>

man/Makefile-server.am: conditionalize make ceph_selinux manpage

http://tracker.ceph.com/issues/13353 Fixes: #13353

Modified-by: Boris Ranto <branto@redhat.com>
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #6171 from SUSE/wip-infernalis-obs

Get infernalis to build in openSUSE Build Service

Reviewed-by: Boris Ranto <branto@redhat.com>

mon: do not remove proxied sessions

A proxied session (see handle_forward) isn't registered, so it doesn't
need remove_session.  Moreover, s->con is null, so it will crash in
remove_session.

Fixes: #13379
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #6181 from athanatos/wip-13323

ReplicatedPG: consider IGNORE_CACHE for all maybe_handle_cache calls

Reviewed-by: Sage Weil <sage@redhat.com>

ceph.spec.in: remove comments regarding ceph UID/GID in SUSE

It is possible that the ceph user/group will not have fixed UID/GID in SUSE.
Instead, it is possible that the ceph package will depend on a separate package
whose sole purpose will be to create the ceph user/group if they do not exist.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph.spec.in: enable OBS post-build-checks to find systemd-tmpfiles

The openSUSE Build Service runs a number of "post-build checks" after the RPMs
have been generated. One of these tests the RPM scriptlets for idempotence.
Without this line in the specfile, the check fails on SLE_12 because it cannot
find the systemd-tmpfiles binary.

http://tracker.ceph.com/issues/13375 Fixes: #13375

Signed-off-by: Nathan Cutler <ncutler@suse.com>

etc/sysconfig/ceph: add CEPH_AUTO_RESTART_ON_UPGRADE

Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph.spec.in: Standardize systemd preun and postun scripts

Currently, the main ceph package and the ceph-radosgw behave
differently on upgrade. This commit unifies their behavior
to the following:

On package removal, disable and stop all related systemd units.

On package upgrade, do nothing unless there is a file /etc/sysconfig/ceph
containing a parameter CEPH_AUTO_RESTART_ON_UPGRADE. If parameter is set
to "yes", restart the systemd units iff they are running.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph.spec.in: fix for out-of-memory errors in OBS

Add "--param ggc-min-expand=20 --param ggc-min-heapsize=32768"
to RPM_OPT_FLAGS, ensuring gcc does not add debug symbols and is
more aggressive about garbage collection.

Thanks to Berthold Gunreben for debugging this issue.

http://tracker.ceph.com/issues/13352 Fixes: #13352

Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph.spec.in: Use _smp_mflags instead of _NPROCESSORS_ONLN

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Revert "install-deps.sh: use %bcond_with selinux on SLE/openSUSE"

This reverts commit 1df2cc2bf625a1f65ef4968d9cf313cc8ee9acc5.

ceph.spec.in: selinux by default on RHEL/Fedora, non-default on SUSE

Surround it with an appropriate conditional.

http://tracker.ceph.com/issues/13324 Fixes: #13324

Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph.spec.in: use %fillup_only macro in SLE/openSUSE

SUSE does not allow /etc/sysconfig/ceph to be owned by the ceph
package.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph.spec.in: distro-conditional groupadd and useradd

The -o option to groupadd/useradd is not recognized in SLE12. For
SLE/openSUSE, follow the openSUSE packaging guidelines.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph.spec.in: move systemd-tmpfiles stuff to ceph-common

Since the ceph user and group are created in the ceph-common
%pre scriptlet, the systemd-tmpfiles --create command needs
to run in that package.

tracker.ceph.com/issues/13346 Fixes: #13346

Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph.spec.in: remove SUSE-specific curses hack

Unclear what this was supposed to fix.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph.spec.in: lttng in SLES12 only

Over in the SUSE sector, we are trying to enable the SLE_12 and openSUSE_13.2
build targets. The lttng/babeltrace stuff is currently available only in
SLE_12.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph.spec.in: drop %insserv_prereq (obsoleted by systemd)

The %post, etc. scriptlets are no longer calling %insserv, et al.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph.spec.in: fix boost-random build dependency for SLE/openSUSE

In Fedora, RHEL and CentOS there is a package called boost-random.
In SLE/openSUSE, there is no such package.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

doc/release-notes: initial v9.1.0 notes

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #6110 from ceph/wip-13239-infernalis

rgw: set default value for env->get() call

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6142 from ceph/wip-13061

ceph.spec.in: Do not always restart the daemons on upgrades/removal

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #6105 from liewegas/wip-13262

mon: ignore messages from disconnected clients; clean up session checks

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #6176 from liewegas/wip-13283

osdc/Objecter: send FULL_TRY and FULL_FORCE ops despite full flag

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

osdc/Objecter: send FULL_TRY and FULL_FORCE ops despite full flag

If a request has the FULL_TRY or FULL_FORCE flag, send it despite a full
pool or cluster.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: allow ping through despite synch/quorum status, with session

Attach sessions to ping messages, too.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: drop ops on closed sessions early

We will throw it out in dispatch_op() anyway.  Remove the ancient (~2009)
reuse_caps logic too, which doesn't make any sense nowadays anyway.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: clean up _ms_dispatch

Lots of cruft here.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: all MonOpRequests have an attached Session

...except in dispatch, where we initially set it up.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: drop any ops from closed sessions in dispatch_op

If we've closed the session we shouldn't waste our time on this op.

Fixes: #13262 (we should not add a subscription to a disconnected session)
Signed-off-by: Sage Weil <sage@redhat.com>

mon: always set up session; move waitlist logic

Unconditionally attach a Session, so that when we push an op back through
dispatch_op it isn't in a weird broken state.

Move the waitinglist check to the bottom.  Use s->global_id == 0 to tell
whether a client has authenticated, and zap sessions based on that.

Signed-off-by: Sage Weil <sage@redhat.com>

mds: avoid emitting cap warnings before evicting session

In the case where a client dies, and another client immediately
tries to access a file locked by the dead client, we would
previously *sometimes* emit a "client.xyz isn't responding to
mclientcaps" warning to the cluster log, right before
evicting the stale session.  This was because the timeout
for the session eviction and the timeout for the
warning message are both 60s.

Fix this by checking the stale sessions before doing the
warning message check in Locker.  If a session is going
to get evicted in this tick, it will already be gone
by the time Locker thinks about emitting the warning
message.

Fixes: #13334
Signed-off-by: John Spray <john.spray@redhat.com>

mon: fix msg leak in resend_routed_requests

MForward takes its own ref so we can drop ours.

Signed-off-by: Sage Weil <sage@redhat.com>

Mon: Fix decoded message leak when this monitor is leader

Fix #13109
Signed-off-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6139 from liewegas/wip-13283

librados: add FULL_TRY flag; EDQUOT != ENOSPC

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librados: document new flag

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #6169 from rzarzynski/wip-13326

rgw: improve handling of already removed buckets in object expirer.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #6168 from SUSE/wip-13333-infernalis

ceph.spec.in: correctly declare systemd dependency for SLE/openSUSE

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

ceph.spec.in: correctly declare systemd dependency for SLE/openSUSE

Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit 7b1882f87509092380d11ec68b504a7bc1f77830)

osd/ReplicatedPG: exempt MDS from the failsafe check, too

This mimics the prior behavior.  We probably want to change this at some
point!

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: improve handling of already removed buckets in object expirer.

Fixes: #13326
Backport: infernalis
Signed-off-by: Radoslaw Zarzynski <rzarzynski@github.com>

Merge pull request #6096 from liewegas/wip-13260

osd/ReplicatedPG: preserve (some) flags when proxying reads

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #6136 from liewegas/wip-13114

osd/librados: fix notify completion race

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

ceph.spec.in: Do not always restart the daemons on removal

There is no need to restart the daemons when removing ceph-selinux
package if SELinux is not enabled.

Fixes: #13061
Signed-off-by: Boris Ranto <branto@redhat.com>

ceph.spec.in: Do not always restart the daemons on upgrades

This patch minimizes the amount of daemon stop/start procedures when
upgrading ceph-selinux package. With this patch, the daemons get
restarted only if SELinux is enabled and the SELinux policy version
changed in the meantime.

Fixes: #13061
Signed-off-by: Boris Ranto <branto@redhat.com>

Merge pull request #6137 from SUSE/wip-13324-infernalis

install-deps.sh: use %bcond_with selinux on SLE/openSUSE

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge remote-tracking branch 'jdurgin/wip-13281' into infernalis

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

ReplicatedPG: consider IGNORE_CACHE for all maybe_handle_cache calls

Otherwise, we could erroneously promote a snap after not promoting
the missing head for which we have a cached obc and ssc.

Fixes: #13323
Signed-off-by: Samuel Just <sjust@redhat.com>

install-deps.sh: use %bcond_with selinux on SLE/openSUSE

http://tracker.ceph.com/issues/13324 Fixes: #13324

Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph_test_rados_api_aio: test pool full gets EDQUOT when FULL_TRY flag is set

Signed-off-by: Sage Weil <sage@redhat.com>