build/ops: in jessie virtualenv is in package virtualenv

The python scripts are not yet compatible with python3, yet it is the
default on jessie. Force the creation of the virtualenv to use python2.7
instead. The wheelhouse is already explicitly populated for both python3
and python2.7 by install-deps.sh, regardless of the default interpreter.

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #8276 from dachary/wip-virtualenv-jewel

python: avoid long paths (part 2)

Reviewed-by: Alfredo Deza <adeza@redhat.com>

python: avoid long paths (part 2)

make CEPH_BUILD_VIRTUALENV=/tmp TESTS=ceph-detect-init/run-tox.sh check

sets the CEPH_DETECT_INIT_VIRTUALENV environment variable

CEPH_BUILD_VIRTUALENV=/tmp make TESTS=ceph-detect-init/run-tox.sh check

does not because Makefile-env.am overrides it.

Do not set CEPH_BUILD_VIRTUALENV in Makefile-env.am so the variable from
the environment can be used. Require that CEPH_BUILD_VIRTUALENV has a
trailing slash so that the default is just an empty variable.

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #8250 from dachary/wip-virtualenv-jewel

python: avoid long paths

Reviewed-by: Alfredo Deza <adeza@redhat.com>

python: avoid long paths

As a temporary measure because it overflows when jenkins tries to build
in a deep path. Run make with:

make CEPH_BUILD_VIRTUALENV=/tmp

and it will use these directories instead of the default.

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #8239 from jcsp/wip-15223

mon: fix mixed-version MDSMonitor

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7934 from ceph/wip-default-tcmalloc-cache-size

etc/sysconfig/ceph: set 128MB tcmalloc cache size

Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #8244 from linuxbox2/jewel-ldap-fixes

rgw_ldap fixes

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #8225 from SUSE/wip-radosgw-description

packaging: align radosgw package description

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #8217 from ceph/wip-rbd-mirror-init

rbd-mirror: systemd and upstart init scripts

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #8245 from theanalyst/fix/delete-zone-msg

rgw: fix error message for zone delete

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #8243 from theanalyst/fix/zone-rm-msg

rgw: typo fix for zone delete message

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix error message for zone delete

Currently if a zone is not a part of a zonegroup, an error message
is printed that zone doesn't exist, and the zone gets deleted anyway.
Since this is exhibited only if the zone isn't a part of a zonegroup
and we allow creation of such zones, clarify that the zone wasn't a
part of zonegroup instead

Fixes: #14951
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw_ldap:  move update of s->perm_mask

This should happen on successful auth, not only when a new user
record is stored.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw_ldap:  two bug fixes

1. only attempt auth on tokens which pass valid() check
2. ignore false positive search result (and fix result in that case)

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #8240 from dillaman/wip-15224

cls_rbd: throw error if set_features mask included non-mutable feature

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rgw: fix zone delete message

Though this error message is hard to reach, it is still wrongly printed
as failed to create instead of failed to delete.

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #8130 from tchaikov/wip-15113

mon: do not panic at seeing requests without session

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

cls_rbd: throw error if set_features mask included non-mutable feature

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mon: fix mixed-version MDSMonitor

Encode old-style until mon quorum features indicate
jewel, forbid creating multiple filesystems, and
when creating a single filesystem use the legacy fscid.

Fixes: #15223
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #8016 from ErwanAliasr1/evelu-15042

rpm,deb: remove conditional BuildRequires for btrfs-progs

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #8214 from athanatos/wip-feature-bits

Deprecate or free up a bunch of feature bits

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8162 from rjfd/wip-15142

rbd-mirror: use the mirroring directory to detect candidate images

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8204 from rjfd/wip-15143

librbd: enable/disable image mirroring automatically for pool mode

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

packaging: align radosgw package description

First, make the Debian package description mention that RGW aims to
implement the Swift API.

Second, replace the RPM package description with the Debian one, both for
consistency and because the Debian one is better.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #8172 from trociny/fix-test-rbd-mirror-position

test: rbd-mirror: compare positions using all fields

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8185 from trociny/wip-rbd-mirror-faster-exit

rbd-mirror: fix long termination due to 30sec wait in main loop

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8221 from jdurgin/wip-test-rbd-mirror

qa: add workunit to run ceph_test_rbd_mirror

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: fix long termination due to 30sec wait in Mirror::run loop

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #8191 from liewegas/wip-15180

qa/workunits/rest/test.py: don't use newfs

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #8101 from jjhuo/bluestore

os/bluestore: small fixes in bluestore StupidAllocator

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8178 from majianpeng/bluefs-closewriter

os/bluestore/BlueFS: Before reap ioct, it should wait io complete

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8220 from dillaman/wip-15202

journal: prevent race injecting new records into overflowed object

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa: add workunit to run ceph_test_rbd_mirror

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #8219 from jdurgin/wip-test-pool-create-cleanup

test_pool_create.sh: put test files in the test dir so they are cleaned up

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

journal: prevent race injecting new records into overflowed object

The recorded added after the overflow will not have been detached
from the object recorder.

Fixes: #15202
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8149 from trociny/test-rbd-mirror

test: rbd-mirror: add "switch to the next tag" test

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #8182 from trociny/wip-rbd-mirror-args

rbd-mirror: make remote context respect env and argv config params

Conflicts:
qa/workunits/rbd/rbd_mirror.sh (asok command format changed)

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8159 from trociny/wip-rbd-mirror-asok

rbd-mirror: use pool/image names in asok commands

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

packaging: added rbd-mirror startup scripts

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

upstart: new rbd-mirror daemon scripts

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

systemd: new ceph-rbd-mirror scripts

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test_pool_create.sh: put test files in the test dir so they are cleaned up

If they aren't cleaned up, tarball gitbuilders are unhappy.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #8216 from ceph/wip-rbd-maint-ops

qa/workunits/rbd: new online maintenance op tests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #8183 from ceph/wip-rgw-fix-mdlog

Wip rgw fix mdlog

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

qa/workunits/rbd: new online maintenance op tests

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8212 from yehudasa/wip-15181

cls/rgw: fix use of timespan

Reviewed-by: Sage Weil <sage@redhat.com>

cls/rgw: fix use of timespan

Need to use make_timespan() when referring to seconds.

Fixes: #15181
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #8209 from liewegas/wip-13992

ceph_test_rados_misc: shorten mount timeout

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph_test_rados_misc: shorten mount timeout

This might fix #13992.

Signed-off-by: Sage Weil <sage@redhat.com>

os/filestore: fix warning

os/filestore/FileStore.cc: In member function ‘int FileStore::_zero(const coll_t&, const ghobject_t&, uint64_t, size_t)’:
os/filestore/FileStore.cc:3294:24: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
       if (offset + len > st.st_size) {
                        ^

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #7932 from theanalyst/mailmap/10.0.4

mailmap for 10.0.4

Reviewed-by: Loic Dachary <ldachary@redhat.com>

qa/workunits/rest/test.py: don't use newfs

It doesn't work anymore.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #8121 from liewegas/wip-15136

mon: remove 'mds setmap'

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

rbd: rbd-mirroring: Added unit tests to test image mirroring behaviour

Signed-off-by: Ricardo Dias <rdias@suse.com>

rbd: rbd-mirroring: Fixed if clause in mirror_image_get_info

This fix targets the case when journaling is not enabled but
someone calls mirror_image_get_info.

Signed-off-by: Ricardo Dias <rdias@suse.com>

rbd: rbd-mirroring: Disable image mirroring when journaling feature is disabled

Signed-off-by: Ricardo Dias <rdias@suse.com>
Fixes: #15143

rbd: rbd-mirroring: Enable image mirroring when journaling feature is enabled

Signed-off-by: Ricardo Dias <rdias@suse.com>

rbd: rbd-mirroring: Enable image mirroring upon create with journaling feature

Signed-off-by: Ricardo Dias <rdias@suse.com>

Merge pull request #8137 from ceph/wip-15146-ceph-dep

rpm: remove sub-package dependencies on "ceph"

Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Jun Huang <hjwsm1989@gmail.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #7838 from rldleblanc/pg_prime-enable

mon: enable 'mon osd prime pg temp' by default

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7939 from yangdongsheng/remove_v1

mon: standardize Ceph removal commands

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7721 from xiexingguo/xxg-wip-fixdc

os/filestore: fix result handling logic of destroy_collection

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7754 from xiexingguo/xxg-wip-14845

osd: fix inaccurate counter and skip over queueing an empty transaction

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7970 from Yan-waller/yj-wip-async

msg/async: _try_send trim already sent for outcoming_bl more efficient

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #8127 from ceph/wip-15122

ceph_test_msgr: reduce test size to fix memory size

Reviewed-by: Sage Weil <sage@redhat.com>

Merge branch 'wip-13990' of git://github.com/tchaikov/ceph

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8160 from yuyuyu101/jewel-spdk

os/bluestore/NVMEDevice: make IO thread using dpdk launch

Merge pull request #8163 from liewegas/wip-15151

osd/ReplicatedPG: clear watches on change after applying repops

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #7359 from songbaisen/song9

mon: consider pool size when creating pool

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7674 from majianpeng/bluestore-sync-transaction

os/bluestore: make bluestore_sync_transaction = true can work.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7860 from majianpeng/bluestores-clone-overlay

os/bluestore: clone overlay data

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7930 from ceph/wip-debian-librgw-tests

debian: package librgw_file* tests

Merge remote-tracking branch 'gh/wip-strict-debian-deps'

Merge pull request #7957 from majianpeng/bluestore-read

os/filestore: require offset == length == 0 for full object read; add test

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #8039 from gregsfortytwo/wip-14962-greg

osd/ReplicatedPG: be more careful about calling publish_stats_to_osd()

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8096 from liewegas/wip-15073

osd: ensure new osdmaps commit before publishing them to pgs

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw: use current period id when it is empty for mdlog list and shard lock/unlock

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #8167 from theanalyst/mstart/rgw_ports

mstart: start rgw on different ports as well

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

osd: assert that we don't try to use an OSDMap > superblock.newest_map

handle_osd_map now adds OSDMaps to the cache as it receives them
but before they are stable.  Ensure that we don't try to use them.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: commit osdmaps before exposing them to PGs

handle_osd_map and the PGs use different sequencers when writing
their updates.  We therefore need to make sure new osdmaps are
committed to disk before we expose them to PGs, lest they update
their info to reference a new osdmap that hasn't actually
committed yet.

This doesn't happen with FileStore because transactions are
ordered when they are queued, but it does affect BlueStore.

Fix by splitting handle_osd_map into two phases, one that just
persists stuff, and the second half that publishes the new maps to
the rest of the OSD.

Fixes: #15073
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #8170 from yehudasa/wip-rgw-sync-fixes-3

Wip rgw sync fixes 3

Reviewed-by: Casey Bodley <cbodley@redhat.com>

os/bluestore/BlueFS: Before reap ioct, it should wait io complete.

When change write mode of FileWrite(in _flush_range) from buffer to
direct. osd met segment fault.
This because call _close_writer don't check related ios whether
complete.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

mon: ignore msg without session attached

there is chance that a connection gets reset, and the session attached
to a message that was sent over this connection is reset. i.e.
con->set_priv(NULL) in Monitor::ms_handle_reset(), but we are about to
handle_command() this message after the message is somehow requeued. so,
not all MonOpRequests have an attached Session, especially those are
re-dispatched. we could check conn->is_connected() everywhere, but
that's error-prone and hard to maintain. so, in this change:

* we need to check for the session before looking at it closer.
* hold a reference to it when holding it, because in another thread,
  ms_handle_reset() could be freeing it.

Fixes: #15113
Signed-off-by: Kefu Chai <kchai@redhat.com>

mon: remove 'mds setmap' command

This command has no discernable purpose, and is difficult and
dangerous to support anyway.

Fixes: #15136
Signed-off-by: Sage Weil <sage@redhat.com>

test: rbd-mirror: different log and asok location for local/remote contexts

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

rbd-mirror: make remote context respect env and argv config params

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #8179 from zhjwpku/zhjwpku

doc/architecture.rst: remove redundant word "across"

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/architecture.rst: remove redundant word "across"

Signed-off-by: Zhao Junwang <zhjwpku@gmail.com>

rbd-mirror: command line options should override environment

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

os/bluestore: rename need_size to want_size in StupidAllocator.

Allocate() could return extent with size < need_size, so rename it per
Sage's suggestion.

Signed-off-by: Jianjian Huo <samuel.huo@gmail.com>

os/bluestore: avoid duplicated searches during allocating free extents

When StupidAllocator searches for free extents, if hint is not 0(most times),
it will search extents starting from hint and up in each applicable bin,
if not found, then search all extents in those bins again.

If the extent to find(>= needs) exists below hint and is not in the choose_bin(),
let's say it's in the bin_X; then during this twice searches, all extents from
hint and up in bins before bin_X will be searched twice. This patch will eliminate
those unnecessary searches.

Signed-off-by: Jianjian Huo <samuel.huo@gmail.com>

features: deprecate CEPH_FEATURE_OSD_PROXY_FEATURES

Can't be removed until CEPH_FeATURE_OSD_MIN_SIZE_RECOVERY is also
ready.

Signed-off-by: Samuel Just <sjust@redhat.com>

features: deprecate CEPH_FEATURE_OSD_MIN_SIZE_RECOVERY

Can't be removed until CEPH_FEATUERE_OSD_PROXY_METADATA goes
as well.

Signed-off-by: Samuel Just <sjust@redhat.com>

features: deprecate CEPH_FEATURE_OSD_TRANSACTION_MAY_LAYOUT

We can't remove this until the other 1<<46's also go.

Signed-off-by: Samuel Just <sjust@redhat.com>

features: deprecate CEPH_FEATURE_OSD_OBJECT_DIGEST

We can't actually reuse this until all of the 1<<46 bits are
ready.

Signed-off-by: Samuel Just <sjust@redhat.com>

features,osd/: deprecate CEPH_FEATURE_OSD_REPOP feature bit

This allows us to rip out the code for supporting the old
repop format.  We can't actually reuse this bit until all
4 1<<46 bits are deprecated for one release.

Signed-off-by: Samuel Just <sjust@redhat.com>

features: deprecate CEPH_FEATURE_OSD_SET_ALLOC_HINT

Signed-off-by: Samuel Just <sjust@redhat.com>

features: deprecate CEPH_FEATURE_PACKED_RECOVERY

Signed-off-by: Samuel Just <sjust@redhat.com>

features: deprecate CEPH_FEATURE_MON_SCRUB

Signed-off-by: Samuel Just <sjust@redhat.com>