Merge remote-tracking branch 'gh/jewel'

Merge pull request #8310 from tchaikov/wip-fix-rbdmap-manpage

makefile: fix rbdmap manpage

Reviewed-by: Sage Weil <sage@redhat.com>

cmake: conditionally install man pages

this matches its counterpart in automake
* install fuse related man pages only if HAVE_LIBFUSE
* install rbd related man pages only if WITH_RBD

Signed-off-by: Kefu Chai <kchai@redhat.com>

autotools: add rbdmap to dist_man_MANS

this fixes the rpm/deb build failure
was introduced by 20c8650

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8038 from badone/wip-crushtool-crash

crushtool: Don't crash when called on a file that isn't a crushmap

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8309 from tchaikov/wip-15240

test/system/rados_list_parallel: print oid if rados_write fails

Reviewed-by: Sage Weil <sage@redhat.com>

xxHash: fix submodule commit

This was inadvertantly rewinded in b9a14312edd6a4edb0f4783ab08ad97e0e9ba03f

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #8211 from liewegas/wip-jewel

doc: draft notes for jewel

Merge pull request #8044 from dzafman/wip-8885

osd: fix dump_ops_in_flight races

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #8156 from liewegas/wip-pg-query

osd/PG: indicate in pg query output whether ignore_history_les would help

Revewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #8177 from majianpeng/signal-handle

global/signal_handler: print thread name in signal handler

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/system/rados_list_parallel: print oid if rados_write fails

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/release-notes: rgw typo

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #8294 from theanalyst/rgw/bucket-quota-check

rgw:Use count fn in RGWUserBuckets for quota check

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge remote-tracking branch 'origin/wip-15065'

Support new standby_for_fscid option, letting us assign daemons to FS clusters.

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge branch 'wip-mfsmap-features' of git://github.com/jcsp/ceph

Clean up some code in the new multi-fs handling. Make its output
more precise and add some debug/repair config options.

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Conflicts:
src/mds/FSMap.cc

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #7952 from ukernel/wip-14360

mds: allow client to request caps when opening file

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #8180 from ukernel/jewel-15167

Be more careful about directory fragmentation and scrubbing

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #8115 from ukernel/jewel-cephfs-data-scan-misc

Minor fixes around data scan in some scenarios

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #8297 from dillaman/wip-cls-rbd-mirror-image-list

cls_rbd: mirror_image_list should return global image id

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #8253 from wido/issue-15119

rgw: Do not send a Content-Type on a '304 Not Modified' response

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #8224 from SUSE/wip-15212

rbdmap: add manpage

Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

cls_rbd: mirror_image_list should return global image id

The global image id is needed to crosslink images when an image is
replicated between multiple clusters.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rgw:Use count fn in RGWUserBuckets for quota check

We already have a count function from RGWUserBuckets class which gives
the total size of the map, use that instead of getting the map and doing
a count.

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

10.1.0

Merge pull request #8285 from dmick/jewel

debian/control: try installing virtualenv first, if it exists

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #8251 from majianpeng/vstart

vstart: make -k with optional mon_num.

Reviewed-by: Kefu Chai <kchai@redhat.com>

vstart: make -k with optional mon_num.

Default mon_num=3, if we change this and using -k to create a new
cluster. It don't work. So we must record the old mon_num.
The same for osd_num/mds_num/rgw_num.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #8288 from dmick/wip-default-virtualenv-path

Makefile-env.am: set a default for CEPH_BUILD_VIRTUALENV

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Makefile-env.am: set a default for CEPH_BUILD_VIRTUALENV

We need to shorten the virtualenv path for non-Jenkins builds
too; gitbuilders were also overflowing the interpreter path limit

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #8235 from trociny/wip-rbd-mirror-asok

rbd-mirror: asok commands to get status and flush on Mirror and Replayer level

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

debian/control: try installing virtualenv first, if it exists

Debian Jessie (at least) now has the virtualenv command in
package 'virtualenv', which depends on python3-virtualenv, and
that's how you get the virtualenv command there.  Earlier
Debian-based releases still use python-virtualenv.  Jessie's
virtualenv command still creates a python2 venv by default.

This will require a followup fix for install-deps.sh
to handle the | correctly, because currently it sends
the string to apt-get, which treats it as a regular expression
alternation of the two package names, so will try to install both.

The problem occurs when packages are installed without Recommended
packages (because jessie's virtualenv package currently Recommends
python-virtualenv); this is the case under pbuilder, and also using
'mk-build-deps --install', and I suspect under other automated
package-building tools.

Note that Build-Requires processing is not specified to
perform the searches in left-to-right order, so even this is
a tenuous workaround.  We probably need distro-specific
debian/ directories.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #8280 from rjfd/wip-15238

rbd-mirror: switch fsid over to mirror uuid

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7976 from ukernel/jewel-15008

Jewel 15008

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #7902 from jcsp/wip-pycephfs-flock

pybind: add flock to libcephfs python bindings

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #7901 from jcsp/wip-pycephfs-fixes

libcephfs: fix python tests and fix getcwd on missing dir

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #8271 from vumrao/wip-vumrao-15244

doc: rgw_region_root_pool option should be in [global]

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #8184 from theanalyst/fix/rgw-admin-zone-msg

rgw: add zone delete to rgw-admin help

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #8264 from cbodley/wip-rgw-admin-errors

radosgw-admin: 'period commit' supplies user-readable error messages

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #8190 from yehudasa/wip-rgw-sync-fixes-4

Wip rgw sync fixes 4

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rbd: rbd-mirroring: Replayer registers in remote journal with mirror_uuid

Signed-off-by: Ricardo Dias <rdias@suse.com>
Fixes: #15238

Merge pull request #8282 from dillaman/wip-15250

rbd: allow librados to prune the command-line for config overrides

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7907 from dachary/wip-subman-facts

script: subscription-manager support

Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #8022 from roidayan/fix_compile_against_latest_accelio

xio: fix compilation against latest accelio

rbd: allow librados to prune the command-line for config overrides

Fixes: #15250
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8279 from dachary/wip-virtualenv-jessie

build/ops: in jessie virtualenv is in package virtualenv

Reviewed-by: Alfredo Deza <adeza@redhat.com>

build/ops: in jessie virtualenv is in package virtualenv

The python scripts are not yet compatible with python3, yet it is the
default on jessie. Force the creation of the virtualenv to use python2.7
instead. The wheelhouse is already explicitly populated for both python3
and python2.7 by install-deps.sh, regardless of the default interpreter.

Signed-off-by: Loic Dachary <loic@dachary.org>

test/rgw: use bucket for data checkpoint instead of data

data checkpoint is not accurate, as data log might not show repeating changes.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync, update and flush high marker

We need to update the high marker even if we skip entries, and eventually
flush it. This is needed so that our position in the bucket index log that
we follow is reflected correctly.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: update data log even if canceling index operation

Need to update the data log anyway, so that it reflects that there were
changes in the bucket index log.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: break out of run_sync() if going down

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix bucket_instance_meta_info mtime json decoding

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #8277 from SUSE/wip-15246

rpm: prefer UID/GID 167 when creating ceph user/group

Reviewed-by: David Disseldorp ddiss@suse.de
Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #8222 from SUSE/wip-14984

systemd: set up environment in rbdmap unit file

Reviewed-by: Boris Ranto <branto@redhat.com>

xio: fix compilation against latest accelio

Do not use C struct initilization style to avoid eclipse errors.
We also don't use c++ struct initalize as members are not specified
and it just happens that in latest accelio version one of the fields
has changed in the middle of the struct.

Signed-off-by: Roi Dayan <roid@mellanox.com>

Merge pull request #8268 from yehudasa/wip-15243

rgw: convert plain object to versioned (with null version) when removing

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #8148 from roidayan/xio-add_prefix_to_xio_logs

xio: add prefix to xio msgr logs

Merge pull request #8227 from roidayan/xio-call_xio_init_first

xio: xio_init needs to be called before any other xio function

Merge pull request #8207 from theanalyst/doc/mstart

doc: very basic doc on mstart

Merge pull request #8237 from oritwas/wip-rgw-fix-mrun

cmake: fix mrun to handle cmake build structure

rgw: convert plain object to versioned (with null version) when removing

Fixes #15243

When removing a plain null versioned object (was created prior to bucket versioning
enabled), need to convert the bucket index representation to a versioned one. This
is needed so that all the versioning mechanics play together.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

radosgw-admin: add explanation for EACCESS on 'realm pull'

References #15200

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: 'period commit' supplies user-readable error messages

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: parse error messages from failed requests

Signed-off-by: Casey Bodley <cbodley@redhat.com>

packaging: rbdmap manpage

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #8273 from wjin/fix

osd: fix log info

Reviewed-by: Kefu Chai <kchai@redhat.com>

RPM: prefer UID/GID 167 when creating ceph user/group

This change affects openSUSE/SLE only.

http://tracker.ceph.com/issues/15246 Fixes: #15246

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #8260 from liewegas/wip-ec-test

test/test-erasure-code.sh: disable pg_temp priming

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #8276 from dachary/wip-virtualenv-jewel

python: avoid long paths (part 2)

Reviewed-by: Alfredo Deza <adeza@redhat.com>

python: avoid long paths (part 2)

make CEPH_BUILD_VIRTUALENV=/tmp TESTS=ceph-detect-init/run-tox.sh check

sets the CEPH_DETECT_INIT_VIRTUALENV environment variable

CEPH_BUILD_VIRTUALENV=/tmp make TESTS=ceph-detect-init/run-tox.sh check

does not because Makefile-env.am overrides it.

Do not set CEPH_BUILD_VIRTUALENV in Makefile-env.am so the variable from
the environment can be used. Require that CEPH_BUILD_VIRTUALENV has a
trailing slash so that the default is just an empty variable.

Signed-off-by: Loic Dachary <loic@dachary.org>

doc: rgw_region_root_pool option should be in [global]
     add new option rgw_zonegroup_root_pool from Jewel.

Fixes: #15244
Signed-off-by: Vikhyat Umrao <vumrao@redhat.com>

osd: fix log info

Signed-off-by: Wei Jin <wjin.cn@gmail.com>

Merge pull request #8250 from dachary/wip-virtualenv-jewel

python: avoid long paths

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #8267 from athanatos/wip-14309

osd: bail out of _committed_osd_maps if we are shutting down

Reviewed-by: Sage Weil <sage@redhat.com>

python: avoid long paths

As a temporary measure because it overflows when jenkins tries to build
in a deep path. Run make with:

make CEPH_BUILD_VIRTUALENV=/tmp

and it will use these directories instead of the default.

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #8266 from athanatos/wip-test-exit-value

Test exit values on test.sh, fix tier.cc

Reviewed-by: Sage Weil <sage@redhat.com>

test/librados/tier.cc: Fix Whiteout tests to force promotion

Otherwise, there may not actually be whiteouts created.

Fixes: 15177
Signed-off-by: Samuel Just <sjust@redhat.com>

qa/workunits/rados/test.sh: check subprocess return values

Fixes: #15165
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #8263 from tchaikov/wip-15031

msg/async: avoid log spam on throttle

Reviewed-by: Sage Weil <sage@redhat.com>

Merge branch 'wip-release-notes' of git://github.com/dachary/ceph into wip-jewel

Conflicts:
doc/release-notes.rst

test/test-erasure-code.sh: disable pg temp priming

If we prime pg_temp, there is a race where this test expects an out OSD
to not be in the acting set but it will be there initially.

Fixes: #15211
Signed-off-by: Sage Weil <sage@redhat.com>

msg/async: avoid log spam on throttle

increase the log level from 1 to 10, the log is printed when message
throttler kicks in.

Fixes: #15031
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8252 from xiexingguo/xxg-wip-15164

doc: fix wrong type of hyphen

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8239 from jcsp/wip-15223

mon: fix mixed-version MDSMonitor

Reviewed-by: Sage Weil <sage@redhat.com>

mds: enable standby_for_fscid without rank

...so that an MDS can specify that it wants
to act within a particular filesystem without
having to be bound to a particular rank.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: add mds_standby_for_fscid config

...and pass it through into MMDSBeacon.

Signed-off-by: John Spray <john.spray@redhat.com>

messages: add MMDSBeacon::standby_for_fscid

For the MDS daemon to tell the mon which filesystem
it wants to a be a standby for, when it is expressing
a standby_for_rank.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: s/standby_for_ns/standby_for_fscid/

Name dates from earlier version of multi-fs
where I was referring to filesystem IDs
as namespace IDs.

Signed-off-by: John Spray <john.spray@redhat.com>

vstart: support creating multiple cephfs filesystems

This is handy if you want to quickly create a cluster
with lots of filesystems, for playing with e.g. what
the ceph status output looks like when multiple filesystems
are in play.

This changes the default filesystem/pool names to be
e.g. cephfs_data_a instead of just `cephfs_data`, but anything
who relied on those names in a vstart environment houldn't have
been (this won't affect teuthology tests, of course, where we
still have a few annoying hardcoded pool names).

Signed-off-by: John Spray <john.spray@redhat.com>

mon: s/mdsmap/fsmap/ in "ceph status"

Signed-off-by: John Spray <john.spray@redhat.com>

mds: plain text prints for FSMap/Filesystem

These were just dumping JSON as a placeholder.

Not strictly necessary to have the extra non-machine-readable
variants of these outputs, but some users will probably appreciate
it, so why not.

Signed-off-by: John Spray <john.spray@redhat.com>

mon: config setting to skip FSMap::sanity

Only for use in extremis, where we have encountered
a bug that generates an invalid FSMap, but the way
to resolve it might be to proceed anyway and stay
up long enough to use commands to fix it.

Fixes: #15063
Signed-off-by: John Spray <john.spray@redhat.com>

mds: pass features through to MDSMap::encode

...from Filesystem.  This will prepare us for if
MDSMap gets any more feature-sensitive encoding
in the future.

Fixes: #15062
Signed-off-by: John Spray <john.spray@redhat.com>

mds: fix mds_info_t::dump

This broke with 4ad8f7254 ('features' gets
output as an item in the 'export_targets' list),
which wasn't the fsmap branch, but fix it while
I'm tidying.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: fix whitespace in is_cluster_available

This got mangled somehow.

Signed-off-by: John Spray <john.spray@redhat.com>

mds/FSMap: pass by ref in update_compat

Coverity: CID 1355578:  Performance inefficiencies  (PASS_BY_VALUE)

Signed-off-by: John Spray <john.spray@redhat.com>

tools: update naming s/handle_mds_map/handle_fs_map/

Signed-off-by: John Spray <john.spray@redhat.com>

mon: name cleanup s/mdsmap_bl/fsmap_bl/

Signed-off-by: John Spray <john.spray@redhat.com>

messages: support features in MFSMap

Not used for anything currently, but let's make
this class pass through the features to FSMap's
encode method for the benefit of the future
generations.

Signed-off-by: John Spray <john.spray@redhat.com>

release-notes: draft v10.1.0 release notes (manual edits)

Signed-off-by: Loic Dachary <loic@dachary.org>