tools/cephfs: fix tmap_upgrade

Move it up to handle the command before we try
and initialize DataScan::driver, because drivers
require a target filesystem, and that doesn't
apply to this command.

Fixes: #15135
Signed-off-by: John Spray <john.spray@redhat.com>

Merge remote-tracking branch 'gh/jewel'

Merge branch 'wip-15086' of git://github.com/SUSE/ceph

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8068 from SUSE/wip-15104

qa/workunits/rbd: use POSIX function definition

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7663 from javacruft/wip-pybind-flags

pybind: Ensure correct python flags are passed

Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7856 from jdurgin/wip-cython-cleanup

configure.ac: update help strings for cython

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge remote-tracking branch 'origin/wip-josh-testing'

Merge pull request #8067 from athanatos/wip-fix-filestore-warning

FileStore: fix initialization order for m_disable_wbthrottle

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

qa/workunits/rbd: use POSIX function definition

In scripts with /bin/sh shebang, we must use POSIX function definitions to
avoid syntax errors on platforms that link /bin/sh to something other than
/bin/bash.

See https://wiki.ubuntu.com/DashAsBinSh#function for more information.

http://tracker.ceph.com/issues/15104 Fixes: #15104

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #8034 from SUSE/wip-15047

packaging: make infernalis -> jewel upgrade work

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

debian: make infernalis -> jewel upgrade work

Attempts to install jewel ceph-common, ceph-mon, ceph-osd, and ceph-base
package over infernalis ceph package fail due to files existing in both.

See comment #4 in the tracker issue for a deeper analysis.

http://tracker.ceph.com/issues/15047 Fixes: #15047

Signed-off-by: Nathan Cutler <ncutler@suse.com>

FileStore: fix initialization order for m_disable_wbthrottle

Signed-off-by: Samuel Just <sjust@redhat.com>

ceph-detect-init: return systemd on Debian Jessie

http://tracker.ceph.com/issues/15086 Fixes: #15086

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Revert "test/time: no need to abs(uint64_t) for comparing"

This reverts commit da0f6608dd6cc3356e727e739575f496c2b6a590.

unittest_time fails:

[==========] Running 4 tests from 2 test cases.
[----------] Global test environment set-up.
[----------] 2 tests from RealClock
[ RUN      ] RealClock.Sanity
[       OK ] RealClock.Sanity (0 ms)
[ RUN      ] RealClock.Conversions
test/common/test_time.cc:125: Failure
Expected: ((Clock::from_double(bd) - brt).count()) < (30U), actual: 18446744073709551595 vs 30
[  FAILED  ] RealClock.Conversions (0 ms)
[----------] 2 tests from RealClock (0 ms total)

[----------] 2 tests from CoarseRealClock
[ RUN      ] CoarseRealClock.Sanity
[       OK ] CoarseRealClock.Sanity (0 ms)
[ RUN      ] CoarseRealClock.Conversions
test/common/test_time.cc:125: Failure
Expected: ((Clock::from_double(bd) - brt).count()) < (30U), actual: 18446744073709551595 vs 30
[  FAILED  ] CoarseRealClock.Conversions (0 ms)
[----------] 2 tests from CoarseRealClock (0 ms total)

[----------] Global test environment tear-down
[==========] 4 tests from 2 test cases ran. (0 ms total)
[  PASSED  ] 2 tests.
[  FAILED  ] 2 tests, listed below:
[  FAILED  ] RealClock.Conversions
[  FAILED  ] CoarseRealClock.Conversions

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #8063 from gregsfortytwo/wip-fix-fsmap-decode

Wip fix fsmap decode

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7752 from somnathr/wip-odsync-write

FileStore: Added O_DSYNC write scheme

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

FSMap: actually include the legacy MDSMap in Filesystems when upgrading

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

FSMap: decode old MDSMaps properly

We accidentally left a few members out that the MDSMap had!

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #8058 from yehudasa/wip-rgw-clean-log

cleanup

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

rgw: silence compilation warnings

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: cleanup of debug messages

use ldout instead of derr and lderr and increase logging level

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #7944 from trociny/wip-rbd-mirror-image-replayer-async

rbd-mirror: ImageReplayer async start/stop

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #8028 from dillaman/wip-librbd-journal-replay

librbd: replaying a journal op post-refresh requires locking

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7483 from liewegas/wip-osd-boot

osd: resolve boot vs NOUP set + clear race

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #8030 from yehudasa/wip-rgw-sync-status

adds the radosgw-admin sync status command that gives a human readable status of the sync process at a specific zone

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #7615 from ceph/wip-14678

rgw: store system object meta in cache when creating it

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #8037 from yehudasa/wip-bucket-link

rgw:bucket link now set the bucket.instance acl (bug fix)

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #8013 from yehudasa/wip-rgw-multisite-fixes

rgw multisite fixes

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #6953 from jcsp/wip-fsmap

Multi-filesystem support

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #7465 from liewegas/wip-promote-throttle

osd: probabilistic cache tier promotion throttling

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge tag 'v10.0.5'

v10.0.5

Conflicts:
debian/changelog

v10.0.5

Signed-off-by: Sage Weil <sage@redhat.com>

debian/changelog: Remove stray 'v' in version

Signed-off-by: Dan Mick <dan.mick@redhat.com>
(cherry picked from commit 0f7730c8f444657d4aaffa28c12bbcd602a2e3b8)

Merge branch 'master' of github.com:ceph/ceph

Merge branch 'debian-stretch-dep-fix' of git://github.com/rldleblanc/ceph

Reviewed-by: Sage Weil <sage@redhat.com>
# Conflicts:
# debian/control

Merge pull request #8045 from tchaikov/wip-scrubstore-cleanup

osd/ScrubStore: remove unused function

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7707 from xiexingguo/xxg-wip-14815

osd: filestore: fix wrong scope of result code for error cases during mount

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7726 from tchaikov/wip-fix-ftbfs-gcc6

test/time: no need to abs(uint64_t) for comparing

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #7955 from ErwanAliasr1/evelu-check-bufferlist

test/bufferlist: Avoid false-positive tests

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7980 from liewegas/wip-14952

mon: initialize last_* timestamps on new pgs to creation time

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #7982 from liewegas/wip-14606

qa/workunits/rest/test.py: add confirmation to 'mds setmap'

Merge pull request #7983 from liewegas/wip-14616

log: do not repeat errors to stderr

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8004 from liewegas/wip-rados-test-stderr

qa/workunits/rados/test.sh: capture stderr too

Merge pull request #8006 from liewegas/wip-big-pages

os/bluestore/KernelDevice: force block size

Merge pull request #8019 from kmroz/wip-upstream-civetweb-low-ports-6

rgw: drop permissions of rgw/civetweb after startup

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8007 from wido/se-mirror-mail

mirrors: Change contact e-mail address for se.ceph.com

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7991 from xiexingguo/xxg-wip-remove-upthrupending

osd: remove up_thru_pending field, which is never used

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8023 from roidayan/xio-remove_unused_variable

xio: remove unused variable

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

test: add rbd-mirror test script

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

test: rbd_mirror_image_replay improve commit position parsing

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

test: ImageReplayer on_stop/on_start error tests

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

rbd-mirror: ImageReplayer async start/stop

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: protect against duplicate journal events

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: protect against journal replay shutdown race condition

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: replaying a journal op post-refresh requires locking

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test: periodic failure in TestJournalReplay.SnapRename

Force the refresh of the image to pick up the new snapshot
name before testing that it successfully was renamed.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7923 from RustShen/master

doc: add cinder backend section to rbd-openstack.rst

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

osd/ScrubStore: remove unused function

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7999 from liewegas/wip-armhf

a few armhf (32-bit build) fixes

Reviewed-by: Dan Mick <dmick@redhat.com>

rgw: store system object meta in cache when creating it

Fixes: #14678
When creating the system object we can easily also get the object's meta, so
keep it in the cache.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw:bucket link now set the bucket.instance acl

Fixes: #11076
Signed-off-by: Zengran Zhang <zhangzengran@h3c.com>

Merge pull request #7891 from rzarzynski/wip-rgw-content-disp

rgw: multiple Swift API compliance improvements for TempURL (Radoslaw Zarzynsk)

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #7990 from x11562/rgw_unused_vector

rgw: remove unused vector

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #8031 from ceph/wip-rgw-usage

rgw: admin api for retrieving usage info (Ji Chen)

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #8036 from yehudasa/wip-rgw-support-json-format-v2

rgw: support json format for admin policy API (Dunrong Huang)

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #7874 from badone/wip-coverity-rgw_lib-uninit

[rgw] RGWLib::env is not used so remove it

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw_admin: policy dump --xml backward compatibility

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

osd: make promote_probability_millis atomic_t

This is almost always read, so it should be cheap.

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: remove unused header file

Signed-off-by: Dunrong Huang <riegamaths@gmail.com>

rgw: support json format output for rados-admin policy command

when use rados-admin to get policy with parameter --format=json:
$ radosgw-admin policy --bucket=m8x --format=json

policy information with json format should be output.

Signed-off-by: Dunrong Huang <riegamaths@gmail.com>

rgw: support json format for admin policy API

Fixes: #14090, #14091
- before this patch:
$ curl -s -X GET http://radosrgw:9090/admin/bucket?format=json&policy&bucket=m8x | xmllint --format -

got XML response with error format

- after this patch:
got correct response with json format

Signed-off-by: Dunrong Huang <riegamaths@gmail.com>

Merge pull request #8027 from liewegas/wip-reweight-fix

test/pybind/test_ceph_argparse: fix reweight-by-utilization tests

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6656 from jmunhoz/wip-bucket-request-payment-usage-log

rgw: add bucket request payment feature usage statistics integration

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: also dump realm in radosgw-admin sync status command

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #8021 from branch-predictor/bp-bench-full-bw

common/obj_bencher.cc: bump the precision of bandwidth field

Reviewed-by: Sage Weil <sage@redhat.com>

FileStore: Added O_DSYNC write scheme

lfn_open can now open file handle in O_DSYNC mode based on the
filestore_odsync_write config option. This should help in getting
more stable and higher performance out in case of SSDs configured as
OSD data store.

Signed-off-by: Somnath Roy <somnath.roy@sandisk.com>

Merge pull request #7675 from vmcloud/wip-12917

rgw: adjust the request_uri to support absoluteURI of http request

test/pybind/test_ceph_argparse: fix reweight-by-utilization tests

Signed-off-by: Sage Weil <sage@redhat.com>

man/8/ceph.rst: remove invalid option for reweight-by-*

--yes-i-really-mean-it is not valid option for reweight-by-*

Signed-off-by: Kefu Chai <kchai@redhat.com>

log: do not repeat errors to stderr

If we get an error writing to the log, log it only once to stderr.
This avoids generating, say, 72 GB of ENOSPC errors in
teuthology.log when /var/log fills up.

Fixes: #14616
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #8025 from roidayan/xio-remove_duplicate_peer_addr_assignment

xio: remove duplicate assignment of peer addr

Reviewed-by: Kefu Chai <kchai@redhat.com>

xio: remove duplicate assignment of peer addr

Signed-off-by: Roi Dayan <roid@mellanox.com>

test: correct journal test failure caused by two merged commits

Commit 21d9595 changed the journal width to 3 objects while commit
a00eae0 changed the signature of the journal create helper.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

xio: remove unused variable

Signed-off-by: Roi Dayan <roid@mellanox.com>

mds: set FSMap encoding version to 6

It was 10, but that wasn't actually necessary,
6 is enough to get ahead of the MDSMap version.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: make FSMap.get_filesystem return const

To make it clearer that anyone modifying a filesystem
should be using modify_filesystem()

Note that FSMap is still a friend of MDSMonitor, as
MDSMap was, so a larger refactor to provide the appropriate
const accessors would also be nice in the future, but that's
nothign to do with multi-filesystems.

Signed-off-by: John Spray <john.spray@redhat.com>

test: update argparse test for rmfailed

rmfailed argument used to be an integer greater than
zero, now any string is acceptable (because we accept
fs:rank role strings here)

Signed-off-by: John Spray <john.spray@redhat.com>

client: update for multi-filesystems

Two main changes:
 * handle client_mds_namespace setting to subscribe
   to a particular MDSMap
 * subscribe to FSMap during `tell` in order to
   look up MDSs irrespective of what fscid they're in.

Signed-off-by: John Spray <john.spray@redhat.com>

tools/cephfs: update for FSMap

Everywhere we used to take a rank, we now
take a role (<fscid:rank>)

Signed-off-by: John Spray <john.spray@redhat.com>

mon: update for multiple filesystems

This is a big logic change throughout MDSMonitor,
and additonally some new commands to handle
per-fs variants of what used to be global things.

Signed-off-by: John Spray <john.spray@redhat.com>

common/obj_bencher.cc: bump the precision of bandwidth field

When calculated final average bandwidth exceeds 999MB, it starts
to get printed in scientific notation. Bump up the bandwidth
field precision so it'll remain in floating-point format for
a bit longer.

Signed-off-by: Piotr Dałek <piotr.dalek@ts.fujitsu.com>

messages: add MFSMap

For mon clients that subscribe to 'fsmap'.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: add FSMap

FSMap encapsulates a collection of MDSMaps, one
for each filesystem, and a collection of standby
MDS daemons that haven't been assigned to
a filesystem yet.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: introduce fs_cluster_id_t type

Unique identifier for a filesystem within
the ceph cluster.  Abbreviated to `fscid`,
deliberately chosen to not conflict
with legacy "fsid" identifiers.

Additionally add mds_role_t type which is
a more general form of rank: ranks are
now only unique within a particular fscid.
Roles are a fscid plus a rank.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: handle get_mds_info as const

Signed-off-by: John Spray <john.spray@redhat.com>

qa: update cephtool test for CephFS multi filesystems

Includes adding test for the enable_multiple flag.

Signed-off-by: John Spray <john.spray@redhat.com>

doc: cephfs admin command list

Creating this skeleton to include the new
"ceph fs ..." commands.  Needs filling out
with prose about purpose and result of
commands.

Signed-off-by: John Spray <john.spray@redhat.com>

messages: fix strings for MMonGetVersionReply

It previously described itself as "mon_check_map_ack"

Signed-off-by: John Spray <john.spray@redhat.com>

java: update libcephfs tests to learn pool name

Don't assume the data/metadata pools are called
"data" and "metadata".

Signed-off-by: John Spray <john.spray@redhat.com>

java: add O_DIRECTORY to libcephfs interface

Signed-off-by: John Spray <john.spray@redhat.com>