unittest_compression_zlib: do not assume buffer will be null terminated

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #8063 from gregsfortytwo/wip-fix-fsmap-decode

Wip fix fsmap decode

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7752 from somnathr/wip-odsync-write

FileStore: Added O_DSYNC write scheme

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

FSMap: actually include the legacy MDSMap in Filesystems when upgrading

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

FSMap: decode old MDSMaps properly

We accidentally left a few members out that the MDSMap had!

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #8058 from yehudasa/wip-rgw-clean-log

cleanup

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

rgw: silence compilation warnings

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: cleanup of debug messages

use ldout instead of derr and lderr and increase logging level

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #7483 from liewegas/wip-osd-boot

osd: resolve boot vs NOUP set + clear race

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #8030 from yehudasa/wip-rgw-sync-status

adds the radosgw-admin sync status command that gives a human readable status of the sync process at a specific zone

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #7615 from ceph/wip-14678

rgw: store system object meta in cache when creating it

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #8037 from yehudasa/wip-bucket-link

rgw:bucket link now set the bucket.instance acl (bug fix)

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #8013 from yehudasa/wip-rgw-multisite-fixes

rgw multisite fixes

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #6953 from jcsp/wip-fsmap

Multi-filesystem support

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #7465 from liewegas/wip-promote-throttle

osd: probabilistic cache tier promotion throttling

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge tag 'v10.0.5'

v10.0.5

Conflicts:
debian/changelog

v10.0.5

Signed-off-by: Sage Weil <sage@redhat.com>

debian/changelog: Remove stray 'v' in version

Signed-off-by: Dan Mick <dan.mick@redhat.com>
(cherry picked from commit 0f7730c8f444657d4aaffa28c12bbcd602a2e3b8)

Merge branch 'master' of github.com:ceph/ceph

Merge branch 'debian-stretch-dep-fix' of git://github.com/rldleblanc/ceph

Reviewed-by: Sage Weil <sage@redhat.com>
# Conflicts:
# debian/control

Merge pull request #8045 from tchaikov/wip-scrubstore-cleanup

osd/ScrubStore: remove unused function

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7707 from xiexingguo/xxg-wip-14815

osd: filestore: fix wrong scope of result code for error cases during mount

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7726 from tchaikov/wip-fix-ftbfs-gcc6

test/time: no need to abs(uint64_t) for comparing

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #7955 from ErwanAliasr1/evelu-check-bufferlist

test/bufferlist: Avoid false-positive tests

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7980 from liewegas/wip-14952

mon: initialize last_* timestamps on new pgs to creation time

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #7982 from liewegas/wip-14606

qa/workunits/rest/test.py: add confirmation to 'mds setmap'

Merge pull request #7983 from liewegas/wip-14616

log: do not repeat errors to stderr

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8004 from liewegas/wip-rados-test-stderr

qa/workunits/rados/test.sh: capture stderr too

Merge pull request #8006 from liewegas/wip-big-pages

os/bluestore/KernelDevice: force block size

Merge pull request #8019 from kmroz/wip-upstream-civetweb-low-ports-6

rgw: drop permissions of rgw/civetweb after startup

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8007 from wido/se-mirror-mail

mirrors: Change contact e-mail address for se.ceph.com

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7991 from xiexingguo/xxg-wip-remove-upthrupending

osd: remove up_thru_pending field, which is never used

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8023 from roidayan/xio-remove_unused_variable

xio: remove unused variable

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7923 from RustShen/master

doc: add cinder backend section to rbd-openstack.rst

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

osd/ScrubStore: remove unused function

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7999 from liewegas/wip-armhf

a few armhf (32-bit build) fixes

Reviewed-by: Dan Mick <dmick@redhat.com>

rgw: store system object meta in cache when creating it

Fixes: #14678
When creating the system object we can easily also get the object's meta, so
keep it in the cache.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw:bucket link now set the bucket.instance acl

Fixes: #11076
Signed-off-by: Zengran Zhang <zhangzengran@h3c.com>

Merge pull request #7891 from rzarzynski/wip-rgw-content-disp

rgw: multiple Swift API compliance improvements for TempURL (Radoslaw Zarzynsk)

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #7990 from x11562/rgw_unused_vector

rgw: remove unused vector

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #8031 from ceph/wip-rgw-usage

rgw: admin api for retrieving usage info (Ji Chen)

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #8036 from yehudasa/wip-rgw-support-json-format-v2

rgw: support json format for admin policy API (Dunrong Huang)

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #7874 from badone/wip-coverity-rgw_lib-uninit

[rgw] RGWLib::env is not used so remove it

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw_admin: policy dump --xml backward compatibility

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

osd: make promote_probability_millis atomic_t

This is almost always read, so it should be cheap.

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: remove unused header file

Signed-off-by: Dunrong Huang <riegamaths@gmail.com>

rgw: support json format output for rados-admin policy command

when use rados-admin to get policy with parameter --format=json:
$ radosgw-admin policy --bucket=m8x --format=json

policy information with json format should be output.

Signed-off-by: Dunrong Huang <riegamaths@gmail.com>

rgw: support json format for admin policy API

Fixes: #14090, #14091
- before this patch:
$ curl -s -X GET http://radosrgw:9090/admin/bucket?format=json&policy&bucket=m8x | xmllint --format -

got XML response with error format

- after this patch:
got correct response with json format

Signed-off-by: Dunrong Huang <riegamaths@gmail.com>

Merge pull request #8027 from liewegas/wip-reweight-fix

test/pybind/test_ceph_argparse: fix reweight-by-utilization tests

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6656 from jmunhoz/wip-bucket-request-payment-usage-log

rgw: add bucket request payment feature usage statistics integration

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: also dump realm in radosgw-admin sync status command

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #8021 from branch-predictor/bp-bench-full-bw

common/obj_bencher.cc: bump the precision of bandwidth field

Reviewed-by: Sage Weil <sage@redhat.com>

FileStore: Added O_DSYNC write scheme

lfn_open can now open file handle in O_DSYNC mode based on the
filestore_odsync_write config option. This should help in getting
more stable and higher performance out in case of SSDs configured as
OSD data store.

Signed-off-by: Somnath Roy <somnath.roy@sandisk.com>

Merge pull request #7675 from vmcloud/wip-12917

rgw: adjust the request_uri to support absoluteURI of http request

test/pybind/test_ceph_argparse: fix reweight-by-utilization tests

Signed-off-by: Sage Weil <sage@redhat.com>

man/8/ceph.rst: remove invalid option for reweight-by-*

--yes-i-really-mean-it is not valid option for reweight-by-*

Signed-off-by: Kefu Chai <kchai@redhat.com>

log: do not repeat errors to stderr

If we get an error writing to the log, log it only once to stderr.
This avoids generating, say, 72 GB of ENOSPC errors in
teuthology.log when /var/log fills up.

Fixes: #14616
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #8025 from roidayan/xio-remove_duplicate_peer_addr_assignment

xio: remove duplicate assignment of peer addr

Reviewed-by: Kefu Chai <kchai@redhat.com>

xio: remove duplicate assignment of peer addr

Signed-off-by: Roi Dayan <roid@mellanox.com>

test: correct journal test failure caused by two merged commits

Commit 21d9595 changed the journal width to 3 objects while commit
a00eae0 changed the signature of the journal create helper.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

xio: remove unused variable

Signed-off-by: Roi Dayan <roid@mellanox.com>

mds: set FSMap encoding version to 6

It was 10, but that wasn't actually necessary,
6 is enough to get ahead of the MDSMap version.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: make FSMap.get_filesystem return const

To make it clearer that anyone modifying a filesystem
should be using modify_filesystem()

Note that FSMap is still a friend of MDSMonitor, as
MDSMap was, so a larger refactor to provide the appropriate
const accessors would also be nice in the future, but that's
nothign to do with multi-filesystems.

Signed-off-by: John Spray <john.spray@redhat.com>

test: update argparse test for rmfailed

rmfailed argument used to be an integer greater than
zero, now any string is acceptable (because we accept
fs:rank role strings here)

Signed-off-by: John Spray <john.spray@redhat.com>

client: update for multi-filesystems

Two main changes:
 * handle client_mds_namespace setting to subscribe
   to a particular MDSMap
 * subscribe to FSMap during `tell` in order to
   look up MDSs irrespective of what fscid they're in.

Signed-off-by: John Spray <john.spray@redhat.com>

tools/cephfs: update for FSMap

Everywhere we used to take a rank, we now
take a role (<fscid:rank>)

Signed-off-by: John Spray <john.spray@redhat.com>

mon: update for multiple filesystems

This is a big logic change throughout MDSMonitor,
and additonally some new commands to handle
per-fs variants of what used to be global things.

Signed-off-by: John Spray <john.spray@redhat.com>

common/obj_bencher.cc: bump the precision of bandwidth field

When calculated final average bandwidth exceeds 999MB, it starts
to get printed in scientific notation. Bump up the bandwidth
field precision so it'll remain in floating-point format for
a bit longer.

Signed-off-by: Piotr Dałek <piotr.dalek@ts.fujitsu.com>

messages: add MFSMap

For mon clients that subscribe to 'fsmap'.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: add FSMap

FSMap encapsulates a collection of MDSMaps, one
for each filesystem, and a collection of standby
MDS daemons that haven't been assigned to
a filesystem yet.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: introduce fs_cluster_id_t type

Unique identifier for a filesystem within
the ceph cluster.  Abbreviated to `fscid`,
deliberately chosen to not conflict
with legacy "fsid" identifiers.

Additionally add mds_role_t type which is
a more general form of rank: ranks are
now only unique within a particular fscid.
Roles are a fscid plus a rank.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: handle get_mds_info as const

Signed-off-by: John Spray <john.spray@redhat.com>

qa: update cephtool test for CephFS multi filesystems

Includes adding test for the enable_multiple flag.

Signed-off-by: John Spray <john.spray@redhat.com>

doc: cephfs admin command list

Creating this skeleton to include the new
"ceph fs ..." commands.  Needs filling out
with prose about purpose and result of
commands.

Signed-off-by: John Spray <john.spray@redhat.com>

messages: fix strings for MMonGetVersionReply

It previously described itself as "mon_check_map_ack"

Signed-off-by: John Spray <john.spray@redhat.com>

java: update libcephfs tests to learn pool name

Don't assume the data/metadata pools are called
"data" and "metadata".

Signed-off-by: John Spray <john.spray@redhat.com>

java: add O_DIRECTORY to libcephfs interface

Signed-off-by: John Spray <john.spray@redhat.com>

include/CompatSet: trivial const-ness fix

Signed-off-by: John Spray <john.spray@redhat.com>

vstart: don't set max_mds to use all daemons

This is only useful if you want to test a
single namespace with no standbys.  It's easier
to initially have max_mds=1 and increase it
later than it is to go through the ritual
of decreasing max_mds and then deactivating
a daemon and waiting for it to stop.

Signed-off-by: John Spray <john.spray@redhat.com>

vstart: don't call "fs new" repeatedly

It was doing it once for each MDS, which is safe
(because it's idempotent) but silly.

Signed-off-by: John Spray <john.spray@redhat.com>

mon: initialize last_* timestamps on new pgs to creation time

Currently, when you create a pool, until the PGs peer you generate a health
error like

            8 pgs are stuck inactive for more than 300 seconds

which is inaccurate and misleading.  Instead, set the timestamps to the
creation time so that warnings don't appear until it's clear they're stuck.

Fixes: #14952
Signed-off-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'origin/jewel'

Conflicts:
src/client/Client.h
Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #7495 from ukernel/jewel-10436

client: flush kernel pagecache before creating snapshot

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

rgw: instruct civetweb to drop permissions

Get uid set in g_ceph_context and pass to civetweb where setuid()/setgid()
will be invoked. This Allows for starting civetweb under privileged port
numbers.

Fixes: #13600
Signed-off-by: Karol Mroz <kmroz@suse.com>

rgw_main: break up frontend config parsing

Before invoking global_init(), determine which frontends are configured.
If Civetweb is to be used, ensure it will be responsible for dropping
permissions after radosgw startup.

Signed-off-by: Karol Mroz <kmroz@suse.com>

global_init: add run_pre_init argument to function signature

Allows running the global_pre_init() separately in order to parse config
options prior to invoking global_init().

Signed-off-by: Karol Mroz <kmroz@suse.com>

global: record target user and group strings

Useful so we don't have to do a reverse lookup when dropping permissions
in civetweb.

Signed-off-by: Karol Mroz <kmroz@suse.com>

Merge pull request #7798 from ukernel/wip-mds-snap

A few more snapshot fixes, mostly around snapshotted inode/dentry tracking

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #7994 from ukernel/jewel-14996

Merge: Fix client handling of "lost" open directories on shutdown

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Modifications to rbd-openstack.rst

Original configuration is not working

'enabled_backends' should be changed accordingly

Signed-off-by: RustShen <rustinpeace@163.com>
Update rbd-openstack.rst

Merge remote-tracking branch 'origin/greg-fs-testing-3-7-damage'

Merge pull request #7686 from david-z/wip-fix-fh-ref-leak

ceph-fuse: Fix potential filehandle ref leak at umount

Reviewed-by: Yan, Zheng <zyan@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #7685 from david-z/wip-fix-put-free-fd-back

[cephfs] fix free fds being exhausted eventually because freed fds are never put back

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #7003 from jcsp/wip-cephfs-tmap-migrate

tools/cephfs: add tmap_upgrade

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Yan, Zheng <zyan@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #8014 from ukernel/wip-15038

mds: fix inode_t::compare()

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #8012 from zmc/jewel

man/8/ceph-disk: fix formatting issue

Reviewed-by: Dan Mick <dmick@redhat.com>

mds: fix inode_t::compare()

Fixes: #15038
Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #7978 from dillaman/wip-14908

librbd: refresh image if required before replaying journal ops

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #8009 from dillaman/wip-14988

test: reproducer for writeback CoW deadlock

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #8005 from dillaman/wip-rbd-lock-fence

pybind: flag an RBD image as closed regardless of result code

Reviewed-by: Josh Durgin <jdurgin@redhat.com>