OSD: allow client throttler to be adjusted on-fly, without restart

This patch allows the osd_client_message_cap and
osd_client_message_size_cap to be adjusted on-fly, using admin socket
functionality.

Fixes: http://tracker.ceph.com/issues/18791
Signed-off-by: Piotr Dałek <piotr.dalek@corp.ovh.com>

Merge pull request #13066 from liewegas/wip-aio-read

os/bluestore: use aio for reads

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #13071 from liewegas/wip-omap-getkeys-rados-test

qa/suites/rados/basic: set low omap limit for rgw workload

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #13150 from tchaikov/wip-mon-cleanup

mon: post-jewel cleanups

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12966 from dillaman/wip-18436

osdc: cache should ignore error bhs during trim

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

Merge pull request #12800 from jcsp/wip-vstart-qasuite

Improve vstart_runner to (optionally) create its own cluster

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #12817 from liupan1111/wip-list-snapshot-time

librbd: support to list snapshot timestamp

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13098 from wido/doc-osd-greace

doc: Document that osd_heartbeat_grace applies to MON and OSD

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #13141 from dillaman/wip-18653

librbd: introduce new constants for tracking max block name prefix

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

librbd: introduce new constants for tracking max block name prefix

Fixes: http://tracker.ceph.com/issues/18653
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13188 from liewegas/wip-rest-fs-name

qa/workunits/rest: use unique pool names for cephfs test

Reviewed-by: Kefu Chai <kchai@redhat.com>

test: regolden formatted-output.t for timestamp metadata of rbd snapshot

Signed-off-by: Pan Liu <pan.liu@istuary.com>

test: Add unit test for getting rbd snapshot timestamp info

Signed-off-by: Pan Liu <pan.liu@istuary.com>

librbd: support to list snapshot time stamp

Fixes: http://tracker.ceph.com/issues/808
Signed-off-by: Pan Liu <pan.liu@istuary.com>

librbd: Add timestamp metadata when creating rbd snapshot

Signed-off-by: Pan Liu <pan.liu@istuary.com>

Merge pull request #12646 from SirishaGuduru/swift-api-auth-doc-change

doc: Fixes radosgw-admin ex: in swift auth section

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

qa/workunits/rest: use unique pool names for cephfs test

Using cephfs_[meta]data collides with the pools that teuthology
already creates if an mds is defined.

This became a (noticeable) problem with 052c3d3f681017d56b5b0ee5cf6f65bffc952a4c

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #13054 from oritwas/wip-rgw-period-update-crash

rgw: fix period update crash

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #13186 from Adirl/fix_rdma_compile

cmake: fix broken RDMA compilation after merge PR #12878

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13003 from Jing-Scott/dev/swift-container-acl-info

rgw: swift: the http referer acl in swift API should be shown

Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #13005 from Jing-Scott/dev/swift-http-url-parse

rgw: swift: The http referer should be parsed to compare in swift API

Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #12874 from Jing-Scott/dev/swift-ref

rgw: the swift container acl should support field .ref

Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

cmake: fix broken RDMA compilation after merge PR #12878

issue: 965984

Change-Id: I2e401acf204069d3907001d276c58d9fb4cd0d58
Signed-off-by: Oren Duer <oren@mellanox.com>

doc: Document that osd_heartbeat_grace applies to MON and OSD

This value has to be set in both the [mon] and [osd] section since
both daemons read this value.

Signed-off-by: Wido den Hollander <wido@42on.com>

Merge pull request #13142 from Adirl/remove_compilation_warning

msg/async/rdma: Remove compilation warning

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #13057 from wido/issue-18635

systemd: Restart Mon after 10s in case of failure

Reviewd-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #13140 from liewegas/wip-bluestore-statfs

os/bluestore: fix statfs to not include DB partition in free space

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #13135 from liewegas/wip-kerneldevice-debug

os/bluestore/KernelDevice: fix debug message

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #13124 from ukernel/wip-18675

client: remove request from session->requests when handling forward

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #13095 from ukernel/wip-18660

mds: skip fragment space check for replayed request

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #13052 from ukernel/wip-16768

mds: don't modify inode that is not projected

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #12973 from ukernel/wip-18578

mds: fix incorrect assertion in Server::_dir_is_nonempty()

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #12825 from stiopaa1/r11124_refuseToDoFsNew

mon/MDSMonitor.cc:refuse fs new on pools with obj

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #12489 from gregsfortytwo/wip-fix-rename-bounds

client: fix the cross-quota rename boundary check conditions

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #12440 from runsisi/wip-fix-fp-exception

include/fs_types: fix unsigned integer overflow

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: only use aio for read if there are >1 blobs

If we have a single blob to read it is not worth the context switch.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlockDevice: allow sync read to accumulate on bl

Do not clear the target bl; instead, append to it.  This makes our
behavior consistent with aio_read, which does the same.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: do read io via aio in parallel

Dispatch all blob reads in parallel via aio.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/KernelDevice: avoid possible use-after-free for ioc->priv

If aio_wake() triggers destruction, ioc->priv might be a
use-after-free (this is the case for the BlueStore read path).

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/NVMEDevice: implement aio_read

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlockDevice: add aio_read API

NVMEDevice not implemented yet.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #13088 from jcsp/wip-16397-unpin

qa: unpin knfs from ubuntu

Reviewed-by: Jeff Layton <jlayton@redhat.com>

mon: assert if pgmap's format_version < 1

we bump the pgmap's format_version to 1 and the monitor's protocol
version to 12 in 2d2aa00. and it is required to upgrade to jewel before
upgrading to kraken or luminous. so we can safely assume that the
PGMonitor's format_version is always greater or equal to 1.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon: assert on CEPH_FEATURE_MON_STATEFUL_SUB

this feature was introduced in jewel. and we are targeting luminous in
master. so it's safe to assume that all OSDs in the cluster have this
feature.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13148 from tchaikov/wip-18692

packaging: install libceph-common.so* not libceph-common.so.*

Reviewed-by: Dan Mick <dmick@redhat.com>

packaging: install libceph-common.so* not libceph-common.so.*

Fixes: http://tracker.ceph.com/issues/18692
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13087 from athanatos/wip-17831-18583

osd/PrimaryLogPG::try_lock_for_read: give up if missing

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #12934 from athanatos/wip-18529

test/pybind/test_rados.py: tolerate TimedOut in test_ping_monitor

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #13110 from dillaman/wip-18666

librbd: prevent self-blacklisting during break lock

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

radosgw/swift: clean up flush / newline behavior.

The current code emits a newline after swift errors, but fails
to account for it when it calculates 'content-length'.  This results in
some clients (go github.com/ncw/swift) producing complaints about the
unsolicited newline such as this,
Unsolicited response received on idle HTTP channel starting with "\n"; err=<nil>

This logic eliminates the newline on flush.  This makes the content length
calculation correct and eliminates the stray newline.

There was already existing separator logic in the rgw plain formatter
that can emit a newline at the correct point.  It had been checking
"len" to decide if previous data had been emitted, but that's reset to 0
by flush().  So, this logic adds a new per-instance variable to separately
track state that it emitted a previous item (and should emit a newline).

Fixes: http://tracker.ceph.com/issues/18473
Signed-off-by: Marcus Watts <mwatts@redhat.com>
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

test/pybind/test_rados.py: tolerate empty output from mon ping

Fixes: http://tracker.ceph.com/issues/18529
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #13007 from Werkov/rgw-fix-tempurl-url-encoding-master

rgw: Use decoded URI when verifying TempURL

Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

os/bluestore: fix statfs to not include DB partition in free space

If we report the DB space as vailable, ceph thinks the OSD can store more
data and will not mark the cluster as full as easily.  And in reality, we
can't actually store data in this space--only metadata.  Avoid the problem
by not reporting it as available.

Fixes: http://tracker.ceph.com/issues/18599
Signed-off-by: Sage Weil <sage@redhat.com>

sync/rdma: Remove warning

Fixed warning - of sending unsigned short int to %x

issue: 944710

Change-Id: I2820e8da77467a4a645d9653b4964e7a2198656b
Signed-off-by: Sarit Zubakov <saritz@mellanox.com>

Merge pull request #12444 from oritwas/wip-rgw-fix-secondary_versioning

rgw: complete versioning enablement after sending it to meta master

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: add check for update return value

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: fix init_zg_from_period when default zone is not set as default

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: we need to reinit the zonegroup after assignment to avoid invalid cct and store

Fixes: http://tracker.ceph.com/issues/18631
Signed-off-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #13074 from liewegas/wip-bluestore-crc-error

os/bluestore: include logical object offset in crc error

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore/KernelDevice: fix debug message

Not even sure why this compiles :/.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12878 from tchaikov/wip-cmake

cmake: link ceph-{mgr,mon,mds,osd} against libcommon statically

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #13122 from orendu/fix-rdma-compilation

cmake: Fix broken async/rdma compilation since move to libceph-common

Reviewed-by: Adir Lev <adirl@mellanox.com>
Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Amir Vadai <amir@vadai.me>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13101 from Adirl/fix_mem_leak

msg/async/rdma: Fix memory leak of OSD

Reviewed-by: Haomai Wang <haomai@xsky.com>

cmake: Fix broken async/rdma compilation since move to libceph-common

Was broken since merge of pull request #12840

Signed-off-by: Oren Duer <oren@mellanox.com>

client: remove request from session->requests when handling forward

Client::handle_client_request_forward() reset request->mds to -1,
it should also remove request from session->requests. Otherwise
Client::kick_requests_closed() get confused.

Fixes: http://tracker.ceph.com/issues/18675
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #13075 from tchaikov/wip-zstd

compressor/zstd: add zstd compression plugin

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: swift: The http referer should be parsed to compare in swift API
The http referer should be parsed to compare with the url set on the container read acl. If we set .r:www.example.com on container read acl, we should parse the hostname 'www.example.com' of the http referer like 'http://www.example.com' from the http request.

Fixes: http://tracker.ceph.com/issues/18685
Signed-off-by: Jing Wenjun <jingwenjun@cmss.chinamobile.com>

librbd: prevent self-blacklisting during break lock

Fixes: http://tracker.ceph.com/issues/18666
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13086 from dillaman/wip-librbd-lock-init-race

librbd: race initializing exclusive lock and configuring IO path

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

compressor/zstd: add zstd compressor

Build/link of zstd itself is maybe not ideal, but it works fine.

Signed-off-by: Sage Weil <sage@redhat.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

add src/zstd submodule

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async/rdma: Fix memory leak of OSD

We can delete qp only in RDMADispatcher::handle_async_event() which call
"erase_qpn" to enable deletion.

issue: 959004

Change-Id: Iab69cb365b37a09e9608d4b3c595e05278bbe021
Signed-off-by: Sarit Zubakov <saritz@mellanox.com>

Merge pull request #13090 from dzafman/wip-18624

Revert "PrimaryLogPG::failed_push: update missing as well"

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13096 from amirv/new_connect_arg

msg/RDMA: Fix broken compilation due to new argument in net.connect()

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #9348 from LiumxNL/fix-mksnap

mon/OSDMonitor: make snaps on tier pool should not be allowed

Merge pull request #12937 from chendave/auth_method

auth: Enhancement for the supported auth methods

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12992 from tchaikov/wip-demangle-on-freebsd

common/BackTrace: demangle on FreeBSD also

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #12620 from varadakari/wip-rocksdb-histogram

os/bluestore: kvdb histogram

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #13031 from dzafman/wip-calc-stats

osd: Calculate degraded and misplaced more accurately

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13032 from liewegas/wip-bluefs-append

os/bluestore/BlueFS: tune flushing of writes

Tested-by: Chagam, Anjaneya <anjaneya.chagam@intel.com>

Merge pull request #13072 from wido/issue-18638

osd: Return correct osd_objectstore in OSD metadata

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #13018 from songbaisen/s11

global: we need to handle the init_on_startup return value when global_init.

Reviewed-by: Kefu Chai <kchai@redhat.com>

msg/RDMA: Fix broken compilation due to new argument in net.connect()

Fixes: 6e4ed291afc3 ("msg: add ms_bind_before_connect to bind before connect")
Change-Id: Ia45f215b5d59dfc8545017518e5162404059829e
Signed-off-by: Amir Vadai <amir@vadai.me>

mds: don't purge strays when mds is in clientreplay state

MDS does not trim log when it's in clientreplay state. If mds hang
at clientreplay state (due to bug), purging strays can submit lots
of log events and create very large mds log.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: skip fragment space check for replayed request

when handling replayed request, stray directory can be different
from the stray directory used by the original request. The fragment
space check for stray directory can fail.

Fixes: http://tracker.ceph.com/issues/18660
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Revert "PrimaryLogPG::failed_push: update missing as well"

This reverts commit dd48b972afde2dfa9ab1a6942c7961750222986d.

Fixes: http://tracker.ceph.com/issues/18624
Signed-off-by: David Zafman <dzafman@redhat.com>

rgw: swift: the http referer acl in swift API should be shown
The container acl about http referer set should be shown in container metadata dump.

Fixes: http://tracker.ceph.com/issues/18665
Signed-off-by: Jing Wenjun <jingwenjun@cmss.chinamobile.com>

qa: unpin knfs from ubuntu

We have an updated nfs-utils that is no longer
generating spurious selinux warnings on CentOS.

Fixes: http://tracker.ceph.com/issues/16397
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #12445 from pritha-srivastava/wip_glob_func

rgw: Added a globbing method for AWS Policies.

Merge pull request #13084 from linuxbox2/wip-librgw-parentref

rgw_file:  interned RGWFileHandle objects need parent refs

Merge pull request #13083 from dachary/wip-15653-crush-weights

crush: verify weights is influenced by the number of replicas

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13076 from dillaman/wip-18617

test: use librados API to retrieve config params

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #12866 from dzafman/wip-18471

qa/workunits/ceph-helpers: wait_for_clean() races with pg creation

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw_file:  interned RGWFileHandle objects need parent refs

RGW NFS fhcache/RGWFileHandle operators assume existence of the
full chain of parents from any object to the its fs_root--this is
a consequence of the weakly-connected namespace design goal, and
not a defect.

This change ensures the invariant by taking a parent ref when
objects are interned (when a parent ref is guaranteed).  Parent
refs are returned when objects are destroyed--essentially by the
invariant, such a ref must exist.

The extra ref is omitted when parent->is_root(), as that node is
not in the LRU cache.

Fixes: http://tracker.ceph.com/issues/18650
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #13038 from linuxbox2/wip-librgw-invalidate

rgw_file: add timed namespace invalidation

librbd: race initializing exclusive lock and configuring IO path

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12916 from Liuchang0812/xmlformatter-fix

common/xmlformatter: turn on underscored and add unittest

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

librbd: improve debug logging for lock / watch state machines

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test: use librados API to retrieve config params

The CephContext object is not ABI-stable, so it is necessary to
use the "conf_get" librados methods to safely retrieve a setting.

Fixes: http://tracker.ceph.com/issues/18617
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

crush: verify weights is influenced by the number of replicas

Refs: http://tracker.ceph.com/issues/15653

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Signed-off-by: Loic Dachary <loic@dachary.org>