Merge pull request #12440 from runsisi/wip-fix-fp-exception

include/fs_types: fix unsigned integer overflow

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13088 from jcsp/wip-16397-unpin

qa: unpin knfs from ubuntu

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #13148 from tchaikov/wip-18692

packaging: install libceph-common.so* not libceph-common.so.*

Reviewed-by: Dan Mick <dmick@redhat.com>

packaging: install libceph-common.so* not libceph-common.so.*

Fixes: http://tracker.ceph.com/issues/18692
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13087 from athanatos/wip-17831-18583

osd/PrimaryLogPG::try_lock_for_read: give up if missing

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #12934 from athanatos/wip-18529

test/pybind/test_rados.py: tolerate TimedOut in test_ping_monitor

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #13110 from dillaman/wip-18666

librbd: prevent self-blacklisting during break lock

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

radosgw/swift: clean up flush / newline behavior.

The current code emits a newline after swift errors, but fails
to account for it when it calculates 'content-length'.  This results in
some clients (go github.com/ncw/swift) producing complaints about the
unsolicited newline such as this,
Unsolicited response received on idle HTTP channel starting with "\n"; err=<nil>

This logic eliminates the newline on flush.  This makes the content length
calculation correct and eliminates the stray newline.

There was already existing separator logic in the rgw plain formatter
that can emit a newline at the correct point.  It had been checking
"len" to decide if previous data had been emitted, but that's reset to 0
by flush().  So, this logic adds a new per-instance variable to separately
track state that it emitted a previous item (and should emit a newline).

Fixes: http://tracker.ceph.com/issues/18473
Signed-off-by: Marcus Watts <mwatts@redhat.com>
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

test/pybind/test_rados.py: tolerate empty output from mon ping

Fixes: http://tracker.ceph.com/issues/18529
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #13007 from Werkov/rgw-fix-tempurl-url-encoding-master

rgw: Use decoded URI when verifying TempURL

Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #12444 from oritwas/wip-rgw-fix-secondary_versioning

rgw: complete versioning enablement after sending it to meta master

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #13074 from liewegas/wip-bluestore-crc-error

os/bluestore: include logical object offset in crc error

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #12878 from tchaikov/wip-cmake

cmake: link ceph-{mgr,mon,mds,osd} against libcommon statically

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #13122 from orendu/fix-rdma-compilation

cmake: Fix broken async/rdma compilation since move to libceph-common

Reviewed-by: Adir Lev <adirl@mellanox.com>
Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Amir Vadai <amir@vadai.me>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13101 from Adirl/fix_mem_leak

msg/async/rdma: Fix memory leak of OSD

Reviewed-by: Haomai Wang <haomai@xsky.com>

cmake: Fix broken async/rdma compilation since move to libceph-common

Was broken since merge of pull request #12840

Signed-off-by: Oren Duer <oren@mellanox.com>

Merge pull request #13075 from tchaikov/wip-zstd

compressor/zstd: add zstd compression plugin

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

librbd: prevent self-blacklisting during break lock

Fixes: http://tracker.ceph.com/issues/18666
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13086 from dillaman/wip-librbd-lock-init-race

librbd: race initializing exclusive lock and configuring IO path

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

compressor/zstd: add zstd compressor

Build/link of zstd itself is maybe not ideal, but it works fine.

Signed-off-by: Sage Weil <sage@redhat.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

add src/zstd submodule

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async/rdma: Fix memory leak of OSD

We can delete qp only in RDMADispatcher::handle_async_event() which call
"erase_qpn" to enable deletion.

issue: 959004

Change-Id: Iab69cb365b37a09e9608d4b3c595e05278bbe021
Signed-off-by: Sarit Zubakov <saritz@mellanox.com>

Merge pull request #13090 from dzafman/wip-18624

Revert "PrimaryLogPG::failed_push: update missing as well"

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13096 from amirv/new_connect_arg

msg/RDMA: Fix broken compilation due to new argument in net.connect()

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #9348 from LiumxNL/fix-mksnap

mon/OSDMonitor: make snaps on tier pool should not be allowed

Merge pull request #12937 from chendave/auth_method

auth: Enhancement for the supported auth methods

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12992 from tchaikov/wip-demangle-on-freebsd

common/BackTrace: demangle on FreeBSD also

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #12620 from varadakari/wip-rocksdb-histogram

os/bluestore: kvdb histogram

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #13031 from dzafman/wip-calc-stats

osd: Calculate degraded and misplaced more accurately

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13032 from liewegas/wip-bluefs-append

os/bluestore/BlueFS: tune flushing of writes

Tested-by: Chagam, Anjaneya <anjaneya.chagam@intel.com>

Merge pull request #13072 from wido/issue-18638

osd: Return correct osd_objectstore in OSD metadata

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #13018 from songbaisen/s11

global: we need to handle the init_on_startup return value when global_init.

Reviewed-by: Kefu Chai <kchai@redhat.com>

msg/RDMA: Fix broken compilation due to new argument in net.connect()

Fixes: 6e4ed291afc3 ("msg: add ms_bind_before_connect to bind before connect")
Change-Id: Ia45f215b5d59dfc8545017518e5162404059829e
Signed-off-by: Amir Vadai <amir@vadai.me>

Revert "PrimaryLogPG::failed_push: update missing as well"

This reverts commit dd48b972afde2dfa9ab1a6942c7961750222986d.

Fixes: http://tracker.ceph.com/issues/18624
Signed-off-by: David Zafman <dzafman@redhat.com>

qa: unpin knfs from ubuntu

We have an updated nfs-utils that is no longer
generating spurious selinux warnings on CentOS.

Fixes: http://tracker.ceph.com/issues/16397
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #12445 from pritha-srivastava/wip_glob_func

rgw: Added a globbing method for AWS Policies.

Merge pull request #13084 from linuxbox2/wip-librgw-parentref

rgw_file:  interned RGWFileHandle objects need parent refs

Merge pull request #13083 from dachary/wip-15653-crush-weights

crush: verify weights is influenced by the number of replicas

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13076 from dillaman/wip-18617

test: use librados API to retrieve config params

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #12866 from dzafman/wip-18471

qa/workunits/ceph-helpers: wait_for_clean() races with pg creation

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw_file:  interned RGWFileHandle objects need parent refs

RGW NFS fhcache/RGWFileHandle operators assume existence of the
full chain of parents from any object to the its fs_root--this is
a consequence of the weakly-connected namespace design goal, and
not a defect.

This change ensures the invariant by taking a parent ref when
objects are interned (when a parent ref is guaranteed).  Parent
refs are returned when objects are destroyed--essentially by the
invariant, such a ref must exist.

The extra ref is omitted when parent->is_root(), as that node is
not in the LRU cache.

Fixes: http://tracker.ceph.com/issues/18650
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #13038 from linuxbox2/wip-librgw-invalidate

rgw_file: add timed namespace invalidation

librbd: race initializing exclusive lock and configuring IO path

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12916 from Liuchang0812/xmlformatter-fix

common/xmlformatter: turn on underscored and add unittest

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

librbd: improve debug logging for lock / watch state machines

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test: use librados API to retrieve config params

The CephContext object is not ABI-stable, so it is necessary to
use the "conf_get" librados methods to safely retrieve a setting.

Fixes: http://tracker.ceph.com/issues/18617
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

crush: verify weights is influenced by the number of replicas

Refs: http://tracker.ceph.com/issues/15653

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Signed-off-by: Loic Dachary <loic@dachary.org>

rgw_file: add timed namespace invalidation

With change, librgw/rgw_file consumers can provide an invalidation
callback, which is used by the library to invalidate directories
whose contents should be forgotten.

The existing RGWLib GC mechanism is being used to drive this.  New
configuration params have been added.  The main configurable is
rgw_nfs_namespace_expire_secs, the expire timeout.

Updated post Yehuda review.

Fixes: http://tracker.ceph.com/issues/18651
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #12956 from liuhongtong/wip-nvmedevice

os/bluestore: fix NVMEDevice::open failure if serial number ends with a …

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Pan liu <liupan1111@gmail.com>
Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

osd: Return correct osd_objectstore in OSD metadata

Do not simply read the configuration value as it might have changed
during OSD startup by reading the type from disk.

Fixes: http://tracker.ceph.com/issues/18638
Signed-off-by: Wido den Hollander <wido@42on.com>

Merge pull request #13055 from optimistyzy/2017123_random_read

os/bluestore/NVMEDevice.cc: fix the random read issue.

Reviewed-by: Haomai Wang <haomai@xsky.com>

os/bluestore/NVMEDevice.cc: use aio_wake in both read and read_random

There are some issues with random read, it seems that
we did not call aio_wake.

Signed-off-by: optimistyzy <optimistyzy@gmail.com>

Merge pull request #12929 from wjwithagen/wip-wjw-empty-cluster_network

common/pick_address.cc: Copy public_netw to cluster_netw if cluster empty

Reviewed-by: Sage Weil <sage@redhat.com>

Merge branch 'master' into wip-wjw-empty-cluster_network

os/bluestore: include logical object offset in crc error message

This is more useful to the user.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12901 from yuyuyu101/wip-msgr-client-bind

msg: client bind

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13036 from SUSE/wip-michal-koutny

mailmap: Michal Koutny affiliation

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Michal Koutný <mkoutny@suse.com>

Merge pull request #12326 from vshankar/wip-15028

librbd: create fewer empty objects during copyup

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

os/bluestore/BlueFS: tolerate longer tail

The important requirement here is that we have enough buffer to write the
whole page.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: increase size threshold before we flush (and generate io)

Having this too high means you might be more bursty. In practice,
though, the commit path is doing explicit syncs on small chunks
anyway.  And compaction work should probably stay reasonably chunky.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: preallocate larger write buffer

Signed-off-by: Sage Weil <sage@redhat.com>

unittest_bit_alloc race fix

Signed-off-by: Ramesh Chander <Ramesh.Chander@sandisk.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13056 from optimistyzy/123_logic

NVMEDevice.cc: cleanup the logic in data_buf_next_sge

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #12938 from LiumxNL/fix-process-osd-failure

mon/OSDMonitor: fix process osd failure

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9419 from songbaisen/a4

osdmaptool: show all the pg map to osds info

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12828 from donglinpeng/dlppull

common/buffer: close pipe fd if set nonblocking fails.

Reviewed-by: Kefu Chai <kchai@redhat.com>

NVMEDevice.cc: cleanup the logic in data_buf_next_sge

Signed-off-by: optimistyzy <optimistyzy@gmail.com>

Merge pull request #12980 from liewegas/wip-osdop-printer

osd/osd_types: clean up OSDOp printers

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

osd/osd_types: clean up OSDOp printers

Specify which ones get offset~length instead of blacklisting those that
don't.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12950 from liewegas/wip-omap-getkeys

osd: extend OMAP_GETKEYS and GETVALS to include a 'more' output field

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #12961 from songbaisen/s10

crush:  when osd_location_hook does not exist, we should exit error.

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12975 from wonzhq/fix-pgstats-down

mon: don't set last_osd_report when the pg stats msg is ignored

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12976 from LiumxNL/wip-170118

osd/OSDMap: get_previous_up_osd_before() may run into endless loop

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12981 from liewegas/wip-crush-move-osd

mon/OSDMonitor: make 'osd crush move ...' work on osds

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13008 from athanatos/wip-clone-range

osd,librados: remove clone_range and associated multi-object cruft

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13033 from dillaman/wip-18618

journal: don't hold future lock during assignment

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

bluestore: Release segs before delete task when write fails

Fix assert failure in ~Task().

Signed-off-by: Hongtong Liu <hongtong.liu@istuary.com>

os/bluestore: fix NVMEDevice::open failure if serial number ends with a number

buf in effect is the serial number in ceph.conf and
the serial number consists of 16 hexadecimal characters.
1. In order to avoid ignoring the numbers, scan buf
with isxdigit.
2. In order to ignore all the potential garbage,
scan buf from the beginning.

Signed-off-by: Hongtong Liu <hongtong.liu@istuary.com>

common/BackTrace: demangle on FreeBSD also

the output on FreeBSD/clang looks like:

1: 0x44bfb3 <_Z3foov+0x413> at /usr/srcs/Ceph/work/ceph/build/bin/unittest_back_trace
2: 0x44c23e <_ZN20BackTrace_Basic_Test8TestBodyEv+0x1e> at /usr/srcs/Ceph/work/ceph/build/bin/unittest_back_trace
3: 0x4d068a <_ZN7testing8internal38HandleSehExceptionsInMethodIfSupportedINS_4TestEvEET0_PT_MS4_FS3_vEPKc+0x7a> at /usr/srcs/Ceph/work/ceph/build/bin/unittest_back_trace
4: 0x4b5977 <_ZN7testing8internal35HandleExceptionsInMethodIfSupportedINS_4TestEvEET0_PT_MS4_FS3_vEPKc+0x77> at /usr/srcs/Ceph/work/ceph/build/bin/unittest_back_trace
...

and update the test accordingly, as FreeBSD/clang uses '<>' to enclose
the mangled function and offset.

also, only demangle the C++ mangled names. those names always start with
"_Z". on FreeBSD, after demangling, "main" is turned into "unsigned
long", which does not make sense.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13051 from chuanhong-wang/dev_conf_clean

common: delete unused conf "filestore_debug_disable_sharded_check"

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

common: delete unused conf "filestore_debug_disable_sharded_check"

Signed-off-by: Chuanhong Wang <chuanhong.wang@163.com>

test: cannot create pool snap on tier

Signed-off-by: Mingxin Liu <mingxin@xsky.com>

OSDMonitor: make pool snaps on cache tier should be not allowed

Signed-off-by: Mingxin Liu <mingxin@xsky.com>

Merge pull request #13042 from dillaman/wip-librbd-compile-warning

librbd: fixed initializer list ordering

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

journal: don't hold future lock during assignment

It's possible that the future raced with its owner and reaches
an empty reference count. This was resulting in the future being
destructed while its lock was still held.

Fixes: http://tracker.ceph.com/issues/18618
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: fixed initializer list ordering

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test / librbd: create few empty objects during flatten

Fixes: http://tracker.ceph.com/issues/15028
Signed-off-by: Venky Shankar <vshankar@redhat.com>

test / librbd: Write ones instead of zeroes to test images

Signed-off-by: Douglas Fuller <dfuller@redhat.com>
Signed-off-by: Venky Shankar <vshankar@redhat.com>

librbd: Create few empty objects during copyup

This is based out of Doug's (@fullerdj) work (PR #9329)
as an attempt to avoid creating empty objects when
flattening an image and otherwise whenever unnecessary.

This gives good optimization benefit when a parent image
is sparsely populated. Moreover, this change is required
for correct behavior when checking disk usage of a clone
(which used to report fully allocated image due to all,
including empty objects being created during flatten).

Signed-off-by: Douglas Fuller dfuller@redhat.com
Signed-off-by: Venky Shankar <vshankar@redhat.com>

librbd: make has_parent() prone to callers from copyup

This is required when CopyupRequest would need to invoke
pre_object_map_update() as part of upcoming changes to
create fewer child image objects whenever possible.

CopyupRequest constructor accepts image extents as an
rvalue forcing the caller to transfer ownership to it
and leaving the original variable in an unspecified
stated making has_parent() return incorrect state when
invoked from CopyupRequest. Therefore, introduce a
private tracking state that can be used in place of
checking emptiness of parent image extents.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

PendingReleaseNotes: mention old clients vs new OSDs

Signed-off-by: Sage Weil <sage@redhat.com>

rados: use bare omap_get_keys op

This handles the client-side looping on 'more' if the OSD limits
the response size.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_rados: pass nullptr for pmore for omap fetch

Assume OSD limits are hugh enough.

Signed-off-by: Sage Weil <sage@redhat.com>

rados: omap bench: pass null for pmore for omap fetch

Assume OSD limits are high enough for us.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_rados_*: use new omap methods

Make these tests pass nullptr for pmore.  This codifies our
assumption that the OSD limits are high enough.

Signed-off-by: Sage Weil <sage@redhat.com>

librados: add omap_get_{keys,vals}2 with pmore output arg

Expose public methods that include a new output argument to indicate
whether there are more keys to fetch or not.

Mark the old interfaces deprecated.

Signed-off-by: Sage Weil <sage@redhat.com>

osdc/Objecter: infer ptruncated on old OSDs via max_entries

If we do not get an explicit 'more' value from the OSD, infer it by
checking whether we got the max requested entries.  On old OSDs, which
don't enforce a limit, this will work.  On new OSDs, we will get the
explicit result.

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: Use decoded URI when verifying TempURL

Instead of calliing url_decode directly, we reuse s->decoded_uri that is
initialized in RGWREST::preprocess().

Fixes: http://tracker.ceph.com/issues/18590
Signed-off-by: Michal Koutný <mkoutny@suse.com>

doc: Michal Koutny affiliation

Signed-off-by: Nathan Cutler <ncutler@suse.com>

PendingReleaseNotes: Specify the change in behaviour of assigning to public_netw

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>