12.0.0

Merge pull request #12979 from theanalyst/scripts/release-notes-tracker

script: ceph-release-notes check orig. issue only for backports

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #13277 from jan--f/janf-teuth-cloud-doc

doc: minor change to a cloud testing paragraph

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #13275 from idryomov/wip-require-osd-features-fix

osd/OSDMap: require OSD features only of OSDs

Reviewed-by: Sage Weil <sage@redhat.com>

doc: minor change to a cloud testing paragraph

Use terminate for the instances to match the terminology used by the
horizon interface.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #13256 from dongbula/correct-cached_crc-comment

osd: correct comment of perfcounter cached_crc in code

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13220 from rzarzynski/wip-rgw-18364-2

rgw: be aware abount tenants on cls_user_bucket -> rgw_bucket conversion

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #12993 from jcsp/wip-cephfs-doc-authcaps

doc: clarify the path restriction mds cap example

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #13242 from Jing-Scott/dev/fix-swift-anon-getobj-error-code

rgw: swift: fix anonymous user's error code of getting object

Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

osd/OSDMap: require OSD features only of OSDs

Fix up commit 1a5cc32f0a3b ("osd/OSDMap: reflect REQUIRE_*_OSDS flag in
required features") -- require_*_osds shouldn't affect older clients.

Fixes: http://tracker.ceph.com/issues/18831
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #13271 from Liuchang0812/fix-file-description-leak

msg/async: fix file description leak in NetHandler

Reviewed-by: Haomai Wang <haomai@xsky.com>

msg/async: fix file description leak in NetHandler

Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

rgw: swift: fix anonymous user's error code of getting object

The openstack swift will return 401 rather than 403 when
the anon user has no permission to get objects.

Fixes: http://tracker.ceph.com/issues/18806
Signed-off-by: Jing Wenjun <jingwenjun@cmss.chinamobile.com>

Merge pull request #13237 from smithfarm/wip-18799

tests: Thrasher: eliminate a race between kill_osd and __init__

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13263 from Adirl/fix_log_spaces

msg/async/rdma: fix log line spacing

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #13257 from tchaikov/wip-less-verbose-ceph-helper

tests: ceph-helpers.sh reduce get_timeout_delays() verbosity

Reviewed-by: Loic Dachary <ldachary@redhat.com>

tests: ceph-helpers.sh reduce get_timeout_delays() verbosity

`set +o` prints out the full command line which is echoed if "xtrace" is
enabled. this increases the verbosity of get_timeout_delays().
in this change, we follow the way of kill_daemons() to kill the extra
output. see aefcf6d.

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: correct comment of perfcounter cached_crc in code

Signed-off-by: lvshuhua <lvshuhua@cmss.chinamobile.com>

Merge pull request #13080 from yuyuyu101/wip-async-fd

msg/async/AsyncConnection: socket's fd can be zero, avoid false assert

Reviewed-by: Kefu Chai <kefu@redhat.com>

Merge pull request #13251 from scuttlemonkey/master

script: adding contributor credits script

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9396 from songbaisen/doc2

doc: fix the librados c api can not compile problem

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13252 from linuxbox2/wip-rgw-setattr

rgw_file:  fix RGWLibFS::setattr for directory objects

rgw_file:  fix RGWLibFS::setattr for directory objects

Fixes: http://tracker.ceph.com/issues/18808
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #13147 from yehudasa/wip-18331

rgw: fix use of marker in List::list_objects()

Merge pull request #12023 from yuyuyu101/wip-msgr-type

msg: allow different ms type for cluster network and public network

Reviewed-by: Adir Lev <adirl@mellanox.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13114 from liewegas/wip-osdmap-required-features

osd: condition object_info_t encoding on required (not up) features

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #13206 from liewegas/wip-osd-enxio

osd: do not send ENXIO on misdirected op by default

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #13213 from ovh/bp-osd-updateable-throttles

osd: allow client throttler to be adjusted on-fly, without restart

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13063 from ovh/wip-mailmap-ovh-contributors

mailmap: Update OVH contributors

Reviewed-by: Loic Dachary <ldachary@redhat.com>

adding contributor credits script

Signed-off-by: Patrick McGarry <pmcgarry@redhat.com>

Merge pull request #13223 from wjwithagen/wip-wjw-freebsd-sphynx

build/install-deps.sh: Add sphinx package for building docs on FreeBSD

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11448 from ledgr/change/mount_fuse_ceph_options_in_fs_mntops_and_python

Rewrite mount.fuse.ceph (to python) and move ceph-fuse options to fs_mntops

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #13194 from smithfarm/wip-16263

tests: ignore bogus ceph-objectstore-tool error in ceph_manager

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

tests: Thrasher: eliminate a race between kill_osd and __init__

If Thrasher.__init__() spawns the do_thrash thread before initializing the
ceph_objectstore_tool property, do_thrash races with the rest
of Thrasher.__init__() and in some cases do_thrash can call kill_osd() before
Trasher.__init__() progresses much further. This can lead to an exception
("AttributeError: Thrasher instance has no attribute 'ceph_objectstore_tool'")
being thrown in kill_osd().

This commit eliminates the race by making sure the ceph_objectstore_tool
attribute is initialized before the do_thrash thread is spawned.

Fixes: http://tracker.ceph.com/issues/18799
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #13234 from ifed01/wip-bluestore-performance-sandbox2017

os/bluestore: fix potential assert in cache _trim method.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12965 from linuxbox2/wip-librgw-expand-argv

rgw_file:  split last argv on ws, if provided

rgw_file:  split last argv on ws, if provided

This is intended to allow an "extra" unparsed argument string
containing various cmdline options to be passed as the last argument
in the argv array of librgw_create(), which nfs-ganesha is
expecting to happen.

While at it, hook env_args() too.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw: be aware abount tenants on cls_user_bucket -> rgw_bucket conversion.

Fixes: http://tracker.ceph.com/issues/18364
Fixes: http://tracker.ceph.com/issues/16355
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

mailmap: Update OVH contributors

Signed-off-by: Bartłomiej Święcki <bartlomiej.swiecki@corp.ovh.com>

Merge pull request #12335 from theanalyst/mailmap/v11.1.0

mailmap updates for v11.1.0

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ceph_test_objectstore: match clone_range src and dst offset

This is what the OSD does, and BlueStore doesn't necessarily handle
all combinations of different offsets.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #13211 from liewegas/wip-clone-offsets

ceph_test_objectstore: match clone_range src and dst offset

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #12904 from ifed01/wip-bluestore-onode-refmap3

os/bluestore: replace Blob ref_map with reference counting

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13192 from liewegas/wip-bluestore-min-alloc-size

os/bluestore: fix min_alloc_size at mkfs time

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: fix potential assert in cache _trim method.

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

msg/async/rdma: fix log line spacing

issue: none

Change-Id: Ieef3284ee673f012331d21fffcb499e22473f62a
Signed-off-by: Adir Lev <adirl@mellanox.com>

OSD: allow client throttler to be adjusted on-fly, without restart

This patch allows the osd_client_message_cap and
osd_client_message_size_cap to be adjusted on-fly, using admin socket
functionality.

Fixes: http://tracker.ceph.com/issues/18791
Signed-off-by: Piotr Dałek <piotr.dalek@corp.ovh.com>

ceph_mds: adopt ms public type

Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge pull request #13158 from dillaman/wip-18594

qa: integrate OpenStack 'gate-tempest-dsvm-full-devstack-plugin-ceph'

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #13225 from tmuthamizhan/wip-ansible-m

qa: do not mention ceph branch explicitly

do not mention ceph branch explicitly

no need to mention ceph_dev_branch explicitly. it will be taken from the
ceph branch value mentioned in the teuthology-suite command

Signed-off-by: Tamil Muthamizhan <tmuthami@redhat.com>

build/install-deps.sh: Add sphinx package for DOCS

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #13009 from liewegas/wip-mempool-bloom-filter

mempool: put bloom_filter in mempool

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #13136 from liewegas/wip-scrub-waitlist

osd: use separate waitlist for scrub

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #12104 from pritha-srivastava/wip_sts_role_rest

rgw: Added code for REST APIs for AWS Roles.

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #13189 from gaurav36/bug-18326-image-rename-non-rbd-pool

rbd: destination pool should be source pool if it is not specified

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12948 from trociny/wip-17020

rbd-mirror: add support for active/passive daemon instances

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13013 from wangzhengyong/master

os/bluestore: remove no use parameter in bluestore_blob_t::map_bl

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12782 from atheism/rgw-orphan-fix-error-messages

rgw: orphan: fix error messages

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13199 from chipitsine/master

mds,mon: Clean issues detected by cppcheck

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13203 from ifed01/wip-bluestore-more-dump

os: extend ObjectStore interface to dump store's performance counters

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12974 from ukernel/wip-multimds-misc

mds: miscellaneous fixes

Reviewed-by: John Spray <john.spray@redhat.com>

qa/suites/rbd: add rbd-mirror-ha test

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

qa/workunits/rbd: tweaks to use RBD_MIRROR_USE_EXISTING_CLUSTER locally

This var is mostly used when running rbd_mirror test scripts on
teuthology.  It can be used locally though to speedup re-running the
tests:

Set a test temp directory:

  export RBD_MIRROR_TEMDIR=/tmp/tmp.rbd_mirror

Run the tests the first time with NOCLEANUP flag (the cluster and
daemons are not stopped on finish):

  RBD_MIRROR_NOCLEANUP=1 ../qa/workunits/rbd/rbd_mirror.sh

Now, to re-run the test without restarting the cluster, run cleanup
with USE_EXISTING_CLUSTER flag:

  RBD_MIRROR_USE_EXISTING_CLUSTER=1 \
    ../qa/workunits/rbd/rbd_mirror_ha.sh cleanup

and then run the tests:

  RBD_MIRROR_USE_EXISTING_CLUSTER=1
    ../qa/workunits/rbd/rbd_mirror_ha.sh

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

qa/workunits/rbd: tests for rbd mirror HA

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

qa/workunits/rbd: allow to start several daemons for the same cluster

by optionally specifyning daemon instance after cluster name and
colon, like:

  start_mirror ${cluster}:${instance}

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

rbd-mirror HA: pool replayer should be started/stopped when lock acquired/released

Fixes: http://tracker.ceph.com/issues/17020
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

rbd-mirror HA: create pool locker / leader class

Fixes: http://tracker.ceph.com/issues/17019
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

rbd-mirror HA: add new lock released/acquired and heartbeat messages

Fixes: http://tracker.ceph.com/issues/17018
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

rbd-mirror: add class for mirror status watcher

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: equality operator for managed_lock::Locker

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: add create_async_context_callback helper

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: missing const qualifiers

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: track in-fly break_lock and get_locker requests in managed lock

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: pass notifier_id to Watcher handle_notify

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: allow watcher derived classes access ioctx, oid and work queue

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

rbd: destination pool should be source pool if it is not specified

Currently if user perform image rename operation and user give pool
name as a optional parameter (--pool=<pool_name>) then currently
its taking this optional pool name for source pool and making
destination pool name default pool name.
With this fix if user provide pool name as a optional pool name
parameter then it  will consider both soruce and destination pool
name as optional parameter pool name.

Fixes: http://tracker.ceph.com/issues/18326
Reported-by: МАРК КОРЕНБЕРГ <socketpair@gmail.com>
Signed-off-by: Gaurav Kumar Garg <garg.gaurav52@gmail.com>

ceph_test_objectstore: match clone_range src and dst offset

This is what the OSD does, and BlueStore doesn't necessarily handle
all combinations of different offsets.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: do not send ENXIO on misdirected op by default

In practice this tends to get bubbled up the stack as an error on
the caller, and they usually do not handle it properly.  For example,
with librbd, this turns into EIO and break the VM.

Instead, this will manifest as a hung op on the client.  That is
also not ideal, but given that the root cause here is generally a
bug, it's not clear what else would be better.

We already log an error in the cluster log, so teuthology runs will
continue to fail.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #13077 from guzhongyan/fix_wrongly_init

librbd: initialize diff parent overlap to zero

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12537 from liewegas/wip-optracker

common/TrackedOp: various cleanups and optimizations

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #13205 from dachary/wip-crush-documentation

crush: API documentation

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: decode min_alloc_size into temp var

Defend against future changes to min_alloc_size.

Signed-off-by: Sage Weil <sage@redhat.com>

crush: API documentation

Signed-off-by: Loic Dachary <loic@dachary.org>

os/bluestore: fix min_alloc_size at mkfs time

It is an ongoing challenge to allow min_alloc_size to be varied on
an existing bluestore instance, and the code paths are not well
tested.  Avoid the complexity entirely by fixing min_alloc_size at
mkfs time.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: add ondisk format version info to super section

Note the version of the ondisk format and the oldest version that is
allowed to read us.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: EIO if we can't read some existing super keys

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #13164 from dzafman/wip-18718

osd: Fix useless MAX(0, unsigned) to prevent out of wack misplaced

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: add submit latency perf counter.

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

test/fio: add performance counters dump on FIO job completion.

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os: add dump_perf_counters method to ObjectStore implementations.

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: fix typos in perf counter public names.

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

test/fio: fix misordered parameter for queue_transaction.

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #13193 from dillaman/wip-18738

journal: stop processing removal after error

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #13069 from SUSE/wip-hammer-retirement

doc: add 0.94.10 and hammer EOL to releases.rst

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Clean issues detected by cppcheck

[src/mds/MDLog.cc:1172] -> [src/mds/MDLog.cc:1173]: (warning) Either the condition 'sle!=NULL' is redundant or there is possible null pointer dereference: sle.
[src/mon/MDSMonitor.cc:2622] -> [src/mon/MDSMonitor.cc:2625]: (warning) Either the condition 'mds_map!=nullptr' is redundant or there is possible null pointer dereference: mds_map.

Signed-off-by: Ilya Shipitsin <chipitsine@gmail.com>

Merge pull request #13151 from smithfarm/wip-ceph-release-notes

tools: ceph-release-notes: ignore low-numbered PRs

Merge pull request #13190 from Adirl/reduce_num_buffers

msg/async/rdma: reduce number of rdma rx/tx buffers

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #13066 from liewegas/wip-aio-read

os/bluestore: use aio for reads

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #13071 from liewegas/wip-omap-getkeys-rados-test

qa/suites/rados/basic: set low omap limit for rgw workload

Reviewed-by: Josh Durgin <jdurgin@redhat.com>