rgw: keep track of written_objs correctly

Fixes: http://tracker.ceph.com/issues/15886
Only add a rados object to the written_objs list if the write
was successful. Otherwise if the write will be canceled for some
reason, we'd remove an object that we didn't write to. This was
a problem in a case where there's multiple writes that went to
the same part. The second writer should fail the write, since
we do an exclusive write. However, we added the object's name
to the written_objs list anyway, which was a real problem when
the old processor was disposed (as it was clearing the objects).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #9152 from liewegas/wip-bitmap-allocator

os/bluestore: bitmap allocator

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: bitmap allocator

Signed-off-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

os/bluestore: pass size to allocator

Signed-off-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

os/bluestore: add config for allocator

Signed-off-by: Ramesh Chander <Ramesh.Chander@sandisk.com

Merge pull request #7787 from wjwithagen/patch-1

clang: common/config.cc basename() requires libgen.h includes

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8689 from runsisi/wip-fix-ev-sk

msg/async/EventSocket: fix missing break statement

Reviewed-by: Haomai Wang <yuyuyu101@163.com>
Reviewed-by: xiexingguo <xie.xingguo@zte.com.cn>

Merge pull request #8717 from ZhenyuLeng/noneed_op_is_discardable

osd: make op_is_discardable check optional

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8961 from xiexingguo/xxg-wip-fixepochtrunc

osd: don't get inc osdmap name truncated

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8985 from badone/wip-iops-typo

common: obj_bencher.cc add missing ":"

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9011 from stiopaa1/osdmap_removeUnneededIncludeFiles

osd/OSDMap.h: remove unneeded include files

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9075 from runsisi/wip-compute-cost

osd/ReplicatedBackend: reuse the already computed cost

Reviewed-by: xiexingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9077 from liewegas/wip-osd-map-cache

osd: instrument map cache hits and misses

Reviewed-by: xiexingguo <xie.xingguo@zte.com.cn>

Merge pull request #9078 from liewegas/wip-clean-temps

mon, osdmap: optimize check and cleanup of pg_temp for new osdmaps

Reviewed-by: xiexingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8298 from wjwithagen/wip-wjw-freebsd-autobuild-2

build: many fixes for freebsd

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8480 from songbaisen/song19

crush: When consider rule size use the correct method.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8974 from liewegas/wip-addr-cleanup

msg/msg_types: clean up entity_addr_t

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9027 from roidayan/make-debs_avoid_creating_recursive_conf_dir

make-debs.sh: avoid creating recursive conf dir

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9144 from vumrao/wip-vumrao-15899

doc : adding explanation of --no-increasing for

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8452 from jcsp/wip-15266

pybind: fix unicode handling in CephFSVolumeClient::purge

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc : adding explanation of --no-increasing for
reweight-by-utilization and test-reweight-by-utilization
commands

Fixes: http://tracker.ceph.com/issues/15899
Signed-off-by: Vikhyat Umrao <vumrao@redhat.com>

doc/changelog: missed v10.2.1 changelog

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #9141 from liewegas/wip-10.2.1

doc/release-notes: v10.2.1 release notes

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc/release-notes: v10.2.1 release notes

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #9135 from flaf/master

doc: improvement of the "tunables" part

Reviewed-by: Sage Weil <sage@redhat.com>

doc: improvement of the "tunables" part

* Some CRUSH_TUNABLES$N references were false.
* The explanations concerning the warning when tunables are
  non-optimal were false or confusing.
* Make the definition of the "default" profile clearer.

Signed-off-by: François Lafont <francois.lafont@ac-versailles.fr>

The function op_is_discardable introduces lock overhead which can
The function op_is_discardable introduces lock overhead which can
not be ignored.It would be better to judge if to discard op when
the connection disconnected as a parameter whether the network is
believable or extremely unlikelihood.

add g_conf osd_discard_disconnected_ops to control whether to
discard op when the connection disconnected
Waiting for test

Signed-off-by: ZhenyuLeng <lengzhenyu@ict.ac.cn>

Merge pull request #9124 from stiopaa1/osd_someMorePrivate

osd/OSD.h: move some data members under private

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9128 from stiopaa1/osd_constOperatorPGQueueable

osd/OSD: pass op() params by const in PGQueueable

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9131 from stiopaa1/compatset_removeConstructor

common: CompatSet.h: Remove unneeded constuctor

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9132 from shinobu-x/wip-shinobu

cls_rbd: fix comment typo

Reviewed-by: Sage Weil <sage@redhat.com>

src/cls/rbd/cls_rbd.cc: fix super trivial typo

CompatSet.h: Remove unneeded constuctor

We have a constuctor taking a std::string so there is no need for
the constuctor taking char*.

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

osd/OSD: pass op() params by const in PGQueueable

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #8767 from jjhuo/org_jianjian

.mailmap and .orgmap: organization affiliation

Reviewed-by: Loic Dachary <ldachary@redhat.com>

osd/OSD.h: move some data members under private

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #9123 from idryomov/wip-qa-rbd-dont-chown

qa: rbd: don't modprobe, chown sysfs files or udevadm settle

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa: rbd/simple_big.sh: drop extraneous sudo

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa: rbd/concurrent.sh: suppress rbd map output

rbd_map_image() echoes the id of the image and is used in command
substitution:

    id=$(rbd_map_image "${image}")

Output from rbd map isn't consumed and clobbers the return.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa: rbd: don't modprobe, chown sysfs files or udevadm settle

This is a followup to 38a572011dbb ("qa: rbd/map-snapshot-io.sh: don't
chown sysfs files") for the rest of the workunits:

- /sys/bus/rbd/add_single_major is used if present, so chown'ing
  /sys/bus/rbd/add doesn't help.  Use sudo on rbd map instead.

- Don't modprobe or udevadm settle - rbd CLI tool takes care of that.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #9070 from liewegas/doc-ct

doc: words of caution about cache tiering

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Nick Fisk <nick@fisk.me.uk>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc/rados: adjust example cache tiering settings

These are saner values.

Signed-off-by: Sage Weil <sage@redhat.com>

doc/rados: words of caution about cache tiering

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #9110 from stiopaa1/osd_moveTickClasses

osd: move tick classes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8887 from jcsp/wip-15689

osdc: send error to recovery waiters on shutdown

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #8835 from xiaoxichen/implement_fsync_in_pybind

pybind: expose fsync in cephfs binding.

Reviewed-by: John Spray <john.spray@redhat.com>

osd/ReplicatedBackend: reuse the already computed cost

no need to compute the cost a second time

Signed-off-by: runsisi <runsisi@zte.com.cn>

osd/OSD.cc: add override to virtual functions

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

osd/OSD: move C_Tick and C_Tick_WithoutOSDLock

Since above classes are implementation detail of OSD class move them
to the .cc file.

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #9079 from liewegas/wip-bluestore-freelist

os/bluestore: switch to bitmap freelist by default

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #8956 from shinobu-x/wip-shinobu

os/bluestore: fix permissions on StupidAllocator.cc, cleanups

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9074 from tchaikov/wip-cmake-manpages

cmake: build man pages

Reviewed-by: Casey Bodley <cbodley@redhat.com>

osd/OSDMap: fix clean_temps and test

For both pg_temp and primary_temp, we want to calculate what the
mapping *will* be after the inc is applied, and if the mapping is
redundant (it matches what we would have gotten anyway), remove
it.  Removing the mapping might mean putting a 'remove' entry in
the inc, or it might mean removing the proposed addition in the
inc.

Update the unit test to test both cases.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMap: consolidate into clean_temps

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMap: move primary_temp checks to check_down_temps

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMap: move redundant pg_temp check to remove_down_temps

Better to do this on the future map value.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMap: move old pool check to remove_down_temps

Better to do this on the projected map state.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMap: minor optimization of remove_down_temps

Signed-off-by: Sage Weil <sage@redhat.com>

cmake: restructure and add missing manpages

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: build man pages

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9024 from trociny/fixup-rbd-mirror-asok

rbd-mirror: don't unregister asok commands if image replayer start failed

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9025 from trociny/fixup-rbd-mirror-status

rbd-mirror: avoid potential deadlock

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9043 from fullerdj/wip-djf-fsx-fix

test/rbd: document fsx option to keep data on success

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9071 from runsisi/wip-rbd-mirror-minor-cleanup

rbd-mirror: minor cleanup

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8924 from xiexingguo/xxg-wip-monitor

mon/Monitor: fix memory leak; add leaked key to tracked_conf_key

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Marcel Lauhoff <lauhoff@uni-mainz.de>

Merge pull request #8952 from badone/wip-i-am-osd-fix

osd: Trivial grammatical fix 'i' to 'I'

Reviewed-by: songbaisen <song.baisen@zte.com.cn>

Merge pull request #8993 from XinzeChi/wip-copy-get

osd: fix ReplicatedPG::fill_in_copy_get logic

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8995 from xiexingguo/xxg-wip-fixpingreply

osd: fix potential access violation during handling PING_REPLY

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8990 from tchaikov/wip-13990

osd: remove all stale osdmaps in handle_osd_map()

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #8998 from kylinstorage/wip-fix-wrong-raw-pointer-conversion

librados: fix wrong raw pointer conversion

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa: rbd/map-snapshot-io.sh: don't chown sysfs files

qa: rbd/map-snapshot-io.sh: don't chown sysfs files

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa: rbd/map-snapshot-io.sh: don't chown sysfs files

/sys/bus/rbd/add_single_major is used if present, so chown'ing
/sys/bus/rbd/add doesn't help.  Use sudo on rbd map instead.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

os/bluestore: switch to bitmap freelist by default

Signed-off-by: Sage Weil <sage@redhat.com>

doc/release-notes: v0.94.7 hammer

Signed-off-by: Sage Weil <sage@redhat.com>

common/address_helper: drop entity_addr_from_sockaddr

entity_addr_t::set_sockaddr() does this.

Signed-off-by: Sage Weil <sage@redhat.com>

msg/xio: use set_sockaddr

Signed-off-by: Sage Weil <sage@redhat.com>

common/address_helper: use set_sockaddr

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #8377 from vshankar/bs-alloc-cleanup

bluefs: use Allocator::create() to instantiate allocator

Reviewed-by: Sage Weil <sage@redhat.com>

pybind: fix unicode handling in CephFSVolumeClient::purge

os.path.join is sensitive to string encoding, but
just doing a straight substitution should not be.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #8482 from majianpeng/bluestore-prefallocate

os/bluestore/BlueStore: preallocate space when use file instead of blockdevice

Reviewed-by: Sage Weil <sage@redhat.com>

msg/msg_types: drop sockaddr_storage from entity_addr_t

This reduces the struct size by a ton!  136 -> 36 bytes.

Encode to sockaddr_storage is now slightly slower.

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async/AsyncConnection: do not use sizeof(entity_addr_t)

Protocol is defined in terms of ceph_entity_addr.

Signed-off-by: Sage Weil <sage@redhat.com>

msg/simple/Pipe: do not use sizeof(entity_addr_t)

The protocol is defined in terms of ceph_entity_addr; use that.

Signed-off-by: Sage Weil <sage@redhat.com>

msg/msg_types: put sockaddr_* fields in a union

No functional change here, just a rename.

Name the fields after the struct prefixes (e.g., sockaddr_in.sin_ -> sin).

Signed-off-by: Sage Weil <sage@redhat.com>

test/librbd/exclusive_lock: use proper entity_addr_t accessors

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #8771 from xiaoxichen/configurable_prefix

pybind: configurable cephfs_vol_client prefix.

Reviewed-by: John Spray <john.spray@redhat.com>

rbd-mirror: fix typo

Signed-off-by: runsisi <runsisi@zte.com.cn>

rbd-mirror: librbd::mirror_peer_list never returns -ENOENT

no need to check -ENOENT specially

Signed-off-by: runsisi <runsisi@zte.com.cn>

rbd-mirror: use the already known pool name directly

no need to try to get the pool name by pool id again

Signed-off-by: runsisi <runsisi@zte.com.cn>

Merge pull request #9067 from trociny/fixup-mirror_image_status_list

cls::rbd: mirror_image_status_list returned max 64 items

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8602 from ajarr/wip-15045

ceph_volume_client: evict client also based on mount path

Reviewed-by: John Spray <john.spray@redhat.com>

doc/rados: drop description of cache modes we don't recommend

Signed-off-by: Sage Weil <sage@redhat.com>

msg/msg_types: dump via sockaddr* operator<<

Signed-off-by: Sage Weil <sage@redhat.com>

tools/rbd/action/Kernel: use sockaddr (not _storage) to stringify

Signed-off-by: Sage Weil <sage@redhat.com>

krbd: use sockaddr instead of sockaddr_storage to print

Signed-off-by: Sage Weil <sage@redhat.com>

msg/msg_types: operator<< for sockaddr

Signed-off-by: Sage Weil <sage@redhat.com>

msg/msg_types: drop addr(), addr_size() accessors

addr() returns a reference to a sockaddr_storage, which we want to drop.

addr_size() is replaced by get_sockaddr_len().

Signed-off-by: Sage Weil <sage@redhat.com>

msg/simple/Pipe: use sockaddr accessors

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async/net_handler: use sockaddr accessors

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async/AsyncMessenger: use sockaddr accessors for bind

Signed-off-by: Sage Weil <sage@redhat.com>