osd/PG: restrict want_acting to up+acting on recovery completion

On recovery completion we recalculate want_acting to see if we
should add recently backfilled osds into acting.  However, at
this point we may have gotten infos from others OSDs outside
of up/acting that could be used for want_acting.  We currently
assert that only up/acting osds are used in
PG::RecoveryState::Active::react(const AdvMap&), so we must
restrict want_acting to up/acting here.

We could remove this restriction, but it would mean

1) checking on every map change that want_acting hasn't been
invalidated, and if so, recalculating want_acting and requesting
a new pg_temp.  Also, presumably

2) on each new info, checking whether we can construct a better
want_acting, and if so, doing it.

That would be a good thing, but is a more complicated change.  In
reality this case comes up very rarely, so simply make our
post-recovery want_acting calculation limit itself to up+acting.

See 1db67c443d84dc5d1ff53cc820fdfd4a2128b680 for the assertion.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #13389 from trociny/wip-rbd-nbd-unused_old_format

rbd-nbd: no need to check image format any more

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13248 from rzarzynski/wip-rgw-18796

rgw: improve handling of illformed Swift's container ACLs.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #12810 from zhangsw/fix-rgw-datasync-errorlog

rgw: error_code in error log is not right when data sync fails.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #13394 from tchaikov/wip-18838

cmake: link consumers of libclient with libcommon

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11128 from tchaikov/wip-16091

mon/MonClient: hunt monitors in parallel

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mon/MonClient: remove unnecessary helper functions

refactor _reopen_session() by removing wrapper around it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/MonClient: remove unnecessary include

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/monclient: hunt for multiple monitor in parallel

* add an option "mon_client_hunt_parallel" for the maxmimum number of parallel
  hunting sessions.

Fixes: http://tracker.ceph.com/issues/16091
Signed-off-by: Steven Dieffenbach <sdieffen@redhat.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/MonClient: mark monc_lock a mutable

so we can label the getters of MonClient with the `const` specifier.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/MonClient: use __func__ for function names

Signed-off-by: Steven Dieffenbach <sdieffen@redhat.com>

client: move monc->set_want_keys() before monc->init()

if monc's tick connect to the mon before monc.set_want_keys() is called,
monc won't ask for the key for MDS service, and hence will fail to
build_authorizer() for MDS service. this change ready us for the
feature of monc-connect-to-mon-in-parallel.

Signed-off-by: Kefu Chai <kchai@redhat.com>

auth: AuthClientHandler::init() pass parameter by const ref

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13149 from liewegas/wip-list-objects

librados: remove legacy object listing API, clean up newer api

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #13409 from xiexingguo/wip-fix-throttler-name

os/bluestore: add "_" prefix for internal methods

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: add "_" prefix for internal methods

These 4 methods are reserved for internal use only.
Prefix them with "_" to keep pace with others.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #13391 from Adirl/ibv_exp

msg/async/rdma: check if exp verbs avail

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #13034 from wjwithagen/wip-wjw-brag

mailmap: Willem Jan Withagen affiliation

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #13209 from wjwithagen/wip-wjw-freebsd-init-ceph

init-ceph: Make init-ceph work under FreeBSD for init-system

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13377 from wjwithagen/wip-wjw-freebsd-jemalloc

CMakeLists.txt: suppress unneeded warning about jemalloc

Merge pull request #13360 from ktdreyer/doc-firewalld-improvements

doc: improve firewalld instructions

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13399 from vumrao/wip-vumrao-18919

rgw: change loglevel to 20 for 'System already converted' message

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #13396 from tchaikov/wip-doc-dedup-refs

doc: remove duplicated references

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #13400 from shinobu-x/wip-18840

doc: Modify Configuring Cinder section

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13405 from linuxbox2/wip-librgw-refcnt

Wip librgw refcnt

Merge pull request #13403 from MichaelEischer/master

doc: Fixes parameter name in rbd configuration on openstack havana/icehouse

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rgw_file: fix hiwat behavior

Removed logic to skip reclaim processing conditionally on hiwat,
this probably meant to be related to a lowat value, which does
not exist.

Having exercised the hiwat reclaim behavior, noticed that the
path which moves unreachable objects to LRU, could and probably
should remove them altogether when q.size exceeds hiwat.  Now
the max unreachable float is lane hiwat, for all lanes.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw_file: refcnt bugfixes

This change includes 3 related changes:

1. add required lock flags for FHCache updates--this is a crash
   bug under concurrent update/lookup

2. omit to inc/dec refcnt on root filehandles in 2 places--the
   root handle current is not on the lru list, so it's not
   valid to do so

3. based on observation of LRU behavior during creates/deletes,
   update (cohort) LRU unref to move objects to LRU when their
   refcount falls to SENTINEL_REFCNT--this cheaply primes the
   current reclaim() mechanism, so very significanty improves
   space use (e.g., after deletes) in the absence of scans
   (which is common due to nfs-ganesha caching)

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw_file:  add refcount dout traces at debuglevel 17

These are helpful for checking RGWFileHandle refcnt invariants.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw_file: add pretty-print for RGWFileHandle

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

doc: Fixes parameter name in rbd configuration on openstack havana/icehouse

The documentation for using rbd together with openstack havana/icehouse
states that the parameter libvirt_disk_cachemodes should be added to
the nova.conf file. However, this is the only parameter that has no
legacy name with a 'libvirt_' prefix. (See
https://github.com/openstack/nova/blob/icehouse-eol/nova/virt/libvirt/driver.py#L252
for the configuration option)
Thus the configured disk_cachemodes were not applied, defaulting to
no caching.

Fixes: #17978
Signed-off-by: Michael Eischer <michael.eischer@fau.de>

rgw: change log level to 20 for 'System already converted' message

Fixes: http://tracker.ceph.com/issues/18919
Signed-off-by: Vikhyat Umrao <vumrao@redhat.com>

Doc: Modify Configuring Cinder section

doc/rbd/rbd-openstack.rst: Add ``volume_backend_name`` in description of
cinder.conf

Fixes: http://tracker.ceph.com/issues/18840
Signed-off-by: Shinobu Kinjo <shinobu@redhat.com>

Merge pull request #13017 from chuanhong-wang/dev_add_perfcount_for_backoff_throttle

osd: adding PerfCounters for backoff throttle

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #13274 from ifed01/wip-bluestore-perf-counters

os/bluestore: add more perf_counters to BlueStore

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13321 from liewegas/wip-kill-sortbitwise-harder

osd: kill sortbitwise

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

doc: remove duplicated references

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13303 from Jing-Scott/fix-swift-cannot-disable-object-versioning

rgw: fix swift cannot disable object versioning

@Jing-Scott  updated, addressing @rzarzynski's change request

Merge pull request #13393 from ddiss/fix_rgw_fcgi_include

rgw: set FCGI_INCLUDE_DIR for cephd_rgw_base

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-fuse: always free allocated fuse_args

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: link consumers of libclient with libcommon

to avoid linking against to both libceph-common and libcommon at the same
time, because both of them will be registered as a provider of lttng
provider.

Fixes: http://tracker.ceph.com/issues/18838
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13382 from xiexingguo/wip-counter-name

os/bluestore: fix wal-queue bytes-counter to keep pace with others

Reviewed-by: Sage Weil <sage@redhat.com>

rgw: set FCGI_INCLUDE_DIR for cephd_rgw_base

cephd_rgw_base build currently fails with fastcgi enabled:
--
In file included from /home/david/ceph/src/rgw/rgw_request.h:13:0,
from /home/david/ceph/src/rgw/rgw_main.cc:53:
/home/david/ceph/src/rgw/rgw_fcgi.h:8:21: fatal error: fcgiapp.h:
No such file or directory
^
--

This is despite the fact that fastcgi was detected and located at
configure time:
build/CMakeCache.txt:FCGI_INCLUDE_DIR:PATH=/usr/include/fastcgi

Fix this by ensuring that the cephd_rgw_base build target correctly uses
FCGI_INCLUDE_DIR.

Fixes: http://tracker.ceph.com/issues/18918
Signed-off-by: David Disseldorp <ddiss@suse.de>

Merge pull request #13392 from Adirl/zero_inline

msg/async/rdma: zero wqe inline

Reviewed-by: Haomai Wang <haomai@xsky.com>

msg/async/rdma: zero wqe inline

we dont use wqe inline

issue: none

Change-Id: Ib187eec3bc79f8caf040fbbfbf0a915186cd58ba
Signed-off-by: Adir Lev <adirl@mellanox.com>

msg/async/rdma: check if exp verbs avail

issue: 975125

Change-Id: I9daa168ca0299887a7238a688508c773b98abde9
Signed-off-by: Adir Lev <adirl@mellanox.com>
Signed-off-by: Oren Duer <oren@mellanox.com>

rgw: fix swift cannot disable object versioning

we should be able to disable object verioning by removing its X-Versions-Location
metadata header by sending an empty key value. this description can be found at
No.8 in http://docs.openstack.org/user-guide/cli-swift-set-object-versions.html.

Fixes: http://tracker.ceph.com/issues/18852
Signed-off-by: Jing Wenjun <jingwenjun@cmss.chinamobile.com>

Merge pull request #13380 from tchaikov/wip-sepia-bt

script/sepia_bt.sh: no need to pass version and sha1 anymore

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #13368 from smithfarm/wip-doc-dev

doc: instructions for provisioning OpenStack VMs ad hoc

Reviewed-by: Loic Dachary <ldachary@redhat.com>

rbd-nbd: no need to check image format any more

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #13367 from tchaikov/wip-qa-jewel-x-singleton

test: Thrasher: update pgp_num of all expanded pools if not yet

Reviewed-by: Sage Weil <sage@redhat.com>

script/sepia_bt.sh: no need to pass version and sha1 anymore

* try to figure out the working directory from the core dump's path
* add -C directory option, so user is able to specify the directory
  for the downloaded packages, and extracted files.

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: fix wal-queue bytes-counter to keep pace with others

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

test: Thrasher: restore changed options after done with thrash

Signed-off-by: Kefu Chai <kchai@redhat.com>

tests: Thrasher: extract _set_config() method

Signed-off-by: Kefu Chai <kchai@redhat.com>

tests: CephManager: add get_config() method

Signed-off-by: Kefu Chai <kchai@redhat.com>

test: Thrasher: update pgp_num of all expanded pools if not yet

otherwise wait_until_healthy will fail after timeout as seeing warning
like:

HEALTH_WARN pool cephfs_data pg_num 182 > pgp_num 172

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13378 from liewegas/wip-luminous

qa/rados/upgrade/jewel-x-singleton: run luminous.yaml at the end

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #13365 from xiexingguo/wip-backoff-refleak

osdc/Objecter: fix possible OSDSession leak on wrong connection

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13363 from jdurgin/wip-req-luminous-error

osd: clarify REQUIRE_LUMINOUS error message

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/rados/upgrade/jewel-x-singleton: run luminous.yaml at the end

renumber too

Signed-off-by: Sage Weil <sage@redhat.com>

CMakeLists.txt: suppress unneeded warningiabout jemalloc

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

doc: instructions for provisioning OpenStack VMs ad hoc

Signed-off-by: Nathan Cutler <ncutler@suse.com>

osd/PrimaryLogPG: unconditionally set PGNLS next to hobject

Drop nibblewise workaround.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: drop get_sort_bitwise() accessors, flag

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: always bitwise

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: drop bitwise arg to MissingLoc::rebuild()

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: make set_last_backfill() assume bitwise

Signed-off-by: Sage Weil <sage@redhat.com>

osd: BackfillInterval is always bitwise

Signed-off-by: Sage Weil <sage@redhat.com>

osd: fix add_source_info

Signed-off-by: Sage Weil <sage@redhat.com>

osd: fix write_blocked_by_scrub

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: drop bitwise from ScrubMap

Signed-off-by: Sage Weil <sage@redhat.com>

common/hobject: eliminate wonky compartors

Signed-off-by: Sage Weil <sage@redhat.com>

common/hobject: remove cmp_* comparators; add normal operators

Fix up callers.

Signed-off-by: Sage Weil <sage@redhat.com>

os: remove nibblewise sort order support

Signed-off-by: Sage Weil <sage@redhat.com>

osd: shutdown if sortbitwise flag is ever cleared

Note we do this only in activate_map so that we can (while booting)
skip past maps that don't have it set.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #13369 from idryomov/wip-doc-kernel-vers

doc: kernel client os-recommendations update

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13229 from trociny/wip-18335

rbd-nbd: check /sys/block/nbdX/size to ensure kernel mapped correctly

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

init-ceph.in: Make init-ceph work under FreeBSD for init-system

 -  FreeBSD find does not have printf option
 -  use the packaged getopt in /usr/local
 -  Only apply user-facts when user ceph is available

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #13292 from tchaikov/wip-script-sepia-bt-support-xenial

script/sepia_bt.sh: support xenial

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

osdc/Objecter: fix possible OSDSession leak on wrong connection

This is introduced by the newly added backoff logic.
Not sure if it will really happen, but just in case.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #12342 from liewegas/wip-backoff

osd,osdc: pg and osd-based backoff

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Affiliation updates

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

doc/dev/rados-client-protocol: describe some of rados protocol, including backoffs

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/thrash: vary backoff settings

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: backoffs on individual objects

Do these midway down do_op.  Reorder the scrub waitlist after the
degraded and unreadable waitlists.

Signed-off-by: Sage Weil <sage@redhat.com>

osdc/Objecter: respect backoffs

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: PG-wide backoffs

Issue at top of do_request.  Release on activation or peering
interval change.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: dispatch OSDBackoff messages into PG workqueue

Signed-off-by: Sage Weil <sage@redhat.com>

osd: split backoffs on PG split

Signed-off-by: Sage Weil <sage@redhat.com>

osd: add pg helpers for managing backoffs

Link Backoff into Session, PG.  Tear them down on session
reset.

Signed-off-by: Sage Weil <sage@redhat.com>

include/ceph_features: RADOS_BACKOFF feature bit

Signed-off-by: Sage Weil <sage@redhat.com>

osdc/Objecter: fix typo

Signed-off-by: Sage Weil <sage@redhat.com>

common/hobject: use hash field for hash<>

No reason to waste CPU recalculating a hash value!

Signed-off-by: Sage Weil <sage@redhat.com>

osd: add MOSDBackoff message type

Signed-off-by: Sage Weil <sage@redhat.com>

osd: move Session class out of OSD

Signed-off-by: Sage Weil <sage@redhat.com>

osdc/Objecter: attach OSDSession to Connection

This lets us avoid an rbtree lookup.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: clarify REQUIRE_LUMINOUS error message

This should be visible at the default log level, so users have some
hope of figuring out the problem.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

doc: ref cephfs docs from os-recommendations

...instead of having potentially conflicting kernel
version information.

Signed-off-by: John Spray <john.spray@redhat.com>

doc: update krbd kernel recommendations

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>