osdc/Objecter: force pg_command ops to ignore overlay

Signed-off-by: Sage Weil <sage@redhat.com>

osdc/Objecter: force pg_read ops to ignore cache overlay

pg_read is only used for PG listing and hit_set_{list,get}; these
operations can't and shouldn't consider the tiering overlay.

This makes the _calc_target behavior with the explicit pgid make sense;
otherwise, what would it mean to try to read pg x.1 from pool x and get
redirected to pg y.1 in pool y?

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMap: make is_acting_osd_shard an explicit spg_t check

Ensure that the ps value is < the pool pg_num.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMap: is_acting_osd_shard

Signed-off-by: Sage Weil <sage@redhat.com>

osd: drop osd_debug_drop_op_probability

This is unused and not terribly useful.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: move internal in-memory types to osd_internal_types.h

Things like ObjectContext and lock state that are internal to the OSD
do not need to be in osd_types and shared with other parts of the code
base.

Notably, this fixes the problem with OpRequest needing things from
osd_types.h (osd_reqid_t for starters).  Others to follow.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: no need to split op waiting lists

Clients are now expected to resend on split, and there is already an
interval change.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: explicitly enumerate ops we can dispatch

This prevents random messages from falling into and OpRequest and
dispatch_op().

Signed-off-by: Sage Weil <sage@redhat.com>

osd: remove MOSDPGMissing

Removed 7c414c5dab318a9212fd09f9637f241b1a2943f6 (pre-bobtail).

Signed-off-by: Sage Weil <sage@redhat.com>

osd: make all fast dispatch ops MOSDFastDispatchOp children

Define common get_spg() and get_map_epoch() methods.

Signed-off-by: Sage Weil <sage@redhat.com>

osdc/Objecter: populate both actual pgid and full has in MOSDOp

New clients need the actual pgid as well as the full hash (as part of the
target hobj).  Old clients only use the full hash value.  We need to pass
both to MOSDOp so it can encode based on the target features.

Signed-off-by: Sage Weil <sage@redhat.com>

messages/MOSDOp: take spg_t, not pg_t, and drop old ctor

Signed-off-by: Sage Weil <sage@redhat.com>

messages/MOSDOp: new encoding w/ actual pgid separate from hobject hash

New clients will see an actual pgid as well as a full has value in the
hobj.  Old clients will continue to see a single (full) hash value.

Signed-off-by: Sage Weil <sage@redhat.com>

messages/MOSDOp: add get_raw_pg()

Many current users expect a full hash value; make that explicit.

Signed-off-by: Sage Weil <sage@redhat.com>

messages/MOSDOp: remove unused reassert_version

Signed-off-by: Sage Weil <sage@redhat.com>

osdc/Objecter: remove reassert_version

We never populate this since we never get an ack.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMap: generalize map_to_pg

So we can do this without constructing an object_locator_t.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: make use of MOSDOp::get_hobj()

Prefer this to get_object_locator() whereever possible.

Signed-off-by: Sage Weil <sage@redhat.com>

message/MOSDOp: build native hobject_t

Drop unneeded snapid_t snapid and object_locator_t, which just duplicate
hobject_t fields.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: fix tracking of last_epoch_split

Note that it is only (currently) important that this value be accurate
on the current OSD since we only use this value (currently) to discard
ops sent before the split.  If we are getting the history from a different
OSD in the cluster that doesn't have an up to date value it doesn't matter
because that implies a primary change and also a client resend.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: discard ops from before the last split

New clients will resend.

Old clients will see a last_force_op_resend (now named
last_force_op_resend_preluminous in latest code) and resend.

We know this because we require that the monitors upgrade to luminous
before the OSDs, and the new mon code sets this field on split.

Signed-off-by: Sage Weil <sage@redhat.com>

osdc/Objecter: resend ops on pg split if osd has CEPH_FEATURE_RESEND_ON_SPLIT

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: make pre-luminous clients resend ops on split

Pre-luminous clients do not understand that a split PG forms a new
interval.  Make them resend ops to work around this.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: discard ops based on either new or old lfor and features

If the client has the new feature bit, use the new field; if they have the
older feature bit, use the old field.

Note that there is no change to the Objecter: last_force_op_resend is
still the "current" field that it should pay attention to.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: last_force_op_resend -> last_force_op_resend_preluminous

Rename the current last_force_op_resend for legacy clients, and add a new
one that only applies to new clients that have the new
CEPH_FEATURE_OSD_NEW_INTERVAL_ON_SPLIT feature.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: add set_last_force_op_resend() accessor and use it

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #13321 from liewegas/wip-kill-sortbitwise-harder

osd: kill sortbitwise

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #13303 from Jing-Scott/fix-swift-cannot-disable-object-versioning

rgw: fix swift cannot disable object versioning

@Jing-Scott  updated, addressing @rzarzynski's change request

Merge pull request #13393 from ddiss/fix_rgw_fcgi_include

rgw: set FCGI_INCLUDE_DIR for cephd_rgw_base

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13382 from xiexingguo/wip-counter-name

os/bluestore: fix wal-queue bytes-counter to keep pace with others

Reviewed-by: Sage Weil <sage@redhat.com>

rgw: set FCGI_INCLUDE_DIR for cephd_rgw_base

cephd_rgw_base build currently fails with fastcgi enabled:
--
In file included from /home/david/ceph/src/rgw/rgw_request.h:13:0,
from /home/david/ceph/src/rgw/rgw_main.cc:53:
/home/david/ceph/src/rgw/rgw_fcgi.h:8:21: fatal error: fcgiapp.h:
No such file or directory
^
--

This is despite the fact that fastcgi was detected and located at
configure time:
build/CMakeCache.txt:FCGI_INCLUDE_DIR:PATH=/usr/include/fastcgi

Fix this by ensuring that the cephd_rgw_base build target correctly uses
FCGI_INCLUDE_DIR.

Fixes: http://tracker.ceph.com/issues/18918
Signed-off-by: David Disseldorp <ddiss@suse.de>

Merge pull request #13392 from Adirl/zero_inline

msg/async/rdma: zero wqe inline

Reviewed-by: Haomai Wang <haomai@xsky.com>

msg/async/rdma: zero wqe inline

we dont use wqe inline

issue: none

Change-Id: Ib187eec3bc79f8caf040fbbfbf0a915186cd58ba
Signed-off-by: Adir Lev <adirl@mellanox.com>

rgw: fix swift cannot disable object versioning

we should be able to disable object verioning by removing its X-Versions-Location
metadata header by sending an empty key value. this description can be found at
No.8 in http://docs.openstack.org/user-guide/cli-swift-set-object-versions.html.

Fixes: http://tracker.ceph.com/issues/18852
Signed-off-by: Jing Wenjun <jingwenjun@cmss.chinamobile.com>

Merge pull request #13380 from tchaikov/wip-sepia-bt

script/sepia_bt.sh: no need to pass version and sha1 anymore

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #13368 from smithfarm/wip-doc-dev

doc: instructions for provisioning OpenStack VMs ad hoc

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #13367 from tchaikov/wip-qa-jewel-x-singleton

test: Thrasher: update pgp_num of all expanded pools if not yet

Reviewed-by: Sage Weil <sage@redhat.com>

script/sepia_bt.sh: no need to pass version and sha1 anymore

* try to figure out the working directory from the core dump's path
* add -C directory option, so user is able to specify the directory
  for the downloaded packages, and extracted files.

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: fix wal-queue bytes-counter to keep pace with others

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

test: Thrasher: restore changed options after done with thrash

Signed-off-by: Kefu Chai <kchai@redhat.com>

tests: Thrasher: extract _set_config() method

Signed-off-by: Kefu Chai <kchai@redhat.com>

tests: CephManager: add get_config() method

Signed-off-by: Kefu Chai <kchai@redhat.com>

test: Thrasher: update pgp_num of all expanded pools if not yet

otherwise wait_until_healthy will fail after timeout as seeing warning
like:

HEALTH_WARN pool cephfs_data pg_num 182 > pgp_num 172

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13378 from liewegas/wip-luminous

qa/rados/upgrade/jewel-x-singleton: run luminous.yaml at the end

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #13365 from xiexingguo/wip-backoff-refleak

osdc/Objecter: fix possible OSDSession leak on wrong connection

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13363 from jdurgin/wip-req-luminous-error

osd: clarify REQUIRE_LUMINOUS error message

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/rados/upgrade/jewel-x-singleton: run luminous.yaml at the end

renumber too

Signed-off-by: Sage Weil <sage@redhat.com>

doc: instructions for provisioning OpenStack VMs ad hoc

Signed-off-by: Nathan Cutler <ncutler@suse.com>

osd/PrimaryLogPG: unconditionally set PGNLS next to hobject

Drop nibblewise workaround.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: drop get_sort_bitwise() accessors, flag

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: always bitwise

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: drop bitwise arg to MissingLoc::rebuild()

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: make set_last_backfill() assume bitwise

Signed-off-by: Sage Weil <sage@redhat.com>

osd: BackfillInterval is always bitwise

Signed-off-by: Sage Weil <sage@redhat.com>

osd: fix add_source_info

Signed-off-by: Sage Weil <sage@redhat.com>

osd: fix write_blocked_by_scrub

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: drop bitwise from ScrubMap

Signed-off-by: Sage Weil <sage@redhat.com>

common/hobject: eliminate wonky compartors

Signed-off-by: Sage Weil <sage@redhat.com>

common/hobject: remove cmp_* comparators; add normal operators

Fix up callers.

Signed-off-by: Sage Weil <sage@redhat.com>

os: remove nibblewise sort order support

Signed-off-by: Sage Weil <sage@redhat.com>

osd: shutdown if sortbitwise flag is ever cleared

Note we do this only in activate_map so that we can (while booting)
skip past maps that don't have it set.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #13369 from idryomov/wip-doc-kernel-vers

doc: kernel client os-recommendations update

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13229 from trociny/wip-18335

rbd-nbd: check /sys/block/nbdX/size to ensure kernel mapped correctly

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13292 from tchaikov/wip-script-sepia-bt-support-xenial

script/sepia_bt.sh: support xenial

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

osdc/Objecter: fix possible OSDSession leak on wrong connection

This is introduced by the newly added backoff logic.
Not sure if it will really happen, but just in case.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #12342 from liewegas/wip-backoff

osd,osdc: pg and osd-based backoff

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc/dev/rados-client-protocol: describe some of rados protocol, including backoffs

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/thrash: vary backoff settings

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: backoffs on individual objects

Do these midway down do_op.  Reorder the scrub waitlist after the
degraded and unreadable waitlists.

Signed-off-by: Sage Weil <sage@redhat.com>

osdc/Objecter: respect backoffs

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: PG-wide backoffs

Issue at top of do_request.  Release on activation or peering
interval change.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: dispatch OSDBackoff messages into PG workqueue

Signed-off-by: Sage Weil <sage@redhat.com>

osd: split backoffs on PG split

Signed-off-by: Sage Weil <sage@redhat.com>

osd: add pg helpers for managing backoffs

Link Backoff into Session, PG.  Tear them down on session
reset.

Signed-off-by: Sage Weil <sage@redhat.com>

include/ceph_features: RADOS_BACKOFF feature bit

Signed-off-by: Sage Weil <sage@redhat.com>

osdc/Objecter: fix typo

Signed-off-by: Sage Weil <sage@redhat.com>

common/hobject: use hash field for hash<>

No reason to waste CPU recalculating a hash value!

Signed-off-by: Sage Weil <sage@redhat.com>

osd: add MOSDBackoff message type

Signed-off-by: Sage Weil <sage@redhat.com>

osd: move Session class out of OSD

Signed-off-by: Sage Weil <sage@redhat.com>

osdc/Objecter: attach OSDSession to Connection

This lets us avoid an rbtree lookup.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: clarify REQUIRE_LUMINOUS error message

This should be visible at the default log level, so users have some
hope of figuring out the problem.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

doc: ref cephfs docs from os-recommendations

...instead of having potentially conflicting kernel
version information.

Signed-off-by: John Spray <john.spray@redhat.com>

doc: update krbd kernel recommendations

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #13249 from liewegas/wip-mon-smooth

mon: smooth io/recovery stats over longer period

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13264 from tchaikov/wip-print-failed-test-name

qa/workunits/rados/test.sh: print test name when it fails

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #13362 from adamemerson/wip-oh-yeah

test: Fix mismatched sign comparison in histogram test

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13344 from gregsfortytwo/wip-osd-discussion-docs

Wip osd discussion docs

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #13342 from athanatos/wip-17831-18583-18809

osd/: don't leak context for Blessed*Context or RecoveryQueueAsync

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

test: Fix mismatched sign comparison in histogram test

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #13137 from jcsp/wip-18661

qa: fix race in Mount.open_background

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #13099 from jcsp/wip-18663

qa/tasks: force umount during kclient teardown

Merge pull request #13281 from theanalyst/release/v12.0.0

doc: v12.0.0 release notes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13302 from YankunLi/lyk_clean_code

rgw: clean redundant code

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13304 from majianpeng/async-tracing

msg/async: cleanup code.

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #13329 from Adirl/fix_febe

msg: Fix calls to Messenger::create with new parameter

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #13351 from tchaikov/wip-init-ceph-newline

init-ceph: print trailing n in "status" output

Reviewed-by: Nathan Cutler <nculter@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13356 from rubenk/rgw-docs-fix-typo

doc: Fix typo and grammar in RGW config reference

Reviewed-by: Sage Weil <sage@redhat.com>

qa/workunits/rados/test.sh: print test name when it fails

we have

2017-02-04T16:15:46.090 INFO:tasks.workunit.client.0.mira032.stdout:error in 22088
2017-02-04T16:15:46.092 INFO:tasks.workunit.client.0.mira032.stderr:bash: line 1: 22092 Alarm clock             ceph_test_rados_api_aio 2>&1
2017-02-04T16:15:46.096 INFO:tasks.workunit.client.0.mira032.stderr:     22093 Done                    | tee ceph_test_rados_api_aio.log
2017-02-04T16:15:46.099 INFO:tasks.workunit.client.0.mira032.stderr:     22094 Done                    | sed "s/^/                  api_aio: /"
2017-02-04T16:15:46.102 INFO:tasks.workunit.client.0.mira032.stderr:+

if a unittest in rados/test.sh fails in teuthology.log, but it would
be desirable to have the failed test name in the line of "error in
22088".

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13258 from trociny/wip-18819

common: possible lockdep false alarm for ThreadPool lock

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13287 from guihecheng/rbd-fix

librbd: fix segfault on EOPNOTSUPP returned while fetching snapshot timestamp

Reviewed-by: Jason Dillaman <dillaman@redhat.com>