osd: tolerate sessionless con in fast dispatch path

We can now get a session cleared from a Connection at any time.  Change
the assert to an if in ms_fast_dispatch to cope.  It's pretty rare, but it
can happen, especially with delay injection.  In particular, a racing
thread can call mark_down() on us.

Fixes: #10209
Backport: giant
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2825 from ceph/wip-9921

SimpleMessenger: Pipe: do not block on takeover while holding global locks

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3046 from yuyuyu101/fix-kinetic

KnieticStore: add necessary header file

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2922 from ceph/wip-da-SCA-20141111

Coverity fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2937 from simon3z/master

common: do not unlock rwlock on destruction

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2788 from ceph/wip-warn-max-pg

mon: warn when there are too many pgs per osd

Reviewed-by: Joao Eduardo Luis <joao@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #2735 from ceph/wip-pg-stat

mon: make 'pg stat -f json' not pg dump; make JSON dump of floats floats and not strings

Reviewed-by: Samuel Just <sjust@redhat.com>

KnieticStore: add necessary header file

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #2855 from ceph/wip-9881

#9881 MDS flush journal asok command

Reviewed-by: Yan, Zheng <zyan@redhat.com>

mds: implement "flush journal" asok

Start a new segment and flush everything older
to the backing store.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: in segment expiry, use end instead of start

Previously we were always leaving the last log
segment un-expired from Journaler's POV, because
we were updating expire_pos to the *start* of the
expired journal segment instead of the *end*.

This was harmless, but cleaning it up makes it
easier to see what's going on during journal flushes
in testing.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: fix MDLog I/O callback without finisher

This could deadlock if another caller was waiting
on a write_head callback while holding mds_lock.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #3037 from stiopaa1/cleanup

added some consts, some cleanup

Reviewed-by: Loic Dachary <ldachary@redhat.com>

added consts, cleanup
Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #3019 from theanalyst/doc/typo/ceph-auth

doc: minor fix in ceph-authtool man page

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3023 from ktdreyer/rpm-rbd-replay-prep-conditional

rpm: unconditionally package rbd-replay-prep(8)

Reviewed-by: Adam Crume <adamcrume@gmail.com>

doc: minor fix in ceph-authtool man page

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #3032 from dachary/wip-osd-log-cosmetics

osd: log message content cosmetics

Reviewed-by: Sahid Orentino Ferdjaoui <sahid.ferdjaoui@redhat.com>

osd: log message content cosmetics

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #2769 from ceph/wip-9557

mds: verify backtrace when fetching dirfrag

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #3027 from majianpeng/fix1

Add "make check" dep python-virtualenv in deps.rpm.txt

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Add "make check" dep python-virtualenv in deps.rpm.txt

In fedora20, When 'make  check' and met those error messages:
/src/test/run-cli-tests: virtualenv not installed, skipping python-using
tests.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #3025 from majianpeng/fix1

Add isa-erasure yasm dep for deps.rpm.txt

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Add isa-erasure yasm dep for deps.rpm.txt

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

rpm: unconditionally package rbd-replay-prep(8)

The build system will conditionally build and install the
rbd-replay-prep utility based on the presence of lttng.

Commit 230c5b8e7f58e2e8be32c3f31c6abcc5bdabc5ca adjusted the RPM
packaging so that we only package and ship the rbd-replay-prep utility
and its man page when we're on a platform that has lttng.

The problem is that this is only half-correct. The build system
unconditionally installs the man page for rbd-replay-prep, regardless of
whether lttng was present or absent, and (by extension,) without regard
for whether we've built and installed the rbd-replay-prep binary or not.

We have two options: either the build system should be adjusted to
conditionally skip installing rbd-replay-prep(8), or else we should just
adjust the RPM packaging to always ship the man page. This commit does
the latter.

Backport: firefly
Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #3022 from leseb/fix-opts

Minor typos and trailing spaces

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Minor typos and trailing spaces

Fix typos and trailing spaces.

Signed-off-by: Sébastien Han <sebastien.han@enovance.com>

Merge pull request #3016 from yuyuyu101/fix-kinetic-support

KineticStore: Fix compile error and add _test_init

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

KineticStore: Fix compile error and add _test_init

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #2995 from nilamdyuti/wip-doc-ceph-disk

doc: Adds man page for ceph-disk utility.

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3021 from tchaikov/wip-fix-typos

doc: fix some typos in ObjectStore.h

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: fix some typos in ObjectStore.h

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

doc: Adds the updated man page for ceph-disk utility.

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

doc: Updates the man page for ceph-disk utility with some changes.

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

Merge pull request #3011 from stiopaa1/someConsts

added some consts

Reviewed-by: Loic Dachary <ldachary@redhat.com>

added some consts
Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

doc: Adds man page for ceph-disk utility.

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

doc: Removes ceph-deploy usage instances from ceph-disk man page.

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

Merge pull request #3003 from dachary/wip-10185-neon

arch: fix neon feature detection

Reviewed-by: Sahid Orentino Ferdjaoui <sahid.ferdjaoui@redhat.com>

arch: fix neon feature detection

The fread function returns the number of elements read, not the number
of bytes.

Introduced by 1cef8339f7a2a503327c20c014e696c91d36281e , after giant.

Add tests for protection against similar regressions.

http://tracker.ceph.com/issues/10185 Fixes: #10185

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3002 from thesues/fix-init

Remove pidfile and asok after stopping ceph

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Remove pidfile and asok after stopping ceph

Signed-off-by: Dongmao Zhang <deanraccoon@gmail.com>

Merge pull request #2872 from dachary/wip-10018-primary-erasure-code-hinfo

osd: erasure code deep scrub must not abort if hinfo is missing

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #2984 from stiopaa1/includeFix

remove unneeded include file

Reviewed-by: Sage Weil <sage@redhat.com>

doc: Updates man page for ceph-disk utility.

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

remove unneeded include file

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #2998 from theanalyst/doc/rgw-s3-bucket-loc

doc: rgw document s3 bucket location features.

Reviewed-by: John Wilkins <jowilkin@redhat.com>

doc: rgw document s3 bucket location features

Fixes: #10142
Reported-by: Neil Levine <nlevine@redhat.com>
Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

doc: Adds man page for ceph-disk utility.

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

Merge pull request #2992 from dachary/wip-10173-autogen

autogen.sh: git submodule sync

Reviewed-by: Sage Weil <sage@redhat.com>

autogen.sh: git submodule sync

It is not enough to git submodule update on a existing git clone: if the
URL of the repository has been changed, the

   git submodule sync

command must be called to synchronizes submodules' remote URL
configuration setting to the value specified in .gitmodules.

http://tracker.ceph.com/issues/10173 Fixes: #10173

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #2936 from ceph/wip-mailmap

Update Inktank attribution to point to Red Hat

Reviewed-by: Loic Dachary <ldachary@redhat.com>

mailmap: Update Warren Usui's attribution to Red Hat

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mailmap: Update Tyler Brekke's attribution to Red Hat

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mailmap: Update Tamil Muthamizhan's attribution to Red Hat

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mailmap: Update Sandon Van Ness' attribution to Red Hat

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mailmap: Update Samuel Just's attribution to Red Hat

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mailmap: Update Patrick McGarry's attribution to Red Hat

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mailmap: Update Noah Watkins' attribution to Red Hat

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mailmap: Update Neil Levine's attribution to Red Hat

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mailmap: Update Mark Nelson's attribution to Red Hat

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mailmap: Update Ken Dreyer's attribution to Red Hat

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mailmap: Update Josh Durgin's attribution to Red Hat

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mailmap: Update John Wilkins' attribution to Red Hat

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mailmap: Update Joao Luis' attribution to Red Hat

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mailmap: Update Ilya Dryomov's attribution to Red Hat

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mailmap: Update Dan Mick's attribution

Point his addresses at Red Hat now, but revert the Inktank->Redhat mappings
so we have more historical accuracy.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mailmap: Update Alfredo Deza's attribution to Red Hat

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mailmap: Update Greg Farnum's attribution to Red Hat

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge remote-tracking branch 'gh/next'

Merge pull request #2983 from tchaikov/wip-fix-lock-dep

lockdep: do not use $CEPH_LOCKDEP for g_lockdep

Reviewed-by: Sage Weil <sage@redhat.com>

lockdep: do not use $CEPH_LOCKDEP for g_lockdep

* a non-zero CEPH_LOCKDEP brings ceph down because g_lockdep_ceph_ctx
  is still being constructed when dout_impl() dereferences it.
* fix a typo in comment.
* remove dead code.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

rgw-admin: create subuser if needed when creating user

Fixes: #10103
Backport: firefly, giant
This turned up after fixing #9973. Earlier we also didn't create the
subuser in this case, but we didn't really read the subuser info when it
was authenticating. Now we do that as required, so we end up failing the
authentication. This only applies to cases where a subuser was created
using 'user create', and not the 'subuser create' command.

Reviewed-by: Sage Weil <sage@redhat.com>
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #2895 from yuyuyu101/wip-10022

AsyncMessenger: Fix several problems

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2980 from ceph/wip-10123

librbd: protect list_children from invalid child pool IoCtxs

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #2981 from yuyuyu101/quick-fix

GenericObjectMap: Avoid master branch compile error

GenericObjectMap: Fix typo

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #2966 from yuyuyu101/wip-10119

KeyValueStore: Fix EC+ KV OSDs crashing

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2978 from majianpeng/fix4

blkdev: using strncpy instead of strcpy.

Reviewed-by: Sage Weil <sage@redhat.com>

blkdev: using strncpy instead of strcpy.

Coverity Scan reported this bug:
> New defect(s) Reported-by: Coverity Scan Showing 1 of 1 defect(s)
> *** CID 1255369:  Copy into fixed size buffer  (STRING_OVERFLOW)
> /common/blkdev.cc: 34 in block_device_support_discard(const char *)()
> 28     bool block_device_support_discard(const char *devname)
> 29     {
> 30       bool can_trim = false;
> 31       char *p = strstr((char *)devname, "sd");
> 32       char name[32] = {0};
> 33
> >>>     CID 1255369:  Copy into fixed size buffer  (STRING_OVERFLOW)
> >>>     You might overrun the 32 byte fixed-size string "name" by
> >>>     copying "p" without checking the length.
> 34       strcpy(name, p);
> 35       for (unsigned int i = 0; i < strlen(name); i++) {
> 36         if(isdigit(name[i])) {
> 37           name[i] = 0;
> 38           break;
> 39         }
>

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #2974 from ceph/wip-sepia-update

Updated sepia hardware list.

Updated sepia hardware list.

Added missing details and additional list of hardware.

Signed-off-by: Sandon Van Ness <sandon@redhat.com>

Merge pull request #2752 from dachary/wip-9817-deep-scrub-start

osd: log when scrub,deep scrub, repair starts

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #2891 from XinzeChi/master

OSD: batch trim for pg_log

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2885 from ceph/wip-flush-snaps

osd/ReplicatedPG: flush snaps immediately

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #2973 from wonzhq/unused-parm

ReplicatedPG: remove unused parameter in function write_update_size_and_usage

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ReplicatedPG: remove unused parameter in function write_update_size_and_usage

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

mds: store backtrace for straydir

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: verify backtrace when fetching dirfrag

Fixes: #9557
Signed-off-by: Yan, Zheng <zyan@redhat.com>

KeyValueStore: Fix parse_header_key

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

osd: deep scrub must not abort if hinfo is missing

Instead it should set read_error.

http://tracker.ceph.com/issues/10018 Fixes: #10018

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

Merge pull request #2926 from ceph/wip-10052

ceph_test_rados_api_tier: fix whiteout cleanup (10052)

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #2967 from ceph/wip-10077

ceph_objectstore_tool shard feature handling

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #2971 from dachary/wip-mailmap

mailmap: Loic Dachary name normalization

mailmap: Loic Dachary name normalization

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #2962 from ceph/wip-9439

osd: Check filter ops for pgls and pgnls

Reviewed-by: Loic Dachary <ldachary@dachary.org>

Merge pull request #2969 from dachary/wip-erasure-code-documentation

doc: erasure-code developer documentation typo

Reviewed-by: Pavan Rallabhandi <Pavan.Rallabhandi@sandisk.com>

doc: erasure-code developer documentation typo

Reported-by: Pavan Rallabhandi <Pavan.Rallabhandi@sandisk.com>
Signed-off-by: Loic Dachary <ldachary@redhat.com>

StoreTest: Add scrub situation to tests

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

KeyValueStore: Add KEY_ENDING sign to the end of key

Keys stored in alphabetical order and need to follow ghobject_t comparison
rule. "generation" and "shard_id" are optional fields for object key, but
a default ghobject with UINT64_MAX generation(by default) will larger than
the same ghobject with other generation. GenericObjectMap rejects to store
generation if generation is UINT64_MAX in order to reduce too much words
in key. So we need to add a MAX sign to the end of key to make ordering
is same with ghobject's comparison rule.

For example:

_GHOBJTOSEQ_:1%e1ds0_head!D168A7E8!!1!!benchmark_last_metadata!head
_GHOBJTOSEQ_:1%e1ds0_head!D168A7E8!!1!!benchmark_last_metadata!head!78!0

The first key should larger than the second in ghobject_t ordering because
of generation. But the first key less than the second in GenericObjectMap.

_GHOBJTOSEQ_:1%e1ds0_head!D168A7E8!!1!!benchmark_last_metadata!head
_GHOBJTOSEQ_:1%e1ds0_head!D168A7E8!!1!!benchmark_last_metadata!head!78!0

After we add KEY_ENDING, '!' is (21) in hexadecimal:

_GHOBJTOSEQ_:1%e1ds0_head!D168A7E8!!1!!benchmark_last_metadata!head(21)78!0(FF)
_GHOBJTOSEQ_:1%e1ds0_head!D168A7E8!!1!!benchmark_last_metadata!head(FF)

Fix bug #10119
Signed-off-by: Haomai Wang <haomaiwang@gmail.com>