rgw: update keystone cache with token info

Fixes: #11125
Backport: hammer, firefly

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #3872 from dachary/wip-7334-partx-test

ceph-disk: test activate on the second journal

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #4014 from ceph/wip-mon-doc

doc/rados/operations/add-or-rm-mons: revise doc a bit to be less confusing

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3922 from ceph/wip-11053

mds: fix assertion caused by system clock backwards

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #3752 from ceph/wip-10744

mds: give up replicas of a stopping mds's stuff

Reviewed-by: Yan, Zheng <zyan@redhat.com>

mds: give up replicas of a stopping mds's stuff

In order for an MDS to make it through stopping when
it had some strays, the other ranks that serviced
the migrate_stray renames must ensure that they
give up any cache objects that belonged to
the stopping MDS, so that the stopping MDS
can finish emptying its cache.

Fixes: #10744
Signed-off-by: John Spray <john.spray@redhat.com>

doc/rados/operations/add-or-rm-mons: revise doc a bit to be less confusing

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #4002 from zhicheng/master

fix doc/dev/network-protocol.rst typo

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4007 from dachary/wip-warnings

fix a few minor compilation warnings

Reviewed-by: Kefu Chai <kchai@redhat.com>

fix a few minor compilation warnings

Signed-off-by: Loic Dachary <ldachary@redhat.com>

fix doc/dev/network-protocol.rst typo

Signed-off-by: Zhicheng Wei <zhicheng@opensourceforge.net>

Merge pull request #3989 from javacruft/wip/fixup-cycles-crossarch

Add support for PPC architecture, provide fallback

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4006 from dachary/wip-documentation

doc: mark the ascii schema as a literal block

Reviewed-by: Yann Dupont <yann@objoo.org>

doc: mark the ascii schema as a literal block

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4005 from dachary/wip-releases

doc: release timeline

Reviewed-by: Yann Dupont <yann@objoo.org>

Merge pull request #3990 from zaitcev/first_contrib

Specify the actual struct used in protocol handshake

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3992 from ceph/wip-package-perf-objectstore

packaging: include ceph_perf_objectstore

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Dan Mick <dmick@redhat.com>

doc: release timeline

Signed-off-by: Loic Dachary <ldachary@redhat.com>

packaging: include ceph_perf_objectstore

The /usr/bin/ceph_perf_objectstore file is installed by default. Prior
to this commit it was missing from the packaging. This caused the RPM to
fail to build in mock.

Add ceph_perf_objectstore to the "ceph-test" RPM and Debian package.

If we end up developing further ceph_perf_* utilities, it would make
sense to glob them all with a wildcard, similar to what we are doing
with all the ceph_test_* utilities in ceph-test.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Specify the actual struct used in protocol handshake

Maybe it was sockaddr_storage in an older version, but it is
definitely entity_addr_t in "ceph v027".

Signed-off-by: Pete Zaitcev <zaitcev@redhat.com>

Add support for PPC architecture, provide fallback

Add high precision cpu cycles support for powerpc and powerpc64.

Provide a fallback for other architectures and warn during
compilation.

Signed-off-by: James Page <james.page@ubuntu.com>

Merge pull request #3957 from xinxinsh/wip-cleanup

os/{Level,Rocks}DBStore: cleanup in_prefix function

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #3985 from fmarsylle/master

doc: s/that will called every time/that will be called every time/

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #3986 from Al3xis-N/master

doc: How to generate an object corpus : should be a subsection

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: How to generate an object corpus : should be a subsection

Fixes: #11099
Signed-off-by: Alexis Normand <n.al3xis@gmail.com>

Merge pull request #3987 from RobinDehu/master

doc: extra \ in CEPH\_AUTH\_UNKNOWN

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: s/that will called every time/that will be called every time/

fixes: #11098

Signed-off-by: florian marsylle <florian.marsylle@hotmail.fr>

doc: extra \ in CEPH\_AUTH\_UNKNOWN

Suppress all \

fixes: #11097

Signed-off-by: DEHU Robin <robindehu@gmail.com>

Merge pull request #3979 from ceph/wip-rbd-map-options

rbd: default map options, new options, misc

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rbd: fix rw option skipping logic

We slip in an extraneous comma if { "rw": "rw" } happens to be the
first map_options pair:

    # ./rbd map -o rw,share foo
    /dev/rbd0

    -> 127.0.0.1:6789 name=admin,key=client.admin,,share rbd foo -

The kernel mount options parser can handle it, but fix it nonetheless.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

rbd: recognize cephx_require_signatures and tcp_nodelay options

Support for these went into 3.19 and 4.0 kernels respectively.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

rbd: add rbd default map options config option

Allow supplying rbd map -o / --options through 'rbd default map
options' config option.  Options specified on the command line take
precedence on a per-option basis, i.e. default and cli options are
merged with a preference for the latter.  For example:

    # grep 'rbd default map options' ceph.conf
            rbd default map options = "ro,nocrc"
    # ./rbd map -o noshare,rw foo
    /dev/rbd0

results in a rw mapping with nocrc,noshare.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

rbd: drop redefining map option warning

For example:

    # ./rbd map -o share,noshare foo
    rbd: warning: redefining map option share: 'share' -> 'noshare'
    /dev/rbd0

mount(8) doesn't do this and it's a bit silly to begin with.  Drop this
in preparation for adding rbd default map options config option.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

rbd: regenerate rbd(8) man page

merge-diff and status commands, --image-features and --image-shared
options, other stuff.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #3924 from ktdreyer/wip-doc-rm-old-ceph-deploy-reference

rm old ceph-deploy reference

Reviewed-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Merge pull request #3965 from linuxbox2/xio-fixes

xio: Xio fixes

Reviewed-by: Kefu Chai <kchai@redhat.com>

Fix XioLoopbackConnection Lifecycle.

Since XioLoopbackConnection is a RefCountedObject, it can't be
an expanded member of XioMessenger.

Fixes cleanup/shutdown errors.

Signed-off-by: Matt Benjamin <matt@cohortfs.com>

Fix CMake build w/o RADOSGW.

Just fixes linkage of dencoder in test.  Fix the false assumption
that cls is only used if WITH_RADOSGW.

Signed-off-by: Matt Benjamin <matt@cohortfs.com>

Merge pull request #3625 from majianpeng/objectcacher

Bugs fix about ObjectCacher

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #3881 from ceph/wip-da-SCA-20150304

SCA fixes

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-disk: more robust parted output parser

In some cases, depending on the implementation or the operating system,

   parted --machine -- /dev/sdh print

may contain empty lines. The current parsing code is fragile and highly
depends on output details. Replace it with code that basically does the
same sanity checks (output not empty, existence of units, existence of
the dev entry) but handles the entire output instead of checking line by
line.

http://tracker.ceph.com/issues/10983 Fixes: #10983

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tools/rados/rados.cc: init some vars in constructor

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw_rados: don't assign value to parameter

Fix for cppcheck issue:
Assignment of function parameter has no effect outside the function.

Remove assignment to need_to_wait before exit.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

src/msg/xio/*: reduce scope of some vars

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw_quota.cc: init variables in constructor

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

TestErasureCodeShec_thread.cc: reduce scope of var

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ObjectStoreTransactionBenchmark.cc: reduce scope of var

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

objectstore/store_test.cc: reduce scope of variable

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw_rados.cc: reduce scope of variable

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw_rados.cc: remove unused string vars

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

librbd/internal.cc: reduce scope of some variables

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

librbd/ObjectMap.cc: reduce scope of variable, prevent redef

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ErasureCodeShec::minimum_to_decode: reduce scope of variables

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_op.cc: fix potential null-deref in strcmp()

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw_main.cc: fix null-deref in case get_storage() fails

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mds/MDS.cc: add missing asserts to check return values

Change existing asserts in set_up_admin_socket() to
check (VAR == 0).

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

TestErasureCodeShec_all.cc: fix sprintf specifier

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw: use static_cast instead of c-style cast

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

osd/ReplicatedPG.cc: use static_cast instead of c-style cast

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

TestErasureCodeShec_thread.cc: use static_cast instead of c-style

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

itest/msgr/test_msgr.cc: prefer ++operator for non-primitive iter

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

tests: ceph-disk.sh /dev/disk/by-partuuid and /dev/loop fixes

On CentOS 6.X and RHEL 6.X:

* /dev/disk/by-partuuid is not updated, support files must be installed
  to create them when udev notices a partition modification.

* /dev/loop is not configured to handle partition tables, the default
  2.6.32 kernel must be configured with loop.max_part=16

Sanity checks are added to verify the above, attempt to fix it and fail
if it cannot be fixed or if a machine reboot is necessary.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3666 from xroche/master

Added ceph_flock() to libcephfs.

Merge pull request #3920 from rajukv/xio-bugfix

XIO: Handle requeue case of XIO messages

Reviewed-by: Kefu Chai <kchai@redhat.com>

XIO: Handle requeue case of XIO messages

If we are not able to send the XIO message using xio_send_msg(),
remove the XIO message from the send Q, before queuing it to the resend
Q. Otherwise, boost will generate a assert.

Signed-off-by: Raju Kurunkad <raju.kurunkad@sandisk.com>

cleanup in_prefix function

Signed-off-by: xinxin shu <xinxin.shu@intel.com>

doc/release-notes: v0.80.8 and .9 changelogs

Signed-off-by: Sage Weil <sage@redhat.com>

tests: ceph-disk paritition creation and partition table in use

The kernel partition table notification (via partprobe, partx, etc.) may
behave differently when a partition is in use. Add a test case that
checks a new journal partition shows as expected, even if another
journal partition is in use.

http://tracker.ceph.com/issues/10987 Refs: #10987

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph-disk.sh test for second journal partition

Create an OSD with data on a disk, journal on another
This will add a new partition to $journal, the previous
one will remain.

Add 200MB to the file supporting the pseudo disk exposed via the loop
device: as we create more OSDs, more 100MB partitions will be created
for journals.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph-disk.sh cosmetic changes and reduced verbosity

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: do not use --journal-dio=true

Direct IO for the journal may cause subtle errors while running the
tests withing a container. Disable Direct IO for all ceph-disk.sh tests.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph-disk.sh may use uuidgen without PATH

Store the absolute path in a variable to find uuidgen even if PATH is
empty.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph-disk.sh can be confused if there are two OSDs

Replace the fragile extaction of the whoami file with an explicit uuid
set for a given OSD. It can conveniently be translated back into an osd
id with ceph osd create uuid if needed.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: teardown on ceph-disk error

The activate_dev error must kill all process with teardown before
attempting to umount the devices when an error happens. Otherwise the
device fails to be unmounted and the loop devices are never freed.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3871 from dachary/wip-install-deps

tests: install-deps.sh install EPEL and RHEL Optional

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

rm old ceph-deploy reference

There's no need to refer to this old version of ceph-deploy. v1.1.3 is
circa June 2013, and since that time, we have shipped much newer version
of ceph-deploy (1.5.22 at the time of this writing). We should simply
state the behavior of modern ceph-deploys here.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #3887 from stiopaa1/feature10891

mon: add "osd pool get $pool all" command

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge remote-tracking branch 'gh/hammer'

Conflicts:
src/gmock

mds: fix assertion caused by system clock backwards

Fixes: #11053
Signed-off-by: Yan, Zheng <zyan@redhat.com>

tests: install-deps.sh install EPEL and RHEL Optional

On CentOS and RHEL, some dependencies come from EPEL.

Always enable RHEL Optional repository for RHEL as packages such as
packages such as libatomic_ops-devel or libedit-devel are only found
there in RHEL 6.5.

http://tracker.ceph.com/issues/11061 Refs: #11061

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3901 from dachary/wip-10922-ceph-deploy-documentation

doc: preparing a device implies activate

Reviewed-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Merge pull request #3907 from dachary/wip-test-delays

tests: increase the delay waiting for a cluster event

Reviewed-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Merge pull request #3914 from ktdreyer/wip-doc-rgw-apache-log-perms

doc: rm Apache log permissions instructions

Reviewed-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Merge pull request #3917 from dachary/wip-rhel6-doc

doc: rhel6.5 is rhel6

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #3890 from XunleiGFS/fix_C_TwoContexts

osdc: fix a memory leak in C_TwoContexts

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #3845 from ceph/wip-wn-rgw-hammer

Wip wn rgw hammer

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3913 from yuyuyu101/wip-async-fix-throttle-deadlock

Wip async fix throttle deadlock

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3911 from athanatos/wip-11057

ReplicatedPG: block write on degraded object if there are waiters

Reviewed-by: Sage Weil <sage@redhat.com>

doc: rhel6.5 is rhel6

RHEL 6.5 is actually  http://ceph.com/rpm-giant/rhel6/

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3909 from dachary/wip-11068-debian-jessie

deb: add zlib1g-dev to Build-Depends for Debian/jessie

Reviewed-by: Yann Dupont <yann@objoo.org>

mon/OSDMonitor: refactor and new key "all" for osd pool get command

This will only output all the values applicable to a given type of pool.
So for example for a pool that is not a tier pool values like HIT_SET_TYPE,
HIT_SET_PERIOD, HIT_SET_COUNT etc. will be ignored.

Fixes: #10891
Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

doc: rm Apache log permissions instructions

On all distros, Apache should be able to write to its own log directory.
If not, that is a severe distro bug that should be reported/fixed.

Remove the references to running "chown" on /var/log/httpd or
/var/log/apache2.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

doc/release-notes: update notes for v0.80.9

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3608 from guangyy/wip-rgw-quota-threads

rgw: make quota/gc thread configurable for starting

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #3795 from rzarzynski/wip-10938

rgw: improve support for X-Timestamp header of Swift API

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Event: Let external events ran without lock/unlock

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

AsyncConnection: Don't block process when throttle is full

If this connection is blocking on throttler and another worker thread
try to acquire this connection's lock, it will deadlock.

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #3668 from ceph/wip-compact-container

Optimize memory usage of MDS

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #3908 from gravitystorm/patch-2

fix typo

Reviewed-by: Loic Dachary <ldachary@redhat.com>

deb: add zlib1g-dev to Build-Depends for Debian/jessie

The zlib1g-dev is installed indirectly for Ubuntu 12.04 or Ubuntu 14.04
but it is only suggested in Debian/jessie. Adding it to the
Build-depends is redundant and harmless for Ubuntu and resolves the
missing dependency for Debian.

http://tracker.ceph.com/issues/11068 Fixes: #11068

Signed-off-by: Loic Dachary <ldachary@redhat.com>