Merge pull request #3901 from dachary/wip-10922-ceph-deploy-documentation

doc: preparing a device implies activate

Reviewed-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Merge pull request #3907 from dachary/wip-test-delays

tests: increase the delay waiting for a cluster event

Reviewed-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Merge pull request #3914 from ktdreyer/wip-doc-rgw-apache-log-perms

doc: rm Apache log permissions instructions

Reviewed-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Merge pull request #3890 from XunleiGFS/fix_C_TwoContexts

osdc: fix a memory leak in C_TwoContexts

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #3913 from yuyuyu101/wip-async-fix-throttle-deadlock

Wip async fix throttle deadlock

Reviewed-by: Sage Weil <sage@redhat.com>

doc: rm Apache log permissions instructions

On all distros, Apache should be able to write to its own log directory.
If not, that is a severe distro bug that should be reported/fixed.

Remove the references to running "chown" on /var/log/httpd or
/var/log/apache2.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

doc/release-notes: update notes for v0.80.9

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3608 from guangyy/wip-rgw-quota-threads

rgw: make quota/gc thread configurable for starting

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #3795 from rzarzynski/wip-10938

rgw: improve support for X-Timestamp header of Swift API

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Event: Let external events ran without lock/unlock

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

AsyncConnection: Don't block process when throttle is full

If this connection is blocking on throttler and another worker thread
try to acquire this connection's lock, it will deadlock.

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #3668 from ceph/wip-compact-container

Optimize memory usage of MDS

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #3908 from gravitystorm/patch-2

fix typo

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: fix typo

Signed-off-by: Andy Allan <github@gravitystorm.co.uk>

Merge pull request #3906 from dachary/wip-mailmap

mailmap: David Zhang affiliation

Reviewed-by: Zhi (David) Zhang <zhangz@yahoo-inc.com>

tests: increase the delay waiting for a cluster event

In some cases the machine running tests may be slow enough that it takes
more than a minute for an OSD to come up. It only happens rarely and
changing the wait period from 60 seconds to 120 seconds. Is so slow
that it takes more than 2 minutes to bring an OSD up, chances are a lot
more will go wrong anyway.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

mailmap: Zhi (David) Zhang affiliation

And name normalization

Reviewed-by: Zhi (David) Zhang <zhangz@yahoo-inc.com>
Signed-off-by: Loic Dachary <ldachary@redhat.com>

doc: preparing a device implies activate

http://tracker.ceph.com/issues/10922 Fixes: #10922

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3900 from dachary/wip-recursive-submodules

autogen.sh: do not submodule --force if git does not support it

autogen.sh: do not submodule --force if git does not support it

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3895 from dachary/wip-recursive-submodules

doc,tests: force checkout of submodules

Merge pull request #3615 from ceph/wip-perf-watch

Performance watching CLI

doc,tests: force checkout of submodules

When updating submodules, always checkout even if the HEAD is the
desired commit hash (update --force) to avoid the following:

* a directory gmock exists in hammer
* a submodule gmock replaces the directory gmock in master
* checkout master + submodule update : gmock/.git is created
* checkout hammer : the gmock directory still contains the .git from
  master because it did not exist at the time and checkout won't
  remove untracked directories
* checkout master + submodule update : git rev-parse HEAD is
  at the desired commit although the content of the gmock directory
  is from hammer

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3892 from cernceph/cephdisk

ceph-disk: activate-[all|journal] should suppress

Backport: hammer, giant, firefly
Reviewed-by: Sage Weil <sage@redhat.com>

tests: sync must also be recursive

In case a submodule URL to a nested submodule changed

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3718 from ceph/wip-10649

mds: new OMAP storage for sessionmap

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

mds: new OMAP storage for sessionmap

Fixes: #10649
Signed-off-by: John Spray <john.spray@redhat.com>

osdc: add omap op perf counters

To avoid bloating the list of counters too much,
group the OMAP operations in to wr,rd,del

Signed-off-by: John Spray <john.spray@redhat.com>

ceph-disk: activate-[all|journal] should suppress

Make the suppress-activate <device> feature work for activate-all
and activate-journal.

Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>

osdc: fix a memory leak in C_TwoContexts

If an ObjectOperation op is cancelled, its destructor is
called and each Context object in out_handler is deleted.
A C_TwoContexts object can be one of these handlers. The
two contexts wrapped in C_TwoContexts must be deleted
as well.

Signed-off-by: Xiong Yiliang <xiongyiliang@xunlei.com>

Merge pull request #3891 from cernceph/doc

doc: osd map cache size is a count, not MB

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: osd map cache size is a count, not MB

Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>

rgw - make starting quota/gc threads configurable

Fixes: 11047
Backport: hammer

Signed-off-by: Guang Yang <yguang@yahoo-inc.com>

Merge pull request #3888 from dachary/wip-gmock-submodule

ceph needs gmock/gtest to be statically linked

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

ceph needs gmock/gtest to be statically linked

The ceph-test package depends on gmock and gtest and needs to statically
link them because packages are not configured to distribute the
corresponding shared library files.

Update the gmock submodule to the version that is configured with
noinst libraries that will be statically linked.

The gmock submodule is modified to reference the branch in which the
ceph specific changes have been done.

http://tracker.ceph.com/issues/11040 Fixes: #11040

Signed-off-by: Loic Dachary <loic@dachary.org>

test: add unit tests for daemonperf formatting

Signed-off-by: John Spray <john.spray@redhat.com>

ceph.in: add 'daemonperf' command

This is inspired by dstat and scripts/perf-watch.py, to
give a convenient live view of an interesting subset
of the performance counters from a Ceph daemon.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: fix type of some stats

These guys were marked as counters, but they were
updated with set(), not inc() -- so they should just
be normal u64s.

Signed-off-by: John Spray <john.spray@redhat.com>

mon: add nicknames to some performance counters

Signed-off-by: John Spray <john.spray@redhat.com>

osd: add nicks to some stats

This is a rough indication of what I find
useful.

Signed-off-by: John Spray <john.spray@redhat.com>

osdc: add perf counter nicknames

Signed-off-by: John Spray <john.spray@redhat.com>

mds: mark stats with nicknames

These are the stats that we would like to appear
in the new perf watch CLI mode.

Signed-off-by: John Spray <john.spray@redhat.com>

common: add optional perf counter nickname

This has two purposes:

* Identify counters that are useful in short/summary views
  of a daemon's performance (only these have a nick set)
* Provide shortened versions of names that are suitable
  for display in a text-mode columnar view.

Signed-off-by: John Spray <john.spray@redhat.com>

common: remove some spurious declarations

Maybe these were defined at some point in history...

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #3886 from ceph/wip-gmock-mirror

.gitmodule_mirrors: add gmock.git mirror

Reviewed-by: Loic Dachary <ldachary@redhat.com>

rgw: improve format of X-Timestamp on Swift objects.

Backport: hammer
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: add support for X-Timestamp on Swift containers.

Fixes: #10938
Backport: hammer
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

.gitmodule_mirrors: add gmock.git mirror

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3846 from ceph/wip-da-gtest-use-submodules

Use git submodules for gmock/gtest

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #3811 from tchaikov/fix-build-debug-xio

Fix build of `--with-debug --enable-xio`

Reviewed-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Merge pull request #3861 from guangyy/wip-11001

rgw: do not pre-fetch data for HEAD requests

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: do not pre-fetch data for HEAD requests

Backport: hammer

Signed-off-by: Guang Yang <yguang@yahoo-inc.com>

Merge pull request #3876 from ceph/wip-rpm-ceph-test-dependencies

#10989 ceph.spec.in: loosen ceph-test's dependencies

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

ceph.spec.in: loosen ceph-test's dependencies

In Debian, the ceph-test package can be installed with any version of
ceph-common.

Prior to this commit, in RHEL, we're much more strict about which
version of the dependencies we require. We depend directly on
librados2/librbd1/libcephfs1 instead of ceph-common, and we also require
the specific versions of these libraries to match the version of
ceph-test.

For testing Ceph, it is nice to have the ability to upgrade the
librados2/librbd1/libcephfs1 libraries on a host without having to also
upgrade the ceph-test package as well.

Remove the version number requirements, and change the dependencies from
librados2/librbd1/libcephfs1 to simply "ceph-common". That will make
/etc/ceph/ and /var/log/ceph present for the tests.

http://tracker.ceph.com/issues/10989 Refs: #10989

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #3804 from Ved-vampir/master

common: add perf counters description

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3869 from tchaikov/print-help-on-osd

ceph.in: print help on 'osd' or 'mon'

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ceph.in: print help on 'osd' or 'mon'

Fixes: #10888
Signed-off-by: Kefu Chai <kchai@redhat.com>

common: add perf counters description

Signed-off-by: Kiseleva Alyona <akiselyova@mirantis.com>

Merge pull request #3865 from ceph/wip-11011

mds: flush immediately in do_open_truncate

Merge pull request #3862 from ceph/wip-da-fix-python

Fix pep8 python issues

Reviewed-by: Kefu Chai <kchai@redhat.com>

mds: flush immediately in do_open_truncate

Previously truncating opens were horribly laggy
for clients when the system was otherwise idle,
as they would wait for the next tick() before
proceeding.

Fixes: #11011
Signed-off-by: John Spray <john.spray@redhat.com>

rados.py: fix pep8 E502, remove redundant '\' in brackets

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rados.py: fix pep E711

Fix for:
- E711 comparison to None should be 'if cond is None:'

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rados.py: fix pep8 indent issues

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rados.py: fix (white)space issues

Fixes for:
- E202 whitespace before '}'
- E221 multiple spaces before operator
- E223 tab before operator
- E226 missing whitespace around arithmetic operator
- E251 unexpected spaces around keyword / parameter equals
- E261 at least two spaces before inline comment
- W293 blank line contains whitespace

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rados.py: fix pep8 E111 indentation (spaces/tab)

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph_rest_api.py: fix pep8 indentation errors

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph_rest_api.py: fix pep8 whitespace issues

Fixes for:
- E226 missing whitespace around arithmetic operator
- E231 missing whitespace after ':' and ','
- E265 block comment should start with '#

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph_argparse.py: pep8 E721 don't compare types, use 'isinstance()'

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

python: fix pep8 'E713 test for membership should be 'not in'

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph_argparse.py: fix pep8 indentation errors

Fix for:
- E126 continuation line over-indented for hanging indent
- E128 continuation line under-indented for visual indent
- E129 visually indented line with same indent as next logical line
- E131 continuation line unaligned for hanging indent

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph_argparse.py: fix pep8 E502, remove redundant '\' in brackets

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph_argparse.py: fix pep8 whitespace issues

Fixes for:
- E231 missing whitespace after ':'
- E231 missing whitespace after ','
- E226 missing whitespace around arithmetic operator

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

cephfs.py: fix pep8 E127/E128 intended lines

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

cephfs.py: pep8 E225. add missing whitespace around operator

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

cephfs.py: fix pep8 E202 whitespaces

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

cephfs.py: fix inline comments

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rbd.py: remove superfluous-parens around if

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

add tox.ini for pep8 with some errors to be ignored

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rbd.py: fix pep8 E101/W191 wrong intended lines

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rbd.py: fix pep8 E128 under-indented lines

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rbd.py: fix pep8 E127 over-indented line

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

python: fix pep8 E302, add missing lines

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Merge pull request #3842 from dachary/wip-docker-build

tests: keep intermediate docker build layers

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #3856 from ceph/wip-10982

RFC Client: do not require successful remount when unmounting

Client: do not require successful remount when unmounting

Fixes: #10982
Signed-off-by: Greg Farnum <gfarnum@redhat.com>

doc/release-notes: add v0.92->0.93 upgrade note

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3831 from dachary/wip-qemu-iotests

Maipo should also use the local qemu clone

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

update some .gitignore files

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

submodules: --recursive needed for gtest in gmock

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

gmock: add git submodule

Add gmock submodule and point to ceph-release-1.7.0 branch.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

gmock: remove in-tree code copy

Step needed to replace by git submodules.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Merge pull request #3818 from rzarzynski/wip-10661

rgw: generate Content-Length in response for POST on Swift containter/object

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #3761 from rzarzynski/wip-10873

rgw: generate the "Date" HTTP header.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: Swift API. Complement the response to "show container details"

OpenStack Object Storage API v1 states that X-Container-Object-Count, X-Container-Bytes-Used and user-defined metadata headers should be included in a response.

Fixes: #10666
Backport: hammer
Reported-by: Ahmad Faheem <ahmad.faheem@ril.com>
Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Signed-off-by: Dmytro Iurchenko <diurchenko@mirantis.com>

Merge remote-tracking branch 'gh/hammer'

Merge pull request #3844 from ceph/wip-10965

Wip 10965

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3843 from ceph/wip-10978

rgw: don't overwrite bucket / object owner when setting acls

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3812 from ceph/wip-10958

librbd: flush pending AIO before resize

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rgw: don't overwrite bucket / object owner when setting acls

Fixes: #10978
Backport: hammer, firefly

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>