test: add unit tests for daemonperf formatting

Signed-off-by: John Spray <john.spray@redhat.com>

ceph.in: add 'daemonperf' command

This is inspired by dstat and scripts/perf-watch.py, to
give a convenient live view of an interesting subset
of the performance counters from a Ceph daemon.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: fix type of some stats

These guys were marked as counters, but they were
updated with set(), not inc() -- so they should just
be normal u64s.

Signed-off-by: John Spray <john.spray@redhat.com>

mon: add nicknames to some performance counters

Signed-off-by: John Spray <john.spray@redhat.com>

osd: add nicks to some stats

This is a rough indication of what I find
useful.

Signed-off-by: John Spray <john.spray@redhat.com>

osdc: add perf counter nicknames

Signed-off-by: John Spray <john.spray@redhat.com>

mds: mark stats with nicknames

These are the stats that we would like to appear
in the new perf watch CLI mode.

Signed-off-by: John Spray <john.spray@redhat.com>

common: add optional perf counter nickname

This has two purposes:

* Identify counters that are useful in short/summary views
  of a daemon's performance (only these have a nick set)
* Provide shortened versions of names that are suitable
  for display in a text-mode columnar view.

Signed-off-by: John Spray <john.spray@redhat.com>

common: remove some spurious declarations

Maybe these were defined at some point in history...

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #3846 from ceph/wip-da-gtest-use-submodules

Use git submodules for gmock/gtest

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #3811 from tchaikov/fix-build-debug-xio

Fix build of `--with-debug --enable-xio`

Reviewed-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Merge pull request #3861 from guangyy/wip-11001

rgw: do not pre-fetch data for HEAD requests

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: do not pre-fetch data for HEAD requests

Backport: hammer

Signed-off-by: Guang Yang <yguang@yahoo-inc.com>

Merge pull request #3876 from ceph/wip-rpm-ceph-test-dependencies

#10989 ceph.spec.in: loosen ceph-test's dependencies

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

ceph.spec.in: loosen ceph-test's dependencies

In Debian, the ceph-test package can be installed with any version of
ceph-common.

Prior to this commit, in RHEL, we're much more strict about which
version of the dependencies we require. We depend directly on
librados2/librbd1/libcephfs1 instead of ceph-common, and we also require
the specific versions of these libraries to match the version of
ceph-test.

For testing Ceph, it is nice to have the ability to upgrade the
librados2/librbd1/libcephfs1 libraries on a host without having to also
upgrade the ceph-test package as well.

Remove the version number requirements, and change the dependencies from
librados2/librbd1/libcephfs1 to simply "ceph-common". That will make
/etc/ceph/ and /var/log/ceph present for the tests.

http://tracker.ceph.com/issues/10989 Refs: #10989

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #3804 from Ved-vampir/master

common: add perf counters description

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3869 from tchaikov/print-help-on-osd

ceph.in: print help on 'osd' or 'mon'

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ceph.in: print help on 'osd' or 'mon'

Fixes: #10888
Signed-off-by: Kefu Chai <kchai@redhat.com>

common: add perf counters description

Signed-off-by: Kiseleva Alyona <akiselyova@mirantis.com>

Merge pull request #3865 from ceph/wip-11011

mds: flush immediately in do_open_truncate

Merge pull request #3862 from ceph/wip-da-fix-python

Fix pep8 python issues

Reviewed-by: Kefu Chai <kchai@redhat.com>

mds: flush immediately in do_open_truncate

Previously truncating opens were horribly laggy
for clients when the system was otherwise idle,
as they would wait for the next tick() before
proceeding.

Fixes: #11011
Signed-off-by: John Spray <john.spray@redhat.com>

rados.py: fix pep8 E502, remove redundant '\' in brackets

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rados.py: fix pep E711

Fix for:
- E711 comparison to None should be 'if cond is None:'

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rados.py: fix pep8 indent issues

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rados.py: fix (white)space issues

Fixes for:
- E202 whitespace before '}'
- E221 multiple spaces before operator
- E223 tab before operator
- E226 missing whitespace around arithmetic operator
- E251 unexpected spaces around keyword / parameter equals
- E261 at least two spaces before inline comment
- W293 blank line contains whitespace

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rados.py: fix pep8 E111 indentation (spaces/tab)

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph_rest_api.py: fix pep8 indentation errors

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph_rest_api.py: fix pep8 whitespace issues

Fixes for:
- E226 missing whitespace around arithmetic operator
- E231 missing whitespace after ':' and ','
- E265 block comment should start with '#

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph_argparse.py: pep8 E721 don't compare types, use 'isinstance()'

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

python: fix pep8 'E713 test for membership should be 'not in'

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph_argparse.py: fix pep8 indentation errors

Fix for:
- E126 continuation line over-indented for hanging indent
- E128 continuation line under-indented for visual indent
- E129 visually indented line with same indent as next logical line
- E131 continuation line unaligned for hanging indent

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph_argparse.py: fix pep8 E502, remove redundant '\' in brackets

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph_argparse.py: fix pep8 whitespace issues

Fixes for:
- E231 missing whitespace after ':'
- E231 missing whitespace after ','
- E226 missing whitespace around arithmetic operator

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

cephfs.py: fix pep8 E127/E128 intended lines

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

cephfs.py: pep8 E225. add missing whitespace around operator

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

cephfs.py: fix pep8 E202 whitespaces

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

cephfs.py: fix inline comments

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rbd.py: remove superfluous-parens around if

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

add tox.ini for pep8 with some errors to be ignored

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rbd.py: fix pep8 E101/W191 wrong intended lines

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rbd.py: fix pep8 E128 under-indented lines

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rbd.py: fix pep8 E127 over-indented line

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

python: fix pep8 E302, add missing lines

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Merge pull request #3842 from dachary/wip-docker-build

tests: keep intermediate docker build layers

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #3856 from ceph/wip-10982

RFC Client: do not require successful remount when unmounting

Client: do not require successful remount when unmounting

Fixes: #10982
Signed-off-by: Greg Farnum <gfarnum@redhat.com>

doc/release-notes: add v0.92->0.93 upgrade note

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3831 from dachary/wip-qemu-iotests

Maipo should also use the local qemu clone

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

update some .gitignore files

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

submodules: --recursive needed for gtest in gmock

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

gmock: add git submodule

Add gmock submodule and point to ceph-release-1.7.0 branch.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

gmock: remove in-tree code copy

Step needed to replace by git submodules.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Merge pull request #3818 from rzarzynski/wip-10661

rgw: generate Content-Length in response for POST on Swift containter/object

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #3761 from rzarzynski/wip-10873

rgw: generate the "Date" HTTP header.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: Swift API. Complement the response to "show container details"

OpenStack Object Storage API v1 states that X-Container-Object-Count, X-Container-Bytes-Used and user-defined metadata headers should be included in a response.

Fixes: #10666
Backport: hammer
Reported-by: Ahmad Faheem <ahmad.faheem@ril.com>
Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Signed-off-by: Dmytro Iurchenko <diurchenko@mirantis.com>

Merge remote-tracking branch 'gh/hammer'

Merge pull request #3844 from ceph/wip-10965

Wip 10965

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3843 from ceph/wip-10978

rgw: don't overwrite bucket / object owner when setting acls

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3812 from ceph/wip-10958

librbd: flush pending AIO before resize

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rgw: don't overwrite bucket / object owner when setting acls

Fixes: #10978
Backport: hammer, firefly

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

tests: keep intermediate docker build layers

Because the docker devmapper backend has an intermittent bug

   https://github.com/docker/docker/issues/4036

the build of the image will sometime fail. It will eventually succeed if
the command is repeated enough times. Not removing the intermediate
layers helps speed up the operation: when repeating the command, only
the layer that failed because of the above bug is rebuilt.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

librbd: flush pending AIO after acquiring lock

There was a potential race condition between a delayed AIO
operation waiting on acquiring a lock and a snap_create
flushing all pending IO.  Since snap_create owned md_lock, the
delayed AIO would not be allowed to complete -- deadlocking the
flush.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: hold snap_lock between clipping IO and registering AIO

In the case where concurrent IO is occurring when a trim resize
operation is initiated, hold the snap_lock between clipping the
IO operation and registering the pending op.  That allows the
resize state machine to properly flush all operations issued
before the clip region was updated.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #3839 from dachary/wip-documentation

doc: ext4 has a journal

Reviewed-by: Simon Guinot <simon.guinot@sequanux.org>

doc: ext4 has a journal

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3834 from linuxbox2/xio-build

cmake:  build fixes

Reviewed-by: Loic Dachary <ldachary@redhat.com>

cmake:  build fixes

* librbd
* test/ceph_dencoder

Signed-off-by: Matt Benjamin <matt@cohortfs.com>

rgw: update makefile to enable civetweb config

Fixes: #10965
Backport: hammer, firefly

Civetweb compilation now includes conf header to enable ipv6.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

civetweb: update submodule

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #3833 from yuyuyu101/temp

TestAsyncDriver: Fix typo in ceph_test_async_driver

TestAsyncDriver: Fix typo in ceph_test_async_driver

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #3832 from tchaikov/fix-doc

doc/install/manual-deployment: fix osd install doc

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc/install/manual-deployment: fix osd install doc

* should be using /var/lib/ceph/osd/{cluster-name}-{osd-num}/sysvinit
  instead of var/lib/ceph/osd/{cluster-name}-{hostname}/sysvinit, thanks
  to Kyle Hutson <kylehutson@ksu.edu>

Fixes: #10957
Signed-off-by: Kefu Chai <kchai@redhat.com>

Maipo should also use the local qemu clone

http://tracker.ceph.com/issues/10975 Fixes: #10975

Signed-off-by: Loic Dachary <loic@dachary.org>

doc/release-notes: fix Takeshi's name

Signed-off-by: Sage Weil <sage@redhat.com>

Merge branch 'hammer' of jenkins.front.sepia.ceph.com:ceph/ceph into hammer

doc/release-notes: fix typo

Signed-off-by: Sage Weil <sage@redhat.com>

doc/release-notes: v0.80.9 firefly

Signed-off-by: Sage Weil <sage@redhat.com>

doc/release-notes: v0.93 release notes

Signed-off-by: Sage Weil <sage@redhat.com>

librbd: moved flush / cache invalidate to resize state machine

Keep async_resize truly async by moving flush and invalidate cache
operations to individual states within the resize state machine.

Fixes: #10958
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: add AIO version of invalidate_cache

Break the existing synchronous behavior into pre- and post-
flush routines using the existing logic.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #3825 from ceph/wip-hammer-gplv2-text

Add GPLv2 text file

Reviewed-by: Loic Dachary <ldachary@redhat.com>

0.93

Add GPLv2 text file

Most of the ceph tree is LGPLv2.1, but there are some files that are
under the full GPLv2.

Add a copy of the GNU General Public License (version 2) to the
distribution. This file was copied verbatim from
https://www.gnu.org/licenses/gpl-2.0.txt

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

rgw: enforce Content-Length in response for POST on Swift cont/obj.

Fixes: #10661
Backport: hammer
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: generate the "Date" HTTP header for civetweb.

Fixes: #10873
Backport: hammer
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

librbd: C_SaferCond memory leak

Unlike the other Context derived classes, C_SaferCond is not
a suicidal object which deletes itself.  Swap heap allocations
of C_SaferCond to stack-based allocations as a result.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #3814 from tchaikov/fix-doc

doc/install/manual-deployment: s/mon/osd/ for osd install doc

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3817 from tchaikov/misc-cosmetic-changes

Misc cosmetic changes

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3816 from fghaas/master

doc: trivial: fix missing newlines in rbd man page

Reviewed-by: Kefu Chai <kchai@redhat.com>

add more constness

Signed-off-by: Kefu Chai <kchai@redhat.com>

OSD: mark internal methods as `private`

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: trivial: fix missing newlines in rbd man page

As it were, the order, stripe_unit and stripe_count options were
followed by descriptions rendered in the same monospace font as the
options themselves. Insert newlines so they are rendered as regular
text, as was the original intention.

Signed-off-by: Florian Haas <florian@hastexo.com>

fix unmatched struct versus class warnings

these warnings come from clang++

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/config: move config_obs_t's dtor back

so one can find the impl of config_obs_t at the expected place.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/install/manual-deployment: s/mon/osd/ for osd install doc

Fixes: #10957
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #3797 from yuyuyu101/wip-reset-check

Messenger: Fix incorrect connect_seq increase when enabling reset_check

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #3813 from tchaikov/add-doc-on-memory-level

doc: add docs about log's "memory level"

doc: add docs about log's "memory level"

Signed-off-by: Kefu Chai <kchai@redhat.com>