librdb: add perf counters descriptions

Sign-off-by: Alyona Kiseleva <akiselyova@mirantis.com>

Merge pull request #4235 from ceph/wip-rbd-unit-tests

tests: exercise all RBD features during unit testing

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4232 from ceph/fix-make-conf

Fix make conf

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4236 from ceph/wip-verify-mds-journal-makefile

mds: update Makefile for removed script

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

mds: update Makefile for removed script

Oversight in "mds: remove verify-mds-journal.sh script"

Signed-off-by: John Spray <john.spray@redhat.com>

tests: exercise all RBD features during unit test

The librbd unit tests currently only test the old image format. Ensure
the new format and its possible features are also tested.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

make: fix with_rbd guard

FileJournal needs stuff from blkdev.c in libcommon.

./.libs/libosd.a(libos_la-FileJournal.o): In function `FileJournal::_open_block_device()':
/home/nwatkins/src/ceph/src/os/FileJournal.cc:139: undefined reference to `get_block_device_size(int, long*)'
/home/nwatkins/src/ceph/src/os/FileJournal.cc:161: undefined reference to `block_device_support_discard(char const*)'
./.libs/libosd.a(libos_la-FileJournal.o): In function `FileJournal::do_discard(long, long)':
/home/nwatkins/src/ceph/src/os/FileJournal.cc:1587: undefined reference to `block_device_discard(int, long, long)'

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

make: add with_rbd guard

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

Merge pull request #4231 from ceph/wip-rm-verify-mds

mds: remove verify-mds-journal.sh script

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

mds: remove verify-mds-journal.sh script

This invokes the MDS in a way that it no longer understands,
and is unused.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #4211 from ceph/wip-11120

osdc/Striper.cc fix stripe_count == 1 && stripe_unit != object_size

Reviewed-by: John Spray <jspray@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4225 from Fale/patch-1

doc: fix the architecture diagram in cephfs.rst

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: fix the architecture diagram in cephfs.rst

Signed-off-by: Fabio Alessandro Locati <fabio@locati.cc>

Merge pull request #4215 from ceph/wip-11258

Wip 11258

Reviewed-by: John Spray <john.spray@redhat.com>

Merge remote-tracking branch 'gh/hammer'

Conflicts:
src/gmock
src/osd/ReplicatedPG.cc

os/KeyValueDB: skip experimental check for test_init

We don't want to spit out the warning twice, and we don't have cct
anyway.

Also test_init is annoying; we should try to kill it.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3878 from ceph/wip-dirty-pg-keys

osd: coalesce into single omap_setkeys for normal writes

Tested-by: Andreas Bluemle <andreas.bluemle@itxperts.de>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #4212 from zhouyuan/isal_2.13

erasure-code: Update ISA-L to 2.13

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4160 from ceph/wip-rocksdb

update rocksdb

mark kinetic experimental

Signed-off-by: Sage Weil <sage@redhat.com>

mark rocksdb experimental

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #4131 from t-miyamae/wip-mshec-documentation-h2

erasure code: add shec's documentation / change default layout

Reviewed-by: Loic Dachary <ldachary@redhat.com>

mds: fix out-of-order messages

When MDS is no longer laggy, it should process deferred messages
first, then process newly received messages.

Fix: #11258
Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #4183 from stiopaa1/OSDMonTrailingSpaces

mon/OSDMonitor: remove trailing whitespaces

Reviewed-by: Kefu Chai <kchai@redhat.com>

erasure-code: Update ISA-L to 2.13

ISA-L 2.13 brings better performance on Avoton (20%). There's no impact on Xeon
platform. The details are in the release notes.

There's a new API ec_encode_data_update() for incremental encoding
and decoding. The other highlevel API keeps the same as in 2.10

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

osdc/Striper.cc fix stripe_count == 1 && stripe_unit != object_size

Fixes:11120
Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #4209 from ceph/wip-java

fix rhel linking error

Backport: giant, firefly, dumpling

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

java: libcephfs_jni.so is in /usr/lib64 on rhel

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

java: dump extra info for ftruncate assertion

Nightly ran and encountered a situation in which fstat following
ftruncate reported a size not equal to the truncated size.

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

Merge pull request #4203 from Vicente-Cheng/fixed-get-mdsmap-assertion

mds: fixed the ceph get mdsmap assertion.

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4173 from ceph/wip-mds-kill-session

Wip mds kill session

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: John Spray <jspray@redhat.com>

Merge pull request #4168 from ceph/wip-11226-dir-fx

RFC client: Hold on to exclusive caps on directories we "own"

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4177 from ceph/wip-11226-extra

Wip 11226 extra

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4189 from ceph/wip-11236

test_libcephfs: do cleanup in ReleaseMounted test

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4192 from ceph/wip-11218

#11218: Multi-MDS fixes for 'damaged' ranks

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4199 from ceph/wip-11250

cls_rbd: fix read past end of bufferlist c_str() in debug log msg

Reviewed-by: Haomai Wang <haomaiwang@gmail.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4194 from athanatos/wip-11199

ReplicatedPG: trim backfill intervals based on peer's last_backfill_star...

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4195 from athanatos/wip-11197

PG: set/clear CREATING in Primary state entry/exit

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4200 from ceph/wip-mds-remove-inode

mds: properly remove inode after purging stray

Reviewed-by: John Spray <john.spray@redhat.com>

Fixed the ceph get mdsmap assertion.

    When we want to get mdsmap, we try to get_version()
    and the return value err = 0 means success.

    The assert verified r == 0. r would not change in this flow.
    It always meet assert and lead mon failure.

    I think this verify should be:
        assert(err == 0)
    It will help to check return value of get_version().

If you have any questions, feel free to let me know.
Thanks!

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

Merge pull request #4193 from larkly/patch-docerror-erasure

doc: Incomplete example in erasure-coded-pool.rst

Reviewed-by: Kefu Chai <kchai@redhat.com>

mds: properly remove inode after purging stray

Signed-off-by: Yan, Zheng <zyan@redhat.com>

cls_rbd: fix read past end of bufferlist c_str() in debug log msg

Fixes: #11250
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #4196 from athanatos/wip-move-code

osd: Wip move code

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4171 from majianpeng/cleanup

common: print hexadecimal for ghobject_t.generation.

Reviewed-by: Sage Weil <sage@redhat.com>

Doc: Incomplete example in erasure-coded-pool.rst

Signed-off-by: Bosse Klykken <larkly@gmail.com>

rocksdb: fix 32-bit build

Signed-off-by: Sage Weil <sage@redhat.com>

Makefile-rocksdb.am: update for latest rocks

Lots of files changed here; rebuilt the list from scratch.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/: Move ReplicatedBackend methods into ReplicatedBackend.cc

Signed-off-by: Samuel Just <sjust@redhat.com>

ReplicatedPG: remove unused C_OnPushCommit

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #4113 from wonzhq/rbd-fix

rbd: misc fixes

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mds: include damaged in MDSMap::dump

Signed-off-by: John Spray <john.spray@redhat.com>

mds: update peer failure response to account for damaged

...by calling get_down_mds_set instead of get_failed_mds_set.

Also remove a redundant if(true) around this region.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: add get_down_mds_set method

For places where we would like to treat failed
and damaged MDS ranks the same, like detecting
when someone has dropped offline.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: account for 'damaged' in MDSMap::is_degraded

Fixes: #11218
Signed-off-by: John Spray <john.spray@redhat.com>

mds: clarify MDBalancer::send_heartbeat

The initial is_degraded() check guarantees that
the 'in' set is equal to the 'up' set.  Later,
this calls get_mds_set and assigns it to a variable
called 'up'.

It's clearer to use get_up_mds_set into the variable
called up (this was confusing when debugging #11218
which was itself a result of is_degraded() ignoring
damaged ranks).

Signed-off-by: John Spray <john.spray@redhat.com>

test_libcephfs: do cleanup in ReleaseMounted test

Fixes: #11236
Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #4188 from XinzeChi/wip-test-1

test: add compile-command for test_ceph_daemon.py

Reviewed-by: Loic Dachary <ldachary@redhat.com>

test: add compile-command for test_ceph_daemon.py

Signed-off-by: Xinze Chi <xmdxcxz@gmail.com>

common: print hexadecimal for ghobject_t.generation & shard_id

In filestore, it use hexadecimal for ghobject_t.generation &
shard_id. Make the same.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

mon/OSDMonitor: remove trailing whitespaces

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #4181 from ceph/wip-rpm-systemd-conditional-syntax

ceph.spec.in: fix _with_systemd conditional

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4182 from ceph/wip-replicatedpg-signed

ReplicatedPG: fix a signed/unsigned comparison warning

Reviewed-by: Samuel Just <sjust@redhat.com>

ReplicatedPG: fix a signed/unsigned comparison warning

This mismatch about whether pool IDs are signed or unsigned is
a persistent annoyance. I'm now casting the unsigned down to signed space
because apparently the OSD is using negative IDs for temporary object
namespaces.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

ceph.spec.in: fix _with_systemd conditional

The RPM conditional needs a "?" in the event that it is not defined.

client: Hold on to exclusive caps on directories we "own"

If a directory is complete, we *really* want to keep the exclusive cap
so that we don't end up needing to do MDS lookup requests on every cache
miss.

Fixes: #11226
Signed-off-by: Greg Farnum <gfarnum@redhat.com>

osd: do not double-write log entries

The pg_log.add() call already dirties the log such that the later
write_log() call will write it.  There is no need to encode it separately
here and then explicitly omap_setkeys() it.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: combine info and log writes into single omap_setkeys

This saves a surprising amount of CPU.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: pass map to write_info instead of txn

No real change yet.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/hammer'

Conflicts:
src/gmock

PG: set/clear CREATING in Primary state entry/exit

Previously, we did not actually set it when we got a pg creation message from
the mon.  It would actually get set on the first start_peering_interval after
that point.  If we don't get that far, but do send a stat update to the mon, we
can end up with 11197.  Instead, let's just set it and clear it upon entry into
and exit from the Primary state.

Fixes: 11197
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #3688 from XinzeChi/wip-clear-transaction

osd: new transcation if needed

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #3870 from majianpeng/cache-tier

Cache tier bug fixs

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3679 from mslovy/wip-findObjectContext-opt

ReplicatedPG:: find_object_context clear extra creation (head,snapdir)

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4065 from majianpeng/pgop-memleak

osd/ReplicatedPG: Fix a memory leak in do_pg_op.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3974 from majianpeng/fgetattr-fix

os/FileStore: For getxattr, enlarge the value size avoid try again.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3604 from yghannam/yazen-testing

crc32c: add aarch64 optimized crc32c implementation

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4073 from jdurgin/wip-lttng-daemon

common, global: use lttng ust functions for handling fork-like calls

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3916 from ceph/wip-rpm-no-ghost-socket-dir

ceph.spec.in: fix handling of /var/run/ceph

Reviewed-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Merge pull request #4000 from ceph/wip-da-SCA-20150312

Fixes for Covertity issues

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4128 from dzafman/wip-11139-hammer

ceph-objectstore-tool: Use exit status 11 for incompatible import attemp...

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #4124 from ceph/wip-omap-clear

osd: fix omap_clear handling

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #4179 from ceph/wip-11065

unit/tox tests structure for ceph-disk

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ceph-disk: add test files to EXTRA_DIST

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-disk: remove double import

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-disk: create initial structure for tox/unit tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>

client: conclude -ENOENT when there is null dentry

Signed-off-by: Yan, Zheng <zyan@redhat.com>

client: don't clear COMPLETE flag when trimming null dentry

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: make sure readdir reply include Fs cap for directory

filelock in LOCK_XSYN state does not allow Fs cap. so client can't
mark directory as complete when handling the readdir reply.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #4170 from ceph/hammer-11205

qa/workunits/fs/misc: fix filelock_interrupt.py

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

qa/workunits/fs/misc: fix filelock_interrupt.py

Handle the case that kernel does not support fcntl.F_OFD_SETLK.
Also fix the code that checks if fnctl fails with errno == EINTR.

Fixes: 11205
Signed-off-by: Yan, Zheng <zyan@redhat.com>
(cherry picked from commit 4ececa3dc4a21b98f61a592da9e2be60a0d71625)

mds: drop replayed requests when seesion is closed

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: fix crash when killing busy session

journal_close_session() has already freed pending_prealloc_inos

Signed-off-by: Yan, Zheng <zyan@redhat.com>

ReplicatedPG: trim backfill intervals based on peer's last_backfill_started

Otherwise, we fail to trim the peer's last_backfill_started and get bug 11199.

1) osd 4 backfills up to 31bccdb2/mira01213209-286/head (henceforth: foo)

2) Interval change happens

3) osd 0 now finds itself backfilling to 4 (lb=foo) and osd.5
(lb=b6670ba2/mira01213209-160/snapdir//1, henceforth: bar)

4) recover_backfill causes both 4 and 5 to scan forward, so 4 has an interval
starting at foo, 5 has an interval starting at bar.

5) Once those have come back, recover_backfill attempts to trim off the
last_backfill_started, but 4's interval starts after that, so foo remains in
osd 4's interval (this is the bug)

7) We serve a copyfrom on foo (sent to 4 as well).

8) We eventually get to foo in the backfilling. Normally, they would have the
same version, but of course we don't update osd.4's interval from the log since
it should not have received writes in that interval. Thus, we end up trying to
recover foo on osd.4 anyway.

9) But, an interval change happens between removing foo from osd.4 and
completing the recovery, leaving osd.4 without foo, but with lb >= foo

Fixes: #11199
Backport: firefly
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #4162 from athanatos/wip-11156

ReplicatedPG::promote_object: check scrubber and block if necessary

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4164 from athanatos/wip-revert-degraded-writes-release-notes

release-notes.rst: update for degraded writes revert

release-notes.rst: update for degraded writes revert

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #4163 from athanatos/wip-revert-degraded-writes

Wip revert degraded writes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4161 from trhoden/cephdeploy_not_new

doc: no longer call out ceph-deploy as new

doc: no longer call out ceph-deploy as new

The tip declaring ceph-deploy as new doesn't really apply anymore.

Signed-off-by: Travis Rhoden <trhoden@redhat.com>