Merge pull request #4244 from dachary/wip-9580

tests for ceph-disk: make journal setup conditional on osd backend

tests: add ceph-disk.sh activate for memstore

Add a test for the activation of the memstore objectstore and verify
that it works without specifying a journal.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: move rados put/get tests in a function

Add the test_pool_read_write function to share the rados put / get test
that demonstrate the osd that has been created can actually be used. Use
it from the both the regular device and dmcrypt tests.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph-disk.sh dmcrypt tests use test_setup_dev_and_run

Instead of duplicating the device construction / destruction logic for
dmcrypt tests, use test_setup_dev_and_run to do it. It is now able to
recover from devmapper leftover which may occur when a cryptsetup test
fails.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: merge ceph-disk.sh activate dmcrypt functions

The activate_dmcrypt_plain_dev_body and activate_dmcrypt_dev_body
functions are almost identical, merge them and differentiate with an
argument.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: add test_setup_dev_and_run to ceph-disk.sh

Move test_activate_dev to test_setup_dev_and_run and make it
run the function given in argument. test_activate_dev calls
test_setup_dev_and_run and no longer needs to implement device
allocation or destruction.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: improve ceph-disk.sh setup/teardown

Address all possible failure cases, when ceph-disk.sh completes or when
it starts with leftover from a previous interrupted run. It is assumed
that ceph-disk.sh will crash at any point.

* umount all mount points that belong to ceph-disk.sh (check the
  absolute path of the directory)
* dmsetup remove all device mapper nodes found to hold a loop device
  that ceph-disks.sh no longer uses
* losetup --detach all loop devices that ceph-disks.sh no longer uses

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: do not use -e in ceph-disk.sh

The tests explicitly return on error when relevant. Add two error cases:

* detect when the allocation of a loop device fails.
* in the outer loop, return immediately if one of the test fails

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: reduce ceph-disk.sh verbosity

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph-disk: follow ceph-osd hints when setting up journal

- If the journal is required, require it.
- If the journal is not allowed, do not allow one to be specified
- If the journal is not wanted, to not set one up by default when none is
provided.

See #9580

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-osd: add --check-wants-journal, --check-allows-journal

Completes OSD side of #9580

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-osd: fix usage

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #4235 from ceph/wip-rbd-unit-tests

tests: exercise all RBD features during unit testing

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4232 from ceph/fix-make-conf

Fix make conf

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4236 from ceph/wip-verify-mds-journal-makefile

mds: update Makefile for removed script

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

mds: update Makefile for removed script

Oversight in "mds: remove verify-mds-journal.sh script"

Signed-off-by: John Spray <john.spray@redhat.com>

tests: exercise all RBD features during unit test

The librbd unit tests currently only test the old image format. Ensure
the new format and its possible features are also tested.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

make: fix with_rbd guard

FileJournal needs stuff from blkdev.c in libcommon.

./.libs/libosd.a(libos_la-FileJournal.o): In function `FileJournal::_open_block_device()':
/home/nwatkins/src/ceph/src/os/FileJournal.cc:139: undefined reference to `get_block_device_size(int, long*)'
/home/nwatkins/src/ceph/src/os/FileJournal.cc:161: undefined reference to `block_device_support_discard(char const*)'
./.libs/libosd.a(libos_la-FileJournal.o): In function `FileJournal::do_discard(long, long)':
/home/nwatkins/src/ceph/src/os/FileJournal.cc:1587: undefined reference to `block_device_discard(int, long, long)'

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

make: add with_rbd guard

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

Merge pull request #4231 from ceph/wip-rm-verify-mds

mds: remove verify-mds-journal.sh script

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

mds: remove verify-mds-journal.sh script

This invokes the MDS in a way that it no longer understands,
and is unused.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #4211 from ceph/wip-11120

osdc/Striper.cc fix stripe_count == 1 && stripe_unit != object_size

Reviewed-by: John Spray <jspray@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4225 from Fale/patch-1

doc: fix the architecture diagram in cephfs.rst

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: fix the architecture diagram in cephfs.rst

Signed-off-by: Fabio Alessandro Locati <fabio@locati.cc>

Merge pull request #4215 from ceph/wip-11258

Wip 11258

Reviewed-by: John Spray <john.spray@redhat.com>

Merge remote-tracking branch 'gh/hammer'

Conflicts:
src/gmock
src/osd/ReplicatedPG.cc

os/KeyValueDB: skip experimental check for test_init

We don't want to spit out the warning twice, and we don't have cct
anyway.

Also test_init is annoying; we should try to kill it.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3878 from ceph/wip-dirty-pg-keys

osd: coalesce into single omap_setkeys for normal writes

Tested-by: Andreas Bluemle <andreas.bluemle@itxperts.de>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #4212 from zhouyuan/isal_2.13

erasure-code: Update ISA-L to 2.13

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4160 from ceph/wip-rocksdb

update rocksdb

mark kinetic experimental

Signed-off-by: Sage Weil <sage@redhat.com>

mark rocksdb experimental

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #4131 from t-miyamae/wip-mshec-documentation-h2

erasure code: add shec's documentation / change default layout

Reviewed-by: Loic Dachary <ldachary@redhat.com>

mds: fix out-of-order messages

When MDS is no longer laggy, it should process deferred messages
first, then process newly received messages.

Fix: #11258
Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #4183 from stiopaa1/OSDMonTrailingSpaces

mon/OSDMonitor: remove trailing whitespaces

Reviewed-by: Kefu Chai <kchai@redhat.com>

erasure-code: Update ISA-L to 2.13

ISA-L 2.13 brings better performance on Avoton (20%). There's no impact on Xeon
platform. The details are in the release notes.

There's a new API ec_encode_data_update() for incremental encoding
and decoding. The other highlevel API keeps the same as in 2.10

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

osdc/Striper.cc fix stripe_count == 1 && stripe_unit != object_size

Fixes:11120
Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #4209 from ceph/wip-java

fix rhel linking error

Backport: giant, firefly, dumpling

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

java: libcephfs_jni.so is in /usr/lib64 on rhel

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

java: dump extra info for ftruncate assertion

Nightly ran and encountered a situation in which fstat following
ftruncate reported a size not equal to the truncated size.

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

Merge pull request #4203 from Vicente-Cheng/fixed-get-mdsmap-assertion

mds: fixed the ceph get mdsmap assertion.

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4173 from ceph/wip-mds-kill-session

Wip mds kill session

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: John Spray <jspray@redhat.com>

Merge pull request #4168 from ceph/wip-11226-dir-fx

RFC client: Hold on to exclusive caps on directories we "own"

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4177 from ceph/wip-11226-extra

Wip 11226 extra

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4189 from ceph/wip-11236

test_libcephfs: do cleanup in ReleaseMounted test

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4192 from ceph/wip-11218

#11218: Multi-MDS fixes for 'damaged' ranks

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4199 from ceph/wip-11250

cls_rbd: fix read past end of bufferlist c_str() in debug log msg

Reviewed-by: Haomai Wang <haomaiwang@gmail.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4194 from athanatos/wip-11199

ReplicatedPG: trim backfill intervals based on peer's last_backfill_star...

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4195 from athanatos/wip-11197

PG: set/clear CREATING in Primary state entry/exit

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4200 from ceph/wip-mds-remove-inode

mds: properly remove inode after purging stray

Reviewed-by: John Spray <john.spray@redhat.com>

Fixed the ceph get mdsmap assertion.

    When we want to get mdsmap, we try to get_version()
    and the return value err = 0 means success.

    The assert verified r == 0. r would not change in this flow.
    It always meet assert and lead mon failure.

    I think this verify should be:
        assert(err == 0)
    It will help to check return value of get_version().

If you have any questions, feel free to let me know.
Thanks!

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

Merge pull request #4193 from larkly/patch-docerror-erasure

doc: Incomplete example in erasure-coded-pool.rst

Reviewed-by: Kefu Chai <kchai@redhat.com>

mds: properly remove inode after purging stray

Signed-off-by: Yan, Zheng <zyan@redhat.com>

cls_rbd: fix read past end of bufferlist c_str() in debug log msg

Fixes: #11250
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #4196 from athanatos/wip-move-code

osd: Wip move code

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4171 from majianpeng/cleanup

common: print hexadecimal for ghobject_t.generation.

Reviewed-by: Sage Weil <sage@redhat.com>

Doc: Incomplete example in erasure-coded-pool.rst

Signed-off-by: Bosse Klykken <larkly@gmail.com>

rocksdb: fix 32-bit build

Signed-off-by: Sage Weil <sage@redhat.com>

Makefile-rocksdb.am: update for latest rocks

Lots of files changed here; rebuilt the list from scratch.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/: Move ReplicatedBackend methods into ReplicatedBackend.cc

Signed-off-by: Samuel Just <sjust@redhat.com>

ReplicatedPG: remove unused C_OnPushCommit

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #4113 from wonzhq/rbd-fix

rbd: misc fixes

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mds: include damaged in MDSMap::dump

Signed-off-by: John Spray <john.spray@redhat.com>

mds: update peer failure response to account for damaged

...by calling get_down_mds_set instead of get_failed_mds_set.

Also remove a redundant if(true) around this region.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: add get_down_mds_set method

For places where we would like to treat failed
and damaged MDS ranks the same, like detecting
when someone has dropped offline.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: account for 'damaged' in MDSMap::is_degraded

Fixes: #11218
Signed-off-by: John Spray <john.spray@redhat.com>

mds: clarify MDBalancer::send_heartbeat

The initial is_degraded() check guarantees that
the 'in' set is equal to the 'up' set.  Later,
this calls get_mds_set and assigns it to a variable
called 'up'.

It's clearer to use get_up_mds_set into the variable
called up (this was confusing when debugging #11218
which was itself a result of is_degraded() ignoring
damaged ranks).

Signed-off-by: John Spray <john.spray@redhat.com>

test_libcephfs: do cleanup in ReleaseMounted test

Fixes: #11236
Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #4188 from XinzeChi/wip-test-1

test: add compile-command for test_ceph_daemon.py

Reviewed-by: Loic Dachary <ldachary@redhat.com>

test: add compile-command for test_ceph_daemon.py

Signed-off-by: Xinze Chi <xmdxcxz@gmail.com>

common: print hexadecimal for ghobject_t.generation & shard_id

In filestore, it use hexadecimal for ghobject_t.generation &
shard_id. Make the same.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

mon/OSDMonitor: remove trailing whitespaces

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #4181 from ceph/wip-rpm-systemd-conditional-syntax

ceph.spec.in: fix _with_systemd conditional

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4182 from ceph/wip-replicatedpg-signed

ReplicatedPG: fix a signed/unsigned comparison warning

Reviewed-by: Samuel Just <sjust@redhat.com>

ReplicatedPG: fix a signed/unsigned comparison warning

This mismatch about whether pool IDs are signed or unsigned is
a persistent annoyance. I'm now casting the unsigned down to signed space
because apparently the OSD is using negative IDs for temporary object
namespaces.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

ceph.spec.in: fix _with_systemd conditional

The RPM conditional needs a "?" in the event that it is not defined.

client: Hold on to exclusive caps on directories we "own"

If a directory is complete, we *really* want to keep the exclusive cap
so that we don't end up needing to do MDS lookup requests on every cache
miss.

Fixes: #11226
Signed-off-by: Greg Farnum <gfarnum@redhat.com>

osd: do not double-write log entries

The pg_log.add() call already dirties the log such that the later
write_log() call will write it.  There is no need to encode it separately
here and then explicitly omap_setkeys() it.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: combine info and log writes into single omap_setkeys

This saves a surprising amount of CPU.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: pass map to write_info instead of txn

No real change yet.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/hammer'

Conflicts:
src/gmock

PG: set/clear CREATING in Primary state entry/exit

Previously, we did not actually set it when we got a pg creation message from
the mon.  It would actually get set on the first start_peering_interval after
that point.  If we don't get that far, but do send a stat update to the mon, we
can end up with 11197.  Instead, let's just set it and clear it upon entry into
and exit from the Primary state.

Fixes: 11197
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #3688 from XinzeChi/wip-clear-transaction

osd: new transcation if needed

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #3870 from majianpeng/cache-tier

Cache tier bug fixs

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3679 from mslovy/wip-findObjectContext-opt

ReplicatedPG:: find_object_context clear extra creation (head,snapdir)

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4065 from majianpeng/pgop-memleak

osd/ReplicatedPG: Fix a memory leak in do_pg_op.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3974 from majianpeng/fgetattr-fix

os/FileStore: For getxattr, enlarge the value size avoid try again.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3604 from yghannam/yazen-testing

crc32c: add aarch64 optimized crc32c implementation

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4073 from jdurgin/wip-lttng-daemon

common, global: use lttng ust functions for handling fork-like calls

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3916 from ceph/wip-rpm-no-ghost-socket-dir

ceph.spec.in: fix handling of /var/run/ceph

Reviewed-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Merge pull request #4000 from ceph/wip-da-SCA-20150312

Fixes for Covertity issues

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4128 from dzafman/wip-11139-hammer

ceph-objectstore-tool: Use exit status 11 for incompatible import attemp...

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #4124 from ceph/wip-omap-clear

osd: fix omap_clear handling

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #4179 from ceph/wip-11065

unit/tox tests structure for ceph-disk

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ceph-disk: add test files to EXTRA_DIST

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-disk: remove double import

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-disk: create initial structure for tox/unit tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>

client: conclude -ENOENT when there is null dentry

Signed-off-by: Yan, Zheng <zyan@redhat.com>

client: don't clear COMPLETE flag when trimming null dentry

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: make sure readdir reply include Fs cap for directory

filelock in LOCK_XSYN state does not allow Fs cap. so client can't
mark directory as complete when handling the readdir reply.

Signed-off-by: Yan, Zheng <zyan@redhat.com>