Make sure s3_utilities are found.

Add FindBin code to rgw/s3 tests to include s3_utilites.pm properly.

Fixes: 7863
Signed-off-by: Warren Usui <warren.usui@inktank.com>

Merge pull request #1527 from ceph/wip-6465

be nitpicky about missing references

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1531 from onlyjob/debian

minor corrections from Debian

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1363 from ceph/wip-3863

Add ceph-client-debug inode lookup

Reviewed-by: Greg Farnum <greg@inktank.com>

minor corrections to package descriptions

minor init.d scripts lintianisation:

 * init.d-script-does-not-source-init-functions
 * init.d-script-does-not-implement-required-option
 * init.d-script-missing-lsb-description

Merge pull request #1528 from ceph/wip-dz-watch-test

Reviewed-by: Samuel Just <sam.just@inktank.com>

qa: Add ceph_multi_stress_watch for rep and ec

Fixes: #7438
Signed-off-by: David Zafman <david.zafman@inktank.com>

Merge pull request #1525 from ceph/wip-listxattr

client: don't include ceph.{file,dir}.layout vxattr in listxattr()

Reviwed-by: Sage Weil <sage@inktank.com>

ensure pybind is in the PYTHONPATH

Signed-off-by: Alfredo Deza <alfredo@deza.pe>

Merge pull request #1523 from ceph/wip-7827

msg: set recv stamp for local delivery

Reviewed-by: Greg Farnum <greg@inktank.com>

be nitpicky about missing references

Signed-off-by: Alfredo Deza <alfredo@deza.pe>

Merge pull request #1526 from ceph/wip-status-function-names

mon: Monitor: change 'status' function names for clarity's sake

Reviewed-by: Sage Weil <sage@inktank.com>

Revert "erasure-code: gf-complete detects SSE at runtime"

This reverts commit dbcf4471838fbf5dce2ee06fc73493f513dbf9f0.

Merge pull request #1524 from dachary/wip-sse

erasure-code: deactivate SSE optimizations

mon: Monitor: s/_mon_status()/get_mon_status()/

'Monitor::_mon_status()' provides status specific to the monitor being
poked.  This is information that does not necessarily relate with
cluster status, and can even be obtained when there's no quorum (via the
admin socket).

We change the function name to make the distinction between mon-specific
status and cluster-status, which is obtained via a different function.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon: Monitor: s/get_status()/get_cluster_status()/

Monitor::get_status() is actually used to obtain 'cluster status' (i.e.,
status about *all* the monitors, maps, IO, overall healthiness).

Just make that clear in the function name, avoiding confusion between
multiple 'status-related' functions in the monitor.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

client: don't include ceph.{file,dir}.layout vxattr in listxattr()

This avoids 'cp -a' modifying layout of new files/directories.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

erasure-code: deactivate SSE optimizations

Workaround until the bug is diagnosed and fixed

http://tracker.ceph.com/issues/7826 refs #7826

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #1521 from ceph/wip-7810

mds: reset connection priv after connection's session is removed

Reviewed-by: Sage Weil <sage@inktank.com>

msg: set recv stamp for local delivery

Fixes: #7827
Signed-off-by: Sage Weil <sage@inktank.com>

mds: reset connection priv after connection's session is removed

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

Merge tag 'v0.78'

v0.78

doc/release-notes: 0.78 final notes

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #1515 from ceph/wip-rgw-less-alloc

Wip rgw less alloc

Reviewed-by: Sage Weil <sage@inktank.com>

doc/release-notes: final 0.78

Signed-off-by: Sage Weil <sage@inktank.com>

rgw: get rid of a memory allocation

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: remove memory allocation

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

0.78

Merge pull request #1512 from dachary/wip-jerasure-sse

 erasure-code: gf-complete detects SSE at runtime

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1513 from ceph/wip-7438

Reviewed-by: Loic Dachary <loic@dachary.org>

Merge pull request #1510 from ceph/wip-pq

osd: some PrioritizedQueue cleanups; dump locking fix; debug pqueue state at high debug level

Reviewed-by: Samuel Just <sam.just@inktank.com>

Revert "ReplicatedPG: disallow trans which atomically create and remove an object"

This reverts commit 9ee108428176c93fe48ae5ae95b160eadabca9b6.

This appears to break rgw in its current state; test and fix soon.

doc/release-notes: 0.78 notes

Signed-off-by: Sage Weil <sage@inktank.com>

Merge remote-tracking branch 'gh/wip-multimds-fixes'

Reviewed-by: Sage Weil <sage@inktank.com>

erasure-code: gf-complete detects SSE at runtime

And the gf-complete code has been updated to rely on the
gf_internal_t::sse flag instead of #if conditionals.

Signed-off-by: Loic Dachary <loic@dachary.org>

autotools: AX_SSE detects the compiler SSE capabilities

For each SSE feature supported by the compiler

* add the corresponding -msse* flag
* define HAVE_SSE*

Remove AX_EXT because it decides based on the CPU capabilities of the
machine compiling the binary which may or may not be the one running
them.

Signed-off-by: Loic Dachary <loic@dachary.org>

test: Add erasure coding to stress watch test

Fixes: #7438
Signed-off-by: David Zafman <david.zafman@inktank.com>

test: Reorg multi_stress_watch to prepare for ec option

If old options are specified maintain backward compatibility

Signed-off-by: David Zafman <david.zafman@inktank.com>

test: Fix ceph_filestore_dump.sh test for new EC pool creation

Signed-off-by: David Zafman <david.zafman@inktank.com>

Merge pull request #1500 from ceph/wip-tid_t

Change tid_t to ceph_tid_t to avoid conflicts with a popular type

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1465 from ceph/wip-s3pm-wusui

Remove code duplication from s3 tests.

Merge pull request #1511 from dachary/wip-exclamation

tests: use ^ instead of ! as invalid char

Reviewed-by: Sage Weil <sage@inktank.com>

tests: use ^ instead of ! as invalid char

Because !!! is incorrectly interpreted as an error by gitbuilder.

Signed-off-by: Loic Dachary <loic@dachary.org>

doc/release-notes: stop confusing sphinx

ERROR: /srv/autobuild-ceph/gitbuilder.git/build/doc/release-notes.rst:594: Unknown target name: "debug".

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #1503 from dachary/wip-jerasure-submodules

 erasure-code: add gf-complete / jerasure submodules

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1505 from ceph/wip-7728

Wip 7728

Reviewed-by: Sage Weil <sage@inktank.com>

Merge remote-tracking branch 'gh/firefly'

Merge pull request #1509 from guangyy/wip-folder-merge

Make it possible for FileStore to be in a state which never do merge and keep it backward compatible

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #1506 from ceph/wip-7755

PG::find_best_info: fix log_tail component

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1507 from ceph/wip-7733

PG::start_peering_interval: always send_notify if !primary

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1508 from ceph/wip-7777

ReplicatedPG::do_op: delay if snapdir is unreadable

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1312 from cernceph/master

rados.py: add aio_remove

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #1504 from ceph/wip-7780

objecter: waive OSDMAP_FULL check for MDS

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #1476 from ceph/wip-scrub-lock

osd: do not queue scrub for non-primary

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #1502 from ceph/wip-flush-journal-asok

wip-flush-journal-asok

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

objecter: waive OSDMAP_FULL check for MDS

The MDS expects to be able to perform writes to OSDs even
if the full ratio has been reached, in order to journal
file deletions to free space.

Fixes: #7780
Signed-off-by: John Spray <john.spray@inktank.com>

qa: test_alloc_hint: flush journal before prodding the FS

OSDs that for some reason get behind on processing their op queue break
expect_alloc_hint_eq(), as it pokes the FS and not the journal.  Fix it
by flushing the journal before proceeding with anything else.

Signed-off-by: Ilya Dryomov <ilya.dryomov@inktank.com>

osd: add flush_journal admin socket command

Add flush_journal admin socket command to be able to flush journal to
the permanent store for online osds.  (For offline osds we already have
ceph-osd --flush-journal.)

Signed-off-by: Ilya Dryomov <ilya.dryomov@inktank.com>

mds: fix typo in MDCache::handle_cache_rejoin_strong

It should invent dirfrags according to the dirfragtree

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

erasure-code: add gf-complete / jerasure submodules

Instead of copying the files in the ceph repository, which is less
convenient.

When building the headers are ignored, even though they do
not exist. When creating the tarbal with make dist, it fails because
they cannot be found. I misread src/gf_int.h to be include/gf_int.h and
wrongfully thought the submodules were to blame. This is why they were
removed shortly after being added.

Signed-off-by: Loic Dachary <loic@dachary.org>

erasure-code: remove copy of gf-complete / jerasure

Signed-off-by: Loic Dachary <loic@dachary.org>

mds: fix NULL pointer dereference in Server::handle_client_rename

Inode's parent dentry can be NULL, which causes segment fault in
Locker::acquire_locks

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: avoid infinite loop in MDLog::submit_entry()

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

Merge pull request #1496 from ceph/wip-7708

mds/Locker: fix null deref on cap import

Merge pull request #1499 from ceph/wip-no-version

Add NO_VERSION to avoid rebuilding ceph_ver.h and relinking

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1498 from ceph/wip-limit-libkeyutils

Makefiles: remove libkeyutils from every binary except two

Reviewed-by: Sage Weil <sage@inktank.com>

mds: fix potential invalid pointer dereference

pin importing inode until subtree importing finishes

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: rdlock dirfragtree lock when renaming directory

This make sure inode importer and inode exporter have consistent
view of dirfrags.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: don't mark scatter locks dirty when dirfrag is dirty

The journal reply code has check that decides which scatter locks
should be marked as dirty. So don't unconditionally mark scatter
locks dirty when dirfrag is dirty

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: trim other MDS' mdsdir during recovery

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: remove wrong assertion

when handling resolve message, it's possible that bound dirfrags
are missing.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

fragtree_t: fix get_leaves_under()

If fragtree is (*^1) and the caller wants leaves under frag 00*.
get_leaves_under() should return empty list.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: improve success rate of subtree exporting

When exporting a subtree, the migrator acquires the required locks,
then freezes the subtree and releases the locks. After subtree is
frozen, it try acquiring the same locks again.

This patch make scatter locks keep in their old states if inode has
exporting dirfrag. It improves the chance that migrator acquires all
required locks when subtree is frozen.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: cleanup CInode::decode_import()

We have bumped protocol version several times, no need to maintain
compatibility for ancient message.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: export file locks together with inode

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: re-issue caps after importing caps (rename)

After importing inode, the issued caps can be less than the caps
client wants. So re-issue caps after importing inode.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: fix open remote dirfrag deadlock

During subtree migration, the importer may need to open subtree
bound dirfrags. Opening subtree bound dirfrags happens after the
exporter freeze the exporting subtee. So the discover message for
opening subtree bound dirfrags should not wait for any freezing
tree/directory, otherwise deadlock can happen.

In MDCache::handle_discover(), there are two cases can cause
discover messages wait for freezing tree/directory. One case is
fetching bare-bone dirfrags. Another case is, when merging dirfrags,
some of the dirfrags are frozen, some are freezing.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: properly initialize MDCache::fragment_info_t

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: avoid issuing zero caps to client

make sure CEPH_CAP_PIN exists in stale caps.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: drop auth pins before waiting for dir unfreeze

Otherwise deadlock can happen

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: fix stack overflow caused by nested dispatch

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: rollback slave request after slave prepare is journalled

Resolve ack message can abort slave requests that are being journalled.
The slave rollback does not handle this case properly. The fix is mark
slave request aborted in this case. The slave rollback code is executed
after slave prepare is safely journalled.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: treat flushing as dirty when rejoining scattered locks

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

Change tid_t to ceph_tid_t to avoid conflicts with a popular type

Signed-off-by: Dan Mick <dan.mick@inktank.com>

Add NO_VERSION to avoid rebuilding ceph_ver.h and relinking

Signed-off-by: Dan Mick <dan.mick@inktank.com>

Makefiles: remove libkeyutils from every binary except two

Only rbd and mount_ceph need secret.c, and only secret.c needs libkeyutils;
remove it from LIBCOMMON_DEPS so it's not a dependency for everything,
remove secret.c from libcommon.a, and add it to mount.ceph/rbd's sources;
add LIBKEYID_LIB to mount.ceph/rbd's LDADD.

Signed-off-by: Dan Mick <dan.mick@inktank.com>

Merge pull request #1497 from dachary/wip-jerasure

remove gf-complete / jerasure sub modules

Reviewed-by: Sage Weil <sage@inktank.com>

remove gf-complete / jerasure sub modules

and add the source files in the tree.

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #1494 from dachary/wip-gitmodule-mirrors

build: add gf-complete/jerasure to gitmodule_mirrors

ReplicatedPG::do_op: delay if snapdir is unreadable

Since all we really need on a snapdir is the context, we really only
need it to be !missing.  However, it might become !missing before it
becomes !unreadable.  That allows ops to end up in the
waiting_for_degraded queue before one in waiting_for_unreadable is
woken, which allows the ops to be reordered.  Rather than reintroduce an
extra waiting_for_missing queue, simply require !unreadable for snapdir
(which implies !misssing).

Fixes: #7777
Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge pull request #1125 from ceph/wip-mon-docs

wip-mon-docs: Better explain required number of monitors & how to troubleshoot a monitor

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
Reviewed-by: Loic Dachary <loic@dachary.org>

mds/Locker: fix null deref on cap import

The session Connection* may be NULL.

Fixes: #7708
Signed-off-by: Sage Weil <sage@inktank.com>

build: add gf-complete/jerasure to gitmodule_mirrors

Signed-off-by: Loic Dachary <loic@dachary.org>

osd: dump priority queue state on dequeue at level 30

Signed-off-by: Sage Weil <sage@inktank.com>

osd: fix OpWQ dump locking

The pqueue is protected by the wq lock, not by qlock; for example, see
OpWQ::_enqueue.  qlock protects the pg_for_processing map only.

Fixes: #7735
Signed-off-by: Sage Weil <sage@inktank.com>

common/PrioritizedQueue: include first item's cost in dump

Signed-off-by: Sage Weil <sage@inktank.com>

common/PrioritizedQueue: constify a few things

Signed-off-by: Sage Weil <sage@inktank.com>

common/PrioritizedQueue: fix remove_by_class() corner case

If i is the first entry, then setting cur = begin() sets us up to point at
something that we are about to delete.  Move the check to the end to avoid
this.

Backport: emperor, dumpling
Signed-off-by: Sage Weil <sage@inktank.com>

doc/dev/cache-pool: describe the tiering agent

Signed-off-by: Sage Weil <sage@inktank.com>