ensure pybind is in the PYTHONPATH

Signed-off-by: Alfredo Deza <alfredo@deza.pe>

be nitpicky about missing references

Signed-off-by: Alfredo Deza <alfredo@deza.pe>

Merge pull request #1521 from ceph/wip-7810

mds: reset connection priv after connection's session is removed

Reviewed-by: Sage Weil <sage@inktank.com>

mds: reset connection priv after connection's session is removed

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

Merge tag 'v0.78'

v0.78

doc/release-notes: 0.78 final notes

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #1515 from ceph/wip-rgw-less-alloc

Wip rgw less alloc

Reviewed-by: Sage Weil <sage@inktank.com>

doc/release-notes: final 0.78

Signed-off-by: Sage Weil <sage@inktank.com>

rgw: get rid of a memory allocation

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: remove memory allocation

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

0.78

Merge pull request #1512 from dachary/wip-jerasure-sse

 erasure-code: gf-complete detects SSE at runtime

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1513 from ceph/wip-7438

Reviewed-by: Loic Dachary <loic@dachary.org>

Merge pull request #1510 from ceph/wip-pq

osd: some PrioritizedQueue cleanups; dump locking fix; debug pqueue state at high debug level

Reviewed-by: Samuel Just <sam.just@inktank.com>

Revert "ReplicatedPG: disallow trans which atomically create and remove an object"

This reverts commit 9ee108428176c93fe48ae5ae95b160eadabca9b6.

This appears to break rgw in its current state; test and fix soon.

doc/release-notes: 0.78 notes

Signed-off-by: Sage Weil <sage@inktank.com>

Merge remote-tracking branch 'gh/wip-multimds-fixes'

Reviewed-by: Sage Weil <sage@inktank.com>

erasure-code: gf-complete detects SSE at runtime

And the gf-complete code has been updated to rely on the
gf_internal_t::sse flag instead of #if conditionals.

Signed-off-by: Loic Dachary <loic@dachary.org>

autotools: AX_SSE detects the compiler SSE capabilities

For each SSE feature supported by the compiler

* add the corresponding -msse* flag
* define HAVE_SSE*

Remove AX_EXT because it decides based on the CPU capabilities of the
machine compiling the binary which may or may not be the one running
them.

Signed-off-by: Loic Dachary <loic@dachary.org>

test: Add erasure coding to stress watch test

Fixes: #7438
Signed-off-by: David Zafman <david.zafman@inktank.com>

test: Reorg multi_stress_watch to prepare for ec option

If old options are specified maintain backward compatibility

Signed-off-by: David Zafman <david.zafman@inktank.com>

test: Fix ceph_filestore_dump.sh test for new EC pool creation

Signed-off-by: David Zafman <david.zafman@inktank.com>

Merge pull request #1500 from ceph/wip-tid_t

Change tid_t to ceph_tid_t to avoid conflicts with a popular type

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1465 from ceph/wip-s3pm-wusui

Remove code duplication from s3 tests.

Merge pull request #1511 from dachary/wip-exclamation

tests: use ^ instead of ! as invalid char

Reviewed-by: Sage Weil <sage@inktank.com>

tests: use ^ instead of ! as invalid char

Because !!! is incorrectly interpreted as an error by gitbuilder.

Signed-off-by: Loic Dachary <loic@dachary.org>

doc/release-notes: stop confusing sphinx

ERROR: /srv/autobuild-ceph/gitbuilder.git/build/doc/release-notes.rst:594: Unknown target name: "debug".

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #1503 from dachary/wip-jerasure-submodules

 erasure-code: add gf-complete / jerasure submodules

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1505 from ceph/wip-7728

Wip 7728

Reviewed-by: Sage Weil <sage@inktank.com>

Merge remote-tracking branch 'gh/firefly'

Merge pull request #1509 from guangyy/wip-folder-merge

Make it possible for FileStore to be in a state which never do merge and keep it backward compatible

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #1506 from ceph/wip-7755

PG::find_best_info: fix log_tail component

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1507 from ceph/wip-7733

PG::start_peering_interval: always send_notify if !primary

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1508 from ceph/wip-7777

ReplicatedPG::do_op: delay if snapdir is unreadable

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1312 from cernceph/master

rados.py: add aio_remove

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #1504 from ceph/wip-7780

objecter: waive OSDMAP_FULL check for MDS

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #1476 from ceph/wip-scrub-lock

osd: do not queue scrub for non-primary

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #1502 from ceph/wip-flush-journal-asok

wip-flush-journal-asok

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

objecter: waive OSDMAP_FULL check for MDS

The MDS expects to be able to perform writes to OSDs even
if the full ratio has been reached, in order to journal
file deletions to free space.

Fixes: #7780
Signed-off-by: John Spray <john.spray@inktank.com>

qa: test_alloc_hint: flush journal before prodding the FS

OSDs that for some reason get behind on processing their op queue break
expect_alloc_hint_eq(), as it pokes the FS and not the journal.  Fix it
by flushing the journal before proceeding with anything else.

Signed-off-by: Ilya Dryomov <ilya.dryomov@inktank.com>

osd: add flush_journal admin socket command

Add flush_journal admin socket command to be able to flush journal to
the permanent store for online osds.  (For offline osds we already have
ceph-osd --flush-journal.)

Signed-off-by: Ilya Dryomov <ilya.dryomov@inktank.com>

mds: fix typo in MDCache::handle_cache_rejoin_strong

It should invent dirfrags according to the dirfragtree

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

erasure-code: add gf-complete / jerasure submodules

Instead of copying the files in the ceph repository, which is less
convenient.

When building the headers are ignored, even though they do
not exist. When creating the tarbal with make dist, it fails because
they cannot be found. I misread src/gf_int.h to be include/gf_int.h and
wrongfully thought the submodules were to blame. This is why they were
removed shortly after being added.

Signed-off-by: Loic Dachary <loic@dachary.org>

erasure-code: remove copy of gf-complete / jerasure

Signed-off-by: Loic Dachary <loic@dachary.org>

mds: fix NULL pointer dereference in Server::handle_client_rename

Inode's parent dentry can be NULL, which causes segment fault in
Locker::acquire_locks

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: avoid infinite loop in MDLog::submit_entry()

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

Merge pull request #1496 from ceph/wip-7708

mds/Locker: fix null deref on cap import

Merge pull request #1499 from ceph/wip-no-version

Add NO_VERSION to avoid rebuilding ceph_ver.h and relinking

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1498 from ceph/wip-limit-libkeyutils

Makefiles: remove libkeyutils from every binary except two

Reviewed-by: Sage Weil <sage@inktank.com>

mds: fix potential invalid pointer dereference

pin importing inode until subtree importing finishes

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: rdlock dirfragtree lock when renaming directory

This make sure inode importer and inode exporter have consistent
view of dirfrags.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: don't mark scatter locks dirty when dirfrag is dirty

The journal reply code has check that decides which scatter locks
should be marked as dirty. So don't unconditionally mark scatter
locks dirty when dirfrag is dirty

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: trim other MDS' mdsdir during recovery

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: remove wrong assertion

when handling resolve message, it's possible that bound dirfrags
are missing.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

fragtree_t: fix get_leaves_under()

If fragtree is (*^1) and the caller wants leaves under frag 00*.
get_leaves_under() should return empty list.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: improve success rate of subtree exporting

When exporting a subtree, the migrator acquires the required locks,
then freezes the subtree and releases the locks. After subtree is
frozen, it try acquiring the same locks again.

This patch make scatter locks keep in their old states if inode has
exporting dirfrag. It improves the chance that migrator acquires all
required locks when subtree is frozen.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: cleanup CInode::decode_import()

We have bumped protocol version several times, no need to maintain
compatibility for ancient message.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: export file locks together with inode

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: re-issue caps after importing caps (rename)

After importing inode, the issued caps can be less than the caps
client wants. So re-issue caps after importing inode.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: fix open remote dirfrag deadlock

During subtree migration, the importer may need to open subtree
bound dirfrags. Opening subtree bound dirfrags happens after the
exporter freeze the exporting subtee. So the discover message for
opening subtree bound dirfrags should not wait for any freezing
tree/directory, otherwise deadlock can happen.

In MDCache::handle_discover(), there are two cases can cause
discover messages wait for freezing tree/directory. One case is
fetching bare-bone dirfrags. Another case is, when merging dirfrags,
some of the dirfrags are frozen, some are freezing.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: properly initialize MDCache::fragment_info_t

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: avoid issuing zero caps to client

make sure CEPH_CAP_PIN exists in stale caps.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: drop auth pins before waiting for dir unfreeze

Otherwise deadlock can happen

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: fix stack overflow caused by nested dispatch

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: rollback slave request after slave prepare is journalled

Resolve ack message can abort slave requests that are being journalled.
The slave rollback does not handle this case properly. The fix is mark
slave request aborted in this case. The slave rollback code is executed
after slave prepare is safely journalled.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: treat flushing as dirty when rejoining scattered locks

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

Change tid_t to ceph_tid_t to avoid conflicts with a popular type

Signed-off-by: Dan Mick <dan.mick@inktank.com>

Add NO_VERSION to avoid rebuilding ceph_ver.h and relinking

Signed-off-by: Dan Mick <dan.mick@inktank.com>

Makefiles: remove libkeyutils from every binary except two

Only rbd and mount_ceph need secret.c, and only secret.c needs libkeyutils;
remove it from LIBCOMMON_DEPS so it's not a dependency for everything,
remove secret.c from libcommon.a, and add it to mount.ceph/rbd's sources;
add LIBKEYID_LIB to mount.ceph/rbd's LDADD.

Signed-off-by: Dan Mick <dan.mick@inktank.com>

Merge pull request #1497 from dachary/wip-jerasure

remove gf-complete / jerasure sub modules

Reviewed-by: Sage Weil <sage@inktank.com>

remove gf-complete / jerasure sub modules

and add the source files in the tree.

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #1494 from dachary/wip-gitmodule-mirrors

build: add gf-complete/jerasure to gitmodule_mirrors

ReplicatedPG::do_op: delay if snapdir is unreadable

Since all we really need on a snapdir is the context, we really only
need it to be !missing.  However, it might become !missing before it
becomes !unreadable.  That allows ops to end up in the
waiting_for_degraded queue before one in waiting_for_unreadable is
woken, which allows the ops to be reordered.  Rather than reintroduce an
extra waiting_for_missing queue, simply require !unreadable for snapdir
(which implies !misssing).

Fixes: #7777
Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge pull request #1125 from ceph/wip-mon-docs

wip-mon-docs: Better explain required number of monitors & how to troubleshoot a monitor

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
Reviewed-by: Loic Dachary <loic@dachary.org>

mds/Locker: fix null deref on cap import

The session Connection* may be NULL.

Fixes: #7708
Signed-off-by: Sage Weil <sage@inktank.com>

build: add gf-complete/jerasure to gitmodule_mirrors

Signed-off-by: Loic Dachary <loic@dachary.org>

osd: dump priority queue state on dequeue at level 30

Signed-off-by: Sage Weil <sage@inktank.com>

osd: fix OpWQ dump locking

The pqueue is protected by the wq lock, not by qlock; for example, see
OpWQ::_enqueue.  qlock protects the pg_for_processing map only.

Fixes: #7735
Signed-off-by: Sage Weil <sage@inktank.com>

common/PrioritizedQueue: include first item's cost in dump

Signed-off-by: Sage Weil <sage@inktank.com>

common/PrioritizedQueue: constify a few things

Signed-off-by: Sage Weil <sage@inktank.com>

common/PrioritizedQueue: fix remove_by_class() corner case

If i is the first entry, then setting cur = begin() sets us up to point at
something that we are about to delete.  Move the check to the end to avoid
this.

Backport: emperor, dumpling
Signed-off-by: Sage Weil <sage@inktank.com>

doc/dev/cache-pool: describe the tiering agent

Signed-off-by: Sage Weil <sage@inktank.com>

PG::start_peering_interval: always send_notify if !primary

Otherwise, we might get into a situation where the primary
forgets about a stray pg.  This is simpler and does not
increase the number of notifies by much.

Fixes: #7733
Signed-off-by: Samuel Just <sam.just@inktank.com>

PG::find_best_info: fix log_tail component

The previous logic should have kept the current best info if it found a
replica which best could log-recover, but p couldn't.  However, the
continue in that loop advanced the inner loop instead of the outer loop
allowing the primary case to take over in cases where best had a longer
tail.  Instead, we will prefer the longer tail regardless of the other
infos to simplify the logic.

Fixes: #7755
Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge pull request #1491 from dachary/wip-submodules

erasure-code: jerasure and gf-complete moved to ceph namespace

Reviewed-by: Alfredo Deza <alfredo.deza@inktank.com>

erasure-code: jerasure and gf-complete moved to ceph namespace

for maintenance purposes

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #1490 from ceph/wip-6806

mon: Properly handle errors from 'cmd_getval()' when needed

Reviewed-by: Sage Weil <sage@inktank.com>

mon: OSDMonitor: don't rely on client-side validation for command arguments

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon: Monitor: handle invalid 'quorum' command argument

Don't rely on client side validation of possible command arguments.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon: Properly handle errors from 'cmd_getval()' when needed

Not handling the error return from cmd_getval() may leave uninitialzied
values, which can cause issues, specially with non-string values.

Fixes: 6806
Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

Merge pull request #1489 from dachary/wip-7757

erasure-code: disable SSE extensions

Reviewed-by: Christophe Courtaut <christophe.courtaut@gmail.com>

erasure-code: disable SSE extensions

Until jerasure-2 / gf-complete is able to detect them at runtime.

http://tracker.ceph.com/issues/7757 refs #7757

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #1488 from dachary/wip-jerasure-2

erasure-code: update jerasure v2 submodule

Reviewed-by: Christophe Courtaut <christophe.courtaut@gmail.com>

erasure-code: update gf-complete v1 submodule

Point to https://bitbucket.org/dachary/gf-complete/branch/v1 which
aggregates the pending pull requests required by Ceph while they are
reviewed.

Signed-off-by: Loic Dachary <loic@dachary.org>

erasure-code: update jerasure v2 submodule

to include the latest compilation warning fixes.

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #1384 from dachary/wip-7599

upgrade jerasure to version 2

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1486 from ceph/wip-cli-err

ceph.in: Better error on bad arg to 'tell'

Reviewed-by: Dan Mick <dan.mick@inktank.com>

Merge pull request #1479 from ceph/wip-rados-outfile

tools/rados: Allow binary file output of omap data

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1480 from ceph/wip-remotebit-dump

mds: Fix remotebit::dump for less common types

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1477 from dachary/wip-erasure-code-profiles

replace properties with erasure code profiles

Reviewed-by: Sage Weil <sage@inktank.com>