xlist: implement copy constructor

...so that I can have a std::map of them.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: health metric for late releasing caps

Follow up on Yan Zheng's "mds: warn clients which
aren't revoking cap" to include a health metric
for this condition as well as the clog messages.

Signed-off-by: John Spray <john.spray@redhat.com>

mon: trigger transaction on MDS health changes

I think this was previously only working as a side effect
of other MDS map changes.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: add a health metric for failure to recall caps

Fixes: #9284
Signed-off-by: John Spray <john.spray@redhat.com>

mds: add state for tracking RECALL progress

To be used later for generating health metrics
for clients which are failing to promptly service
CEPH_SESSION_RECALL_STATE messages.

Signed-off-by: John Spray <john.spray@redhat.com>

xlist: implement const_iterator

Signed-off-by: John Spray <john.spray@redhat.com>

client: fix trim_caps for inodes in root

Previously client would fail to release caps for files
in the root directory in response to CEPH_SESSION_RECALL_STATE
messages.

Signed-off-by: John Spray <john.spray@redhat.com>

client: failure injection for cap release

Used for simulating a buggy client that trips
the error detection in #9282 (warn clients
which aren't revoking caps)

Signed-off-by: John Spray <john.spray@redhat.com>

client: fix potentially invalid read in trim_caps

trim_dentry can potentially free an inode, so get/put
it around the block where we use the inode's dn_set.

Signed-off-by: John Spray <john.spray@redhat.com>

client: more precise cap trimming

Two fixes:
 * Client would unlink everything it could, instead of just
   meeting its goal, because caps.size() doesn't change until
   dentries are cleaned up later.  Take account of the trimmed
   count in the while() condition to fix that.
 * Don't count the root ino as trimmed, as although it has no
   dentries (of course), we will never give up the cap.

With this change, the client will now precisely achieve the number
of caps requested in CEPH_SESSION_RECALL_STATE messages.

Signed-off-by: John Spray <john.spray@redhat.com>

client: fix crash in trim_caps

In a75af4c2, procedure was added to invalidate root's dentries
if the trimming failed to free enough caps.  This would sometimes
crash because root->dir wasn't necessarily open.

Fix by only doing it if root dir is open, though I suspect this
may not be the end of it...

Signed-off-by: John Spray <john.spray@redhat.com>

Merge remote-tracking branch 'gh/giant'

Merge pull request #2481 from sjahl/master

doc: fixes a formatting error on ops/crush-map

doc: fixes a formatting error on ops/crush-map

Signed-off-by: Stephen Jahl <stephenjahl@gmail.com>

Merge pull request #2467 from majianpeng/fix3

buffer: In rebuild_page_aligned for the last ptr is page aligned, no need call rebuild().

Reviewed-by: Loic Dachary <loic-201408@dachary.org>

Merge pull request #2478 from ceph/wip-9445

global: fix hang when segv happens inside logging code

Reviewed-by: Loic Dachary <loic-201408@dachary.org>

Merge pull request #2477 from ceph/wip-client-msg-leak

client: fix a message leak

mds: update segment references during journal rewrite

... to avoid leaving log events that reference log
segments by offsets which no longer exist.

Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 386f2d7c829422695a1b1f41bd3f17ca3eef1f61)
Reviewed-by: Greg Farnum <greg@inktank.com>

Merge pull request #2469 from ceph/wip-9427-rewrite

mds: update segment references during journal rewrite

Reviewed-by: Greg Farnum <greg@inktank.com>

log: add simple test to verify an internal SEGV doesn't hang

Test that the segv injection works.

Test that a segv while logging something doesn't hang when the signal
handlers are installed.  Note that this fails/hangs without the previous
fix.

Signed-off-by: Sage Weil <sage@redhat.com>

client: fix a message leak

Signed-off-by: John Spray <john.spray@redhat.com>

global/signal_handler: do not log if SEGV originated inside log code

Signed-off-by: Sage Weil <sage@redhat.com>

log: add Log::is_inside_log_lock()

Signed-off-by: Sage Weil <sage@redhat.com>

mds: update segment references during journal rewrite

... to avoid leaving log events that reference log
segments by offsets which no longer exist.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #2471 from ceph/wip-9446

mon: fix MDS health detail output

Reviewed-by: Sage Weil <sage@redhat.com>

mon: fix MDS health detail output

I fat fingered a couple of things here.

Signed-off-by: John Spray <john.spray@redhat.com>

buffer: Add a test for bufferlist::rebuild_page_aligned

Make the last prt of bufferlist which is page-aligned don't change anything
after rebuild_page_aligned.

Signed-off-by: Ma Jianpeng <jianpeng.ma@intel.com>

buffer: In rebuild_page_aligned for the last ptr is page aligned, no need call rebuild().

This only happen for the last ptr. Because rebuild() don't change the len
of ptr, so if last ptr isn't page-size aligned but is page aligned, the
rebuild() don't change anything.

Signed-off-by: Ma Jianpeng <jianpeng.ma@intel.com>

Merge pull request #2468 from dachary/wip-always-create-pidfile

daemons: write pid file even when told not to daemonize

Reviewed-by: Loic Dachary <loic-201408@dachary.org>
Reviewed-by: Sage Weil <sage@redhat.com>

daemons: write pid file even when told not to daemonize

systemd wants to run daemons in foreground, but daemons wouldn't write
out the pid file with -f.  Fixed.

Signed-off-by: Alexandre Oliva <oliva@gnu.org>

Merge pull request #2464 from dachary/wip-9429-bench

erasure-code: fix erasure_code_benchmark goop

Reviewed-by: Janne Grunau <j@jannau.net>

Merge pull request #2416 from xiaoxichen/make_crush_private

Change CrushWrapper::crush to private

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2450 from dachary/wip-9413-erasure-code-version-check

erasure-code: mon, osd etc. depend on the plugins

erasure-code: fix erasure_code_benchmark goop

Using a stringstream that is only displayed on error when calling the
erasure code factory, instead of cerr. The user expects the output to be
clean when there is no error.

http://tracker.ceph.com/issues/9429 Fixes: #9429

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

Merge pull request #2409 from apeters1971/wip-ec-isa-table-cache-refac-master

EC-ISA: add intelligent table cache

Reviewed-by: Loic Dachary <loic-201408@dachary.org>

doc: Fixed syntax error.

Signed-off-by: John Wilkins <jowilki@redhat.com>

doc: Updated authentication notes. Fixed syntax error.

Signed-off-by: John Wilkins <jowilki@redhat.com>

Merge pull request #2459 from ceph/wip-7934

test: Fix ceph_test_rados_watch_notify to delete the pools it creates

Reviewed-by: Loic Dachary <loic-201408@dachary.org>

Merge pull request #2463 from ceph/wip-mds-beacon

mds: a couple fixes for the beacons

Reviewed-by: John Spray <john.spray@redhat.com>

mds: sleep in progress thread if laggy and waiting_for_nolaggy waiters

If we have nolaggy waiters but are laggy we want to sleep.  Otherwise,
we will just spin and spam the log ...

Signed-off-by: Sage Weil <sage@redhat.com>

mds/Beacon: do not reconnect to mon in quick succession

Wait at least one beacon interval between mon session resets.

Fixes: #9428
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2460 from ceph/wip-client-ll-ref

client: include ll_ref when printing inode

client: include ll_ref when printing inode

Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #2444 from wonzhq/read-recency

osd: set min_read_recency_for_promote to default 1 when doing upgrade

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2449 from majianpeng/fix3

fix two bugs about perfcounter

Reviewed-by: Sage Weil <sage@redhat.com>

test: Fix ceph_test_rados_watch_notify to delete the pools it creates

Fixes: #7934
Signed-off-by: David Zafman <dzafman@redhat.com>

ReplicatedPG: Make perfcounter record the read-size for
 async-read.

Signed-off-by: Ma Jianpeng <jianpeng.ma@intel.com>

ReplicatedPG: record correctly subop for perfcounter.

In log_subop_stats, it omit to record the counter of subop.

Signed-off-by: Ma Jianpeng <jianpeng.ma@intel.com>

Merge pull request #2454 from athanatos/wip-9269

FileStore: report l_os_j_lat as commit latency

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2453 from athanatos/wip-9220

Objecter::_recalc_linger_op: resend for any acting set change

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2443 from ceph/wip-9241

osdc/Objecter: drop bad session nref assert

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #2293 from ceph/wip-hitset-bytes

osd: improve agent calculation by factoring out hit_set bytes used properly

Reviewed-by: Samuel Just <sam.just@inktank.com>

erasure-code: mon, osd etc. depend on the plugins

Since the erasure code plugin version check has been introduced,
whenever a library/binary that can load plugin needs to be recompiled,
the erasure code plugins must also be considered. If the reason for
recompiling the library/binary is a new commit, the plugins will fail to
load.

The dependency is not based on source compilation and a shared library
dependency on liberasure-code.la is added instead. This library is
uniformly used whenever a plugin is to be loaded and therefore covers
all library/binaries that need it.

http://tracker.ceph.com/issues/9413 Fixes: #9413

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

Merge pull request #2451 from ceph/wip-osdc-leak

osdc/Objecter: fix leak of MStatfsReply

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #2447 from reclosedev/s3_colon_in_access_key

[rgw][s3] Allow colon ':' in access key

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

[rgw][s3] Allow colon ':' in access key

When access key contains ':', e.g. `some_info:for_user',
authorization header looks like:

"AWS some_info:for_user:request_signature"

so `auth_str.find(':')` result in auth_id = "some_info",
auth_sign = "for_user:request_signature".

auth_str.rfind(':') solves this issue.

Signed-off-by: Roman Haritonov <reclosedev@gmail.com>

osdc/Objecter: fix leak of MStatfsReply

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2448 from ceph/wip-replay-locking

mds: fix replay locking

Reviewed-by: John Spray <john.spray@redhat.com>

EC-ISA: avoid usage of table cache lock outside the class implementation by introducing the setEncodingTable/setEncodingCoefficient methods

EC-ISA: add intelligent table cache

mds: fix replay locking

When replaying EImportFinish/EFragment event, the replay thread may call
MDS::queue_waiters. MDS::queue_waiters() requires its caller to hold the
mds_lock. Otherwise assert(waiter_mutex == __null || waiter_mutex->is_locked())
in Cond::Signal() will be tiggered.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

osd: set min_read_recency_for_promote to default 1 when doing upgrade

When upgrading from a build without the promotion on 2nd read feature,
should set min_read_recency_for_promote to the default value 1, instead
of 0.

Signed-off-by: Zhiqiang Wang <wonzhq@hotmail.com>

Change CrushWrapper::crush to private

Currently in CrushWrapper, the member "struct crush_map *crush"  is a public member,
so people can break the encapsulation and manipulate directly to the crush structure.

This is not a good practice for encapsulation and will lead to inconsistent if code
mix use the CrushWrapper API and crush C API.A simple example could be:
1.some code use crush_add_rule(C-API) to add a rule, which will not set the have_rmap flag to false in CrushWrapper
2.another code using CrushWrapper trying to look up the newly added rule by name will get a -ENOENT.

This patch move CrushWrapper::crush to private, together with three reverse map(type_rmap, name_rmap, rule_name_rmap)
and also change codes accessing the CrushWrapper::crush to make it compile.

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

osdc/Objecter: drop bad session nref assert

This is a bad assert.  Specifically, handle_osd_op_reply may still be
holding the session ref while it is calling the completion for a previous
request.  This is safe: it is only holding the session ref after it dropped
the global map rwlock because of the per-session completion locks.  The
request in question was already marked completed by the time our thread
took the session lock.

Fixes: #9241
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2433 from ceph/wip-rbd-force-write-back

rbd should use write-back when caching is enabled

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

osd/ClassHandler: fix build

Broken by 70ce400a8b4e0f5a20e6ea9877784998cdbb9a2d.

Signed-off-by: Sage Weil <sage@redhat.com>

FileStore: report l_os_j_lat as commit latency

l_os_commit_lat is actually the commit cycle latency.

Fixes: #9269
Backport: firefly
Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge pull request #2441 from ceph/wip-9365

osd/ClassHandler: improve error logging

Reviewed-by: Samuel Just <sam.just@inktank.com>

osd/ClassHandler: improve error logging

Fixes: #9365
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2412 from dachary/wip-9370-flush-logs

tests: flush logs before grepping them

Merge pull request #2434 from dachary/wip-9381-erasure-code-rpm

packaging: add to RPM packages isa and lrc

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2427 from ceph/wip-9362

librados: do not write to user buffer after timeout

Reviewed-by: Greg Farnum <greg@inktank.com>

Merge pull request #2437 from athanatos/wip-9339

ReplicatedPG: create max hitset size

Reviewed-by: Sage Weil <sage@redhat.com>

osdc/Objecter: revoke rx_buffer on op_cancel

If we cancel a read, revoke the rx buffers to avoid a use-after-free and/or
other undefined badness by using user buffers that may no longer be
present.

Fixes: #9362
Backport: firefly, dumpling
Reported-by: Matthias Kiefer <matthias.kiefer@1und1.de>
Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_rados_api_io: add read timeout test

Verify we don't receive data after a timeout.

Based on reproducer for #9362 written by
Matthias Kiefer <matthias.kiefer@1und1.de>.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_rados_api_*: expose nspace

Signed-off-by: Sage Weil <sage@redhat.com>

Objecter::_recalc_linger_op: resend for any acting set change

Fixes: #9220
Backport: firefly
Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge pull request #2439 from athanatos/wip-9054

Wip 9054

Reviewed-by: Sage Weil <sage@redhat.com>

ReplicatedPG:start_flush send a second delete

Suppose we start with the following in the cache pool:

30:[29,21,20,15,10,4]:[22(21), 15(15,10), 4(4)]+head

The object doesn't exist at 29 or 20.

First, we flush 4 leaving the backing pool with:

3:[]+head

Then, we begin to flush 15 with a delete with snapc 4:[4] leaving the
backing pool with:

4:[4]:[4(4)]

Then, we finish flushing 15 with snapc 9:[4] with leaving the backing
pool with:

9:[4]:[4(4)]+head

Next, snaps 10 and 15 are removed causing clone 10 to be removed leaving
the cache with:

30:[29,21,20,4]:[22(21),4(4)]+head

We next begin to flush 22 by sending a delete with snapc 4(4) since
prev_snapc is 4 <---------- here is the bug

The backing pool ignores this request since 4 < 9 (ORDERSNAP) leaving it
with:

9:[4]:[4(4)]

Then, we complete flushing 22 with snapc 19:[4] leaving the backing pool
with:

19:[4]:[4(4)]+head

Then, we begin to flush head by deleting with snapc 22:[21,20,4] leaving
the backing pool with:

22[21,20,4]:[22(21,20), 4(4)]

Finally, we flush head leaving the backing pool with:

30:[29,21,20,4]:[22(21*,20*),4(4)]+head

When we go to flush clone 22, all we know is that 22 is dirty, has snaps
[21], and 4 is clean. As part of flushing 22, we need to do two things:
1) Ensure that the current head is cloned as cloneid 4 with snaps [4] by
sending a delete at snapc 4:[4].
2) Flush the data at snap sequence < 21 by sending a copyfrom with snapc
20:[20,4].

Unfortunately, it is possible that 1, 1&2, or 1 and part of the flush
process for some other now non-existent clone have already been
performed.  Because of that, between 1) and 2), we need to send
a second delete ensuring that the object does not exist at 20.

Fixes: #9054
Backport: firefly
Related: 66c7439ea0888777b5cfc08bcb0fbd7bfd8653c3
Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge pull request #2256 from ceph/wip-9005

doc: add calamari to the api/index section.

Reviewed-by: John Wilkins <jowilkin@redhat.com>

Revert "ReplicatedPG:start_flush send a second delete"

This reverts commit a525bf3190a712fdceb018267a0aa08e90c44e62.

Conflicts:
src/osd/ReplicatedPG.cc

Revert "osd/ReplicatedPG: avoid dereferencing iterator at end()"

This reverts commit ded1cf4a93d15ca3be312d296c3ab956109587e3.

ReplicatedPG: create max hitset size

Otherwise, hit_set_create could create an unbounded size hitset
object.

Fixes: #9339
Backport: firefly
Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge pull request #2436 from athanatos/wip-9293

PG::init: clear rollback info for backfill as well

Merge pull request #2435 from athanatos/wip-9179

PG: wait until we've build the missing set to discover_all_missing

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2430 from yuyuyu101/kvstore-scrub-inconsistence

Kvstore scrub inconsistence

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2429 from yuyuyu101/kvstore-fix-superblock

KeyValueStore: Fix upgrade from non-exist superblock OSD version error

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2428 from ceph/wip-yasm

fix yasm checks for x32

Reviewed-by: Daniel Schepler <dschepler@gmail.com>

packaging: add all erasure code plugins to RPM packages

It means distributing a few plugins that are only used for unit testing
but it does not use much disk space and this is otherwise harmless.

Explicitly listing which plugins are to be installed is problematic
because some of them (isa for now and maybe more later) are not
available for all architectures. Properly maintaining the list of
plugins to install would therefore mean exactly matching which
architecture has which plugins.

http://tracker.ceph.com/issues/9381 Fixes: #9381

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

Merge pull request #2305 from majianpeng/fix5

Make ceph_erasure_code_benchmark more robust.

Reviewed-by: Loic Dachary <loic-201408@dachary.org>

erasure-code: Avoid k/m that we set are not equal in erasure-plugin.

Because erasure-plugin has default k/m and can autotune if k or m
invalid. Check k/m they are the same as we want.

Signed-off-by: Ma Jianpeng <jianpeng.ma@intel.com>

KeyValueStore: Fix scrub causing inconsistence state

Fix #8589
Backport #Firefly
Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

ObjectStore Test: Add test for simulating scrub behavior

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

KeyValueStore: Fix upgrade from non-exist superblock OSD version error

If user upgrade OSD from Firefly or lower than 0.84, it won't get valid
keyvaluestore backend.

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #2378 from swamireddy/wip-orgnaizationmap

mailmap updates

Reviewed-by: Loic Dachary <loic-201408@dachary.org>

 mailmap: M Ranga Swami Reddy affiliation

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>

mailmap: John Spray name normalization

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>
Reviewed-by: John Spray <jspray@redhat.com>

mailmap: Marco Garcês affiliation

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>
Reviewed-by: Marco Garcês <marco@garces.cc>

mailmap: Erik Logtenberg affiliation

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>
Reviewed-by: Erik Logtenberg <erik@logtenberg.eu>

mailmap: François Lafont affiliation

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>