DBObjectMap: restructure map_header_in_use locking

Tieing map_header_in_use to a _Header is a mistake since ownership of
the map_header_in_use can move (clone).  Instead, grab the
map_header_in_use entry at the top of each call chain and release at the
end.  To facilitate that, we introduce a MapHeaderLock object and
require that it be passed to the various map header manipulation
methods.

Fixes: #9326
Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge pull request #2454 from athanatos/wip-9269

FileStore: report l_os_j_lat as commit latency

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2453 from athanatos/wip-9220

Objecter::_recalc_linger_op: resend for any acting set change

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2443 from ceph/wip-9241

osdc/Objecter: drop bad session nref assert

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #2293 from ceph/wip-hitset-bytes

osd: improve agent calculation by factoring out hit_set bytes used properly

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #2451 from ceph/wip-osdc-leak

osdc/Objecter: fix leak of MStatfsReply

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #2447 from reclosedev/s3_colon_in_access_key

[rgw][s3] Allow colon ':' in access key

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

[rgw][s3] Allow colon ':' in access key

When access key contains ':', e.g. `some_info:for_user',
authorization header looks like:

"AWS some_info:for_user:request_signature"

so `auth_str.find(':')` result in auth_id = "some_info",
auth_sign = "for_user:request_signature".

auth_str.rfind(':') solves this issue.

Signed-off-by: Roman Haritonov <reclosedev@gmail.com>

osdc/Objecter: fix leak of MStatfsReply

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2448 from ceph/wip-replay-locking

mds: fix replay locking

Reviewed-by: John Spray <john.spray@redhat.com>

mds: fix replay locking

When replaying EImportFinish/EFragment event, the replay thread may call
MDS::queue_waiters. MDS::queue_waiters() requires its caller to hold the
mds_lock. Otherwise assert(waiter_mutex == __null || waiter_mutex->is_locked())
in Cond::Signal() will be tiggered.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

osdc/Objecter: drop bad session nref assert

This is a bad assert.  Specifically, handle_osd_op_reply may still be
holding the session ref while it is calling the completion for a previous
request.  This is safe: it is only holding the session ref after it dropped
the global map rwlock because of the per-session completion locks.  The
request in question was already marked completed by the time our thread
took the session lock.

Fixes: #9241
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2433 from ceph/wip-rbd-force-write-back

rbd should use write-back when caching is enabled

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

osd/ClassHandler: fix build

Broken by 70ce400a8b4e0f5a20e6ea9877784998cdbb9a2d.

Signed-off-by: Sage Weil <sage@redhat.com>

FileStore: report l_os_j_lat as commit latency

l_os_commit_lat is actually the commit cycle latency.

Fixes: #9269
Backport: firefly
Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge pull request #2441 from ceph/wip-9365

osd/ClassHandler: improve error logging

Reviewed-by: Samuel Just <sam.just@inktank.com>

osd/ClassHandler: improve error logging

Fixes: #9365
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2412 from dachary/wip-9370-flush-logs

tests: flush logs before grepping them

Merge pull request #2434 from dachary/wip-9381-erasure-code-rpm

packaging: add to RPM packages isa and lrc

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2427 from ceph/wip-9362

librados: do not write to user buffer after timeout

Reviewed-by: Greg Farnum <greg@inktank.com>

Merge pull request #2437 from athanatos/wip-9339

ReplicatedPG: create max hitset size

Reviewed-by: Sage Weil <sage@redhat.com>

osdc/Objecter: revoke rx_buffer on op_cancel

If we cancel a read, revoke the rx buffers to avoid a use-after-free and/or
other undefined badness by using user buffers that may no longer be
present.

Fixes: #9362
Backport: firefly, dumpling
Reported-by: Matthias Kiefer <matthias.kiefer@1und1.de>
Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_rados_api_io: add read timeout test

Verify we don't receive data after a timeout.

Based on reproducer for #9362 written by
Matthias Kiefer <matthias.kiefer@1und1.de>.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_rados_api_*: expose nspace

Signed-off-by: Sage Weil <sage@redhat.com>

Objecter::_recalc_linger_op: resend for any acting set change

Fixes: #9220
Backport: firefly
Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge pull request #2439 from athanatos/wip-9054

Wip 9054

Reviewed-by: Sage Weil <sage@redhat.com>

ReplicatedPG:start_flush send a second delete

Suppose we start with the following in the cache pool:

30:[29,21,20,15,10,4]:[22(21), 15(15,10), 4(4)]+head

The object doesn't exist at 29 or 20.

First, we flush 4 leaving the backing pool with:

3:[]+head

Then, we begin to flush 15 with a delete with snapc 4:[4] leaving the
backing pool with:

4:[4]:[4(4)]

Then, we finish flushing 15 with snapc 9:[4] with leaving the backing
pool with:

9:[4]:[4(4)]+head

Next, snaps 10 and 15 are removed causing clone 10 to be removed leaving
the cache with:

30:[29,21,20,4]:[22(21),4(4)]+head

We next begin to flush 22 by sending a delete with snapc 4(4) since
prev_snapc is 4 <---------- here is the bug

The backing pool ignores this request since 4 < 9 (ORDERSNAP) leaving it
with:

9:[4]:[4(4)]

Then, we complete flushing 22 with snapc 19:[4] leaving the backing pool
with:

19:[4]:[4(4)]+head

Then, we begin to flush head by deleting with snapc 22:[21,20,4] leaving
the backing pool with:

22[21,20,4]:[22(21,20), 4(4)]

Finally, we flush head leaving the backing pool with:

30:[29,21,20,4]:[22(21*,20*),4(4)]+head

When we go to flush clone 22, all we know is that 22 is dirty, has snaps
[21], and 4 is clean. As part of flushing 22, we need to do two things:
1) Ensure that the current head is cloned as cloneid 4 with snaps [4] by
sending a delete at snapc 4:[4].
2) Flush the data at snap sequence < 21 by sending a copyfrom with snapc
20:[20,4].

Unfortunately, it is possible that 1, 1&2, or 1 and part of the flush
process for some other now non-existent clone have already been
performed.  Because of that, between 1) and 2), we need to send
a second delete ensuring that the object does not exist at 20.

Fixes: #9054
Backport: firefly
Related: 66c7439ea0888777b5cfc08bcb0fbd7bfd8653c3
Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge pull request #2256 from ceph/wip-9005

doc: add calamari to the api/index section.

Reviewed-by: John Wilkins <jowilkin@redhat.com>

Revert "ReplicatedPG:start_flush send a second delete"

This reverts commit a525bf3190a712fdceb018267a0aa08e90c44e62.

Conflicts:
src/osd/ReplicatedPG.cc

Revert "osd/ReplicatedPG: avoid dereferencing iterator at end()"

This reverts commit ded1cf4a93d15ca3be312d296c3ab956109587e3.

ReplicatedPG: create max hitset size

Otherwise, hit_set_create could create an unbounded size hitset
object.

Fixes: #9339
Backport: firefly
Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge pull request #2436 from athanatos/wip-9293

PG::init: clear rollback info for backfill as well

Merge pull request #2435 from athanatos/wip-9179

PG: wait until we've build the missing set to discover_all_missing

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2430 from yuyuyu101/kvstore-scrub-inconsistence

Kvstore scrub inconsistence

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2429 from yuyuyu101/kvstore-fix-superblock

KeyValueStore: Fix upgrade from non-exist superblock OSD version error

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2428 from ceph/wip-yasm

fix yasm checks for x32

Reviewed-by: Daniel Schepler <dschepler@gmail.com>

packaging: add all erasure code plugins to RPM packages

It means distributing a few plugins that are only used for unit testing
but it does not use much disk space and this is otherwise harmless.

Explicitly listing which plugins are to be installed is problematic
because some of them (isa for now and maybe more later) are not
available for all architectures. Properly maintaining the list of
plugins to install would therefore mean exactly matching which
architecture has which plugins.

http://tracker.ceph.com/issues/9381 Fixes: #9381

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

Merge pull request #2305 from majianpeng/fix5

Make ceph_erasure_code_benchmark more robust.

Reviewed-by: Loic Dachary <loic-201408@dachary.org>

erasure-code: Avoid k/m that we set are not equal in erasure-plugin.

Because erasure-plugin has default k/m and can autotune if k or m
invalid. Check k/m they are the same as we want.

Signed-off-by: Ma Jianpeng <jianpeng.ma@intel.com>

KeyValueStore: Fix scrub causing inconsistence state

Fix #8589
Backport #Firefly
Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

ObjectStore Test: Add test for simulating scrub behavior

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

KeyValueStore: Fix upgrade from non-exist superblock OSD version error

If user upgrade OSD from Firefly or lower than 0.84, it won't get valid
keyvaluestore backend.

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #2378 from swamireddy/wip-orgnaizationmap

mailmap updates

Reviewed-by: Loic Dachary <loic-201408@dachary.org>

 mailmap: M Ranga Swami Reddy affiliation

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>

mailmap: John Spray name normalization

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>
Reviewed-by: John Spray <jspray@redhat.com>

mailmap: Marco Garcês affiliation

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>
Reviewed-by: Marco Garcês <marco@garces.cc>

mailmap: Erik Logtenberg affiliation

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>
Reviewed-by: Erik Logtenberg <erik@logtenberg.eu>

mailmap: François Lafont affiliation

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>

Merge pull request #2422 from ceph/wip-6494

Enforce cache size on read requests

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

configure: do not try yasm on x32

Signed-off-by: Sage Weil <sage@redhat.com>

doc: Fixed broken links and clarified terms for new authentication docs.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Minor cleanup.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

Merge pull request #2424 from ceph/wip-9366

lttng: Fix 'make tag' when configured --without-lttng

debian: only B-R yasm on amd64

Make yasm dependency amd64 only, it isn?t used elsewhere
but breaks x32 (which is mis-detected as amd64)

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2390 from ceph/wip-9311

Parallelize RBD import/export

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #2426 from ceph/wip-doc-qemu

doc: Added more robust usage example for uid and conf for QEMU.

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge remote-tracking branch 'gh/next'

doc/release-notes: v0.85

Signed-off-by: Sage Weil <sage@redhat.com>

doc: Added more robust usage example for uid and conf for QEMU.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Clean up syntax to suppress warnings.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Clean up syntax.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

Merge pull request #2421 from ceph/wip-paxos-dout

mon/Paxos: don't spam log with is_readable at dout level 1

Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>

lttng: Fix 'make tag' when configured --without-lttng

Signed-off-by: Adam Crume <adamcrume@gmail.com>

mon/Paxos: make is_readable debug output show result

Signed-off-by: Sage Weil <sage@redhat.com>

mon/Paxos: don't spam log with is_readable at dout level 1

Backport: firefly, dumpling
Reported-by: Aanchal Agrawal <Aanchal.Agrawal@sandisk.com>
Signed-off-by: Sage Weil <sage@redhat.com>

0.85

Merge pull request #2395 from ceph/wip-9282

mds: warn clients which aren't revoking caps

Merge pull request #2391 from ceph/wip-mds-lock

Wip mds lock

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #2404 from ceph/wip-9358

osdc/Journaler: fix try_read_entry()/wait_for_readable() locking

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #2365 from infernix/centos5

Include types.h after stdint.h to accommodate for older distributions.

Reviewed-by: Loic Dachary <loic-201408@dachary.org>

Include types.h after stdint.h to accomodate for older distributions. This fixes compilation on CentOS 5.

Signed-off-by: Gerben Meijer <infernix@gmail.com>

rbd should use write-back when caching is enabled

librbd now defaults to write-through until the first flush
is received.  For rbd, force the use of write-through.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #2415 from johnkeepmoving/fix-off-error-in-rbd.cc

correct error of repeatedly adding io_size to off in do_bench_write

Reviewed-by: Loic Dachary <loic-201408@dachary.org>

correct error of repeatedly adding io_size to off in do_bench_write

Signed-off-by: jiangheng <jiangheng0511@gmail.com>

Enforce cache size on read requests

In-flight cache reads were not previously counted against
new cache read requests, which could result in very large
cache usage.  This effect is most noticeable when writing
small chunks to a cloned image since each write requires
a full object read from the parent.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #2396 from ceph/wip-SCA-20140902

SCA and Coverity fixes

Reviewed-by: Loic Dachary <loic@dachary.org>

Merge pull request #2398 from xiaoxichen/refactor_lrc_crush

Refactor ErasureCodeLrc::create_ruleset

Reviewed-by: Loic Dachary <loic@dachary.org>

tests: flush logs before grepping them

Otherwise the test races with the daemon writing the logs and can
sometime fail.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

Merge pull request #2410 from dachary/wip-9368

erasure-code: init function is extern "C"

erasure-code: init function is extern "C"

Otherwise it is not found and the real cause obscured by the test
setup. The thread returns immediately and the timeout test sometime
fails.

http://tracker.ceph.com/issues/9368 Fixes: #9368

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

doc: Update logging path to reflect log rotate script default path.

Fixes: #9351
Signed-off-by: John Wilkins <john.wilkins@inktank.com>

Merge pull request #2407 from ceph/wip-formatter-va

formatter: clean up dump_format()

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #2408 from theanalyst/doc/contributing

doc: Add a contributing file at the root of repo

Reviewed-by: Loic Dachary <loic@dachary.org>

doc: Add a contributing file at the root of repo

Github shows up a `guidelines for contributing` while opening a new pull
request for any repository with a CONTRIBUTING file at the repo
root, which allows for a quick overview to contribute to the
repository.

Currently this file just links to the Submitting Patches and the
documentation start guide.

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #2406 from alram/master

rgw: add .log to default log path

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add .log to default log path

Fixes: #9353
Signed-off-by: Alexandre Marangone <alexandre.marangone@inktank.com>

formatter: clean up dump_format()

Create a common dump_format_va() function, and make all the different
variants call it.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #2329 from theanalyst/f/s3_bucket_loc

WIP: S3: Implementation for S3 Get Bucket Location

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #2405 from dachary/wip-workunits

qa: scripts to help test ceph-qa-suite

Reviewed-by: Cheng Cheng <ccheng.leo@gmail.com>

qa: scripts to help test ceph-qa-suite

This workunit will be used by tests as a placeholder that always return
true. This is helpful in tests when a script from the qa/workunits
directory is mandatory but we do not care about testing anything.  For
an example of how it can be used, check
https://github.com/ceph/ceph-qa-suite/pull/120

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

osdc/Journaler: fix try_read_entry()/wait_for_readable() locking

These two functions should be protected by the Journal::mutex. Furthermore,
wait_for_readable() is racy. The journal may have already become readable
when it is called.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: warn clients which aren't revoking caps

Add a list to Locker to track revoking caps. print a warning message if client does not
release caps within the given time.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

rgw: Implementation for S3 Get Bucket Location

S3 API supports getting the location for a bucket, which gives out one
of those geographic zones (US-WEST-1, EU for eg). Also it returns an
empty string for the default region.
(http://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketGETlocation.html)

Since LocationConstraint corresponds to regions in our case, this API
returns the region, for the "default" region empty string is returned

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

test/formatter: Add tests for dump_format_ns

Adding basic unit test to test the new formatter class'
dump_format_ns. Since the functionality only affects XML (and other
implementations mimic dump_format exactly), tests are added for these.
`fmt.close_section()` is avoided in the tests as this calls an
assert (and there is no section to close) and this triggers a test
failure.

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

rgw_formats: dump_format_ns implementation

Since the base formatter was extended with a dump_format_ns class,
implementing this here. For now, this is exactly same as dump_format, as
the concept of ns in json is not used.

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

Formatter: add a dump_format_ns to base Formatter

`dump_format_ns` is a generic formatter to dump a simple format along
with a namespace. It is `dump_format` with an optional ns. This also
extends the XML formatter with this functionality. This allows creation
of xml tags with ns and a specified format. The JSON Format doesn't
define a ns, and here the functionality is exactly same as that of
`dump_format`

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

Refactor ErasureCodeLrc::create_ruleset

Refactor ErasureCodeLrc::create_ruleset
1.Use CrushWrapper instead of crush C-api.
2.Make ruleset_id == rule_id, consist as the behavior in
  CrushWrapper::add_simple_ruleset().

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

Merge pull request #2375 from ceph/wip-8648

mds: clear objects' dirty flags after log segment is expired

Merge pull request #2393 from dachary/wip-warning

erasure-code: warning fix

Reviewed-by: David Zafman <dzafman@redhat.com>

mds: clear objects' dirty flags after log segment is expired

When standby-replay MDS detects a log segment is expired, it should check
the expired segment's dirty lists and clear corresponding objects' dirty
bits. Otherwise these objects will be pinned in the standby-replay MDS's
cache forever.

Fixes: #8648
Signed-off-by: Yan, Zheng <zyan@redhat.com>