erasure-code: fix BlaumRoth sanity check on w

For the BlaumRoth technique, w+1 must be a prime. This sanity check is
present in the sample code from the jerasure library but was not
implemented in the jerasure plugin.

The default value for w is also changed to match this constraint. The
unit tests is changed to always use the default w instead of forcing it
to w.

http://tracker.ceph.com/issues/6754 Fixes: #6754

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

erasure-code: refactor jerasure Liberation parameters checks

Create a virtual function to check for each parameter for Liberation
and its derivations Liber8tion and BlaumRoth. The code is moved around
but is unmodified.

A function reseting the parameters to sane defaults when an error
occurs is introduced. It is slightly different from the code that was
duplicated in Liberation and BlaumRoth. In both cases it now resets w.
It is a noop for Liber8tion because the user cannot modify this
parameter which is always set to the default.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

Merge pull request #2305 from majianpeng/fix5

Make ceph_erasure_code_benchmark more robust.

Reviewed-by: Loic Dachary <loic-201408@dachary.org>

erasure-code: Avoid k/m that we set are not equal in erasure-plugin.

Because erasure-plugin has default k/m and can autotune if k or m
invalid. Check k/m they are the same as we want.

Signed-off-by: Ma Jianpeng <jianpeng.ma@intel.com>

Merge pull request #2378 from swamireddy/wip-orgnaizationmap

mailmap updates

Reviewed-by: Loic Dachary <loic-201408@dachary.org>

 mailmap: M Ranga Swami Reddy affiliation

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>

mailmap: John Spray name normalization

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>
Reviewed-by: John Spray <jspray@redhat.com>

mailmap: Marco Garcês affiliation

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>
Reviewed-by: Marco Garcês <marco@garces.cc>

mailmap: Erik Logtenberg affiliation

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>
Reviewed-by: Erik Logtenberg <erik@logtenberg.eu>

mailmap: François Lafont affiliation

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>

Merge pull request #2422 from ceph/wip-6494

Enforce cache size on read requests

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

doc: Fixed broken links and clarified terms for new authentication docs.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Minor cleanup.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

Merge pull request #2424 from ceph/wip-9366

lttng: Fix 'make tag' when configured --without-lttng

Merge pull request #2390 from ceph/wip-9311

Parallelize RBD import/export

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #2426 from ceph/wip-doc-qemu

doc: Added more robust usage example for uid and conf for QEMU.

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge remote-tracking branch 'gh/next'

doc/release-notes: v0.85

Signed-off-by: Sage Weil <sage@redhat.com>

doc: Added more robust usage example for uid and conf for QEMU.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Clean up syntax to suppress warnings.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Clean up syntax.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

Merge pull request #2421 from ceph/wip-paxos-dout

mon/Paxos: don't spam log with is_readable at dout level 1

Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>

lttng: Fix 'make tag' when configured --without-lttng

Signed-off-by: Adam Crume <adamcrume@gmail.com>

mon/Paxos: make is_readable debug output show result

Signed-off-by: Sage Weil <sage@redhat.com>

mon/Paxos: don't spam log with is_readable at dout level 1

Backport: firefly, dumpling
Reported-by: Aanchal Agrawal <Aanchal.Agrawal@sandisk.com>
Signed-off-by: Sage Weil <sage@redhat.com>

0.85

Merge pull request #2395 from ceph/wip-9282

mds: warn clients which aren't revoking caps

Merge pull request #2391 from ceph/wip-mds-lock

Wip mds lock

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #2404 from ceph/wip-9358

osdc/Journaler: fix try_read_entry()/wait_for_readable() locking

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #2365 from infernix/centos5

Include types.h after stdint.h to accommodate for older distributions.

Reviewed-by: Loic Dachary <loic-201408@dachary.org>

Include types.h after stdint.h to accomodate for older distributions. This fixes compilation on CentOS 5.

Signed-off-by: Gerben Meijer <infernix@gmail.com>

Merge pull request #2415 from johnkeepmoving/fix-off-error-in-rbd.cc

correct error of repeatedly adding io_size to off in do_bench_write

Reviewed-by: Loic Dachary <loic-201408@dachary.org>

correct error of repeatedly adding io_size to off in do_bench_write

Signed-off-by: jiangheng <jiangheng0511@gmail.com>

Enforce cache size on read requests

In-flight cache reads were not previously counted against
new cache read requests, which could result in very large
cache usage.  This effect is most noticeable when writing
small chunks to a cloned image since each write requires
a full object read from the parent.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #2396 from ceph/wip-SCA-20140902

SCA and Coverity fixes

Reviewed-by: Loic Dachary <loic@dachary.org>

Merge pull request #2398 from xiaoxichen/refactor_lrc_crush

Refactor ErasureCodeLrc::create_ruleset

Reviewed-by: Loic Dachary <loic@dachary.org>

Merge pull request #2410 from dachary/wip-9368

erasure-code: init function is extern "C"

erasure-code: init function is extern "C"

Otherwise it is not found and the real cause obscured by the test
setup. The thread returns immediately and the timeout test sometime
fails.

http://tracker.ceph.com/issues/9368 Fixes: #9368

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

doc: Update logging path to reflect log rotate script default path.

Fixes: #9351
Signed-off-by: John Wilkins <john.wilkins@inktank.com>

Merge pull request #2407 from ceph/wip-formatter-va

formatter: clean up dump_format()

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #2408 from theanalyst/doc/contributing

doc: Add a contributing file at the root of repo

Reviewed-by: Loic Dachary <loic@dachary.org>

doc: Add a contributing file at the root of repo

Github shows up a `guidelines for contributing` while opening a new pull
request for any repository with a CONTRIBUTING file at the repo
root, which allows for a quick overview to contribute to the
repository.

Currently this file just links to the Submitting Patches and the
documentation start guide.

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #2406 from alram/master

rgw: add .log to default log path

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add .log to default log path

Fixes: #9353
Signed-off-by: Alexandre Marangone <alexandre.marangone@inktank.com>

formatter: clean up dump_format()

Create a common dump_format_va() function, and make all the different
variants call it.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #2329 from theanalyst/f/s3_bucket_loc

WIP: S3: Implementation for S3 Get Bucket Location

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #2405 from dachary/wip-workunits

qa: scripts to help test ceph-qa-suite

Reviewed-by: Cheng Cheng <ccheng.leo@gmail.com>

qa: scripts to help test ceph-qa-suite

This workunit will be used by tests as a placeholder that always return
true. This is helpful in tests when a script from the qa/workunits
directory is mandatory but we do not care about testing anything.  For
an example of how it can be used, check
https://github.com/ceph/ceph-qa-suite/pull/120

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

osdc/Journaler: fix try_read_entry()/wait_for_readable() locking

These two functions should be protected by the Journal::mutex. Furthermore,
wait_for_readable() is racy. The journal may have already become readable
when it is called.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: warn clients which aren't revoking caps

Add a list to Locker to track revoking caps. print a warning message if client does not
release caps within the given time.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

rgw: Implementation for S3 Get Bucket Location

S3 API supports getting the location for a bucket, which gives out one
of those geographic zones (US-WEST-1, EU for eg). Also it returns an
empty string for the default region.
(http://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketGETlocation.html)

Since LocationConstraint corresponds to regions in our case, this API
returns the region, for the "default" region empty string is returned

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

test/formatter: Add tests for dump_format_ns

Adding basic unit test to test the new formatter class'
dump_format_ns. Since the functionality only affects XML (and other
implementations mimic dump_format exactly), tests are added for these.
`fmt.close_section()` is avoided in the tests as this calls an
assert (and there is no section to close) and this triggers a test
failure.

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

rgw_formats: dump_format_ns implementation

Since the base formatter was extended with a dump_format_ns class,
implementing this here. For now, this is exactly same as dump_format, as
the concept of ns in json is not used.

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

Formatter: add a dump_format_ns to base Formatter

`dump_format_ns` is a generic formatter to dump a simple format along
with a namespace. It is `dump_format` with an optional ns. This also
extends the XML formatter with this functionality. This allows creation
of xml tags with ns and a specified format. The JSON Format doesn't
define a ns, and here the functionality is exactly same as that of
`dump_format`

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

Refactor ErasureCodeLrc::create_ruleset

Refactor ErasureCodeLrc::create_ruleset
1.Use CrushWrapper instead of crush C-api.
2.Make ruleset_id == rule_id, consist as the behavior in
  CrushWrapper::add_simple_ruleset().

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

Merge pull request #2375 from ceph/wip-8648

mds: clear objects' dirty flags after log segment is expired

Merge pull request #2393 from dachary/wip-warning

erasure-code: warning fix

Reviewed-by: David Zafman <dzafman@redhat.com>

mds: clear objects' dirty flags after log segment is expired

When standby-replay MDS detects a log segment is expired, it should check
the expired segment's dirty lists and clear corresponding objects' dirty
bits. Otherwise these objects will be pinned in the standby-replay MDS's
cache forever.

Fixes: #8648
Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #2399 from ceph/wip-8366

doc: adds a navigational menu for docs

Reviewed-by: John Wilkins <john.wilkins@inktank.com>

doc: fix missing bracket

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

doc: attempt to get the ayni JS into all head tags

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

OSDMonitor.cc: fix potential division by zero

Fix for:

CID 1232602 (#1 of 1): Division or modulo by zero (DIVIDE_BY_ZERO)
 divide_by_zero: In expression num_pg_copies / num_osds, division
 by expression num_osds which may be zero has undefined behavior.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

objectstore/store_test.cc: fix unintentional integer overflow

CID 1232603 (#1 of 1): Unintentional integer overflow (OVERFLOW_BEFORE_WIDEN)
 overflow_before_widen: Potentially overflowing expression
 objs_per_folder * folders_range(rng) with type int (32 bits,
 signed) is evaluated using 32-bit arithmetic before being
 used in a context which expects an expression of type
 uint64_t (64 bits, unsigned). To avoid overflow, cast
 either operand to uint64_t before performing the multiplication.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

os/HashIndex.cc: fix unintentional integer overflow

CID 1232604 (#1 of 1): Unintentional integer overflow (OVERFLOW_BEFORE_WIDEN)
 overflow_before_widen: Potentially overflowing expression
 (unsigned int)abs(this->merge_threshold) * 16U * this->split_multiplier
 with type unsigned int (32 bits, unsigned) is evaluated using 32-bit
 arithmetic before being used in a context which expects an expression of
 type uint64_t const (64 bits, unsigned). To avoid overflow, cast either
 operand to uint64_t const before performing the multiplication.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mount.ceph.c: free memory before return

CID 1219610 (#1 of 1): Resource leak (RESOURCE_LEAK)
 leaked_storage: Variable buf going out of scope leaks the
  storage it points to.
 leaked_storage: Variable mount_path going out of scope leaks
  the storage it points to.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

st_rados_create_pool.cc: cleanup before return from run()

CID 739605 (#1 of 1): Resource leak (RESOURCE_LEAK)
 leaked_storage: Variable io_ctx going out of scope leaks the
 storage it points to.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rados_list_parallel.cc: cleanup before return from run()

Fix for:

CID 717112 (#1 of 1): Resource leak (RESOURCE_LEAK)
 leaked_storage: Variable io_ctx going out of scope leaks
 the storage it points to.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

erasure-code: warning fix

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

Merge pull request #2392 from majianpeng/fix1

FileStore: Remove unused code.

Reviewed-by: Loic Dachary <loic@dachary.org>

FileStore: Remove unused code.

Signed-off-by: Ma Jianpeng <jianpeng.ma@intel.com>

Merge pull request #2373 from ceph/wip-9026

client: vxattr support for stat

Reviewed-by: Greg Farnum <greg@inktank.com>

mds: don't take mds_lock when doing journal operations

Now journaler and objecter have internal locks to protect themself, there is
no need to take the mds_lock before using journaler and objecter.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

librbd.cc: add check to prevent null deref in rbd_list()

Fix for :

CID 1192467 (2 of 2): Explicit null dereferenced (FORWARD_NULL)
 var_deref_model: Passing null pointer ibuf to rbd_list,
 which dereferences it.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Parallelize RBD import/export

Use librbd aio_read/_write calls where possible to improve speed.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

PendingIO.cc: use static_cast instead of c-style cast

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

FileStore.cc: use static_cast instead of c-style cast

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Merge remote-tracking branch 'origin/next'

Merge pull request #2377 from ceph/wip-session-stats

mds: cap and lease count in `session ls` asok

Reviewed-by: Yan, Zheng <ukernel@gmail.com>

Merge pull request #2376 from ceph/wip-request-count

mds: Fix Session::get_request_count

Reviewed-by: Yan, Zheng <ukernel@gmail.com>

client: vxattr support for rstat

copy the kernel client code (which uses table to present vxattrs)

Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #2363 from ceph/wip-9285

osd/ReplicatedPG: do not evict blocked objects

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #2343 from ceph/wip-9151

Wip 9151 + 9152

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2362 from ceph/wip-9294

osd/ReplicatedPG: avoid dereferencing iterator at end()

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #2322 from ceph/wip-doc-authentication

doc: Refactored authentication docs.

Reviewed-by: John Wilkins <john.wilkins@inktank.com>

doc: Provided additional detail on daemon-specific caps syntax.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Updated Keyring settings from comments and ceph-deploy defaults.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

Merge pull request #2372 from dachary/wip-disk-thread-ioprio

documentation: document disk thread ioprio options

Reviewed-by: Volker Voigt <volker.voigt@1und1.de>

documentation: disk thread ioprio options

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

client/Client.cc: fix null pointer checks for 'in'

Cleanup checks for null pointer dereference of 'in' use a
wider code block the cover 'in'.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mon: add MDS metric metadata to health detail

This is a bit wonky because the mon health structure expects
a string, and we have a key-val structure, but it's better
to output this somehow than to have it rot as a purely internal
thing.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: add thread to progress queues outside dispatch

This speeds up processing of queued waiters.

Fixes: #9252
Signed-off-by: John Spray <john.spray@redhat.com>

mon: handle beacon health metrics in MDSMonitor

Fixes: #9151
Signed-off-by: John Spray <john.spray@redhat.com>

mds: populate health metrics in beacon

Signed-off-by: John Spray <john.spray@redhat.com>

messages: add health fields to MMDSBeacon

Signed-off-by: John Spray <john.spray@redhat.com>

mds: use HeartbeatMap to control beacons

...so that if something hogs mds_lock without
pinging the heartbeat map, we will stop sending
beacons to the mon and it will have the chance
to recognise that we are laggy.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: const getters in MDLog

Signed-off-by: John Spray <john.spray@redhat.com>

include/CompatSet: more const-ness

Signed-off-by: John Spray <john.spray@redhat.com>

mds: promote Beacon to be a Dispatcher

This allows it to handle its own MSG_MDS_BEACON
messages from the mon, outside of mds_lock.

This is less important than the sending of beacons
being outside the lock, but still nice to have and
gets all the beacon messaging in one place.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: extract beacon state from MDS lock

For clarity, give it its own class for the
state protected by the separate lock too.

Fixes: #9152
Signed-off-by: John Spray <john.spray@redhat.com>

include/CompatSet: make merge const()

With usual shenanigans to deal with [] on map
not being const-safe.

Signed-off-by: John Spray <john.spray@redhat.com>