osdc/Objecter: revoke rx_buffer on op_cancel

If we cancel a read, revoke the rx buffers to avoid a use-after-free and/or
other undefined badness by using user buffers that may no longer be
present.

Fixes: #9362
Backport: firefly, dumpling
Reported-by: Matthias Kiefer <matthias.kiefer@1und1.de>
Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_rados_api_io: add read timeout test

Verify we don't receive data after a timeout.

Based on reproducer for #9362 written by
Matthias Kiefer <matthias.kiefer@1und1.de>.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_rados_api_*: expose nspace

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2436 from athanatos/wip-9293

PG::init: clear rollback info for backfill as well

Merge pull request #2435 from athanatos/wip-9179

PG: wait until we've build the missing set to discover_all_missing

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2430 from yuyuyu101/kvstore-scrub-inconsistence

Kvstore scrub inconsistence

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2429 from yuyuyu101/kvstore-fix-superblock

KeyValueStore: Fix upgrade from non-exist superblock OSD version error

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2428 from ceph/wip-yasm

fix yasm checks for x32

Reviewed-by: Daniel Schepler <dschepler@gmail.com>

Merge pull request #2305 from majianpeng/fix5

Make ceph_erasure_code_benchmark more robust.

Reviewed-by: Loic Dachary <loic-201408@dachary.org>

erasure-code: Avoid k/m that we set are not equal in erasure-plugin.

Because erasure-plugin has default k/m and can autotune if k or m
invalid. Check k/m they are the same as we want.

Signed-off-by: Ma Jianpeng <jianpeng.ma@intel.com>

KeyValueStore: Fix scrub causing inconsistence state

Fix #8589
Backport #Firefly
Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

ObjectStore Test: Add test for simulating scrub behavior

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

KeyValueStore: Fix upgrade from non-exist superblock OSD version error

If user upgrade OSD from Firefly or lower than 0.84, it won't get valid
keyvaluestore backend.

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #2378 from swamireddy/wip-orgnaizationmap

mailmap updates

Reviewed-by: Loic Dachary <loic-201408@dachary.org>

 mailmap: M Ranga Swami Reddy affiliation

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>

mailmap: John Spray name normalization

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>
Reviewed-by: John Spray <jspray@redhat.com>

mailmap: Marco Garcês affiliation

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>
Reviewed-by: Marco Garcês <marco@garces.cc>

mailmap: Erik Logtenberg affiliation

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>
Reviewed-by: Erik Logtenberg <erik@logtenberg.eu>

mailmap: François Lafont affiliation

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>

Merge pull request #2422 from ceph/wip-6494

Enforce cache size on read requests

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

configure: do not try yasm on x32

Signed-off-by: Sage Weil <sage@redhat.com>

doc: Fixed broken links and clarified terms for new authentication docs.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Minor cleanup.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

Merge pull request #2424 from ceph/wip-9366

lttng: Fix 'make tag' when configured --without-lttng

debian: only B-R yasm on amd64

Make yasm dependency amd64 only, it isn?t used elsewhere
but breaks x32 (which is mis-detected as amd64)

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2390 from ceph/wip-9311

Parallelize RBD import/export

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #2426 from ceph/wip-doc-qemu

doc: Added more robust usage example for uid and conf for QEMU.

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge remote-tracking branch 'gh/next'

doc/release-notes: v0.85

Signed-off-by: Sage Weil <sage@redhat.com>

doc: Added more robust usage example for uid and conf for QEMU.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Clean up syntax to suppress warnings.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Clean up syntax.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

Merge pull request #2421 from ceph/wip-paxos-dout

mon/Paxos: don't spam log with is_readable at dout level 1

Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>

lttng: Fix 'make tag' when configured --without-lttng

Signed-off-by: Adam Crume <adamcrume@gmail.com>

mon/Paxos: make is_readable debug output show result

Signed-off-by: Sage Weil <sage@redhat.com>

mon/Paxos: don't spam log with is_readable at dout level 1

Backport: firefly, dumpling
Reported-by: Aanchal Agrawal <Aanchal.Agrawal@sandisk.com>
Signed-off-by: Sage Weil <sage@redhat.com>

0.85

Merge pull request #2395 from ceph/wip-9282

mds: warn clients which aren't revoking caps

Merge pull request #2391 from ceph/wip-mds-lock

Wip mds lock

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #2404 from ceph/wip-9358

osdc/Journaler: fix try_read_entry()/wait_for_readable() locking

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #2365 from infernix/centos5

Include types.h after stdint.h to accommodate for older distributions.

Reviewed-by: Loic Dachary <loic-201408@dachary.org>

Include types.h after stdint.h to accomodate for older distributions. This fixes compilation on CentOS 5.

Signed-off-by: Gerben Meijer <infernix@gmail.com>

Merge pull request #2415 from johnkeepmoving/fix-off-error-in-rbd.cc

correct error of repeatedly adding io_size to off in do_bench_write

Reviewed-by: Loic Dachary <loic-201408@dachary.org>

correct error of repeatedly adding io_size to off in do_bench_write

Signed-off-by: jiangheng <jiangheng0511@gmail.com>

Enforce cache size on read requests

In-flight cache reads were not previously counted against
new cache read requests, which could result in very large
cache usage.  This effect is most noticeable when writing
small chunks to a cloned image since each write requires
a full object read from the parent.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #2396 from ceph/wip-SCA-20140902

SCA and Coverity fixes

Reviewed-by: Loic Dachary <loic@dachary.org>

Merge pull request #2398 from xiaoxichen/refactor_lrc_crush

Refactor ErasureCodeLrc::create_ruleset

Reviewed-by: Loic Dachary <loic@dachary.org>

Merge pull request #2410 from dachary/wip-9368

erasure-code: init function is extern "C"

erasure-code: init function is extern "C"

Otherwise it is not found and the real cause obscured by the test
setup. The thread returns immediately and the timeout test sometime
fails.

http://tracker.ceph.com/issues/9368 Fixes: #9368

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

doc: Update logging path to reflect log rotate script default path.

Fixes: #9351
Signed-off-by: John Wilkins <john.wilkins@inktank.com>

Merge pull request #2407 from ceph/wip-formatter-va

formatter: clean up dump_format()

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #2408 from theanalyst/doc/contributing

doc: Add a contributing file at the root of repo

Reviewed-by: Loic Dachary <loic@dachary.org>

doc: Add a contributing file at the root of repo

Github shows up a `guidelines for contributing` while opening a new pull
request for any repository with a CONTRIBUTING file at the repo
root, which allows for a quick overview to contribute to the
repository.

Currently this file just links to the Submitting Patches and the
documentation start guide.

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #2406 from alram/master

rgw: add .log to default log path

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add .log to default log path

Fixes: #9353
Signed-off-by: Alexandre Marangone <alexandre.marangone@inktank.com>

formatter: clean up dump_format()

Create a common dump_format_va() function, and make all the different
variants call it.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #2329 from theanalyst/f/s3_bucket_loc

WIP: S3: Implementation for S3 Get Bucket Location

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #2405 from dachary/wip-workunits

qa: scripts to help test ceph-qa-suite

Reviewed-by: Cheng Cheng <ccheng.leo@gmail.com>

qa: scripts to help test ceph-qa-suite

This workunit will be used by tests as a placeholder that always return
true. This is helpful in tests when a script from the qa/workunits
directory is mandatory but we do not care about testing anything.  For
an example of how it can be used, check
https://github.com/ceph/ceph-qa-suite/pull/120

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

osdc/Journaler: fix try_read_entry()/wait_for_readable() locking

These two functions should be protected by the Journal::mutex. Furthermore,
wait_for_readable() is racy. The journal may have already become readable
when it is called.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: warn clients which aren't revoking caps

Add a list to Locker to track revoking caps. print a warning message if client does not
release caps within the given time.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

rgw: Implementation for S3 Get Bucket Location

S3 API supports getting the location for a bucket, which gives out one
of those geographic zones (US-WEST-1, EU for eg). Also it returns an
empty string for the default region.
(http://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketGETlocation.html)

Since LocationConstraint corresponds to regions in our case, this API
returns the region, for the "default" region empty string is returned

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

test/formatter: Add tests for dump_format_ns

Adding basic unit test to test the new formatter class'
dump_format_ns. Since the functionality only affects XML (and other
implementations mimic dump_format exactly), tests are added for these.
`fmt.close_section()` is avoided in the tests as this calls an
assert (and there is no section to close) and this triggers a test
failure.

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

rgw_formats: dump_format_ns implementation

Since the base formatter was extended with a dump_format_ns class,
implementing this here. For now, this is exactly same as dump_format, as
the concept of ns in json is not used.

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

Formatter: add a dump_format_ns to base Formatter

`dump_format_ns` is a generic formatter to dump a simple format along
with a namespace. It is `dump_format` with an optional ns. This also
extends the XML formatter with this functionality. This allows creation
of xml tags with ns and a specified format. The JSON Format doesn't
define a ns, and here the functionality is exactly same as that of
`dump_format`

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

Refactor ErasureCodeLrc::create_ruleset

Refactor ErasureCodeLrc::create_ruleset
1.Use CrushWrapper instead of crush C-api.
2.Make ruleset_id == rule_id, consist as the behavior in
  CrushWrapper::add_simple_ruleset().

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

Merge pull request #2375 from ceph/wip-8648

mds: clear objects' dirty flags after log segment is expired

Merge pull request #2393 from dachary/wip-warning

erasure-code: warning fix

Reviewed-by: David Zafman <dzafman@redhat.com>

mds: clear objects' dirty flags after log segment is expired

When standby-replay MDS detects a log segment is expired, it should check
the expired segment's dirty lists and clear corresponding objects' dirty
bits. Otherwise these objects will be pinned in the standby-replay MDS's
cache forever.

Fixes: #8648
Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #2399 from ceph/wip-8366

doc: adds a navigational menu for docs

Reviewed-by: John Wilkins <john.wilkins@inktank.com>

doc: fix missing bracket

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

doc: attempt to get the ayni JS into all head tags

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

OSDMonitor.cc: fix potential division by zero

Fix for:

CID 1232602 (#1 of 1): Division or modulo by zero (DIVIDE_BY_ZERO)
 divide_by_zero: In expression num_pg_copies / num_osds, division
 by expression num_osds which may be zero has undefined behavior.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

objectstore/store_test.cc: fix unintentional integer overflow

CID 1232603 (#1 of 1): Unintentional integer overflow (OVERFLOW_BEFORE_WIDEN)
 overflow_before_widen: Potentially overflowing expression
 objs_per_folder * folders_range(rng) with type int (32 bits,
 signed) is evaluated using 32-bit arithmetic before being
 used in a context which expects an expression of type
 uint64_t (64 bits, unsigned). To avoid overflow, cast
 either operand to uint64_t before performing the multiplication.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

os/HashIndex.cc: fix unintentional integer overflow

CID 1232604 (#1 of 1): Unintentional integer overflow (OVERFLOW_BEFORE_WIDEN)
 overflow_before_widen: Potentially overflowing expression
 (unsigned int)abs(this->merge_threshold) * 16U * this->split_multiplier
 with type unsigned int (32 bits, unsigned) is evaluated using 32-bit
 arithmetic before being used in a context which expects an expression of
 type uint64_t const (64 bits, unsigned). To avoid overflow, cast either
 operand to uint64_t const before performing the multiplication.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mount.ceph.c: free memory before return

CID 1219610 (#1 of 1): Resource leak (RESOURCE_LEAK)
 leaked_storage: Variable buf going out of scope leaks the
  storage it points to.
 leaked_storage: Variable mount_path going out of scope leaks
  the storage it points to.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

st_rados_create_pool.cc: cleanup before return from run()

CID 739605 (#1 of 1): Resource leak (RESOURCE_LEAK)
 leaked_storage: Variable io_ctx going out of scope leaks the
 storage it points to.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rados_list_parallel.cc: cleanup before return from run()

Fix for:

CID 717112 (#1 of 1): Resource leak (RESOURCE_LEAK)
 leaked_storage: Variable io_ctx going out of scope leaks
 the storage it points to.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

erasure-code: warning fix

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

Merge pull request #2392 from majianpeng/fix1

FileStore: Remove unused code.

Reviewed-by: Loic Dachary <loic@dachary.org>

FileStore: Remove unused code.

Signed-off-by: Ma Jianpeng <jianpeng.ma@intel.com>

Merge pull request #2373 from ceph/wip-9026

client: vxattr support for stat

Reviewed-by: Greg Farnum <greg@inktank.com>

mds: don't take mds_lock when doing journal operations

Now journaler and objecter have internal locks to protect themself, there is
no need to take the mds_lock before using journaler and objecter.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

librbd.cc: add check to prevent null deref in rbd_list()

Fix for :

CID 1192467 (2 of 2): Explicit null dereferenced (FORWARD_NULL)
 var_deref_model: Passing null pointer ibuf to rbd_list,
 which dereferences it.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Parallelize RBD import/export

Use librbd aio_read/_write calls where possible to improve speed.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

PendingIO.cc: use static_cast instead of c-style cast

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

FileStore.cc: use static_cast instead of c-style cast

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Merge remote-tracking branch 'origin/next'

Merge pull request #2377 from ceph/wip-session-stats

mds: cap and lease count in `session ls` asok

Reviewed-by: Yan, Zheng <ukernel@gmail.com>

Merge pull request #2376 from ceph/wip-request-count

mds: Fix Session::get_request_count

Reviewed-by: Yan, Zheng <ukernel@gmail.com>

client: vxattr support for rstat

copy the kernel client code (which uses table to present vxattrs)

Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #2363 from ceph/wip-9285

osd/ReplicatedPG: do not evict blocked objects

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #2343 from ceph/wip-9151

Wip 9151 + 9152

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2362 from ceph/wip-9294

osd/ReplicatedPG: avoid dereferencing iterator at end()

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #2322 from ceph/wip-doc-authentication

doc: Refactored authentication docs.

Reviewed-by: John Wilkins <john.wilkins@inktank.com>

doc: Provided additional detail on daemon-specific caps syntax.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Updated Keyring settings from comments and ceph-deploy defaults.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

Merge pull request #2372 from dachary/wip-disk-thread-ioprio

documentation: document disk thread ioprio options

Reviewed-by: Volker Voigt <volker.voigt@1und1.de>

documentation: disk thread ioprio options

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

client/Client.cc: fix null pointer checks for 'in'

Cleanup checks for null pointer dereference of 'in' use a
wider code block the cover 'in'.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>