WorkQueue: Taking care of potential race condition during pause()
Introduced two variables to keep track of number of threads paused
and drained during threadpool pause/drain. The pause()/drain() call
is waiting till number of pause/drain threads equals to toral number of
thread pool threads.

Signed-off-by: Somnath Roy <somnath.roy@sandisk.com>

OSD:Derived sharded queue implementation is changed
All the threadpool related stuff like stop/pause/drain etc. are not
handled by sharded queue anymore. All it is implementing are related to
processing,enqueue , signaling of waiting threads and shard queue status.
The pg ordering is been taken care of by introducing a map <pg,op> in each
shard.

Signed-off-by: Somnath Roy <somnath.roy@sandisk.com>

ceph-common: The Sharded threadpool worker logic changed
Now, the _process() of the derived queue is processing one request
at a time and the outer loop is controlled by the sharded threadpool.
The stop/pause/drain functionalities are controlled by the sharded TP.

Signed-off-by: Somnath Roy <somnath.roy@sandisk.com>

OSD: Sharded Op worker queue implementation for handling OSD ops
This is the implementation for the client of the
sharded thread pool/sharded workQ. Removed the op_wq class and now OSD
ops are going through sharded workqueue model which is used by the
sharded threadpool. Derived ShardedOpWQ implementation has a data structure
called ShardData which has it's own lock/cond and storage. ShardedOpWQ
holds a vector of that and the size of the vector is a config option.
During enqueue operation on the queue, the ops are sharded across these
ShardData based on pg hash % number of shards.
Similarly, in the _process function the sharded thread pool threads are
divided across ShardData based on thread index % number of shards

Signed-off-by: Somnath Roy <somnath.roy@sandisk.com>

ceph-common: Implementation of the sharded threadpool.
Threadpool will only be having a single work queue and internally the
work queue will be having multiple storage data structures.
Based on some logic (which is derived class implementation specific)
the work queue will shard the requests among these storage structures.
Each storage will be guarded by finer grained sunchronization
objects. Sharded threadpool threads will be assigned to work on a
shard based on some algorithm which is again derived class implementation
specific.

Signed-off-by: Somnath Roy <somnath.roy@sandisk.com>

Merge remote-tracking branch 'gh/next'

rgw: calc md5 and compare if user provided appropriate header

Fixes: #8436
Backport: firefly

This was broken in ddc2e1a8e39a5c6b9b224c3eebd1c0e762ca5782. The fix
resurrects and old check that was dropped.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1888 from ceph/wip-8472

msg_types.h: Don't use strncmp for comparing sockaddr_in.

Reviewed-by: Loic Dachary <loic@dachary.org>

msg_types.h: Don't use strncmp for comparing sockaddr_in.

Fixes: #8472
Signed-off-by: Dan Mick <dan.mick@inktank.com>

Merge pull request #1886 from nereocystis/pools-doc

doc: update pools documentation

Reviewed-by: Loic Dachary <loic@dachary.org>

doc: update pools documentation

set=>get in "Get Pool Values"
grep for 'replicated size', not 'rep size'

Signed-off-by: Kevin Dalley <kevin@kelphead.org>

Merge pull request #1884 from nereocystis/glossary-rados

doc: Define RADOS in glossary

doc: Define RADOS in glossary

Signed-off-by: Kevin Dalley <kevin@kelphead.org>

doc: Added osd pool default size setting example.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Moved redundant text out of quick-common.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

Merge pull request #1882 from ceph/wip-8465

osd: fix bad is_active() assert in share_map()

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #1840 from yuyuyu101/wip-8381

Fix set_alloc_hint op cause KeyValueStore crash problem

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #1874 from ceph/wip-8292

better error reporting on incompatible device requirements

Reviewed-by: Dan Mick <dan.mick@inktank.com>

osd: fix bad is_active() assert in share_map()

We may be is_stopping() here if we are racing with shutdown().

Fixes: #8465
Backport: firefly, dumpling
Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #1880 from ailzhang/patch-1

rest-api: key missing for per "rx" and "rwx"

Backport: firefly
Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1847 from ceph/wip-8259

README, Makefile: make install-deps easier

Reviewed-by: Loic Dachary <loic@dachary.org>

Merge pull request #1860 from ceph/wip-filestore

osd: filestore backend improvements

Reviewed-by: Loic Dachary <loic@dachary.org>

Merge pull request #1864 from dachary/wip-filestore

unit tests for #1860

Merge pull request #1879 from ceph/wip-8321

fix incompatible `Counter` usage on Python 2.6

Reviewed-by: Loic Dachary <loic@dachary.org>

doc: Made additional changes s/CRUSH/Ceph for pull request #1855

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

Merge pull request #1855 from kazhang/patch-1

doc: Update architecture.rst:Calculating PG IDs

Merge pull request #1866 from cmattson/radosgwdoc

doc: Improvements to radosgw docs

Merge pull request #1881 from hsdhillon/master

doc: Fixes broken hyperlinks

doc: Fixes broken hyperlinks

Makefile: make install-deps rule

This only works if autoconf is happy, but it's better than nothing.

Fixes: #8259
Signed-off-by: Sage Weil <sage@inktank.com>

debian: improve package dep list

This is based on the Build-Depends in debian/control.

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #1812 from ceph/wip-smart-df

mon: show writeable space in 'ceph df'

Reviewed-by: Samuel Just <sam.just@inktank.com>

mon: set min_size to data chunk count for erasure pools

Make the min_size value meaningful for erasure pools.

Signed-off-by: Sage Weil <sage@inktank.com>

mon: include 'max avail' in df output

Include an estimate of the maximum writeable space for each pool.  Note
that this value is a conservative estimate for that pool based on the
most-full OSD.  It is also potentially misleading as it is the available
space if *all* new data were written to this pool; one cannot (generally)
add up the available space for all pools.

Signed-off-by: Sage Weil <sage@inktank.com>

mon: right justify df values

Signed-off-by: Sage Weil <sage@inktank.com>

vstart.sh: -e to create an ec pool called 'ec'

Signed-off-by: Sage Weil <sage@inktank.com>

crush: add get_rule_weight_map

Calculate a weight map of OSDs for a given rule.

Signed-off-by: Sage Weil <sage@inktank.com>

rest-api: key missing for per "rx" and "rwx"

commit 85a1cf31e6 and db266a3fb2 introduce new per "rx" and "rwx", but key missing for per "rx" and "rwx" in permmap

Signed-off-by: Ailing Zhang <ailzhang@cisco.com>

Merge pull request #1878 from ceph/wip-map-advance

osd: fix map advance limit to handle map gaps

Reviewed-by: Samuel Just <sam.just@inktank.com>

remove unused variables, gets all tox envs passing

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

add backport of collections.Counter for python2.6

Using Raymond Hettinger's MIT backport

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

intial take on ceph-brag-client and tox. Python 2.7 passes

Signed-off-by: Alfredo Deza <alfredo@deza.pe>

Merge pull request #1853 from dachary/wip-mailmap

mailmap updates

each commit has been submitted to the author for review

mailmap: Aristoteles Neto affiliation

and name normalization

Reviewed-by: Aristoteles Neto <aristoteles.neto@webdrive.co.nz>
Signed-off-by: Loic Dachary <loic@dachary.org>

mailmap: Chris Glass affiliation

Reviewed-by: Chris Glass <tribaal@gmail.com>
Signed-off-by: Loic Dachary <loic@dachary.org>

mailmap: Christian Theune affiliation

Reviewed-by: Christian Theune <ct@gocept.com>
Signed-off-by: Loic Dachary <loic@dachary.org>

mailmap: Shawn Edwards affiliation

Signed-off-by: Loic Dachary <loic@dachary.org>

mailmap: Ilya Dryomov name normalization

Signed-off-by: Loic Dachary <loic@dachary.org>

mailmap: Stuart Longland affiliation

Signed-off-by: Loic Dachary <loic@dachary.org>

mailmap: Walter Huf affiliation

Reviewed-by: Walter J. Huf <hufman@gmail.com>
Signed-off-by: Loic Dachary <loic@dachary.org>

mailmap: Michael Riederer affiliation

and name normalization

Signed-off-by: Loic Dachary <loic@dachary.org>

mailmap: Stefan Eilemann affiliation

Signed-off-by: Loic Dachary <loic@dachary.org>

mailmap: Stephen F Taylor affiliation

Signed-off-by: Loic Dachary <loic@dachary.org>

mailmap: Daniel J. Hofmann affiliation

Reviewed-by: Daniel J. Hofmann <daniel@trvx.org>
Signed-off-by: Loic Dachary <loic@dachary.org>

mailmap: Kevin Dalley affiliation

Reviewed-by: Kevin Dalley <kevin@kelphead.org>
Signed-off-by: Loic Dachary <loic@dachary.org>

mailmap: Florent Flament affiliation

and name normalization

Reviewed-by: Florent Flament <florent.flament@cloudwatt.com>
Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #1877 from ceph/wip-tier-doc

doc: Fix cache tier docs

Reviewed-by: Loic Dachary <loic@dachary.org>

doc: Fix cache tier docs

The instructions for 'osd tier add' were plain
wrong, claiming that the cache mode could
be passed as a third argument.

Signed-off-by: John Spray <john.spray@inktank.com>

Merge pull request #1856 from dachary/wip-pg-documentation

documentation: adapt PG formula for erasure coded pools

Reviewed-By: Christophe Courtaut <christophe.courtaut@gmail.com>

Merge pull request #1867 from dachary/wip-8324

documentation: update pid file description

Reviewed-by: Florent Flament <florent.flament@cloudwatt.com>

os: FileStore::create_backend unit tests

Assert that the fs_type is mapped to a backend that claims to have the
matching name.

Signed-off-by: Loic Dachary <loic@dachary.org>

os: ObjectStore::collect_metadata unit tests

And an emacs-executable reminder about how to run a specific test.

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #1813 from ceph/wip-old-out

mon: restore osd weight after auto-mark out and then osd restart

Reviewed-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Loic Dachary <loic@dachary.org>

Merge pull request #1811 from ceph/wip-perf-atomic

perf_counter: use atomics instead of mutex

Reviewed-by: Greg Farnum <greg@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #1873 from dachary/wip-8440

mon: fix set cache_target_full_ratio

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #1876 from atwardowski/patch-4

Fix Documentation Typo

Reviewed-by: Loic Dachary <loic@dachary.org>

Fix Documentation Typo

Signed-off-by: Adam Twardowski <adam.twardowski@gmail.com>

doc: Added primary affinity. Some minor rewrap edits.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

better error reporting on incompatible device requirements

Signed-off-by: Alfredo Deza <alfredo@deza.pe>

Merge pull request #1739 from ceph/wip-journal-tool

cephfs-journal-tool

Merge pull request #1824 from dachary/wip-documentation

erasure-code: fix URL in developer notes

Reviewed-by: koleosfuscus <koleosfuscus@yahoo.com>

mon: fix set cache_target_full_ratio

It was a noop because it was incorrectly using the variable n. Add a
test to protect against regression.

http://tracker.ceph.com/issues/8440 Fixes: #8440

Reported-by: Geoffrey Hartz <hartz.geoffrey@gmail.com>
Signed-off-by: Loic Dachary <loic@dachary.org>

osd: fix map advance limit to handle map gaps

The recent change in cf25bdf6b0090379903981fe8cee5ea75efd7ba0 would stop
advancing after some number of epochs, but did not take into consideration
the possibilty that there are missing maps.  In that case, it is impossible
to advance past the gap.

Fix this by increasing the max epoch as we go so that we can always get
beyond the gap.

Signed-off-by: Sage Weil <sage@inktank.com>

documentation: update pid file description

The pid file behavior changed over time and the description is misleading.

http://tracker.ceph.com/issues/8324 Fixes: #8324

Signed-off-by: Loic Dachary <loic@dachary.org>

Improvements to radosgw docs
- Correct a typo (`rados-admin`) in admin docs
- Reorder sections in config so "above example" is in expected position

os/FileStore: include filestore backend, f_type in osd metadata

Report the FileStore backend and underlying fs type as part of the osd
metadata shared with the monitor.

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #1861 from atwardowski/patch-3

Fix Doc Typo

Reviewed-by: Loic Dachary <loic@dachary.org>

Merge pull request #1846 from ceph/wip-8342

init-ceph: keep trying other OSDs if we fail to set CRUSH position on one OSD

Reviewed-by: Loic Dachary <loic@dachary.org>
Reviewed-by: Dmitry Smirnov <onlyjob@debian.org>

Merge pull request #1858 from ceph/wip-8428

rgw: check appropriate entity permission on put_metadata

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Fix Doc Typo

Signed-off-by: Adam Twardowski <adam.twardowski@gmail.com>

os/FileStore: refactor backend instantiation

Use a factory method for creating the backend so that the logic is not
duplicated in mkfs() and _detect_fs().  Make the other fs-dependent logic
(xattr limits) key off of the fs_type magic.

Signed-off-by: Sage Weil <sage@inktank.com>

os/FileStore: fix journal-less operation

A few different fixes here:

- make the op_seq correct on start when there is no replay
- make an explicit 'no journal' case in queue_transaction (one that
  actually works).  Note that the 'trailing journal' mode is probably
  still broken.
- add a -J flag to vstart.sh for journalless operation

Signed-off-by: Sage Weil <sage@inktank.com>

rgw: check appropriate entity permission on put_metadata

Fixes: #8428
Backport: firefly

Cannot use verify_object_permission() to test acls, as the operation
here might either be on object or on bucket.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

documentation: adapt PG formula for erasure coded pools

Signed-off-by: Loic Dachary <loic@dachary.org>

Update architecture.rst:Calculating PG IDs

To Calculate PG ID, if I didn't get it wrong, CRUSH calculates the hash modulo
the number of PGs instead of OSDs, according to osd/osd_types.cc:963
ceph_stable_mod(pg.ps(), pg_num, pg_num_mask).

Signed-off-by: Kai Zhang <zakir.exe@gmail.com>

Merge pull request #1854 from ceph/wip-java-build

cephfs-java: build against older jni headers

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #1851 from FlorentFlament/master

documentation: update osd pool default size from 2 to 3.

Reviewed-By: Christophe Courtaut <christophe.courtaut@gmail.com>

documentation: update osd pool default size from 2 to 3.

Fixes: #7725
Signed-off-by: Florent Flament <florent.flament@cloudwatt.com>

Merge pull request #1815 from ceph/osd-metadata

osd: include osd_objectstore in metadata reported to mon

Reviewed-by: Loic Dachary <loic@dachary.org>

Merge pull request #1793 from ceph/wip-multimds

Speed up several cross-MDS operations by reducing the number of two-phase commit disk accesses we have to go through.

Reviewed-by: Greg Farnum <greg@inktank.com>

cephfs-java: build against older jni headers

Older versions of the JNI interface expected non-const parameters
to their memory move functions. It's unpleasant, but won't actually
change the memory in question, to do a cast_const in order to satisfy
those older headers. (And even if it *did* modify the memory, that
would be okay given our single user.)

Signed-off-by: Greg Farnum <greg@inktank.com>

Merge pull request #1838 from ceph/wip-client-time

fs: include client timestamp in all mds requests

Reviewed-by: Greg Farnum <greg@inktank.com>

Merge pull request #1835 from ceph/wip-8332

HashIndex: in cleanup, interpret missing dir as completed merge

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1834 from ceph/wip-8334

Wip 8334

Reviewed-by: Sage Weil <sage@inktank.com>

doc/release-notes: v0.67.9

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #1848 from yuyuyu101/librbd-flush

Avoid extra check for clean object

Backport: firefly, dumpling
Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1849 from ceph/wip-mon-get-version

mon: set MMonGetVersionReply tid

Reviewed-by: Sage Weil <sage@inktank.com>

Avoid extra check for clean object

We needn't to check clean object via buffer state, skip the clean object.

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

mon: set MMonGetVersionReply tid

Currently we don't set MMonGetVersionReply tid even if the original
MMonGetVersion message had a non-zero tid.  This is bad for the kernel
client, which has the infrastructure in place that relies on tids to
lookup message buffers and contexts.  To kick off transitioning away
from the workaround, set MMonGetVersionReply tid to the tid of the
original MMonGetVersion message.

Signed-off-by: Ilya Dryomov <ilya.dryomov@inktank.com>

README: move package dependencies into separate files

Sorted, with a short, simple command to paste.  Avoid the duplication of
the lists.

Signed-off-by: Sage Weil <sage@inktank.com>