osd: tests for osd bench

Check that the safeguards against timeouts work as expected and that the
default hardcoded values are valid.

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #1961 from ceph/wip-6703

support dmcrypt partitions when activating

Reviewed-by: Loic Dachary <loic@dachary.org>

Merge pull request #2007 from GRyall/master

doc: Fixed Typo in pools documentation - replaced '-' with '_' in eg

Reviewed-by: John Wilkins john.wilkins@inktank.com

doc: Fixed Typo in pools documentation - replaced '-' with '_' in example set-quota comands.

Merge pull request #1997 from ceph/wip-tiermsg

mon: name instead of id in "has tiers" message

Reviewed-by: Loic Dachary <loic@dachary.org>

mon: name instead of id in "has tiers" message

Instead of "Pool foo has tiers 1 2" print
"Pool foo has tiers bar baz".

Signed-off-by: John Spray <jspray@redhat.com>

Merge pull request #1988 from dachary/wip-warnings

test: fix -Wsign-compare warnings

Reviewed-by: Sahid Orentino Ferdjaoui <sahid.ferdjaoui@cloudwatt.com>

Merge pull request #1993 from dachary/fix_ut

autotools: avoid check_SCRIPTS duplication

Reviewed-by: Wei Luo <luowei@yahoo-inc.com>

Merge pull request #1995 from sjahl/master

doc: Fixed an incorrect flag in radosgw admin docs.

doc: Fixed an incorrect flag in radosgw admin docs.

Example use a "--gen-key" flag, which doesn't exist. Changed it to
"--gen-access-key".

Signed-off-by: Stephen Jahl <stephenjahl@gmail.com>

autotools: avoid check_SCRIPTS duplication

The check_SCRIPTS content must be added to EXTRA_DIST, otherwise it will
not be included by make dist and it won't be possible to run make check
successfully.

One solution would be to add $(check_SCRIPTS) to EXTRA_DIST to avoid
duplication and help with long term maintenance. However, $(srcdir) is
not supported in the content of the check_SCRIPTS variable.

A GNU Make variable substitution (patsubst) is used to prepend $(srcdir)
to each script, only when used in the EXTRA_DIST variable.

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #1992 from luowei62/fix_ut

Fix dist package run unit test failed.

Reviewed-by: Loic Dachary <loic@dachary.org>

Merge pull request #1989 from dachary/wip-8622-rados-alignment

tests: prevent gitbuilder trigger in test-erasure-code.sh

Reviewed-by: Lluis Pamies-Juarez <lluis@pamies.cat>

Fix dist package run unit test failed.

Signed-off-by: Wei Luo <luowei@yahoo-inc.com>

tests: prevent gitbuilder trigger in test-erasure-code.sh

Use the INFO keywork instead of WARNING so that gitbuilder does not
confuse this with a warning worth fixing.

Signed-off-by: Loic Dachary <loic@dachary.org>

test: fix -Wsign-compare warnings

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #1987 from dachary/wip-8622-rados-alignment

erasure-code: verify that rados put enforces alignment

Reviewed-by: Lluis Pamies-Juarez <lluis@pamies.cat>

erasure-code: verify that rados put enforces alignment

http://tracker.ceph.com/issues/8622 refs: #8622

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #1983 from dachary/wip-osd-config

tests: remove spurious and harmless find command

Reviewed-by: Florent Flament <florent.flament@cloudwatt.com>

Merge pull request #1984 from llpamies/wip-8622

enforce rados put aligment

Reviewed-by: Loic Dachary <loic@dachary.org>

Merge pull request #1943 from athanatos/wip-8507

Wip 8507

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1986 from ceph/wip-scrub-sleep

osd: introduce simple sleep during scrub

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #1985 from concubidated/master

ReplicatedPG: Fix 'ajusted' typo

Reviewed-by: Sage Weil <sage@inktank.com>

ReplicatedPG: 'ajusted' typo

Signed-off-by: <Tyler Brekke> tyler.brekke@inktank.com

rgw-admin: Fix the 'show log' command

Fixes: #8553
Reviewed-by: Yehuda Sadeh <yehuda@inktank.com>
Signed-off-by: Sylvain Munaut <s.munaut@whatever-company.com>

enforce rados put aligment

Signed-off-by: Lluis Pamies-Juarez <lluis.pamies-juarez@hgst.com>

tests: remove spurious and harmless find command

Signed-off-by: Loic Dachary <loic@dachary.org>

osd: introduce simple sleep during scrub

This option is similar to osd_snap_trim_sleep: simply inject an optional
sleep in the thread that is doing scrub work.  This is a very kludgey and
coarse knob for limiting the impact of scrub on the cluster, but can help
until we have a more robust and elegant solution.

Only sleep if we are in the NEW_CHUNK state to avoid delaying processing of
an in-progress chunk.  In this state nothing is blocked on anything.
Conveniently, chunky_scrub() requeues itself for each new chunk.

Backport: firefly, dumpling
Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #1968 from ceph/wip-ceph-isatty

ceph: output prompt only if stdin is tty

Reviewed-by: Sage Weil <sage@inktank.com>

support dmcrypt partitions when activating

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

Merge pull request #1947 from ceph/wip-import-purge

Wip import purge

Reviewed-by: Greg Farnum <greg@inktank.com>

Merge pull request #1970 from dachary/wip-erasure-code-profile-default

Revert "erasure-code: create default profile if necessary"

Revert "erasure-code: create default profile if necessary"

This reverts commit 360de6adf24ef9a1287d169416d5e9bbb72afbec.

Conflicts:
src/mon/OSDMonitor.cc

mon/OSDMonitor: fix build error

This broke with 360de6adf24ef9a1287d169416d5e9bbb72afbec.

Signed-off-by: Sage Weil <sage@inktank.com>

osd: verify osd config sanity checks

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #1951 from sahid/ut-lru

Adds tests to include/lru.h + fixes small issues

Reviewed-by: Loic Dachary <loic@dachary.org>
Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1954 from dachary/wip-8307-erasure-code-profile-implicit-creation

erasure code profile implicit creation

Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>

Merge pull request #1964 from ceph/wip-osd-configs

osd: add sanity check/warning on a few key configs

Reviewed-by: Loic Dachary <loic@dachary.org>

Merge pull request #1966 from dachary/wip-erasure-code-profile-default

DNM: erasure-code: create default profile if necessary

Reviewed-by: Sage Weil <sage@inktank.com>

PendingReleaseNotes: note about keyvaluestore-dev on-disk format change

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #1941 from yuyuyu101/wip-8564

Fix write operation on a deleted object in the same transaction(KeyValueStore)

Reviewed-by: Sage Weil <sage@inktank.com>

common: Enforces the methods lru_pin() and lru_unpin()

If lru_*pin() is called twice, the counter will be incr/decr
incorrectly since it will count more/less pinned objects than
there is and so corrupts the balancing (lru_adjust()).

Signed-off-by: Sahid Orentino Ferdjaoui <sahid.ferdjaoui@cloudwatt.com>

common: Fixes issue with lru_clear() + add new test

The method lru_clear() must set attribute lru_num to zero
after lru_top, lru_bot and lru_mid are reseted. indeed, lru_num
is the total number of elements found in all of them.

Also the test insures the good behavior of the method
lru_adjust() - lru_touch() calls lru_adjust every time
to balance lru_top and lru_bot by the value of lru_midpoint.

Signed-off-by: Sahid Orentino Ferdjaoui <sahid.ferdjaoui@cloudwatt.com>

common: Adds simple tests to verify good behavior

Adds new tests to the lib include/lru.h to verify methods
insert_*.

Signed-off-by: Sahid Orentino Ferdjaoui <sahid.ferdjaoui@cloudwatt.com>

ceph: output prompt only if stdin is tty

In loop mode, output prompt only if stdin is tty, i.e. if we are
interactive.

Signed-off-by: Ilya Dryomov <ilya.dryomov@inktank.com>

erasure-code: create default profile if necessary

http://tracker.ceph.com/issues/8601

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #1960 from dachary/wip-test-kill-race

tests: prevent kill race condition

Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>

osd: add sanity check/warning on a few key configs

Warn when certain config values are set to bad values.

Backport: firefly, dumpling
Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #1949 from ceph/wip-7774-3

Wip 7774 3

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #1915 from ceph/wip-8452-2

rgw: set meta object in extra flag when initializing it

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

osd: remove non const get_erasure_code_profile

It is not used and will unexpectedly create a new entry if selected when
the const version is intended.

Signed-off-by: Loic Dachary <loic@dachary.org>

tests: prevent kill race condition

When trying to kill a daemon, keep its pid in a variable instead of
retrieving it from the pidfile multiple times. It prevents the following
race condition:

  * try to kill ceph-mon
  * ceph-mon is in the process of dying and removed its pidfile
  * try to kill ceph-mon fails because the pidfile is not found
  * another ceph-mon is spawned and fails to bind the port
    because the previous ceph-mon is still holding it

Signed-off-by: Loic Dachary <loic@dachary.org>

osd: improve osd pool create error message readability

Add std::endl to separate independent error messages related to erasure
code.

Signed-off-by: Loic Dachary <loic@dachary.org>

erasure-code: consistent argument parsing for profiles

Remove the = from the goodchars of the erasure_code_profile argument of
osd pool create so that it is consistent with the goodchars of osd
erasure-code-profile set / rm.

Signed-off-by: Loic Dachary <loic@dachary.org>

erasure-code: OSDMonitor::get_erasure_code is a const

If it is not, the non const version of OSDMap::get_erasure_code_profile
is called and a profile is created as a side effect, which is not
intended.

http://tracker.ceph.com/issues/8307 refs: #8307

Signed-off-by: Loic Dachary <loic@dachary.org>

erasure-code: pool create must not create profiles

If a non existent profile is provided as an argument to osd pool create,
it must exit on error and not create the profile as a side effect.

http://tracker.ceph.com/issues/8307 refs: #8307

Signed-off-by: Loic Dachary <loic@dachary.org>

qa: extend cephfs_journal_tool_smoke

To provide coverage for the 'import' command
resulting in the same contents as before an
export/import cycle.

Signed-off-by: John Spray <john.spray@redhat.com>

qa: set +x on cephfs_journal_tool_smoke

Signed-off-by: John Spray <john.spray@redhat.com>

tools/cephfs: Purge trailing objects during import

Reviewed-by: Gregory Farnum <greg@inktank.com>
Signed-off-by: John Spray <john.spray@inktank.com>

tools/cephfs: error handling in journal_export

Was failing on unreadable journal even if
only trying to do an import.

Was calling messenger destructor without init
on early failures due to early instantiation
of Dumper.

Signed-off-by: John Spray <john.spray@inktank.com>

tools/cephfs: Clean up waits in Dumper

Remove some LOC by using C_SaferCond.

Signed-off-by: John Spray <john.spray@inktank.com>

Merge pull request #1955 from ceph/wip-8585

rgw: set a default data extra pool name

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #1933 from ceph/wip-8556

osd: do not require EC feature bit from clients.

Reviewed-by: Greg Farnum <greg@inktank.com>

osd/OSDMap: do not require ERASURE_CODE feature of clients

Just because an EC pool exists in the cluster does not mean tha tthe client
has to support the feature:

1) The way client IO is initiated is no different for EC pools than for
   replicated pools.
2) People may add an EC pool to an existing cluster with old clients and
   locking those old clients out is very rude when they are not using the
   new pool.
3) The only direct client user of EC pools right now is rgw, and the new
   versions already need to support various other features like CRUSH_V2
   in order to work.  These features are present in new kernels.

Fixes: #8556
Backport: firefly
Signed-off-by: Sage Weil <sage@inktank.com>

osd/OSDMap: make get_features() take an entity type

Make the helper that returns what features are required of the OSDMap take
an entity type argument, as the required features may vary between
components in the cluster.

Backport: firefly
Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #1957 from ceph/wip-doc-os-recommendations

doc: Updated the OS Recommendations for Firefly.

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

doc: Updated the OS Recommendations for Firefly.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Updated the example configuration.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Updated doc for more recent versions.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

Merge pull request #1939 from ceph/wip-mon

mon: a few basic perfcounter guages

Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>

Merge pull request #1956 from ceph/wip-fsx-sizeof

test_librbd_fsx: fix sign-compare gcc warning

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1948 from steveftaylor/6700

Fix for bug #6700

Reviewed-by: Alfredo Deza <alfredo.deza@inktank.com>

test_librbd_fsx: fix sign-compare gcc warning

{read,write}bdy are signed, sizeof returns unsigned, gcc complains.

Signed-off-by: Ilya Dryomov <ilya.dryomov@inktank.com>

Merge pull request #1937 from yuyuyu101/keyvaluestore-strip-size

Add strip size and set_alloc_hint op support for KeyValueStore

Reviewed-by: Sage Weil <sage@inktank.com>

rgw: set a default data extra pool name

Fixes: #8585
Have a default name for the data extra pool, otherwise it would be empty
which means that it'd default to the data pool name (which is a problem
with ec backends).

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

doc: Made mention of "incomplete" status.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

Merge pull request #1953 from ceph/wip-xattr-spillout

Enable and fix the existing xattr spillout code.

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1897 from somnathr/wip-sharded-threadpool

Reviewed-by: Greg Farnum <greg@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>

qa: add an fsx run which turns on kernel debugging

Signed-off-by: Greg Farnum <greg@inktank.com>

FileStore: remove the user_only param from _fgetattrs

Nobody sets it to true, so remove it!

Signed-off-by: Greg Farnum <greg@inktank.com>

FileStore: remove user_only options from getattrs through the ObjectStore stack

This sort of awareness belongs at a higher level in the stack -- as
evidenced by nobody using the option at this level. Remove it from the
implementations and the interface

Signed-off-by: Greg Farnum <greg@inktank.com>

FileStore: do not use user_only in collection_getattrs

There's no particular reason why any of the callers of collection_getattrs
want to avoid looking at Ceph's internal xattrs.
It looks like this flag (set in 1862ddd88548fd4609f4fa9715dbad42a84d3775) was
set this way by mistake.
And finally, we don't actually set xattrs on collections anymore, anyway.

Signed-off-by: Greg Farnum <greg@inktank.com>

FileStore: remove dead code

This case wasn't reachable when it was introduced, and it's still not.

Signed-off-by: Greg Farnum <greg@inktank.com>

FileStore: set XATTR_NO_SPILL_OUT when creating new files.

Fixes: #8205
Backport: firefly

Signed-off-by: Greg Farnum <greg@inktank.com>

FileStore: make _clone() copy spill out marker

Previously we were not doing so, and that resulted in unpredictable loss
of xattrs from the client's perspective.

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

Add xattr test to ceph_test_objectstore

Add xattr synthetic test to test ObjectStore xattr related interfaces.

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

rgw: chain to multiple cache entries in one call

This ensures that chained cache entries that depend on more than one raw
cache entry (bucket info cache depends on both the bucket entry point
and on the bucket info object), are chained and created atomically.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: chain binfo_cache to multiple cache entries

Need to chain it to both the entry point and to the instance.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: bucket info uses cache chaining

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: user info uses cache chaining

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: an infrastructure for hooking into the raw cache

Extend the RGWCache so that we can chain other caches to it so that when
data is invalidated it notifies them.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: cache bucket info

This is really a partial implementation, so should only be used for
testing.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: cache decoded user info

Instead of accessing the raw user info data, cache the decoded
structure.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

PG: Added a const spg_t member to the PG class
The const spg_t member is been insantiated from constructor
and now get_pgid() can reference this to return a spg_t instance
without the need of pg_info (thus not requiring to acquire pg_lock).

Signed-off-by: Somnath Roy <somnath.roy@sandisk.com>

doc: Added a tip to install NTP when running multiple monitors.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Added additional background to address additional states.

Covers 1-node cluster issues, fewer OSDs than replicas, and
active + clean + inconsistent states in greater detail.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Added background discussion to clock settings.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

OSD: The thread pool variable name changed
The variable names are more meaningful now. op_tp -> osd_tp and
op_sharded_tp -> osd_op_tp

Signed-off-by: Somnath Roy <somnath.roy@sandisk.com>

ShardedTP: The config option changed
The config option for sharded threadpool is changed to
osd_op_num_threads_per_shard instead of osd_op_num_sharded_pool_threads.
Along with osd_op_num_shards this will be much more user friendly while
configuring the number of op threads for the osd.

Signed-off-by: Somnath Roy <somnath.roy@sandisk.com>

ShardedTP: Changes related to conforming to Ceph coding guidelines

Signed-off-by: Somnath Roy <somnath.roy@sandisk.com>