Give meaningful error when submodules are not checked out

Signed-off-by: Adam Crume <adamcrume@gmail.com>

osd: Only normalize extent if op uses extent

Code in ReplicatedPG was reading and modifying ceph_osd_op.extent,
even though that is a member of a union and only valid for some ops.

Signed-off-by: Adam Crume <adamcrume@gmail.com>

Merge pull request #1998 from ceph/wip-disk-ioprio

osd: allow io priority for (background) disk threadpool to be set

Backport: firefly, dumpling
Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #2015 from ceph/wip-8610

osd: ignore CRUSH_ITEM_NONE in compat_must_dispatch_immediately

Reviewed-by: Samuel Just <sam.just@inktank.com>

osd: ignore CRUSH_ITEM_NONE in compat_must_dispatch_immediately

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #1979 from ceph/wip-backfill-priority

osd: improve backfill prioritization

Reviewed-by: Samuel Just <sam.just@inktank.com>

osd: allow io priority to be set for the disk_tp

The disk_tp covers scrubbing, pg deletion, and snap trimming

Signed-off-by: Sage Weil <sage@inktank.com>

common/WorkQueue: allow io priority to be set for wq

Signed-off-by: Sage Weil <sage@inktank.com>

common/Thread: allow io priority to be set for a Thread

Ideally, set this before starting the thread.  If you set it after, we
could potentially race with create() itself.

Signed-off-by: Sage Weil <sage@inktank.com>

common/io_priority: wrap ioprio_set() and gettid()

Signed-off-by: Sage Weil <sage@inktank.com>

Merge remote-tracking branch 'gh/wip-misc-fixes'

Conflicts:
src/mon/Monitor.cc

Reviewed-by: Sage Weil <sage@inktank.com>
Reviewed-by: Loic Dachary <loic@dachary.org>

Merge pull request #2009 from kevincox/election-int

Use sized integer.

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #2011 from dachary/wip-osd-bench

osd: tests for osd bench

Reviewed-by: Huang Jun <hjwsm1989@gmail.com>

osd: tests for osd bench

Check that the safeguards against timeouts work as expected and that the
default hardcoded values are valid.

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #1974 from hjwsm1989/osd-bench

  Fix EINVAL err when use "ceph tell osd.* bench"

Reviewed-by: Loic Dachary <loic@dachary.org>

Use sized integer.

Signed-off-by: Kevin Cox <kevincox@kevincox.ca>

Merge pull request #1961 from ceph/wip-6703

support dmcrypt partitions when activating

Reviewed-by: Loic Dachary <loic@dachary.org>

Merge pull request #2007 from GRyall/master

doc: Fixed Typo in pools documentation - replaced '-' with '_' in eg

Reviewed-by: John Wilkins john.wilkins@inktank.com

doc: Fixed Typo in pools documentation - replaced '-' with '_' in example set-quota comands.

Merge pull request #1997 from ceph/wip-tiermsg

mon: name instead of id in "has tiers" message

Reviewed-by: Loic Dachary <loic@dachary.org>

osd/osd_types.cc: dump correct pg_log_entry_t member variable

We were dumping 'version' twice: for 'version' and 'prior_version'.  We
do have a member variable called 'prior_version', which is assumed to
be the correct one to be dumped.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

osd: use appropriate json types instead of stream when dumping info

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon: name instead of id in "has tiers" message

Instead of "Pool foo has tiers 1 2" print
"Pool foo has tiers bar baz".

Signed-off-by: John Spray <jspray@redhat.com>

Merge pull request #1988 from dachary/wip-warnings

test: fix -Wsign-compare warnings

Reviewed-by: Sahid Orentino Ferdjaoui <sahid.ferdjaoui@cloudwatt.com>

osd: ECMsgTypes: dump json bools instead of strings

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

osd: have 'tid' dumped as a json unsigned int where appropriate

Most of the places dumping it would dump it as a string. Others as an int.
Just a couple would dump it as an unsigned int.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

Merge pull request #1993 from dachary/fix_ut

autotools: avoid check_SCRIPTS duplication

Reviewed-by: Wei Luo <luowei@yahoo-inc.com>

mon: dump 'epoch' as json unsigned instead of int

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon: PGMonitor: dump 'epoch' as json int instead of string

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon: OSDMonitor: dump 'up' as json array instead of string

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon: OSDMonitor: dump 'acting' as json array instead of string

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

Merge pull request #1995 from sjahl/master

doc: Fixed an incorrect flag in radosgw admin docs.

doc: Fixed an incorrect flag in radosgw admin docs.

Example use a "--gen-key" flag, which doesn't exist. Changed it to
"--gen-access-key".

Signed-off-by: Stephen Jahl <stephenjahl@gmail.com>

autotools: avoid check_SCRIPTS duplication

The check_SCRIPTS content must be added to EXTRA_DIST, otherwise it will
not be included by make dist and it won't be possible to run make check
successfully.

One solution would be to add $(check_SCRIPTS) to EXTRA_DIST to avoid
duplication and help with long term maintenance. However, $(srcdir) is
not supported in the content of the check_SCRIPTS variable.

A GNU Make variable substitution (patsubst) is used to prepend $(srcdir)
to each script, only when used in the EXTRA_DIST variable.

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #1992 from luowei62/fix_ut

Fix dist package run unit test failed.

Reviewed-by: Loic Dachary <loic@dachary.org>

Merge pull request #1989 from dachary/wip-8622-rados-alignment

tests: prevent gitbuilder trigger in test-erasure-code.sh

Reviewed-by: Lluis Pamies-Juarez <lluis@pamies.cat>

Fix dist package run unit test failed.

Signed-off-by: Wei Luo <luowei@yahoo-inc.com>

tests: prevent gitbuilder trigger in test-erasure-code.sh

Use the INFO keywork instead of WARNING so that gitbuilder does not
confuse this with a warning worth fixing.

Signed-off-by: Loic Dachary <loic@dachary.org>

test: fix -Wsign-compare warnings

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #1987 from dachary/wip-8622-rados-alignment

erasure-code: verify that rados put enforces alignment

Reviewed-by: Lluis Pamies-Juarez <lluis@pamies.cat>

erasure-code: verify that rados put enforces alignment

http://tracker.ceph.com/issues/8622 refs: #8622

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #1983 from dachary/wip-osd-config

tests: remove spurious and harmless find command

Reviewed-by: Florent Flament <florent.flament@cloudwatt.com>

Merge pull request #1984 from llpamies/wip-8622

enforce rados put aligment

Reviewed-by: Loic Dachary <loic@dachary.org>

Merge pull request #1943 from athanatos/wip-8507

Wip 8507

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1986 from ceph/wip-scrub-sleep

osd: introduce simple sleep during scrub

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #1985 from concubidated/master

ReplicatedPG: Fix 'ajusted' typo

Reviewed-by: Sage Weil <sage@inktank.com>

ReplicatedPG: 'ajusted' typo

Signed-off-by: <Tyler Brekke> tyler.brekke@inktank.com

rgw-admin: Fix the 'show log' command

Fixes: #8553
Reviewed-by: Yehuda Sadeh <yehuda@inktank.com>
Signed-off-by: Sylvain Munaut <s.munaut@whatever-company.com>

enforce rados put aligment

Signed-off-by: Lluis Pamies-Juarez <lluis.pamies-juarez@hgst.com>

tests: remove spurious and harmless find command

Signed-off-by: Loic Dachary <loic@dachary.org>

osd: prioritize backfill based on *how* degraded

Instead of prioritizing all degraded PGs equally, prioritize those that
have the fewest acting members (the most degraded).

Drop the priority constants.

Define our max priority independently of AsyncReserver, which uses a host
unsigned.

Signed-off-by: Sage Weil <sage@inktank.com>

osd: add osd_min_recovery_priority tunable

Add a tunable to control the minimum priority recovery item that we will
grant a reservation for.  The priority is lower for less-degraded PGs.
This allows a saavy admin to only backfill the least-redundant PGs.

Signed-off-by: Sage Weil <sage@inktank.com>

common/AsyncReserver: add a min_priority knob

To do reserve items that are < the min priority.  Default to 0.

Signed-off-by: Sage Weil <sage@inktank.com>

osd: fix word sizes in MBackfillReserve

Fix these fields to 32 bits.

Signed-off-by: Sage Weil <sage@inktank.com>

osd: introduce simple sleep during scrub

This option is similar to osd_snap_trim_sleep: simply inject an optional
sleep in the thread that is doing scrub work.  This is a very kludgey and
coarse knob for limiting the impact of scrub on the cluster, but can help
until we have a more robust and elegant solution.

Only sleep if we are in the NEW_CHUNK state to avoid delaying processing of
an in-progress chunk.  In this state nothing is blocked on anything.
Conveniently, chunky_scrub() requeues itself for each new chunk.

Backport: firefly, dumpling
Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #1968 from ceph/wip-ceph-isatty

ceph: output prompt only if stdin is tty

Reviewed-by: Sage Weil <sage@inktank.com>

mon: Monitor: complex code deduplication algorithm applied to tick()

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon: Monitor: rework tick() so we don't calculate the same thing over and again

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

support dmcrypt partitions when activating

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

  Fix EINVAL err when use "ceph tell osd.* bench"

Signed-off-by: huangjun <hjwsm1989@gmail.com>

Merge pull request #1947 from ceph/wip-import-purge

Wip import purge

Reviewed-by: Greg Farnum <greg@inktank.com>

Merge pull request #1970 from dachary/wip-erasure-code-profile-default

Revert "erasure-code: create default profile if necessary"

Revert "erasure-code: create default profile if necessary"

This reverts commit 360de6adf24ef9a1287d169416d5e9bbb72afbec.

Conflicts:
src/mon/OSDMonitor.cc

mon/OSDMonitor: fix build error

This broke with 360de6adf24ef9a1287d169416d5e9bbb72afbec.

Signed-off-by: Sage Weil <sage@inktank.com>

osd: verify osd config sanity checks

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #1951 from sahid/ut-lru

Adds tests to include/lru.h + fixes small issues

Reviewed-by: Loic Dachary <loic@dachary.org>
Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1954 from dachary/wip-8307-erasure-code-profile-implicit-creation

erasure code profile implicit creation

Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>

Merge pull request #1964 from ceph/wip-osd-configs

osd: add sanity check/warning on a few key configs

Reviewed-by: Loic Dachary <loic@dachary.org>

Merge pull request #1966 from dachary/wip-erasure-code-profile-default

DNM: erasure-code: create default profile if necessary

Reviewed-by: Sage Weil <sage@inktank.com>

PendingReleaseNotes: note about keyvaluestore-dev on-disk format change

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #1941 from yuyuyu101/wip-8564

Fix write operation on a deleted object in the same transaction(KeyValueStore)

Reviewed-by: Sage Weil <sage@inktank.com>

common: Enforces the methods lru_pin() and lru_unpin()

If lru_*pin() is called twice, the counter will be incr/decr
incorrectly since it will count more/less pinned objects than
there is and so corrupts the balancing (lru_adjust()).

Signed-off-by: Sahid Orentino Ferdjaoui <sahid.ferdjaoui@cloudwatt.com>

common: Fixes issue with lru_clear() + add new test

The method lru_clear() must set attribute lru_num to zero
after lru_top, lru_bot and lru_mid are reseted. indeed, lru_num
is the total number of elements found in all of them.

Also the test insures the good behavior of the method
lru_adjust() - lru_touch() calls lru_adjust every time
to balance lru_top and lru_bot by the value of lru_midpoint.

Signed-off-by: Sahid Orentino Ferdjaoui <sahid.ferdjaoui@cloudwatt.com>

common: Adds simple tests to verify good behavior

Adds new tests to the lib include/lru.h to verify methods
insert_*.

Signed-off-by: Sahid Orentino Ferdjaoui <sahid.ferdjaoui@cloudwatt.com>

ceph: output prompt only if stdin is tty

In loop mode, output prompt only if stdin is tty, i.e. if we are
interactive.

Signed-off-by: Ilya Dryomov <ilya.dryomov@inktank.com>

erasure-code: create default profile if necessary

http://tracker.ceph.com/issues/8601

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #1960 from dachary/wip-test-kill-race

tests: prevent kill race condition

Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>

osd: add sanity check/warning on a few key configs

Warn when certain config values are set to bad values.

Backport: firefly, dumpling
Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #1949 from ceph/wip-7774-3

Wip 7774 3

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #1915 from ceph/wip-8452-2

rgw: set meta object in extra flag when initializing it

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

osd: remove non const get_erasure_code_profile

It is not used and will unexpectedly create a new entry if selected when
the const version is intended.

Signed-off-by: Loic Dachary <loic@dachary.org>

tests: prevent kill race condition

When trying to kill a daemon, keep its pid in a variable instead of
retrieving it from the pidfile multiple times. It prevents the following
race condition:

  * try to kill ceph-mon
  * ceph-mon is in the process of dying and removed its pidfile
  * try to kill ceph-mon fails because the pidfile is not found
  * another ceph-mon is spawned and fails to bind the port
    because the previous ceph-mon is still holding it

Signed-off-by: Loic Dachary <loic@dachary.org>

osd: improve osd pool create error message readability

Add std::endl to separate independent error messages related to erasure
code.

Signed-off-by: Loic Dachary <loic@dachary.org>

erasure-code: consistent argument parsing for profiles

Remove the = from the goodchars of the erasure_code_profile argument of
osd pool create so that it is consistent with the goodchars of osd
erasure-code-profile set / rm.

Signed-off-by: Loic Dachary <loic@dachary.org>

erasure-code: OSDMonitor::get_erasure_code is a const

If it is not, the non const version of OSDMap::get_erasure_code_profile
is called and a profile is created as a side effect, which is not
intended.

http://tracker.ceph.com/issues/8307 refs: #8307

Signed-off-by: Loic Dachary <loic@dachary.org>

erasure-code: pool create must not create profiles

If a non existent profile is provided as an argument to osd pool create,
it must exit on error and not create the profile as a side effect.

http://tracker.ceph.com/issues/8307 refs: #8307

Signed-off-by: Loic Dachary <loic@dachary.org>

qa: extend cephfs_journal_tool_smoke

To provide coverage for the 'import' command
resulting in the same contents as before an
export/import cycle.

Signed-off-by: John Spray <john.spray@redhat.com>

qa: set +x on cephfs_journal_tool_smoke

Signed-off-by: John Spray <john.spray@redhat.com>

tools/cephfs: Purge trailing objects during import

Reviewed-by: Gregory Farnum <greg@inktank.com>
Signed-off-by: John Spray <john.spray@inktank.com>

tools/cephfs: error handling in journal_export

Was failing on unreadable journal even if
only trying to do an import.

Was calling messenger destructor without init
on early failures due to early instantiation
of Dumper.

Signed-off-by: John Spray <john.spray@inktank.com>

tools/cephfs: Clean up waits in Dumper

Remove some LOC by using C_SaferCond.

Signed-off-by: John Spray <john.spray@inktank.com>

Merge pull request #1955 from ceph/wip-8585

rgw: set a default data extra pool name

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #1933 from ceph/wip-8556

osd: do not require EC feature bit from clients.

Reviewed-by: Greg Farnum <greg@inktank.com>

osd/OSDMap: do not require ERASURE_CODE feature of clients

Just because an EC pool exists in the cluster does not mean tha tthe client
has to support the feature:

1) The way client IO is initiated is no different for EC pools than for
   replicated pools.
2) People may add an EC pool to an existing cluster with old clients and
   locking those old clients out is very rude when they are not using the
   new pool.
3) The only direct client user of EC pools right now is rgw, and the new
   versions already need to support various other features like CRUSH_V2
   in order to work.  These features are present in new kernels.

Fixes: #8556
Backport: firefly
Signed-off-by: Sage Weil <sage@inktank.com>

osd/OSDMap: make get_features() take an entity type

Make the helper that returns what features are required of the OSDMap take
an entity type argument, as the required features may vary between
components in the cluster.

Backport: firefly
Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #1957 from ceph/wip-doc-os-recommendations

doc: Updated the OS Recommendations for Firefly.

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

doc: Updated the OS Recommendations for Firefly.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Updated the example configuration.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Updated doc for more recent versions.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

Merge pull request #1939 from ceph/wip-mon

mon: a few basic perfcounter guages

Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>