doc/release-notes: sort

meh

Signed-off-by: Sage Weil <sage@inktank.com>

doc/release-notes: fix indentation; sigh

Signed-off-by: Sage Weil <sage@inktank.com>

doc/release-notes: v0.73

Signed-off-by: Sage Weil <sage@inktank.com>

PendingReleaseNotes: note CRUSH and hashpspool default changes

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #930 from ceph/wip-hashpspool

enable hashpspool by default

Reviewed-by: Samuel Just <sam.just@inktank.com>

Revert "Partial revert "mon: osd pool set syntax relaxed, modify unit tests""

This reverts commit e80ab94bf44e102fcd87d16dc11e38ca4c0eeadb.

We accept non-CephInt arguments again, now that we've got the monitors
handling differing APIs intelligently.

Signed-off-by: Greg Farnum <greg@inktank.com>
Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon/OSDMonitor: take 'osd pool set ...' value as a string again

We ran into problems before when we made this a string because a mixed
cluster of mons might forward a client request with the wrong schema.
To make this work, we make the new code understand both the new and
old schema, and also backport a change to emperor and dumpling to
handle the new schema.

For the previous attempt to do this, see:
 337195f04653eed8e8f153a5b074f3bd48408998
 2fe0d0d97af95c22db80800f5b9da51f672d9407

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

Merge pull request #925 from ceph/wip-mon-api

Merge in changes to unify the API presented by the monitors and handle changes gracefully.

(Upgrade tests) Tested-by: Tamil Muthamizhan <tamil.muthamizhan@inktank.com>

Reviewed-by: Sage Weil <sage@inktank.com>
Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>

osd: enable HASHPSPOOL by default

Much like the CRUSH tunables, this first appears in kernel v3.9.

Unlike the CRUSH tunables, it does not appear in Ceph until v0.64
(post cuttlefish, pre dumpling).

Signed-off-by: Sage Weil <sage@inktank.com>

mon: if we're the leader, don't validate command matching

Classic-format commands never match our leader command set!

Signed-off-by: Greg Farnum <greg@inktank.com>

mon: by default, warn if some members of the quorum are "classic"

Signed-off-by: Greg Farnum <greg@inktank.com>

MemStore: update for the new ObjectStore interface

68fdcfa1cc249af859400a2ce4590fefbb2f525b changed the ObjectStore
interface in the 'next' branch, which was merged into master by
e5a02c33e23e4fbdc7bf0f16a5bbff61f4e37186. Unfortunately the
Memstore (added via the master branch) was not corrected for this
interface change.

Signed-off-by: Greg Farnum <greg@inktank.com>
Reviewed-by: David Zafman <david.zafman@inktank.com>

Merge branch 'next'

Merge pull request #927 from dachary/wip-crush-test

crush: remove crushtool test leftover

crush: remove crushtool test leftover

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #920 from dachary/wip-man

man: Ceph is also an object store

Reviewed-by: Sage Weil <sage@inktank.com>

Elector: use monitor's encoded command sets instead of our own

Signed-off-by: Greg Farnum <greg@inktank.com>

Merge pull request #865 from ceph/wip-doc-build-cluster

Wip doc build cluster

Monitor: encode and expose mon command sets

Signed-off-by: Greg Farnum <greg@inktank.com>

man: update man/ from doc/man/8

As explained in admin/manpage-howto.txt

Signed-off-by: Loic Dachary <loic@dachary.org>

man: Ceph is also an object store

Replace

   Ceph distributed file system

with

   Ceph distributed storage system

to help reduce the idea that Ceph is just a file system.

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #923 from dachary/wip-crush-test

CrushTester patches and documentation

Reviewed-by: Sage Weil <sage@inktank.com>

os/MemStore: do on_apply_sync callback synchronously

We can easily deadlock if we put this in the Finisher thread behind other
work; do it synchronously!

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Yehuda Sadeh <yehuda@inktank.com>

v0.73

Elector: keep a list of classic mons instead of each mon's commands

We aren't actually using the sets, so don't bother keeping them.

Signed-off-by: Greg Farnum <greg@inktank.com>

crush: implement --show-bad-mappings for indep

Support the presence of ITEM_NONE device numbers in the indep mapping as
proof of a bad mapping. Implement the associated unit tests.

Signed-off-by: Loic Dachary <loic@dachary.org>

crush: add unitest for crushtool --show-bad-mappings

Signed-off-by: Loic Dachary <loic@dachary.org>

crush: remove scary message string

The string is no longer used and can be removed.

Signed-off-by: Loic Dachary <loic@dachary.org>

crush: document the --test mode of operations

Signed-off-by: Loic Dachary <loic@dachary.org>

Monitor: Elector: share the classic command set if we have a classic mon

The leader now checks to see if any monitors did not provide their
command set, and if so, shares the list of "classic" commands instead
of his own set. This will prevent users from seeing different commands
(depending on whether they connect to an old or new mon) while
performing upgrades, and will make it really obvious if they forgot
to upgrade one of the monitors!

Signed-off-by: Greg Farnum <greg@inktank.com>

Elector: share local command set when deferring

We're about to use this at a basic level, to identify when we have
"classic" monitors in-quorum, but could also do something more
sophisticated like a set intersection on the commands.

Signed-off-by: Greg Farnum <greg@inktank.com>

Monitor: import MonCommands.h from original Dumpling and expose it

If the Elector doesn't receive a set of commands from the elected leader, it
assumes the monitor is "classic" and uses the Dumpling command set as
the leader set.

Signed-off-by: Greg Farnum <greg@inktank.com>

Monitor: validate incoming commands against the leader's set too

Then check against our own, and forward if we don't recognize it
or for some reason don't match.

Signed-off-by: Greg Farnum <greg@inktank.com>

Monitor: disseminate leader's command set instead of our own

Signed-off-by: Greg Farnum <greg@inktank.com>

Elector: transmit local api on election win, accept leader's on loss

If we're the leader, just point to our local set. Disseminating these
will let peons advertise the full command set supported by the leader.
INCOMPLETE: does not yet handle winning Electors who do not send a command set.

Signed-off-by: Greg Farnum <greg@inktank.com>

messages: make room for passing supported monitor commands in MMonElection

We're going to use this space to let leader tell everybody what
commands it supports.

Signed-off-by: Greg Farnum <greg@inktank.com>

Monitor: pull command mapping out of _allowed_command()

We want to be able to validate commands against both the leader and
local command sets, so make that functionality generic.

Signed-off-by: Greg Farnum <greg@inktank.com>

Merge pull request #918 from ceph/port/misc

Misc portability patches

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #922 from dachary/wip-crush-choose-tries

crush: fix map->choose_tries boundary test

Reviewed-by: Sage Weil <sage@inktank.com>

crush: --show-utilization* implies --show-statistics

--show-utilization* outputs only if --show-statistics is set, which is
confusing. Instead of failing, set --show-statistics to avoid the
confusion.

Signed-off-by: Loic Dachary <loic@dachary.org>

Monitor: add a separate leader_supported_commands

This isn't used yet, but will be shortly.

Signed-off-by: Greg Farnum <greg@inktank.com>

Monitor: expose local monitor commands to other compilation units

Signed-off-by: Greg Farnum <greg@inktank.com>

MonCommand: add operator== and operator!=

Signed-off-by: Greg Farnum <greg@inktank.com>

MonCommand: support encode/decode

Signed-off-by: Greg Farnum <greg@inktank.com>

encoding: fix [encode|decode]_array_nohead

We want to actually encode each element and keep it, rather than
writing each one at the position after the array end!

Signed-off-by: Greg Farnum <greg@inktank.com>

crush: add CrushTester accessors

Signed-off-by: Loic Dachary <loic@dachary.org>

crush: output --show-bad-mappings on err

Instead of using stdout so that it displays well when used in
conjunction with --show-statistics

Signed-off-by: Loic Dachary <loic@dachary.org>

crush: fix map->choose_tries boundary test

CrushWrapper::start_choose_profile allocates map->choose_tries with
choose_total_tries elements. When crush_choose_firstn sets a value, it
tests against map->choose_local_tries which could lead to memory
corruption if map->choose_total_tries is smaller than
map->choose_local_tries.

Another indesirable but non fatal side effect is that the output crushtool
--show-choose-tries will be truncated to choose_local_tries which is
set to a lower value than choose_total_tries by the default tuneables.

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #869 from ceph/wip-crush

crush changes for erasure coding

Reviewed-by: Loic Dachary <loic@dachary.org>
Reviewed-by: Samuel Just <sam.just@inktank.com>

librbd: remove unused private variable

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

TrackedOp: remove unused private variable

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

librbd: rename howmany to avoid conflict

A howmany macro exists on some platforms in standard headers, but there
really isn't any sort of standard that I've found. We just avoid the
conflict entirely this way.

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

Merge pull request #917 from ceph/port/compat

compat: define replacement TEMP_FAILURE_RETRY

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #919 from ceph/port/fdatasync

wbthrottle: use feature check for fdatasync

Reviewed-by: Sage Weil <sage@inktank.com>

wbthrottle: use feature check for fdatasync

Checking for fdatasync uses the same approach as the qemu configure
script. The relevant commit is d1722a27f552a22561104210e0afad4577878e53.
Here is a copy of the commit message which explains the check:

Under Darwin, a symbol exists for the fdatasync() function, so that our
link test succeeds. However _POSIX_SYNCHRONIZED_IO is set to '-1'.

According to POSIX:2008, a value of -1 means the feature is not
supported.
A value of 0 means supported at compilation time, and a value greater 0
means supported at both compilation and run time.

Enable fdatasync() only if _POSIX_SYNCHRONIZED_IO is '>0'.

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

rados_sync: fix mismatched tag warning

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

rados_sync: remove unused private variable

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

mon: check for sys/vfs.h existence

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

make: increase maximum template recursion depth

With clang on OSX spirit blows up without this.

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

compat: define replacement TEMP_FAILURE_RETRY

Not all platforms have it.

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

Merge remote-tracking branch 'gh/wip-fix-3x'

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge remote-tracking branch 'gh/wip-fix-tunables'

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

crush/CrushCompiler: make current set of tunables 'safe'

We can reenable this error the next time we add new tunables.

Signed-off-by: Sage Weil <sage@inktank.com>

crushtool: remove scary tunables messages

Signed-off-by: Sage Weil <sage@inktank.com>

crush/CrushCompiler: start with legacy tunables when compiling

Ensure that a crush file always compiled deterministically, even though
the default values for *new* maps has changed.

Signed-off-by: Sage Weil <sage@inktank.com>

crush: add indep data set to cli tests

This will help us catch things if we break the mapping.

Signed-off-by: Sage Weil <sage@inktank.com>

osdmaptool: fix cli tests for 3x

Signed-off-by: Sage Weil <sage@inktank.com>

osd: default to 3x replication

3x is the recommendation; it should be the default too.

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #913 from dachary/wip-crush-unittest

CrushWrapper::move_bucket unittest and minor fixes

Reviewed-by: Sage Weil <sage@inktank.com>

Revert "osd: default to 3x replication"

This reverts commit cb26fbde52f31b449af60acce3ced34e593d6e1e.

Fix unit tests and do integration tests first; this may have unexpected
consequences.

crush: detach_bucket must test item >= 0 not > 0

Since detach_bucket is a private helper solely used by move_bucket which
contains another ( correct ) safeguard, the code cannot be reached and
the problem can never happen. If another function uses detach_bucket,
it may happen.

Signed-off-by: Loic Dachary <loic@dachary.org>

crush: remove obsolete comments from link_bucket

Probably copy/pasted from move_bucket.

Signed-off-by: Loic Dachary <loic@dachary.org>

crush: remove redundant code from move_bucket

The following was introduced in 2012 by a2d0cff1b071bed84ac439e4fcf9ddfb936f89c8

  // un-set the device name so we can use add_item later
  build_rmap(name_map, name_rmap);
  name_map.erase(id);
  name_rmap.erase(id_name);

when insert_item refused to move a bucket for which a name already
exists. It was changed in 2013 by
4e2557a038dc1e8c68993ad8571d74e2eb8ea90a and now supports it. The
TestCrushWrapper unittest for move_bucket pass.

Signed-off-by: Loic Dachary <loic@dachary.org>

crush: unittest CrushWrapper::move_bucket

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #888 from ceph/wip-crush-tunables

default to bobtail-era crush tunables.

Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>

Merge pull request #903 from ceph/wip-memstore

memstore: reference ObjectStore backend

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #907 from ceph/wip-3x

osd: default to 3x replication

crush/mapper: dump indep partial progression for debugging

...if DEBUG_INDEP is #defined.

Signed-off-by: Sage Weil <sage@inktank.com>

PendingReleaseNotes: note change of CRUSH indep mode in release notes

Signed-off-by: Sage Weil <sage@inktank.com>

crush: add feature CRUSH_V2 for new indep mode and SET_*_TRIES rule steps

Signed-off-by: Sage Weil <sage@inktank.com>

crush: CHOOSE_LEAF -> CHOOSELEAF throughout

This aligns the internal identifier names with the user-visible names in
the decompiled crush map language.

Signed-off-by: Sage Weil <sage@inktank.com>

osd/OSDMap: fix feature calculation for CACHEPOOL

We need to include the faeture in the mask.

Signed-off-by: Sage Weil <sage@inktank.com>

crush/CrushCompiler: [de]compile set_choose[leaf]_tries rule step

Signed-off-by: Sage Weil <sage@inktank.com>

crush/CrushWrapper: set chooseleaf_tries to 5 for 'simple' indep rules

When making a generic indep rule, set the recursive retry to 5.  This gives
better overall results.

Signed-off-by: Sage Weil <sage@inktank.com>

crush/mapper: add SET_CHOOSE_TRIES rule step

Since we can specify the recursive retries in a rule, we may as well also
specify the non-recursive tries too for completeness.

Signed-off-by: Sage Weil <sage@inktank.com>

crush/mapper: apply chooseleaf_tries to firstn mode too

Parameterize the attempts for the _firstn choose method, and apply the
rule-specified tries count to firstn mode as well.  Note that we have
slightly different behavior here than with indep:

 If the firstn value is not specified for firstn, we pass through the
 normal attempt count.  This maintains compatibility with legacy behavior.
 Note that this is usually *not* actually N^2 work, though, because of the
 descend_once tunable.  However, descend_once is unfortunately *not* the
 same thing as 1 chooseleaf try because it is only checked on a reject but
 not on a collision.  Sigh.

 In contrast, for indep, if tries is not specified we default to 1
 recursive attempt, because that is simply more sane, and we have the
 option to do so.  The descend_once tunable has no effect for indep.

Signed-off-by: Sage Weil <sage@inktank.com>

crush/mapper: fix up the indep tests

Fix indentation.
Simplify+fix the changed vs moved calculation.
Use the new SET_CHOOSE_LEAF_TRIES command.

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #886 from ceph/wip-6922

Fix some pg_num change return codes and make them more resistant to mis-use

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #909 from dachary/wip-crush-unittest

more CrushWrapper unittest

crush: unittest CrushWrapper::get_immediate_parent

Signed-off-by: Loic Dachary <loic@dachary.org>

crush: unittest CrushWrapper::update_item

Signed-off-by: Loic Dachary <loic@dachary.org>

crush: unittest s/std::string/string/

Signed-off-by: Loic Dachary <loic@dachary.org>

crush: unittest use const instead of define

And reduce the depth of the hierarchy because three levels of buckets
capture the same cases as four levels.

Signed-off-by: Loic Dachary <loic@dachary.org>

crush: unittest CrushWrapper::check_item_loc

Signed-off-by: Loic Dachary <loic@dachary.org>

crush: unittest remove useless c->create()

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge remote-tracking branch 'origin/next'

osd: default to 3x replication

3x is the recommendation; it should be the default too.

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #901 from dachary/wip-crush-unittest

crush: check for invalid names in loc[]

Reviewed-by: Sage Weil <sage@inktank.com>

crush: check for invalid names in loc[]

Add the is_valid_crush_loc helper to test for invalid crush names in
insert_item and update_item, before performing any side
effect. Implement the associated unit tests.

Signed-off-by: Loic Dachary <loic@dachary.org>

osd: queue pg deletion after on_removal txn

The removal is normally so slow that these don't really race, but they
could.

Signed-off-by: Sage Weil <sage@inktank.com>