rgw: Conditional PUT on ETag

Fixes: #8562
Signed-off-by: Ray Lv <raylv@yahoo-inc.com>

Merge pull request #3155 from athanatos/wip-8011

ReplicatedPG: fail a non-blocking flush if the object is being scrubbed

Reviewed-by: Sage Weil <sage@redhat.com>

ReplicatedPG: fail a non-blocking flush if the object is being scrubbed

Fixes: #8011
Backport: firefly, giant
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #3154 from athanatos/wip-10059

Wip 10059

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3156 from athanatos/wip-10150

ReplicatedPG::scan_range: an object can disappear between the list and t...

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3157 from athanatos/wip-10157

PGLog: include rollback_info_trimmed_to in (read|write)_log

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3158 from athanatos/wip-10259

osd_types: op_queue_age_hist and fs_perf_stat should be in osd_stat_t::o...

Reviewed-by: Sage Weil <sage@redhat.com>

ReplicatedPG::scan_range: an object can disappear between the list and the attr get

The first item in the range is often last_backfill, upon which writes
can be occuring.  It's trimmed off on the primary side anyway.

Fixes: 10150
Backport: dumpling, firefly, giant
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #2579 from yuandong1222/cache-hboject-filestore

Cache hobject filestore_key to avoid massive _reverse_nibbles calls

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3018 from tchaikov/wip-fix-safe_read_file

safe_io: do not set ending \0 in safe_read_file()

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3070 from ceph/wip-10080

SimpleMessenger: allow RESETSESSION whenever we forget an endpoint

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3150 from ceph/wip-10288

mon: fix `fs ls` on peons

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #3152 from dachary/wip-10296-journal-discard

os: do not attempt to detect discard by default

Reviewed-by: Sage Weil <sage@redhat.com>

tests: do not ignore ceph-disk.sh::test_activate_dev failure

The status of the test commands must be tested instead of being
silently ignored.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

os: do not attempt to detect discard by default

If the journal_discard option is false, the journal should no try to
guess if the device has discard capabilities.

http://tracker.ceph.com/issues/10296 Refs: #10296

Signed-off-by: Loic Dachary <ldachary@redhat.com>

mon: fix `fs ls` on peons

This was incorrectly using pending_mdsmap instead
of mdsmap.  We didn't notice in test because of
single-mon configurations.

Fixes: #10288
Backport: giant

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #3149 from ceph/wip-typo

mds: fix log typo

Reviewed-by: Loic Dachary <ldachary@redhat.com>

mds: fix log typo

s/trvserse/traverse/

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #3137 from thesues/fix-strtol

Fix libstriprados::remove, use strtoll insdead of strtol

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3038 from ceph/wip-mds-readonly

Wip mds readonly

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #3145 from XinzeChi/wip-finisher

Finisher: call cond signal if necessary in function queue

Reviewed-by: Sage Weil <sage@redhat.com>

Finisher: call signal if necessary in function queue

Signed-off-by: Xinze Chi <xmdxcxz@gmail.com>

Merge pull request #3119 from ceph/wip-8797

rados.py: remove Rados.__del__(); it just causes problems

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

Call Rados.shutdown() explicitly before exit

This is mostly a demonstration of good behavior, as the resources will
be reclaimed on exit anyway.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

rados.py: remove Rados.__del__(); it just causes problems

Recent versions of Python contain a change to thread shutdown that
causes ceph to hang on exit; see http://bugs.python.org/issue21963.
As it turns out, this is relatively easy to avoid by not spawning
threads on exit, as Rados.__del__() will certainly do by calling
shutdown(); I suspect, but haven't proven, that the problem is
that shutdown() tries to start() a threading.Thread() that never
makes it all the way back to signal start().

Also add a PendingReleaseNote and extra doc comments to clarify.

Fixes: #8797
Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #3129 from ceph/wip-gfcomplete-dirty

gf-complete: update submodule to newest upstream

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3071 from ceph/wip-10164

mds: drop dirty dentries in deleted directory

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #3135 from XinzeChi/wip-3

os/FileStore.cc: insert not empty list<Context*> to op_finisher/ondisk_finisher

Reviewed-by: Sage Weil <sage@redhat.com>

os/FileStore.cc: insert not empty list<Context*> to op_finisher/ondisk_finisher.

Signed-off-by: Xinze Chi <xmdxcxz@gmail.com>

Merge pull request #3140 from leseb/sample-ceph.conf

ceph.conf: update sample

Reviewed-by: Sage Weil <sage@redhat.com>

ceph.conf: update sample

This commit adds:

* client admin socket
* change some default
* re-indent

Signed-off-by: Sébastien Han <sebastien.han@enovance.com>

Fix libstriprados::remove, use strtoll insdead of strtol

Signed-off-by: Dongmao Zhang <deanraccoon@gmail.com>

Merge pull request #3120 from ceph/wip-10010

ceph-osd: don't close stderr if not daemonizing

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

mds: set dirfrag version when fetching dirfrag is skipped

Signed-off-by: Yan, Zheng <zyan@redhat.com>

ceph-osd: remove extra close of stderr

Otherwise, one loses log messages when running with -f or -d.  When
daemonizing, stderr is already closed in global_init_postfork_finish.

Fixes: #10010, #10113, #9810
Signed-off-by: Dan Mick <dan.mick@redhat.com>

gf-complete: update submodule to newest upstream

Keep the tree cleaner!

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #3073 from cholcombe973/get_pool_values_docs

Updating docs for Get Pool Value

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Modifying the docs to add the Get pool commands to match the CLI.
Signed-off-by: Chris Holcombe <chris.holcombe@nebula.com>

Merge pull request #3106 from danderson/asan-fixes

Asan fixes

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3113 from wonzhq/name-norm

mailmap: Zhiqiang Wang name normalization

Reviewed-by: Loic Dachary <ldachary@redhat.com>

mailmap: Zhiqiang Wang name normalization

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

Fix stack buffer overflow reported by ASan.

Signed-off-by: David Anderson <dave@natulte.net>

Fix alloc-dealloc mismatch reported by ASan (new[] vs. delete).

Signed-off-by: David Anderson <dave@natulte.net>

Fix alloc-dealloc mismatch reported by ASan (malloc vs. C++ delete).

Signed-off-by: David Anderson <dave@natulte.net>

Merge pull request #3103 from danderson/ceph-dev-mode-python-envvar

ceph: respect the PYTHON environment variable for dev mode.

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ceph: respect the PYTHON environment variable for dev mode.

On OSes where `python` is python3, dev mode's re-exec makes the
ceph tool fail. The standard way to fix this is by exporting
the PYTHON envvar pointing to the python2 interpreter.

Signed-off-by: Dave Anderson <dave@natulte.net>

Merge pull request #3099 from sileht/sileht/python-rados-object-lock

python-rados: Add object lock support

Reviewed-by: Loic Dachary <ldachary@redhat.com>

python-rados: Add object lock support

This change adds to the python binding the support of:
 - rados_lock_exclusive
 - rados_lock_shared
 - rados_unlock

http://tracker.ceph.com/issues/6114 Refs: #6114

Signed-off-by: Mehdi Abaakouk <sileht@redhat.com>

Merge pull request #3035 from dachary/wip-10197-arch-arm

arch: add support for HW_CAP based neon runtime detection

Reviewed-by: Sage Weil <sage@redhat.com>

osd_types: op_queue_age_hist and fs_perf_stat should be in osd_stat_t::operator==

Fixes: 10259
Backport: giant, firefly, dumpling
Signed-off-by: Samuel Just <sjust@redhat.com>

common/ceph_context: don't import std namespace

This was broken by 7fed5dee4f96a83d1d6914f6fc0895bba2d15b99

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3020 from dachary/wip-10192-objectstore-tool-lookup

objectstore tool lookup by name & --op list filter

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

ceph_objectstore_tool: Accept json object with --pgid instead of array

It isn't anticipated that anyone would use this but keeps backward compatible

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_objectstore_tool: Improve object spec parsing error messages

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_objectstore_tool: Fix errors messages in newer code

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_objectstore_tool: Remove extraneous endl on error throw messages

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_objectstore_tool: Add --format and --pretty-format support

--pretty-format defaults true
Add --format so xml output can be requested
--op list defaults to single line of json per object
To override this more human readable output use --pretty-format=false
Add testing of --op list special handling

Signed-off-by: David Zafman <dzafman@redhat.com>

.gitignore: add ceph_perf_objectstore

Signed-off-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2987 from yuyuyu101/wip-9888

AsyncMessenger: Async event threads can shared by all AsyncMessenger

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3043 from yuyuyu101/objectstore-perf

Objectstore::Transaction Perf Program

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3094 from yuandong1222/bufferlist-get_contiguous

common: bufferlist::get_contiguous return 0 when param len == 0

Reviewed-by: Sage Weil <sage@redhat.com>

common: bufferlist::get_contiguous return 0 when param len == 0

Change-Id: I3eaee45b12c634687cd1b294b3c264aed8cab03a
Signed-off-by: Dong Yuan <yuandong1222@gmail.com>

ceph_objectstore_tool: Strip _* (always _head) from pgid in list entry output

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_objectstore_tool: BUG: --op list wasn't including snapshots

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_objectstore_tool: For terminal output of list one object per line

Instead of a parsable array make it easier to cut and paste listed objects

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_objectstore_tool: In error case umount objectstore

Signed-off-by: David Zafman <dzafman@redhat.com>

objectstore_tool: test --op list variants

Signed-off-by: Loic Dachary <ldachary@redhat.com>

objectstore_tool: parse new object description format

The object format changed from

    {json object}

to

    [pgid,{json object}]

The parser is updated accordingly. If the --pgid is present, check that
it equals the pgid from the object description.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

objectstore_tool: filter --op list and explore all PGs

The positional object name is used to filter the output of --op list and
only show the objects with a matching name. If both the object name and
the pgid are omitted, all objects from all PGs are displayed.

The output format is changed from

    {"oid":"GROUP","key":"","snapid":-2,
     "hash":2659194943,"max":0,"pool":0,"namespace":""}

to

    [["0.7_head",{"oid":"GROUP","key":"","snapid":-2,
                  "hash":2659194943,"max":0,"pool":0,
                  "namespace":""}]]

where the first member is the pgid where the object is found.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

objectstore_tool: lookup objects by name

If the object is not a parsable JSON string, assume an object name and
look it up in all the PGs. If multiple objects have the same name, only
apply the command to one of them. It is primarily useful in a test
environment where the names of the tests objects are known and only a
small number of objects exists. It replaces the following:

    path='--data-path dev/osd0 --journal-path dev/osd0.journal'
    for pgid in $(./ceph_objectstore_tool $path --op list-pgs) ; do
      object=$(./ceph_objectstore_tool $path --pgid $pgid --op list |
               grep '"oid":"NAME"')
      test -n "$object" && break
    done
    ./ceph_objectstore_tool $path --pgid $pgid "$object" remove

with:

    ./ceph_objectstore_tool $path NAME remove

http://tracker.ceph.com/issues/10192 Fixes: #10192

Signed-off-by: Loic Dachary <ldachary@redhat.com>

objectstore_tool: refactor list-lost and fix-lost

Abstract out the PG exploration loops and encapsulate the list-lost and
fix-lost semantic in a callable object.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

objectstore_tool: update usage strings

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Cache hobject filestore_key to avoid massive _reverse_nibbles calls

Signed-off-by: Dong Yuan <yuandong1222@gmail.com>

common/Initialize.h: add header file to Makefile.am

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

mds: drop dentry leases in deleted directory

we will never link dentry in deleted directory to inode, so it's
OK to remove client leases prematurely. This optimization allows
us to purge deleted directory inode earlier.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: disallow flush dentry/journal when MDS is readonly

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: properly unregister asok commands

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: drop dirty dentries in deleted directory

opened dirfrags and null dirty dentries in deleted directory inode
prevent MDCache::eval_stray() from purging the delete inode.

It's safe to not commit null dirty dentries in deleted directory to
corresponding dirfrag objects, because these dirfrag objects will be
deleted soon.

Fixes: #10164
Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #3081 from dachary/wip-10246-centos7

doc: add centos 7 to ceph-deploy installation

Reviewed by: John Wilkins <jowilkin@redhat.com>

Merge pull request #3031 from dachary/wip-10199-config-get-xml

common: admin sock output XML elements whitespace

Reviewed-by: Sahid Orentino Ferdjaoui <sahid.ferdjaoui@redhat.com>

arch: add support for HW_CAP based neon runtime detection

Rename the files from neon to arm to reflect the fact that it's related
to arm processors and also because NEON was renamed ASIMD later. The
NEON and ASIMD features are mutually exclusive. 32bits binaries will get
NEON and never ASIMD, if they run on ARMv7 or ARMv8. 64bits binaries
will only run on ARMv8 and get ASIMD and never NEON.

The flag remains with _neon and no other flag is introduced since there
is no risk of confusion. Besides people who care usually know NEON but
are not yet aware of the ASIMD renaming. Keeping the _neon name probably
saves some questions.

http://tracker.ceph.com/issues/10197 Fixes: #10197

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #2959 from ceph/wip-10104

rados.py: fix misnamed 'wait_*' routines, add true wait/wait-with-cb

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #2985 from CzBiX/quick-fix

rgw: check endpoints is not empty before use it.

This fixes a regression due to commit 3c4b782bcb5460523982a04b1e5c7963e08a1687

Reported-by: Mykola Golub <to.my.trociny@gmail.com>

pybind/test_rados: add test for wait_for_complete_and_cb()

Also, factor out some common code between the two tests

Signed-off-by: Dan Mick <dan.mick@redhat.com>

rados.py: fix misnamed 'wait_*' routines, add true wait/wait-with-cb

Also modify aio_read test for wait: write an object, take its active set
down, try to aio_read; verify read doesn't complete until active set is
allowed back up

Fixes: #10104
Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #3076 from ceph/wip-mon-mdsdne

MDSMonitor.cc: fix assertion caused by MDSMap::STATE_DNE state

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #3072 from ceph/wip-10231-gperftools-location

heap_profiler: support new gperftools header locations

gitbuilders are happy

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3084 from ceph/wip-rados-include

librados: Fix rados include problem reported by "sponce"

Reviewed-by: Sage Weil <sage@redhat.com>

librados: Fix rados include problem reported by "sponce"

Caused by a03f85a8e7fab296ea2df70a929a1c5e4aa0f7fb

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #3059 from dachary/wip-10125-radosgw-init

rgw: run radosgw as apache with systemd

Reviewed-by: Sage Weil <sage@redhat.com>

rgw: run radosgw as apache with systemd

Same as sysv.

http://tracker.ceph.com/issues/10125 Fixes: #10125

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #2997 from ceph/wip-snapmapper

osd: require SNAPMAPPER feature; drop tolerate_collection_add_enoent

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #2979 from ceph/wip-kill-category

osd: kill 'category' from pg_stats_t

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #2694 from dachary/wip-9728-jerasure-neon

erasure-code: jerasure support for NEON

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3082 from scuttlemonkey/wip-doc-10247

Changed os-recommendation table sorting to remove implied bias.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3048 from ceph/wip-10209

osd: tolerate sessionless con in fast dispatch path

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #3049 from dachary/wip-10211-erasure-code-buffer-alignement

erasure-code: enforce chunk size alignment

Reviewed-by: Samuel Just <sjust@redhat.com>

doc/release-notes: v0.89

Signed-off-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/next'

Changed os-recommendation table sorting to remove implied bias.

Fixes: #10247
signed-of by: Patrick McGarry <pmcgarry@redhat.com>