Merge pull request #3933 from dachary/wip-10617-firefly

osd: do not ignore deleted pgs on startup

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3932 from dachary/wip-10546-firefly

mon: Monitor: fix timecheck rounds period

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3931 from dachary/wip-10512-firefly

osd: requeue blocked op before flush it was blocked on

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3930 from dachary/wip-10497-firefly

librados: Translate operation flags from C APIs

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3955 from dachary/wip-10059-firefly

PG: always clear_primary_state

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3954 from dachary/wip-10014-firefly

ObjectStore: Don't use largest_data_off to calc data_align.

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3947 from dachary/wip-9555-firefly

osd: check that source OSD is valid for MOSDRepScrub

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3964 from dachary/wip-10157-firefly

PGLog: include rollback_info_trimmed_to in (read|write)_log

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3960 from dachary/wip-6003-firefly

FileJournal: fix journalq population in do_read_entry()

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3950 from dachary/wip-9985-firefly

Get the currently atime of the object in cache pool for eviction

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Xinze Chi <xmdxcxz@gmail.com>

Merge pull request #3935 from dachary/wip-10723-firefly

Fix memory leak in python rados bindings

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3866 from ceph/wip-cot-firefly

Backport ceph-objectstore-tool changes to firefly

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3996 from dzafman/wip-10676

Fix ceph command manpage to match ceph -h

Reviewed-by: Xinxin Shu <xinxin.shu5040@gmail.com>

Merge pull request #3927 from dachary/wip-10351-firefly

mount.ceph: avoid spurious error message

Reviewed-by: Yan, Zheng <zyan@redhat.com>

doc: Minor fixes to ceph command manpage

Fixes: #10676
Signed-off-by: David Zafman <dzafman@redhat.com>

doc: Fix ceph command manpage to match ceph -h (firefly)

Improve synopsis section
Fixes: #10676
Signed-off-by: David Zafman <dzafman@redhat.com>

rados.py: keep reference to python callbacks

If we don't keep a reference to these, the librados aio calls will
segfault since the python-level callbacks will have been garbage
collected. Passing them to aio_create_completion() does not take a
reference to them. Keep a reference in the python Completion object
associated with the request, since they need the same lifetime.

This fixes a regression from 60b019f69aa0e39d276c669698c92fc890599f50.

Fixes: #10775
Backport: dumpling, firefly, giant
Signed-off-by: Josh Durgin <jdurgin@redhat.com>
(cherry picked from commit 36d37aadbbbece28d70e827511f1a473d851463d)

doc: Changes format style in ceph to improve readability as html.

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>
(cherry picked from commit 8b796173063ac9af8c21364521fc5ee23d901196)

osd: check that source OSD is valid for MOSDRepScrub

Make sure the message we got from the peer OSD is valid.  Specifically,
this avoids a race like this:

- A marks down B
- B sends MOSDRepScrub
- A accepts connection from B as new
- A replies to scrub
- B crashes because msgr seq 1 < expected seq 1000+

See #8880 for the most recent fix for requests.

Fixes: #9555
Backport: giant, firefly
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 847e5e102522d651aa9687a54aaafcebf3afc596)

Conflicts:
    src/osd/OSD.cc: require functions first argument is now a reference

osd: pass Message* to most require_* helpers

These do nothing but op->get_req(); pass the Message* explicitly so that
non-OpRequest callers can use them.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 707a111d53efb09b3471dd3788b86d2bfee4e96f)

Conflicts:
src/osd/OSD.cc
src/osd/OSD.h
        significantly changes had to be made but in a simple way and
stays within the scope of the original commit

PGLog: include rollback_info_trimmed_to in (read|write)_log

Fixes: #10157
Backport: firefly, giant
Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit 1fe8b846641486cc294fe7e1d2450132c38d2dba)

Conflicts:
src/osd/PGLog.cc
        in the context coll_t::META_COLL was replaced with META_COLL

FileJournal: fix journalq population in do_read_entry()

Fixes: 6003
Backport: dumpling, firefly, giant
Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit bae1f3eaa09c4747b8bfc6fb5dc673aa6989b695)

Conflicts:
src/os/FileJournal.cc
        because reinterpret_cast was added near two hunks after firefly

osd/ReplicatedPG: fix compile error

From 1fef4c3d541cba360738437420ebfa2447d5802e.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 4a9ad7dc2da6f4fa6a64235776a3f1d2799aef60)

PG: always clear_primary_state on new interval, but only clear pg temp if not primary

Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit f692bfe076b8ddb679c6d1a6ea78cc47f0876326)

PG: always clear_primary_state when leaving Primary

Otherwise, entries from the log collection process might leak into the next
epoch, where we might end up choosing a different authoritative log.  In this
case, it resulted in us not rolling back to log entries on one of the replicas
prior to trying to recover from an affected object due to the peer_missing not
being cleared.

Fixes: #10059
Backport: giant, firefly, dumpling
Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit c87bde64dfccb5d6ee2877cc74c66fc064b1bcd7)

ObjectStore: Don't use largest_data_off to calc data_align.

If largest_data_off % CEPH_PAGE_SIZE != 0, the get_data_aligment return
a erro value. This make the FileJouranl::align_bl to memcopy much data.

Tested-by: Sage Weil <sage@redhat.com>
Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>
(cherry picked from commit a1aa70f2f21339feabfe9c1b3c9c9f97fbd53c9d)

Get the currently atime of the object in cache pool for eviction

Because if there are mutiple atime in agent_state for the same object, we should use the recently one.

Signed-off-by: Xinze Chi <xmdxcxz@gmail.com>
(cherry picked from commit 1fef4c3d541cba360738437420ebfa2447d5802e)

Merge pull request #3828 from dachary/wip-10425-firefly

librados: fix resources leakage in RadosClient::connect()

Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #3826 from dachary/wip-10257-firefly

mon: PGMonitor: several stats output error fixes

Reviewed-by: Joao Eduardo Luis <joao@redhat.com>

Merge pull request #3824 from dachary/wip-10353-firefly

crush: set_choose_tries = 100 for erasure code rulesets

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3823 from dachary/wip-10787-firefly

mon/OSDMonitor: do not trust small values in osd epoch cache

Reviewed-by: Joao Eduardo Luis <joao@redhat.com>

Merge pull request #3915 from dachary/wip-10080-firefly

SimpleMessenger: allow RESETSESSION whenever we forget an endpoint

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Fix memory leak in python rados bindings

A circular reference was inadvertently created when using the
CFUNCTYPE binding for callbacks for the asynchronous i/o callbacks.
This commit refactors the usage of the callbacks such that the
Ioctx object does not have a class reference to the callbacks.

Fixes: #10723
Backport: giant, firefly, dumpling
Signed-off-by: Billy Olsen <billy.olsen@gmail.com>
Reviewed-by: Dan Mick <dmick@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
(cherry picked from commit 60b019f69aa0e39d276c669698c92fc890599f50)

osd: do not ignore deleted pgs on startup

These need to get instantiated so that we can complete the removal process.

Fixes: #10617
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 879fd0c192f5d3c6afd36c2df359806ea95827b8)

mon: Monitor: fix timecheck rounds period

Fixes: #10546
Backports: dumpling?,firefly,giant

Signed-off-by: Joao Eduardo Luis <joao@redhat.com>
(cherry picked from commit 2e749599ac6e1060cf553b521761a93fafbf65bb)

osd: requeue blocked op before flush it was blocked on

If we have request A (say, cache-flush) that blocks things, and then
request B that gets blocked on it, and we have an interval change, then we
need to requeue B first, then A, so that the resulting queue will keep
A before B and preserve the order.

This was observed on this firefly run:

  ubuntu@teuthology:/a/sage-2015-01-09_21:43:43-rados-firefly-distro-basic-multi/694675

Backport: giant, firefly
Fixes: #10512
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 11bdfb4131ecac16d4a364d651c6cf5d1d28c702)

librados: Translate operation flags from C APIs

The operation flags in the public C API are a distinct enum
and need to be translated to Ceph OSD flags, like as happens in
the C++ API. It seems like the C enum and the C++ enum consciously
use the same values, so I reused the C++ translation function.

Signed-off-by: Matthew Richards <mattjrichards@gmail.com>
(cherry picked from commit 49d114f1fff90e5c0f206725a5eb82c0ba329376)

Conflicts:
src/librados/librados.cc
        comes from lttng tracepoints introduced after firefly

mount.ceph: avoid spurious error message

/etc/mtab in most modern distributions is a symbol link to
/proc/self/mounts.

Fixes: #10351
Signed-off-by: Yan, Zheng <zyan@redhat.com>
(cherry picked from commit bdd0e3c4bda97fe18487a58dd173a7dff752e1a2)

SimpleMessenger: allow RESETSESSION whenever we forget an endpoint

In the past (e229f8451d37913225c49481b2ce2896ca6788a2) we decided to disable
reset of lossless Pipes, because lossless peers resetting caused trouble and
they can't forget about each other. But they actually can: if mark_down()
is called.

I can't figure out how else we could forget about a remote endpoint, so I think
it's okay if we tell them we reset in order to clean up state. That's desirable
so that we don't get into strange situations with out-of-whack counters.

Fixes: #10080
Backport: giant, firefly, dumpling

Signed-off-by: Greg Farnum <gfarnum@redhat.com>
(cherry picked from commit 8cd1fdd7a778eb84cb4d7161f73bc621cc394261)

0.80.9

osd: Get pgid ancestor from last_map when building past intervals

Fixed OSD::build_past_intervals_parallel() and PG::generate_past_intervals()

Fixes: #10430
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 0c5b66da7a9ba516340d06d9e806beb9d1040d0e)

Conflicts:
src/osd/OSD.cc

osd: Pass oldest_map when generating past intervals

From load_pgs() the superblock hasn't been published yet
so we need to retrieve the value of oldest_map depending on the context.

Fixes: #10427
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 7fb721c1ceb39b38ca2e653299bcf51e109411d7)

Conflicts:
src/osd/OSD.cc

osd: build_push_op() handle short reads so recipient doesn't crash

Fixes: #8121
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit b9a04846d18e1c6621d7f2794ec1fae02875bed2)

ceph_objectstore_tool: fix check_output on python2.6

* backported the subprocess.check_output from python2.7

Fixes: #10756
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 15350a088d84bc6fc664f0d3f5d09b35f58b2144)

Conflicts:
src/test/ceph_objectstore_tool.py

ceph_objectstore_tool: For import get object_info_t available for each object

Add object_info_t to object_begin so we have at object create time
This will be useful for importing from multiple erasure coded exports

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 05d916ed12f361da48ef73953bcc0cef465fcc2a)

ceph_objectstore_tool: Handle import of pg which OSD has split

Fail import if import data doesn't include OSDMap and can't find it locally
See if local map can be read for import's epoch
Jump to current epoch like a split would if local map not present

Fixes: #9781
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit afda6e4f3b98cc1773fd014583dfb5e1f214a939)

Conflicts:
src/tools/ceph_objectstore_tool.cc

ceph_objectstore_tool: On import following a split skip objects no longer in pg

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 68b27e25a3729566c3a22c0b71f70f7f3aca29a3)

Conflicts:
src/tools/ceph_objectstore_tool.cc

ceph_objectstore_tool: Verify that object pool and pgid pool match

Also, earlier check for invalid --pgid with import op

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit da3be80497a1b1878ee5d2015f8840d202a83aa2)

ceph_objectstore_tool: Check for splits and fail import if there were splits

Add osdmap into metadata_section
On export put metadata_section before file data

Fixes: #9780
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 19fdeea8b67091ed044ebce25799d3237b4d734a)

ceph_objectstore_tool: Add special exit code for test infrastructure

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit b1f12f09c0211b608178f5ca2e292ab1765ce620)

ceph_objectstore_tool: Check that pool exists before allowing import

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit f24f646d870703b7b79563cdbc20920248be6142)

Conflicts:
src/tools/ceph_objectstore_tool.cc

ceph_objectstore_tool: Check cluster_fsid before allowing an import

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 196c8112dc9a6e8780b05d6c579cd7fbd5b07589)

ceph_objectstore_tool: Allow the metadata_section to be anywhere in the export

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 62dd912f1192b28700a15b02507a8c9efd710cb5)

Conflicts:
src/tools/ceph_objectstore_tool.cc

ceph_objectstore_tool: import-rados shouldn't import internal namespace objects

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit f727d2eaf50b0351feb9f12dcd65d50fb6eff7e9)

ceph_objectstore_tool: Get g_ceph_context available to import-rados

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit ddc4613ec71752e5dccbbfe6dc078b86f0f186a9)

Conflicts:
src/tools/ceph_objectstore_tool.cc

ceph_objectstore_tool: Fix import-rados skipping of snapshots

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit fe936026ed87c9f95f3b7ad235b24c22e8de5f55)

ceph_objectstore_tool: read_fd() doesn't handle ^D from tty stdin, don't allow

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 5cb692528e3ac0ebea3f1714b3ac43f69d176888)

ceph-objectstore-tool: Remove --pretty-format and use new --format options

Call new_formatter() with --format specified argument

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 22b71744bb0cb56434d5f6214ccea7d81f771860)

Conflicts:
src/tools/ceph_objectstore_tool.cc

ceph_objectstore_tool: Describe super_ver values

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 0aeba0f216a54390288b5e3d6147deb31877f744)

ceph_objectstore_tool.cc: reduce scope of variable

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
(cherry picked from commit 78542f9a901c05e627b53b5306ea604be3bc82e8)

ceph_objectstore_tool.cc: prefer ++operator for non-primitive iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
(cherry picked from commit 252fc03ba21c7b09922598a8d40997fc639bb994)

ceph_objectstore_tool: Prevent tool from transitioning struct_v on rm-past-intervals

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit f056bdf93980a0a8e6157dbb124a79389a8f1a3c)

ceph_objectstore_tool: Accept json object with --pgid instead of array

It isn't anticipated that anyone would use this but keeps backward compatible

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 59b423e2e8846b098326fdec440de46b8e3d2769)

ceph_objectstore_tool: Improve object spec parsing error messages

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit de6384fda183801c16af1b61ed36eaed289bb4f6)

ceph_objectstore_tool: Fix errors messages in newer code

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit eae7c02fded460f6c8aaf18aa83d2730b89e0eb1)

ceph_objectstore_tool: Remove extraneous endl on error throw messages

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 32c832f0c62259a492d1c934c56ac165496763a0)

ceph_objectstore_tool: Add --format and --pretty-format support

--pretty-format defaults true
Add --format so xml output can be requested
--op list defaults to single line of json per object
To override this more human readable output use --pretty-format=false
Add testing of --op list special handling

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit cca85a534fe488ae314400e8faad55a758609467)

ceph_objectstore_tool: Strip _* (always _head) from pgid in list entry output

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 5eacd3c5f39766cb8be6b3251d139d16431cf6b6)

ceph_objectstore_tool: BUG: --op list wasn't including snapshots

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit b617ee2d45886ec6b3dc0db0edbf814ea5748083)

ceph_objectstore_tool: For terminal output of list one object per line

Instead of a parsable array make it easier to cut and paste listed objects

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 5a66db9418aeed31fec98999c5053dab357d9c1e)

ceph_objectstore_tool: In error case umount objectstore

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit bc6ea9cb8a2b86c73f7f15bc46145177ccf91f4b)

objectstore_tool: test --op list variants

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit ddba2676c4c48c2a556f5d4ffd817bfe134a9153)

objectstore_tool: parse new object description format

The object format changed from

    {json object}

to

    [pgid,{json object}]

The parser is updated accordingly. If the --pgid is present, check that
it equals the pgid from the object description.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit df9d5c5cfd8b0ff793647a592c7661965cef5c92)

objectstore_tool: filter --op list and explore all PGs

The positional object name is used to filter the output of --op list and
only show the objects with a matching name. If both the object name and
the pgid are omitted, all objects from all PGs are displayed.

The output format is changed from

    {"oid":"GROUP","key":"","snapid":-2,
     "hash":2659194943,"max":0,"pool":0,"namespace":""}

to

    [["0.7_head",{"oid":"GROUP","key":"","snapid":-2,
                  "hash":2659194943,"max":0,"pool":0,
                  "namespace":""}]]

where the first member is the pgid where the object is found.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit c69aaceac7f370e5369d511bf17898adc338ae43)

objectstore_tool: lookup objects by name

If the object is not a parsable JSON string, assume an object name and
look it up in all the PGs. If multiple objects have the same name, only
apply the command to one of them. It is primarily useful in a test
environment where the names of the tests objects are known and only a
small number of objects exists. It replaces the following:

    path='--data-path dev/osd0 --journal-path dev/osd0.journal'
    for pgid in $(./ceph_objectstore_tool $path --op list-pgs) ; do
      object=$(./ceph_objectstore_tool $path --pgid $pgid --op list |
               grep '"oid":"NAME"')
      test -n "$object" && break
    done
    ./ceph_objectstore_tool $path --pgid $pgid "$object" remove

with:

    ./ceph_objectstore_tool $path NAME remove

http://tracker.ceph.com/issues/10192 Fixes: #10192

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit 7c1165f96391821c00cca1ac04b3433dbec6bb6e)

Conflicts:
src/tools/ceph_objectstore_tool.cc

objectstore_tool: refactor list-lost and fix-lost

Abstract out the PG exploration loops and encapsulate the list-lost and
fix-lost semantic in a callable object.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit d9e747b1bdb53d1fe543ef311e3db35fb78d8051)

objectstore_tool: update usage strings

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit a90233c8b79ae6c035b5169c7f8809f853631689)

Conflicts:
src/tools/ceph_objectstore_tool.cc

ceph_objectstore_tool: Add feature called set-allow-sharded-objects

Uses --op set-allow-sharded-objects option
This operation will be rejected if on the target OSD's osdmap there is
    at least one OSD which does not support ERASURE CODES.
Prompt the user that they could import if sharded state allowed
Prompt the user to use new feature if sharded state found inconsistent

Fixes: #10077
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit f3dab446fc8e58b3b3d9334b8c38722e73881b9e)

Conflicts:
src/tools/ceph_objectstore_tool.cc

ceph_objectstore_tool: Add utility routine get_osdmap()

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit b3021b0d3362000b5938d406ed2e032a8eb38405)

Conflicts:
src/tools/ceph_objectstore_tool.cc

ceph_objectstore_tool: Clear ...INCOMPAT_SHARDS from feature if exporting replicated pg

Don't require importing OSD to have shards feature for replicated pg

http://tracker.ceph.com/issues/10077 Fixes: #10077

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 86baf2d38170ef19de2dd5e9ce3f280237d8474d)

tests: ceph_objectstore_tool.py test all variants of export/import

Handle change of error message text

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 9e53c3554c71121f2e2dd0234b5232da37ad5a1b)

ceph_objectstore_tool: Make --file option consistent by treating "-" as stdout/stdin

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 8c87f3284f87d1121218cb7f41edc81b74c9df29)

tests: ceph_objectstore_tool.py fix list-attr for erasure code

Adding testing of xattr for erasure coded shards
Fix error message when finding an unexpected xattr key

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit cbecab477a70782f2f69258f035e78fb5c829810)

Conflicts:
src/test/ceph_objectstore_tool.py

tests: ceph_objectstore_tool.py check for malformed JSON for erasure code objs

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 40717aa4c399e87d2c3e32038f78788eb213f87d)

tests: ceph_objectstore_tool.py fix off by 1 ATTR_OBJS handling

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit eaf1d1e35243566c46b478788e79e0ebf7583015)

tests: ceph_objectstore_tool.py skip if /dev/tty fails

Some environments do not have a /dev/tty. When opening /dev/tty fails,
skip the test instead of returning an error.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit 4c94f1778fdf483e9e0b62f89f7e46e78aeeebf3)

tests: ceph_objectstore_tool.py encapsulate init-ceph stop

Call init-ceph in kill_daemons and add a call to kill_daemon when main
returns on error so that it never leaves daemons hanging.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit fc435ff3623d196dd7cc375302acd8dfc9eb59fd)

tests: ceph_objectstore_tool.py main returns

Instead of calling sys.exit() the main function returns the desired exit
code.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit 0f3d7b1315f2b5595047d8bd13949ed0d9194bfa)

tests: ceph_objectstore_tool.py replace stop.sh with init-ceph

The stop.sh will stop all ceph-* processes. Use the init-ceph script
instead to selectively kill the daemons run by the vstart.sh cluster
used for ceph_objectstore_tool.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit e8f34bd62bf282144b8851fb9764cf4429a49c25)

tests: ceph_objectstore_tool.py use a dedicated directory

Set CEPH_DIR to a directory that is specific to ceph_objectstore_tool so
that it can run in parallel with other vstart.sh clusters.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit e303d1afde58e68c1f587962010da9e1f1278bc3)

Conflicts:
src/test/ceph_objectstore_tool.py

tests: ceph_objectstore_tool.py run faster by default

By default use only a small number of objects to speed up the tests. If
the argument "big" is given, use a large number of objects as it may
help find some problems.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit 235257c257aea98b770d9637957818c8aeb6c745)

tests: ceph_objectstore_tool.py run mon and osd on specific port

By default vstart.sh runs MDS but they are not needed for the tests,
only run mon and osd instead. Instead of using the default vstart.sh
port which may conflict with a already running vstart.sh, set the
CEPH_PORT=7400 which is not used by any other test run with make check.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit f04d4e7539bc8c1b6cf94db815f9dcdecc52faa2)

tests: ceph_objectstore_tool.py can use a WARN cluster

The tests do not need HEALTH_OK exclusively, a HEALTH_WARN cluster can
also run them successfully.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit 783378c019aaac36d542e1b12c0d64196ea21012)

tests: ceph_objectstore_tool.py use env python

Using #/usr/bin/env python instead of a hard coded path is more flexible
and can also be used to run from virtualenv.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit 74506d2506d03d05935cbe342fef9dc1d9022a13)

ceph_objectstore_tool: Fixes to make import work again

The is_pg() call is now true even for pgs pending removal, fix broken
    finish_remove_pgs() by removing is_pg() check.
Need to add create_collection() to the initial transaction on import

Fixes: #10090
Signed-off-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 5ce09198bf475e5c3a2df26232fa04ba9912b103)

Conflicts:
src/tools/ceph_objectstore_tool.cc

ceph_objectstore_tool: Accept CEPH_ARGS environment arguments

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 10fe7cfe561f91717f2ac2e13aeecc06a903704e)

ceph_objectstore_tool: MemStore needs a CephContext

Pass g_ceph_context to ObjectStore::create() needed by MemStore

Fixes: #9661
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 0b155d00c542f0d8b8b5b0324dac4b3cf7ff37b5)

ceph_objectstore_tool: Rename generated binary to ceph-objectstore-tool

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 4f72ba545e204a24a55adead43c61cb1d4394381)

Conflicts:
debian/ceph-test.install
src/.gitignore
src/test/ceph-helpers.sh (doesn't exist in firefly)
src/test/ceph_objectstore_tool.py
src/tools/ceph_objectstore_tool.cc

vstart.sh: warn less

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit a69b8450f642af91a352d0de4378d93828291933)

ceph_objectstore_tool: When exporting to stdout, don't cout messages

Fixes: #10128
Caused by a2bd2aa7

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 0d5262ac2f69ed3996af76a72894b1722a27b37d)
(cherry picked from commit 6cb9a2499cac2645e2cc6903ab29dfd95aac26c7)