ceph_objectstore_tool.cc: prefer ++operator for non-primitive iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
(cherry picked from commit 252fc03ba21c7b09922598a8d40997fc639bb994)

ceph_objectstore_tool: Prevent tool from transitioning struct_v on rm-past-intervals

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit f056bdf93980a0a8e6157dbb124a79389a8f1a3c)

ceph_objectstore_tool: Accept json object with --pgid instead of array

It isn't anticipated that anyone would use this but keeps backward compatible

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 59b423e2e8846b098326fdec440de46b8e3d2769)

ceph_objectstore_tool: Improve object spec parsing error messages

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit de6384fda183801c16af1b61ed36eaed289bb4f6)

ceph_objectstore_tool: Fix errors messages in newer code

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit eae7c02fded460f6c8aaf18aa83d2730b89e0eb1)

ceph_objectstore_tool: Remove extraneous endl on error throw messages

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 32c832f0c62259a492d1c934c56ac165496763a0)

ceph_objectstore_tool: Add --format and --pretty-format support

--pretty-format defaults true
Add --format so xml output can be requested
--op list defaults to single line of json per object
To override this more human readable output use --pretty-format=false
Add testing of --op list special handling

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit cca85a534fe488ae314400e8faad55a758609467)

ceph_objectstore_tool: Strip _* (always _head) from pgid in list entry output

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 5eacd3c5f39766cb8be6b3251d139d16431cf6b6)

ceph_objectstore_tool: BUG: --op list wasn't including snapshots

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit b617ee2d45886ec6b3dc0db0edbf814ea5748083)

ceph_objectstore_tool: For terminal output of list one object per line

Instead of a parsable array make it easier to cut and paste listed objects

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 5a66db9418aeed31fec98999c5053dab357d9c1e)

ceph_objectstore_tool: In error case umount objectstore

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit bc6ea9cb8a2b86c73f7f15bc46145177ccf91f4b)

objectstore_tool: test --op list variants

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit ddba2676c4c48c2a556f5d4ffd817bfe134a9153)

objectstore_tool: parse new object description format

The object format changed from

    {json object}

to

    [pgid,{json object}]

The parser is updated accordingly. If the --pgid is present, check that
it equals the pgid from the object description.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit df9d5c5cfd8b0ff793647a592c7661965cef5c92)

objectstore_tool: filter --op list and explore all PGs

The positional object name is used to filter the output of --op list and
only show the objects with a matching name. If both the object name and
the pgid are omitted, all objects from all PGs are displayed.

The output format is changed from

    {"oid":"GROUP","key":"","snapid":-2,
     "hash":2659194943,"max":0,"pool":0,"namespace":""}

to

    [["0.7_head",{"oid":"GROUP","key":"","snapid":-2,
                  "hash":2659194943,"max":0,"pool":0,
                  "namespace":""}]]

where the first member is the pgid where the object is found.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit c69aaceac7f370e5369d511bf17898adc338ae43)

objectstore_tool: lookup objects by name

If the object is not a parsable JSON string, assume an object name and
look it up in all the PGs. If multiple objects have the same name, only
apply the command to one of them. It is primarily useful in a test
environment where the names of the tests objects are known and only a
small number of objects exists. It replaces the following:

    path='--data-path dev/osd0 --journal-path dev/osd0.journal'
    for pgid in $(./ceph_objectstore_tool $path --op list-pgs) ; do
      object=$(./ceph_objectstore_tool $path --pgid $pgid --op list |
               grep '"oid":"NAME"')
      test -n "$object" && break
    done
    ./ceph_objectstore_tool $path --pgid $pgid "$object" remove

with:

    ./ceph_objectstore_tool $path NAME remove

http://tracker.ceph.com/issues/10192 Fixes: #10192

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit 7c1165f96391821c00cca1ac04b3433dbec6bb6e)

Conflicts:
src/tools/ceph_objectstore_tool.cc

objectstore_tool: refactor list-lost and fix-lost

Abstract out the PG exploration loops and encapsulate the list-lost and
fix-lost semantic in a callable object.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit d9e747b1bdb53d1fe543ef311e3db35fb78d8051)

objectstore_tool: update usage strings

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit a90233c8b79ae6c035b5169c7f8809f853631689)

Conflicts:
src/tools/ceph_objectstore_tool.cc

ceph_objectstore_tool: Add feature called set-allow-sharded-objects

Uses --op set-allow-sharded-objects option
This operation will be rejected if on the target OSD's osdmap there is
    at least one OSD which does not support ERASURE CODES.
Prompt the user that they could import if sharded state allowed
Prompt the user to use new feature if sharded state found inconsistent

Fixes: #10077
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit f3dab446fc8e58b3b3d9334b8c38722e73881b9e)

Conflicts:
src/tools/ceph_objectstore_tool.cc

ceph_objectstore_tool: Add utility routine get_osdmap()

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit b3021b0d3362000b5938d406ed2e032a8eb38405)

Conflicts:
src/tools/ceph_objectstore_tool.cc

ceph_objectstore_tool: Clear ...INCOMPAT_SHARDS from feature if exporting replicated pg

Don't require importing OSD to have shards feature for replicated pg

http://tracker.ceph.com/issues/10077 Fixes: #10077

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 86baf2d38170ef19de2dd5e9ce3f280237d8474d)

tests: ceph_objectstore_tool.py test all variants of export/import

Handle change of error message text

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 9e53c3554c71121f2e2dd0234b5232da37ad5a1b)

ceph_objectstore_tool: Make --file option consistent by treating "-" as stdout/stdin

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 8c87f3284f87d1121218cb7f41edc81b74c9df29)

tests: ceph_objectstore_tool.py fix list-attr for erasure code

Adding testing of xattr for erasure coded shards
Fix error message when finding an unexpected xattr key

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit cbecab477a70782f2f69258f035e78fb5c829810)

Conflicts:
src/test/ceph_objectstore_tool.py

tests: ceph_objectstore_tool.py check for malformed JSON for erasure code objs

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 40717aa4c399e87d2c3e32038f78788eb213f87d)

tests: ceph_objectstore_tool.py fix off by 1 ATTR_OBJS handling

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit eaf1d1e35243566c46b478788e79e0ebf7583015)

tests: ceph_objectstore_tool.py skip if /dev/tty fails

Some environments do not have a /dev/tty. When opening /dev/tty fails,
skip the test instead of returning an error.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit 4c94f1778fdf483e9e0b62f89f7e46e78aeeebf3)

tests: ceph_objectstore_tool.py encapsulate init-ceph stop

Call init-ceph in kill_daemons and add a call to kill_daemon when main
returns on error so that it never leaves daemons hanging.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit fc435ff3623d196dd7cc375302acd8dfc9eb59fd)

tests: ceph_objectstore_tool.py main returns

Instead of calling sys.exit() the main function returns the desired exit
code.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit 0f3d7b1315f2b5595047d8bd13949ed0d9194bfa)

tests: ceph_objectstore_tool.py replace stop.sh with init-ceph

The stop.sh will stop all ceph-* processes. Use the init-ceph script
instead to selectively kill the daemons run by the vstart.sh cluster
used for ceph_objectstore_tool.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit e8f34bd62bf282144b8851fb9764cf4429a49c25)

tests: ceph_objectstore_tool.py use a dedicated directory

Set CEPH_DIR to a directory that is specific to ceph_objectstore_tool so
that it can run in parallel with other vstart.sh clusters.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit e303d1afde58e68c1f587962010da9e1f1278bc3)

Conflicts:
src/test/ceph_objectstore_tool.py

tests: ceph_objectstore_tool.py run faster by default

By default use only a small number of objects to speed up the tests. If
the argument "big" is given, use a large number of objects as it may
help find some problems.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit 235257c257aea98b770d9637957818c8aeb6c745)

tests: ceph_objectstore_tool.py run mon and osd on specific port

By default vstart.sh runs MDS but they are not needed for the tests,
only run mon and osd instead. Instead of using the default vstart.sh
port which may conflict with a already running vstart.sh, set the
CEPH_PORT=7400 which is not used by any other test run with make check.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit f04d4e7539bc8c1b6cf94db815f9dcdecc52faa2)

tests: ceph_objectstore_tool.py can use a WARN cluster

The tests do not need HEALTH_OK exclusively, a HEALTH_WARN cluster can
also run them successfully.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit 783378c019aaac36d542e1b12c0d64196ea21012)

tests: ceph_objectstore_tool.py use env python

Using #/usr/bin/env python instead of a hard coded path is more flexible
and can also be used to run from virtualenv.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit 74506d2506d03d05935cbe342fef9dc1d9022a13)

ceph_objectstore_tool: Fixes to make import work again

The is_pg() call is now true even for pgs pending removal, fix broken
    finish_remove_pgs() by removing is_pg() check.
Need to add create_collection() to the initial transaction on import

Fixes: #10090
Signed-off-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 5ce09198bf475e5c3a2df26232fa04ba9912b103)

Conflicts:
src/tools/ceph_objectstore_tool.cc

ceph_objectstore_tool: Accept CEPH_ARGS environment arguments

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 10fe7cfe561f91717f2ac2e13aeecc06a903704e)

ceph_objectstore_tool: MemStore needs a CephContext

Pass g_ceph_context to ObjectStore::create() needed by MemStore

Fixes: #9661
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 0b155d00c542f0d8b8b5b0324dac4b3cf7ff37b5)

ceph_objectstore_tool: Rename generated binary to ceph-objectstore-tool

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 4f72ba545e204a24a55adead43c61cb1d4394381)

Conflicts:
debian/ceph-test.install
src/.gitignore
src/test/ceph-helpers.sh (doesn't exist in firefly)
src/test/ceph_objectstore_tool.py
src/tools/ceph_objectstore_tool.cc

vstart.sh: warn less

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit a69b8450f642af91a352d0de4378d93828291933)

ceph_objectstore_tool: When exporting to stdout, don't cout messages

Fixes: #10128
Caused by a2bd2aa7

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 0d5262ac2f69ed3996af76a72894b1722a27b37d)
(cherry picked from commit 6cb9a2499cac2645e2cc6903ab29dfd95aac26c7)

ceph_objectstore_tool.cc: prefer ++operator for non-primitive iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
(cherry picked from commit a5468abe4459159e8a9f7a4f21d082bb414e1cdd)

Test: fixing a compile warning in ceph_objectstore_tool.cc

For the compiler's sake:
tools/ceph_objectstore_tool.cc:2547:15: warning: ‘r’ may be used
uninitialized in this function [-Wmaybe-uninitialized]

Signed-off-by: Zhiqiang Wang <wonzhq@hotmail.com>
(cherry picked from commit c3e1466b46076f133b62f98e2c0b712bdde0e119)

ceph_objectstore_tool: Bug fixes and test improvements

ceph_objectgstore_tool:
Fix bugs in the way collection_list_partial() was being called
which caused objects to be seen over and over again.

Unit test:
Fix get_objs() to walk pg tree for pg with sub-directories
Create more objects to test object listing code
Limit number of larger objects
Limit number of objects which get attributes and omaps

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit a03f719eb3a46f410550afce313e6720e0c27946)

ceph_objectstore_tool, test: Implement import-rados feature and unit test code

Fixes: #8276
Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 23ec93a86140c4b271b45d87c62682288079cbba)

test: ceph_objectstore_tool unit test improvements

Add namespaces to testing
Increase filesize so export will have multiple chunks
Put json for each object into the db dict

Signed-off-by: David Zafman <david.zafman@inktank.com>

ceph_objectstore_tool: Add operation "rm-past-intervals" for testing purposes

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 3694068b67fd625495c4511390bc5bcbfbbd28f5)

ceph_objectstore_tool: Add past_intervals to export/import code

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 9325ec15d4b89c5537cbcbda4c7594e25dc6e7df)

ceph_objectstore_tool: Minor improvements

Make all non-error non-debug output to stdout
Fix a message

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit a2bd2aa7babb4ad45ba21c70f9d179fda27742aa)

ceph_objectstore_tool, test: Add list-pgs operations and unit test case

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit f01e334c697057158354f0ce5ecff6d6ba8e2704)

Complete replacement of ceph_filestore_tool and ceph_filestore_dump
with unified ceph_objectstore_tool

Move list-lost-objects and fix-lost-objects features from
  ceph_filestore_tool to ceph_objectstore_tool as list-lost, fix-lost
Change --type to --op for info, log, export...operations
Add --type for the ObjectStore type (defaults to filestore)
Change --filestore-path to --data-path
Update installation, Makefile.am, and .gitignore
Fix and rename test case to match
  Add some additional invalid option checks

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 83fbc91e5c4e52cc1513f34908f99d2ac3b930df)

Renames and removal towards a unified ceph_objectstore_tool

Rename ceph_filestore_dump.cc and ceph_filestore_dump.py
Remove ceph_filestore_tool.cc

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 77864193a1162393ade783480aee39a232934377)

ceph_filestore_dump: Add set-omaphdr object command

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit b4d95cc85af9af64d33d541cd69c5f28fd45423b)

ceph_filestore_dump: Add get-omaphdr object command

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 30c0f3114e665acdd99e64bf0d2a7399b33e8d61)

ceph_filestore_dump: Add rm-omap object command

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 0fc6bd2777edf24a044f454beacf1647cc52f9fe)

ceph_filestore_dump: Add set-omap object command

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 50cd57e902fe508f98f63fea30626780b07561d9)

ceph_filestore_dump: Add get-omap object command

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit b50c43ce5e52f5bbcb3684f6793d50f34ed741d1)

ceph_filestore_dump: Add rm-attr object command

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 465d77733c7499fbd65bebe7141895714c625e0d)

ceph_filestore_dump: Add set-attr object command

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 95554e03dcb74b6d74b2f1b2891b3570abb187b8)

ceph_filestore_dump: Add get-attr object command

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 55d43c0e20fc853daec134449b9954248fd7ef31)

ceph_filestore_dump: Add set-bytes object command

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 3c24d1f46a624d0a053ad234997a1f8c8b036db5)

ceph_filestore_dump: Add get-bytes object command

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 869dd92cc8ec29a3a684f88c335d359f225bba24)

ceph_filestore_dump: Add list-omap object command

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 48890c7741d76cf92b5f589f49378ca57292e88b)

ceph_filestore_dump: Add list-attrs object command

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 00c6b0673288ca76fe144575b7af76eaa36f5857)

ceph_filestore_dump: Add --type list to output objects in a pg in json

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 844dabb7f311e68eba0293ae9ca4c68521745877)

ceph_filestore_dump: Add remove object command and usage for new commands

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 605caec64b036f8ab5ae451d7e9e7515d414f28e)

ceph_filestore_dump: Add utility function get_fd_data()

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit d4a9dafe442f139562497d746f80ba49faa954e8)

ceph_filestore_dump: Fail import/export with a tty for stdin/stdout

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 7520e504cf2cdd3de2f236acb2cbf8a5016e6698)

ceph_filstore_dump: Save if stdout is a tty and add routine to clean binary strings

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 3a574cc78b0e3ec6d8dd0c39ee20e7a54ad64056)

common: Add missing ghobject_t::decode() for json

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit c05f895d15a1d0e78ff5e9ae1a83f0a5424103d0)

Changes:
Adjusted for older shard_t

ceph_filestore_dump: Add --skip-journal-replay and --skip-mount-omap

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 2e9dcb256509e7c921556202052f0cc1dcd91398)

os: Add optional flags to generic ObjectStore creation (SKIP_JOURNAL_REPLAY
and SKIP_MOUNT_OMAP)

Only FileStore cares about these flags, so passed on during create()

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 3d9fde9d92322cd8ac3e3d8bcbf5b0a01ef0528b)

Conflicts:
src/os/FileStore.cc

ceph_filestore_dump: Improve debug output by showing actual offset

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 44b261d5d1b36528bfbcb37dbd866b615e14be99)

ceph_filestore_dump: Use cerr now that we aren't closing stderr

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 087c0f9d31e0f3d5bae7eac6231978105a71677e)

common,ceph_filestore_dump: Add ability for utilities to suppress library dout output

Suppress dout output with CODE_ENVIRONMENT_UTILITY_NODOUT
ceph_filestore_dump turns on dout output if --debug specified
When used it can still be enable with --log-to-stderr --err-to-stderr

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit f7f9b251fc377651d8da4cbfd1942c3b86f3247e)

ceph_filestore_dump: Export omap in batches for large omap case

New function get_omap_batch()
Create a TYPE_OMAP section for each batch

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 501dd3c05b8983159a289b021943cb828b908f53)

ceph_filestore_dump: Remove unused bufferlist databl

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 398b418e2b9f8260bcfacac8bcebea5beffcceca)

ceph_filestore_dump.cc: cleanup includes

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
(cherry picked from commit 8620609884243596d35b69c571d2da751e63cf2b)

Merge pull request #3852 from dachary/wip-10965-rgw-firefly

rgw: enable ipv6 in civetweb

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3853 from dachary/wip-10907-rgw-firefly

rgw: pass civetweb configurables to civetweb

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3851 from dachary/wip-10978-rgw-firefly

rgw: don't overwrite bucket / object owner when setting acls

Reviewed-by: Loic Dachary <ldachary@redhat.com>

rgw: don't overwrite bucket / object owner when setting acls

Fixes: #10978
Backport: hammer, firefly

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit eb13f2d4b60c031f16139f7cc4237c012644dd78)

Conflicts:
src/rgw/rgw_op.cc : trivial s/.empty()/== NULL/

rgw: pass civetweb configurables to civetweb

Fixes: #10907
Backport: firefly

Pass any configurables in the rgw frontends config line to civetweb.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 986d7554426764a149621ba733c5c075b94e0431)

civetweb: update submodule

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit f28fe744285117a1715eac0d08911cdb37285103)

rgw: update makefile to enable civetweb config

Fixes: #10965
Backport: hammer, firefly

Civetweb compilation now includes conf header to enable ipv6.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit caa90225bad9fe5e9c275e6189b3396b4d396e3f)

Conflicts:
src/rgw/Makefile.am
           radosgw_CFLAGS = -Icivetweb/include
        was not yet changed into
           radosgw_CFLAGS = -I$(srcdir)/civetweb/include
        in firefly

Merge pull request #3820 from tchaikov/firefly-pg-leak-10421

osd: fix PG leak in SnapTrimWQ._clear()

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3772 from ceph/wip-10883-firefly

osd: Fix FileJournal wrap to get header out first

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3521 from dzafman/wip-10676

Backport doc fixes that appear to apply to firefly

Reviewed-by: Loic Dachary <ldachary@redhat.com>

osd: fix PG leak in SnapTrimWQ._clear()

Fixes: #10421
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 01e154d592d6cdbf3f859cf1b4357e803536a6b4)

Merge pull request #3730 from ceph/wip-firefly-flock

backport ceph-fuse file locking patches to Firefly

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #3406 from ceph/wip-10299-firefly

librbd: complete all pending aio ops prior to closing image

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librbd: complete all pending aio ops prior to closing image

It was possible for an image to be closed while aio operations
were still outstanding.  Now all aio operations are tracked and
completed before the image is closed.

Fixes: #10299
Backport: giant, firefly, dumpling
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #3404 from ceph/wip-10270-firefly

librbd: gracefully handle deleted/renamed pools

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3410 from ceph/wip-9854-firefly

osdc: Constrain max number of in-flight read requests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

osd: Fix FileJournal wrap to get header out first

Correct and restore assert that was removed

Cause by f46b1b473fce0322a672b16c7739e569a45054b6
Fixes: #10883
Backport: dumpling, firefly, giant

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 970bb4901f93575709421b5b25c3eff213de61b8)

qa/workunits/fs/misc: Add a workunit for file lock interruption

Signed-off-by: Yan, Zheng <zyan@redhat.com>
(cherry picked from commit ac92c455a9aa19e4288acdf0c9a746e03a640efb)

mds: fix neighbor lock check

Signed-off-by: Yan, Zheng <zyan@redhat.com>
(cherry picked from commit b0e6e85aa08ea74cd209aad04f3f0bf991761e12)

client: use finisher to abort MDS request

When a request is interrupted, libfuse first locks an internal mutex,
then calls the interrupt callback. libfuse need to lock the same mutex
when unregistering interrupt callback. We unregister interrupt callback
while client_lock is locked, so we can't acquiring the client_lock in
the interrupt callback.

Signed-off-by: Yan, Zheng <zyan@redhat.com>
(cherry picked from commit 09699454e729592d426aeff5b578697e850af12e)

Conflicts:
src/client/Client.cc
src/client/Client.h

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

client: use atomic variable to track reference of MetaRequeset

this allow us to increase reference count of MetaRequest while not holding
the client_lock

Signed-off-by: Yan, Zheng <zyan@redhat.com>
(cherry picked from commit e464a7765230c504b3e3b88bcb1106f67b7c3eb9)

client: allow interrupting blocked file lock operation

This commit introduce two new types of setfilelock request. Unlike
setfilelock (UNLOCK) request, these two new types of setfilelock request
do not drop locks that have alread been acquired, they only interrupt
blocked setfilelock request.

Signed-off-by: Yan, Zheng <zyan@redhat.com>
(cherry picked from commit 4134c149d3759dd6a3aaa1a353b77bbfe8e9491b)

client: register callback for fuse interrupt

libfuse allows program to reigster a callback for interrupt. When a file
system operation is interrupted, the fuse kernel driver sends interupt
request to libfuse. libfuse calls the interrupt callback when receiving
interrupt request.

Signed-off-by: Yan, Zheng <zyan@redhat.com>
(cherry picked from commit 289e8b4a7efa1ae6427115af9bbe541c9e1f0e90)