OpRequest: release the message throttle when unregistered

We don't want messages in the OpTracker history hanging on to
message throttle.

Fixes: #14248
Backport: hammer, firefly
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #7114 from dillaman/wip-14234-jewel

librbd: reduce verbosity of common error condition logging

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge branch 'wip-14237-jewel' of git://github.com/dillaman/ceph into jewel

qa/workunits/rbd: import_export should use clean temp space

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7113 from dillaman/wip-14168-jewel

journal: disconnect watch after watch error

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/workunits/rbd: import_export should use clean temp space

Fixes: #14237
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: reduce verbosity of common error condition logging

Fixes: #14234
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: disconnect watch after watch error

Fixes: #14168
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Revert "rgw: use smart pointer for C_Reinitwatch"

This commit introduced regressions, issues #14173, #14180.

The problem is that the context deletes itself after execution,
so using a shared_ptr to hold it is problematic, as we'll eventually
have a double free. Also, it doesn't solve the underlying leak issue.

This reverts commit e9daed221e7c6fd84aef78c3d5fc18470b916fcb.

Merge pull request #7101 from dillaman/wip-14221-jewel

qa/workunits/rbd: rbd-nbd test should use sudo for map/unmap ops

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/workunits/rbd: rbd-nbd test should use sudo for map/unmap ops

Fixes: #14221
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6933 from jcsp/wip-14048

mds: tear down connections from `tell` commands

Merge pull request #7043 from dillaman/wip-14170-jewel

librbd: do not ignore self-managed snapshot release result

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librbd: do not ignore self-managed snapshot release result

Fixes: #14170
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7042 from dillaman/wip-14164-jewel

librbd: properly handle replay of snap remove RPC message

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7040 from dillaman/wip-14092-jewel

librbd: ensure librados callbacks are flushed prior to destroying

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7035 from dillaman/wip-14122-jewel

librbd: clear error when older OSD doesn't support image flags

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librbd: properly handle replay of snap remove RPC message

Fixes: #14164
Backport: infernalis
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: ensure librados callbacks are flushed prior to destroying image

Fixes: #14092
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: clear error when older OSD doesn't support image flags

Fixes: #14122
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7002 from dillaman/wip-14092-jewel

tests: flush op work queue prior to destroying MockImageCtx

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

tests: flush op work queue prior to destroying MockImageCtx

Fixes: #14092
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6815 from badone/wip-13993

common: log: Assign LOG_DEBUG priority to syslog calls

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6979 from dillaman/wip-14123-jewel

librbd: correct issues discovered when cache is disabled

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #6974 from dillaman/wip-14108-jewel

librbd: correct issues discovered during teuthology testing

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librbd: skip journal event if write/discard was no-op

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: object map ENOENT optimizations should complete asynchronously

This ensures a consistent lock ordering between the optimized path and
a librados callback. Remove the cache writebach handler's custom finisher
and re-use the op_work_queue.

Fixes: #14123
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: update exclusive lock state when shutting down

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: extra request required if journal enabled w/ caching

Fixes: #14108
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: stop the copyup thread during shutdown

Fixes: #14107, #14108
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6972 from dillaman/wip-14106-jewel

rbd: merge_diff test should use new --object-size parameter instead of --order

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/workunits/rbd: use --object-size instead of --order

Fixes: #14106
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: process crushed, rbd creating with striping parameters

Signed-off-by: Na Xie <xie.na@h3c.com>

log: Log.cc: Assign LOG_DEBUG priority to syslog calls

Fixes: #13993
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge remote-tracking branch 'gh/jewel'

Merge pull request #6932 from aiicore/test_fix

tests: allow object corpus readable test to skip specific incompat instances

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6941 from jcsp/wip-14029

mds: fix setvxattr (broken in a536d114)

Merge pull request #6948 from ceph/revert-6939-bp-remove-bz2dev

Revert "makefiles: remove bz2-dev from dependencies"

Reviewed-by: Dan Mick <dmick@redhat.com>

Revert "makefiles: remove bz2-dev from dependencies"

Merge pull request #6692 from SUSE/wip-13860

ceph.spec.in: add BuildRequires: systemd

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #6939 from branch-predictor/bp-remove-bz2dev

makefiles: remove bz2-dev from dependencies

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6943 from smithfarm/wip-doc-dev-typo

doc: fix typo in developer guide

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: fix typo in developer guide

Signed-off-by: Nathan Cutler <ncutler@suse.com>

mds: fix setvxattr (broken in a536d114)

There was an accidental move of this line
when adding the MAY_SET_POOL check, which
was causing setxattr to proceed before
it had the right locks, and thereby apply
its checks on bad data (symptom was failing
to detect that the file had data written to it).

Fixes: #14029
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #6821 from H3C/wip-cephfs-f11322

ceph-fuse: add process to ceph-fuse --help

Reviewed-by: John Spray <john.spray@redhat.com>

makefiles: remove bz2-dev from dependencies

The only thing that uses bzip2-devel is RocksDB, and it's optional, not
requirement. Drop the bzip2-devel/libbz2-dev dependency entirely, and
let RocksDB use it only if it is already present.

Fixes: #13981
Signed-off-by: Piotr Dałek <piotr.dalek@ts.fujitsu.com>

ceph-fuse: add process to ceph-fuse --help

Signed-off-by: Wei Feng <feng.wei@h3c.com>

Merge pull request #6934 from smithfarm/wip-14070

doc: Update list of admin/build-doc dependencies

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: Update list of admin/build-doc dependencies

http://tracker.ceph.com/issues/14070 Fixes: #14070

Signed-off-by: Nathan Cutler <ncutler@suse.com>

mds: tear down connections from `tell` commands

We can identify sessions that were never really
opened, just created when a client sent in
a 'tell' message.  When we see one of those, mark
the associated connection disposable when responding
to the command.

Fixes: #14048
Signed-off-by: John Spray <john.spray@redhat.com>

test/encoding/readable.sh: add non-whole type skip

With this modification there will be possibility to skip only one/few
object/s of particular type, not whole type as it was before.

Before:
- To skip whole TYPE create file named TYPE in forward_incompat
directory.

Now:
- To skip whole TYPE create file or empty directory named TYPE in
forward_incompat directory.
- To skip one/few object/s of TYPE create directory named TYPE in
forward_incompat and put into symbolic links to objects that
you wantt to skip.

Signed-off-by: Igor Podoski <igor.podoski@ts.fujitsu.com>

Merge pull request #6910 from dachary/wip-dev-guide-ceph-qa-suite

doc: dev: introduction to tests

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #6927 from tchaikov/wip-fix-ftbfs

rgw: fix the build failure

Reviewed-by: Loic Dachary <ldachary@redhat.com>

rgw: remove unused variable

to fix the compiler warning of

 rgw/rgw_formats.cc: In member function 'virtual void
 RGWFormatter_Plain::dump_format_va(const char*, const char*, bool, const
 char*, __va_list_tag*)':
 rgw/rgw_formats.cc:143:15: warning: unused variable 'format'
 [-Wunused-variable]
    const char *format;

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: fix the build failure

* s/bucket_name_str/bucket_name/
* the member variable name of `req_state` was changed to `bucket_name`
  in f7ca00a. but some commits was still using the old name before the
  commit got merged.
* fixes for the tenant related change
* also fixes a typo

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6859 from dillaman/wip-librbd-async-refresh

librbd: image refresh code paths converted to async state machines

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librbd: refresh object map after rollback

An image refresh will no longer trigger an automatic object map
reload, therefore explicitly reload the object map after rollback.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: refresh image asynchronously from watch/notify path

The AIO path already ensures asynchronous image refreshes, but the
watch/notify path for maintenance ops was still using the sync refresh
path.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: the journal state machine is now asynchronous

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: adjust test cases to refactored librbd

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: all object map methods are now async

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: IO ops will now asynchronously refresh the image if needed

This avoids blocking the AIO path with the previous synchronous
refresh path.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

WorkQueue: PointerWQ now supports out-of-band processing

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: flush all in-flight IO on block write and shut down

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: add shut down support to the AIO work queue

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: integrate new async image state machines

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: ImageWatcher should release lock upon request

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: implement image state tracker for open/close/refresh/etc

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: rename ImageRefresh to ImageState

This class handles more than just refreshing the image from
disk.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: object map lifespan now controlled by state machines

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: decouple ImageWatcher from exclusive lock management

Use new ExclusiveLock state machine to handle all the proper
transitions between lock states.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: separate image refresh into async state machines

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: exclusive lock state machines test cases

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: separate exclusive lock handling into async state machines

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librados_test_stub: additional definitions and mock functions

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: mock test cases for new async object map lock/unlock/refresh ops

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: async object map lock/unlock/refresh state machines

Creating async versions to support an async image refresh

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librados_test_stub: additional mocked IoCtx methods

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: convert object_map::InvalidateRequest to template

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: support updated async cls_rbd API

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls_rbd: async version of class helpers added

New async versions of get_id, get_immutable_metadata,
get_mutable_metadata, get_flags, get_stripe_unit_count,
snapshot_list, and old_snapshot_list.  These are needed
by the new librbd async ImageCtx op state machines.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: fix missing header dependencies for parent_types.h

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librados_test_stub: implemented ObjectReadOperation::stat

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: restrict rebuild object map test case

It should only run when the object map is enabled for the image.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: moved object name helpers to librbd::util namespace

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: integrate cleaned up librbd AIO callback adapter

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: cleanup librbd AIO completion callbacks

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: cleanup Context and librados AIO completion callbacks

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6643 from rzarzynski/wip-rgw-slo-rzarzynski

- rgw: swift bulk delete (Radoslaw Zarzynski)
- rgw: static large objects (Radoslaw Zarzynski, Yehuda Sadeh)

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

10.0.1

Merge pull request #6925 from dillaman/wip-13633

librbd: optionally validate new RBD pools for snapshot support

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

tests: new integration test for validating new RBD pools

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: optionally validate RBD pool configuration (snapshot support)

Fixes: #13633
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6917 from majianpeng/journal-iohint

Journal iohint for librbd

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #6923 from dillaman/wip-14060

librbd: correct race conditions discovered during unit testing

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

journal: avoid race between flush request and in-flight flushes

Previously, if the journal object was empty, a flush request could
complete instantly even if there is an in-flight 'record safe'
callback.

Fixes: #14060
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: fix test case race condition for maintenance requests

Fixes: #14060
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6922 from liewegas/wip-cephfs-posix

doc/cephfs/posix: update

Merge pull request #6899 from branch-predictor/bp-fix-no-rbd-build

rbd: fix build with "--without-rbd"

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rbd: fix build with "--without-rbd"

It's not possible to build Ceph configured with "--without-rbd", as
tests and FileStore depends on components provided by RBD. This changeset
fixes this.

Fixes: #14058
Signed-off-by: Piotr Dałek <piotr.dalek@ts.fujitsu.com>

doc/cephfs/posix: update

Signed-off-by: Sage Weil <sage@redhat.com>