Merge pull request #6057 from rahul1aggarwal/master

Bug #13207:  Rados Gateway: Anonymous user is able to read bucket with authenticated read ACL

doc/releases: add v0.80.11 to release table

Signed-off-by: Sage Weil <sage@redhat.com>

doc/release-notes: final v0.80.11 notes

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #6374 from dachary/wip-release-notes-firefly

release-notes: draft v0.80.11 release notes

13207: Rados Gateway: Anonymous user is able to read bucket with authenticated read ACL

Signed-off-by: root <rahul.1aggarwal@gmail.com>

Merge pull request #6654 from tchaikov/wip-unsigned-missing-scrub

osd: note down the number of missing clones

Reviewed-by: David Zafman <dzafman@redhat.com>

osd: note down the number of missing clones

Caused by 2e0bb0ae9498fd9a0c49f1b81bf0c86f7a7cac42
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6637 from Vicente-Cheng/update-ceph-disk-manual-page

doc: Update ceph-disk manual page with new feature deactivate/destroy.

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: Update ceph-disk manual page to remove some option description.

  - remove `--mark-init` and `--activate-key` from options because
    they are not options that are common to all subcommands.

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

doc: Update ceph-disk manual page with new feature deactivate/destroy.

  - Update the ceph-disk manual page

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

Merge pull request #6545 from dzafman/wip-dry-fixes

Fix --dry-run for many ceph-objectstore-tool operations

Includes adding missing tests for xattr and omap
Add testing of --dry-run for most operations
Minor cleanup of rados command output

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6640 from david-z/wip-async-msg-crash-cephfs

Cephfs will crash if enabling async msg because of an assertion

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #6482 from herver/wip-13709

rgw: fix reload on non Debian systems.

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #5662 from ceph/wip-scrub-basic

ScrubStack and "tag path" command

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Yan, Zheng <zyan@redhat.com>

Cephfs will crash if enabling async msg because of an assertion

Signed-off-by: Zhi Zhang <zhangz.david@outlook.com>

Merge pull request #6541 from dillaman/wip-11287

librbd: integrate journaling support for IO operations

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librbd: copy operation needs to use AIO work queue for writes

This ensures that the exclusive lock is properly acquired prior to
writing to the new image.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: simplify IO flush handling

Add a new convenience method to ImageCtx for handling flush
requests and cleanup flush handling with dealing with the cache.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: possible deadlock attempting to drain parent image WQs

Ensure all AIO to the parent image is properly flushed and assert
that all work queues are empty before closing the parent image.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

WorkQueue: PointerWQ drain no longer waits for other queues

If another (independent) queue was processing, drain could
block waiting.  Instead, allow drain to exit quickly if
no items are being processed and the queue is empty for
the current WQ.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #5867 from Vicente-Cheng/ceph-disk-destroy-implementation

ceph-disk: Add destroy and deactivate option

ceph-disk now understands deactivate to shutdown an OSD and umount the partition. It will remain in a "down" state until it is activated again with --reactivate. If a deactivated OSD is destroyed with ceph-disk destroy, it will be permanently removed from the cluster and from the crushmap.

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ceph-disk: remove the redundant try except and minor nits

  - remove the redundant try except

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

tests: ceph-disk: add wait_for_osd_down() in ceph-disk-test.py of qa

  - add wait_for_osd_down() to avoid the side effect of deactivate

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

tests: ceph-disk: modify the ceph-disk qa test cases

  - minor correct for the latest ceph-disk qa test cases

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

tests: ceph-disk: improve the unit test to coverage all deactivate/destroy function.

  - rework the unit test to coverage all deactivate/destroy function
  - make test item simple and easier to read

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

ceph-disk: improve the device query stage on deactivate/destroy feature.

  - Make code path much easier to get device info. (get little bit overhead)
  - Let some error rasie the correct execption
  - for dmcrypt device, we unmap on the deactivate stage. (consist with activate)

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

tests: ceph-disk: add deactivate/reactivate/destroy test cases.

  - using the deactivate/destroy feature to destroy osd
  - test reactivate option when the osd goes deactive
  - add check_osd_status to check osd status when osd goes up

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

tests: ceph-disk: add some unittest functions to coverage destroy/deactivate feature.

  - Add new unittest functions to coverage the part of dmcrypt/mpath handling.

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

ceph-disk: modify the destroy/deactivate behavior to handle dmcrypt/mpath feature.

  - modify the behavior to handle dmcrypt/mpath osd
  - add some functions to get the information of dmcrypt osd
  - fixed the logging format Use `%(filename)s` instead of `%(filename)`

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

tests: ceph-disk: Make unit test coverage all ceph-disk destroy/deactivate feature

  - Add some unit test to coverage all destroy/deactivate feature.
  - Do some minor modifications on the ceph-diskw

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

tests: ceph-disk: modified the destroy_osd test function.

  - use the new implementation (ceph-disk deactivate/destroy) instead of step by step remove

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

ceph-disk: Implement unittest for ceph-disk deactivate/destroy feature

  - The unit tests cover all the implemention about deactivate/destroy.

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

ceph-disk: add --reactivate option, modify parameter about deactivate and destroy

  - add `--reactivate` option (activate do no-op without reactivate with deactive flag)
  - for consistency, make both deactivate and destroy take the device/partition name
  - add `--deactivate-by-id` to deactivate and destroy for ease of use

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

ceph-disk: add `--mark-out` option on deactivate feature.

  - Using `--mark-out` option to mark osd out when
    deactivate this osd instead of always mark osd out.

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

ceph-disk: use `ceph osd dump` to check osd status

  - Before deactivated osd, we need to know the status
    including IN/OUT, UP/DOWN on this osd.

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

ceph-disk: add destroy feature

  Implement destroy option on ceph-disk.

  - remove OSD from CRUSH map
  - remove OSD cephx key
  - deallocate OSD ID
  - destroy data (with --zap option)

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

ceph-disk: add deactivate feature

  Implement deactivate option on ceph-disk.

  - stop ceph-osd service if needed (If osd still in osd map, make it out first)
  - remove 'ready', 'active', and INIT-specific files
  - remove gpt partition type and change partition name (prevent triggered by udev)
  - create deactive flag
  - umount device and remove mount point

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

Merge pull request #6607 from dillaman/wip-13784

rbd: support negative boolean command-line optionals

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #6606 from dillaman/wip-13806

rbd: add missing command aliases to refactored CLI

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rbd: support negative boolean command-line optionals

Fixes: #13784
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: new rbd CLI command aliases

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: add missing command aliases to refactored CLI

Fixes: #13806
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6590 from ceph/wip-rbd-user-option

rbd: accept --user, refuse -i command-line optionals

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6593 from trociny/fixup-rbd-refactor

rbd: stripe unit/count set incorrectly from config

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6594 from chenji-kael/patch-2

Update .mailmap

Update .mailmap

rbd: stripe unit/count set incorrectly from config

(after rbd-refactor fixup)

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #6549 from kylinstorage/global_flags

fix: use right init_flags to finish CephContext

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #6575 from ceph/update-infernalis-releasenotes

doc: update infernalis release notes

Merge pull request #6484 from XinzeChi/wip-journal-optimization

os: write file journal optimization

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #6371 from liewegas/wip-osd-less-crush

osd: avoid calculating crush mapping for most ops

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #6416 from rohanmars/wip-solaris-port

librados: Solaris port

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6439 from XinzeChi/wip-merge-tnx

osd: merge local_t and op_t txn to single one

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6547 from xiaoxichen/fix1

osd: check do_shutdown before do_restart

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6566 from kylinstorage/wip-fix-wrong-use-parenthesis

osd: fix wrong use of right parenthesis in localized read logic

Reviewed-by: Sage Weil <sage@redhat.com>

rbd: accept --user, refuse -i

--user is the equivalent of --id (mentioned in docs and ceph -h) and
used by at least a couple of qa scripts and teuthology.  Add it to the
list of valid rbd options - it's handled inside global_init() before
Shell:execute() is called.

OTOH allowing -i is bogus - global_init() wouldn't process it for
clients (CEPH_ENTITY_TYPE_CLIENT), which rbd tool is one of.  Drop it.

Signed-off-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #6584 from dachary/wip-test

tests: ignore test-suite.log

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6583 from ceph/wip-krbd-map-args

rbd: unbreak rbd map + cephx_sign_messages option

Reviewed-by: Sage Weil <sage@redhat.com>

tests: ignore test-suite.log

Now that the tests at the top level directory are run via TESTS and not
check-local:, the test-suite.log file is created and must bit
gitignored.

Signed-off-by: Loic Dachary <loic@dachary.org>

rbd: recognize cephx_sign_messages option

Signed-off-by: Ilya Dryomov <idryomov@redhat.com>

rbd: unbreak rbd map CLI

Previously, if snapname wasn't specified we would pass NULL to
krbd_map(), which was a cue for it to use "-" as a snapshot name.  With
the new rbd CLI, "" is passed in; same goes for map options.

Change krbd_map() accordingly and update its other user.

Signed-off-by: Ilya Dryomov <idryomov@redhat.com>

tests: fix test case using new api

Signed-off-by: Xinze Chi <xinze@xsky.com>

Merge pull request #6574 from yuyuyu101/fix-broken-kinects

kv/KineticStore: fix broken split_key

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6571 from dachary/wip-test-run-cli

tests: restore run-cli-tests

Merge pull request #6578 from dachary/wip-13785-debian-rbd-replay

build/ops: rbd-replay moved from ceph-test-dbg to ceph-common-dbg

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #4737 from kylinstorage/wip-temp-based-object-eviction

osd: improve temperature calculation for cache tier agent

Reviewed-by: Sage Weil

Merge pull request #6422 from xiexingguo/xxg-wip13639

librados: fix potential null pointer access when do pool_snap_list

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #6486 from XinzeChi/wip-multiple-finisher

osd: FileStore: support multiple ondisk finish and apply finishers

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #6518 from kylinstorage/wip-trivial-optimization

osd: optimize scrub subset_last_update calculation

Reviewed-by: Sage Weil <sage@redhat.com>

build/ops: rbd-replay moved from ceph-test-dbg to ceph-common-dbg

http://tracker.ceph.com/issues/13785 Fixes: #13785

Signed-off-by: Loic Dachary <loic@dachary.org>

tests: avoid bashism

The shell used by the cli tests is not always bash. Not using the
here-word is also more readable in this specific case.

Signed-off-by: Loic Dachary <loic@dachary.org>

rbd: hardcode application name into help

Avoid dynamically detecting the application name and instead hardcode
the rbd CLI name into the help output.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

ceph-objectstore-tool: Add dry-run checking to ops missing it

Update test to check most operations

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Remove unused function (cleanup)

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Remove redundant test output

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Verify replicated PG beyond just data after vstart

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Fix verify() used after import to also check xattr and omap

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Add test cases for xattr and omap ceph-objectstore-tool operations

Signed-off-by: David Zafman <dzafman@redhat.com>

kv/KineticStore: Fix broken split_key

Introduced by PR #6312

Signed-off-by: Haomai Wang <haomai@xsky.com>

Update infernalis release notes

Update infernalis release notes

Signed-off-by: Vasu Kulkarni vasu@redhat.com

tests: restore run-cli-tests

e4ca468 moved src/test/run-cli-tests from check-local to check_SCRIPTS
but did not add it to the TESTS variable.

Signed-off-by: Loic Dachary <loic@dachary.org>

SIGPIPE suppression for platforms without SO_NOSIGPIPE or MSG_NOSIGNAL

Signed-off-by: Rohan Mars <code@rohanmars.com>

radosgw-admin: fix cli tests

Signed-off-by: Sage Weil <sage@redhat.com>

osdmaptool: fix cli tests

Signed-off-by: Sage Weil <sage@redhat.com>

crushtool: fix cli tests

Signed-off-by: Sage Weil <sage@redhat.com>

crushtool: fix cli test help

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #6532 from dachary/wip-mailmap

mailmap: Ubuntu Kylin name changed to Kylin Cloud

Reviewed-by: Li Wang <li.wang@kylin-cloud.com>

Merge pull request #5848 from storage-zuiwanyuan/wip-nonblock-connect

msg/async: support of non-block connect in async messenger

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #6478 from yuyuyu101/wip-13666

msg/async: let receiver ack message ASAP

Reviewed-by: Sage Weil <sage@redhat.com>

osd: fix wrong use of right parenthesis

Signed-off-by: Jie Wang <jie.wang@kylin-cloud.com>
Reviewed-by: Li Wang <li.wang@kylin-cloud.com>

msg/async: support of non-block connect in async messenger

Fixes: #12802
Signed-off-by: Jianhui Yuan <zuiwanyuan@gmail.com>

Merge pull request #6534 from kylinstorage/wip-trivial-scrub-cleanup

osd: clarify the scrub result report

Reviewed-by: Kefu Chai <kchai@redhat.com>

scrub: clarify the result report

It may happen that the authoritative object
such that auth.size != be_get_ondisk_size(auth_oi.size),
in that case, clarify the error report.

Signed-off-by: Li Wang <li.wang@kylin-cloud.com>

journal: avoid holding lock while marking ops are complete

There is a possibility of a race if the JournalTrimmer destructor is
waiting for an async op to complete but the op is flagged as complete
while the mutex is being held.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cmake: updates for refactored librbd IO path

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: flush journal entries prior to releasing lock

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: only erase IO events after they are marked safe

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: librbd updates for C++11

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: verify librbd journal entries are written to disk

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: treat flush op as a write op

This will ensure that the journal is properly opened to handle
appending events.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>