scripts: ceph-release-notes for development versions

Relax the requirements for titles and issues when --strict is not set.
Collect and display the merge messages when they are not the same as the
PR title.

Signed-off-by: Loic Dachary <loic@dachary.org>

release-notes: draft v10.0.0 release notes

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #6603 from ukernel/wip-13782

client: avoid creating orphan object in Client::check_pool_perm()

Reviewed-by: John Spray <john.spray@redhat.com>

Merge branch 'osd-op-create' of git://github.com/majianpeng/ceph

Conflicts:
src/osd/ReplicatedPG.cc

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6489 from xiexingguo/xxg-wip-13715

librados: fix pool alignment API overflow issue

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #6492 from cxwshawn/wip-perf-fix

osd: change mutex to spinlock to optimize thread context switch.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6550 from cernceph/wip-deepscrub

osd: randomize deep scrubbing

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6568 from ghost/wip-scrub-error

scrub: do not assign value if read error

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #6609 from liewegas/wip-newstore

newstore: misc updates (including kv and os/fs stuff)

Merge pull request #6622 from liewegas/wip-13796

osd: partial revert of "ReplicatedPG: result code not correctly set in some cases."

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #6658 from dachary/wip-releases

doc: add v0.80.11 to the release timeline

Reviewed-by: Nathan Cutler <ncutler@suse.cz>

Merge pull request #6057 from rahul1aggarwal/master

Bug #13207:  Rados Gateway: Anonymous user is able to read bucket with authenticated read ACL

doc: add v0.80.11 to the release timeline

Signed-off-by: Loic Dachary <loic@dachary.org>

doc/releases: add v0.80.11 to release table

Signed-off-by: Sage Weil <sage@redhat.com>

doc/release-notes: final v0.80.11 notes

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #6374 from dachary/wip-release-notes-firefly

release-notes: draft v0.80.11 release notes

13207: Rados Gateway: Anonymous user is able to read bucket with authenticated read ACL

Signed-off-by: root <rahul.1aggarwal@gmail.com>

Merge pull request #6654 from tchaikov/wip-unsigned-missing-scrub

osd: note down the number of missing clones

Reviewed-by: David Zafman <dzafman@redhat.com>

osd: note down the number of missing clones

Caused by 2e0bb0ae9498fd9a0c49f1b81bf0c86f7a7cac42
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6637 from Vicente-Cheng/update-ceph-disk-manual-page

doc: Update ceph-disk manual page with new feature deactivate/destroy.

Reviewed-by: Loic Dachary <ldachary@redhat.com>

RadosClient: reimplement the pool alignment methods using the new ones

so we don't less duplicated code and the logic reveals the problem of the old methods in a more visible way.
Fixes: #13715
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

doc: Update ceph-disk manual page to remove some option description.

  - remove `--mark-init` and `--activate-key` from options because
    they are not options that are common to all subcommands.

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

doc: Update ceph-disk manual page with new feature deactivate/destroy.

  - Update the ceph-disk manual page

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

Merge pull request #6545 from dzafman/wip-dry-fixes

Fix --dry-run for many ceph-objectstore-tool operations

Includes adding missing tests for xattr and omap
Add testing of --dry-run for most operations
Minor cleanup of rados command output

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6640 from david-z/wip-async-msg-crash-cephfs

Cephfs will crash if enabling async msg because of an assertion

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #6482 from herver/wip-13709

rgw: fix reload on non Debian systems.

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #5662 from ceph/wip-scrub-basic

ScrubStack and "tag path" command

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Yan, Zheng <zyan@redhat.com>

Cephfs will crash if enabling async msg because of an assertion

Signed-off-by: Zhi Zhang <zhangz.david@outlook.com>

osd: partial revert of "ReplicatedPG: result code not correctly set in some cases."

This reverts the parts of commit 3b5d54bec0f0556b7dd6e7ea7e6df02ec71f97d9
that change omap return values (i.e., do not just assert, but
change what the client sees).

Fixes: #13796
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #6541 from dillaman/wip-11287

librbd: integrate journaling support for IO operations

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librbd: copy operation needs to use AIO work queue for writes

This ensures that the exclusive lock is properly acquired prior to
writing to the new image.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: simplify IO flush handling

Add a new convenience method to ImageCtx for handling flush
requests and cleanup flush handling with dealing with the cache.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: possible deadlock attempting to drain parent image WQs

Ensure all AIO to the parent image is properly flushed and assert
that all work queues are empty before closing the parent image.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

WorkQueue: PointerWQ drain no longer waits for other queues

If another (independent) queue was processing, drain could
block waiting.  Instead, allow drain to exit quickly if
no items are being processed and the queue is empty for
the current WQ.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #5867 from Vicente-Cheng/ceph-disk-destroy-implementation

ceph-disk: Add destroy and deactivate option

ceph-disk now understands deactivate to shutdown an OSD and umount the partition. It will remain in a "down" state until it is activated again with --reactivate. If a deactivated OSD is destroyed with ceph-disk destroy, it will be permanently removed from the cluster and from the crushmap.

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ceph-disk: remove the redundant try except and minor nits

  - remove the redundant try except

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

tests: ceph-disk: add wait_for_osd_down() in ceph-disk-test.py of qa

  - add wait_for_osd_down() to avoid the side effect of deactivate

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

tests: ceph-disk: modify the ceph-disk qa test cases

  - minor correct for the latest ceph-disk qa test cases

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

tests: ceph-disk: improve the unit test to coverage all deactivate/destroy function.

  - rework the unit test to coverage all deactivate/destroy function
  - make test item simple and easier to read

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

ceph-disk: improve the device query stage on deactivate/destroy feature.

  - Make code path much easier to get device info. (get little bit overhead)
  - Let some error rasie the correct execption
  - for dmcrypt device, we unmap on the deactivate stage. (consist with activate)

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

tests: ceph-disk: add deactivate/reactivate/destroy test cases.

  - using the deactivate/destroy feature to destroy osd
  - test reactivate option when the osd goes deactive
  - add check_osd_status to check osd status when osd goes up

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

tests: ceph-disk: add some unittest functions to coverage destroy/deactivate feature.

  - Add new unittest functions to coverage the part of dmcrypt/mpath handling.

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

ceph-disk: modify the destroy/deactivate behavior to handle dmcrypt/mpath feature.

  - modify the behavior to handle dmcrypt/mpath osd
  - add some functions to get the information of dmcrypt osd
  - fixed the logging format Use `%(filename)s` instead of `%(filename)`

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

tests: ceph-disk: Make unit test coverage all ceph-disk destroy/deactivate feature

  - Add some unit test to coverage all destroy/deactivate feature.
  - Do some minor modifications on the ceph-diskw

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

tests: ceph-disk: modified the destroy_osd test function.

  - use the new implementation (ceph-disk deactivate/destroy) instead of step by step remove

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

ceph-disk: Implement unittest for ceph-disk deactivate/destroy feature

  - The unit tests cover all the implemention about deactivate/destroy.

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

ceph-disk: add --reactivate option, modify parameter about deactivate and destroy

  - add `--reactivate` option (activate do no-op without reactivate with deactive flag)
  - for consistency, make both deactivate and destroy take the device/partition name
  - add `--deactivate-by-id` to deactivate and destroy for ease of use

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

ceph-disk: add `--mark-out` option on deactivate feature.

  - Using `--mark-out` option to mark osd out when
    deactivate this osd instead of always mark osd out.

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

ceph-disk: use `ceph osd dump` to check osd status

  - Before deactivated osd, we need to know the status
    including IN/OUT, UP/DOWN on this osd.

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

ceph-disk: add destroy feature

  Implement destroy option on ceph-disk.

  - remove OSD from CRUSH map
  - remove OSD cephx key
  - deallocate OSD ID
  - destroy data (with --zap option)

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

ceph-disk: add deactivate feature

  Implement deactivate option on ceph-disk.

  - stop ceph-osd service if needed (If osd still in osd map, make it out first)
  - remove 'ready', 'active', and INIT-specific files
  - remove gpt partition type and change partition name (prevent triggered by udev)
  - create deactive flag
  - umount device and remove mount point

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

Merge pull request #6607 from dillaman/wip-13784

rbd: support negative boolean command-line optionals

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #6606 from dillaman/wip-13806

rbd: add missing command aliases to refactored CLI

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rbd: support negative boolean command-line optionals

Fixes: #13784
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

os/newstore: disable rocksdb compression

This has been shown to be problematic for performance on the
monitor.  Note that this takes us from ~170/bytes per onode to
~540/bytes per onode.  (The encoded onode_t is 390 bytes, not
including the key name.)

Signed-off-by: Sage Weil <sage@redhat.com>

common: mirror leveldb default tuning w/ rocksdb

Signed-off-by: Sage Weil <sage@redhat.com>

mon: disabled rocksdb compression when used as the backend

This significantly reduced CPU utilization on the bigbang scale
testing cluster at CERN.  Note that it is already disabled for
leveldb by default (in ceph_mon.cc).

Signed-off-by: Sage Weil <sage@redhat.com>

os/fs/FS: fix zero()'s PUNCH_HOLE incancation

We get EOPNOTSUPP unconditionally without KEEP_SIZE.

Signed-off-by: Sage Weil <sage@redhat.com>

os/fs/FS: fix zero() return value on fallback

Signed-off-by: Sage Weil <sage@redhat.com>

os/RocksDBStore: set up $path.wal -> $path symlink

If $path.wal doesn't exist, create it and symlink it to $path.
Set wal_dir to that.  This makes it easy to move the wal content
elsewhere later, or to pre-create the .wal dir.

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: distinguish between db open and create

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: remove newstore_db_path option

It is simpler to have fixed locations and symlinks.

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: newstore_backend_options -> newstore_rocksdb_options

This way we can have default settings per-backend.  Also note that
this is what we currently do with leveldb on the mon and osd.

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: set rocksdb default options

 max_write_buffer_number=16
 min_write_buffer_number_to_merge=6

This cuts the amount of short-lived WAL data that gets
rewritten by roughly a factor of 6.

Signed-off-by: Sage Weil <sage@redhat.com>

tests: new rbd CLI command aliases

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: add missing command aliases to refactored CLI

Fixes: #13806
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

osd: scrub if load below daily avg and decreasing

Store a daily loadavg and use as an upper limit on when to allow
scrubs. Also track the 15 minute loadavg and only scrub when the
loadavg is decreasing (i.e. 1m < 15m).

Backports: hammer, infernalis

Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>

osd: randomize deep scrubbing

osd_scrub_interval_randomize_ratio works to randomize the shallow scrubs
but doesn't prevent a thundering herd of deep scrubs every osd_deep_scrub_interval.
Add the option osd_deep_scrub_randomize_ratio which defines the rate at which scrubs
will randomly turn into deep scrubs.

Backports: hammer, infernalis

Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>
Signed-off-by: Herve Rousseau <herve.rousseau@cern.ch>

Merge pull request #6590 from ceph/wip-rbd-user-option

rbd: accept --user, refuse -i command-line optionals

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6593 from trociny/fixup-rbd-refactor

rbd: stripe unit/count set incorrectly from config

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

client: avoid creating orphan object in Client::check_pool_perm()

Fixes: #13782
Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #6594 from chenji-kael/patch-2

Update .mailmap

scrub: do not assign value if read error for ECBackend

Signed-off-by: Li Wang <li.wang@kylin-cloud.com>

scrub: do not assign value if read error for ReplicatedBackend

Signed-off-by: Li Wang <li.wang@kylin-cloud.com>

osdservice: state changed to atomic_t to decrease thread context switch.

    service.is_stopping is in the key IO path, hot call, better use spinlock.

Signed-off-by: Xiaowei Chen <chen.xiaowei@h3c.com>

osd: change mutex to spinlock to optimize thread context switch.

Signed-off-by: Xiaowei Chen <chen.xiaowei@h3c.com>

Update .mailmap

rbd: stripe unit/count set incorrectly from config

(after rbd-refactor fixup)

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #6549 from kylinstorage/global_flags

fix: use right init_flags to finish CephContext

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #6575 from ceph/update-infernalis-releasenotes

doc: update infernalis release notes

Merge pull request #6484 from XinzeChi/wip-journal-optimization

os: write file journal optimization

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #6371 from liewegas/wip-osd-less-crush

osd: avoid calculating crush mapping for most ops

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #6416 from rohanmars/wip-solaris-port

librados: Solaris port

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6439 from XinzeChi/wip-merge-tnx

osd: merge local_t and op_t txn to single one

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6547 from xiaoxichen/fix1

osd: check do_shutdown before do_restart

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6566 from kylinstorage/wip-fix-wrong-use-parenthesis

osd: fix wrong use of right parenthesis in localized read logic

Reviewed-by: Sage Weil <sage@redhat.com>

rbd: accept --user, refuse -i

--user is the equivalent of --id (mentioned in docs and ceph -h) and
used by at least a couple of qa scripts and teuthology.  Add it to the
list of valid rbd options - it's handled inside global_init() before
Shell:execute() is called.

OTOH allowing -i is bogus - global_init() wouldn't process it for
clients (CEPH_ENTITY_TYPE_CLIENT), which rbd tool is one of.  Drop it.

Signed-off-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #6584 from dachary/wip-test

tests: ignore test-suite.log

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6583 from ceph/wip-krbd-map-args

rbd: unbreak rbd map + cephx_sign_messages option

Reviewed-by: Sage Weil <sage@redhat.com>

tests: ignore test-suite.log

Now that the tests at the top level directory are run via TESTS and not
check-local:, the test-suite.log file is created and must bit
gitignored.

Signed-off-by: Loic Dachary <loic@dachary.org>

rbd: recognize cephx_sign_messages option

Signed-off-by: Ilya Dryomov <idryomov@redhat.com>

rbd: unbreak rbd map CLI

Previously, if snapname wasn't specified we would pass NULL to
krbd_map(), which was a cue for it to use "-" as a snapshot name.  With
the new rbd CLI, "" is passed in; same goes for map options.

Change krbd_map() accordingly and update its other user.

Signed-off-by: Ilya Dryomov <idryomov@redhat.com>

tests: fix test case using new api

Signed-off-by: Xinze Chi <xinze@xsky.com>

Merge pull request #6574 from yuyuyu101/fix-broken-kinects

kv/KineticStore: fix broken split_key

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6571 from dachary/wip-test-run-cli

tests: restore run-cli-tests

Merge pull request #6578 from dachary/wip-13785-debian-rbd-replay

build/ops: rbd-replay moved from ceph-test-dbg to ceph-common-dbg

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #4737 from kylinstorage/wip-temp-based-object-eviction

osd: improve temperature calculation for cache tier agent

Reviewed-by: Sage Weil

Merge pull request #6422 from xiexingguo/xxg-wip13639

librados: fix potential null pointer access when do pool_snap_list

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #6486 from XinzeChi/wip-multiple-finisher

osd: FileStore: support multiple ondisk finish and apply finishers

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #6518 from kylinstorage/wip-trivial-optimization

osd: optimize scrub subset_last_update calculation

Reviewed-by: Sage Weil <sage@redhat.com>