Merge pull request #16454 from liewegas/wip-fix-ceph-scrub

qa/tasks/ceph_manager: wait for osd to start after objectstore-tool sequence

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16028 from jcsp/wip-mgr-commands

mon: load mgr commands at runtime

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16415 from xiexingguo/wip-health-mon

mon: a few health fixes

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16409 from liupan1111/wip-restructure-bs

os/bluestore: move aio.h/cc from fs dir to bluestore dir

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16328 from neha-ojha/recovery_sleep_default

osd: Add recovery sleep configuration option for HDDs and SSDs

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16470 from tchaikov/wip-watch-audit-channel

qa/workunits/cephtool/test.sh: fix test to watch audit channel

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

qa/workunits/cephtool/test.sh: fix test to watch audit channel

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/reg11184: import run

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16429 from liewegas/wip-jewel-x

qa/suites/upgrade/jewel-x: misc fixes for new health checks

Merge pull request #16374 from majianpeng/bluestore-double-count-kv-queue-lat

os/bluestore/BlueStore: Avoid double counting state_kv_queued_lat.

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #16423 from liewegas/wip-ls

mon: '* list' -> '* ls'

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16434 from badone/wip-fix-deep-age-reporting

mon: Fix deep_age copy paste error

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16444 from tchaikov/wip-test-osd-stat

qa/workunits/cephtool/test.sh: "ceph osd stat" output changed, update accordingly

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

qa/tasks/ceph_manager: wait for osd to start after objectstore-tool sequence

Fixes: http://tracker.ceph.com/issues/20705
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #14287 from oritwas/wip-rgw-usage

rgw: usage

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #15673 from yehudasa/wip-20107

cls/refcount: store and use list of retired tags

Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #16421 from tchaikov/wip-19939

messages/: always set header.version in encode_payload()

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #16418 from theanalyst/doc-rgw-tenant-names

doc: rgw clarify limitations when creating tenant names

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Pritha Srivastava <prsrivas@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #16443 from wjwithagen/bug-wjw-qa-test-reorder

cephtool/test.sh: Only delete a test pool when no longer needed.

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #16451 from tchaikov/wip-qa-reg11184

qa/tasks/reg11184: use literal 'foo' instead pool_name

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/tasks/reg11184: use literal 'foo' instead pool_name

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16412 from wido/zabbix-release-notes

doc: Add Zabbix ceph-mgr plugin to PendingReleaseNotes

Reviewed-by: Kefu Chai <kchai@redhat.com>

release notes: Add Zabbix ceph-mgr module to PendingReleaseNotes

Signed-off-by: Wido den Hollander <wido@42on.com>

doc: rgw clarify limitations when creating tenant names

We only allow alphanumeric and underscore characters in tenant names
according to the validation in `RGWHandler_REST::validate_tenant_name`

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

qa/workunits/cephtool/test.sh: "ceph osd stat" output changed, update test accordingly

Signed-off-by: Kefu Chai <kchai@redhat.com>
Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

cephtool/test.sh: Only delete a test pool when no longer needed.

the pool_getset pool is deleted before all tests on it are complete

4: /home/jenkins/workspace/ceph-master/qa/workunits/cephtool/test.sh:1990: test_mon_osd_pool_set:  ceph osd pool delete pool_get
set pool_getset --yes-i-really-really-mean-it
4: pool 'pool_getset' removed
4: /home/jenkins/workspace/ceph-master/qa/workunits/cephtool/test.sh:1992: test_mon_osd_pool_set:  ceph osd pool get rbd crush_r
ule
4: /home/jenkins/workspace/ceph-master/qa/workunits/cephtool/test.sh:1992: test_mon_osd_pool_set:  grep 'crush_rule: '
4: crush_rule: replicated_rule
4: /home/jenkins/workspace/ceph-master/qa/workunits/cephtool/test.sh:1994: test_mon_osd_pool_set:  ceph -f json osd pool get poo
l_getset compression_mode
4: Error ENOENT: unrecognized pool 'pool_getset'

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #16403 from wjwithagen/bug-wjw-ceph-osd-stat

test: ceph osd stat out has changed, fix tests for that

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16428 from idryomov/wip-krbd-luminous-thrash

qa: thrash tests for backoff and upmap

Reviewed-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #16424 from tchaikov/wip-assert-in-c

include/assert: test c++ before using static_cast<>

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16400 from liewegas/wip-crush-rule-id

crush: s/ruleset/id/ in decompiled output; prevent compilation when ruleset != id

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #16402 from liewegas/wip-20105

ceph_test_rados_api_watch_notify: flush after unwatch

Reviewed-by: Kefu Chai <kchai@redhat.com>

mon: hide 'osd crush rule list' (we prefer 'ls')

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16433 from dmick/wip-ccache

do_cmake.sh: fix syntax for /bin/sh (doesn't have +=)

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mon/HealthMonitor: fix summary of monitor clock skew

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon: Fix deep_age copy paste error

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

do_cmake.sh: fix syntax for /bin/sh (doesn't have +=)

Signed-off-by: Dan Mick <dan.mick@redhat.com>

librados: 'pg ls' is not a mgr command

Note that this breaks the command *during* the mon upgrade from jewel ->
luminous, which is slightly annoying, but means we avoid proxying via the
mon after upgrade is complete, which is good and IMO more important.

In the future we may want librados to cache the command descriptions so
that commands can be directed automatically.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: restore mgr command lookup so that we can still proxy to mgr

Since mgr commands aren't in the main mon_commands array now, we need to
explicitly look up commands there too.  This restores the behavior
implemented below in which we forward misdirected mgr commands to the mon.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: constify _get_moncommand

Signed-off-by: Sage Weil <sage@redhat.com>

doc: update osd recovery sleep description

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #16309 from fangyuxiangGL/fix-admin-rest-api

rgw: modify email to empty by admin RESTful api doesn't work

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15952 from jdurgin/wip-peering-deletes

osd: process deletes during recovery instead of peering

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16376 from jecluis/wip-mon-fix-loglast

mon/LogMonitor: don't read list's end() for log last

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16419 from Abhishekvrshny/fix-err-msg

rgw: fix error message in removing bucket with --bypass-gc flag

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

doc: add osd recovery sleep for HDDs and SSDs

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #16431 from jeffgreenca/master

doc: Fixed a typo in yum repo filename script

Merge pull request #15763 from dillaman/wip-pool-tags

mon: support pool application metadata key/values

Reviewed-by: Sage Weil <sage@redhat.com>

doc: Fixed a typo in yum repo filename script

Signed-off-by: Jeff Green <jeff.green.ca@gmail.com>

PendingReleaseNotes: included details for the pool tags

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

doc: fixed warning on rbd quick start link reference

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

doc: document pool tags in rados pool operations doc

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test/librados_test_stub: added new application metadata APIs

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mon: disable application health warnings for upgrade test cases

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test: skip pool application metadata tests if OSDs not at min luminous

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test: enable pool applications for new pools

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mon: disable application metadata on cache tier pools

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: add new 'pool init' action for initializing a RBD pool

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rgw: auto-tag created pools as owned by rgw

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librados: added async application enable API

RGW has numerous pool creation calls, one of which utilizes
an async interface. This adds support for RGW's use-case.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librados: new API to manipulate pool application metadata

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mon: auto-enable CephFS on pools when using "fs new"/"fs add_data_pool"

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mon: default 'rbd' pool to the 'rbd' application

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mon: heuristics for auto-enabling pool applications upon upgrade

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mon: health warning if in-use pools don't have application enabled

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mon: added new "osd pool application" commands

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mon: store application metadata in pg_pool_t

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mon: 'auth list' -> 'auth ls'

Signed-off-by: Sage Weil <sage@redhat.com>

mon: 'config-key list' -> 'config-key ls'

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16275 from linuxbox2/wip-rgw-readdir-cookie

rgw_file: permit dirent offset computation

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #16368 from theanalyst/fix/rgw-reshard-pool-ns

rgw: use a namespace for rgw reshard pool for upgrades as well

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #16395 from jecluis/wip-fix-mon-mgr-bootstrap

mon/AuthMonitor: generate bootstrap-mgr key on upgrade

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16425 from smithfarm/wip-ceph-w

doc: PendingReleaseNotes: "ceph -w" behavior has changed drastically

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16314 from tchaikov/wip-doc-replace-osd

doc: add instructions for replacing an OSD

Reviewed-by: Alfredo Deza <adeza@redhat.com>

qa/workunits/cephtool/test.sh: ceph osd stat out has changed, fix tests for that

The output of ceph osd stat has changed,
It printed:

cluster b370a29d-9287-4ca3-ab57-3d824f65e339
 health HEALTH_OK
 monmap e1: 1 mons at {ceph1=10.0.0.8:6789/0}, election epoch 2, quorum 0 ceph1
 osdmap e63: 2 osds: 2 up, 2 in
  pgmap v41338: 952 pgs, 20 pools, 17130 MB data, 2199 objects
        115 GB used, 167 GB / 297 GB avail
             952 active+clean

but now the osdmap line has gone and thus this no longer works:
qa/workunits/cephtool/test.sh:1944:
old_pgs=$(ceph osd pool get $TEST_POOL_GETSET pg_num | sed -e 's/pg_num: //')
new_pgs=$(($old_pgs+$(ceph osd stat | grep osdmap | awk '{print $3}')*32))

4: qa/workunits/cephtool/test.sh: line 1945: 10+*32: syntax errotoken is "*32")

 - And parse the output in json , with jq, for better reliability

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

include/assert: test c++ before using static_cast<>

this partially reverts 2e7c72d.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: PendingReleaseNotes: "ceph -w" behavior has changed drastically

Signed-off-by: Joao Eduardo Luis <joao@suse.de>
Signed-off-by: Nathan Cutler <ncutler@suse.com>

messages/: always set header.version in encode_payload()

we encode the payload w/o the writelock even can_write == NOWRITE, if
the message "can_fast_prepare". in that case, the "feature" of the
connection is 0, as no handshake happens yet. so the header.version is
always set to a version compatible with pre-luminous. but when the
message is re-encoded when the connection is re-established with feature
with luminous, the header.version is not set back to HEADER_VERSION.
that's why the message's encoding is not consistent with header.version
sometimes.

in this change, we always set the header.version in encode_payload(), so
it's consistent even after connection reset and message re-encoding.

Fixes: http://tracker.ceph.com/issues/19939
Signed-off-by: Kefu Chai <kchai@redhat.com>

mon: define static mgr_commands at mkfs time

This closes a window between mkfs and when the first mgr goes active
where *no* mgr commands are defined, and things like 'pg dump' fail.  We
do not get the default set of commands defined by modules, but we get
everything else.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr: move mgr_commands to separate compilation unit

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonCommands: std::

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MgrMonitor: mark mgr commands with FLAG_MGR

Signed-off-by: Sage Weil <sage@redhat.com>

qa/doc: update for "mgr tell" no longer needed

Signed-off-by: John Spray <john.spray@redhat.com>

mon: load mgr commands dynamically

So that the list of commands includes python modules,
thus allowing python-provided commands to be invoked
by the CLI with out a `tell mgr` prefix.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: transmit command descriptions to mgr in activating beacon

The mgr already sends a beacon to the mon immediately
after loading python modules in Mgr::init, to indicate
that it is now available.  Use that beacon to transmit
the command descriptions.

The monitor should handle this beacon by persisting
the command descriptions before persisting the updated
mgrmap that indicates that the mgr is now active.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: use MonCommand for command descriptions

...and update the MonCommand encoding so that we
can readily send vectors of them around.

Signed-off-by: John Spray <john.spray@redhat.com>

encoding: remove encode_array_nohead

This was just a for loop.  No longer needed for
MonCommands, and the usage in memstore/PageSet
was just iterating over char* and should never have
been there to begin with.

Signed-off-by: John Spray <john.spray@redhat.com>

rgw: fix error message in removing bucket with --bypass-gc flag

Fixes: http://tracker.ceph.com/issues/20688
Signed-off-by: Abhishek Varshney <abhishek.varshney@flipkart.com>

Merge pull request #16411 from smithfarm/wip-crn-regression

tools: ceph-release-notes: refactor and fix regressions

Reviewed-By: Kefu Chai <kchai@redhat.com>
Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

qa/suites/krbd: add luminous thrash tests

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/suites/krbd: reorganize thrash tests

- factor out install and ceph into ceph/ceph.yaml
- pg_num thrashing + 20 minute health timeout for thrashosds
- common thrashosds-health.yaml whitelist
- drop iozone workload

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/suites/krbd: heavier rbd_fio workload

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/tasks/rbd_fio: dump fio options before starting

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/tasks/rbd_fio: support libaio engine

Want to set iodepth and do direct AIO.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

tools: ceph-release-notes: match Reviewed-by more liberally

Signed-off-by: Nathan Cutler <ncutler@suse.com>

mon/HealthMonitor: fix mon_warn_on_osd_down_out_interval_zero does not work

And the output is wrong...

cluster:
    id:     b979e20d-6441-46b4-8663-954e1e8ce01d
    health: HEALTH_WARN
            1 osds down
            mon %names has mon_osd_down_out_interval set to 0

Now:
 health: HEALTH_WARN
            mon a is low on available space
            mon a has mon_osd_down_out_interval set to 0

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/HealthMonitor: fix wrong health level

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/HealthMonitor: fix regex formatting

Was:
./bin/ceph -s
  cluster:
    id:     0f704e51-f496-4812-a782-f6bcc490a109
    health: HEALTH_ERR
            mon%plurals% %names% %isorare% low on available space

Now:
 ./bin/ceph -s
  cluster:
    id:     1bb2cc6d-ba6f-467a-b811-994101a42749
    health: HEALTH_ERR
            mon a is low on available space

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

tools: ceph-release-notes: fix sorted() key lambda regression

https://github.com/ceph/ceph/pull/16261 ported the script to Python 3, but it
retained the 2-argument version of the sorted() key function - in Python 3 the
key function takes only one argument.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

tools: ceph-release-notes: refactor and fix regression

This commit refactors the logic for determining the PR title and merge message,
and fixes a regression introduced by https://github.com/ceph/ceph/pull/16277

Signed-off-by: Nathan Cutler <ncutler@suse.com>

test: add a couple lost+delete unit tests

Signed-off-by: Josh Durgin <jdurgin@redhat.com>