qa/tasks/reg11184: use literal 'foo' instead pool_name

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16403 from wjwithagen/bug-wjw-ceph-osd-stat

test: ceph osd stat out has changed, fix tests for that

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16428 from idryomov/wip-krbd-luminous-thrash

qa: thrash tests for backoff and upmap

Reviewed-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #16424 from tchaikov/wip-assert-in-c

include/assert: test c++ before using static_cast<>

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16400 from liewegas/wip-crush-rule-id

crush: s/ruleset/id/ in decompiled output; prevent compilation when ruleset != id

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #16402 from liewegas/wip-20105

ceph_test_rados_api_watch_notify: flush after unwatch

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16433 from dmick/wip-ccache

do_cmake.sh: fix syntax for /bin/sh (doesn't have +=)

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

do_cmake.sh: fix syntax for /bin/sh (doesn't have +=)

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #16309 from fangyuxiangGL/fix-admin-rest-api

rgw: modify email to empty by admin RESTful api doesn't work

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15952 from jdurgin/wip-peering-deletes

osd: process deletes during recovery instead of peering

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16376 from jecluis/wip-mon-fix-loglast

mon/LogMonitor: don't read list's end() for log last

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16419 from Abhishekvrshny/fix-err-msg

rgw: fix error message in removing bucket with --bypass-gc flag

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #16431 from jeffgreenca/master

doc: Fixed a typo in yum repo filename script

Merge pull request #15763 from dillaman/wip-pool-tags

mon: support pool application metadata key/values

Reviewed-by: Sage Weil <sage@redhat.com>

doc: Fixed a typo in yum repo filename script

Signed-off-by: Jeff Green <jeff.green.ca@gmail.com>

PendingReleaseNotes: included details for the pool tags

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

doc: fixed warning on rbd quick start link reference

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

doc: document pool tags in rados pool operations doc

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test/librados_test_stub: added new application metadata APIs

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mon: disable application health warnings for upgrade test cases

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test: skip pool application metadata tests if OSDs not at min luminous

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test: enable pool applications for new pools

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mon: disable application metadata on cache tier pools

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: add new 'pool init' action for initializing a RBD pool

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rgw: auto-tag created pools as owned by rgw

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librados: added async application enable API

RGW has numerous pool creation calls, one of which utilizes
an async interface. This adds support for RGW's use-case.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librados: new API to manipulate pool application metadata

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mon: auto-enable CephFS on pools when using "fs new"/"fs add_data_pool"

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mon: default 'rbd' pool to the 'rbd' application

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mon: heuristics for auto-enabling pool applications upon upgrade

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mon: health warning if in-use pools don't have application enabled

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mon: added new "osd pool application" commands

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mon: store application metadata in pg_pool_t

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #16275 from linuxbox2/wip-rgw-readdir-cookie

rgw_file: permit dirent offset computation

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #16368 from theanalyst/fix/rgw-reshard-pool-ns

rgw: use a namespace for rgw reshard pool for upgrades as well

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #16395 from jecluis/wip-fix-mon-mgr-bootstrap

mon/AuthMonitor: generate bootstrap-mgr key on upgrade

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16425 from smithfarm/wip-ceph-w

doc: PendingReleaseNotes: "ceph -w" behavior has changed drastically

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16314 from tchaikov/wip-doc-replace-osd

doc: add instructions for replacing an OSD

Reviewed-by: Alfredo Deza <adeza@redhat.com>

qa/workunits/cephtool/test.sh: ceph osd stat out has changed, fix tests for that

The output of ceph osd stat has changed,
It printed:

cluster b370a29d-9287-4ca3-ab57-3d824f65e339
 health HEALTH_OK
 monmap e1: 1 mons at {ceph1=10.0.0.8:6789/0}, election epoch 2, quorum 0 ceph1
 osdmap e63: 2 osds: 2 up, 2 in
  pgmap v41338: 952 pgs, 20 pools, 17130 MB data, 2199 objects
        115 GB used, 167 GB / 297 GB avail
             952 active+clean

but now the osdmap line has gone and thus this no longer works:
qa/workunits/cephtool/test.sh:1944:
old_pgs=$(ceph osd pool get $TEST_POOL_GETSET pg_num | sed -e 's/pg_num: //')
new_pgs=$(($old_pgs+$(ceph osd stat | grep osdmap | awk '{print $3}')*32))

4: qa/workunits/cephtool/test.sh: line 1945: 10+*32: syntax errotoken is "*32")

 - And parse the output in json , with jq, for better reliability

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

include/assert: test c++ before using static_cast<>

this partially reverts 2e7c72d.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: PendingReleaseNotes: "ceph -w" behavior has changed drastically

Signed-off-by: Joao Eduardo Luis <joao@suse.de>
Signed-off-by: Nathan Cutler <ncutler@suse.com>

rgw: fix error message in removing bucket with --bypass-gc flag

Fixes: http://tracker.ceph.com/issues/20688
Signed-off-by: Abhishek Varshney <abhishek.varshney@flipkart.com>

Merge pull request #16411 from smithfarm/wip-crn-regression

tools: ceph-release-notes: refactor and fix regressions

Reviewed-By: Kefu Chai <kchai@redhat.com>
Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

qa/suites/krbd: add luminous thrash tests

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/suites/krbd: reorganize thrash tests

- factor out install and ceph into ceph/ceph.yaml
- pg_num thrashing + 20 minute health timeout for thrashosds
- common thrashosds-health.yaml whitelist
- drop iozone workload

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/suites/krbd: heavier rbd_fio workload

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/tasks/rbd_fio: dump fio options before starting

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/tasks/rbd_fio: support libaio engine

Want to set iodepth and do direct AIO.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

tools: ceph-release-notes: match Reviewed-by more liberally

Signed-off-by: Nathan Cutler <ncutler@suse.com>

tools: ceph-release-notes: fix sorted() key lambda regression

https://github.com/ceph/ceph/pull/16261 ported the script to Python 3, but it
retained the 2-argument version of the sorted() key function - in Python 3 the
key function takes only one argument.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

tools: ceph-release-notes: refactor and fix regression

This commit refactors the logic for determining the PR title and merge message,
and fixes a regression introduced by https://github.com/ceph/ceph/pull/16277

Signed-off-by: Nathan Cutler <ncutler@suse.com>

test: add a couple lost+delete unit tests

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

osd/PrimaryLogPG: guard lost_delete missing_loc change by feature flag

With deletes during recovery instead of during log processing, we need
to keep the entry in missing_loc.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

TestPGLog: add unit tests for rebuilding missing set

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

PGLog, PrimaryLogPG: rebuild the missing set when the OSDMap flag is set

The recovery_deletes flag will only be set once, by the 'ceph osd
require-osd-release luminous' command.

This matches how we rebuild the missing set when reading it off disk in
read_log_and_missing().

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

osd_types, PGLog: encode missing based on features

Store whether the missing set should contain deletes, so that
persisted versions can be rebuilt if needed. Make missing_item
versioned, since it's persisted by the pg_log as an individual omap
value.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

osd_types, Objecter: make recovery_deletes feature create a new interval

This is needed to create a single place to regenerate the missing set
- at the start of a new interval where support for recovery deletes
changed.

The missing set is otherwise not cleared, so it would need to be
rebuilt in arbitrary places if e.g. an osd not supporting it went down
and restarted with support, or if we used a feature flag command to
trigger rebuilding the missing set.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

OSDMap, OSDMonitor: automatically set recovery deletes for luminous

Once the required osd release is luminous, all osds must support
recovery deletes, so set the flag then. This avoids an extra manual
step in luminous upgrades.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

OSDMap, OSDMonitor: add flag for all osds supporting recovery deletes

Just like sortbitwise, this can only be toggled on, and once on osds
that do not support it are not allowed to boot.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

osd: add incompat superblock feature for deletes during recovery

On-disk missing sets would need to be regenerated if downgraded from
luminous to kraken.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

include/ceph_features.h: add feature bit for handling deletes during recovery

The BLKIN feature bit was actually unused - it was a remnant from
earlier versions of the blkin work, but all the encoding is handled by
struct-level versioning in the version that merged.

Use bit 60 (unused in any prior version) so that recovery deletes
could potentially be backported.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

osd/PGBackend: include min_epoch in RecoveryDelete messages

This matches ordering with other recovery messages, and may speed up
processing a bit.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

osd/PGLog: reset complete_to when appending lost_delete entries

Since lost_deletes queue recovery directly, and don't go through
activate_not_complete(), our complete_to iterator may still point at
log.end() (a list iterator pointing to .end() will still point to
.end() after a push_back().). Reset it to point before these new
lost_delete entries. This is needed now that lost_deletes are
performed during recovery, instead of inline when merging logs.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #16386 from mikulely/rgw-cleanup

rgw: drop unused find_replacement() and some function docs

Reviewed-by: Jos Collin <jcollin@redhat.com>

rgw: miis trivial cleanup

+ drop outdated & misplaced function description
+ drop unused func find_placement

Signed-off-by: Jiaying Ren <jiaying.ren@umcloud.com>

doc: add instructions for replacing an OSD

* 8/ceph.rst: `rm` subcommand removes osd from osdmap, not the cluster.
  the latter is more ambiguous in different contexts.
* rados/operations/add-or-rm-osds.rst: add a subsection of "Replacing an
  OSD". update the subsection of "Removing the OSD" with "ceph osd
  purge" command
* release-notes.rst: link from it to the new subsection in add-or-rm-osds.rst

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16391 from Yan-waller/wip-walle-0718radososd

osd: combine conditional statements

Reviewed-by: Jos Collin <jcollin@redhat.com>

ceph_test_rados_api_watch_notify: flush after unwatch

After we unwatch, we need to make sure the finisher thread has flushed
its work before we tear down the test.  Otherwise, the notify callback
may dereference the test object looking for a member and segfault.

Probably-Fixes: http://tracker.ceph.com/issues/20105
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16401 from bstillwell/releases-doc-update-20170718

Documentation updates for July 2017 releases

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #16372 from dillaman/wip-20643

cls/rbd: trash_list should be iterable

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #16397 from liewegas/wip-20647

mon/MgrMonitor: fix standby addition to mgrmap

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Documentation updates for July 2017 releases

Updates for three new releases this month.  Two new Jewel releases
(10.2.8 and 10.2.9) and one new Luminous release candidate (12.1.1).

Signed-off-by: Bryan Stillwell <bstillwell@godaddy.com>

Merge pull request #16350 from ceph/wip-cd-docs-update

[docs/quick-start]: update quick start to add a note for mgr create command for luminous+ builds

crush: s/ruleset/id/ in decompiled output

Moving away from the 'ruleset' terminology.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MgrMonitor: fix standby addition to mgrmap

Inadvertantly removed by ba45fba01c96bcae5b55c50a6076bb1e879d58b9

Fixes: http://tracker.ceph.com/issues/20647
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15432 from dachary/wip-osd-new

ceph-disk: support osd new

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge remote-tracking branch 'gh/luminous'

Merge pull request #15837 from fangyuxiangGL/placement-rule-err

rgw: fix err when copy object in bucket with specified placement rule

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mon/AuthMonitor: generate bootstrap-mgr key on upgrade

We were adding the entity, but no key. By generating a key, we avoid
making things really weird.

Fixes: http://tracker.ceph.com/issues/20666
Signed-off-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #16291 from ZVampirEM77/wip-acl-grants-num-limit

rgw: acl grants num limit

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #16310 from ZVampirEM77/wip-acl-reqdata-too-large

rgw: req xml params size limitation error msg

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #16324 from yehudasa/wip-20620

rgw: delete object in error path

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15940 from smithfarm/wip-20425

build/ops: rpm: put mgr python build dependencies in make_check bcond

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #16283 from xiexingguo/wip-osd-segfault

mon, osd: misc fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16353 from tchaikov/wip-20605

mon: add force-create-pg back

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16356 from liewegas/wip-mgr-early-health

mon/MgrMonitor: only induce mgr epoch shortly after mkfs

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #16359 from liewegas/wip-cli-stdinout

ceph: allow '-' with -i and -o for stdin/stdout

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #16364 from liupan1111/wip-fix-clean

os/bluestore: use reference to avoid string copy

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16366 from smithfarm/wip-babeltrace-typo

build/ops: rpm: fix typo WTIH_BABELTRACE

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #16261 from tchaikov/wip-ceph-release-notes-py3

tools: ceph-release-notes: port it to py3

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #16380 from mikulely/doc-index-typos

test,rgw: fix rgw placement rule pool config option

Reviewed-by: Jos Collin <jcollin@redhat.com>

test,rgw: fix rgw placement rule pool config option

actually we're checking '--index-pool' & `--max-buckets` during
the argument parsing.

same as 3af3f4e84ca226f63f7491eb64195d07419a3e1e

Signed-off-by: Jiaying Ren jiaying.ren@umcloud.com

osd: combine conditional statement

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

Merge PR 16068 into master

* refs/remotes/upstream/pull/16068/head:
mds: wait auth pinned objects when deactivating mds
mds: fix "wait for stray manager to start"
ceph: kick purge queue when mds active
mds: properly do null snapflush
mds: force client to flush data when waiting on LOCK_SNAP_SYNC lock
mds: don't issue caps for frozen inode

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR 16202 into master

* refs/remotes/upstream/pull/16202/head:
doc: add some docs about 'cephfs-data-scan scan_links'
mds/FSMap.cc: remember stopped mds when resetting filesystem
tools/cephfs: handle removed dentries when replaying journal

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR 15969 into master

* refs/remotes/upstream/pull/15969/head:
mds/events: Initialize Non-static class members in ESession
mds/events: Initialize Non-static class members in fullbit
mds/events: Initialize Non-static class members in EImportStart
mds/events: Initialize Non-static class members in EFragment

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR 16153 into master

* refs/remotes/upstream/pull/16153/head:
mds: fixup may be an unclaimed tick_event

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR 16041 into master

* refs/remotes/upstream/pull/16041/head:
mds: “*dir” print repeatedly
MDS: tick_event repeated assignment

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR 16039 into master

* refs/remotes/upstream/pull/16039/head:
common/config_opts: drop unused opt(mds_kill_journal_at)

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR 15982 into master

* refs/remotes/upstream/pull/15982/head:
mds/MDSMap: use find in is_data_pool

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>