mon: health warning if in-use pools don't have application enabled

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mon: added new "osd pool application" commands

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mon: store application metadata in pg_pool_t

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #16275 from linuxbox2/wip-rgw-readdir-cookie

rgw_file: permit dirent offset computation

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #16368 from theanalyst/fix/rgw-reshard-pool-ns

rgw: use a namespace for rgw reshard pool for upgrades as well

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #16395 from jecluis/wip-fix-mon-mgr-bootstrap

mon/AuthMonitor: generate bootstrap-mgr key on upgrade

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16425 from smithfarm/wip-ceph-w

doc: PendingReleaseNotes: "ceph -w" behavior has changed drastically

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16314 from tchaikov/wip-doc-replace-osd

doc: add instructions for replacing an OSD

Reviewed-by: Alfredo Deza <adeza@redhat.com>

doc: PendingReleaseNotes: "ceph -w" behavior has changed drastically

Signed-off-by: Joao Eduardo Luis <joao@suse.de>
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #16411 from smithfarm/wip-crn-regression

tools: ceph-release-notes: refactor and fix regressions

Reviewed-By: Kefu Chai <kchai@redhat.com>
Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

tools: ceph-release-notes: match Reviewed-by more liberally

Signed-off-by: Nathan Cutler <ncutler@suse.com>

tools: ceph-release-notes: fix sorted() key lambda regression

https://github.com/ceph/ceph/pull/16261 ported the script to Python 3, but it
retained the 2-argument version of the sorted() key function - in Python 3 the
key function takes only one argument.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

tools: ceph-release-notes: refactor and fix regression

This commit refactors the logic for determining the PR title and merge message,
and fixes a regression introduced by https://github.com/ceph/ceph/pull/16277

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #16386 from mikulely/rgw-cleanup

rgw: drop unused find_replacement() and some function docs

Reviewed-by: Jos Collin <jcollin@redhat.com>

rgw: miis trivial cleanup

+ drop outdated & misplaced function description
+ drop unused func find_placement

Signed-off-by: Jiaying Ren <jiaying.ren@umcloud.com>

doc: add instructions for replacing an OSD

* 8/ceph.rst: `rm` subcommand removes osd from osdmap, not the cluster.
  the latter is more ambiguous in different contexts.
* rados/operations/add-or-rm-osds.rst: add a subsection of "Replacing an
  OSD". update the subsection of "Removing the OSD" with "ceph osd
  purge" command
* release-notes.rst: link from it to the new subsection in add-or-rm-osds.rst

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16391 from Yan-waller/wip-walle-0718radososd

osd: combine conditional statements

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #16401 from bstillwell/releases-doc-update-20170718

Documentation updates for July 2017 releases

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #16372 from dillaman/wip-20643

cls/rbd: trash_list should be iterable

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #16397 from liewegas/wip-20647

mon/MgrMonitor: fix standby addition to mgrmap

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Documentation updates for July 2017 releases

Updates for three new releases this month.  Two new Jewel releases
(10.2.8 and 10.2.9) and one new Luminous release candidate (12.1.1).

Signed-off-by: Bryan Stillwell <bstillwell@godaddy.com>

Merge pull request #16350 from ceph/wip-cd-docs-update

[docs/quick-start]: update quick start to add a note for mgr create command for luminous+ builds

mon/MgrMonitor: fix standby addition to mgrmap

Inadvertantly removed by ba45fba01c96bcae5b55c50a6076bb1e879d58b9

Fixes: http://tracker.ceph.com/issues/20647
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15432 from dachary/wip-osd-new

ceph-disk: support osd new

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge remote-tracking branch 'gh/luminous'

Merge pull request #15837 from fangyuxiangGL/placement-rule-err

rgw: fix err when copy object in bucket with specified placement rule

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mon/AuthMonitor: generate bootstrap-mgr key on upgrade

We were adding the entity, but no key. By generating a key, we avoid
making things really weird.

Fixes: http://tracker.ceph.com/issues/20666
Signed-off-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #16291 from ZVampirEM77/wip-acl-grants-num-limit

rgw: acl grants num limit

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #16310 from ZVampirEM77/wip-acl-reqdata-too-large

rgw: req xml params size limitation error msg

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #16324 from yehudasa/wip-20620

rgw: delete object in error path

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15940 from smithfarm/wip-20425

build/ops: rpm: put mgr python build dependencies in make_check bcond

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #16283 from xiexingguo/wip-osd-segfault

mon, osd: misc fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16353 from tchaikov/wip-20605

mon: add force-create-pg back

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16356 from liewegas/wip-mgr-early-health

mon/MgrMonitor: only induce mgr epoch shortly after mkfs

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #16359 from liewegas/wip-cli-stdinout

ceph: allow '-' with -i and -o for stdin/stdout

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #16364 from liupan1111/wip-fix-clean

os/bluestore: use reference to avoid string copy

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16366 from smithfarm/wip-babeltrace-typo

build/ops: rpm: fix typo WTIH_BABELTRACE

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #16261 from tchaikov/wip-ceph-release-notes-py3

tools: ceph-release-notes: port it to py3

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #16380 from mikulely/doc-index-typos

test,rgw: fix rgw placement rule pool config option

Reviewed-by: Jos Collin <jcollin@redhat.com>

test,rgw: fix rgw placement rule pool config option

actually we're checking '--index-pool' & `--max-buckets` during
the argument parsing.

same as 3af3f4e84ca226f63f7491eb64195d07419a3e1e

Signed-off-by: Jiaying Ren jiaying.ren@umcloud.com

osd: combine conditional statement

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

Merge PR 16068 into master

* refs/remotes/upstream/pull/16068/head:
mds: wait auth pinned objects when deactivating mds
mds: fix "wait for stray manager to start"
ceph: kick purge queue when mds active
mds: properly do null snapflush
mds: force client to flush data when waiting on LOCK_SNAP_SYNC lock
mds: don't issue caps for frozen inode

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR 16202 into master

* refs/remotes/upstream/pull/16202/head:
doc: add some docs about 'cephfs-data-scan scan_links'
mds/FSMap.cc: remember stopped mds when resetting filesystem
tools/cephfs: handle removed dentries when replaying journal

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR 15969 into master

* refs/remotes/upstream/pull/15969/head:
mds/events: Initialize Non-static class members in ESession
mds/events: Initialize Non-static class members in fullbit
mds/events: Initialize Non-static class members in EImportStart
mds/events: Initialize Non-static class members in EFragment

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR 16153 into master

* refs/remotes/upstream/pull/16153/head:
mds: fixup may be an unclaimed tick_event

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR 16041 into master

* refs/remotes/upstream/pull/16041/head:
mds: “*dir” print repeatedly
MDS: tick_event repeated assignment

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR 16039 into master

* refs/remotes/upstream/pull/16039/head:
common/config_opts: drop unused opt(mds_kill_journal_at)

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR 15982 into master

* refs/remotes/upstream/pull/15982/head:
mds/MDSMap: use find in is_data_pool

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR 15981 into master

* refs/remotes/upstream/pull/15981/head:
mds: reinstate daemonperf counters

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR 16200 into master

* refs/remotes/upstream/pull/16200/head:
qa: thrash max_mds and deactivate ranks

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR 15979 into master

* refs/remotes/upstream/pull/15979/head:
Ignore unmatched rstat errors from MDS during rebuild testing

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR 16288 into master

* refs/remotes/upstream/pull/16288/head:
qa/cephfs: don't use int() to convert string of float point number

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #16375 from batrick/docker-core-debug

script: add docker core dump debugger

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

script: add docker core dump debugger

This script sets up a docker container with all the packages used for your Ceph
(test) branch. From there, you can gdb the core dump if you have access to the
test cluster.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #16367 from Yan-waller/wip-walle-0715osdmon

mon: Fix output text and doc

Reviewed-by: Jos Collin <jcollin@redhat.com>

mon: cleanups

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

Merge pull request #16203 from shashalu/raise-s3policy-debug-level

rgw: raise debug level of RGWPostObj_ObjStore_S3::get_policy

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #16373 from jcsp/wip-dashboard-osds

mgr/dashboard: add OSD list view

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

update ceph-deploy quick start to add a note for mgr create command

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #15674 from chardan/jfw-wip-20240-printf-format

libradosstriper: fix format injection vulnerability

Reviewed-by: Kefu Chai <kchai@redhat.com>

12.1.1

mgr/dashboard: add OSD list view

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #15667 from rzarzynski/wip-rgw-fix-nullptrderef-in-rgw-admin

rgw: fix potential null pointer dereference in rgw_admin.

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #16251 from linuxbox2/wip-mgr-nfs-svcmap

rgw_file:  add service map registration

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16341 from liewegas/wip-20631

osd: restart boot process if waiting for luminous mons

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15807 from zhangsw/feature-lifecycle-date

rgw: S3 lifecycle now supports expiration date

Reviewed-by: Daniel Gryniewicz <dang@fprintf.net>

cls/rbd: trash_list should be iterable

Fixes: http://tracker.ceph.com/issues/20643
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

PendingReleaseNotes: add a note on RGW's pool change for resharding

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: move reshard pool to ns in log pool

Since the pool was introduced only in Luminous dev and RC releases we
can probably upgrade without the need to bump up the the struct version
numbers. This needs a pending release notes entry

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: dump the reshard pool in rgw zone params json

So that the zone get/put commands display the reshard pool as well

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

mon/OSDMonitor: fix "ceph osd pool get rbd all --format=json-pretty"

Two problems:
(1) MIN_WRITE_RECENCY_FOR_PROMOTE is a tier-only option.
(2) should automatically filter out unset pool options, otherwise it will
    keep outputing rubbish:

{
    "pool": "rbd",
    "pool_id": 3,
    "min_write_recency_for_promote": 0
}
{
    "pool": "rbd",
    "pool_id": 3,
    "fast_read": 0
}
{
    "pool": "rbd",
    "pool_id": 3
}
{
    "pool": "rbd",
    "pool_id": 3
}
{
    "pool": "rbd",
    "pool_id": 3
}
{
    "pool": "rbd",
    "pool_id": 3,
    "csum_type": "???"
}

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/OSDMonitor: drop unnecessary write permission for "crush get-tunable" command

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/OSD: filter out deprecated meta for bluestore

Journal path is filestore related...

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/OSDMonitor: cleanup last_osd_report if osd does not exist

In case we might want to reuse the same slot later.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/OSD: gracefully shutdown on error exit during init

This can avoid crashes as below:

  0> 2017-07-12 09:34:47.427438 7f320ce61b80 -1 /home/xxg/build/ceph-dev/src/common/HeartbeatMap.cc: In function 'ceph::HeartbeatMap::~HeartbeatMap()'
thread 7f320ce61b80 time 2017-07-12 09:34:47.422986
/home/xxg/build/ceph-dev/src/common/HeartbeatMap.cc: 39: FAILED assert(m_workers.empty())

 ceph version 12.1.0-702-gc5b99af (c5b99af) luminous (rc)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110) [0x7f320d8ba7f0]
 2: (ceph::HeartbeatMap::~HeartbeatMap()+0xf8) [0x7f320d9be0a8]
 3: (CephContext::~CephContext()+0x40c) [0x7f320d9a648c]
 4: (CephContext::put()+0xe6) [0x7f320d9a6776]
 5: (main()+0xad3) [0x7f320d282953]
 6: (__libc_start_main()+0xf5) [0x7f32094cfb15]
 7: (()+0x4964c9) [0x7f320d31f4c9]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

ceph: allow '-' with -i and -o for stdin/stdout

Signed-off-by: Sage Weil <sage@redhat.com>

mon: add force-create-pg back

and now it's "ceph osd force-create-pg'

Fixes: http://tracker.ceph.com/issues/20605
Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: clarify when encountering eacces for reshard list

Specify that we don't have access to the reshard pool when encountering
EACCESS.
TODO: get rgw's name and add that in the log message

Fixes http://tracker.ceph.com/issues/20289

Signed-off-by: Karol Mroz <kmroz@suse.de>
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

build/ops: rpm: fix typo WTIH_BABELTRACE

Introduced by b331898ea9aefc547265b388dddbc388417184fe

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #16339 from majianpeng/test-fio-print-more-perfcounter

test/fio: print all perfcounters rather than objectstore itself.

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: use reference to void string copy

Signed-off-by: Pan Liu <wanjun.lp@alibaba-inc.com>

Merge pull request #16346 from liewegas/wip-20602

mon: skip crush smoke test when running under valgrind

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16302 from liewegas/wip-mds-dup-alerts

mon/MDSMonitor: fix segv when multiple MDSs raise same alert

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

build/ops: rpm: socat is only needed for "make check"

Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: rpm: put mgr python build dependencies in make_check bcond

Fixes: http://tracker.ceph.com/issues/20425
Signed-off-by: Nathan Cutler <ncutler@suse.com>
Signed-off-by: Tim Serong <tserong@suse.com>

Merge pull request #16189 from bassam/pr-msgr-bind-addr

mon: add support public_bind_addr option

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16349 from liewegas/wip-vstart-bind

vstart.sh: bind restful, dashboard to ::, not 127.0.0.1

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16329 from joscollin/wip-cleanup-crush-warning

crush: silence warning from -Woverflow

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16345 from jcsp/wip-watch-channel

ceph.in: filter out audit from ceph -w

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16315 from majianpeng/bluestore-misc-fix

os/bluestore: misc fix and cleanups

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16351 from liewegas/wip-mgr-init-debug

mgr,mon: debug init and mgrdigest subscriptions

Merge pull request #16358 from liupan1111/wip-fix-client

test/msgr: fixed the hang issue for perf_msg_client

Reviewed-by: Haomai Wang <haomai@xsky.com>

test/msgr: fixed the hang issue for perf_msg_client

Signed-off-by: Pan Liu <wanjun.lp@alibaba-inc.com>

qa/suites/rados/rest/mgr-restful: simplify

Use default port; don't bother setting bind addr.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MgrMonitor: only induce mgr epoch shortly after mkfs

For early clusters, if there isn't an active manager, we eventually want
to trigger a health warning by rolling over the mgrmap epoch.  We don't
want to do that if we have no active/available manager after that.  Fix
by checking ever_had_active_mgr here.

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: acl grants num limit

According to AWS S3 in this document[1], an ACL can have up to 100
grants.

If the nums of grants is larger than 100, S3 will return like following:
400
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>MalformedACLError</Code><Message>The XML you provided was not well-formed or did not validate against our published schema</Message><RequestId>10EC67824572C378</RequestId><HostId>AWL3NnQChs/HCfOTu5MtyEc9uzRuxpYMhmvXQry2CovCcuxO2/tMqY1zGoWOur86ipQt3v/WEiA=</HostId></Error>

Now if the nums of request acl grants is larger than the maximum allowed, rgw will return
like following:
400
<?xml version="1.0" encoding="UTF-8"?><Error><Code>MalformedACLError</Code><Message>The request is rejected, because the acl grants number you requested is larger than the maximum 101 grants allowed in an acl.</Message><BucketName>222</BucketName><RequestId>tx000000000000000000017-00596b5fad-101a-default</RequestId><HostId>101a-default-default</HostId></Error>

The maximum number of acl grants can be configured in config file with the configuration item:

rgw_acl_grants_max_num

[1] http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html

Signed-off-by: Enming Zhang <enming.zhang@umcloud.com>

rgw: req xml params size limitation error msg

Now rgw will return like following:

400
<?xml version="1.0" encoding="UTF-8"?><Error><Code>MalformedXML</Code><Message>The XML you provided was larger than the maximum 2048 bytes allowed.</Message><BucketName>333</BucketName><RequestId>tx000000000000000000009-00596a1331-101a-default</RequestId><HostId>101a-default-default</HostId></Error>

Signed-off-by: Enming Zhang <enming.zhang@umcloud.com>

Merge pull request #15587 from wjwithagen/wip-wjw-ceph-disk-is_diskdevice

ceph-disk/ceph_disk/main.py: Replace ST_ISBLK() test by is_diskdevice()

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16347 from tchaikov/wip-test-ceph-disk

tests: ceph-disk: use communicate() instead of wait() for output

Reviewed-by: Loic Dachary <ldachary@redhat.com>

mon/MonCommand: drop unnecessary write permission

since "log last" does not ask for it.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>