librados: 'pg ls' is not a mgr command

Note that this breaks the command *during* the mon upgrade from jewel ->
luminous, which is slightly annoying, but means we avoid proxying via the
mon after upgrade is complete, which is good and IMO more important.

In the future we may want librados to cache the command descriptions so
that commands can be directed automatically.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: restore mgr command lookup so that we can still proxy to mgr

Since mgr commands aren't in the main mon_commands array now, we need to
explicitly look up commands there too.  This restores the behavior
implemented below in which we forward misdirected mgr commands to the mon.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: constify _get_moncommand

Signed-off-by: Sage Weil <sage@redhat.com>

mon: define static mgr_commands at mkfs time

This closes a window between mkfs and when the first mgr goes active
where *no* mgr commands are defined, and things like 'pg dump' fail.  We
do not get the default set of commands defined by modules, but we get
everything else.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr: move mgr_commands to separate compilation unit

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonCommands: std::

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MgrMonitor: mark mgr commands with FLAG_MGR

Signed-off-by: Sage Weil <sage@redhat.com>

qa/doc: update for "mgr tell" no longer needed

Signed-off-by: John Spray <john.spray@redhat.com>

mon: load mgr commands dynamically

So that the list of commands includes python modules,
thus allowing python-provided commands to be invoked
by the CLI with out a `tell mgr` prefix.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: transmit command descriptions to mgr in activating beacon

The mgr already sends a beacon to the mon immediately
after loading python modules in Mgr::init, to indicate
that it is now available.  Use that beacon to transmit
the command descriptions.

The monitor should handle this beacon by persisting
the command descriptions before persisting the updated
mgrmap that indicates that the mgr is now active.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: use MonCommand for command descriptions

...and update the MonCommand encoding so that we
can readily send vectors of them around.

Signed-off-by: John Spray <john.spray@redhat.com>

encoding: remove encode_array_nohead

This was just a for loop.  No longer needed for
MonCommands, and the usage in memstore/PageSet
was just iterating over char* and should never have
been there to begin with.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #16411 from smithfarm/wip-crn-regression

tools: ceph-release-notes: refactor and fix regressions

Reviewed-By: Kefu Chai <kchai@redhat.com>
Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

tools: ceph-release-notes: match Reviewed-by more liberally

Signed-off-by: Nathan Cutler <ncutler@suse.com>

tools: ceph-release-notes: fix sorted() key lambda regression

https://github.com/ceph/ceph/pull/16261 ported the script to Python 3, but it
retained the 2-argument version of the sorted() key function - in Python 3 the
key function takes only one argument.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

tools: ceph-release-notes: refactor and fix regression

This commit refactors the logic for determining the PR title and merge message,
and fixes a regression introduced by https://github.com/ceph/ceph/pull/16277

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #16386 from mikulely/rgw-cleanup

rgw: drop unused find_replacement() and some function docs

Reviewed-by: Jos Collin <jcollin@redhat.com>

rgw: miis trivial cleanup

+ drop outdated & misplaced function description
+ drop unused func find_placement

Signed-off-by: Jiaying Ren <jiaying.ren@umcloud.com>

Merge pull request #16391 from Yan-waller/wip-walle-0718radososd

osd: combine conditional statements

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #16401 from bstillwell/releases-doc-update-20170718

Documentation updates for July 2017 releases

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #16372 from dillaman/wip-20643

cls/rbd: trash_list should be iterable

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #16397 from liewegas/wip-20647

mon/MgrMonitor: fix standby addition to mgrmap

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Documentation updates for July 2017 releases

Updates for three new releases this month.  Two new Jewel releases
(10.2.8 and 10.2.9) and one new Luminous release candidate (12.1.1).

Signed-off-by: Bryan Stillwell <bstillwell@godaddy.com>

Merge pull request #16350 from ceph/wip-cd-docs-update

[docs/quick-start]: update quick start to add a note for mgr create command for luminous+ builds

mon/MgrMonitor: fix standby addition to mgrmap

Inadvertantly removed by ba45fba01c96bcae5b55c50a6076bb1e879d58b9

Fixes: http://tracker.ceph.com/issues/20647
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15432 from dachary/wip-osd-new

ceph-disk: support osd new

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge remote-tracking branch 'gh/luminous'

Merge pull request #15837 from fangyuxiangGL/placement-rule-err

rgw: fix err when copy object in bucket with specified placement rule

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #16291 from ZVampirEM77/wip-acl-grants-num-limit

rgw: acl grants num limit

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #16310 from ZVampirEM77/wip-acl-reqdata-too-large

rgw: req xml params size limitation error msg

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #16324 from yehudasa/wip-20620

rgw: delete object in error path

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15940 from smithfarm/wip-20425

build/ops: rpm: put mgr python build dependencies in make_check bcond

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #16283 from xiexingguo/wip-osd-segfault

mon, osd: misc fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16353 from tchaikov/wip-20605

mon: add force-create-pg back

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16356 from liewegas/wip-mgr-early-health

mon/MgrMonitor: only induce mgr epoch shortly after mkfs

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #16359 from liewegas/wip-cli-stdinout

ceph: allow '-' with -i and -o for stdin/stdout

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #16364 from liupan1111/wip-fix-clean

os/bluestore: use reference to avoid string copy

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16366 from smithfarm/wip-babeltrace-typo

build/ops: rpm: fix typo WTIH_BABELTRACE

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #16261 from tchaikov/wip-ceph-release-notes-py3

tools: ceph-release-notes: port it to py3

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #16380 from mikulely/doc-index-typos

test,rgw: fix rgw placement rule pool config option

Reviewed-by: Jos Collin <jcollin@redhat.com>

test,rgw: fix rgw placement rule pool config option

actually we're checking '--index-pool' & `--max-buckets` during
the argument parsing.

same as 3af3f4e84ca226f63f7491eb64195d07419a3e1e

Signed-off-by: Jiaying Ren jiaying.ren@umcloud.com

osd: combine conditional statement

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

Merge PR 16068 into master

* refs/remotes/upstream/pull/16068/head:
mds: wait auth pinned objects when deactivating mds
mds: fix "wait for stray manager to start"
ceph: kick purge queue when mds active
mds: properly do null snapflush
mds: force client to flush data when waiting on LOCK_SNAP_SYNC lock
mds: don't issue caps for frozen inode

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR 16202 into master

* refs/remotes/upstream/pull/16202/head:
doc: add some docs about 'cephfs-data-scan scan_links'
mds/FSMap.cc: remember stopped mds when resetting filesystem
tools/cephfs: handle removed dentries when replaying journal

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR 15969 into master

* refs/remotes/upstream/pull/15969/head:
mds/events: Initialize Non-static class members in ESession
mds/events: Initialize Non-static class members in fullbit
mds/events: Initialize Non-static class members in EImportStart
mds/events: Initialize Non-static class members in EFragment

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR 16153 into master

* refs/remotes/upstream/pull/16153/head:
mds: fixup may be an unclaimed tick_event

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR 16041 into master

* refs/remotes/upstream/pull/16041/head:
mds: “*dir” print repeatedly
MDS: tick_event repeated assignment

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR 16039 into master

* refs/remotes/upstream/pull/16039/head:
common/config_opts: drop unused opt(mds_kill_journal_at)

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR 15982 into master

* refs/remotes/upstream/pull/15982/head:
mds/MDSMap: use find in is_data_pool

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR 15981 into master

* refs/remotes/upstream/pull/15981/head:
mds: reinstate daemonperf counters

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR 16200 into master

* refs/remotes/upstream/pull/16200/head:
qa: thrash max_mds and deactivate ranks

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR 15979 into master

* refs/remotes/upstream/pull/15979/head:
Ignore unmatched rstat errors from MDS during rebuild testing

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR 16288 into master

* refs/remotes/upstream/pull/16288/head:
qa/cephfs: don't use int() to convert string of float point number

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #16375 from batrick/docker-core-debug

script: add docker core dump debugger

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

script: add docker core dump debugger

This script sets up a docker container with all the packages used for your Ceph
(test) branch. From there, you can gdb the core dump if you have access to the
test cluster.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #16367 from Yan-waller/wip-walle-0715osdmon

mon: Fix output text and doc

Reviewed-by: Jos Collin <jcollin@redhat.com>

mon: cleanups

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

Merge pull request #16203 from shashalu/raise-s3policy-debug-level

rgw: raise debug level of RGWPostObj_ObjStore_S3::get_policy

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #16373 from jcsp/wip-dashboard-osds

mgr/dashboard: add OSD list view

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

update ceph-deploy quick start to add a note for mgr create command

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #15674 from chardan/jfw-wip-20240-printf-format

libradosstriper: fix format injection vulnerability

Reviewed-by: Kefu Chai <kchai@redhat.com>

12.1.1

mgr/dashboard: add OSD list view

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #15667 from rzarzynski/wip-rgw-fix-nullptrderef-in-rgw-admin

rgw: fix potential null pointer dereference in rgw_admin.

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #16251 from linuxbox2/wip-mgr-nfs-svcmap

rgw_file:  add service map registration

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16341 from liewegas/wip-20631

osd: restart boot process if waiting for luminous mons

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15807 from zhangsw/feature-lifecycle-date

rgw: S3 lifecycle now supports expiration date

Reviewed-by: Daniel Gryniewicz <dang@fprintf.net>

cls/rbd: trash_list should be iterable

Fixes: http://tracker.ceph.com/issues/20643
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mon/OSDMonitor: fix "ceph osd pool get rbd all --format=json-pretty"

Two problems:
(1) MIN_WRITE_RECENCY_FOR_PROMOTE is a tier-only option.
(2) should automatically filter out unset pool options, otherwise it will
    keep outputing rubbish:

{
    "pool": "rbd",
    "pool_id": 3,
    "min_write_recency_for_promote": 0
}
{
    "pool": "rbd",
    "pool_id": 3,
    "fast_read": 0
}
{
    "pool": "rbd",
    "pool_id": 3
}
{
    "pool": "rbd",
    "pool_id": 3
}
{
    "pool": "rbd",
    "pool_id": 3
}
{
    "pool": "rbd",
    "pool_id": 3,
    "csum_type": "???"
}

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/OSDMonitor: drop unnecessary write permission for "crush get-tunable" command

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/OSD: filter out deprecated meta for bluestore

Journal path is filestore related...

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/OSDMonitor: cleanup last_osd_report if osd does not exist

In case we might want to reuse the same slot later.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/OSD: gracefully shutdown on error exit during init

This can avoid crashes as below:

  0> 2017-07-12 09:34:47.427438 7f320ce61b80 -1 /home/xxg/build/ceph-dev/src/common/HeartbeatMap.cc: In function 'ceph::HeartbeatMap::~HeartbeatMap()'
thread 7f320ce61b80 time 2017-07-12 09:34:47.422986
/home/xxg/build/ceph-dev/src/common/HeartbeatMap.cc: 39: FAILED assert(m_workers.empty())

 ceph version 12.1.0-702-gc5b99af (c5b99af) luminous (rc)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110) [0x7f320d8ba7f0]
 2: (ceph::HeartbeatMap::~HeartbeatMap()+0xf8) [0x7f320d9be0a8]
 3: (CephContext::~CephContext()+0x40c) [0x7f320d9a648c]
 4: (CephContext::put()+0xe6) [0x7f320d9a6776]
 5: (main()+0xad3) [0x7f320d282953]
 6: (__libc_start_main()+0xf5) [0x7f32094cfb15]
 7: (()+0x4964c9) [0x7f320d31f4c9]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

ceph: allow '-' with -i and -o for stdin/stdout

Signed-off-by: Sage Weil <sage@redhat.com>

mon: add force-create-pg back

and now it's "ceph osd force-create-pg'

Fixes: http://tracker.ceph.com/issues/20605
Signed-off-by: Kefu Chai <kchai@redhat.com>

build/ops: rpm: fix typo WTIH_BABELTRACE

Introduced by b331898ea9aefc547265b388dddbc388417184fe

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #16339 from majianpeng/test-fio-print-more-perfcounter

test/fio: print all perfcounters rather than objectstore itself.

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: use reference to void string copy

Signed-off-by: Pan Liu <wanjun.lp@alibaba-inc.com>

Merge pull request #16346 from liewegas/wip-20602

mon: skip crush smoke test when running under valgrind

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16302 from liewegas/wip-mds-dup-alerts

mon/MDSMonitor: fix segv when multiple MDSs raise same alert

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

build/ops: rpm: socat is only needed for "make check"

Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: rpm: put mgr python build dependencies in make_check bcond

Fixes: http://tracker.ceph.com/issues/20425
Signed-off-by: Nathan Cutler <ncutler@suse.com>
Signed-off-by: Tim Serong <tserong@suse.com>

Merge pull request #16189 from bassam/pr-msgr-bind-addr

mon: add support public_bind_addr option

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16349 from liewegas/wip-vstart-bind

vstart.sh: bind restful, dashboard to ::, not 127.0.0.1

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16329 from joscollin/wip-cleanup-crush-warning

crush: silence warning from -Woverflow

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16345 from jcsp/wip-watch-channel

ceph.in: filter out audit from ceph -w

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16315 from majianpeng/bluestore-misc-fix

os/bluestore: misc fix and cleanups

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16351 from liewegas/wip-mgr-init-debug

mgr,mon: debug init and mgrdigest subscriptions

Merge pull request #16358 from liupan1111/wip-fix-client

test/msgr: fixed the hang issue for perf_msg_client

Reviewed-by: Haomai Wang <haomai@xsky.com>

test/msgr: fixed the hang issue for perf_msg_client

Signed-off-by: Pan Liu <wanjun.lp@alibaba-inc.com>

qa/suites/rados/rest/mgr-restful: simplify

Use default port; don't bother setting bind addr.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MgrMonitor: only induce mgr epoch shortly after mkfs

For early clusters, if there isn't an active manager, we eventually want
to trigger a health warning by rolling over the mgrmap epoch.  We don't
want to do that if we have no active/available manager after that.  Fix
by checking ever_had_active_mgr here.

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: acl grants num limit

According to AWS S3 in this document[1], an ACL can have up to 100
grants.

If the nums of grants is larger than 100, S3 will return like following:
400
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>MalformedACLError</Code><Message>The XML you provided was not well-formed or did not validate against our published schema</Message><RequestId>10EC67824572C378</RequestId><HostId>AWL3NnQChs/HCfOTu5MtyEc9uzRuxpYMhmvXQry2CovCcuxO2/tMqY1zGoWOur86ipQt3v/WEiA=</HostId></Error>

Now if the nums of request acl grants is larger than the maximum allowed, rgw will return
like following:
400
<?xml version="1.0" encoding="UTF-8"?><Error><Code>MalformedACLError</Code><Message>The request is rejected, because the acl grants number you requested is larger than the maximum 101 grants allowed in an acl.</Message><BucketName>222</BucketName><RequestId>tx000000000000000000017-00596b5fad-101a-default</RequestId><HostId>101a-default-default</HostId></Error>

The maximum number of acl grants can be configured in config file with the configuration item:

rgw_acl_grants_max_num

[1] http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html

Signed-off-by: Enming Zhang <enming.zhang@umcloud.com>

rgw: req xml params size limitation error msg

Now rgw will return like following:

400
<?xml version="1.0" encoding="UTF-8"?><Error><Code>MalformedXML</Code><Message>The XML you provided was larger than the maximum 2048 bytes allowed.</Message><BucketName>333</BucketName><RequestId>tx000000000000000000009-00596a1331-101a-default</RequestId><HostId>101a-default-default</HostId></Error>

Signed-off-by: Enming Zhang <enming.zhang@umcloud.com>

Merge pull request #15587 from wjwithagen/wip-wjw-ceph-disk-is_diskdevice

ceph-disk/ceph_disk/main.py: Replace ST_ISBLK() test by is_diskdevice()

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16347 from tchaikov/wip-test-ceph-disk

tests: ceph-disk: use communicate() instead of wait() for output

Reviewed-by: Loic Dachary <ldachary@redhat.com>

mon/MonCommand: drop unnecessary write permission

since "log last" does not ask for it.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/OSDMap: kill dead structure "struct qi"

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #16334 from wjwithagen/wjw-bug-stringyfy

core:" Stringify needs access to << before reference" src/include/stringify.h

Reviewed-by: Jos Collin <jcollin@redhat.com>

osd/OSDMap: allow bidirectional swap of pg-upmap-items

This is useful when we also want an even distribution of pg primaries across osds.
For example:
Was:
[0 1 2]

By applying bidirectional swap of pg-upmap-items mapping [[0,1],[1,0]], now:
[1 0 2]

Thus we successfully decrease the number of primaries of osd.0 by 1 without
affecting the current (even) distribution of global pgs.

Real exmaple:
./bin/ceph pg ls-by-pool rbd
PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES LOG DISK_LOG STATE        STATE_STAMP                VERSION REPORTED UP      UP_PRIMARY ACTING  ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP                LAST_DEEP_SCRUB DEEP_SCRUB_STAMP
3.0           0                  0        0         0       0     0   0        0 active+clean 2017-07-12 15:14:45.083441     0'0    29:13 [0,1,3]          0 [0,1,3]              0        0'0 2017-07-12 15:14:14.515989             0'0 2017-07-12 15:14:14.515989

./bin/ceph osd pg-upmap-items 3.0 0 1 1 0 3 5
set 3.0 pg_upmap_items mapping to [0->1,1->0,3->5]

./bin/ceph pg ls-by-pool rbd
PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES LOG DISK_LOG STATE        STATE_STAMP                VERSION REPORTED UP      UP_PRIMARY ACTING  ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP                LAST_DEEP_SCRUB DEEP_SCRUB_STAMP
3.0           0                  0        0         0       0     0   0        0 active+clean 2017-07-12 15:16:22.648424     0'0    33:13 [1,0,5]          1 [1,0,5]              1        0'0 2017-07-12 15:14:14.515989             0'0 2017-07-12 15:14:14.515989

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>