OSDMonitor: check mon_max_pool_pg_num when set pool pg_num

Check the mon_max_pool_pg_num when creating pools, but did not

check in the the modification. So we can create a pool and then

modify pg_num exceeds the mon_max_pool_pg_num limit.

By following these steps can verify this problem

step 1: set  mon_max_pool_pg_num = 65536

step 2: ceph osd pool create testpool 65536 65536

step 3: ceph osd pool set testpool pg_num 65537

Signed-off-by: chenhg <c744402859@gmail.com>

Merge pull request #16488 from liewegas/wip-skewed

common/options: remove mon_warn_osd_usage_min_max_delta from options.cc too

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #16490 from jdurgin/wip-20739

osd/PrimaryLogPG: skip deleted missing objects in pg[n]ls

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16501 from liewegas/wip-fix-mgr-commands

mon: do not dereference empty mgr_commands

Merge pull request #16500 from liewegas/wip-compact-sudo

qa/workunits/cephtool/test.sh: add sudo for daemon compact

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15991 from dillaman/wip-rbd-auth-profile

mon,osd: new rbd-based cephx cap profiles

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16432 from liewegas/wip-mgr-daemonstate-lock

mgr: add per-DaemonState lock

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #16460 from liewegas/wip-mgr-metadata

mon: add mgr metdata commands, and overall 'versions' command for all daemon versions

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16465 from xiexingguo/wip-shutup-noscrub-warn

mon: stop issuing not-[deep]-scrubbed warnings if disabled

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16477 from tchaikov/wip-health-mon

mon/HealthMonitor: trigger a proposal if stat updated

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #16478 from xiexingguo/wip-drop-empty-message

mon/HealthMonitor: avoid sending unnecessary MMonHealthChecks to leader

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16480 from jcsp/wip-rm-convert

common: remove config opt conversion utility

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

mon: do not dereference empty mgr_commands

This may be empty before the initial quorum commits the static command
list.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/cephtool/test.sh: add sudo for daemon compact

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR 16390 into master

* refs/remotes/upstream/pull/16390/head:
mds: cleanup with unique_ptr
mds: add to get version in the mds_commands
mds: cleanup

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR 16379 into master

* refs/remotes/upstream/pull/16379/head:
qa: fix MDS_CLIENT_RECALL copy error

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR 16337 into master

* refs/remotes/upstream/pull/16337/head:
mds: clear dirty rstat flag on non-auth inode when finishing scatter update

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR 16280 into master

* refs/remotes/upstream/pull/16280/head:
Client: mkdirs bugs when two clients send mkdir request for a same dir

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR 16278 into master

* refs/remotes/upstream/pull/16278/head:
MDSMonitor: show laggy MDSs at higher debug level

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR 16270 into master

* refs/remotes/upstream/pull/16270/head:
mds: improve wording of mdsmap dne logging

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR 16232 into master

* refs/remotes/upstream/pull/16232/head:
MDS: update the mlogger of mds in function check_memory_usage

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR 16231 into master

* refs/remotes/upstream/pull/16231/head:
Client: get rid of variable "butnot" in "client::get_caps"

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR 16226 into master

* refs/remotes/upstream/pull/16226/head:
qa: wait for OSDMap to propagate for snap purge

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR 16149 into master

* refs/remotes/upstream/pull/16149/head:
mds: misc cleanup

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR 15942 into master

* refs/remotes/upstream/pull/15942/head:
client: add unlock_fh_pos when abort

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

osd/PrimaryLogPG: skip deleted missing objects in pg[n]ls

Fixes: http://tracker.ceph.com/issues/20739
Signed-off-by: Josh Durgin <jdurgin@redhat.com>

PendingReleaseNotes: added blurb for new RBD cap profiles

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

osd: expand profile caps upon construction to avoid potential race

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits/rbd: rbd-mirror tests should use 'mirror' user

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/suites/rbd: mirroring tests should use rbd cap profiles

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits/rbd: devstack test should use auth profiles

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

doc: include details for new RBD cap profiles

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mon,osd: drop the "allow" prefix for profile caps

The use of the a profile implies that it sets everything up. If
support for deny is ever added, it doesn't make much sense to
deny a profile.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

osd: new 'rbd'/'rbd-read-only' osd cap profiles

The 'rbd' profile provides read-only class access on all pools
to the 'rbd_children' object and write access to the optionally
specified pool. The 'rbd-read-only' profile does as its name
implies.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

osd: primitive cephx osd profile cap support

The two new example profiles are read-only and read-write

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mon: added 'rbd' profile

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mon: support regex-based restrictions on command caps

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #16445 from tchaikov/wip-16895

test: create asok files in a temp directory under $TMPDIR

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

common/options: remove mon_warn_osd_usage_min_max_delta from options.cc too

Signed-off-by: Sage Weil <sage@redhat.com>

test: create asok files in a temp directory under $TMPDIR

to shorten the pathname of unix domain socket created for admin socket,
so it does not exceed the limit of 107 on GNU/Linux:

* ceph-helper.sh: the temp directory is named ${TMPDIR:-/tmp}/ceph-asok.$$
* vstart.sh: the temp directory is named `mktemp -u -d "${TMPDIR:-/tmp}/ceph-asok.XXXXXX"`

Fixes: http://tracker.ceph.com/issues/16895
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16408 from yanghonggang/master

osd: fix pg ref leaks when osd shutdown

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #16449 from smithfarm/wip-obsolete-xio

build/ops: rpm: Drop legacy libxio support

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16453 from liewegas/wip-workloadgen

crush: enforce buckets-before-rules rule

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #16455 from liewegas/wip-20667

osd,mds,mgr: do not dereference null rotating_keys

Reviewed-by: Bassam Tabbara <Bassam.Tabbara@Quantum.com>

Merge pull request #16459 from jdurgin/wip-20704

osd: fix a couple bugs with persisting the missing set when it contains deletes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16461 from liewegas/wip-skewed

mon/PGMap: remove skewed utilizatoin warning

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/PGMap: remove skewed utilizatoin warning

This has a few problems:

1- It does not do it's analysis over CRUSH rule roots/classes, which
means that an innocent user of classes will see skewed usage (bc hdds are
more full than ssds, say)

2- It does not take degraded clusters into account, which means the warning
will appear when a fresh OSD is added.

See http://tracker.ceph.com/issues/20730

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/DaemonState: add per DaemonState lock

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16406 from dmick/wip-mgr-counters

mgr: perf schema fns/change notification and Prometheus plugin

Reviewed-by: Sage Weil <sage@redhat.com>

doc/man/8/ceph: document 'ceph mgr *' commands

Signed-off-by: Sage Weil <sage@redhat.com>

mgr: use rvalue ref for MMgrBeacon metadata

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/cephtool/test.sh: add a few tests

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ServiceMap: fix typo in service status summary

Signed-off-by: Sage Weil <sage@redhat.com>

mon: 'versions' command to show running versions for daemons of all types

Easier!

Signed-off-by: Sage Weil <sage@redhat.com>

mon: factor out count_metadata w/ counter helper

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MgrMonitor: 'mgr metadata', 'mgr versions', 'mgr count-metadata'

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MgrMonitor: store mgr daemon metadata

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16484 from jecluis/wip-20371-qa-suites

qa: flush out monc's dropped msgs on msgr failure injection

qa: flush out monc's dropped msgs on msgr failure injection

We have a few open tickets regarding the mgr being down during suites
involving messenger failure injection. There are a few suspicions that
this may be related with the monclient, but we'll need more logs to
validate those suspicions and, more, to validate we're actually fixing
the issue.

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #16448 from linuxbox2/wip-supriti-ord

rgw_file: properly & |'d flags

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #16475 from amitkumar50/branchAffliation

Add amitkumar50 affiliation to .organizationmap

Reviewed-by: Jos Collin <jcollin@redhat.com>

Add amitkumar50 affiliation to .organizationmap

Signed-off-by: Amit Kumar <amitkuma@redhat.com>

Merge pull request #16430 from yuriw/wip_add_luminous

qa: Added luminous to the mix in schedule_subset.sh

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

common: remove config opt conversion utility

Now that the new options.cc stuff has landed, this is no longer
needed.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #16211 from jcsp/wip-options-jcsp

common: Revamp config option definitions

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16313 from Songweibin/wip-mirror-peer

rbd: make it more understandable when adding peer returns error

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #16462 from wjwithagen/wjw-warning-unused-unsetdumpable

common: prevent unset_dumpable from generating warnings

Reviewed-by: Kefu Chai <kchai@redhat.com>

mon/HealthMonitor: avoid sending unnecessary MMonHealthChecks to leader

If there is no historic warnings and no new warnings is generated,
skip sending MMonHealthChecks(for peon) or updating quorum_checks(for leader).

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #16454 from liewegas/wip-fix-ceph-scrub

qa/tasks/ceph_manager: wait for osd to start after objectstore-tool sequence

Reviewed-by: Kefu Chai <kchai@redhat.com>

mon/HealthMonitor: trigger a proposal if stat updated

leader should always propose if the peon update it with new health stats

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/options: update for latest added

Signed-off-by: John Spray <john.spray@redhat.com>

common: fix erasure_code_dir definition

This was missing its `safe` flag, causing some attempts
to set it during testing to fail.

Signed-off-by: John Spray <john.spray@redhat.com>

test: use config set_val_or_die instead of set_val

...in places that the return code was not being checked.

This fixes cases where an error in the config schema or the
value being passed in would cause weird failures beacuse the
set_val had not taken effect.

Signed-off-by: John Spray <john.spray@redhat.com>

common/options: fix overflowing 64 bit literals

This manifested as a failure in objectstore tool test_fuse.sh

Signed-off-by: John Spray <john.spray@redhat.com>

test: update daemon_config.cc for conf changes

This was only partially updated in previous commits
for --num-clients, --num-open-files.

Also update int validation test to reflect that values
are now 64 bit internally.

Signed-off-by: John Spray <john.spray@redhat.com>

common: update options.cc for latest master

Signed-off-by: John Spray <john.spray@redhat.com>

common: fix Option::is_safe

Signed-off-by: John Spray <john.spray@redhat.com>

common: remove usage of set_daemon_default for now

The code in common_preinit is still there to override
these settings as appropriate.

The set_daemon_default stuff was breaking ceph-conf tests (because
you would get the client-side defaults when asking about an OSD's
settings), and md_config_t isn't properly identifying daemons
using code_env yet.

Ticket to add it back in:
http://tracker.ceph.com/issues/20627

Signed-off-by: John Spray <john.spray@redhat.com>

common: run validator on all defaults

RBD relies on this behaviour to get the int-ized
form for rbd_default_features.

Signed-off-by: John Spray <john.spray@redhat.com>

rgw: set mins on options to avoid unneeded max()

Signed-off-by: John Spray <john.spray@redhat.com>

common: update options.cc for master

Signed-off-by: John Spray <john.spray@redhat.com>

common: move validation in Option and add a test

Signed-off-by: John Spray <john.spray@redhat.com>

test: update md_config_t unit test

Signed-off-by: John Spray <john.spray@redhat.com>

common: revert public_addr setting to a string

This was a string in the old schema, and tests
depended on that -- if we want to change its type
let's do that separately to the infrastructure changes.

Signed-off-by: John Spray <john.spray@redhat.com>

tools: avoid max() calls on rbd config options

These were awkward for typing of the '1' literal vs.
the int64_t settings.  The whole max() thing is also
unnecessary now, if we set proper bounds on the option
definitions.

Signed-off-by: John Spray <john.spray@redhat.com>

common/options: separate "service" from "tags"

Signed-off-by: John Spray <john.spray@redhat.com>

common: enforce min/max/enum on config options

Signed-off-by: John Spray <john.spray@redhat.com>

include/uuid: give uuid_d a < comparator

This is only there so that a variant<> containing
a uuid (amongst other types) can have operator< acting on it.

Signed-off-by: John Spray <john.spray@redhat.com>

common: update options.cc for latest additions

Signed-off-by: John Spray <john.spray@redhat.com>

mon: implement `config set`

Signed-off-by: John Spray <john.spray@redhat.com>

mds: implement `config set`

Signed-off-by: John Spray <john.spray@redhat.com>

osd: implement `config set`

This is a friendlier replacement for injectargs.

Signed-off-by: John Spray <john.spray@redhat.com>

common: pass up error strings from set_val

This is so that we can use it as a replacment
for the ugly injectargs.

Signed-off-by: John Spray <john.spray@redhat.com>

common: implement "config help" to output schema

Signed-off-by: John Spray <john.spray@redhat.com>

rgw: treat config ints as 64 bit

In the new world, all int config values are 64 bit.  When
doing a min() like this, need both sides in the same type.

Signed-off-by: John Spray <john.spray@redhat.com>

common: respect daemon defaults in config options

Signed-off-by: John Spray <john.spray@redhat.com>

common: re-add unsigned config option type

As long as some options are being consumed
via md_config_t:: members, various users
of (unsigned) int values will get compile warnings
when they e.g. compare them with other unsigned values.

Signed-off-by: John Spray <john.spray@redhat.com>

common: reinstate config opt validators

Signed-off-by: John Spray <john.spray@redhat.com>

common: sync up options.cc with latest master

Signed-off-by: John Spray <john.spray@redhat.com>

common/config: hook in new Option config descriptions

The C++ class member fields continue to exist for
settings defined in common/legacy_config_opts.h, but
all the schema information is coming from common/options.cc
now.

The values in md_config_t::values are automatically
copied into the C++ class member fields for legacy config
options as needed.

Signed-off-by: John Spray <john.spray@redhat.com>