mon: do not dereference empty mgr_commands

This may be empty before the initial quorum commits the static command
list.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16408 from yanghonggang/master

osd: fix pg ref leaks when osd shutdown

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #16449 from smithfarm/wip-obsolete-xio

build/ops: rpm: Drop legacy libxio support

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16453 from liewegas/wip-workloadgen

crush: enforce buckets-before-rules rule

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #16455 from liewegas/wip-20667

osd,mds,mgr: do not dereference null rotating_keys

Reviewed-by: Bassam Tabbara <Bassam.Tabbara@Quantum.com>

Merge pull request #16459 from jdurgin/wip-20704

osd: fix a couple bugs with persisting the missing set when it contains deletes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16461 from liewegas/wip-skewed

mon/PGMap: remove skewed utilizatoin warning

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/PGMap: remove skewed utilizatoin warning

This has a few problems:

1- It does not do it's analysis over CRUSH rule roots/classes, which
means that an innocent user of classes will see skewed usage (bc hdds are
more full than ssds, say)

2- It does not take degraded clusters into account, which means the warning
will appear when a fresh OSD is added.

See http://tracker.ceph.com/issues/20730

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16406 from dmick/wip-mgr-counters

mgr: perf schema fns/change notification and Prometheus plugin

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16484 from jecluis/wip-20371-qa-suites

qa: flush out monc's dropped msgs on msgr failure injection

qa: flush out monc's dropped msgs on msgr failure injection

We have a few open tickets regarding the mgr being down during suites
involving messenger failure injection. There are a few suspicions that
this may be related with the monclient, but we'll need more logs to
validate those suspicions and, more, to validate we're actually fixing
the issue.

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #16448 from linuxbox2/wip-supriti-ord

rgw_file: properly & |'d flags

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #16475 from amitkumar50/branchAffliation

Add amitkumar50 affiliation to .organizationmap

Reviewed-by: Jos Collin <jcollin@redhat.com>

Add amitkumar50 affiliation to .organizationmap

Signed-off-by: Amit Kumar <amitkuma@redhat.com>

Merge pull request #16430 from yuriw/wip_add_luminous

qa: Added luminous to the mix in schedule_subset.sh

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #16211 from jcsp/wip-options-jcsp

common: Revamp config option definitions

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16313 from Songweibin/wip-mirror-peer

rbd: make it more understandable when adding peer returns error

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #16462 from wjwithagen/wjw-warning-unused-unsetdumpable

common: prevent unset_dumpable from generating warnings

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16454 from liewegas/wip-fix-ceph-scrub

qa/tasks/ceph_manager: wait for osd to start after objectstore-tool sequence

Reviewed-by: Kefu Chai <kchai@redhat.com>

common/options: update for latest added

Signed-off-by: John Spray <john.spray@redhat.com>

common: fix erasure_code_dir definition

This was missing its `safe` flag, causing some attempts
to set it during testing to fail.

Signed-off-by: John Spray <john.spray@redhat.com>

test: use config set_val_or_die instead of set_val

...in places that the return code was not being checked.

This fixes cases where an error in the config schema or the
value being passed in would cause weird failures beacuse the
set_val had not taken effect.

Signed-off-by: John Spray <john.spray@redhat.com>

common/options: fix overflowing 64 bit literals

This manifested as a failure in objectstore tool test_fuse.sh

Signed-off-by: John Spray <john.spray@redhat.com>

test: update daemon_config.cc for conf changes

This was only partially updated in previous commits
for --num-clients, --num-open-files.

Also update int validation test to reflect that values
are now 64 bit internally.

Signed-off-by: John Spray <john.spray@redhat.com>

common: update options.cc for latest master

Signed-off-by: John Spray <john.spray@redhat.com>

common: fix Option::is_safe

Signed-off-by: John Spray <john.spray@redhat.com>

common: remove usage of set_daemon_default for now

The code in common_preinit is still there to override
these settings as appropriate.

The set_daemon_default stuff was breaking ceph-conf tests (because
you would get the client-side defaults when asking about an OSD's
settings), and md_config_t isn't properly identifying daemons
using code_env yet.

Ticket to add it back in:
http://tracker.ceph.com/issues/20627

Signed-off-by: John Spray <john.spray@redhat.com>

common: run validator on all defaults

RBD relies on this behaviour to get the int-ized
form for rbd_default_features.

Signed-off-by: John Spray <john.spray@redhat.com>

rgw: set mins on options to avoid unneeded max()

Signed-off-by: John Spray <john.spray@redhat.com>

common: update options.cc for master

Signed-off-by: John Spray <john.spray@redhat.com>

common: move validation in Option and add a test

Signed-off-by: John Spray <john.spray@redhat.com>

test: update md_config_t unit test

Signed-off-by: John Spray <john.spray@redhat.com>

common: revert public_addr setting to a string

This was a string in the old schema, and tests
depended on that -- if we want to change its type
let's do that separately to the infrastructure changes.

Signed-off-by: John Spray <john.spray@redhat.com>

tools: avoid max() calls on rbd config options

These were awkward for typing of the '1' literal vs.
the int64_t settings.  The whole max() thing is also
unnecessary now, if we set proper bounds on the option
definitions.

Signed-off-by: John Spray <john.spray@redhat.com>

common/options: separate "service" from "tags"

Signed-off-by: John Spray <john.spray@redhat.com>

common: enforce min/max/enum on config options

Signed-off-by: John Spray <john.spray@redhat.com>

include/uuid: give uuid_d a < comparator

This is only there so that a variant<> containing
a uuid (amongst other types) can have operator< acting on it.

Signed-off-by: John Spray <john.spray@redhat.com>

common: update options.cc for latest additions

Signed-off-by: John Spray <john.spray@redhat.com>

mon: implement `config set`

Signed-off-by: John Spray <john.spray@redhat.com>

mds: implement `config set`

Signed-off-by: John Spray <john.spray@redhat.com>

osd: implement `config set`

This is a friendlier replacement for injectargs.

Signed-off-by: John Spray <john.spray@redhat.com>

common: pass up error strings from set_val

This is so that we can use it as a replacment
for the ugly injectargs.

Signed-off-by: John Spray <john.spray@redhat.com>

common: implement "config help" to output schema

Signed-off-by: John Spray <john.spray@redhat.com>

rgw: treat config ints as 64 bit

In the new world, all int config values are 64 bit.  When
doing a min() like this, need both sides in the same type.

Signed-off-by: John Spray <john.spray@redhat.com>

common: respect daemon defaults in config options

Signed-off-by: John Spray <john.spray@redhat.com>

common: re-add unsigned config option type

As long as some options are being consumed
via md_config_t:: members, various users
of (unsigned) int values will get compile warnings
when they e.g. compare them with other unsigned values.

Signed-off-by: John Spray <john.spray@redhat.com>

common: reinstate config opt validators

Signed-off-by: John Spray <john.spray@redhat.com>

common: sync up options.cc with latest master

Signed-off-by: John Spray <john.spray@redhat.com>

common/config: hook in new Option config descriptions

The C++ class member fields continue to exist for
settings defined in common/legacy_config_opts.h, but
all the schema information is coming from common/options.cc
now.

The values in md_config_t::values are automatically
copied into the C++ class member fields for legacy config
options as needed.

Signed-off-by: John Spray <john.spray@redhat.com>

log/SubsystemMap: return a size_t from get_num

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: tighten initialization of keyring setting

We should not proceed if setting this fails.  Also
the meta=false setting is not what you want when
calling into set_val after initialization, I'm not sure
how that ever worked!

Signed-off-by: John Spray <john.spray@redhat.com>

common: separate config opts definitions from subsys definitions

Previously these were all in one header and inclusions of it
got really verbose from everyone having to define SUBSYS and OPTION
macros even though they only wanted to pick out one or the other.

Also, this separates the subsys.h stuff (staying) from the legacy
config opt definitions (transitional, will go).

Signed-off-by: John Spray <john.spray@redhat.com>

common: remove ::validate definitions

These will be replaced by validate methods
on Option subclasses that need them.  The code
that was in these files moved to options.[h|cc]

Signed-off-by: John Spray <john.spray@redhat.com>

common: hook in 'safe' flag to new config options

It's a poor substitute for real a concurrency solution
but for the moment carry it forward so that the options
structure can replace the list of config_option
in md_config_t.

Signed-off-by: John Spray <john.spray@redhat.com>

common: s/config_opts/legacy_config_opts/

Name change reflects that newly added config opts
don't need to go here.

Signed-off-by: John Spray <john.spray@redhat.com>

common: use a vector instead of array in options.cc

Avoids retro 80s style blank entry at the end :-)

Signed-off-by: John Spray <john.spray@redhat.com>

common: rip out option code generation bit

We can have a legacy (static field) config object
that includes fields from config_opts.h, and
then switch to using dynamic get() for newly
added options, so that we don't need to do
code generation for the new config infrastructure.

Signed-off-by: John Spray <john.spray@redhat.com>

g++ ../src/convert.cc -o cv -I../src -I.
./cv > /tmp/foo
...and copy that into options.cc
bin/generate_option_header > ../src/common/config_fields.h

Signed-off-by: Sage Weil <sage@redhat.com>

common/options: more constness

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/options: specialize on non bool integeral types

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/options: new options infrastructure

Define schema for config options.  Helper to generate a header fragment
to declare the types.

Unlike the old config_opts.h approach, we will not intialize values in
the header.  This avoids a recompile if there is a change and also allows
us to specify different defaults and do parsing and validation at runtime.
Instead, we'll intialize values in the constructure of the containing
class.

Signed-off-by: Sage Weil <sage@redhat.com>

common: remove max_open_files config option

The only user is init-ceph, and it can get at it via ceph-conf without
the option being declared here.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-syn: parse --num-client locally; remove global option

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16028 from jcsp/wip-mgr-commands

mon: load mgr commands at runtime

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16415 from xiexingguo/wip-health-mon

mon: a few health fixes

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Revert "Test-enable XIO messenger and IB/RDMA support"

This reverts commit 71cbc832ded17212d840afa709700a2ca1a498b9.

Signed-off-by: Nathan Cutler <ncutler@suse.com>
Conflicts:
ceph.spec.in (trivial resolution)

Merge pull request #16409 from liupan1111/wip-restructure-bs

os/bluestore: move aio.h/cc from fs dir to bluestore dir

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16328 from neha-ojha/recovery_sleep_default

osd: Add recovery sleep configuration option for HDDs and SSDs

Reviewed-by: Sage Weil <sage@redhat.com>

common: prevent unset_dumpable from generating warnings

 - Let the struct at least have a CTR

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #16470 from tchaikov/wip-watch-audit-channel

qa/workunits/cephtool/test.sh: fix test to watch audit channel

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

qa/workunits/cephtool/test.sh: fix test to watch audit channel

Signed-off-by: Sage Weil <sage@redhat.com>

doc/mgr: add Prometheus plugin docs

Signed-off-by: Dan Mick <dan.mick@redhat.com>

mgr/index.rst: clarify which toplevel items are plugins

Signed-off-by: Dan Mick <dan.mick@redhat.com>

doc/mgr/zabbix.rst: fix subheadings, add subsubheadings

(otherwise subheadings appear as top-level items in the TOC, because
they're marked just like the title)

Signed-off-by: Dan Mick <dan.mick@redhat.com>

qa: remove workloadgen test

The CRUSH rule creation is busted (rules and buckets out of order), but
after I fix that it doesn't seem to run right anyway.  Remove it.
We get the mon thrasher coverage from rados/monthrash already; I don't
think this is adding meaningful coverage for the amount of effort it takes
to maintain.

Signed-off-by: Sage Weil <sage@redhat.com>

crush/CrushCompiler: enforce buckets before rules rule

Signed-off-by: Sage Weil <sage@redhat.com>

osd: bail out if no map yet

If we get pings before finishing our startup we may not yet have an
osdmap.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/reg11184: import run

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PGLog: force write of may_include_deletes after split

When a collection is split this needs to be persisted again. Normally
this is only persisted when the missing set is rebuilt during a new
interval when it previous did not include deletes, but during split we
keep the in-memory missing set may_include_deletes flag, but do not
rebuild the missing set.

Fixes: http://tracker.ceph.com/issues/20704
Signed-off-by: Josh Durgin <jdurgin@redhat.com>

ceph_objectstore_tool: always write may_include_deletes if it's set

This prevents us from importing a missing set without also setting the
may_include_deletes_in_missing omap value if appropriate.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

msg: restore peer addr checks

This was dropped in bf4938567943c80345966f9c5a3bdc75a913175b but should
not have been.  Since we are advertising the addr and not the bind
addr there is no reason to disable this check.

Signed-off-by: Sage Weil <sage@redhat.com>

osd,mds,mgr: do not dereference null rotating_keys

Immediately after we bind to a port, but before we have set up our
auth infrastructure, we may get incoming connections.  Deny them.  Since
we are not yet advertising ourselves these are peers trying to connect
to old instances of daemons, not us.

This triggers now because of bf4938567943c80345966f9c5a3bdc75a913175b.
Previously, the peer would see we were a different addr and drop the
connection.  Now, it continues.

Fixes: http://tracker.ceph.com/issues/20667
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16429 from liewegas/wip-jewel-x

qa/suites/upgrade/jewel-x: misc fixes for new health checks

Merge pull request #16374 from majianpeng/bluestore-double-count-kv-queue-lat

os/bluestore/BlueStore: Avoid double counting state_kv_queued_lat.

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #16423 from liewegas/wip-ls

mon: '* list' -> '* ls'

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16434 from badone/wip-fix-deep-age-reporting

mon: Fix deep_age copy paste error

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16444 from tchaikov/wip-test-osd-stat

qa/workunits/cephtool/test.sh: "ceph osd stat" output changed, update accordingly

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

qa/tasks/ceph_manager: wait for osd to start after objectstore-tool sequence

Fixes: http://tracker.ceph.com/issues/20705
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #14287 from oritwas/wip-rgw-usage

rgw: usage

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #15673 from yehudasa/wip-20107

cls/refcount: store and use list of retired tags

Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #16421 from tchaikov/wip-19939

messages/: always set header.version in encode_payload()

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #16418 from theanalyst/doc-rgw-tenant-names

doc: rgw clarify limitations when creating tenant names

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Pritha Srivastava <prsrivas@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #16443 from wjwithagen/bug-wjw-qa-test-reorder

cephtool/test.sh: Only delete a test pool when no longer needed.

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #16451 from tchaikov/wip-qa-reg11184

qa/tasks/reg11184: use literal 'foo' instead pool_name

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/tasks/reg11184: use literal 'foo' instead pool_name

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16412 from wido/zabbix-release-notes

doc: Add Zabbix ceph-mgr plugin to PendingReleaseNotes

Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw_file: properly & |'d flags

Found by "Supriti Singh" <Supriti.Singh@suse.com>.

Fixes http://tracker.ceph.com/issues/20663

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

release notes: Add Zabbix ceph-mgr module to PendingReleaseNotes

Signed-off-by: Wido den Hollander <wido@42on.com>

doc: rgw clarify limitations when creating tenant names

We only allow alphanumeric and underscore characters in tenant names
according to the validation in `RGWHandler_REST::validate_tenant_name`

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

qa/workunits/cephtool/test.sh: "ceph osd stat" output changed, update test accordingly

Signed-off-by: Kefu Chai <kchai@redhat.com>
Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>