mon/ConfigKeyService: wait for quorum

Among other things, this prevents a mgr activation stall if it
happens to request config-keys from a mon that is out of
quorum.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MgrMonitor: show delta

This is mostly because I couldn't debug a weird state where
it kept updating but it appeared the addr was not changing
(always '-').

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MgrMap: implement dump, add to ceph-dencoder

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MgrMonitor: print MgrMap to log on each change

The other services do this.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MgrMonitor: health warn/err if no active mgr

Start warning once mons are luminous; start erroring once
require_luminous is set in osdmap.  Allow a grace period for
mgr to restart or standby to take over before we turn a warning
into an error.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMap: set initial pg timestamp as osdmap modified stamp

Currently the PGMap may register the new pg in multiple places: the
mgr or mon, and the timestamp is when the mon or mgr gets around to
recording it into its PGMap.  Make this deterministic by using the OSDMap
mtime (which is when the PG was *actually* created).

This fixes a problem where the pg create command has one timestamp (from
the mon) and the pgmap on the mgr has another, so an observer will see the
last_scrub_stamp (and other timestamps) change when nothing has actually
happened.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MgrMonitor: do not tick() if not active or leader

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/MgrClient: assume missing MgrMap means no acces to mgr at all

If we get as far as authenticating and have no MgrMap that implies the
mon didn't provide us one (despite our request) and we have no access to
the mgr at all.

Signed-off-by: Sage Weil <sage@redhat.com>

librados: return error from mgr start_command

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/dump_stuck.py: stop making assertions about 'health' report

Health comes from teh mon, while the pg stats come from teh mgr, so they
may be out of sync.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/DaemonServer: log commands to cluster + audit logs

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/MgrStandby: add LogClient

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/DaemonServer: debug on shutdown

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/DaemonServer: whitespace

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/MgrClient: cope with disappearing perf_counters

Perfcounters can go away; deal with it.  This collapses
into a single loop, but it means that the mgr might
stop getting certain counters without explicit
notification.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/upgrade/jewel-x: don't initially start mgr daemons

Signed-off-by: Sage Weil <sage@redhat.com>

qa/releases/luminous.yaml: set up mgr key; start mgr

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/upgrade: don't initially start mgr daemon

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph.py: add 'skip_mgr_daemons' option

For upgrades

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph.py: give everyone mgr caps

Signed-off-by: Sage Weil <sage@redhat.com>

mgr: do not crash if FSMap doesn't have 'addr' metadata

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/MgrClient: do not assert on control-c

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/Session: keep inst, not addr

For parity with log messages from other daemons etc.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr: move MgrSession to its own header

Signed-off-by: Sage Weil <sage@redhat.com>

mon/AuthMonitor: fix mgr mon caps to 'allow profile mgr'

Signed-off-by: Sage Weil <sage@redhat.com>

systemd/ceph-mgr@.service: fix mgr mon cap

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: fix mgr caps

Signed-off-by: Sage Weil <sage@redhat.com>

mon/AuthMonitor: add 'mgr' caps to keyring on upgrade

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonCap: add bootstrap-mgr profile

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-authtool: check 'mgr' section too

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/MgrClient: fix uninitialize variable

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/MgrClient: move lock init into .h

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/Mgr: less noisy about ms_dispatch

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites: run mgr daemon(s)

Everything up upgrade/, which will be slightly tricky.

Signed-off-by: Sage Weil <sage@redhat.com>

test/osd/osd-scrub-repair.sh: cope with mon/mgr asynchrony

Need to wait for mon to reflect pg state as reported by
mgr.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/MgrClient: throttle connection attempts

Max of 1 per second.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/cephtool/test.sh: test_auth_profiles must include mgr

For 'pg ...' commands

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonCap: pass entity type (mgr, mon) to is_capable

This lets us define different meanings for profiles on
the mon and mgr.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MgrMonitor: enforce mgr caps

Require 'allow x' for mgr beacon.  Verify fsid.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr: include cluster fsid in MMgrBeacon

Prevent ceph-mgr from talking to the wrong cluster!

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonCap: expand 'profile mgr' caps

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/DaemonServer: enforce caps

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/DaemonServer: attach MgrSession to connections

Reuse MonCap; it is suitably generic.

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: debug mgrc

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: give client.admin mgr 'allow *'

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: give osd mgr = 'allow profile osd'

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: pass ceph.conf to ceph-mgr

Signed-off-by: Sage Weil <sage@redhat.com>

test/osd/osd-*.sh: run mgr daemon for tests

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/ceph-helpers: start and stop mgr daemons

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-mgr: threads after fork

MgrStandby contains an Objecter with a ceph_timer and
associated thread.  Create it after we fork.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: forward mgr commands; cap them

Put the completions on a finisher to avoid deadlock.
(MgrClient lock is inside mon_lock due to start_command;
completion cannot retake mon_lock while holding MgrClient's
internal lock.)

The mon has a cap on the nubmer of client message bytes it hold
in memory.  We do not want to eat those up and DOS the mon
because the mgr is not available and all mgr commands are
blocked.

Define the limit as a fraction of mon_client_bytes to avoid user
misconfiguration.

Return EAGAIN if we hit the limit.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: add a finisher

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MgrMonitor: feed our MgrClient MgrMaps

We don't have a MonClient so we don't subscribe to these in the
usual way.  This will do.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: add a MgrClient

Signed-off-by: Sage Weil <sage@redhat.com>

mon: generate tickets for mgr service

This is kludgey because the mon is 'special' when it comes to
auth, but it works!

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/MgrClient: remove unused cond signal helpers

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/MgrClient: don't block command submission if no session yet

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/MgrClient: resend pending commands on reconnect

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/DaemonServer: implement 'pg [scrub|deep-scrub|repair] ...'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/DaemonServer: use process_pg_map_command helper for most things

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/DaemonServer: use stringstream ds in local scope

Signed-off-by: Sage Weil <sage@redhat.com>

mgr: don't crash on bad auth msg from client

Signed-off-by: Sage Weil <sage@redhat.com>

osd: accept MOSDScrub messages from ceph-mgr

Signed-off-by: Sage Weil <sage@redhat.com>

msg/Connection: implement peer_is_mgr()

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/DaemonServer: set messenger policy; set throttles

Put clients, osds, mons, and mdss in different buckets.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: move most PGMonitor commands into PGMap.{cc} helper

We don't need private access to PGMap members, so a simple
function will do.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: move dump_stuck_pg_stats from PGMonitor -> PGMap

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMap: constify a few things

Signed-off-by: Sage Weil <sage@redhat.com>

pybind/ceph_argparse: send MGR-flagged commands to ceph-mgr

Signed-off-by: Sage Weil <sage@redhat.com>

mgr: add generic Formatter for handle_command

Signed-off-by: Sage Weil <sage@redhat.com>

mgr: include MgrCommands.h in mgr_commands

Signed-off-by: Sage Weil <sage@redhat.com>

mon,mgr: move "pg ..." commands to MgrCommands, but keep #including on mon

These will become mgr commands.  Move them untouched to
MgrCommands.h, but for now include them in the mon_commands with
the new MGR flag on them.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: hide the MGR flag on mon commands until require_luminous is set

Signed-off-by: Sage Weil <sage@redhat.com>

mon: add a MGR flag to mon commands

The flag will indicate that a command should be sent to ceph-mgr
instead of ceph-mon (rados_mgr_command instead of
rados_mon_command).

For legacy clients, we'll have to make the mon forward the message
on behalf of the client.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #14122 from hrchu/rgwadmin-doc-quota

doc: correct the quota section

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #14078 from tchaikov/wip-19258

mgr: shutdown py_modules in Mgr::shutdown()

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #13337 from ifed01/wip-bluestore-minimiza-blobs2

os/bluestore: optimize blob usage when doing appends/overwrites

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14223 from tchaikov/wip-rbd-nbd-ftbfs

rbd-nbd: s/cpp_error/cpp_strerror/ to fix FTBFS

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Pan Liu <liupan1111@gmail.com>

mgr: shutdown py_modules in Mgr::shutdown()

Fixes: http://tracker.ceph.com/issues/19258
Signed-off-by: Kefu Chai <kchai@redhat.com>

rbd-nbd: s/cpp_error/cpp_strerror/ to fix FTBFS

the build failure was introduced by ff4dcf0

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14079 from liupan1111/wip-support-signal-handler

rbd-nbd: support signal handle for SIGHUP, SIGINT and SIGTERM.

Reviewed-by: Dongsheng Yang <dongsheng.yang@easystack.cn>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #14184 from SUSE/wip-mgr-rest-timezone

pybind/mgr/rest: don't set timezone to Chicago

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14199 from andihit/doc-update-default-osd-op-thread-timeout

doc: change osd_op_thread_timeout default value to 15

Reviewed-by: Sage Weil <sage@redhat.com>

doc: rgw: correct the quota section

Add the missing option and fix typo.

Fixes: http://tracker.ceph.com/issues/19397
Signed-off-by: Chu, Hua-Rong <hrchu@cht.com.tw>

Merge pull request #14203 from cbodley/wip-rgw-slo-auth-identity

rgw: fix build of conflict after auth rework

Reviewed-by: Sage Weil <sage@redhat.com>

rgw: fix build of conflict after auth rework

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #13764 from guihecheng/rgwfs-errcode

rgw_file: fix non-posix errcode EINVAL to ENAMETOOLONG

Merge pull request #14108 from sven15/doc-fix-pools

doc: typo in hit_set_search_last_n

Reviewed-by: John Wilkins <jowilkin@redhat.com>

Merge pull request #14128 from yehudasa/wip-19367

cls/user: cls_user_bucket backward compatibility

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13987 from shinobu-x/wp-sk-clean-up-timeindexlistctx-20170316

cls/timeindex: clean up cls_timeindex_client.h|cc

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14009 from yuyuyu101/wip-objecter-ping

msg/async/AsyncConnection: keepalive objecter ping connection to avoid timeout

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14133 from runsisi/wip-pg-state-cleanup

osd/PG: some minor cleanups

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #14089 from ceph/wip-rgw-doc-changing-master

doc/rgw: instructions for changing multisite master zone

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #14090 from cbodley/wip-rgw-test-delete-notempty

test/rgw: test_bucket_delete_notempty in test_multi.py

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #13561 from cbodley/wip-19027

rgw: allow system users to read SLO parts

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

doc: change osd_op_thread_timeout default value to 15

Signed-off-by: Andreas Gerstmayr <andreas.gerstmayr@catalysts.cc>

Merge pull request #13984 from liewegas/wip-osdmap-remap

mon,osd: explicitly remap some pgs

Reviewed-by: Josh Durgin <jduring@redhat.com>

Merge pull request #13687 from Liuchang0812/wip-19106-eliminate-unsafe-option

common/config: eliminate config_t::set_val unsafe option

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13855 from yuyuyu101/wip-bluestore-rm-range

kv/RocksDBStore: implement rm_range_keys operator interface and test

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13879 from liewegas/wip-19076

osd/ReplicatedBackend: clear pull source once we are done with it

Reviewed-by: Sage Weil <sage@redhat.com>