Merge pull request #15387 from jecluis/wip-20123

mon/PGMap: show %used in formatted output

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15508 from tchaikov/wip-ceph.in-cleanup

ceph.in: drop the compatiiblity to handle non json commands

Merge pull request #15560 from dmick/master

mgr/DaemonServer.cc: log daemon type string as well as id

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15562 from tchaikov/wip-no-assert.h

common,osdc: remove atomic_t completely

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15567 from Fabian-Gruenbichler/pull/logrotatesync

debian: sync logrotate packaging with downstream

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15578 from jcsp/wip-mgr-cleanup

mgr: cleanup, stop clients sending in perf counters

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15582 from liewegas/wip-20226

osd: take PGRef for recovery sleep wakeup event

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15405 from liewegas/wip-rest-test

pybind/mgr/restful: improve cert handling; work with vstart

Reviewed-by: Boris Ranto <branto@redhat.com>

debian/control: Build-Depends on python-pecan

This is needed for make check (so that restful mgr module can load).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15557 from liewegas/wip-mgr-respawn

mgr/MgrStandby: respawn when deactivated

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15301 from shashalu/rgw-ssl

rgw: add "rgw_verify_ssl" config

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15257 from shashalu/fix-rgw-admin-retcode

rgw: fix radosgw-admin retcode

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14950 from fangyuxiangGL/set-obj-accounted-size

rgw: set object accounted size correctly

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/Mgr: fix deadlock in load_config

Signed-off-by: Sage Weil <sage@redhat.com>

pybind/mgr/dashboard: get_localized_config for server_{addr,port}

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/rest/test_mgr_rest_api.py: improvements

Signed-off-by: Sage Weil <sage@redhat.com>

pybind/mgr/restful: store each key+pass in a separate key

Signed-off-by: Sage Weil <sage@redhat.com>

pybind/mgr/restful: prefix commands with 'restful'

And use - instead of _, following the ceph convention.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr: add get_config_prefix

Fetch a dict of all config options with a given prefix.

Signed-off-by: Sage Weil <sage@redhat.com>

pybind/mgr/restful: localize key/crt keys

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/PyState: add get_mgr_id() to module interface

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/PyModules: prefix by mgr/, not mgr/$id/

If modules want per-instance state, they can include
the mgr id in their portion of the key name.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/ConfigKeyService: more useful status message

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: start up mgr restful API

We try to stagger the ports that mgr modules bind to so that
concurrent vstart instances can run with consecutive ports but
the services will not collide.  Yes, this is awkward.

Signed-off-by: Sage Weil <sage@redhat.com>

pybind/mgr/restful: all crt/key or filename in config-key

Signed-off-by: Sage Weil <sage@redhat.com>

ceph.spec.in: add python-pecan as BuildDepends too

For install-deps.sh's benefit.

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: load 'restful' (not 'rest') mgr module

Signed-off-by: Sage Weil <sage@redhat.com>

test/vstart_wrapper.sh: fix MGR_PYTHON_PATH

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: start mgr dashboard, echo URL(s)

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15507 from zhangsw/fix-rgw-logtrim-bug

rgw: datalog trim and mdlog trim handles the result returned by osd incorrectly.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15102 from majianpeng/blockdevice-pmem

os/bluestore/BlockDevice: support pmem device as bluestore backend

Reviewed-by: Sage Weil <sage@redhat.com>

osd: take PGRef for recovery sleep wakeup event

The raw PG* may not remain valid when the event fires.

Fixes: http://tracker.ceph.com/issues/20226
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15577 from jcsp/wip-dashboard-cleanup

mgr: dashboard code cleanup

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: remove confusing SyncObject stuff

This was a hangover from when these wrapper classes
were borrowed from Calamari, which used these
versions/equality functions to work out when
to go fetch data from the ceph cluster.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr/dashboard: fix server view on funny version format

Signed-off-by: John Spray <john.spray@redhat.com>

mgr/dashboard: fix error handling in remote_view_cache

Signed-off-by: John Spray <john.spray@redhat.com>

mgr/dashboard: keep a global librados instance

...so that classes that need one aren't creating
their own all the time.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #15566 from baruza/wip-doc-20199

doc: Change the default values of some OSD options

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15485 from ceph/wip-hdd-throttle-cost

common/config_opts.h: Lower HDD throttle cost.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15203 from Liuchang0812/wip-kill-broken-links

doc: kill some broken links

Reviewed-by: Kefu Chai <kchai@redhat.com>

debian: sync logrotate packaging with downstream

both Ubuntu and Debian put the logrotate script into
ceph-common, to ensure that radosgw logs are rotated as
well.

to prevent duplicate logrotate scripts handling the same log
files, and to minimize the delta between upstream and
downstream packaging, sync this change back upstream.

Fixes: http://tracker.ceph.com/issues/19938
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>

mgr: drop PyModules::lock around module shutdown calls

This deadlocked if the module's shutdown call was trying
to take a pythonland lock that was held by a thread
that was in turn trying to call into C++ land and take
PyModules::lock.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: don't let TYPE_CLIENT entities send stats

Anyone instantiating librados instances ended up
appearing as a weird entry in DaemonStateIndex.

Come back to this at the point we want e.g. stats
from RGWs or NFS gateways.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: don't log successful ceph_config_get as err

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #15575 from liewegas/wip-pgnls-note

PendingReleaseNotes: notes on whiteouts vs pgnls

Merge pull request #15309 from ifed01/wip-bluestore-boost-statfs

os/bluestore: keep statfs replica in RAM to avoid expensive KV retrieval

Reviewed-by: Sage Weil <sage@redhat.com>

PendingReleaseNotes: notes on whiteouts vs pgnls

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15532 from dillaman/wip-rbd-mirror-stress

rbd-mirror: lock loss during sync should wait for in-flight copies

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #15518 from dillaman/wip-rbd-missing-default-pool

rbd: removed hardcoded default pool

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

os/bluestore: introduce a debug switch to bypass kv update.

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: introduce debug parameter to bypass bdev verification

E.g. this allows to backup WAL/DB volumes after preconditioning and quickly rollback to that state when needed.

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: keep statfs replica in memory to avoid expensive KV access

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: move volatile_statfs struct out of TransContext

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #15433 from wjwithagen/wip-wjw-prettytable

ceph-disk: add setting for external py-modules for tox-testing

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: Change the default values of some OSD options

These values were changed to reflect this change a3f9cba

Fixes: http://tracker.ceph.com/issues/20199
Signed-off-by: Bara Ancincova <bara@redhat.com>

ceph-disk: use system modules if needed

 - prettytable usage was introduced in:
https://github.com/ceph/ceph/commit/3fa8bb1
   It is in the install-deps.sh file to be installed, but
   it is not per default installed in the testenvironment
   So allow tox tests to use "external" modules

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

tools/rbd_mirror: do not include ununsed header

Signed-off-by: Kefu Chai <kchai@redhat.com>

client: s/atomic_t/std::atomic<>

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake,debian,rpm: remove atomic_t completely

Signed-off-by: Kefu Chai <kchai@redhat.com>

msg,client,os,journal: do not include unused header

Signed-off-by: Kefu Chai <kchai@redhat.com>

msg/async: s/atomic_t/atomic<>/

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: rgw_rados: s/atomic_t/atomic<>/

Signed-off-by: Kefu Chai <kchai@redhat.com>

msg,common: do not include assert.h everywhere to clobber <assert.h>

* reverts c93daaf92b643a06a822558973d377733dbd9d85
* add "#include "include/assert.h" to where <cassert> or <assert.h> are
  included, or avoid including them if possible.

Signed-off-by: Kefu Chai <kchai@redhat.com>

osdc/Objecter: s/atomic_t/std::atomic<>/

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/ceph_context: s/atomic_t/std::atomic<>/

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/DaemonServer.cc: log daemon type string as well as id

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #14866 from chardan/jfw-wip-halflife_atomic_t-varan

common,test: migrate atomic_t to std::atomic

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15520 from liewegas/wip-max-object-size

osd: reduce rados_max_object_size from 100 GB -> 128 MB

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

osd: reduce rados_max_object_size from 100 GB -> 128 MB

I'm not sure why we had this so huge!  128 MB is a pretty reasonable size.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15230 from chardan/jfw-wip-halflife_atomic_t-xio

xio: migrate atomic_t to std::atomic<>

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15516 from javacruft/i386-cmake-fixes

cmake: misc fixes for build on i386

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15525 from wjwithagen/wip-wjw-clang-testosdmap

test: osd/TestOSDMap.cc: fix Clang complain about promotion

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/MgrStandby: respawn when deactivated

- It is ugly to unwind all of the Mgr state so that we can reactivate
  later.
- It is perhaps impossible to do shut down the python state reliably.
- Respawning provides a clean state and is reliable.

This mostly just copies MDSServer::respawn().

Fixes: http://tracker.ceph.com/issues/19595
Fixes: http://tracker.ceph.com/issues/19549
Signed-off-by: Sage Weil <sage@redhat.com>

rbd: removed hardcoded default rbd pool name

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-nbd: removed default rbd pool name

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-replay: removed default rbd pool name

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

pybind/rbd: module should depend on librbd not rbd CLI

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

common: new 'rbd_default_pool' configuration option

The 'rbd' pool will no longer be automatically created. Allow
the user to specify a custom default RBD pool name.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: narrowed scope of exclusive lock op tracking

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: partial revert of 649b29d

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #15498 from adamemerson/wip-fixation

test: Fix narrowing conversion
rgw: Support certain archaic and antiquated distributions

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15436 from liewegas/wip-versions

mon: {mon,osd,mds} {versions,count-metadata}

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15501 from liewegas/wip-blocked-is-err

mon/PGMap: call blocked requests ERR not WARN

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

mon/PGMap: call requests blocked for 128x as long ERR not WARN

- rename the option (max -> warn)
- add an err_..._ratio multiplier
- switch to HEALTH_ERR once requests are blocked long enough
- make the error ratio high (default is 32*128s -> about an hour) so that
we don't trigger on a heavily loaded cluster.

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: Support certain archaic and antiquated distributions

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

test: Fix narrowing conversion

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

os/bluestore/BlockDevice:support pmem device as bluestore backend.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #15531 from liewegas/wip-osd-pglog

osd: rename osd -> osd_pglog; include pglog-related bufferlists

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

osd/osd_types: put pg_log_entry_t::extra_reqids in osd_pglog mempool

Signed-off-by: Sage Weil <sage@redhat.com>

osd: put pg_log_entry_t::snaps in osd_pglog mempool

Signed-off-by: Sage Weil <sage@redhat.com>

osd: map ObjectModDesc to osd_pglog pool

This can consume significant space for EC pools.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: move pglog to osd_pglog mempool

This actually everything in the osd mempool, so it's basically a rename.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15413 from tchaikov/wip-rgw-boost-cleanup

rgw: drop asio/{yield,coroutine}.hpp replacements

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15434 from yuyuyu101/wip-lz4

compressor: add LZ4 support

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15512 from jcsp/wip-mgrmap-epoch

mon: don't prefix mgr summary with epoch number

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15437 from dachary/wip-bluestore

ceph-disk: add --filestore argument, default to --bluestore

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15547 from wjwithagen/wip-wjw-freebsd-get_device_by_fd

common/blkdev.cc: propagate get_device_by_fd to different OSes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15435 from liewegas/wip-bluestore-warning

os/bluestore: fix warning

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #15371 from liewegas/wip-mon-client-features

mon: track features from connect clients, and use it to gate set-require-min-compat-client

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

common/blkdev.cc: propagate get_device_by_fd to different OSes

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>