Merge pull request #15323 from fangyuxiangGL/compress_bug_fix

rgw: get wrong content when download object with specific range with compression

Reviewed-by: Adam Kupczyk <akupczyk@mirantis.com>
Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #15610 from wjwithagen/wip-wjw-freebsd-py-pecan

core: Install Pecan for FreeBSD

Merge pull request #15574 from dillaman/wip-20223

pybind/rbd: OSError should be picklable

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #15576 from dachary/wip-ceph-disk

ceph-disk: sensible default for block.db

Reviewed-by: Sage Weil <sage@redhat.com>

core: Install Pecan for FreeBSD

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #15609 from stiwari/wip-19879

Merge pull request from stiwari/wip-19879

Merge pull request #12851 from zhangsw/fix-rgw-bucketcheck-bug

rgw: bucket index check in radosgw-admin removes valid index.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #15606 from stiwari/fix_1

Documentation Fixes for http://tracker.ceph.com/issues/19879

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #15588 from liewegas/wip-20216

pybind/mgr/dashboard: monkeypatch os.exit to stop cherrypy from taking down mgr

Reviewed-by: John Spray <jspray@redhat.com>

Merge pull request #15550 from jcsp/wip-doc-cephfs-objsize

doc/cephfs: mention RADOS object size limit

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

ceph-disk: sensible default for block.db

If bluestore_block_db_size is unset or zero, fallback to a sensible
default which is block_size / 100. With a minimum of 1GB.

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #13395 from dang/wip-quota-cache-doc

Document RGW quota cache options

Merge pull request #15568 from guihecheng/rgw_file-cleanup-names

rgw: rgw_file cleanup names

Merge pull request #15541 from guihecheng/rgw_file-cleanup

rgw_file: replace raw fs->fh_lru.unref with predefined fs->unref

Merge pull request #15558 from Linbing/wip-osd-optimize-send-failues

osd: when osd in not in failure_pending, we don't need to get osd inst from osdmap.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15223 from badone/wip-ceph_test_admin_socket_output

tests: Add integration tests for admin socket output

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15573 from liewegas/wip-20208

mon/PGMap: fix osd_epoch update when removing osd_stat

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Loic Dachary <loic@dachary.org>

Merge pull request #15594 from idryomov/wip-krbd-rados-ls

qa: update krbd_data_pool.sh to match the new rados ls behavior

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #15600 from ceph/luminous

12.0.3

Merge pull request #15180 from theanalyst/doc/rgw-dev-s3tests

doc: dev improve the s3tests doc to reflect current scripts

Merge pull request #15570 from joscollin/wip-cleanup-testmsgr-warning

test/msgr: silence warnings from -Wsign-compare

Reviewed-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #15100 from kestrels/wip-crc32c-fastzero2

crc32c: Add ppc64le fast zero optimized assembly.

Reviewed-By: Piotr Dałek <piotr.dalek@corp.ovh.com>

Merge pull request #15387 from jecluis/wip-20123

mon/PGMap: show %used in formatted output

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15508 from tchaikov/wip-ceph.in-cleanup

ceph.in: drop the compatiiblity to handle non json commands

Merge pull request #15560 from dmick/master

mgr/DaemonServer.cc: log daemon type string as well as id

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15562 from tchaikov/wip-no-assert.h

common,osdc: remove atomic_t completely

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15567 from Fabian-Gruenbichler/pull/logrotatesync

debian: sync logrotate packaging with downstream

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15578 from jcsp/wip-mgr-cleanup

mgr: cleanup, stop clients sending in perf counters

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15582 from liewegas/wip-20226

osd: take PGRef for recovery sleep wakeup event

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

pybind/mgr/dashboard: monkeypatch os.exit to stop cherrypy from taking down mgr

Fixes: http://tracker.ceph.com/issues/20216
Suggested-by: Tim Serong <tserong@suse.com>
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15405 from liewegas/wip-rest-test

pybind/mgr/restful: improve cert handling; work with vstart

Reviewed-by: Boris Ranto <branto@redhat.com>

debian/control: Build-Depends on python-pecan

This is needed for make check (so that restful mgr module can load).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15557 from liewegas/wip-mgr-respawn

mgr/MgrStandby: respawn when deactivated

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15301 from shashalu/rgw-ssl

rgw: add "rgw_verify_ssl" config

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15257 from shashalu/fix-rgw-admin-retcode

rgw: fix radosgw-admin retcode

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14950 from fangyuxiangGL/set-obj-accounted-size

rgw: set object accounted size correctly

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/Mgr: fix deadlock in load_config

Signed-off-by: Sage Weil <sage@redhat.com>

pybind/mgr/dashboard: get_localized_config for server_{addr,port}

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/rest/test_mgr_rest_api.py: improvements

Signed-off-by: Sage Weil <sage@redhat.com>

pybind/mgr/restful: store each key+pass in a separate key

Signed-off-by: Sage Weil <sage@redhat.com>

pybind/mgr/restful: prefix commands with 'restful'

And use - instead of _, following the ceph convention.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr: add get_config_prefix

Fetch a dict of all config options with a given prefix.

Signed-off-by: Sage Weil <sage@redhat.com>

pybind/mgr/restful: localize key/crt keys

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/PyState: add get_mgr_id() to module interface

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/PyModules: prefix by mgr/, not mgr/$id/

If modules want per-instance state, they can include
the mgr id in their portion of the key name.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/ConfigKeyService: more useful status message

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: start up mgr restful API

We try to stagger the ports that mgr modules bind to so that
concurrent vstart instances can run with consecutive ports but
the services will not collide.  Yes, this is awkward.

Signed-off-by: Sage Weil <sage@redhat.com>

pybind/mgr/restful: all crt/key or filename in config-key

Signed-off-by: Sage Weil <sage@redhat.com>

ceph.spec.in: add python-pecan as BuildDepends too

For install-deps.sh's benefit.

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: load 'restful' (not 'rest') mgr module

Signed-off-by: Sage Weil <sage@redhat.com>

test/vstart_wrapper.sh: fix MGR_PYTHON_PATH

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: start mgr dashboard, echo URL(s)

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15507 from zhangsw/fix-rgw-logtrim-bug

rgw: datalog trim and mdlog trim handles the result returned by osd incorrectly.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15102 from majianpeng/blockdevice-pmem

os/bluestore/BlockDevice: support pmem device as bluestore backend

Reviewed-by: Sage Weil <sage@redhat.com>

osd: take PGRef for recovery sleep wakeup event

The raw PG* may not remain valid when the event fires.

Fixes: http://tracker.ceph.com/issues/20226
Signed-off-by: Sage Weil <sage@redhat.com>

qa: update krbd_data_pool.sh to match the new rados ls behavior

See commit e9973adf1fb7 ("PendingReleaseNotes: notes on whiteouts vs
pgnls").

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #15577 from jcsp/wip-dashboard-cleanup

mgr: dashboard code cleanup

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: remove confusing SyncObject stuff

This was a hangover from when these wrapper classes
were borrowed from Calamari, which used these
versions/equality functions to work out when
to go fetch data from the ceph cluster.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr/dashboard: fix server view on funny version format

Signed-off-by: John Spray <john.spray@redhat.com>

mgr/dashboard: fix error handling in remote_view_cache

Signed-off-by: John Spray <john.spray@redhat.com>

mgr/dashboard: keep a global librados instance

...so that classes that need one aren't creating
their own all the time.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #15566 from baruza/wip-doc-20199

doc: Change the default values of some OSD options

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15485 from ceph/wip-hdd-throttle-cost

common/config_opts.h: Lower HDD throttle cost.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15203 from Liuchang0812/wip-kill-broken-links

doc: kill some broken links

Reviewed-by: Kefu Chai <kchai@redhat.com>

debian: sync logrotate packaging with downstream

both Ubuntu and Debian put the logrotate script into
ceph-common, to ensure that radosgw logs are rotated as
well.

to prevent duplicate logrotate scripts handling the same log
files, and to minimize the delta between upstream and
downstream packaging, sync this change back upstream.

Fixes: http://tracker.ceph.com/issues/19938
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>

mgr: drop PyModules::lock around module shutdown calls

This deadlocked if the module's shutdown call was trying
to take a pythonland lock that was held by a thread
that was in turn trying to call into C++ land and take
PyModules::lock.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: don't let TYPE_CLIENT entities send stats

Anyone instantiating librados instances ended up
appearing as a weird entry in DaemonStateIndex.

Come back to this at the point we want e.g. stats
from RGWs or NFS gateways.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: don't log successful ceph_config_get as err

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #15575 from liewegas/wip-pgnls-note

PendingReleaseNotes: notes on whiteouts vs pgnls

Merge pull request #15309 from ifed01/wip-bluestore-boost-statfs

os/bluestore: keep statfs replica in RAM to avoid expensive KV retrieval

Reviewed-by: Sage Weil <sage@redhat.com>

PendingReleaseNotes: notes on whiteouts vs pgnls

Signed-off-by: Sage Weil <sage@redhat.com>

pybind/rbd: OSError should be picklable

Fixes: http://tracker.ceph.com/issues/20223
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mon/PGMap: erase osd_epochs entry too when we erase osd_stat entry

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ClusterState: dump pgmap and inc at dout 30

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15532 from dillaman/wip-rbd-mirror-stress

rbd-mirror: lock loss during sync should wait for in-flight copies

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #15518 from dillaman/wip-rbd-missing-default-pool

rbd: removed hardcoded default pool

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

os/bluestore: introduce a debug switch to bypass kv update.

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: introduce debug parameter to bypass bdev verification

E.g. this allows to backup WAL/DB volumes after preconditioning and quickly rollback to that state when needed.

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: keep statfs replica in memory to avoid expensive KV access

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: move volatile_statfs struct out of TransContext

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

test/msgr: silence warnings from -Wsign-compare

The following warning appears during make in the latest code:

ceph/src/test/msgr/test_msgr.cc:1474:46: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
     CHECK_AND_WAIT_TRUE(srv_dispatcher.count > last + 1);
                         ~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~
ceph/src/test/msgr/test_msgr.cc:54:9: note: in definition of macro ‘CHECK_AND_WAIT_TRUE’
     if (expr)                           \
         ^~~~
ceph/src/test/msgr/test_msgr.cc:1475:30: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
     if (srv_dispatcher.count == last) {

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #15433 from wjwithagen/wip-wjw-prettytable

ceph-disk: add setting for external py-modules for tox-testing

Reviewed-by: Loic Dachary <ldachary@redhat.com>

rgw_file: use predefined format_child_name to replace raw string concatenation

Signed-off-by: Gui Hecheng <guihecheng@cmss.chinamobile.com>

doc: Change the default values of some OSD options

These values were changed to reflect this change a3f9cba

Fixes: http://tracker.ceph.com/issues/20199
Signed-off-by: Bara Ancincova <bara@redhat.com>

ceph-disk: use system modules if needed

 - prettytable usage was introduced in:
https://github.com/ceph/ceph/commit/3fa8bb1
   It is in the install-deps.sh file to be installed, but
   it is not per default installed in the testenvironment
   So allow tox tests to use "external" modules

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

rgw_file: pass bucketname instead of uri for bucket create/delete

Signed-off-by: Gui Hecheng <guihecheng@cmss.chinamobile.com>

tools/rbd_mirror: do not include ununsed header

Signed-off-by: Kefu Chai <kchai@redhat.com>

client: s/atomic_t/std::atomic<>

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake,debian,rpm: remove atomic_t completely

Signed-off-by: Kefu Chai <kchai@redhat.com>

msg,client,os,journal: do not include unused header

Signed-off-by: Kefu Chai <kchai@redhat.com>

msg/async: s/atomic_t/atomic<>/

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: rgw_rados: s/atomic_t/atomic<>/

Signed-off-by: Kefu Chai <kchai@redhat.com>

msg,common: do not include assert.h everywhere to clobber <assert.h>

* reverts c93daaf92b643a06a822558973d377733dbd9d85
* add "#include "include/assert.h" to where <cassert> or <assert.h> are
  included, or avoid including them if possible.

Signed-off-by: Kefu Chai <kchai@redhat.com>

osdc/Objecter: s/atomic_t/std::atomic<>/

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/ceph_context: s/atomic_t/std::atomic<>/

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/DaemonServer.cc: log daemon type string as well as id

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #14866 from chardan/jfw-wip-halflife_atomic_t-varan

common,test: migrate atomic_t to std::atomic

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd: when osd is not in failure_pending, we don't need to get osd inst from osdmap.

Signed-off-by: linbing <linbing@t2cloud.net>

Merge pull request #15520 from liewegas/wip-max-object-size

osd: reduce rados_max_object_size from 100 GB -> 128 MB

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

osd: reduce rados_max_object_size from 100 GB -> 128 MB

I'm not sure why we had this so huge!  128 MB is a pretty reasonable size.

Signed-off-by: Sage Weil <sage@redhat.com>