ceph_test_rados_api_*: wait for snap trim on ENOENT during cleanup

Assume that an ENOENT removing an object during cleanup means the
head is a whiteout because there are clones.  When that happens
just sleep for a bit and then retry in the hopes that snap
trimming has happened.

Time out (and fail) after retrying for 10 minutes' worth of tries.

Fixes: http://tracker.ceph.com/issues/19948
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15604 from liewegas/wip-rest-test-qa

qa/suites/rados/rest: test restful mgr module

Reviewed-by: Boris Ranto <branto@redhat.com>

qa/suites/rados/rest/mgr-restful: test on centos

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15584 from liewegas/wip-20219

mon/MgrStatMonitor: keep mgrstat version ahead of pgmon

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #15620 from tchaikov/wip-20249

test/vstart_wrapper.sh: display_log on test failure

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

qa/workunits/rest/test_mgr_rest_api.py: tolerate old distros

Work with older requests and/or missing urllib3.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/rest/mgr-restful: test restful module

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15634 from liewegas/wip-pyformatter

mgr/PyFormatter: implement dump_format_va

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15580 from liewegas/wip-bluestore-memory

os/bluestore: configure rocksdb cache via bluestore_cache_kv_ratio

Reviewed-by: Mark Nelson <mnelson@redhat.com>

mgr/PyFormatter: implement dump_format_va

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15624 from Adirl/wip

msg/async/rdma: check if fin message completed

Reviewed-by: Haomai Wang <haomai@xsky.com>

msg/async/rdma: check if fin message completed

Change-Id: I9cb1e523be7b21127503a9252b09331a37bea204
Signed-off-by: Alexander Mikheev <alexm@mellanox.com>
Signed-off-by: Adir Lev <adirl@mellanox.com>

mon/PGMonitor: assert PGMonitor stops proposing after luminous

One hacky exception here for the rebuild-mondb case.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MgrStatMonitor: keep mgrstat version ahead of pgmon

Pre-luminous, statfs and getpoolstats come from pgmon, and clients will
see and track the paxos version for that.  After the switchover, those
move to mgrstatmon.  If a clients sends a version ahead of the current
mgrstat version, it will block until the paxos version catches up, which
may be years for an aged cluster.

Fix by keeping mgrstat version ahead of pgmon's every time it updates.
This only really matters right at the transition point (we don't update
before that, and after that pgmon doesn't update).

Fixes: http://tracker.ceph.com/issues/20219
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #14753 from theanalyst/v12.0.2/mailmap

doc: mailmap for v12.0.2

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #15328 from jimifm/wip-RGWPutObj

rgw/rgw_op:Prevents memory leaks when calling func swift_versioning_copy() fails

Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #15323 from fangyuxiangGL/compress_bug_fix

rgw: get wrong content when download object with specific range with compression

Reviewed-by: Adam Kupczyk <akupczyk@mirantis.com>
Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

qa: timeout if flush_pg_stats() takes too long

a "timeout" which defaults to 300 seconds is added to flush_pg_stats()

Signed-off-by: Kefu Chai <kchai@redhat.com>

mailmap: Yunfei Guan affiliation

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

mailmap: Andrey Parfenov affiliation

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

mailmap: Yao Zongyou affiliation

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

mailmap: Patrick Dinnen affiliation

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

mailmap: Henrik Korkuc affiliation

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

mailmap: Sven Seeberg affiliation

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

mailmap: Mohamad Gebai affiliation

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

mailmap: Fabian Grünbichler affiliation

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

mailmap: Vedant Nanda affiliation

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

mailmap: Andrew Solomon affiliation

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

mailmap: Yu Shengzuo affiliation

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

mailmap: Yanchun Bai affiliation

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

mailmap: Tone Zhang affiliation

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

mailmap: Liu Hong affiliation

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

mailmap: Liu Hong affiliation

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

mailmap: Li Hongjie name normalization

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

mailmap: Josh Durgin name normalization

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

mailmap: Jesse Williamson name normalization

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

mailmap: Hongtong Liu name normalization

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

mailmap: He Chuang affiliation

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

mailmap: Chu Hua-Rong name normalization

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

qa: remove duplicated flush_pg_stats() helper

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/vstart_wrapper.sh: display_log on test failure

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15610 from wjwithagen/wip-wjw-freebsd-py-pecan

core: Install Pecan for FreeBSD

Merge pull request #15574 from dillaman/wip-20223

pybind/rbd: OSError should be picklable

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #15576 from dachary/wip-ceph-disk

ceph-disk: sensible default for block.db

Reviewed-by: Sage Weil <sage@redhat.com>

core: Install Pecan for FreeBSD

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #15609 from stiwari/wip-19879

Merge pull request from stiwari/wip-19879

Merge pull request #12851 from zhangsw/fix-rgw-bucketcheck-bug

rgw: bucket index check in radosgw-admin removes valid index.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #15606 from stiwari/fix_1

Documentation Fixes for http://tracker.ceph.com/issues/19879

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #15588 from liewegas/wip-20216

pybind/mgr/dashboard: monkeypatch os.exit to stop cherrypy from taking down mgr

Reviewed-by: John Spray <jspray@redhat.com>

Merge pull request #15550 from jcsp/wip-doc-cephfs-objsize

doc/cephfs: mention RADOS object size limit

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

ceph-disk: sensible default for block.db

If bluestore_block_db_size is unset or zero, fallback to a sensible
default which is block_size / 100. With a minimum of 1GB.

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #13395 from dang/wip-quota-cache-doc

Document RGW quota cache options

Merge pull request #15568 from guihecheng/rgw_file-cleanup-names

rgw: rgw_file cleanup names

Merge pull request #15541 from guihecheng/rgw_file-cleanup

rgw_file: replace raw fs->fh_lru.unref with predefined fs->unref

Merge pull request #15558 from Linbing/wip-osd-optimize-send-failues

osd: when osd in not in failure_pending, we don't need to get osd inst from osdmap.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15223 from badone/wip-ceph_test_admin_socket_output

tests: Add integration tests for admin socket output

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15573 from liewegas/wip-20208

mon/PGMap: fix osd_epoch update when removing osd_stat

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Loic Dachary <loic@dachary.org>

os/bluestore: set bluestore_cache_kv_ratio to configure rocksdb cache size

Devote 40% to kv (rocksdb), 50% to metadata (onodes etc), 10% to data.

Note that if we don't consume the data portion (e.g., no cache hints) that
the onode metadata will "borrow" that space.

Signed-off-by: Sage Weil <sage@redhat.com>

kv/RocksDBStore: rocksdb_cache_row_ratio

Specify portion of cache to devote to row cache (the rest goes to the
block cache).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15594 from idryomov/wip-krbd-rados-ls

qa: update krbd_data_pool.sh to match the new rados ls behavior

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #15600 from ceph/luminous

12.0.3

Merge pull request #15180 from theanalyst/doc/rgw-dev-s3tests

doc: dev improve the s3tests doc to reflect current scripts

kv/RocksDBStore: allow cache_size to be specified explicitly

(Not via a config option)

Signed-off-by: Sage Weil <sage@redhat.com>

kv/RocksDBStore: make rocksdb_cache_type tunable (lru or clock)

"[Clock is] Similar to NewLRUCache, but create a cache based on CLOCK
algorithm with better concurrent performance in some cases. See
[cache]/clock_cache.cc for more detail."

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15570 from joscollin/wip-cleanup-testmsgr-warning

test/msgr: silence warnings from -Wsign-compare

Reviewed-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #15100 from kestrels/wip-crc32c-fastzero2

crc32c: Add ppc64le fast zero optimized assembly.

Reviewed-By: Piotr Dałek <piotr.dalek@corp.ovh.com>

Merge pull request #15387 from jecluis/wip-20123

mon/PGMap: show %used in formatted output

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15508 from tchaikov/wip-ceph.in-cleanup

ceph.in: drop the compatiiblity to handle non json commands

Merge pull request #15560 from dmick/master

mgr/DaemonServer.cc: log daemon type string as well as id

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15562 from tchaikov/wip-no-assert.h

common,osdc: remove atomic_t completely

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15567 from Fabian-Gruenbichler/pull/logrotatesync

debian: sync logrotate packaging with downstream

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15578 from jcsp/wip-mgr-cleanup

mgr: cleanup, stop clients sending in perf counters

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15582 from liewegas/wip-20226

osd: take PGRef for recovery sleep wakeup event

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

pybind/mgr/dashboard: monkeypatch os.exit to stop cherrypy from taking down mgr

Fixes: http://tracker.ceph.com/issues/20216
Suggested-by: Tim Serong <tserong@suse.com>
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15405 from liewegas/wip-rest-test

pybind/mgr/restful: improve cert handling; work with vstart

Reviewed-by: Boris Ranto <branto@redhat.com>

debian/control: Build-Depends on python-pecan

This is needed for make check (so that restful mgr module can load).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15557 from liewegas/wip-mgr-respawn

mgr/MgrStandby: respawn when deactivated

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15301 from shashalu/rgw-ssl

rgw: add "rgw_verify_ssl" config

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15257 from shashalu/fix-rgw-admin-retcode

rgw: fix radosgw-admin retcode

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14950 from fangyuxiangGL/set-obj-accounted-size

rgw: set object accounted size correctly

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/Mgr: fix deadlock in load_config

Signed-off-by: Sage Weil <sage@redhat.com>

pybind/mgr/dashboard: get_localized_config for server_{addr,port}

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/rest/test_mgr_rest_api.py: improvements

Signed-off-by: Sage Weil <sage@redhat.com>

pybind/mgr/restful: store each key+pass in a separate key

Signed-off-by: Sage Weil <sage@redhat.com>

pybind/mgr/restful: prefix commands with 'restful'

And use - instead of _, following the ceph convention.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr: add get_config_prefix

Fetch a dict of all config options with a given prefix.

Signed-off-by: Sage Weil <sage@redhat.com>

pybind/mgr/restful: localize key/crt keys

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/PyState: add get_mgr_id() to module interface

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/PyModules: prefix by mgr/, not mgr/$id/

If modules want per-instance state, they can include
the mgr id in their portion of the key name.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/ConfigKeyService: more useful status message

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: start up mgr restful API

We try to stagger the ports that mgr modules bind to so that
concurrent vstart instances can run with consecutive ports but
the services will not collide.  Yes, this is awkward.

Signed-off-by: Sage Weil <sage@redhat.com>

pybind/mgr/restful: all crt/key or filename in config-key

Signed-off-by: Sage Weil <sage@redhat.com>

ceph.spec.in: add python-pecan as BuildDepends too

For install-deps.sh's benefit.

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: load 'restful' (not 'rest') mgr module

Signed-off-by: Sage Weil <sage@redhat.com>

test/vstart_wrapper.sh: fix MGR_PYTHON_PATH

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: start mgr dashboard, echo URL(s)

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15507 from zhangsw/fix-rgw-logtrim-bug

rgw: datalog trim and mdlog trim handles the result returned by osd incorrectly.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15102 from majianpeng/blockdevice-pmem

os/bluestore/BlockDevice: support pmem device as bluestore backend

Reviewed-by: Sage Weil <sage@redhat.com>

osd: take PGRef for recovery sleep wakeup event

The raw PG* may not remain valid when the event fires.

Fixes: http://tracker.ceph.com/issues/20226
Signed-off-by: Sage Weil <sage@redhat.com>

qa: update krbd_data_pool.sh to match the new rados ls behavior

See commit e9973adf1fb7 ("PendingReleaseNotes: notes on whiteouts vs
pgnls").

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>