Merge pull request #14486 from dachary/wip-multiweight

crush: weight_set and id remapping

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14556 from liewegas/wip-pgupmap

osd: pg-remap -> pg-upmap

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #14554 from ukernel/wip-19630

mds: fix stray creation/removal notification

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #14433 from cbodley/wip-rgw-multi-py

test/rgw: refactor test_multi.py for use in qa suite

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #14623 from liewegas/wip-bluestore-compression

qa/objectstore: test bluestore with aggressive compression

Merge pull request #14642 from linuxbox2/wip-rgw-poolcreate-erange

rgw: error more verbosely in RGWRados::create_pool

rgw: error more verbosely in RGWRados::create_pool

Adds detail when pool creation fails with ERANGE, which can be
caused by pool settings.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #14637 from ovh/bp-test-osd-dup-warn-on-low-limit

test/osd/osd-dup.sh: warn on low open file limit

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14631 from idryomov/wip-krbd-data-pool-fix

qa: krbd_data_pool.sh: account for rbd_info metadata object

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #14385 from smithfarm/wip-release-script-strip-periods

tools: ceph-release-notes: strip trailing punctuation

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

test/osd/osd-dup.sh: warn on low open file limit

This test fails badly when open file limit is low. Increasing to around
1536 seems to be doing the trick, so warn the user with appropriate
message and try to proceed anyway.

Signed-off-by: Piotr Dałek <piotr.dalek@corp.ovh.com>

Merge pull request #14599 from smithfarm/wip-crushtool-mappings

doc: mention --show-mappings in crushtool manpage

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #14483 from wjwithagen/wip-wjw-freebsd-ceph-disk-list

src/ceph-disk/ceph_disk/main.py: Make 'ceph-disk list' work on FreeBSD

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

test/rgw: rewrite test_multi.py in terms of rgw_multi

logging changes inspired by Abhishek Lekshmanan <abhishek@suse.com>

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: add tests in rgw_multi/tests.py based on multisite.py

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: add rgw_multi/multisite.py

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14614 from wjwithagen/wip-wjw-cephtool-test-mon.sh

qa/workunits/cephtool/test.sh: Be more liberal in testing health-output.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14106 from theanalyst/release/12.0.1

doc: add 12.0.1 release notes

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/objecstore/bluestore-comp: test bluestore with compression on

Enable aggressive compression.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/objectstore/bluestore: drop old option

This test code has been removed.

Signed-off-by: Sage Weil <sage@redhat.com>

qa: krbd_data_pool.sh: account for rbd_info metadata object

... added in commit a16beba78435 ("librbd: image create validates that
pool supports overwrites").

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

test/rgw: add skeleton rgw_multi module

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: update and fixes for test-rgw-multisite.sh

the script was incomplete and unused, but it seems useful in itself
to bring up a simple multisite cluster without having to go through
test_multi.py. it's also a good test for functions in the other
test-rgw-*.sh scripts

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: test_multi.py uses cluster_id for calls

this makes the underlying scripts more flexible, because they don't
depend on having a cluster name in the zgX-cY format

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14619 from cbodley/wip-rgw-encryption-build-fix

rgw: remove extra RGWMPObj in rgw_multi.h

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

rgw: remove extra RGWMPObj in rgw_multi.h

two separate PRs had done refactoring around RGWMPObj, and it ended up
in two different places. remove the one in rgw_multi.h, because
rgw_rados.h now depends on its definition

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14418 from songbaisen/m20

mon: fix synchronise pgmap with others

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14436 from dachary/wip-19513-ceph-disk-device-class

ceph-disk,osd: add support for crush device classes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14464 from ceph/wip-systemd

qa/tasks: use sudo to check ceph health for systemd test

Merge pull request #14615 from dillaman/wip-19636

librbd: corrected resize RPC message backwards compatibility

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #14381 from joscollin/wip-cleanup-osdmap-rangefor

osd: Cleanup-Updated OSDMap.cc with C++11 style range-for loops

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14431 from liewegas/wip-mgrclient-leak

mgr/MgrClient: fix reconnect event leak

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14507 from tchaikov/wip-19590

mgr: always free allocated MgrPyModule

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

osd: pg-remap -> pg-upmap

'remap' is to non-specific a name.  In particular, it
sounds like it is related to the 'remapped' PG state
but in reality it is not related.

'upmap' or 'pg-upmap' is more specific: it maps a pgid
to the 'up' set value (or item)

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #13597 from cbodley/wip-s3tests-crypto

qa/rgw: add configuration for server-side encryption tests

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #13483 from ceph/wip-rgw-encryption-doc

doc: rgw server-side encryption and barbican

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #11049 from aclamk/wip-rgw-s3-crypto-sse-kms-plugin

rgw: s3 server-side encryption (SSE-C, SSE-KMS)

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #14415 from smithfarm/wip-19556

tests: Thrasher: handle "OSD has the store locked" gracefully

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14456 from tchaikov/wip-rocksdb-freebsd-build

rocksdb: sync with upstream

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #14609 from linuxbox2/wip-rgw-file-wattrs

rgw_file:  pre-compute unix attrs in write_finish()

Merge pull request #14488 from dzafman/wip-15789

Give requested scrub work a higher priority

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

librbd: corrected resize RPC message backwards compatibility

Commit d1f2c557 incorrectly changed the order of variables within
the payload. This resulted in breaking the resize RPC message
with older versions of Ceph.

Fixes: http://tracker.ceph.com/issues/19636
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #14600 from tchaikov/wip-19390

debian: package ceph.logroate properly

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

rgw_file:  pre-compute unix attrs in write_finish()

New serialized Unix attrs need to reflect the change being made,
and should be reverted if the change fails.

Fixes: http://tracker.ceph.com/issues/19653
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #14396 from jcsp/wip-19550

qa: re-enable ENOSPC tests for kclient

Merge pull request #14589 from jcsp/wip-19640

client: refine fsync/close writeback error handling

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #14272 from jcsp/wip-vstart-fixup

qa: fix test_standby_for_invalid_fscid with vstart_runner

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #14196 from jcsp/wip-cephfs-relnotes

PendingReleaseNotes: recent cephfs changes

Merge pull request #14105 from jcsp/wip-pretty-tell

mds: pretty json from `tell` commands

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #14104 from jcsp/wip-18509

mds: include advisory `path` field in damage

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #14164 from jcsp/wip-16842-mitigation

Mitigation for #16842, validate sessions after load

Reviewed-by: Yan, Zheng <zyan@redhat.com>

client: refine fsync/close writeback error handling

Previously, errors stuck indelibly to the inode, which
meant that a close call would see an error even if the
user already dutifully fsync()'d and handled it.

We should emit each error only once per file handle.

Signed-off-by: John Spray <john.spray@redhat.com>

/workunits/cephtool/test.sh: Be more liberal in testing health-output.

Sometimes I get output like:
   HEALTH_ERR 2 pgs stuck unclean; Full ratio(s) out of order

Which goes away over time. So it is a transit issue

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #14585 from optimistyzy/414

bluestore/NVMEDEVICE: update SPDK to version 17.03

Reviewed-by: Haomai Wang <haomai@xsky.com>

doc: mention --show-mappings in crushtool manpage

Fixes: http://tracker.ceph.com/issues/19649
Signed-off-by: Loic Dachary <ldachary@redhat.com>
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #12535 from ceph/wip-rgw-multisite-teuthology

rgw: multisite enabled over multiple clusters
Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14555 from yaozongyou/fix-readme-notconsistent

README.md: fix build instructions inconsistent.

Reviewed-by: Kefu Chai <kchai@redhat.com>

crush: implement CrushWrapper::dump(choose_args)

Signed-off-by: Loic Dachary <loic@dachary.org>

crush: disable modification API when choose_args is not empty

Adding, removing or move items / buckets via the CrushWrapper API when
choose_args is not empty is unlikely to produce the desired outcome. The
caller should instead add, remove or move items / buckets in a
decompiled crushmap, update the associated choose_arg and upload the new
crushmap.

Signed-off-by: Loic Dachary <loic@dachary.org>

crush: compile/decompile crush_choose_arg_map

A map of crush_choose_arg_map is added to the crushmap text syntax. The
key is an integer matching a pool number.

Signed-off-by: Loic Dachary <loic@dachary.org>

debian: package ceph.logroate properly

see also "man dh_installlogrotate"

Fixes: http://tracker.ceph.com/issues/19390
Signed-off-by: Kefu Chai <kchai@redhat.com>

crush: add per pool choose_args when calling do_rule

If there is no crush_choose_arg_map for a given pool (the default) a
NULL pointer is given instead and crush_do_rule behavior remains
unchanged.

Signed-off-by: Loic Dachary <loic@dachary.org>

crush: implement weight and id overrides for straw2

bucket_straw2_choose needs to use weights that may be different from
weight_items. For instance to compensate for an uneven distribution
caused by a low number of values. Or to fix the probability biais
introduced by conditional probabilities (see
http://tracker.ceph.com/issues/15653 for more information).

We introduce a weight_set for each straw2 bucket to set the desired
weight for a given item at a given position. The weight of a given item
when picking the first replica (first position) may be different from
the weight the second replica (second position). For instance the weight
matrix for a given bucket containing items 3, 7 and 13 could be as
follows:

          position 0   position 1

item 3     0x10000      0x100000
item 7     0x40000       0x10000
item 13    0x40000       0x10000

When crush_do_rule picks the first of two replicas (position 0), item 7,
3 are four times more likely to be choosen by bucket_straw2_choose than
item 13. When choosing the second replica (position 1), item 3 is ten
times more likely to be choosen than item 7, 13.

By default the weight_set of each bucket exactly matches the content of
item_weights for each position to ensure backward compatibility.

bucket_straw2_choose compares items by using their id. The same ids are
also used to index buckets and they must be unique. For each item in a
bucket an array of ids can be provided for placement purposes and they
are used instead of the ids. If no replacement ids are provided, the
legacy behavior is preserved.

Signed-off-by: Loic Dachary <loic@dachary.org>

crush: cleanup test memory leaks

Signed-off-by: Loic Dachary <loic@dachary.org>

crush: do not use TREE in tests

It is bugous and unsupported.

Signed-off-by: Loic Dachary <loic@dachary.org>

bluestore/NVMEDEVICE: update SPDK to version 17.03

Do some minor changes:

1 Restrict the total DPDK memory used by an osd instance.
change the name from bluestore_spdk_socket_mem to
bluestore_spdk_mem.

2 use spdk_env_init instead of rte_eal_init. The reason is that
SPDK lib invokes rte_eal_init which reduces the initilization
paramter conversion and check, also spdk 17.03 invokes
spdk_vtophys_register_dpdk_mem() (which is an internal function)
in spdk_env_init, and this func must be called.

Signed-off-by: optimistyzy <optimistyzy@gmail.com>

osd: Give requested scrub work a higher priority

Once started we now queue scrub work at higher priority than
scheduled scrubs.

Fixes: http://tracker.ceph.com/issues/15789
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #14591 from tchaikov/wip-readme-headings

README.md: use github heading syntax to mark the headings

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #13968 from dzafman/wip-15912-followon

osd,mon: misc full fixes and cleanups

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14440 from liewegas/wip-status-flags

osd/OSDMap: hide require_*_osd and sortbitwise flags

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

test: Test health check output for full ratios

Test out of order ratios summary and details
Test various full osd conditions summary and details

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #14561 from linuxbox2/wip-rgw-reread-dir

rgw_file: fix readdir after dirent-change

README.md: use github heading syntax to mark the headings

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14466 from fangyuxiangGL/bi

rgw: bucket stats display bucket index type

Reviewed-by: liuchang0812 <liuchang0812@gmail.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

mon, osd: Add detailed full information for now in the mon

Show ceph health doc output in the correct order

Signed-off-by: David Zafman <dzafman@redhat.com>

mon: Issue warning or error if a full ratio out of order

The full ratios should be in this order: nearfull, backfillfull, full, failsafe full

Signed-off-by: David Zafman <dzafman@redhat.com>

mon: Always fix-up full ratios when specified incorrectly in config

Signed-off-by: David Zafman <dzafman@redhat.com>

mon: Use currently configure full ratio to determine available space

This is a bug that would not adjust available space based on the
currently configured full ratio, but rather the mon_osd_full_ratio
default initial value.

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: check_full_status() remove bogus comment and use equivalent computation

We actually compute kb_used as the kb - kb_avail.  We don't have the
statfs() system call issue of non-privileged f_bavail vs f_bfree.  It
was assumed that used was really like (blocks - f_bfree).  It is not.

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Check whether any OSD is full before starting recovery

Add event RecoveryTooFull to move to NotRecovering state

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Add PG state and flag for too full for recovery

New state machine state NotRecovering
New PG state PG_STATE_RECOVERY_TOOFULL

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Add check_osdmap_full() to check for shard OSD fullness

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Rename backfill_request_* to recovery_request_*

To be used by both recovery and backfill

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Check failsafe full and crash on push/pull

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Revamp injectfull op to support all full states

Use check_* for injectable full checks
Use is_* to just test simple cur_state

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Handle backfillfull_ratio just like nearfull and full

Add BACKFILLFULL as a local OSD cur_state
Notify monitor of this new fullness state

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: For testing full disks add injectfull socket command

Signed-off-by: David Zafman <dzafman@redhat.com>

common: Bump ratio for backfillfull from 85% to 90%

Signed-off-by: David Zafman <dzafman@redhat.com>

common: Remove unused config option osd_recovery_threads

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: too_full_for_backfill() returns ostream for reason

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Fail-safe full is a hard stop even for mds

We can't allow OSD to become non-startable even if mds
could be writing as part of file removals.

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Fix intended test flow and restore nearfull-ratio

This is inconsequential but seems to have always been wrong since original
commit 6cafb0e3e0bc5f992c2483a46cb00e83dca035cc

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Switch from pg to osd for set-*-ratio commands

Testing of 6422e0a220fb3f32ccae50e0c7e52dc9984685c6

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Remove unused argument to clear_queued_recovery

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Increase osd_backfill_retry_interval to 30 seconds

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: cleanup comment

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Fix log message

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #12536 from ilc/cls_rgw_minor_refactor

cls/rgw: Clean up the "magic string" usage in the cls layer for RGW.

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

mds: fix stray creation/removal notification

In MDCache::scan_stray_dir(), notify_stray_created() is called for
both null and non-null dentries. This is wrong, it should be called
only for non-null dentries.

Calling notify_stray_created() in MDCache::get_or_create_stray_dentry()
is racy too. MDS can create wrong stray dentry because target dentry
linkage can change before rmdir/rename lock the dentry lock.

Fixes: http://tracker.ceph.com/issues/19630
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #14046 from songbaisen/m7

mgr: optimization some judgment and adjust the debug remove value in register_new_pgs

Reviewed-by: John Spray <john.spray@redhat.com>