Merge pull request #11623 from dillaman/wip-cls-rbd-group

librbd: ensure consistency groups will gracefully fail on older OSDs

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #11703 from dillaman/wip-17750

test: skip TestLibRBD.DiscardAfterWrite if skip partial discard enabled

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #11714 from dachary/wip-17757-eio

tests: disable failing tests

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

tests: disable failing unittest_erasure_code_plugin_jerasure

While figuring out what's wrong with it.

Refs: http://tracker.ceph.com/issues/17561

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: disable failing test-erasure-eio.sh

While figuring out what's wrong with it.

Refs: http://tracker.ceph.com/issues/17757

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #11382 from yuyuyu101/wip-fix-shadow

osd/PGBackend: fix collection_list shadow return value

Reviewed-by: Kefu Chai <kchai@redhat.com>
tchaikov

Merge pull request #11547 from javeme/check-pgid

osd/command tell: check pgid at the right time

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11203 from stiopaa1/osd_pg_replaceCountPart1

osd/PG.cc: prevent repeated searching of map/set

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11242 from stiopaa1/osd_replicatedpg_removeCount1

osd/ReplicatedPG: remove unneeded use of count

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11365 from badone/wip-lsb_release-remove

common: Remove the runtime dependency on lsb_release

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Ken Dreyer (Red Hat) <kdreyer@redhat.com>

test: skip TestLibRBD.DiscardAfterWrite if skip partial discard enabled

Fixes: http://tracker.ceph.com/issues/17750
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11249 from wido/osd-flush-journal-shutdown

osd: Flush Journal on shutdown

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #11686 from ceph/wip-ceph-rest-api-new-addr

ceph-rest-api: understand the new style entity_addr_t representation

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11530 from markhpc/wip-rocksdb_tuning

common/config_opts.h: Optimized RocksDB WAL settings.

Merge pull request #11688 from stiopaa1/mds_snapserver_addConstToMemFun

mds/SnapServer: add const to member function

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

mds/SnapServer: add const to member function

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

ceph-rest-api: understand the new style entity_addr_t representation

the new entity_addr_t representation was introduced by 3561eb3, we
should update ceph_rest_api.py accordingly.

Fixes: http://tracker.ceph.com/issues/17742
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10993 from Vicente-Cheng/fix-alloc-choose_tries

crush: make counting of choose_tries consistent

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11246 from badone/wip-set_xattr_limits_via_conf-fix

os/filestore: Fix erroneous WARNING: max attr too small

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11614 from runsisi/wip-fix-PK11_FreeSymKey-segfault

auth: fix NULL pointer access when trying to delete CryptoAESKeyHandler instance

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11654 from hjwsm1989/fix

mon: print the num_pools and num_objects in 'ceph -s -f json/json-p…

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11639 from atheism/mgr-PyModules-remove-duplicated-else-if

mgr: PyModules.cc: remove duplicated if condition for fs_map

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11659 from tchaikov/wip-delete-cct-later

librados: postpone cct deletion

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11314 from jcsp/wip-mgr-flake8

pybind: clean up mgr stuff for flake8

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11650 from wjwithagen/wip-wjw-log-loging

Log: Replace namespace log with logging

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11680 from tchaikov/wip-17728

test/ceph_test_msgr: do not use Message::middle for holding transient…

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #11397 from dzafman/wip-scrub-misc

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/ceph_test_msgr: do not use Message::middle for holding transient data

Message::middle is used for holding encoded data, so we we can not stuff
it with payload and leave the "payload" field empty. this change
refactors the ceph_test_msgr by introducing a Payload class which
encodes all test data in it.

Fixes: http://tracker.ceph.com/issues/17728
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11677 from ceph/wip-rocksdb-no-march-native

rocksdb: do not build with --march=native

Reviewed-by: Bassam Tabbara <bassam.tabbara@quantum.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #11678 from tchaikov/wip-silence-rbd-warning

librbd: silence the unused variable warning

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: silence the unused variable warning

Signed-off-by: Kefu Chai <kchai@redhat.com>

rocksdb: do not build with --march=native

we can not assume that building host has the same arch (or the same cpu
capability) with the target machine.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11663 from stiopaa1/mds_mdslog_addConstToMemFunc

mds/MDLog: add const to member functions

Merge pull request #11668 from Enproduktion/master

doc: Fixed mapping error in legacy mds command

Merge pull request #11446 from ukernel/wip-cephfs-scan-links

tools/cephfs: add scan_links command which fixes linkages errors

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11609 from stiopaa1/mds_snaprealm_unneededuseofcount

mds/SnapRealm: remove unneeded use of count

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11625 from stiopaa1/mds_damageTable_removeUnneededCount

mds/DamageTable.cc: remove unneeded use of count

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #9304 from dzafman/wip-13937

osd: handle EC recovery read errors

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11645 from dillaman/wip-librbd-discard-after-write

test: new librbd discard after write test case

Merge pull request #11670 from stiopaa1/mds_mutation_addConstToMemFun

mds/Mutation.h: add const to member functions

Merge pull request #11608 from atheism/mds-duplicated-check

client: Client.cc: remove duplicated op type checking against CEPH_MD…

client: Client.cc: correct the op type checking for CEPH_MDS_OP_CREATE

Signed-off-by: Weibing Zhang <zhangweibing@unitedstack.com>

global: Always log loading erasure code for tests

Fix for broken test-erasure-code.sh and test-erasure-eio.sh

Signed-off-by: David Zafman <dzafman@redhat.com>

test.sh: Make check for flags more robust

Low space broke test, saw "flags nearfull,pauserd,pausewr...."

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Remove extra objectstore_tool call which causes a recovery

Caused by: 70e000a9a42c50eda48f5d3b9e432ffc3a70f75b

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Handle object removals in a non-racey way

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Fix hang on unfound object after mark_unfound_lost is done

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Handle recovery read errors

Fixes: http://tracker.ceph.com/issues/13937
Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Fix log messages

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: CLEANUP: Remove unused pending_read member

Remove unused struct

Signed-off-by: David Zafman <dzafman@redhat.com>

Revert "test: Disable tests due to recovery race"

This reverts commit 5bc55338f5a1645bc651811fae2f89ad855ff86e.

Conflicts:
src/test/Makefile.am (no longer exists)
src/test/erasure-code/Makefile.am (no longer exists)

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #11582 from batrick/ceph-dumpable

core: set dumpable flag after setuid

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

mds/Mutation.h: add const to member functions

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

doc: Fixed mapping error in legacy mds command

Signed-off-by: Malte Fiala <maltefiala@produktion.io>

Merge pull request #11666 from tchaikov/wip-cmake

cmake: remove include/Makefile.am

cmake: remove include/Makefile.am

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/filestore: Fix erroneous WARNING: max attr too small

When we read the conf values from the conf file we call handle_conf_change which
calls set_xattr_limits_via_conf but this is before m_fs_type is initialised so
we fall through to the default comaprison with
filestore_max_xattr_value_size_other (1024) and erroneously issue the warning.
This happens at every startup but subsequent calls once the filestore is mounted
proceed as expected. Don't call set_xattr_limits_via_conf unless the filestore
is mounted.

Fixes: http://tracker.ceph.com/issues/17420
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #11638 from tchaikov/wip-clang

include/enc: make clang happy

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11658 from hjwsm1989/doc

doc: change the osd_max_backfills default to 1

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mds/MDLog: add const to member functions

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #10887 from michaelsevilla/mantle

fs: Mantle: A Programmable Metadata Load Balancer

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11641 from vshankar/rbd-data-pool-assert

librbd: possible assert failure creating image when using data pool

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

common/cmdparse,msg/async: remove unnecessary std::move()

Signed-off-by: Kefu Chai <kchai@redhat.com>

librados: postpone cct deletion

RefCountedObject::put() prints log using cct, but RadosClient
relinquishes the reference to cct too early, so when its
`RefCountedObject` member variables print log when they are
destructed, the cct is already freed. hence we have a segfault.

to address this problem, we need postpone destructure of it after all
other RefCountedObject member variables are destroyed.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11570 from tchaikov/wip-revise-SubmittingPatches

doc: reformat SubmittingPatches with more rst syntax

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #11630 from cbodley/wip-rgw-curl-upgrade

rgw: remove suggestion to upgrade libcurl
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

osd/PGBackend: fix collection_list shadow return value

Fixes: http://tracker.ceph.com/issues/17713
Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge pull request #11385 from bassamtabbara/wip-rocksdb-external-project

cmake: use external project for rocksdb

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: remove ending period in section titles

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: change the osd_max_backfills default to 1
Fixes: http://tracker.ceph.com/issues/17701
Signed-off-by: huangjun <hjwsm1989@gmail.com>

Merge pull request #11653 from ktdreyer/doc-epel-instructions

doc: standardize EPEL instructions

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: use external project for rocksdb

This commit makes rocksdb an external cmake project. This ensures
that rocksdb is no longer built from src/rocksdb, and that make clean
works correctly.

Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>

pickup new rocksdb with support for linux cmake

This picksup a new version of rocksdb with support for
linux cmake.

Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>

Merge pull request #11619 from dachary/wip-17662-ceph-disk

ceph-disk: allow using a regular file as a journal

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11652 from ktdreyer/doc-mgr-syntax

doc: fix mgr literal block rST syntax

Reviewed-by: Kefu Chai <kchai@redhat.com>

  mon: print the num_pools and num_objects in 'ceph -s -f json/json-pretty'.
Fixes: http://tracker.ceph.com/issues/17703
Signed-off-by: huangjun <hjwsm1989@gmail.com>

librbd: assert failure when using data pool

This fixes a silly assert that's hit during image creation
(cli/api) when the data pool specified is same as the pool
specified by -p/--pool option (or the default).

Signed-off-by: Venky Shankar <vshankar@redhat.com>

doc: standardize EPEL instructions

Prior to this change, the documentation pages contained different ways
to enable EPEL. Pick a simple, secure (https) way and standardize on
that.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

doc: fix mgr literal block rST syntax

Prior to this change, there were double-colon ("::") characters in the
HTML docs to indicate literal blocks.

The proceeding paragraphs were not pre-formatted because there was no
newline between the double-colon and the to-be-pre-formatted paragraph.

Move the "::" characters inline to simplify the syntax and make Sphinx
properly interpret these texts as literal blocks.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #11594 from cbodley/wip-rgw-init-bucket-sync

rgw: fix for passing temporary in InitBucketSyncStatus

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #11505 from cbodley/wip-17571

rgw: clean up RGWShardedOmapCRManager on early return

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #11504 from cbodley/wip-17568

rgw: RGWSimpleRadosReadCR tolerates empty reads

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

mantle: re-design and fixes
- fix legacy encoding in mds map and add balancer to dumps
- fix blocking rados read and remove temporary files
- fix beacon message spamming

Signed-off-by: Michael Sevilla <mikesevilla3@gmail.com>

mantle: store balancer in RADOS, balancer version in MDSMap
- add docs and sample balancer (greedy-spill)

Signed-off-by: Michael Sevilla <mikesevilla3@gmail.com>

mantle: write balancer policies in Lua

Introduces Mantle, a programmable metadata load balancer. Policies for making
migration decisions are written in Lua but the Migrator and Balancer modules
still do fragmentation and migration. If the Lua balancer fails, control falls
back to the original balancer implementation.

Signed-off-by: Michael Sevilla <mikesevilla3@gmail.com>

Merge pull request #11352 from dachary/wip-17526-rados-bench

doc: document hostname constraints for rados bench

reviewed-by: John Wilkins <jowilkin@redhat.com>

Merge pull request #11602 from dillaman/wip-17665

doc: clarify QEMU rbd cache settings

reviewed-by: John Wilkins <jowilkin@redhat.com>

Log: Replace namespace log with loging

 - This to avoid a nameing confilct that Clang finds with
   std::log from cmath

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #9825 from liewegas/wip-addrvec

msg: add entity_addr_t types; add new entity_addrvec_t type

Reviewed-by: Kefu Chai <kchai@redhat.com>

test: new librbd discard after write test case

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11517 from ukernel/wip-17562

Fix #17562 (backtrace check fails when scrubbing directory created by fsstress)

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11593 from ukernel/wip-17275

osdc/ObjectCacher: wake up dirty stat waiters after removing buffers

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11604 from stiopaa1/mds_inotable_addOverride

mds/InoTable.h: add override to virtual functs

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11613 from stiopaa1/mds_cdir_removeUnneededUseofCount

mds/CDir.cc: remove unneeded use of count

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11611 from batrick/i17670

mds: check if down mds is known

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11191 from renhwztetecs/renhw-wip-add-libcephfs

libcephfs: add ceph_fsetattr&&ceph_lchmod&&ceph_lutime

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11560 from ukernel/wip-mds-context-wrapping

MDS: reduce usage of context wrapper

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11345 from jcsp/wip-mgr-assert

messages: fix out of range assertion

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr: PyModules.cc: remove duplicated if condition for fs_map

Signed-off-by: Weibing Zhang <zhangweibing@unitedstack.com>

test/encoding: fix the build with clang

put the overloaded operator<<() into namespace std, so clang's name
resolution is able to find it.

Signed-off-by: Kefu Chai <kchai@redhat.com>