vstart.sh: move rgw configuration to client.rgw section

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

Merge pull request #18122 from liewegas/wip-monthrash-peering

qa/suites/rados/monthrash: tolerate PG_AVAILABILITY during mon thrashing

Merge pull request #18047 from gregsfortytwo/wip-20416-bitwise-assert

osd: make the PG's SORTBITWISE assert a more generous shutdown

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18049 from liewegas/wip-kv-iterator

kv/KeyValueDB: add column family

Reviewed-by: Adam Emerson <aemerson@redhat.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #18156 from theanalyst/doc-changelog-rename

doc: rename changelog with a .txt extension

Merge pull request #17442 from tchaikov/wip-test-throttle

test/throttle: kill tests exercising dtor of Throttle classes

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

doc: rename changelog with a .txt extension

Since this is used to x-ref in the release notes

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #18148 from theanalyst/doc-10.2.10-release-notes

doc: release notes update for 10.2.10

Reviewed-By: Nathan Cutler <ncutler@suse.com>
Reviewed-By: Sage Weil <sage@redhat.com>

Merge pull request #18155 from smithfarm/wip-doc-releases-tweak

doc: improve links in doc/releases.rst

doc: improve links in doc/releases.rst

Signed-off-by: Nathan Cutler <ncutler@suse.com>

doc: update releases table with v10.2.10 Jewel

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

doc: add release notes to v10.2.10 jewel

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #18153 from liewegas/wip-lts

doc/releases: drop LTS/stable line from second chart

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

doc/releases: tweaks

Signed-off-by: Sage Weil <sage@redhat.com>

doc/releases: drop LTS/stable line from second chart

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18152 from ceph/wip-fix-doc-exceptions

doc raise exceptions with a base class

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-By: Kefu Chai <kchai@redhat.com>

Merge pull request #18117 from liewegas/wip-lts

doc/releases: update release cycle docs

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

doc/releases: update release cycle docs

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18150 from xiexingguo/wip-21618

qa/standalone/scrub/osd-scrub-repair.sh: drop omap_digest flag

Reviewed-by: Sage Weil <sage@redhat.com>

doc parse two lines for getting the context

So that signatures can get parsed when they are split like:

PG::RecoveryState::RepWaitBackfillReserved::react(
  const RemoteReservationCanceled &evt)

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #18151 from theanalyst/doc-changelog-10.2.10

doc: Adding changelog for 10.2.10

doc raise exceptions with a base class

Although valid in very old Python, it is no longer possible to raise
"bare" exceptions without a class.

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc: Adding changelog for 10.2.10

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

qa/standalone/scrub/osd-scrub-repair.sh: drop omap_digest flag

This is no longer set if we are backed by bluestore, which we are by
default.  See be078c8b7b131764caa28bc44452b8c5c2339623

Signed-off-by: Sage Weil <sage@redhat.com>
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

test/common/unittest_throttle: drop tests exercising dtors

quote from Adam,

> let's just remove all the destructor tests. Since it's just exercising
> an assertion, it's most likely to be complexity that doesn't buy us
> much.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18064 from tchaikov/wip-vectors-equal

osd: replace vectors_equal() with operator==(vector<>, vector<>)

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge PR #18032 into master

* refs/pull/18032/head:
Merge commit 'refs/pull/4/head' of github.com:batrick/ceph into ptl-tool-2
script/ptl-tool.py: Label Validation
Merge commit 'refs/pull/2/head' of github.com:batrick/ceph into ptl-tool-2
Merge commit 'refs/pull/1/head' of github.com:batrick/ceph into ptl-tool-2
script/ptl-tool.py: Fix the Comment example
script/ptl-tool.py: Fixed error message
ptl-tool: make more configurable and usable

Reviewed-by: Yuri Weinstein <yweins@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

githubmap: update contributors

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge commit 'refs/pull/4/head' of github.com:batrick/ceph into ptl-tool-2

* refs/pull/4/head:
  script/ptl-tool.py: Label Validation

Merge pull request #18134 from liewegas/wip-21660

src/messages/MOSDMap: reencode OSDMap for older clients

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #17974 from jcsp/wip-doc-mgr-iptables

doc: update firewall doc to mention ceph-mgr

Reviewed-by: Kefu Chai <kchai@redhat.com>

src/messages/MOSDMap: reencode OSDMap for older clients

We explicitly select which missing bits trigger a reencode.  We
already had jewel and earlier covered, but kraken includes all of
the previously mentioned bits but not SERVER_LUMINOUS.  This
prevents kraken clients from decoding luminous maps.

Fixes: http://tracker.ceph.com/issues/21660
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18070 from liewegas/wip-21613

osd: fix replica/backfill target handling of REJECT

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

kv/RocksDBStore: fix default CF handling

We need to close the default CF handle *only* if we explicitly opened it.

Also, take care to use that handle wherever we can.  This is mostly
paranoia--I'm not sure if rocksdb cares if you mix the handle you opened
with DefaultColumnFamily(), which is different.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: make the PG's SORTBITWISE assert a more generous shutdown

We want to stop working if we get activated while sortbitwise is not set
on the cluster, but we might have old maps where it wasn't if the flag
was changed recently. And doing it in the PG code was a bit silly anyway.

Instead check SORTBITWISE in the main OSDMap handling code prior to
prepublishing it. Let it go through if we aren't active at the time.

Fixes: http://tracker.ceph.com/issues/20416
Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #17939 from mkogan1/wip-rgw-21196

rgw: update the usage read iterator in truncated scenario

Merge pull request #18028 from liewegas/wip-pick-ifname

common/pick_address: add {public,cluster}_network_interface option

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #18091 from vasilyMellanox/gid_idx_fix

msg/async/rdma: uninitialized variable fix

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #18092 from tchaikov/wip-cmake-cleanup

cmake: do not link against unused or duplicated libraries

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18094 from xiexingguo/wip-tracker-21618

qa/standalone/scrub/osd-scrub-repair.sh: add extents flag into object_info_t

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17907 from amitkumar50/cov-rgw-10

rgw: Ignoring the returned error

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #18106 from kmroz/wip-keystone-revocator-thread-assert

rgw: stop/join TokenCache revoke thread only if started.

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc/releases: drop LTS vs stable from chart; update luminous eol data

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18120 from liewegas/wip-bluefs-allocation

os/bluestore: make bluefs behave better near enospc

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #18114 from dillaman/wip-21670

librbd: list_children should not attempt to refresh image

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #18118 from liewegas/wip-21470

os/bluestore: fix another aio stall/deadlock

Reviewed-by: Varada Kari <varada.kari@gmail.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #18105 from gregsfortytwo/wip-sigpipe-porting

msg: reimplement sigpipe blocking

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17710 from Songweibin/wip-add-pybind

pybind/rados: add support open_ioctx2 API

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #18079 from liewegas/wip-21618

qa/standalone/scrub/osd-scrub-repair: no -y to diff

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #18088 from dillaman/wip-21629

osd: additional protection for out-of-bounds EC reads

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: use separate CF for omap and deferred

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: define PREFIX_ALLOC_BITMAP

This was hard-coded to 'b'.

Signed-off-by: Sage Weil <sage@redhat.com>

kv/RocksDBStore: simplify CF merge operator routing

Merge operators always map to a prefix, just like a CF, so we can link
directly to the operator without doing any lookup.

Signed-off-by: Sage Weil <sage@redhat.com>

kv/RocksDBStore: add CF support

- detect and use existing CFs on open

Signed-off-by: Jianjian Huo <jianjian.huo@ssi.samsung.com>
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/monthrash: tolerate PG_AVAILABILITY during mon thrashing

Signed-off-by: Sage Weil <sage@redhat.com>

script/ptl-tool.py: Label Validation

Validated the label and verified the existence of the label at the beginning of build_branch().
This will avoid:
* Assigning invalid testing labels like 'abcd' and other existing labels like 'bug', 'cleanup' etc; as testing labels
* Stops spamming the Label list in the Ceph repo by creating new (invalid format) labels everytime. The labels can be created manually in github and ptl-tool only uses them.

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge commit 'refs/pull/2/head' of github.com:batrick/ceph into ptl-tool-2

* refs/pull/2/head:
  script/ptl-tool.py: Fixed error message

Merge commit 'refs/pull/1/head' of github.com:batrick/ceph into ptl-tool-2

* refs/pull/1/head:
  script/ptl-tool.py: Fix the Comment example

os/bluestore: add bluestore_bluefs_min_free

We need at least ~1GB free so we can write out new SSTs (which are 256MB
each).

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: crash on enospc

We were passing this error back to rocksdb, which would then crash (much)
later with a corrupted SST file!  No good.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: use normal Context for async deferred_try_submit

I'm not quite sure why the FunctionContext did not ever execute on the
finisher thread (perhaps the [&] captured some state on the stack that it
shouldn't have?).  In any case, using a traditional Context here appears
to resolve the problem (of the async deferred_try_submit() never executing,
leading to a bluestore stall/deadlock).

Fixes: http://tracker.ceph.com/issues/21470
Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: wake kv thread when blocking on deferred_bytes

We need to wake the kv thread whenever setting deferred_aggressive to
ensure that txns with deferred io that have committed but haven't submitted
their deferred writes get submitted.  This aligns us with the other
users of deferred_aggressive (e.g., _osr_drain_all).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18099 from dillaman/wip-21628

librbd: journal should ignore -EILSEQ errors from compare-and-write

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

osd/PG: add explicit CANCEL MBackfillReserve type

This lets us distinguish between REJECT and CANCEL for clarity.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: separate event for RemoteReservationCanceled

Right now we transparently map a RemoteReservationRejected into a
*Canceled event because this what peers send over the wire.  Even
once new peers start sending and explicit CANCEL, old peers will
still do so, so we'll maintain this mapping for a while.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #17959 from gaosibei/branch2

RGW: Multipart upload may double the quota

osd/PG: separate verb 'Reject' from passive 'rejected'

This reduces pg->reject_reservation() callsites from 2 to 1 and
makes the state transitions a bit more explicit.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18087 from liu-chunmei/fix_dpdk_build

build: fix dpdk build error

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #18115 from mogeb/rados-doc

doc: add osd_max_object_size in osd configuration

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18072 from dragonylffly/wip-optimize-assert

assert(false)->ceph_abort()

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #18073 from dragonylffly/wip-check-monitor-address-configuration

mon: check monitor address configuration

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18082 from Liuchang0812/wip-21624

os/bluestore: make BlueStore opened by start_kv_only umountable

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18083 from Liuchang0812/wip-21625

tool: ceph-kvstore-tool doesn't umount BlueStore properly

Reviewed-by: Sage Weil <sage@redhat.com>

fix dpdk build error.
    when ./do_cmake.sh add  -DWITH_DPDK=ON -DUSE_CRYPTOPP=ON -DWITH_NSS=OFF -DWITH_TESTS=OFF
    if build debian package, need remove some file froms install since we disabled TESTS.

Signed-off-by: chunmei <chunmei.liu@intel.com>

doc: add osd_max_object_size in osd configuration

Having osd_max_write_size in the doc without osd_max_object_size
is confusing.

Signed-off-by: Mohamad Gebai <mgebai@suse.com>

Merge pull request #18110 from ceph/wip-volume-api-change

ceph-volume lvm api refactor/move

Reviewed By: Andrew Schoen <aschoen@redhat.com>

Merge pull request #17380 from ceph/wip-21189

mgr/status: format byte quantities in base 2 multiples

Reviewed-by: Kefu Chai <kchai@redhat.com>

librbd: list_children should not attempt to refresh image

The snap_lock is being held when this method is invoked, which can
result in a deadlock.

Fixes: http://tracker.ceph.com/issues/21670
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17746 from elliswuuuuuuuu/wip-prometheus

ceph-mgr: can not change prometheus port for mgr

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Revert "REVERT msgr: force use of the SIGPIPE stopper for testing"

This reverts commit a76181df4a71ad15aee8dd2e4adcebce8330fae4.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

REVERT msgr: force use of the SIGPIPE stopper for testing

Note that since this sets MSG_NOSIGNAL to 0 it won't take effect
on the ::send[msg] invocations

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

msgr: add a mechanism for Solaris to avoid dying on SIGPIPE

This is fairly clean: we define an RAII object in the Messenger.h on
Solaris, and "declare" it with a macro in the implementations. There's
no code duplication and on Linux it's just entirely compiled out.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Revert "SIGPIPE suppression for platforms without SO_NOSIGPIPE or MSG_NOSIGNAL"

This reverts commit 131deb39769c1187c334ee84f552d3be01f1751b. It added
code that shouldn't have entered the project repo.

Conflicts:
src/msg/async/AsyncConnection.cc
src/msg/async/AsyncConnection.h
src/msg/simple/Pipe.cc
src/msg/simple/Pipe.h

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #18089 from liewegas/wip-localpool-min-size

mgr/localpool: default to 3x; allow min_size adjustment

Reviewed-by: John Spray <john.spray@redhat.com>

mgr/localpool: fix rule selection

The 'osd pool create' arg parsing is broken; the rule name for
'ceph osd pool create $name $numpgs replicated $rulename' is passed
via the erasure_code_profile param.  Too many req=false options
without a way to disambiguate them.

Work around it by passing both 'rule' and 'erasure_code_profile'
keys, so that if/when the hack in OSDMonitor.cc is removed it will
still work.  Blech.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-volume tests move lvm api tests into its new test module

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests refactor lvm api import for test_listing

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests refactor lvm api import for api tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests refactor lvm api import for activate tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests refactor lvm api import in conftest

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume devices.lvm refactor importing api to come from api/lvm.py

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume move lvm/api.py to api/lvm.py so disk can consume it

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume create an api module to hold common api modules

Signed-off-by: Alfredo Deza <adeza@redhat.com>

script/ptl-tool.py: Fix the Comment example

The example shown in the comment needs export to work as expected.

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #18048 from badone/wip-boost-mirrors

make-dist,cmake: Try multiple URLs to download boost before failing

Reviewed-by: Kefu Chai <kchai@redhat.com>

make-dist,cmake: Try multiple URLs to download boost before failing

Remove SPOF during boost download for make-dist and later cmake > 3.7

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

rgw: stop/join TokenCache revoke thread only if started.

Thread::join triggers an assert otherwise.

Fixes http://tracker.ceph.com/issues/21666

Signed-off-by: Karol Mroz <kmroz@suse.de>

Merge pull request #18097 from dillaman/wip-21663

qa/workunits/rbd: fixed variable name for resync image id

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

librbd: journal should ignore -EILSEQ errors from compare-and-write

Fixes: http://tracker.ceph.com/issues/21628
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/standalone/scrub/osd-scrub-repair: no -y to diff

With -y you can't see the entire line when it is long, which is
needed to identify the diff failure in
http://tracker.ceph.com/issues/21618

Instead, let the interactive user specify the option if they want it.

Signed-off-by: Sage Weil <sage@redhat.com>