qa/cephfs: test ec data pool

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/fs/basic_functional/clusters: more osds

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18141 from adamemerson/wip-log-time

log: Use the coarse real time clock in log timestamps

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17976 from adamemerson/wip-warning-sign

mgr,osd,kv: Fix various warnings for Clang and GCC7

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

log: Mark timestamps for coarseness and print appropriately

Have each timestamp know whether it's coarse or fine. Print six figures
for fine stamps and three for coarse.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

log: Add log_coarse_timestamps option

And set up an observer in CephContext to coarsen/refine time stamps as
requested.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

config: Add templated _get_val for use in observers

When notified by a config observer, the config lock is already held,
create a _get_val template to manipulate newstyle options when
notified.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

log: Use log_clock, defaulting to coarse time

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

log: Add optional msg parameter to create_entry and test with it

The tests should, really, get their time from the same source as the
log, and also probably use the same interface to creating entries that
other people do. So add an optional message argument and have the
tests use that rather than using the new operator directly.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

log: Create log_clock to replace use of utime_t

We create a new time type to support using coarse timestamps (for
performance) by default while still allowing the user to switch to
fine timestamps.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #18122 from liewegas/wip-monthrash-peering

qa/suites/rados/monthrash: tolerate PG_AVAILABILITY during mon thrashing

Merge pull request #18047 from gregsfortytwo/wip-20416-bitwise-assert

osd: make the PG's SORTBITWISE assert a more generous shutdown

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18049 from liewegas/wip-kv-iterator

kv/KeyValueDB: add column family

Reviewed-by: Adam Emerson <aemerson@redhat.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #18156 from theanalyst/doc-changelog-rename

doc: rename changelog with a .txt extension

Merge pull request #17442 from tchaikov/wip-test-throttle

test/throttle: kill tests exercising dtor of Throttle classes

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

doc: rename changelog with a .txt extension

Since this is used to x-ref in the release notes

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #18148 from theanalyst/doc-10.2.10-release-notes

doc: release notes update for 10.2.10

Reviewed-By: Nathan Cutler <ncutler@suse.com>
Reviewed-By: Sage Weil <sage@redhat.com>

Merge pull request #18155 from smithfarm/wip-doc-releases-tweak

doc: improve links in doc/releases.rst

doc: improve links in doc/releases.rst

Signed-off-by: Nathan Cutler <ncutler@suse.com>

doc: update releases table with v10.2.10 Jewel

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

doc: add release notes to v10.2.10 jewel

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #18153 from liewegas/wip-lts

doc/releases: drop LTS/stable line from second chart

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

doc/releases: tweaks

Signed-off-by: Sage Weil <sage@redhat.com>

doc/releases: drop LTS/stable line from second chart

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18152 from ceph/wip-fix-doc-exceptions

doc raise exceptions with a base class

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-By: Kefu Chai <kchai@redhat.com>

Merge pull request #18117 from liewegas/wip-lts

doc/releases: update release cycle docs

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

doc/releases: update release cycle docs

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18150 from xiexingguo/wip-21618

qa/standalone/scrub/osd-scrub-repair.sh: drop omap_digest flag

Reviewed-by: Sage Weil <sage@redhat.com>

doc parse two lines for getting the context

So that signatures can get parsed when they are split like:

PG::RecoveryState::RepWaitBackfillReserved::react(
  const RemoteReservationCanceled &evt)

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #18151 from theanalyst/doc-changelog-10.2.10

doc: Adding changelog for 10.2.10

doc raise exceptions with a base class

Although valid in very old Python, it is no longer possible to raise
"bare" exceptions without a class.

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc: Adding changelog for 10.2.10

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

qa/standalone/scrub/osd-scrub-repair.sh: drop omap_digest flag

This is no longer set if we are backed by bluestore, which we are by
default.  See be078c8b7b131764caa28bc44452b8c5c2339623

Signed-off-by: Sage Weil <sage@redhat.com>
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

test/common/unittest_throttle: drop tests exercising dtors

quote from Adam,

> let's just remove all the destructor tests. Since it's just exercising
> an assertion, it's most likely to be complexity that doesn't buy us
> much.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18064 from tchaikov/wip-vectors-equal

osd: replace vectors_equal() with operator==(vector<>, vector<>)

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge PR #18032 into master

* refs/pull/18032/head:
Merge commit 'refs/pull/4/head' of github.com:batrick/ceph into ptl-tool-2
script/ptl-tool.py: Label Validation
Merge commit 'refs/pull/2/head' of github.com:batrick/ceph into ptl-tool-2
Merge commit 'refs/pull/1/head' of github.com:batrick/ceph into ptl-tool-2
script/ptl-tool.py: Fix the Comment example
script/ptl-tool.py: Fixed error message
ptl-tool: make more configurable and usable

Reviewed-by: Yuri Weinstein <yweins@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

githubmap: update contributors

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge commit 'refs/pull/4/head' of github.com:batrick/ceph into ptl-tool-2

* refs/pull/4/head:
  script/ptl-tool.py: Label Validation

Merge pull request #18134 from liewegas/wip-21660

src/messages/MOSDMap: reencode OSDMap for older clients

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #17974 from jcsp/wip-doc-mgr-iptables

doc: update firewall doc to mention ceph-mgr

Reviewed-by: Kefu Chai <kchai@redhat.com>

src/messages/MOSDMap: reencode OSDMap for older clients

We explicitly select which missing bits trigger a reencode.  We
already had jewel and earlier covered, but kraken includes all of
the previously mentioned bits but not SERVER_LUMINOUS.  This
prevents kraken clients from decoding luminous maps.

Fixes: http://tracker.ceph.com/issues/21660
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18070 from liewegas/wip-21613

osd: fix replica/backfill target handling of REJECT

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

kv/RocksDBStore: fix default CF handling

We need to close the default CF handle *only* if we explicitly opened it.

Also, take care to use that handle wherever we can.  This is mostly
paranoia--I'm not sure if rocksdb cares if you mix the handle you opened
with DefaultColumnFamily(), which is different.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: make the PG's SORTBITWISE assert a more generous shutdown

We want to stop working if we get activated while sortbitwise is not set
on the cluster, but we might have old maps where it wasn't if the flag
was changed recently. And doing it in the PG code was a bit silly anyway.

Instead check SORTBITWISE in the main OSDMap handling code prior to
prepublishing it. Let it go through if we aren't active at the time.

Fixes: http://tracker.ceph.com/issues/20416
Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #17939 from mkogan1/wip-rgw-21196

rgw: update the usage read iterator in truncated scenario

Merge pull request #18028 from liewegas/wip-pick-ifname

common/pick_address: add {public,cluster}_network_interface option

Reviewed-by: Dan Mick <dmick@redhat.com>

build: Fix a couple uninitialized variable warnings

For GCC7.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

kv: Fix overloaded virtual warning in KeyValueDBMemory

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rbd_mirror: Fix overloaded virtual warning in Clang++

Just pull in the original function with a using.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

build: Turn off address-of-packed-member warning

This is newly added in recent versions of Clang and probably not
something we want to stop doing.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

{mgr,osd}: Remove pessimizing moves

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #18091 from vasilyMellanox/gid_idx_fix

msg/async/rdma: uninitialized variable fix

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #18092 from tchaikov/wip-cmake-cleanup

cmake: do not link against unused or duplicated libraries

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18094 from xiexingguo/wip-tracker-21618

qa/standalone/scrub/osd-scrub-repair.sh: add extents flag into object_info_t

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17907 from amitkumar50/cov-rgw-10

rgw: Ignoring the returned error

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #18106 from kmroz/wip-keystone-revocator-thread-assert

rgw: stop/join TokenCache revoke thread only if started.

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc/releases: drop LTS vs stable from chart; update luminous eol data

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18120 from liewegas/wip-bluefs-allocation

os/bluestore: make bluefs behave better near enospc

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #18114 from dillaman/wip-21670

librbd: list_children should not attempt to refresh image

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #18118 from liewegas/wip-21470

os/bluestore: fix another aio stall/deadlock

Reviewed-by: Varada Kari <varada.kari@gmail.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #18105 from gregsfortytwo/wip-sigpipe-porting

msg: reimplement sigpipe blocking

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17710 from Songweibin/wip-add-pybind

pybind/rados: add support open_ioctx2 API

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #18079 from liewegas/wip-21618

qa/standalone/scrub/osd-scrub-repair: no -y to diff

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #18088 from dillaman/wip-21629

osd: additional protection for out-of-bounds EC reads

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: use separate CF for omap and deferred

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: define PREFIX_ALLOC_BITMAP

This was hard-coded to 'b'.

Signed-off-by: Sage Weil <sage@redhat.com>

kv/RocksDBStore: simplify CF merge operator routing

Merge operators always map to a prefix, just like a CF, so we can link
directly to the operator without doing any lookup.

Signed-off-by: Sage Weil <sage@redhat.com>

kv/RocksDBStore: add CF support

- detect and use existing CFs on open

Signed-off-by: Jianjian Huo <jianjian.huo@ssi.samsung.com>
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/monthrash: tolerate PG_AVAILABILITY during mon thrashing

Signed-off-by: Sage Weil <sage@redhat.com>

script/ptl-tool.py: Label Validation

Validated the label and verified the existence of the label at the beginning of build_branch().
This will avoid:
* Assigning invalid testing labels like 'abcd' and other existing labels like 'bug', 'cleanup' etc; as testing labels
* Stops spamming the Label list in the Ceph repo by creating new (invalid format) labels everytime. The labels can be created manually in github and ptl-tool only uses them.

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge commit 'refs/pull/2/head' of github.com:batrick/ceph into ptl-tool-2

* refs/pull/2/head:
  script/ptl-tool.py: Fixed error message

Merge commit 'refs/pull/1/head' of github.com:batrick/ceph into ptl-tool-2

* refs/pull/1/head:
  script/ptl-tool.py: Fix the Comment example

os/bluestore: add bluestore_bluefs_min_free

We need at least ~1GB free so we can write out new SSTs (which are 256MB
each).

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: crash on enospc

We were passing this error back to rocksdb, which would then crash (much)
later with a corrupted SST file!  No good.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: use normal Context for async deferred_try_submit

I'm not quite sure why the FunctionContext did not ever execute on the
finisher thread (perhaps the [&] captured some state on the stack that it
shouldn't have?).  In any case, using a traditional Context here appears
to resolve the problem (of the async deferred_try_submit() never executing,
leading to a bluestore stall/deadlock).

Fixes: http://tracker.ceph.com/issues/21470
Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: wake kv thread when blocking on deferred_bytes

We need to wake the kv thread whenever setting deferred_aggressive to
ensure that txns with deferred io that have committed but haven't submitted
their deferred writes get submitted.  This aligns us with the other
users of deferred_aggressive (e.g., _osr_drain_all).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18099 from dillaman/wip-21628

librbd: journal should ignore -EILSEQ errors from compare-and-write

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

osd/PG: add explicit CANCEL MBackfillReserve type

This lets us distinguish between REJECT and CANCEL for clarity.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: separate event for RemoteReservationCanceled

Right now we transparently map a RemoteReservationRejected into a
*Canceled event because this what peers send over the wire.  Even
once new peers start sending and explicit CANCEL, old peers will
still do so, so we'll maintain this mapping for a while.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #17959 from gaosibei/branch2

RGW: Multipart upload may double the quota

osd/PG: separate verb 'Reject' from passive 'rejected'

This reduces pg->reject_reservation() callsites from 2 to 1 and
makes the state transitions a bit more explicit.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18087 from liu-chunmei/fix_dpdk_build

build: fix dpdk build error

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #18115 from mogeb/rados-doc

doc: add osd_max_object_size in osd configuration

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18072 from dragonylffly/wip-optimize-assert

assert(false)->ceph_abort()

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #18073 from dragonylffly/wip-check-monitor-address-configuration

mon: check monitor address configuration

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18082 from Liuchang0812/wip-21624

os/bluestore: make BlueStore opened by start_kv_only umountable

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18083 from Liuchang0812/wip-21625

tool: ceph-kvstore-tool doesn't umount BlueStore properly

Reviewed-by: Sage Weil <sage@redhat.com>

fix dpdk build error.
    when ./do_cmake.sh add  -DWITH_DPDK=ON -DUSE_CRYPTOPP=ON -DWITH_NSS=OFF -DWITH_TESTS=OFF
    if build debian package, need remove some file froms install since we disabled TESTS.

Signed-off-by: chunmei <chunmei.liu@intel.com>

doc: add osd_max_object_size in osd configuration

Having osd_max_write_size in the doc without osd_max_object_size
is confusing.

Signed-off-by: Mohamad Gebai <mgebai@suse.com>

Merge pull request #18110 from ceph/wip-volume-api-change

ceph-volume lvm api refactor/move

Reviewed By: Andrew Schoen <aschoen@redhat.com>

Merge pull request #17380 from ceph/wip-21189

mgr/status: format byte quantities in base 2 multiples

Reviewed-by: Kefu Chai <kchai@redhat.com>

librbd: list_children should not attempt to refresh image

The snap_lock is being held when this method is invoked, which can
result in a deadlock.

Fixes: http://tracker.ceph.com/issues/21670
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17746 from elliswuuuuuuuu/wip-prometheus

ceph-mgr: can not change prometheus port for mgr

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Revert "REVERT msgr: force use of the SIGPIPE stopper for testing"

This reverts commit a76181df4a71ad15aee8dd2e4adcebce8330fae4.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

REVERT msgr: force use of the SIGPIPE stopper for testing

Note that since this sets MSG_NOSIGNAL to 0 it won't take effect
on the ::send[msg] invocations

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

msgr: add a mechanism for Solaris to avoid dying on SIGPIPE

This is fairly clean: we define an RAII object in the Messenger.h on
Solaris, and "declare" it with a macro in the implementations. There's
no code duplication and on Linux it's just entirely compiled out.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Revert "SIGPIPE suppression for platforms without SO_NOSIGPIPE or MSG_NOSIGNAL"

This reverts commit 131deb39769c1187c334ee84f552d3be01f1751b. It added
code that shouldn't have entered the project repo.

Conflicts:
src/msg/async/AsyncConnection.cc
src/msg/async/AsyncConnection.h
src/msg/simple/Pipe.cc
src/msg/simple/Pipe.h

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #18089 from liewegas/wip-localpool-min-size

mgr/localpool: default to 3x; allow min_size adjustment

Reviewed-by: John Spray <john.spray@redhat.com>

mgr/localpool: fix rule selection

The 'osd pool create' arg parsing is broken; the rule name for
'ceph osd pool create $name $numpgs replicated $rulename' is passed
via the erasure_code_profile param.  Too many req=false options
without a way to disambiguate them.

Work around it by passing both 'rule' and 'erasure_code_profile'
keys, so that if/when the hack in OSDMonitor.cc is removed it will
still work.  Blech.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-volume tests move lvm api tests into its new test module

Signed-off-by: Alfredo Deza <adeza@redhat.com>