Merge PR #24098 into master

* refs/pull/24098/head:
Rename "include/assert.h"

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #24112 into master

* refs/pull/24112/head:
osd: avoid unncecessary ref-counting in OSD::enqueue_op.

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #24090 from cbodley/wip-crimson-concurrent-dispatch

crimson/net: concurrent dispatch for SocketMessenger

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #24092 into master

* refs/pull/24092/head:
mgr: Change signature of PyString_AsString to match return

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #24057 into master

* refs/pull/24057/head:
src/common: add a unit test (bufferlist.sha1())
osd, src/common: return sha1 value if zero-length buffer.
src/common/buffer.cc: remove unnecessary copy in sha1()

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #24072 into master

* refs/pull/24072/head:
osd/PG: kill __queue_query__
osd/OSD: deprecate __project_pg_history__

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #24104 into master

* refs/pull/24104/head:
mgr/diskprediction: add prototype diskprediction module

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #24103 from b-ranto/wip-dashboard-http

mgr/dashboard: Do not require cert for http

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #24015 from ricardoasmarques/wip-update-selection-on-refresh

mgr/dashboard: More configs for table `updateSelectionOnRefresh`

Reviewed-by: Kanika Murarka <kmurarka@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Rename SSLCherryPyConfig

The class does not always set up SSL, dropping the SSL bit.

Fixes: http://tracker.ceph.com/issues/36069
Signed-off-by: Boris Ranto <branto@redhat.com>

mgr/dashboard: Do not require cert for http

The ceph dashboard currently requires a SSL certificate even if it is
not running in the SSL mode since it is always querying for the
certificate file/key pair.

This patch fixes the behaviour by querying for the certificate file/key
only if it is running in the SSL mode.

Fixes: http://tracker.ceph.com/issues/36069
Signed-off-by: Boris Ranto <branto@redhat.com>

Merge pull request #24031 from rzarzynski/wip-common-bl-drop-0c

common: drop the unused zero-copy facilities in ceph::bufferlist

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #24120 into master

* refs/pull/24120/head:
doc: Fix Typos of Developer Guide

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #24105 from dillaman/wip-24516

librbd: object map improperly flagged as invalidated

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #23981 from cbodley/wip-35830

rgw: raise debug level on redundant data sync error messages

Merge pull request #24088 from dzafman/wip-35982

qa/standalone: Standalone test corrections

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

common: get_data() of buffer_raw isn't virtual anymore.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common: make the get_data() of buffer_raw interface final.

This is just to ensure the just dropped buffer::raw_pipe
was the solely user of this facility. After successful
validation, we can drop `virtual` on the method entirely.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common: drop can_zero_copy() from ceph::buffer interfaces.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common: drop zero_copy_to_fd() from ceph::buffer interfaces.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common: drop write_fd_zero_copy() from bufferlist.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common: drop buffer::raw_pipe altogether with its factory method.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common: drop buffer::list::read_fd_zero_copy.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common, test: drop unit tests for 0c in ceph::bufferlist.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mgr/diskprediction: add prototype diskprediction module

This module is written by Rick Chen <rick.chen@prophetstor.com> and
provides both a built-in local predictor and a cloud mode that queries
a cloud service (provided by ProphetStor) to predict device failures.

Signed-off-by: Rick Chen <rick.chen@prophetstor.com>
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #24087 from rzarzynski/wip-common-buffer_raw-drop-page-align-getters

common: drop/mark-as-final getters of buffer::raw for palign.

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #24061 from liewegas/wip-35923

osd/PG: write pg epoch when resurrecting pg after delete vs merge race

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #23306 from penglaiyxy/msg_branch

msg: ceph_abort() when there are enough accepter errors in msg server

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23955 from wjwithagen/wjw-fix-ceph-helpers.sh

test: Start using GNU awk and fix archiving directory

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: Fix Typos of Developer Guide

Signed-off-by: Li Bingyang <li.bingyang1@zte.com.cn>

Merge PR #24023 into master

* refs/pull/24023/head:
mds: prevent rctime from going back

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #24022 into master

* refs/pull/24022/head:
client: update ctime when modifying file content

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #24048 into master

* refs/pull/24048/head:
mds: use common send message mechanisms
mds: remove dead code

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #24053 into master

* refs/pull/24053/head:
mds: translate CEPH_MDS_STATE_NULL state name as null

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #21992 into master

* refs/pull/21992/head:
qa: extend timeout for SessionMap flush

Reviewed-by: John Spray <john.spray@redhat.com>

librbd: do not invalidate object map when attempting to delete non-existent snapshot

If duplicate snapshot remove requests are received by the lock owner from a peer
client, the first request will remove the object map. If the second request
arrives while the first is in-progress, it will again attempt to remove the
object map but fail to load it since it's already been deleted. This incorrectly
results in the next object map being flagged as invalid.

Fixes: http://tracker.ceph.com/issues/24516
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: converted object map snapshot remove state machine to new style

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #24029 from smithfarm/wip-35922

cmake: no libradosstriper headers if WITH_LIBRADOSSTRIPER=OFF

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jesse Williamson <jwilliamson@suse.de>

Merge pull request #24036 from dillaman/wip-24688

rbd-mirror: optionally extract peer secrets from config-key

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #24063 from dillaman/wip-24140

rbd-mirror: prevent creation of clones when parents are syncing

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #24108 from clebergnu/rm_qa_btrfs_binary

qa/btrfs/test_rmdir_async_snap: remove binary file

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23989 from dzafman/wip-35846

tools: ceph-objectstore-tool: Allow target level as first positional argument

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

osd: avoid unncecessary ref-counting in OSD::enqueue_op.

```
-    4,30%     0,32%  msgr-worker-0  ceph-osd  [.] OSD::enqueue_op
   - 3,98% OSD::enqueue_op
      + 2,40% OSD::ShardedOpWQ::_enqueue
      + 0,42% OpRequest::mark_flag_point
        0,37% operator new
        0,36% PerfCounters::tinc
      + 0,21% Mutex::Unlock
```

Annotate:
```
OSD::enqueue_op  /home/perf/src/ceph-rzarzynski3/build/bin/ceph-osd
       │93           if( px != 0 ) intrusive_ptr_add_ref( px );
       │       test   %rbx,%rbx
       │95   _ZN13PGOpQueueableC4E5spg_t():
       │175    explicit PGOpQueueable(spg_t pg) : pgid(pg) {}
  0,70 │       mov    %cl,0x18(%rax)
       │177  _ZN8PGOpItemC4E5spg_tN5boost13intrusive_ptrI9OpRequestEE():
       │204    PGOpItem(spg_t pg, OpRequestRef op) : PGOpQueueable(pg), op(op) {
       │       mov    %rdx,(%rax)
       │206  _ZN5boost13intrusive_ptrI9OpRequestEC4ERKS2_():
       │     ↓ je     166
       │94   _ZNSt13__atomic_baseIiEppEv():
 48,25 │       lock   addl   $0x1,0x30(%rbx)
       │166:   mov    0x158(%rbp),%ecx
       │       mov    0x15c(%rbp),%edx
       │299  _ZN17ShardedThreadPool9ShardedWQI11OpQueueItemE5queueEOS1_():
       │669        tp->set_wq(this);
       │670      }
       │671      ~ShardedWQ() override {}
       │
       │673      void queue(T&& item) {
       │674        _enqueue(std::move(item));
  2,80 │       mov    %r12,%rsi
```

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/net: messenger uses dispatch gate for authorizer callbacks

this also removes the checks for null dispatcher - start() must be
called with a valid dispatcher before getting here

Signed-off-by: Casey Bodley <cbodley@redhat.com>

crimson/net: messenger uses dispatch gate for accept/connect callbacks

the dispatcher can throw to reject a new connection, so these callbacks
were moved ahead of the exception handling

Signed-off-by: Casey Bodley <cbodley@redhat.com>

crimson/net: messenger uses dispatch gate for reset callbacks

Signed-off-by: Casey Bodley <cbodley@redhat.com>

crimson/net: add unit test for concurrent dispatch

Signed-off-by: Casey Bodley <cbodley@redhat.com>

crimson/net: concurrent dispatch for SocketMessenger

Signed-off-by: Casey Bodley <cbodley@redhat.com>

crimson/net: don't return null from Connection::read_message()

SocketConnection::read_message() now loops until it has a message with
valid sequence number. this means SocketMessenger::dispatch() doesn't
have to handle the null message case

Signed-off-by: Casey Bodley <cbodley@redhat.com>

crimson/net: use std::optional instead of boost

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/btrfs/test_rmdir_async_snap: remove binary file

On a quick look at the source code, I noticed this binary file, which
looks like was committed by mistake.

Signed-off-by: Cleber Rosa <crosa@redhat.com>

Merge pull request #23650 from tchaikov/wip-dpdk-cmake

cmake: fix the build WITH_DPDK=ON

Reviewed-by: Casey Bodley <cbodley@redhat.com>

librbd: test_flags helper should require snap id parameter

The HEAD and snapshots have potentially different flag states
since object maps get invalidated per revision.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #24097 from cyx1231st/fix-seastar-msgr-read-racing

crimson/net: Fix racing for promise on_message

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/net: Fix racing for promise on_message

read_tags_until_next_message() will forward the ready future and create
a new promise for on_message, which assumes there is already a
read_message() holding the previous promise, but it is not true.

Signed-off-by: Yingxin <yingxin.cheng@intel.com>

Merge pull request #23972 from badone/wip-documentation-add-12.2.8-release

doc/releases/schedule: Add Luminous 12.2.8

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #24099 from badone/wip-mimic-13-2-0-date-correction

doc/releases: Mark Mimic first release as June

Reviewed-by: Nathan Cutler <ncutler@suse.com>

osd/PG: kill __queue_query__

which has no consumbers...

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/OSD: deprecate __project_pg_history__

__project_pg_history__ does an inverse traverse of the series
of osdmaps passed in to get a pg's pg_history_t filled, which
can become super inefficient if the osdmap list to check is
very long.
E.g., in one of our clusters, we've observed it took approximate
10s for a PG to finish it's projecting:
```
2018-08-27 13:51:58.694823 7f1e1335a700 15 osd.9 823276 project_pg_history 34.6e9 from 821893 to 823276, start ec=380829/380829 l
is/c 820412/820412 les/c/f 820413/820413/0 821785/821785/821785
2018-08-27 13:52:08.634230 7f1e1335a700 15 osd.9 823276 project_pg_history 34.6e9 acting|up changed in 822265 from [57]/[57] 57/5
7 -> [58,57]/[58,57] 58/58
2018-08-27 13:52:08.634244 7f1e1335a700 15 osd.9 823276 project_pg_history 34.6e9 up changed in 822265 from [57] 57 -> [58,57] 58
2018-08-27 13:52:08.634248 7f1e1335a700 15 osd.9 823276 project_pg_history 34.6e9 primary changed in 822265
2018-08-27 13:52:08.634250 7f1e1335a700 15 osd.9 823276 project_pg_history end ec=380829/380829 lis/c 820412/820412 les/c/f 82041
3/820413/0 822265/822265/822265
```
Quote from Sage:
> let's just kill this off entirely, and make the handle_pg_query_nopg
reply unconditionally. Or, maybe, do a single sloppy check to see if
the primary has changed since the original epoch... if the osdmap
happens to be in cache... or not.
The querying end will discard the reply if it is out of date from it's
perspective, so it doesn't matter, and I suspect the overhead of doing
the check is larger than the overhead of sending a query reply that gets ignored.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
Signed-off-by: yanjun <yan.jun8@zte.com.cn>

doc/releases: Mark Mimic first release as June

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Rename "include/assert.h"

This conflicts with the system assert.h so rename and change includes to
reflect the new name.

Fixes: http://tracker.ceph.com/issues/35682
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #24094 from libingyang-zte/master

doc: Fix Typos of Developer Guide

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

doc: Fix Typos of Developer Guide

Signed-off-by: Li Bingyang <li.bingyang1@zte.com.cn>

mgr: Change signature of PyString_AsString to match return

PyUnicode_AsUTF8 now returns 'const char*'

Fixes: http://tracker.ceph.com/issues/35984
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

common: drop is_page_aligned() of buffer::raw entirely.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

test: osd-backfill-stats.sh: Fix subtests to get primary which can change

Fixes: http://tracker.ceph.com/issues/35982
Signed-off-by: David Zafman <dzafman@redhat.com>

test: Fix for error message changed in ceph-objectstore-tool

Fixes: http://tracker.ceph.com/issues/35982
Caused by: 6bd682f53dfe0b2f7c31b5c1ba081afb72f1dd6c

Signed-off-by: David Zafman <dzafman@redhat.com>

common: drop/mark-as-final getters of buffer::raw for palign.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #23896 from Songweibin/wip-group-snap-rollback

rbd: add group snap rollback method

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #24077 from alfredodeza/wip-rm35970

ceph-volume  fix zap not working with LVs

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume tests.functional comment why not use --destroy when zapping lvs in filestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional use --destroy when zapping lvs in bluestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

qa/workunits/rbd: new test for rbd-mirror peer config-key secrets

Fixes: http://tracker.ceph.com/issues/24688
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

doc/rbd: update mirroring to include new peer secret config-key actions

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mon: added bootstrap-rbd-mirror/rbd-mirror mon profiles

The rbd-mirror daemon will require permission to read config-keys
from the "rbd/mirror/" prefix. These new profiles (and the new
associated bootstrap-rbd-mirror user) provide the required
permissions.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: mirror pool peer commands now support mon_host/key overrides

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #24024 from p-na/fix-flaky-qa-tests

mgr/dashboard: Fix flaky QA tests

Reviewed-by: Ricardo Dias <rdias@suse.com>

ceph-volume lvm.zap use remove_pv only for physical devices

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #23660 from falcon78921/wip-multisite-docs

doc: updated multisite documentation

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23995 from tchaikov/wip-add-ceph-libboost-repo

install-deps,run-make-check: use ceph-libboost repo

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Erwan Velu <erwan@redhat.com>

src/common: add a unit test (bufferlist.sha1())

Signed-off-by: Myoungwon Oh <omwmw@sk.com>

Merge pull request #24067 from libingyang-zte/master

doc: Fix typos in Developer Guide

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd, src/common: return sha1 value if zero-length buffer.

Signed-off-by: Myoungwon Oh <omwmw@sk.com>

doc: Fix Spelling Error in Developer Guide

Signed-off-by: Li Bingyang <li.bingyang1@zte.com.cn>

Merge pull request #23919 from adamemerson/wip-end-to-warnings

Clean up warnings

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24018 from dzafman/wip-35912

qa/standalone: Minor test improvements

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24060 from alfredodeza/wip-rm35913

ceph-volume batch: allow --osds-per-device, default it to 1

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

qa: extend timeout for SessionMap flush

Fixes: https://tracker.ceph.com/issues/24129
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-volume tests.lvm.strategies initial filestore validation

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.tests conftest fixtures for custom vgs

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.devices.lvm add bluestore strategy cases

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests verify strategy validators for minimum size

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.batch.filestore allow osds-per-device

Signed-off-by: Alfredo Deza <adeza@redhat.com>

rbd-mirror: connect to remote cluster using optional mon_host/key values

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: prevent creation of clones when parents are syncing

This will prevent a possible race condition where a thrashing rbd-mirror
daemon in mid-sync with a parent image would result in the deletion of
all snapshots when it restarts the sync.

Fixes: http://tracker.ceph.com/issues/24140
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls_cas: Fix format string warning

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge PR #24025 into master

* refs/pull/24025/head:
mon/OSDMonitor: reject pg_num -> 0

Reviewed-by: huangjun <huangjun@xsky.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #21611 into master

* refs/pull/21611/head:
osd/ECBackend: don't get result code of subchunk-read overwritten

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge PR #23273 into master

* refs/pull/23273/head:
os/bluestore: handle spurious read errors

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #23694 into master

* refs/pull/23694/head:
mon: remove pre-luminous health output compatibility

Reviewed-by: Sage Weil <sage@redhat.com>