librbd: write-around cache should honor FUA op flag

If force unit access is specified, the cache should be bypassed just
like other non-optimized IO through the cache.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: support new bench 'full-seq' io pattern option

This ensures IOs are issued in sequential order regardless of the current
IO thread setting.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: enable the simple IO scheduler by default

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: switch to write-around cache policy by default

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: new alternative write-around cache layer

This is a replacement for the ObjectCacher-based writethrough/
writeback cache. It permits a configurable maximum byte size
of in-flight writes (bytes) to the OSDs where the writes are
immediately completed back to the caller. Any IO errors are
propagated on the next flush request.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: pass journal tid through flush object dispatch spec

This permits moving the journal flush and commit logic down to the
journal object dispatch layer.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: remove special case for journal replay via writeback

With the future addition of a writearound cache (and to a much
lesser extent the IO scheduler), the code will become easier
to maintain via a single IO path that periodically injects IO
flushes to ensure the data is consistently committed to disk.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: moved object cacher writeback handler to cache sub-directory

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

common/buffer: hexdump should skip lines that are the same as previous

Previously, it would only skip similar lines if the data was all zeroed.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #27504 from tchaikov/wip-qa-use-python36-for-py3-tests

qa/tasks/ceph_deploy: install python3.6 instead of python3.4 for py3 tests

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

Merge PR #27323 into master

* refs/pull/27323/head:
msg/async: avoid creating unnecessary AsyncConnectionRef

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #27444 from ifed01/wip-ifed-fix-21312

os/bluestore: call fault_range prior to looking for blob to reuse

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #27445 from ifed01/wip-ifed-fix-39143

os/ceph-bluestore-tool: bluefs-bdev-expand asserts if no WAL

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #27438 from Devp00l/wip-issue-37701

mgr/dashboard: Removes distracting search behavior

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #27348 from Devp00l/wip-issue-39086

mgr/dashboard: Enable read only users to read again

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

qa/tasks/ceph_deploy: install python3.6 instead of python3.4 for py3 tests

EPEL7 has switched over to python3.6 as the main python3. and we started
packaging python bindings for python3.6 since
https://github.com/ceph/ceph-build/pull/1283

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #27495 into master

* refs/pull/27495/head:
doc/releases/nautilus: fix config update step

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc/releases/nautilus: fix config update step

'ceph ... > /etc/ceph/ceph.conf' fails because bash truncates ceph.conf
before the ceph cli invocation is able to use it to connect to the cluster.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #27290 from sebastian-philipp/rook-system-namespace

mgr/rook: Support other system namespaces

Reviewed-by: Blaine Gardner <blaine.gardner@suse.com>
Reviewed-by: Rohan Gupta <rohanrgupta1996@gmail.com>
Reviewed-by: Travis Nielsen <tnielsen@redhat.com>

Merge pull request #27408 from nathan-weinberg/reformat

mgr/dashboard: Updated existing E2E tests to match new format

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge PR #27490 into master

* refs/pull/27490/head:
PendingReleaseNotes: note on python3.6 changes

Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

PendingReleaseNotes: note on python3.6 changes

Fixes: http://tracker.ceph.com/issues/39164
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27406 from rhcs-dashboard/fix-39125-master

mgr/dashboard: Add whitelist to guard

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #27289 from sebastian-philipp/rook-fix-osd-create

mgr/rook: Fix creation of bluestore OSDs

Reviewed-by: Blaine Gardner <blaine.gardner@suse.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Travis Nielsen <tnielsen@redhat.com>

Merge PR #27464 into master

* refs/pull/27464/head:
msg/async/frames_v2.h: fix warning

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #27357 from dillaman/wip-39021

librbd: fixed several race conditions related to copyup

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #27466 from wjwithagen/wjw-fix-/src/test/rgw/CMakeLists.txt

build/rgw: unittest_rgw_dmclock_scheduler does not need Boost_LIBRARIES

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27468 from tchaikov/wip-rpm-python3

rpm: add "Provides: python3-*" for python packages and cleanup

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

rpm: add "Provides: python3-*" for python packages

so user can install python3-rados, instead of python36-rados, without
specifying the minor version of python. also, we should not break our
teuthology tests with this naming scheme change. for instance, our
cephfs qa suite installs `python3-cephfs` for testing the `cephfs-shell`

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27309 from pritha-srivastava/wip-rgw-non-existent-object

rgw: Evaluating bucket policies also while reading permissions for an…

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

rpm: always use 3 as the default python3_{pkgversion,version}

as we have the rpm macros installed in
https://github.com/ceph/ceph-build/pull/1283 .

Signed-off-by: Kefu Chai <kchai@redhat.com>

build/rgw: unittest_rgw_dmclock_scheduler does not need Boost_LIBRARIES

Redo fix, also see #26799

Otherwise linking could error like:
```
/usr/local/bin/ld: /usr/local/lib/libboost_python27.so: undefined reference to `PyUnicodeUCS4_FromEncodedObject'
/usr/local/bin/ld: /usr/local/lib/libboost_python27.so: undefined reference to `PyNumber_InPlaceDivide'
........
/usr/local/bin/ld: /usr/local/lib/libboost_python27.so: undefined reference to `PyStaticMethod_Type'
/usr/local/bin/ld: /usr/local/lib/libboost_python27.so: undefined reference to `PyTuple_Size'
c++: error: linker command failed with exit code 1 (use -v to see invocation)
gmake[2]: *** [src/test/rgw/CMakeFiles/unittest_rgw_dmclock_scheduler.dir/build.make:147: bin/unittest_rgw_dmclock_scheduler] Error 1
```

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

msg/async/frames_v2.h: fix warning

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #27455 from wjwithagen/wjw-fix-test_weighted_shuffle.cc

test: Add `#include <array>`

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27431 from sebastian-philipp/rook-doc-update-mon

doc/mgr/orchestrator_cli: Rook orch supports mon update

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge pull request #25985 from l-vo/update_php_doc

doc/radosgw: update documentation examples with the current S3 PHP client

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge PR #27051 into master

* refs/pull/27051/head:
mds: open import bounding dirfrags in batch
mds: remove superfluous error in StrayManager::advance_delayed()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #27412 into master

* refs/pull/27412/head:
doc: add doc for blacklisting older CephFS clients

Reviewed-by: Barbora Ančincová <bara@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #27194 from trociny/wip-38661

librbd: do not allow to deep copy migrating image

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #27268 from trociny/wip-38364

librbd: support EC data pool images sparsify

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: Enable read only users to read again

The dashboards Prometheus receiver API needed to receive a POST from the
frontend, but POSTs aren't allowed by default by any read only
users. As a result the receiver API call had thrown a 403 error which
redirected the user to the 403 error page. Now you can get the last
notifications via GET. This prevents the redirection for read only
users, as a result they can get the last notifications and also
see all other allowed pages again.

Fixes: https://tracker.ceph.com/issues/39086
Signed-off-by: Stephan Müller <smueller@suse.com>

Merge pull request #27452 from tchaikov/wip-crimson-auth-server

crimson: move dummy impl of AuthServer to DummyAuth

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mgr/dashboard: Add whitelist to guard

After PR https://github.com/ceph/ceph/pull/26572, when RGW is not
  configured, accessing /rgw drop-down (daemons, users or buckets)
  results in nothing apparently happening (not even an error).

  Under the curtains, what is happening is that the ModuleStatusGuard
  has redirected the route to the rgw/501, but as this route is now
  under parent rgw route handler, which sets CanActivateChild guards,
  this results in a new ModuleStatusGuard invokation, a subsequent
  failure and a new redirection to rgw/501.

  Several approaches could be taken here:
  - Remove error pages from lazy-loaded modules. Probably it does not
  make sense to have a 501 page per component.
  - Add some whitelist to avoid this kind of loop (e.g.: 501, or any
      error page).
  - Set a max number of redirections (cautionary measure).

Fixes: https://tracker.ceph.com/issues/39125
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

mgr/rook: Support other system namespaces

Fixes http://tracker.ceph.com/issues/38799

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

test: Add includefile for arrays

Otherwise Clang will complain on FreeBSD:
```
/home/jenkins/workspace/ceph-master/src/test/test_weighted_shuffle.cc:9:23: error: implicit instantiation of undefined template 'std::__1::array<char, 5>'
  std::array<char, 5> choices{'a', 'b', 'c', 'd', 'e'};
                      ^
/usr/include/c++/v1/__tuple:223:64: note: template is declared here
template <class _Tp, size_t _Size> struct _LIBCPP_TEMPLATE_VIS array;
                                                               ^
/home/jenkins/workspace/ceph-master/src/test/test_weighted_shuffle.cc:10:22: error: implicit instantiation of undefined template 'std::__1::array<int, 5>'
  std::array<int, 5> weights{100, 50, 25, 10, 1};
                     ^
/usr/include/c++/v1/__tuple:223:64: note: template is declared here
template <class _Tp, size_t _Size> struct _LIBCPP_TEMPLATE_VIS array;
```

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

crimson/mon: implement AuthServer methods

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/auth: move dummy impl of AuthServer to DummyAuth

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27416 from tchaikov/wip-crimson-auth

crimson: update osd when peer gets authenticated

Reviewed-by: Samuel Just <sjust@redhat.com>

crimson: update osd when peer gets authenticated

* common/auth_handler.h: add an abstract class of AuthHandler, the one who is interested in
  an authenticated peer should implement this class
* mon/MonClient: let mon::Client implement AuthServer, as it has access the keyring. it
  will update the registered AuthHandler if the client (peer) is
  authenticated.
* osd: implement AuthHandler class. we will keep track of the connected
  sessions along their caps in a follow-up change.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27426 from tchaikov/wip-crimson-setup-msgr

crimson/osd: consolidate the code to initialize msgrs

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #27286 from tchaikov/wip-strncpy-with-nul

osdc/Objecter: always add `\0' after strncpy()

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #27429 from cyx1231st/wip-seastar-msgr-perf2

test/crimson: include writes in perf_crimson/async_server

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/crimson: wrap perf parameters as structured data

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

test/crimson: perf_crimson/async_server write test support

Allow server to have write/send workload with specified block size.

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

Merge pull request #27422 from iotcg/doc_pg

doc: fix parameter to set pg autoscale mode

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #27427 into master

* refs/pull/27427/head:
cephfs-shell: Use colorama module instead of colorize

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #27369 into master

* refs/pull/27369/head:
docs: Add note to execute cephfs-shell

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #27256 into master

* refs/pull/27256/head:
mds: drop reconnect message from non-existent session

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #27214 into master

* refs/pull/27214/head:
mds/server: check directory split after rename.

Reviewed-by: Zheng Yan <zyan@redhat.com>

msg/async: avoid creating unnecessary AsyncConnectionRef

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #27415 into master

* refs/pull/27415/head:
qa: decouple session map test from simple msgr
msg/async: move connection ref
msg/async: dec active connections when marked down

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #27414 into master

* refs/pull/27414/head:
msg/async: resolve gcc warning

Reviewed-by: Ricardo Dias <rdias@suse.com>

qa: decouple session map test from simple msgr

Instead of looking at the number of threads (used by the simple messenger) to
judge the coming and going of connections, use the (async) messenger perf
counters.

Plus some other minor improvements.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

msg/async: move connection ref

Trivial optimization.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

msg/async: dec active connections when marked down

Otherwise, tests can't tell when a connection is stopped until it's eventually
"lazily" deleted. This should be safe since the perf counter is manipulating an
atomic value.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

os/ceph-bluestore-tool: bluefs-bdev-expand asserts if no WAL

Fixes: https://tracker.ceph.com/issues/39143
Signed-off-by: Igor Fedotov <ifedotov@suse.com>

doc: add doc for blacklisting older CephFS clients

Fixes: http://tracker.ceph.com/issues/39130
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

librbd: force a copyup object map update if migration completes

If a live-migration copyup was in-progress when the copyup completes,
force the object map to be updated.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: cleaned up snap/parent lock release in read-from-parent path

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test/librbd: initial unit test for copyup state machine

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: copyup should restart delayed ops against the same object

This avoids the potential for a race condition where an in-flight
copyup is removed from the in-flight copyup list and a subsequent
IO against the same object causes a second in-flight copyup.

Fixes: http://tracker.ceph.com/issues/39021
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: merge copyup object map update states

The object map HEAD and HEAD/snapshot update states have been
simplified and merged into a single state. This also fixes
several potential race conditions and an issue where CoR might
incorrectly mark the HEAD object has exists+dirty.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: refactor copyup state machine to match current code style

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: properly hold snap/parent locks during IO

The ImageCtx::parent pointer was dereferenced without holding the lock
which could lead to a crash. The ImageCtx::migration_info structure
was also dereferenced without holding a lock.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: empty IO requests should complete in new context

This will avoid the possibility of a lock being held by the caller
and having the IO immediately complete back to the calling state
machine.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: deep-copy object copy should register an in-flight op

When handling live migrations, the source image is the parent
image of the destination image. To prevent the parent image from
being closed while a request is in-flight, the object copy
state machine now registers an async operation with the source
image.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: deep-copy object copy should delay computing object existence

This prevent a potential recursive lock on the destination image's
snap_lock.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

os/bluestore: call fault_range properly prior to looking for blob to reuse

Fixes: https://tracker.ceph.com/issues/21312
Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #27281 into master

* refs/pull/27281/head:
script/ceph-release-notes: alternate merge commit format

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #26782 from iotcg/master

vstart.sh: fix CEPH_PORT check and cleanups

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #27437 into master

* refs/pull/27437/head:
vstart: add an alias for cephfs-shell to vstart_environment.sh

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #27419 into master

* refs/pull/27419/head:
common: add --log-early command line option

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27091 from yuvalif/s2_pubsub_api_new

S3 compatible pubsub API

Merge PR #27374 into master

* refs/pull/27374/head:
mgr/volume: set cephfs metadata bias at 4x
mgr/volume: default to 16 PGs (min) for metadata pool

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #27349 into master

* refs/pull/27349/head:
qa/distros/supported/ubuntu_latest: 16.04 -> 18.04
qa/distros/supported/centos_latest: 7.5 -> 7.6
qa/distros: add centos 7.6

Reviewed-by: David Galloway <dgallowa@redhat.com>

Merge PR #27386 into master

* refs/pull/27386/head:
os/filestore/FileJournal: note EIO events
os/filestore: make note of EIO errors when we see them
os/filestore: note devname for later use
global/signal_handler: avoid core dump on EIO
os/bluestore/KernelDevice: note EIO metadata on aio EIO
global: add hook to annotate crash report with EIO information

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #27337 into master

* refs/pull/27337/head:
msg/async: add timeout for connections which are not yet ready
msg: rename ms_tcp_read_timeout to ms_connection_idle_timeout

Reviewed-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: Removes distracting search behavior

The behavior was to not search anything if the search included a
subset of a column name, with the thought in mind that the user would
like to search a specific column for a value.

Now the column name is only ignored after you typed in ':' in order to see
the data in the table again before specifying a more concrete value to
search for.

Fixes: https://tracker.ceph.com/issues/37701
Signed-off-by: Stephan Müller <smueller@suse.com>

vstart: add an alias for cephfs-shell to vstart_environment.sh

Signed-off-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #27341 from liewegas/wip-learn-addr-from-peer

msg/async/ProtocolV[12]: add ms_learn_addr_from_peer

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #27340 from tchaikov/wip-cmt-more-chatty

ceph-monstore-tool: print out caps when rebuilding monstore

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #27352 from liewegas/wip-deferred-log-start

common: start logging for non-global_init users

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27324 from batrick/async-msgr-clear

msg/async: use faster clear method to delete containers

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #26806 from xiexingguo/wip-repair-eio-rep

osd: automatically repair replicated replica on pulling error

Reviewed-by: David Zafman <dzafman@redhat.com>

mgr/rook: Fix creation of bluestore OSDs:

See https://rook.io/docs/rook/v0.9/ceph-cluster-crd.html
    Storage Configuration: Specific devices

Fixes http://tracker.ceph.com/issues/39062

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

docs: Add note to execute cephfs-shell

Signed-off-by: Varsha Rao <varao@redhat.com>

doc/mgr/orchestrator_cli: Rook orch supports mon update

Fixes: http://tracker.ceph.com/issues/39137
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

cephfs-shell: Use colorama module instead of colorize

Remove references to colorize attributes. As in cmd2 module, colorize attribute
is deleted. Instead use colorama module.

Fixes: https://tracker.ceph.com/issues/38996
Signed-off-by: Varsha Rao <varao@redhat.com>

crimson/osd: consolidate the code to initialize msgrs

no need to repeat them in difference places. we should just ready them
in main.cc where they are created.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27417 from tchaikov/wip-rpm-python36-el7

rpm: use python 3.6 as the default python3

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>