Merge PR #27010 into master

* refs/pull/27010/head:
mon/OSDMonitor: allow 'osd pool set pgp_num_actual'

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #26812 into master

* refs/pull/26812/head:
msg/async V2: optimize check state by replace.

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #27006 from tchaikov/wip-crimson-os

crimson/os: do not fail if fsid file exists when mkfs

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #26996 from neha-ojha/wip-2-36739

osd/PG: skip rollforward when !transaction_applied during append_log()

Reviewed-by: Xie Xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #26056 into master

* refs/pull/26056/head:
mds: check earlier if directories are already exported
mds: dont print auth trees if they are too many
mds: dont print subtrees if they are too many/big

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #26662 into master

* refs/pull/26662/head:
qa: update kclient testing to RHEL 7.6

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #26817 into master

* refs/pull/26817/head:
mds: avoid revoking Fsx from loner during directory fragmentation

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #26895 into master

* refs/pull/26895/head:
mds: convert unnecessary list usage to vector
mds: convert get_*dirfrags to use vector
mds: convert iterator loop to generic loop
mds: list to std::list
mds: constantize dump_loads

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #26931 from trociny/wip-38706

librbd: make flush be queued by QOS throttler

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #27049 from trociny/wip-rbd-deep-copy-cleanup

librbd: fix typo in deep_copy::ObjectCopyRequest::compute_read_ops

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

crimson/os: do not fail if fsid file exists when mkfs

instead of passing a random uuid down to CyanStore::mkfs(), let it
create one if necessary.

Signed-off-by: chunmei Liu <chunmei.liu@intel.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: implement CyanStore::get_attr()

it is used for retrieving object_info and snapset attributes.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #27042 into master

* refs/pull/27042/head:
qa: add krbd_discard_granularity.t test
qa: update and rename krbd_discard_1b.t

Reviewed-by: Dongsheng Yang <dongsheng.yang@easystack.cn>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #27004 from tchaikov/wip-crimson-osdmap

crimson/osd: crimson/osd: do not load fullmap.0

Reviewed-by: Liu-Chunmei <chunmei.liu@intel.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

librbd: fix typo in deep_copy::ObjectCopyRequest::compute_read_ops

The second arg for interval_set insert method is the inserting
interval lenth, while the end position was provided. It still
worked correctly, because the end position value is always larger
than the truncated length.

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #26973 from smithfarm/wip-nautilus-backports

script: enable nautilus in backport scripts

Reviewed-by: Sebastian Wagner <swagner@suse.com>

qa: add krbd_discard_granularity.t test

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge PR #27019 into master

* refs/pull/27019/head:
doc/releases/nautilus: add reference to msgr2 config update section
doc/cephfs/nfs: update ceph.conf example
doc/releases/nautilus: final upgrade note updates

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26772 from sebastian-philipp/orchestrator-rm-stateful-rule

mgr/orchestrator: Remove `(add|test|remove)_stateful_service_rule`

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Tim Serong <tserong@suse.com>

Merge tag 'v14.2.0'

v14.2.0

Merge pull request #26977 from p-na/wip-pna-fix-tox-env-vars

mgr/dashboard: Fix env vars of `run-tox.sh`

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26095 from bbc/s3secretcache

rgw: Added caching for S3 credentials retrieved from keystone

Merge pull request #26564 from theanalyst/wip-reshard-lc-fixes

rgw: LC: handle resharded buckets

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27001 from linuxbox2/wip-rgw-delimchar

rgw: fix signed char truncation in delimiter check

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa: update and rename krbd_discard_1b.t

Passing 1 for alloc_size is no longer allowed.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge remote-tracking branch 'gh/nautilus'

Merge pull request #26457 from mogeb/osd-stop-steady-clock

osd: use steady clock in prepare_to_stop()

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Fix env vars of `run-tox.sh`

Fixes: http://tracker.ceph.com/issues/38798
Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

14.2.0

doc/releases/nautilus: add reference to msgr2 config update section

Signed-off-by: Sage Weil <sage@redhat.com>

doc/cephfs/nfs: update ceph.conf example

and reference 'ceph config generate-minimal-conf'

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #26956 into nautilus

* refs/pull/26956/head:
doc/releases/nautilus: more detail around ceph-volume step

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #27013 from badone/wip-nautilus-CA-change

qa/ceph-ansible: Move to Nautilus

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #27011 from liewegas/wip-redirect-mimic

qa/suites/upgrade/mimic-x/parallel: enable all classes

Reviewed-by: Myoungwon Oh <omwmw@sk.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27018 from liewegas/wip-38294

qa/suites/upgrade/luminous-x: disable c-o-t import/export tests between versions

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #26960 into nautilus

* refs/pull/26960/head:
ceph_release: nautilus is will be 'stable' for v14.2.0

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge PR #27017 into master

* refs/pull/27017/head:
osd: transpose two wait lists in comment

Reviewed-by: Sage Weil <sage@redhat.com>

doc/releases/nautilus: final upgrade note updates

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suitse/upgrade/luminous-x/stress-split-erasure-code: disable c-o-t tests

The stress-split thrasher already had this off, but the ec variant did
not.  We don't support ceph-objectstore-tool exports/imports between major
versions.

Fixes: http://tracker.ceph.com/issues/38294
Signed-off-by: Sage Weil <sage@redhat.com>

osd: transpose two wait lists in comment

see PrimaryLogPG::do_request(), we check for

1. is_peered(), then
2. flushes_in_progress, then
3. is_active()
4. scrubber.is_chunky_scrub_active() && write_blocked_by_scrub(head) in
   PrimaryLogPG::do_op() which is called by PrimaryLogPG::do_request().

while in PrimaryLogPG::on_change()

we requeue the waiting request in the reversed order,

1. requeue_ops(waiting_for_peered);
2. requeue_ops(waiting_for_flush);
3. requeue_ops(waiting_for_active);

so the comment is not in sync with the code. in this change,
"waiting_for_active" and "waiting_for_flush" are transposed in the
comment explaining blocked request wait lists.

also, sync the pre-conditions of "waiting_for_peered" and
"waiting_for_flush" with "PrimaryLogPG::do_request()"

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #26822 into master

* refs/pull/26822/head:
osd/PG: adjust pg history on fabricated merge target if necessary

Reviewed-by: Neha Ojha <nojha@redhat.com>

ceph_test_rados_api_tier_cxx: remove mimic checks

These checks don't work when the cluster is newer than mimic.

Instead, let's just try not to run this test on a pre-mimic cluster.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #27002 into nautilus

* refs/pull/27002/head:
mgr/telemetry: check for errors when sending report

Reviewed-by: Sage Weil <sage@redhat.com>

qa/suites/upgrade/mimic-x/parallel: enable all classes

This fixes the redirect tiering tests, which need the 'cas' class.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_rados_api_tier_cxx: remove luminous check

It was a flawed check, since it would fail for anything newer than
luminous.  And in any case, we don't need it anymore.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: allow 'osd pool set pgp_num_actual'

Normally we let the mgr control pgp_num_actual for us in a nice, safe, controlled
way.  However, it is very conservative, and only makes changes if all PGs are healthy.
There are situations where the user wants to be move aggressive than this.

For example, if you have a pool with many PGs (say, 4096) and set pg_num_target to a
small number like 4, the mgr will adjust pgp_num way down.  This can lead to an OSD
hitting max_pgs_per_osd.  That prevents the PGs from being active+clean, however,
which prevents the mgr adjusting pgp_num back up even if the user sets the target to
a larger value.

This patch lets the user directly adjust pgp_num_actual.  Note that we still do
not expose access to pg_num_actual, since there are much stricter conditions that
must be true in order to safely make downward adjustments.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #27000 into nautilus

* refs/pull/27000/head:
osd: stringify as int

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26975 from xiexingguo/wip-mon-pri-lost

mon/MonMap: always set mon priority; add it to dump

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26903 from xiexingguo/wip-ds-auth

mgr/DaemonServer: handle caps more carefully

Reviewed-by: Sage Weil <sage@redhat.com>

crimson/osd: do not load fullmap.0

fullmap.0 is an empty map, it used when we try to rebase inc_map.1.

Signed-off-by: chunmei Liu <chunmei.liu@intel.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: reuse load_map() in store_maps()

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: add OSD::load_map()

so we can reuse it when we need to load an non-cached osdmap.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: use boost::make_counting_iterator()

no need to repeat the type of the iterator.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: advance thru maps sequentially

we should not do this in parallel.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/telemetry: check for errors when sending report

There was no error checking, and the server has been failing for
some time, but no one noticed.  Oops.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

qa/ceph-ansible: Move to Nautilus

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

osd/PG: !transaction_applied is true for async_recovery_targets as well

Signed-off-by: Neha Ojha <nojha@redhat.com>

rgw: fix signed char truncation in delimiter check

Fixes: https://tracker.ceph.com/issues/24821
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge PR #26563 into nautilus

* refs/pull/26563/head:
doc/ceph-volume/lvm/batch: fix typos

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #26994 into nautilus

* refs/pull/26994/head:
doc: add scrub link for nautilus release notes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #26822 into nautilus

* refs/pull/26822/head:
osd/PG: adjust pg history on fabricated merge target if necessary

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #26986 into nautilus

* refs/pull/26986/head:
mgr: Make 3rd parameter optional in ceph_get_module_option

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #26949 into nautilus

* refs/pull/26949/head:
mgr/dashboard: add test_module_selftest to API tests

Reviewed-by: Volker Theile <vtheile@suse.com>

osd: stringify as int

Otherwise we get

$ hexdump -C require_osd_release
00000000  0e 0a                                             |..|
00000002

Signed-off-by: Sage Weil <sage@redhat.com>

mds: convert unnecessary list usage to vector

The idea of this change is to use a more allocation efficient structure.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: convert get_*dirfrags to use vector

The idea of this change is to use a more allocation efficient structure.

For reasons I don't understand, this patch caused CInode::get_nested_dirfrags
and CInode::get_subtree_dirfrags to fail to compile when in-lined in the
header. The problem was that the inlined methods tried to access the CDir
internals when CDir is an incomplete types. What confuses me is that those
inlined methods ever compiled. In any case, I have moved the methods to the
CInode.cc source to avoid the issue.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc/releases/nautilus: more detail around ceph-volume step

Recommend host reboots to verify ceph-volume adoption worked.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #26997 from cbodley/wip-38008

doc/rgw: document placement targets and storage classes

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge PR #26995 into nautilus

* refs/pull/26995/head:
osd/PG: fix pg merge check for rc clusters

Reviewed-by: Neha Ojha <nojha@redhat.com>

doc/rgw: document storage class configuration

Fixes: http://tracker.ceph.com/issues/38008
Signed-off-by: Casey Bodley <cbodley@redhat.com>

doc/rgw: document placement target configuration

Fixes: http://tracker.ceph.com/issues/24508
Signed-off-by: Casey Bodley <cbodley@redhat.com>

mds: convert iterator loop to generic loop

To pave the way to use vectors.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: list to std::list

Cleaning up namespaces.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: constantize dump_loads

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

osd/PG: fix pg merge check for rc clusters

If a cluster had a pg merge pending before last_pg_merge_meta was
introduced then the source_pgid will be pg_t().  If that's the case,
skip these new checks.

Likewise, if we decode a legacy pg_pool_t, put the old merge les/lec
values into the correct location.

Signed-off-by: Sage Weil <sage@redhat.com>

doc: add scrub link for nautilus release notes

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

osd/PG: change error to dout in _scan_rollback_obs()

This message is now expected during scrubbing, since we do not delete
objects during a rollforward when !transaction_applied.

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge PR #26958 into nautilus

* refs/pull/26958/head:
mon: avoid using naked pointer
msg/async: do not "return std::move(local_var)"
mon: do not assert on bad auth payload

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: João Eduardo Luís <joao@suse.de>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #26959 into nautilus

* refs/pull/26959/head:
mgr/crash: include entity (instead of daemon) in 'crash ls'

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge PR #26965 into nautilus

* refs/pull/26965/head:
ms/async/ProtocolV2: add ms_die_on_bug and assert rxbuf/txbuf don't get big
msg/async/ProtocolV2: do not reenable pre_auth buffering on from reset_recv_state

Reviewed-by: Ricardo Dias <rdias@suse.com>

osd/PG: skip rollforward when !transaction_applied during append_log()

Earlier, we did pg_log.roll_forward(&handler), when
!transaction_applied, which advanced the crt and trimmed the entries
in rollforward(). Due to this, during _merge_object_divergent_entries(),
when we tried to rollback entries, those objects were not found in the
backend, and thus we hit this bug http://tracker.ceph.com/issues/36739.

With this change, we are advancing the crt value, without deleting the
objects, so that _merge_object_divergent_entries() does not fail
because of deleted objects.

Fixes: http://tracker.ceph.com/issues/36739
Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge PR #26974 into nautilus

* refs/pull/26974/head:
doc: add cephfs scrub related admin document

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #26984 from ricardoasmarques/remove-iscsi-gateway-param-nautilus

mgr/dashboard: Drop iSCSI gateway name parameter

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge PR #26937 into master

* refs/pull/26937/head:
doc: add package for Golang

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

librbd: when requeueing QOS throttled request queue it back

to preserve order of throttled requests, which is important for flush.

Signed-off-by: Mykola Golub <mgolub@suse.com>

common: add "requeue_back" WorkQueue method

and rename "requeue" method to "requeue_front" for clarity.

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #26981 from linuxbox2/wip-nfs-sysobj

rgw: nfs: svc-enable RGWLib

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #26863 from linuxbox2/wip-rgw-delimiter

rgw: prefix-delimiter listing: support >1 character delimiter

mgr: Make 3rd parameter optional in ceph_get_module_option

Fixes: https://tracker.ceph.com/issues/38766
File "/usr/share/ceph/mgr/mgr_module.py", line 863, in _handle_command
2019-03-15T11:28:31.324 INFO:tasks.ceph.mgr.x.mira117.stderr:    return self.handle_command(inbuf, cmd)
2019-03-15T11:28:31.324 INFO:tasks.ceph.mgr.x.mira117.stderr:  File "/usr/share/ceph/mgr/selftest/module.py", line 114, in handle_command
2019-03-15T11:28:31.324 INFO:tasks.ceph.mgr.x.mira117.stderr:    self._self_test()
2019-03-15T11:28:31.325 INFO:tasks.ceph.mgr.x.mira117.stderr:  File "/usr/share/ceph/mgr/selftest/module.py", line 213, in _self_test
2019-03-15T11:28:31.325 INFO:tasks.ceph.mgr.x.mira117.stderr:    self._self_test_config()
2019-03-15T11:28:31.325 INFO:tasks.ceph.mgr.x.mira117.stderr:  File "/usr/share/ceph/mgr/selftest/module.py", line 313, in _self_test_config
2019-03-15T11:28:31.325 INFO:tasks.ceph.mgr.x.mira117.stderr:    assert self.get_module_option_ex("foo", "bar") is None
2019-03-15T11:28:31.325 INFO:tasks.ceph.mgr.x.mira117.stderr:  File "/usr/share/ceph/mgr/mgr_module.py", line 942, in get_module_option_ex
2019-03-15T11:28:31.325 INFO:tasks.ceph.mgr.x.mira117.stderr:    r = self._ceph_get_module_option(module, key)
2019-03-15T11:28:31.325 INFO:tasks.ceph.mgr.x.mira117.stderr:TypeError: ceph_get_module_option() takes exactly 3 arguments (2 given)

Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #26954 from linuxbox2/wip-nfs-null-path

rgw: nfs: skip empty (non-POSIX) path segments

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

mgr/dashboard: Drop iSCSI gateway name parameter

iSCSI gateway name will now be obtained from the
ceph-iscsi `/api/sysinfo/hostname` endpoint

Signed-off-by: Ricardo Marques <rimarques@suse.com>

rgw: nfs: svc-enable RGWLib

Add minimal svc_sys_obj.h boilerplate to RGWLibRequest.  Fix a
trivial illegal access from RGWPutObjProcessor's RadosWriter
when the request object is not stack allocated.

Fixes: https://tracker.ceph.com/issues/38769
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #26971 from theanalyst/nautilus-rgw-enoent-mdlog

rgw_sync: drop ENOENT error logs from mdlog

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #26957 from ceph/wip-bz1666822

ceph-volume: look for rotational data in lsblk

Reviewed-by: Alfredo Deza <adeza@redhat.com>

mon: avoid using naked pointer

to save our trouble to delete auth_handler in the error handling paths

Signed-off-by: Kefu Chai <kchai@redhat.com>

msg/async: do not "return std::move(local_var)"

it prevents copy elision. and both GCC and Clang warn like

warning: moving a local object in a return statement prevents copy
elision [-Wpessimizing-move]

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon: do not assert on bad auth payload

If we get garbage, fail to authenticate--do not assert out and crash.

Signed-off-by: Sage Weil <sage@redhat.com>

ms/async/ProtocolV2: add ms_die_on_bug and assert rxbuf/txbuf don't get big

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async/ProtocolV2: do not reenable pre_auth buffering on from reset_recv_state

This is specifically bad because we call reset_recv_state from
reuse_connection, which turns buffering back on on an already-authenticated
session.

Instead, reenable it only when we set the state to START_CONNECT.  (On
the accepting side, it is a fresh connection, so it starts out true.)
Also, we want to *disable* it on the connection we are reusing, which
might be in a pre-auth state, while we are in a post-auth state.

Fixes: http://tracker.ceph.com/issues/38746
Signed-off-by: Sage Weil <sage@redhat.com>

script: enable nautilus in ceph-release-notes

Signed-off-by: Nathan Cutler <ncutler@suse.com>