doc: add dev notes for mon's full osdmap pruning

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #19811 from neha-ojha/wip-async-recovery

osd: async recovery

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #20922 from idryomov/wip-osdmap-encoding-fix

osd/OSDMap: fix HAVE_FEATURE logic in encode()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/mimic-dev2'

# Conflicts:
# qa/standalone/osd/repro_long_log.sh

Merge pull request #20917 from liewegas/wip-no-query-debug

osd: debug dispatch_context cases where queries not sent

Reviewed-by: Josh Durgin <jduring@redhat.com>

Merge pull request #20918 from liewegas/wip-less-many

qa/suites/rados: fewer msgr failures

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20919 from liewegas/wip-cot

qa/standalone/osd/repro_long_log: no-mon-config for cot

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

PG: dout cleanup

Signed-off-by: Neha Ojha <nojha@redhat.com>

PG: add async recovery and backfill to pg dout() prefix

Signed-off-by: Neha Ojha <nojha@redhat.com>

PG: use clear_location() to clear out missing_loc

Signed-off-by: Neha Ojha <nojha@redhat.com>

PG: use get_recovery_read to prevent race during recovery deletes

Signed-off-by: Neha Ojha <nojha@redhat.com>

PG: handle async targets with missing clone in _rollback_to()

Signed-off-by: Neha Ojha <nojha@redhat.com>

ECBackend: send correct stats to async_recovery_targets

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa: modify TEST_recovery_sizeup() to handle async recovery

Signed-off-by: Neha Ojha <nojha@redhat.com>

PG: want set size should not be below min_size

Signed-off-by: Neha Ojha <nojha@redhat.com>

PG: clear missing_loc only during async_recovery

Signed-off-by: Neha Ojha <nojha@redhat.com>

PG: add location to missing_loc only during async_recovery

Signed-off-by: Neha Ojha <nojha@redhat.com>

PG: clear out old missing_loc in issue_repop

Signed-off-by: Neha Ojha <nojha@redhat.com>

PG: handle degraded object in _rollback_to()

Signed-off-by: Neha Ojha <nojha@redhat.com>

ECBackend: update pg_log missing for async_recovery_targets

Signed-off-by: Neha Ojha <nojha@redhat.com>

PG: skip primary in peer_missing in rebuild()

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge PR #20880 into wip-sage-testing-20180315.163409

* refs/pull/20880/head:
"Correct" license from LGPL2 to LGPL2.1

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #20910 from ceph/fix_callback_plugins_dir

ceph-volume: fix action plugins path in tests

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #20897 from ifed01/wip-ifed-discard-on-mkfs

os/bluestore: do SSD discard on mkfs

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20895 from tchaikov/wip-drop-unused-options

common/options: drop unused options

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #20445 from aclamk/wip-assert-smaller

common: Make code to invoke assert() smaller.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20887 from liewegas/wip-kill-backlog

osd/osd_types: remove backlog type for pg_log_entry_t

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/OSDMap: fix HAVE_FEATURE logic in encode()

Currently clients that don't have SERVER_LUMINOUS end up being fed
v6 (i.e. luminous) encoding because they also don't have SERVER_MIMIC.
This was introduced in commit 553048fbf97a ("osd/OSDMap: track newly
removed and purged snaps in each epoch").

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/standalone/osd/repro_long_log: no-mon-config for cot

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados: fewer msgr failures

500 is a bit much... e.g., enough to hit timeouts forming mon quorum.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: debug dispatch_context cases where queries not sent

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-volume: fix action plugins path in tox

because of this commit [1] in ceph-ansible tests in ceph-volume needs to
be modified accordingly.

[1] https://github.com/ceph/ceph-ansible/commit/60d4b75f519c03fca91384f231b071793f582376

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #20836 from xiexingguo/wip-balancer-05

pybind/mgr/balancer: add sanity check against empty adjusted_map

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20857 from zealoussnow/wip-leozhang

osd/bluestore: delete unused variable in KernelDevice

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #20690 from myoungwon/wip-manifest-num-counter

osd: add num_object_manifest

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20652 from shangfufei/wip-common

src/common: update some ms_* options to be more consistent

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #20886 from liewegas/wip-kill-min-max

include: kill MIN and MAX macros

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #20321 from jcsp/wip-22918

mgr: initialize PyModuleRegistry sooner

Reviewed-by: Kefu Chai <kchai@redhat.com>

PG: bug fix in is_degraded_or_backfilling_object()

Signed-off-by: Neha Ojha <nojha@redhat.com>

PG: update pg_log missing for async_recovery_targets

Signed-off-by: Neha Ojha <nojha@redhat.com>

PG: reset async_recovery_targets after recovery

Signed-off-by: Neha Ojha <nojha@redhat.com>

PG: call choose_acting when there are async_recovery_targets

Signed-off-by: Neha Ojha <nojha@redhat.com>

PG: mark recovery started when fail to get recovery_read_lock

Signed-off-by: Neha Ojha <nojha@redhat.com>

PG: set delete flag in needs_recovery_map for deletes

Signed-off-by: Neha Ojha <nojha@redhat.com>

PG: update missing_loc for async_recovery_targets

Signed-off-by: Neha Ojha <nojha@redhat.com>

PG: do not block writes on async_recovery_targets

Signed-off-by: Neha Ojha <nojha@redhat.com>

PG: modify should_send_op to handle async recovery

Signed-off-by: Neha Ojha <nojha@redhat.com>

osd: rename ECSubWrite parameter backfill to backfill_or_async_recovery

Signed-off-by: Neha Ojha <nojha@redhat.com>

PG: cleanup and fixes for async recovery

Signed-off-by: Neha Ojha <nojha@redhat.com>

osd: introduce acting_recovery_backfill shards

Signed-off-by: Neha Ojha <nojha@redhat.com>

PG: initial changes required for async recovery

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #20901 from neha-ojha/wip-fix-mgr

 qa/tasks/ceph_manager: use set_config on revived osd

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

 qa/tasks/ceph_manager: use set_config on revived osd

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #20824 from ceph/wip-rm23259

ceph-volume process the abspath of the executable first

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #20220 from dzafman/wip-calc-stats3

osd: Improve recovery stat handling by using peer_missing and missing_loc info

Reviewed-by: Sage Weil <sage@redhat.com>

test: osd-backfill-stats.sh parallel osd-recovery-stats.sh check() changes

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Use "(est)" in log message when an osd doesn't have peer_missing

Consolidate check() code and common script code
TEST_recovery_multi() wasn't reliable due to delayed peer_missing

Signed-off-by: David Zafman <dzafman@redhat.com>

test: osd-recovery-stats.sh: New test with different missing objs on multiple OSDs

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Fix _update_calc_stats() to use peer_missing and missing_by_count

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Changes to handle sharding for EC in missing_by_count

Signed-off-by: David Zafman <dzafman@redhat.com>

osd/PrimaryLogPG: update missing_loc on peer recover

The MissingLoc structure already tracks locations of any missing object
on any acting OSD, and is updated on_local_recover and on_global_recover.
Update it on_peer_recover too so that the loc information (and derivative
missing_by_count info) is correct.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: MissingLoc: track location counts by upset and other

For every object in missing_loc, calculate how many known locations are in
the upset and all others, and call that loc_count_t.  Keep
a map of loc_count_t to the number of missing objects with that
distribution.

Signed-off-by: Sage Weil <sage@redhat.com>
Signed-off-by: David Zafman <dzafman@redhat.com>

test: Correction for better degraded/misplaced handling

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Always build peer_missing (including strays) for _update_calc_stats()

Fixes: http://tracker.ceph.com/issues/22837
Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Fix for handling more than 1 missing target

Fix test case to test more than 1 target

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #20751 from s0nea/wip-mgr-dashboard_rbd_add

mgr/dashboard_v2: Add RBD create functionality to the Python backend

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

os/bluestore: trim SSD devices on mkfs

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #20878 from ceph/wip-rm23337

ceph-volume document multipath support

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

osd/osd_types: remove backlog type for pg_log_entry_t

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #20804 from Devp00l/cephfs-template-fix

mgr/dashboard_v2: Fix cephfs template table usage

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #20783 from tspmelo/mgr-perf-counters

mgr/dashboard_v2: add mgr to the list of perf counters

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #20728 from jcsp/wip-dashboard-can_run

mgr/dashboard v2: implement can_run method

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

common/options: drop unused options

async_compressor_* have never been used, so drop them.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Add RBD create functionality

This commit adds a basic RBD create functionality to the REST API. It enables
the POST request of the http://<host>:<port>/api/rbd API endpoint.

The commit adds also related tests.

Signed-off-by: Tatjana Dehler <tdehler@suse.com>

Merge PR #20700 into master

* refs/pull/20700/head:
client: use either dentry_invalidate_cb or remount_cb to invalidate kernel dcache

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #20719 into master

* refs/pull/20719/head:
client: add new ceph_mount_perms_set function

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #20780 into master

* refs/pull/20780/head:
doc/cephfs: mention "ceph fs set-default"
mds: include legacy_client_fscid in FSMap::dump

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

include: kill MIN and MAX macros

Use std::min and std::max instead.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: MIN -> std::min

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #20881 into master

* refs/pull/20881/head:
messages/MClientRequest: fix a typo so we don't output 'setfilelockrule' in logs

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20832 from liewegas/wip-fix-do-copy-range-replay

os/filestore: fix do_copy_range replay bug

Reviewed-by: Kefu Chai <kchai@redhat.com>

"Correct" license from LGPL2 to LGPL2.1

Back in commit 7469f26a33c072169d6ed929352eda06562ffe3f when the COPYING
file was added to the repository and the project was "licensed," the
license file was

                     GNU LESSER GENERAL PUBLIC LICENSE
                          Version 2.1, February 1999

However, the license was abbreviated as LGPL2 in various places (pretty
much everywhere, actually), due to my own ignorance/carelessness.  (I was
distinguishing it from shiny new version 3, released 6 months earlier.)

Version 2 of LGPL is the "Library General Public License," published in
June 1991: https://www.gnu.org/licenses/old-licenses/lgpl-2.0.html

Note that this ambiguity persists for most of the project's lifetime.  For
example, the original debian/copyright in 8adc9dac1d61f34ce85e3cbad1478e3e6ffe2b8e
said "LGPL2" bug in 4545f8b929cc562a1231c0ceb067e9d8e58a0de7 (2009, a year
later) it is changed to LGPL2.1 to satisfy Debian's lintian checks.  (It's
remained LGPL2.1 ever since.)

"Correct" the record by changing LGPL2 references to LGPL2.1.

Signed-off-by: Sage Weil <sage@redhat.com>

messages/MClientRequest: fix a typo so we don't output 'setfilelockrule' in logs

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

ceph-volume lvm.prepare simplify help menu with bluestore default flags

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.create simplify help menu with bluestore default flags

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume document multipath support

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #20864 from ceph/wip-cv-env-flags-fix

ceph-volume tests use granular env vars for vagrant

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #20724 from mdw-at-linuxbox/wip-master-rgw-fixbuf

rgw: fix memory fragmentation problem reading data from client.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20872 from liewegas/wip-deterministic-cli-test

test/cli/osdmaptool/test-map-pgs.t: remove nondetermimistic test

Reviewed-by: Alfredo Deza <adeza@redhat.com>

mgr/dashboard_v2: add mgr to the list of perf counters

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #20739 from Jeegn-Chen/wip-reject-sse-mpcp

rgw: reject encrypted object COPY before supported

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

test/cli/osdmaptool/test-map-pgs.t: remove nondetermimistic test

This fails randomly; determinism is good!

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #20811 from tspmelo/fix-table-details

mgr/dashboard_v2: fix and improve table details

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #20757 from tspmelo/wip-chart-tooltip

mgr/dashboard_v2: Improve charts tooltips

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

ceph-volume tests use granular env vars for vagrant

Fixes the error that would inject an extra argument to vagrant when the
environment variable is not present

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #20830 from Songweibin/wip-cleanup

osd: two cleanups

Reviewed-by: Luo Kexue <luo.kexue@zte.com.cn>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #20838 from badone/test_dump_pgstate_history-check-for-empty-output

tests: Check for empty output in test_dump_pgstate_history

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20839 from guzhongyan/tree-from-cleanup

mon/OSDMonitor: clean up cmd 'osd tree-from'

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd/bluestore: delete unused variable in KernelDevice

Signed-off-by: Leo Zhang <nguzcf@gmail.com>

Merge pull request #20831 from zhsj/fix-spelling

misc: fix various spelling errors

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: iliul <liul.stone@gmail.com>