Merge pull request #20922 from idryomov/wip-osdmap-encoding-fix

osd/OSDMap: fix HAVE_FEATURE logic in encode()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/mimic-dev2'

# Conflicts:
# qa/standalone/osd/repro_long_log.sh

Merge pull request #20917 from liewegas/wip-no-query-debug

osd: debug dispatch_context cases where queries not sent

Reviewed-by: Josh Durgin <jduring@redhat.com>

Merge pull request #20918 from liewegas/wip-less-many

qa/suites/rados: fewer msgr failures

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20919 from liewegas/wip-cot

qa/standalone/osd/repro_long_log: no-mon-config for cot

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #20880 into wip-sage-testing-20180315.163409

* refs/pull/20880/head:
"Correct" license from LGPL2 to LGPL2.1

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #20910 from ceph/fix_callback_plugins_dir

ceph-volume: fix action plugins path in tests

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #20897 from ifed01/wip-ifed-discard-on-mkfs

os/bluestore: do SSD discard on mkfs

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20895 from tchaikov/wip-drop-unused-options

common/options: drop unused options

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #20445 from aclamk/wip-assert-smaller

common: Make code to invoke assert() smaller.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20887 from liewegas/wip-kill-backlog

osd/osd_types: remove backlog type for pg_log_entry_t

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/OSDMap: fix HAVE_FEATURE logic in encode()

Currently clients that don't have SERVER_LUMINOUS end up being fed
v6 (i.e. luminous) encoding because they also don't have SERVER_MIMIC.
This was introduced in commit 553048fbf97a ("osd/OSDMap: track newly
removed and purged snaps in each epoch").

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/standalone/osd/repro_long_log: no-mon-config for cot

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados: fewer msgr failures

500 is a bit much... e.g., enough to hit timeouts forming mon quorum.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: debug dispatch_context cases where queries not sent

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-volume: fix action plugins path in tox

because of this commit [1] in ceph-ansible tests in ceph-volume needs to
be modified accordingly.

[1] https://github.com/ceph/ceph-ansible/commit/60d4b75f519c03fca91384f231b071793f582376

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #20836 from xiexingguo/wip-balancer-05

pybind/mgr/balancer: add sanity check against empty adjusted_map

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20857 from zealoussnow/wip-leozhang

osd/bluestore: delete unused variable in KernelDevice

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #20690 from myoungwon/wip-manifest-num-counter

osd: add num_object_manifest

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20652 from shangfufei/wip-common

src/common: update some ms_* options to be more consistent

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #20886 from liewegas/wip-kill-min-max

include: kill MIN and MAX macros

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #20321 from jcsp/wip-22918

mgr: initialize PyModuleRegistry sooner

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20901 from neha-ojha/wip-fix-mgr

 qa/tasks/ceph_manager: use set_config on revived osd

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

 qa/tasks/ceph_manager: use set_config on revived osd

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #20824 from ceph/wip-rm23259

ceph-volume process the abspath of the executable first

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #20220 from dzafman/wip-calc-stats3

osd: Improve recovery stat handling by using peer_missing and missing_loc info

Reviewed-by: Sage Weil <sage@redhat.com>

test: osd-backfill-stats.sh parallel osd-recovery-stats.sh check() changes

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Use "(est)" in log message when an osd doesn't have peer_missing

Consolidate check() code and common script code
TEST_recovery_multi() wasn't reliable due to delayed peer_missing

Signed-off-by: David Zafman <dzafman@redhat.com>

test: osd-recovery-stats.sh: New test with different missing objs on multiple OSDs

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Fix _update_calc_stats() to use peer_missing and missing_by_count

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Changes to handle sharding for EC in missing_by_count

Signed-off-by: David Zafman <dzafman@redhat.com>

osd/PrimaryLogPG: update missing_loc on peer recover

The MissingLoc structure already tracks locations of any missing object
on any acting OSD, and is updated on_local_recover and on_global_recover.
Update it on_peer_recover too so that the loc information (and derivative
missing_by_count info) is correct.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: MissingLoc: track location counts by upset and other

For every object in missing_loc, calculate how many known locations are in
the upset and all others, and call that loc_count_t.  Keep
a map of loc_count_t to the number of missing objects with that
distribution.

Signed-off-by: Sage Weil <sage@redhat.com>
Signed-off-by: David Zafman <dzafman@redhat.com>

test: Correction for better degraded/misplaced handling

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Always build peer_missing (including strays) for _update_calc_stats()

Fixes: http://tracker.ceph.com/issues/22837
Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Fix for handling more than 1 missing target

Fix test case to test more than 1 target

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #20751 from s0nea/wip-mgr-dashboard_rbd_add

mgr/dashboard_v2: Add RBD create functionality to the Python backend

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

os/bluestore: trim SSD devices on mkfs

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #20878 from ceph/wip-rm23337

ceph-volume document multipath support

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

osd/osd_types: remove backlog type for pg_log_entry_t

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #20804 from Devp00l/cephfs-template-fix

mgr/dashboard_v2: Fix cephfs template table usage

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #20783 from tspmelo/mgr-perf-counters

mgr/dashboard_v2: add mgr to the list of perf counters

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #20728 from jcsp/wip-dashboard-can_run

mgr/dashboard v2: implement can_run method

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

common/options: drop unused options

async_compressor_* have never been used, so drop them.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Add RBD create functionality

This commit adds a basic RBD create functionality to the REST API. It enables
the POST request of the http://<host>:<port>/api/rbd API endpoint.

The commit adds also related tests.

Signed-off-by: Tatjana Dehler <tdehler@suse.com>

Merge PR #20700 into master

* refs/pull/20700/head:
client: use either dentry_invalidate_cb or remount_cb to invalidate kernel dcache

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #20719 into master

* refs/pull/20719/head:
client: add new ceph_mount_perms_set function

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #20780 into master

* refs/pull/20780/head:
doc/cephfs: mention "ceph fs set-default"
mds: include legacy_client_fscid in FSMap::dump

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

include: kill MIN and MAX macros

Use std::min and std::max instead.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: MIN -> std::min

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #20881 into master

* refs/pull/20881/head:
messages/MClientRequest: fix a typo so we don't output 'setfilelockrule' in logs

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20832 from liewegas/wip-fix-do-copy-range-replay

os/filestore: fix do_copy_range replay bug

Reviewed-by: Kefu Chai <kchai@redhat.com>

"Correct" license from LGPL2 to LGPL2.1

Back in commit 7469f26a33c072169d6ed929352eda06562ffe3f when the COPYING
file was added to the repository and the project was "licensed," the
license file was

                     GNU LESSER GENERAL PUBLIC LICENSE
                          Version 2.1, February 1999

However, the license was abbreviated as LGPL2 in various places (pretty
much everywhere, actually), due to my own ignorance/carelessness.  (I was
distinguishing it from shiny new version 3, released 6 months earlier.)

Version 2 of LGPL is the "Library General Public License," published in
June 1991: https://www.gnu.org/licenses/old-licenses/lgpl-2.0.html

Note that this ambiguity persists for most of the project's lifetime.  For
example, the original debian/copyright in 8adc9dac1d61f34ce85e3cbad1478e3e6ffe2b8e
said "LGPL2" bug in 4545f8b929cc562a1231c0ceb067e9d8e58a0de7 (2009, a year
later) it is changed to LGPL2.1 to satisfy Debian's lintian checks.  (It's
remained LGPL2.1 ever since.)

"Correct" the record by changing LGPL2 references to LGPL2.1.

Signed-off-by: Sage Weil <sage@redhat.com>

messages/MClientRequest: fix a typo so we don't output 'setfilelockrule' in logs

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

ceph-volume lvm.prepare simplify help menu with bluestore default flags

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.create simplify help menu with bluestore default flags

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume document multipath support

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #20864 from ceph/wip-cv-env-flags-fix

ceph-volume tests use granular env vars for vagrant

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #20724 from mdw-at-linuxbox/wip-master-rgw-fixbuf

rgw: fix memory fragmentation problem reading data from client.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20872 from liewegas/wip-deterministic-cli-test

test/cli/osdmaptool/test-map-pgs.t: remove nondetermimistic test

Reviewed-by: Alfredo Deza <adeza@redhat.com>

mgr/dashboard_v2: add mgr to the list of perf counters

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #20739 from Jeegn-Chen/wip-reject-sse-mpcp

rgw: reject encrypted object COPY before supported

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

test/cli/osdmaptool/test-map-pgs.t: remove nondetermimistic test

This fails randomly; determinism is good!

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #20811 from tspmelo/fix-table-details

mgr/dashboard_v2: fix and improve table details

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #20757 from tspmelo/wip-chart-tooltip

mgr/dashboard_v2: Improve charts tooltips

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

ceph-volume tests use granular env vars for vagrant

Fixes the error that would inject an extra argument to vagrant when the
environment variable is not present

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #20830 from Songweibin/wip-cleanup

osd: two cleanups

Reviewed-by: Luo Kexue <luo.kexue@zte.com.cn>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #20838 from badone/test_dump_pgstate_history-check-for-empty-output

tests: Check for empty output in test_dump_pgstate_history

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20839 from guzhongyan/tree-from-cleanup

mon/OSDMonitor: clean up cmd 'osd tree-from'

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd/bluestore: delete unused variable in KernelDevice

Signed-off-by: Leo Zhang <nguzcf@gmail.com>

Merge pull request #20831 from zhsj/fix-spelling

misc: fix various spelling errors

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: iliul <liul.stone@gmail.com>

Merge pull request #20777 from wjwithagen/wjw-mgr_dashboard_test

test: only test dashboard_v2 when it is enabled

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #20848 from liewegas/wip-mds-a

qa/suites: mds.0 -> mds.a

Reviewed-by: Yuri Weinstein <yweins@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-volume process the abspath of the executable first

So that it can always log the full path to the executable when it is
logging the 'Running command:' line

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #20849 from ceph/wip-cv-env-flags

ceph-volume tests add optional flags for vagrant

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume tests add optional flags for vagrant

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #20827 from jdurgin/wip-pg-log-trim-errors

osd: treat successful and erroroneous writes the same for log trimming

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/dashboard_v2: fix and improve table details

There was a problem with the current implementation of the table details, when
compiled in production mode, where the method that created the details element
would throw an exception.

Besides fixing that, this commit also improves the way we define the details.
Now instead of sending the component name, through the 'detailsComponent' attr,
we need to get the current selection via 'updateSelection' attr and declare the
detail component inside the cdTable and add the cdTableDetail attr to that
element.

Signed-off-by: Tiago Melo <tmelo@suse.com>

qa/suites/hadoop: mds.0 -> mds.a

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/ceph-deploy: mds.0 -> mds.a

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/powercycle: mds.0 -> mds.a

Signed-off-by: Sage Weil <sage@redhat.com>

PendingReleaseNotes: dummy change

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: implement can_run checks for dashboard v2

Should be especially handy in development environments
for giving a clear message for people who have forgotten
to build frontend bits.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr/dashboard_v2: improve health page charts tooltips

Extracted the charts into a new component and it is now using a new tooltip.

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard_v2: improve tooltip of cephfsChart

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard_v2: extract cephfs chart into a component

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard_v2: improve tooltip of cdSparkline

Added a new scss and class for the tooltip.

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #20444 from cbodley/wip-qa-ssl

qa/rgw: verify suite tests civetweb with ssl

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #20823 from rjfd/wip-dashboard-pools

mgr/dashboard_v2: Pool controller

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: support test task as CLI arg in rub-backend-api-tests.sh

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: Pool controller implementation and tests

Signed-off-by: Ricardo Dias <rdias@suse.com>

Merge pull request #20821 from tspmelo/fix-configuration-page

mgr/dashboard_v2: fix cluster configuration page

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #20806 from Devp00l/table-column-toggle

mgr/dashboard_v2: Add toggle able columns

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #20797 from rjfd/wip-dashboard-log-requests

mgr/dashboard: HTTP request logging

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #20765 from sebastian-philipp/dashboard_v2-no-detail_route

mgr/dashboard_v2: Removed unused `tools.detail_route()`

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #20772 from votdev/fix_osd_details

mgr/dashboard v2: Don't show details if multiple OSDs are selected

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

qa/workunits/mon/crush_ops.sh: test osd tree-from

Signed-off-by: Gu Zhongyan <guzhongyan@360.cn>

mon/OSDMonitor: clean up cmd 'osd tree-from'
Handle 'bucket not exist' and 'not a bucket' case

Signed-off-by: Gu Zhongyan <guzhongyan@360.cn>

rgw: reject encrypted object COPY before supported

Current implementation does not follow S3 spec and even
may result in data corruption silently when copying
multipart objects accorss pools. So reject COPY operations
on encrypted objects before it is fully functional.

Fixes: http://tracker.ceph.com/issues/23232
Signed-off-by: Jeegn Chen <jeegnchen@gmail.com>

pybind/mgr/balancer: add sanity check against empty adjusted_map

E.g., because all OSDs are marked as **out**.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>