PG: add async recovery and backfill to pg dout() prefix

Signed-off-by: Neha Ojha <nojha@redhat.com>

PG: use clear_location() to clear out missing_loc

Signed-off-by: Neha Ojha <nojha@redhat.com>

PG: use get_recovery_read to prevent race during recovery deletes

Signed-off-by: Neha Ojha <nojha@redhat.com>

PG: handle async targets with missing clone in _rollback_to()

Signed-off-by: Neha Ojha <nojha@redhat.com>

ECBackend: send correct stats to async_recovery_targets

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa: modify TEST_recovery_sizeup() to handle async recovery

Signed-off-by: Neha Ojha <nojha@redhat.com>

PG: want set size should not be below min_size

Signed-off-by: Neha Ojha <nojha@redhat.com>

PG: clear missing_loc only during async_recovery

Signed-off-by: Neha Ojha <nojha@redhat.com>

PG: add location to missing_loc only during async_recovery

Signed-off-by: Neha Ojha <nojha@redhat.com>

PG: clear out old missing_loc in issue_repop

Signed-off-by: Neha Ojha <nojha@redhat.com>

PG: handle degraded object in _rollback_to()

Signed-off-by: Neha Ojha <nojha@redhat.com>

ECBackend: update pg_log missing for async_recovery_targets

Signed-off-by: Neha Ojha <nojha@redhat.com>

PG: skip primary in peer_missing in rebuild()

Signed-off-by: Neha Ojha <nojha@redhat.com>

PG: bug fix in is_degraded_or_backfilling_object()

Signed-off-by: Neha Ojha <nojha@redhat.com>

PG: update pg_log missing for async_recovery_targets

Signed-off-by: Neha Ojha <nojha@redhat.com>

PG: reset async_recovery_targets after recovery

Signed-off-by: Neha Ojha <nojha@redhat.com>

PG: call choose_acting when there are async_recovery_targets

Signed-off-by: Neha Ojha <nojha@redhat.com>

PG: mark recovery started when fail to get recovery_read_lock

Signed-off-by: Neha Ojha <nojha@redhat.com>

PG: set delete flag in needs_recovery_map for deletes

Signed-off-by: Neha Ojha <nojha@redhat.com>

PG: update missing_loc for async_recovery_targets

Signed-off-by: Neha Ojha <nojha@redhat.com>

PG: do not block writes on async_recovery_targets

Signed-off-by: Neha Ojha <nojha@redhat.com>

PG: modify should_send_op to handle async recovery

Signed-off-by: Neha Ojha <nojha@redhat.com>

osd: rename ECSubWrite parameter backfill to backfill_or_async_recovery

Signed-off-by: Neha Ojha <nojha@redhat.com>

PG: cleanup and fixes for async recovery

Signed-off-by: Neha Ojha <nojha@redhat.com>

osd: introduce acting_recovery_backfill shards

Signed-off-by: Neha Ojha <nojha@redhat.com>

PG: initial changes required for async recovery

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #20901 from neha-ojha/wip-fix-mgr

 qa/tasks/ceph_manager: use set_config on revived osd

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

 qa/tasks/ceph_manager: use set_config on revived osd

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #20824 from ceph/wip-rm23259

ceph-volume process the abspath of the executable first

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #20220 from dzafman/wip-calc-stats3

osd: Improve recovery stat handling by using peer_missing and missing_loc info

Reviewed-by: Sage Weil <sage@redhat.com>

test: osd-backfill-stats.sh parallel osd-recovery-stats.sh check() changes

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Use "(est)" in log message when an osd doesn't have peer_missing

Consolidate check() code and common script code
TEST_recovery_multi() wasn't reliable due to delayed peer_missing

Signed-off-by: David Zafman <dzafman@redhat.com>

test: osd-recovery-stats.sh: New test with different missing objs on multiple OSDs

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Fix _update_calc_stats() to use peer_missing and missing_by_count

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Changes to handle sharding for EC in missing_by_count

Signed-off-by: David Zafman <dzafman@redhat.com>

osd/PrimaryLogPG: update missing_loc on peer recover

The MissingLoc structure already tracks locations of any missing object
on any acting OSD, and is updated on_local_recover and on_global_recover.
Update it on_peer_recover too so that the loc information (and derivative
missing_by_count info) is correct.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: MissingLoc: track location counts by upset and other

For every object in missing_loc, calculate how many known locations are in
the upset and all others, and call that loc_count_t.  Keep
a map of loc_count_t to the number of missing objects with that
distribution.

Signed-off-by: Sage Weil <sage@redhat.com>
Signed-off-by: David Zafman <dzafman@redhat.com>

test: Correction for better degraded/misplaced handling

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Always build peer_missing (including strays) for _update_calc_stats()

Fixes: http://tracker.ceph.com/issues/22837
Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Fix for handling more than 1 missing target

Fix test case to test more than 1 target

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #20751 from s0nea/wip-mgr-dashboard_rbd_add

mgr/dashboard_v2: Add RBD create functionality to the Python backend

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #20878 from ceph/wip-rm23337

ceph-volume document multipath support

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #20804 from Devp00l/cephfs-template-fix

mgr/dashboard_v2: Fix cephfs template table usage

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #20783 from tspmelo/mgr-perf-counters

mgr/dashboard_v2: add mgr to the list of perf counters

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #20728 from jcsp/wip-dashboard-can_run

mgr/dashboard v2: implement can_run method

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: Add RBD create functionality

This commit adds a basic RBD create functionality to the REST API. It enables
the POST request of the http://<host>:<port>/api/rbd API endpoint.

The commit adds also related tests.

Signed-off-by: Tatjana Dehler <tdehler@suse.com>

Merge PR #20700 into master

* refs/pull/20700/head:
client: use either dentry_invalidate_cb or remount_cb to invalidate kernel dcache

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #20719 into master

* refs/pull/20719/head:
client: add new ceph_mount_perms_set function

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #20780 into master

* refs/pull/20780/head:
doc/cephfs: mention "ceph fs set-default"
mds: include legacy_client_fscid in FSMap::dump

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #20881 into master

* refs/pull/20881/head:
messages/MClientRequest: fix a typo so we don't output 'setfilelockrule' in logs

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

messages/MClientRequest: fix a typo so we don't output 'setfilelockrule' in logs

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

ceph-volume lvm.prepare simplify help menu with bluestore default flags

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.create simplify help menu with bluestore default flags

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume document multipath support

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #20864 from ceph/wip-cv-env-flags-fix

ceph-volume tests use granular env vars for vagrant

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #20724 from mdw-at-linuxbox/wip-master-rgw-fixbuf

rgw: fix memory fragmentation problem reading data from client.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20872 from liewegas/wip-deterministic-cli-test

test/cli/osdmaptool/test-map-pgs.t: remove nondetermimistic test

Reviewed-by: Alfredo Deza <adeza@redhat.com>

mgr/dashboard_v2: add mgr to the list of perf counters

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #20739 from Jeegn-Chen/wip-reject-sse-mpcp

rgw: reject encrypted object COPY before supported

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

test/cli/osdmaptool/test-map-pgs.t: remove nondetermimistic test

This fails randomly; determinism is good!

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #20811 from tspmelo/fix-table-details

mgr/dashboard_v2: fix and improve table details

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #20757 from tspmelo/wip-chart-tooltip

mgr/dashboard_v2: Improve charts tooltips

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

ceph-volume tests use granular env vars for vagrant

Fixes the error that would inject an extra argument to vagrant when the
environment variable is not present

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #20830 from Songweibin/wip-cleanup

osd: two cleanups

Reviewed-by: Luo Kexue <luo.kexue@zte.com.cn>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #20838 from badone/test_dump_pgstate_history-check-for-empty-output

tests: Check for empty output in test_dump_pgstate_history

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20839 from guzhongyan/tree-from-cleanup

mon/OSDMonitor: clean up cmd 'osd tree-from'

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20831 from zhsj/fix-spelling

misc: fix various spelling errors

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: iliul <liul.stone@gmail.com>

Merge pull request #20777 from wjwithagen/wjw-mgr_dashboard_test

test: only test dashboard_v2 when it is enabled

Reviewed-by: John Spray <john.spray@redhat.com>

ceph-volume process the abspath of the executable first

So that it can always log the full path to the executable when it is
logging the 'Running command:' line

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #20849 from ceph/wip-cv-env-flags

ceph-volume tests add optional flags for vagrant

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume tests add optional flags for vagrant

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #20827 from jdurgin/wip-pg-log-trim-errors

osd: treat successful and erroroneous writes the same for log trimming

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/dashboard_v2: fix and improve table details

There was a problem with the current implementation of the table details, when
compiled in production mode, where the method that created the details element
would throw an exception.

Besides fixing that, this commit also improves the way we define the details.
Now instead of sending the component name, through the 'detailsComponent' attr,
we need to get the current selection via 'updateSelection' attr and declare the
detail component inside the cdTable and add the cdTableDetail attr to that
element.

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: implement can_run checks for dashboard v2

Should be especially handy in development environments
for giving a clear message for people who have forgotten
to build frontend bits.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr/dashboard_v2: improve health page charts tooltips

Extracted the charts into a new component and it is now using a new tooltip.

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard_v2: improve tooltip of cephfsChart

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard_v2: extract cephfs chart into a component

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard_v2: improve tooltip of cdSparkline

Added a new scss and class for the tooltip.

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #20444 from cbodley/wip-qa-ssl

qa/rgw: verify suite tests civetweb with ssl

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #20823 from rjfd/wip-dashboard-pools

mgr/dashboard_v2: Pool controller

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: support test task as CLI arg in rub-backend-api-tests.sh

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: Pool controller implementation and tests

Signed-off-by: Ricardo Dias <rdias@suse.com>

Merge pull request #20821 from tspmelo/fix-configuration-page

mgr/dashboard_v2: fix cluster configuration page

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #20806 from Devp00l/table-column-toggle

mgr/dashboard_v2: Add toggle able columns

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #20797 from rjfd/wip-dashboard-log-requests

mgr/dashboard: HTTP request logging

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #20765 from sebastian-philipp/dashboard_v2-no-detail_route

mgr/dashboard_v2: Removed unused `tools.detail_route()`

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #20772 from votdev/fix_osd_details

mgr/dashboard v2: Don't show details if multiple OSDs are selected

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

qa/workunits/mon/crush_ops.sh: test osd tree-from

Signed-off-by: Gu Zhongyan <guzhongyan@360.cn>

mon/OSDMonitor: clean up cmd 'osd tree-from'
Handle 'bucket not exist' and 'not a bucket' case

Signed-off-by: Gu Zhongyan <guzhongyan@360.cn>

rgw: reject encrypted object COPY before supported

Current implementation does not follow S3 spec and even
may result in data corruption silently when copying
multipart objects accorss pools. So reject COPY operations
on encrypted objects before it is fully functional.

Fixes: http://tracker.ceph.com/issues/23232
Signed-off-by: Jeegn Chen <jeegnchen@gmail.com>

test: Fail upon flush_pg_stats timeout

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Fix log messages

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #20809 from idryomov/wip-librbd-discard-create-truncate

librbd: create+truncate for whole-object layered discards

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20243 from ceph/wip-cd-volume-entry

qa/tests: add openstack volume info + lvs for ceph-volume

misc: fix various spelling errors

Signed-off-by: Shengjing Zhu <i@zhsj.me>

Merge pull request #20826 from liewegas/wip-fix-mstart-config

vstart.sh: fix mstart variables

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

osd: direct use of 'shard' without repeated calls

Signed-off-by: songweibin <song.weibin@zte.com.cn>

osd: drop unused header in OSD.cc

Signed-off-by: songweibin <song.weibin@zte.com.cn>

Merge pull request #20819 from ygtzf/wip-install-deps

install-deps.sh: fix an error condition expression

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20419 from tchaikov/wip-22748

common/config: add units to options

Reviewed-by: John Spray <john.spray@redhat.com>