doc/rbd: update mirroring to include new peer secret config-key actions

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mon: added bootstrap-rbd-mirror/rbd-mirror mon profiles

The rbd-mirror daemon will require permission to read config-keys
from the "rbd/mirror/" prefix. These new profiles (and the new
associated bootstrap-rbd-mirror user) provide the required
permissions.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: mirror pool peer commands now support mon_host/key overrides

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: connect to remote cluster using optional mon_host/key values

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: extract optional peer mon_host/key values from MON

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: cluster watcher doesn't need to resolve pool names

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: renamed 'peer_t' type to 'PeerSpec'

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: renamed 'types' files to 'Types'

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #23941 from trociny/wip-23445

librbd: flatten operation should use object map

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #22675 from nuyhah/fix_block_name_prefix

librbd: block_name_prefix is not created randomly

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #23790 from iridescent-rsy/myfix

librbd: remove template declaration of a non-template function

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #23853 from zy751713126/add_protect_snapshot_list

rbd: add protected in snap list

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #24009 from trociny/wip-test-migration-other-pool

test/librbd: migration tests did not delete additional pool

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: block_name_prefix is not created randomly

fixes: http://tracker.ceph.com/issues/24634
Signed-off-by: hyun ha hyun.ha@navercorp.com

Merge pull request #23702 from Rubab-Syed/rgw-orchestrator-rook

mgr/orchestrator: Add RGW service support

Reviewed-by: John Spray <john.spray@redhat.com>

Merge PR #23908 into master

* refs/pull/23908/head:
client: retry remount on dcache invalidation failure

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #23691 into master

* refs/pull/23691/head:
client:two ceph-fuse client, one can not list out files created by another.

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #23323 into master

* refs/pull/23323/head:
client: set f_files to the total number of files in the filesystem

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23314 into master

* refs/pull/23314/head:
mds: include session uptime when diplaying session list
mds: track average uptime of sessions

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23954 from zy751713126/clear_snap_limit

rbd: simplified code to remove do_clear_limit function

Merge PR #23198 into master

* refs/pull/23198/head:
mds: optimize acquiring locks
mds: reduce memory fragments of EMetaBlob
mds: optimize memory allocation in CInode::encode_inodestat
mds: convert MDSCacheObject::ref_map to flat_map

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23199 into master

* refs/pull/23199/head:
script: backport-create-issue: optionally take list of issue numbers

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23728 into master

* refs/pull/23728/head:
doc: suppored features of kernel client

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23824 into master

* refs/pull/23824/head:
doc/cephfs: add notes on application best practices

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

script: backport-create-issue: optionally take list of issue numbers

Make the script optionally take a comma-separated list of issue numbers.
(Could be just one issue.)

Before this patch, backport-create-issue script insisted on looping over all
issues in Pending Backport status. This was cumbersome in cases when only
one issue (or a couple issues) needed to be processed.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #21379 from pritha-srivastava/wip-rgw-user-policy

rgw: User Policy

Merge pull request #23497 from noahdesu/insights

mgr/insights: insights reporting module

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #23146 from jcsp/wip-progress

mgr/progress: improve+test OSD out handling

Reviewed-by: Noah Watkins <nwatkins@redhat.com>

Merge pull request #23897 from votdev/improve_delete_modal

mgr/dashboard: Make deletion dialog more touch device friendly

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #24016 from votdev/bug_35907

mgr/dashboard: Progress bar does not stop in TableKeyValueComponent

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge PR #22987 into master

* refs/pull/22987/head:
common,rgw: rename sha1_digest_t
osd: decrement old chunk's reference count if the chunk has a reference.
src/test: add a unit test
osd: using fingerprint OID if fingerprint is set
osd: add flag interfaces in chunk_info_t
common/buffer.cc: add sha1 fingerprint
osd: add fingerprint property
mon: add a command to set fingerprint algorithm

Merge PR #24006 into master

* refs/pull/24006/head:
osd/OSD: clear ping_history on heartbeat_reset
mon/OSDMonitor: share new maps with even non-active osds

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #24010 into master

* refs/pull/24010/head:
osd/OSD: kick right merge source
mgr/DaemonServer: split should respect inflight creating pgs

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23464 from wido/influx-optimize

mgr/influx: Use Queue to store points which need to be written

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #24019 from alfredodeza/wip-rm34535

ceph-volume  batch carve out lvs for bluestore

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

doc: suppored features of kernel client

Fixes: http://tracker.ceph.com/issues/27209
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #23957 from tchaikov/wip-crimson-logging

common,auth,crimson: add logging to crimson

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Liu-Chunmei <chunmei.liu@intel.com>

Merge pull request #23992 from badone/wip-librados-client-unique-ptr-compile-error

librados: Include memory for unique_ptr definition

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23723 from xiexingguo/wip-list-missing

osd/PrimaryLogPG: rename list_missing -> list_unfound command

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23921 from croit/fix-35544

osd/OSDMap: add osd status to utilization dumper

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23488 from xiaomanh/master

tools: correct the description of Allowed options in osdomap tool

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Make deletion dialog more touch device friendly

* Refactor deletion dialog
* Add directives.module.ts to be able to use 'autofocus' in deletion dialog

Signed-off-by: Volker Theile <vtheile@suse.com>

qa: add task for progress module

Signed-off-by: John Spray <john.spray@redhat.com>

qa: add 4th OSD to mgr test cluster

This is useful for testing progress module.

Signed-off-by: John Spray <john.spray@redhat.com>

qa: add tests for progress module

Signed-off-by: John Spray <john.spray@redhat.com>

mgr/progress: no progress event on unmoved pgs

PGs may not be moved on osd out, if there is no suitable
location for them to move to.  In this situation
it doesn't make sense to have a progress event, as the
health warnings adequately communicate the situation.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr/progress: fix PgRecoveryEvent completion cases

The event was previously not getting moved to the completed
list.  There are a couple more cases too:
 - When some pgs go away (a pool is removed) during the event
 - When the OSD comes back in after going out

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: expose osdmap pg_to_up_acting_osds

It's not efficient to have python calling this
O(pg_num) times to find the pgs for an OSD, but
I'm just shooting for something functional for now.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #23910 from votdev/improve_autofocus_directive

mgr/dashboard: Refactor autofocus directive

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

rbd: add protected in snap list

Signed-off-by: Zheng Yin <zhengyin@cmss.chinamobile.com>

Merge pull request #23752 from ifed01/wip-ifed-fix-garbage-test

os/tests: fix garbageCollection test case from store_test suite.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24021 from libingyang-zte/master

doc: Fix Spelling Error of Radosgw

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

doc: Fix Spelling Error of Radosgw

Signed-off-by: Li Bingyang <li.bingyang1@zte.com.cn>

qa/tasks/mgr: whitelist insights test health checks

these drive health history tracking tests.

Signed-off-by: Noah Watkins <nwatkins@redhat.com>

Merge pull request #23602 from smanjara/wip-test-netem

qa: Task to emulate network delay and packet drop between two given h…

mgr/dashboard: Progress bar does not stop in TableKeyValueComponent

Fixes: https://tracker.ceph.com/issues/35907
Signed-off-by: Volker Theile <vtheile@suse.com>

ceph-volume tests.util verify Disk objects don't change state with divisions

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.disk fix an issue where Disk objects would mutate on div operations

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.prepare add a helper to get block.db sizes from ceph.conf

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.batch.bluestore add TODOs for custom fast/slow devices

Signed-off-by: Alfredo Deza <adeza@redhat.com>

-f  ceph-volume lvm.batch.bluestore validation and reporting with VG reuse

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/cephfs: add notes on application best practices

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #23997 from batrick/multimds-qa-broken-symlink

qa: fix symlink

Merge PR #23965 into master

* refs/pull/23965/head:
doc/dev/msgr2: better formatting
doc/dev/msgr2: clarify padding alignment
doc/dev/msgr2: tweak message flow handshake
doc/dev/msgr2: remove stream concept, streamline auth

Reviewed-by: Ricardo Dias <rdias@suse.com>

ceph-volume lvm.batch fix error reporting, Device objects aren't strings

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.batch.bluestore validation and reporting with VG reuse

Reworks the bluestore validation and reporting to account for reusable
VGs from fast devices, and adds validation calls to ensure the new way
to calculate this process will work.

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.batch.filestore capture SizeAllocationErrors

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.batch make sure data devices don't have existing LVs on bluestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

librbd: flatten operation should use object map

If the object is known to exist in the image, the copy-up operation
can be skipped for that object.

Fixes: http://tracker.ceph.com/issues/23445
Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: add object_may_not_exist helper

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #21271 from cbodley/wip-rgw-beast-async

rgw: beast frontend reworks pause/stop and yields during body io

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge PR #23845 into master

* refs/pull/23845/head:
osd/OSDMap: include age in up and in counts for ceph status
mon/OSDMonitor: set new_last_{up,in}_change
osd/OSDMap: store last_up_change and last_in_change
mgr/MgrMap: include mgr age in map printer
mon/MgrMap: track active_changed timestamp
mon: include mon quorum age in status
include/utime: add utimespan_str helper

Reviewed-by: John Spray <john.spray@redhat.com>

Merge PR #23949 into master

* refs/pull/23949/head:
mon/OSDMonitor: invalidate max_failed_since on cancel_report

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #23968 into master

* refs/pull/23968/head:
dout: add basic prefix providers
dout: add DoutPrefixPipe for composing prefix providers

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #23971 into master

* refs/pull/23971/head:
cls/numops: fix cls_numops.cc log add to mul

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #23975 into master

* refs/pull/23975/head:
common/buffer.cc: add create_small_page_aligned to avoid mem waste when apply for small mem in big page size(e.g. 64k) OS

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #23939 from votdev/bug_35685

mgr/dashboard: Fix bug in user form when changing password

Reviewed-by: Stephan Müller <smueller@suse.com>

osd/OSD: kick right merge source

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

test/librbd: migration tests did not delete additional pool

Instead of creating an additional pool on every migration test
that needs it, create a pool on the test case setup and reuse.

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #23839 from trociny/wip-migration-commit-race

librbd: fix potential live migration after commit issues due to not refreshed image header

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

osd/OSD: clear ping_history on heartbeat_reset

Because the old connections are gone, and hence we should not
leave behind a long list of obsolete ping_history there, which
is misleading...

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/OSDMonitor: share new maps with even non-active osds

OSDs may not be aware of their deadness and trapped at
an obsolete map in which they were still marked as up:

```
host        osd     down_at     stuck_at
ceph-03     9       e712        e711
ceph-03     13      e700        e699
ceph-03     28      e697        e696
ceph-03     48      e697        e696
ceph-03     52      e707        e704
ceph-03     61      e710        e708
ceph-03     73      e712        e710
ceph-03     77      e708        e707

ceph-05     12      e711        e710
ceph-05     21      e703        e702
ceph-05     24      e700        e699
ceph-05     29      e703        e699
ceph-05     41      e711        e710
ceph-05     53      e711        e710
ceph-05     72      e712        e711

```

In https://github.com/ceph/ceph/pull/23958 an OSD will ping monitor
periodically now if it is stuck at __wait_for_healthy__. But in the
above case OSDs are still considering themselves as __active__ and
hence should miss that fixer.

Since these OSDs might be still able to contact with monitors (
otherwise there is no way for them to be marked up again) and send
beacons contiguously, we can simply get them out of the trap by
sharing some new maps with them.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
Signed-off-by: runsisi <runsisi@zte.com.cn>

mgr/dashboard: Refactor autofocus directive

Refactor the autofocus directive and add some unit tests.

Signed-off-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Unable to edit user when making an accidental change to the password field

Fixes: https://tracker.ceph.com/issues/35685
Signed-off-by: Volker Theile <vtheile@suse.com>

mgr/DaemonServer: split should respect inflight creating pgs

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #23993 from badone/wip-fedora-build-Cython3-error

rpm: Fix Fedora error "No matching package to install: 'Cython3'"

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23833 from falcon78921/wip-docs-34539

doc/rados: fixed hit set type link

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24000 from libingyang-zte/master

doc: Fix Spelling Error of Radosgw

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

doc: fixed hit set type link

Fixed reference link for hit set type value. Restructured wording in description.
Fixes: https://tracker.ceph.com/issues/34539
Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>

doc: Fix Spelling Error of Radosgw

Signed-off-by: Li Bingyang <li.bingyang1@zte.com.cn>

qa: fix symlink

Introduced-by: 6ac1882dc49bc714390d81e582d5febc9bb5e549
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23895 from xiexingguo/wip-more-async-fixes

osd/PrimaryLogPG: update missing_loc more carefully

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #23958 from xiexingguo/wip-heartbeat-stuck

osd/OSD: ping monitor if we are stuck at __waiting_for_healthy__

Reviewed-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: invalidate max_failed_since on cancel_report

max_failed_since might reference the very failure-report which is to be
cancelled. We can simply invalidate it here to make **get_failed_since()**
recalculate if necessary.

Fixes: http://tracker.ceph.com/issues/35860
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #23449 into master

* refs/pull/23449/head:
osd/OSDMap: cleanup: s/tmpmap/nextmap/
qa/standalone/osd/osd-backfill-stats: fixes
osd/OSDMap: clean out pg_temp mappings that exceed pool size
mon/OSDMonitor: clean temps and upmaps in encode_pending, efficiently
osd/OSDMapMapping: do not crash if acting > pool size

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #23984 into master

* refs/pull/23984/head:
mon: test if gid exists in pending for prepare_beacon

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

osd/OSDMap: cleanup: s/tmpmap/nextmap/

Be consistent with OSDMap.h

Signed-off-by: Sage Weil <sage@redhat.com>

qa/standalone/osd/osd-backfill-stats: fixes

Grep from the primary's log, not every osd's log.

For the backfill_remapped task in particular, after the pg_temp change it
just so happens that the primary changes across the pool size change and
thus two different primaries do (some) backfill.  Fix that test to pass
the correct primary.

Other tests are unaffected as they do not (happen to) trigger a primary
change and already satisfied the (removed) check that only one OSD does
backfill.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMap: clean out pg_temp mappings that exceed pool size

If the pool size is reduced, we can end up with pg_temp mappings that are
too big.  This can trigger bad behavior elsewhere (e.g., OSDMapMapping,
which assumes that acting and up are always <= pool size).

Fixes: http://tracker.ceph.com/issues/26866
Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: clean temps and upmaps in encode_pending, efficiently

- do not rebuild the next map when we already have it
- do this work in encode_pending, not create_pending, so we get bad
values before they are published.

Signed-off-by: Sage Weil <sage@redhat.com>