doc: add a deduplication document

Signed-off-by: Myoungwon Oh <omwmw@sk.com>

Merge pull request #28078 from sebastian-philipp/kubejacker-fix-mgr-plugins

scripts/kubejacker: Fix mgr_plugins target for centos

Reviewed-by: Tim Serong <tserong@suse.com>

Merge pull request #28155 from alimaredia/wip-remove-mon-valgrind-check-in-rgw-suite

qa: remove mon valgrind check in rgw verfiy suite

mgr/dashboard: Bug Fix Error on Mgr Dashboard NFS (#28075)

mgr/dashboard: Bug Fix Error on Mgr Dashboard NFS

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #28169 from bk201/wip-39971

mgr/dashboard: fix some performance data are not displayed

Reviewed-by: Kanika Murarka <kmurarka@redhat.com>

Merge pull request #28181 from smithfarm/wip-39981

do_cmake.sh: use bash

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #28093 from dillaman/wip-39703

librbd: use custom allocator for aligned boost::lockfree::queue

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #28147 from bk201/wip-39933

mgr/dashboard: fix backend error when updating RBD interlocked features

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

mgr/dashboard: Fix tcmu-runner perf counters page (#28154)

mgr/dashboard: Fix tcmu-runner perf counters page

Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #26518 from linuxbox2/wip-rgw-lc-prefix-2

rgw: lifecycle: alternate solution to prefix_map conflict

rgw: lifecycle: fix object iteration for >1 same-prefix rules

Rules of like-prefix must visit every matching object.  A next
logical step is to combine tag checking for rules of like-prefix.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw: lifecycle: fix RGWLifecycleConfiguration::valid()

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw: lifecycle: alternate solution to prefix_map conflict

A rule containing a tag filter but no constraining prefix is
legal, as is a sequence of >1 such rules.

As currently designed, enforcement requires rules to be
installed in prefix_map as well as rule_map, though only the
latter is encoded.

Fixes: http://tracker.ceph.com/issues/37879
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

do_cmake.sh: use bash

Fixes: http://tracker.ceph.com/issues/39981
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #27996 from jan--f/package-prometheus-alerts-suse

ceph.spec.in: package prometheus default alerts for SUSE

Reviewed-by: Nathan Cutler <ncutler@suse.com>

mgr/dashboard: fix some performance data are not displayed

Several Grafana dashboards were updated and their uids are changed.
Update corresponding uids in angular templates. Also fix that when a user
clicks performance details of an OSD, wrong OSD metrics are displayed.

Fixes: http://tracker.ceph.com/issues/39971
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

Merge pull request #28094 from tchaikov/wip-install-deps-gcc-8

install-deps.sh: use gcc-8 on xenial and trusty

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #28079 from tchaikov/wip-crimson-rados-cleanup

crimson: support pgnls and delete op

Reviewed-off-by: Samuel Just <sjust@redhat.com>

qa: remove mon valgrind check in rgw verfiy suite

memory leaks in the monitor are causing a significant
percentage of jobs run in the rgw verify suite to
fail even though the jobs succeeded before hand.

See: http://tracker.ceph.com/issues/38827 for
root cause

Signed-off-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #27653 from tspmelo/wip-nfs-pseudo

mgr/dashboard: Improve NFS Pseudo pattern message

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #28023 from rjfd/wip-fix-ceph-mgr-systemd

systemd: ceph-mgr: set MemoryDenyWriteExecute to false

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #28044 from dillaman/wip-librbd-io-perf

librbd: tweaks to increase IOPS and reduce CPU usage

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #28067 from yuriw/wip-yuriw-crontab-master

qa/tests: added new client-upgrade-*-nautilus suites for jewel, luminous, mimic

mgr/dashboard: Fix tcmu-runner perf counters page

Fixes: https://tracker.ceph.com/issues/39954
Signed-off-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #27681 from tspmelo/wip-charts

mgr/dashboard: Unify the look of dashboard charts

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #28113 from hjwsm1989/fix-krb-compile-error

auth/krb: fix Kerberos compile error

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: fix backend error when updating RBD interlocked features

Now object-map and fast-diff features are interlocked. Allow
enabling/disabling these two features together without raising an already
enabled/disabled error.

Fixes: http://tracker.ceph.com/issues/39933
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

mgr/dashboard: Bug Fix Error on Mgr Dashboard NFS

This change fixes error on NFS Ganesha management dashboard that throws an error when request daemon list.

Fixes: https://tracker.ceph.com/issues/39709
Signed-off-by: Nur Faizin <nur.faizin91@gmail.com>

auth/krb: fix Kerberos compile error

Fixes: http://tracker.ceph.com/issues/39948
Signed-off-by: huangjun <huangjun@xsky.com>

Merge PR #28080 into master

* refs/pull/28080/head:
cephfs-shell: Remove whitespace after '('
cephfs-shell: Remove unused 'flag' variable
cephfs-shell: Fix flake8 invalid escape sequence warning
cephfs-shell: Remove unnecessary empty lines
cephfs-shell: Fix flake8 bare 'except' warnings

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #27822 into master

* refs/pull/27822/head:
cephfs-shell: Fix mkdir relative path error

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/tests: added new client-upgrade-*-nautilus suites for jewel, luminous, mimic

NOTE:

!!!! three suites below MUST use --suite-branch jewel, luminous, mimic (see https://tracker.ceph.com/issues/24021)
ref: https://github.com/ceph/ceph/pull/27983; https://github.com/ceph/ceph/pull/27934; https://github.com/ceph/ceph/pull/28027
--filter "ubuntu_16.04,centos_7.6,rhel_7.6" - test ONLY supported distro BEFORE mimic
--filter "ubuntu_16.04,ubuntu_18.04,centos_7.6,rhel_7.6" - test ONLY supported distro AFTER mimic

Merge after https://github.com/ceph/ceph/pull/28027

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

librbd: use custom allocator for aligned boost::lockfree::queue

If tcmalloc is in-use as the allocator and its version is less than 2.6.2,
it might be missing support for 'aligned_alloc'. This can result in the
glibc version of 'aligned_alloc' being used to allocate memory that is
then freed by tcmalloc -- resulting in a crash.

Fixes: http://tracker.ceph.com/issues/39703
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: Unify the look of dashboard charts

Fixes: http://tracker.ceph.com/issues/39384
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #28092 from tchaikov/wip-dashboard-package

mgr/dashboard: Update tar to v4.4.8

Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #28043 from jan--f/update-grafana-dashboards

Update grafana dashboards

Reviewed-by: kmurarka@redhat.com
Reviewed-by: epuertat@redhat.com

Merge PR #27377 into master

* refs/pull/27377/head:
qa: fs Ignore getfattr errors for ceph.dir.pin

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #27594 into master

* refs/pull/27594/head:
mgr/volumes: allow creation/deletion of FS subvolume group snapshots
mgr/volumes: allow creation/deletion of FS subvolume groups
mgr/volumes: allow creation/deletion of snapshots of FS subvolumes
mgr/volumes: allow fetching path of FS subvolumes
mgr/volumes: use the fs_subvolume module
mgr/volumes: add fs_subvolume module

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #27483 into master

* refs/pull/27483/head:
cephfs-shell: add quota management

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #27753 into master

* refs/pull/27753/head:
cephfs-shell: Add stat command

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #27872 into master

* refs/pull/27872/head:
cephfs-shell: teuthology tests

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

install-deps.sh: use gcc-8 on xenial and trusty

to be consistent with #27183 and https://github.com/ceph/ceph-build/pull/1286

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/volumes: allow creation/deletion of FS subvolume group snapshots

... using `ceph fs subvolumegroup snapshot create/rm` commands.

Signed-off-by: Ramana Raja <rraja@redhat.com>

mgr/volumes: allow creation/deletion of FS subvolume groups

... using `ceph fs subvolumegroup create/rm` commands.
FS subvolume groups are parent directories of FS subvolumes. They
can be directly mapped to OpenStack Manila share groups.

Signed-off-by: Ramana Raja <rraja@redhat.com>

mgr/volumes: allow creation/deletion of snapshots of FS subvolumes

... using `ceph fs subvolume snapshot create/rm` commands.

Signed-off-by: Ramana Raja <rraja@redhat.com>

mgr/volumes: allow fetching path of FS subvolumes

... using `ceph fs subvolume getpath` command.

Signed-off-by: Ramana Raja <rraja@redhat.com>

mgr/volumes: use the fs_subvolume module

... instead of src/pybind/ceph_volume_client.py to manage CephFS
subvolumes using `ceph fs subvolume` set of commands.

Signed-off-by: Ramana Raja <rraja@redhat.com>

mgr/volumes: add fs_subvolume module

fs_subvolume module provisions and manages CephFS subvolumes, which are
CephFS subdirectories with a desired layout and quota. Its code is
heavily borrowed from, src/pybind/ceph_volume_client.py

Fixes: http://tracker.ceph.com/issues/39610
Signed-off-by: Ramana Raja <rraja@redhat.com>

monitoring: update Grafana dashboards

Fix various panels that used outdated metric names, cluncky or
unnecessary label_replace calls. Also unify the style of many panels.

Fixes: http://tracker.ceph.com/issues/39652
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

mgr/dashboard: Update tar to v4.4.8

This update fixes an Arbitrary File Overwrite issue (CVE-2018-20834).
see https://nvd.nist.gov/vuln/detail/CVE-2018-20834

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28085 from rzarzynski/wip-crimson-no-atomics-in-refcntobj

crimson, common: RefCountedObj doesn't use atomics in Seastar builds.

Reviewed-by: Kefu Chai <kchai@redhat.com>

librbd: avoid bl manipulation on single object read/writes

If a read or write only spans a single object and buffer extent,
there isn't any need to build a new bufferlist from a full sub-bl.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: switch to lightweight striper for hot IO path

Optimize non-complex IO requests using the new lightweight object
striper that avoids heap allocations for small requests.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: drop data oid parameter from IO path

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: helper method to generate data object OIDs on-demand

The new generator avoids multiple string length and copy operations
that are currently required in the Striper.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

osdc/Striper: implemented a lightweight version of StripedReadResult

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

osdc/Striper: implemented a lightweight version of file_to_extents

The previous versions of file_to_extents resulted in numerous heap
allocations for maps, vectors, and strings -- in addition to using
a CPU inefficient oid name formater.

librbd doesn't require the use of object_t and it would be better to
avoid any heap allocations on the IO path for the common case.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

common/Finisher: avoid memory re-allocations for finisher queue

Since there is only a single thread associated with the Finisher,
swap between two queues to avoid the need to re-allocate a new
vector for each iteration through the loop. Also replace the
condition broadcast with a signal since there is only a single
thread to wake up.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #27999 from trociny/wip-rbd-sparse_copyup

librbd: clone copy-on-write operations should preserve sparseness

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

crimson/osd: add crimson_debug_pg_always_active option

before crimson is able to peer, we should leave this option ON.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: add preliminary delete support

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: add pgnls support

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28071 from tchaikov/wip-crimson-perf-regres

crimson,osd: performance fixes

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson, common: RefCountedObj doesn't use atomics in SeaStar builds.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mgr/dashboard: Improve NFS Pseudo pattern message

Fixes: http://tracker.ceph.com/issues/39327
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #27270 from theanalyst/beast-fe-ipv6

rgw: normalize v6 endpoint behaviour for the beast frontend

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #26281 from smithfarm/wip-git-describe-long

make-dist: set version number only once

Reviewed-by: Tim Serong <tserong@suse.com>

Merge pull request #28045 from cbodley/wip-39635

cls/user: cls_user_set_buckets_info overwrites creation_time

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #27980 from cbodley/wip-24991

rgw: delete_obj_index() takes mtime for bilog

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #28031 from ivancich/wip-rgw-admin-unordered

rgw: allow radosgw-admin to list bucket w --allow-unordered

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #27676 from mkogan1/wip-rgw-39380

rgw: fix default_placement containing "/" when storage_class is standard

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27984 from cbodley/wip-test-rgw-multisite-sh

test/rgw: fix test-rgw-multisite.sh script for creating multisite clusters

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #27807 from cyb70289/utf8-encode

common: optimize encode_utf8

Reviewed-by: Kefu Chai <kchai@redhat.com>

cephfs-shell: Remove whitespace after '('

This patch fixes flake8 whitespace error.

Fixes: https://tracker.ceph.com/issues/39717
Signed-off-by: Varsha Rao <varao@redhat.com>

cephfs-shell: Remove unused 'flag' variable

This patch fixes flake8 unused local variable error.

Fixes: https://tracker.ceph.com/issues/39717
Signed-off-by: Varsha Rao <varao@redhat.com>

cephfs-shell: Fix flake8 invalid escape sequence warning

Backslash-character pair is not a valid escape sequence, since Python 3.6
version. Prefixing character 'r' to the escape sequence, fixes the warning.

Fixes: https://tracker.ceph.com/issues/39717
Signed-off-by: Varsha Rao <varao@redhat.com>

cephfs-shell: Remove unnecessary empty lines

This patch fixes flake8 too many blank lines error.

Fixes: https://tracker.ceph.com/issues/39717
Signed-off-by: Varsha Rao <varao@redhat.com>

cephfs-shell: Fix flake8 bare 'except' warnings

Instead of bare 'except', catch just libcephfs exceptions.

Fixes: https://tracker.ceph.com/issues/39717
Signed-off-by: Varsha Rao <varao@redhat.com>

Merge pull request #28051 from dillaman/wip-39521

librbd: fix issues with object-map/fast-diff feature interlock

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #28063 from dillaman/wip-39656

qa/workunits/rbd: use https protocol for devstack git operations

Reviewed-by: Mykola Golub <mgolub@suse.com>

scripts/kubejacker: Fix mgr_plugins target for centos

Also use the latest master as default FROM

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

cmake: define BOOST_SP_DISABLE_THREADS for crimson

because we are single-threaded

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/osd_internal_types: bring moving semantics to ObjectState.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: simplify write handling in PG::do_osd_ops.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

librbd: clone copy-on-write operations should preserve sparseness

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #27870 from theanalyst/rgw-objexp-fixes-cli

rgw: object expirer fixes

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #27926 from tchaikov/wip-38627

pybind/mgr: convert str to int using "int()"

Reviewed-by: Noah Watkins <noahwatkins@gmail.com>

Merge pull request #27914 from dzafman/wip-39304

osd: Include dups in copy_after() and copy_up_to()

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #27961 from dzafman/wip-39582

osd: Output Base64 encoding of CRC header if binary data present

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd: Fix merge_log_dups() not remove dup corresponding to log.tail

The log tail is not inclusive of the log, so not an overlap
Fix unittest_pglog to verify proper operation

Signed-off-by: David Zafman <dzafman@redhat.com>

test: unittest_pglog: Additional test cases for trim, copy_up_to and copy_after

Signed-off-by: David Zafman <dzafman@redhat.com>

test: unittest_pglog: Remove irrelevant config values

osd_min_pg_log_entries and osd_max_pg_log_entries are not involved
in the log handling code being tested.

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Adding standalone test of log copy handling

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge PR #27782 into master

* refs/pull/27782/head:
os/bluestore: protect BlueFS::FileReader::buf
os/bluestore: log 'buffered' in KernelDevice::read_random
os/bluestore: support RocksDB prefetching at BlueFS
os/bluestore: call BlueFS::_init_logger earlier.
tools/ceph-objectstore-tool: dump perf counters on competion.
os/bluestore: optimize mapping offset to extent in BlueFS.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #27928 into master

* refs/pull/27928/head:
mon/OSDMonitor: require nautilus to set pg_autoscale_mode
mon/OSDMonitor: allow pg_num to increase when require_osd_release < N

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #27877 into master

* refs/pull/27877/head:
mon: paxos: introduce new reset_pending_committing_finishers for safety

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21722 from mslovy/wip-parial-recovery

osd: partial recovery strategy based on PGLog

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #28060 from ceph/wip-bz1666822

ceph-volume: use the Device.rotational property instead of sys_api

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: skip missing interpreters when running tox tests

This will skip tests for versions of python that don't exist on the node
that's running the tests.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: use the Device.rotational property instead of sys_api

Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1666822

Signed-off-by: Andrew Schoen <aschoen@redhat.com>