Merge pull request #26341 from sebastian-philipp/dashboard-merge-orchclient

mgr/dashboard: Fix iSCSI + Rook issues

Reviewed-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: Subscribe to changes when RequiredIf is used (#28231)

mgr/dashboard: Subscribe to changes when RequiredIf is used

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #28078 from sebastian-philipp/kubejacker-fix-mgr-plugins

scripts/kubejacker: Fix mgr_plugins target for centos

Reviewed-by: Tim Serong <tserong@suse.com>

Merge pull request #28155 from alimaredia/wip-remove-mon-valgrind-check-in-rgw-suite

qa: remove mon valgrind check in rgw verfiy suite

mgr/dashboard: Subscribe to changes when RequiredIf is used

FormControls that use 'requiredIf' validators depend on the state of other
controls.
To prevent that some of the state changes are not tracked, we now automatically
subscribe to the prerequsited controls and update the validation of the parent
control.

Fixes: https://tracker.ceph.com/issues/40017
Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Bug Fix Error on Mgr Dashboard NFS (#28075)

mgr/dashboard: Bug Fix Error on Mgr Dashboard NFS

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #28169 from bk201/wip-39971

mgr/dashboard: fix some performance data are not displayed

Reviewed-by: Kanika Murarka <kmurarka@redhat.com>

Merge pull request #28181 from smithfarm/wip-39981

do_cmake.sh: use bash

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #28093 from dillaman/wip-39703

librbd: use custom allocator for aligned boost::lockfree::queue

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

mgr/rook: Raise proper exception in `describe_service`

`describe_service` is called often and raising
  assertions will pollute the log file with tracebacks.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: Load iSCSI config from local store and orchestrator

As iSCSI is not supported by Rook, users need a way to manage
  ceph-iscsi without Rook knowledge

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: Merge OrchClient with OrchestratorClientMixin

Fixed missing error propagation from
  the orchestrator to the dashboard

Adapted orchestrator integration in NFS

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #28147 from bk201/wip-39933

mgr/dashboard: fix backend error when updating RBD interlocked features

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

mgr/dashboard: Fix tcmu-runner perf counters page (#28154)

mgr/dashboard: Fix tcmu-runner perf counters page

Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #26518 from linuxbox2/wip-rgw-lc-prefix-2

rgw: lifecycle: alternate solution to prefix_map conflict

rgw: lifecycle: fix object iteration for >1 same-prefix rules

Rules of like-prefix must visit every matching object.  A next
logical step is to combine tag checking for rules of like-prefix.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw: lifecycle: fix RGWLifecycleConfiguration::valid()

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw: lifecycle: alternate solution to prefix_map conflict

A rule containing a tag filter but no constraining prefix is
legal, as is a sequence of >1 such rules.

As currently designed, enforcement requires rules to be
installed in prefix_map as well as rule_map, though only the
latter is encoded.

Fixes: http://tracker.ceph.com/issues/37879
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

do_cmake.sh: use bash

Fixes: http://tracker.ceph.com/issues/39981
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #27996 from jan--f/package-prometheus-alerts-suse

ceph.spec.in: package prometheus default alerts for SUSE

Reviewed-by: Nathan Cutler <ncutler@suse.com>

mgr/dashboard: fix some performance data are not displayed

Several Grafana dashboards were updated and their uids are changed.
Update corresponding uids in angular templates. Also fix that when a user
clicks performance details of an OSD, wrong OSD metrics are displayed.

Fixes: http://tracker.ceph.com/issues/39971
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

Merge pull request #28094 from tchaikov/wip-install-deps-gcc-8

install-deps.sh: use gcc-8 on xenial and trusty

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #28079 from tchaikov/wip-crimson-rados-cleanup

crimson: support pgnls and delete op

Reviewed-off-by: Samuel Just <sjust@redhat.com>

qa: remove mon valgrind check in rgw verfiy suite

memory leaks in the monitor are causing a significant
percentage of jobs run in the rgw verify suite to
fail even though the jobs succeeded before hand.

See: http://tracker.ceph.com/issues/38827 for
root cause

Signed-off-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #27653 from tspmelo/wip-nfs-pseudo

mgr/dashboard: Improve NFS Pseudo pattern message

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #28023 from rjfd/wip-fix-ceph-mgr-systemd

systemd: ceph-mgr: set MemoryDenyWriteExecute to false

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #28044 from dillaman/wip-librbd-io-perf

librbd: tweaks to increase IOPS and reduce CPU usage

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #28067 from yuriw/wip-yuriw-crontab-master

qa/tests: added new client-upgrade-*-nautilus suites for jewel, luminous, mimic

mgr/dashboard: Fix tcmu-runner perf counters page

Fixes: https://tracker.ceph.com/issues/39954
Signed-off-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #27681 from tspmelo/wip-charts

mgr/dashboard: Unify the look of dashboard charts

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #28113 from hjwsm1989/fix-krb-compile-error

auth/krb: fix Kerberos compile error

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: fix backend error when updating RBD interlocked features

Now object-map and fast-diff features are interlocked. Allow
enabling/disabling these two features together without raising an already
enabled/disabled error.

Fixes: http://tracker.ceph.com/issues/39933
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

mgr/dashboard: Bug Fix Error on Mgr Dashboard NFS

This change fixes error on NFS Ganesha management dashboard that throws an error when request daemon list.

Fixes: https://tracker.ceph.com/issues/39709
Signed-off-by: Nur Faizin <nur.faizin91@gmail.com>

auth/krb: fix Kerberos compile error

Fixes: http://tracker.ceph.com/issues/39948
Signed-off-by: huangjun <huangjun@xsky.com>

Merge PR #28080 into master

* refs/pull/28080/head:
cephfs-shell: Remove whitespace after '('
cephfs-shell: Remove unused 'flag' variable
cephfs-shell: Fix flake8 invalid escape sequence warning
cephfs-shell: Remove unnecessary empty lines
cephfs-shell: Fix flake8 bare 'except' warnings

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #27822 into master

* refs/pull/27822/head:
cephfs-shell: Fix mkdir relative path error

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/tests: added new client-upgrade-*-nautilus suites for jewel, luminous, mimic

NOTE:

!!!! three suites below MUST use --suite-branch jewel, luminous, mimic (see https://tracker.ceph.com/issues/24021)
ref: https://github.com/ceph/ceph/pull/27983; https://github.com/ceph/ceph/pull/27934; https://github.com/ceph/ceph/pull/28027
--filter "ubuntu_16.04,centos_7.6,rhel_7.6" - test ONLY supported distro BEFORE mimic
--filter "ubuntu_16.04,ubuntu_18.04,centos_7.6,rhel_7.6" - test ONLY supported distro AFTER mimic

Merge after https://github.com/ceph/ceph/pull/28027

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

librbd: use custom allocator for aligned boost::lockfree::queue

If tcmalloc is in-use as the allocator and its version is less than 2.6.2,
it might be missing support for 'aligned_alloc'. This can result in the
glibc version of 'aligned_alloc' being used to allocate memory that is
then freed by tcmalloc -- resulting in a crash.

Fixes: http://tracker.ceph.com/issues/39703
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: Unify the look of dashboard charts

Fixes: http://tracker.ceph.com/issues/39384
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #28092 from tchaikov/wip-dashboard-package

mgr/dashboard: Update tar to v4.4.8

Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #28043 from jan--f/update-grafana-dashboards

Update grafana dashboards

Reviewed-by: kmurarka@redhat.com
Reviewed-by: epuertat@redhat.com

Merge PR #27377 into master

* refs/pull/27377/head:
qa: fs Ignore getfattr errors for ceph.dir.pin

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #27594 into master

* refs/pull/27594/head:
mgr/volumes: allow creation/deletion of FS subvolume group snapshots
mgr/volumes: allow creation/deletion of FS subvolume groups
mgr/volumes: allow creation/deletion of snapshots of FS subvolumes
mgr/volumes: allow fetching path of FS subvolumes
mgr/volumes: use the fs_subvolume module
mgr/volumes: add fs_subvolume module

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #27483 into master

* refs/pull/27483/head:
cephfs-shell: add quota management

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #27753 into master

* refs/pull/27753/head:
cephfs-shell: Add stat command

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #27872 into master

* refs/pull/27872/head:
cephfs-shell: teuthology tests

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

install-deps.sh: use gcc-8 on xenial and trusty

to be consistent with #27183 and https://github.com/ceph/ceph-build/pull/1286

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/volumes: allow creation/deletion of FS subvolume group snapshots

... using `ceph fs subvolumegroup snapshot create/rm` commands.

Signed-off-by: Ramana Raja <rraja@redhat.com>

mgr/volumes: allow creation/deletion of FS subvolume groups

... using `ceph fs subvolumegroup create/rm` commands.
FS subvolume groups are parent directories of FS subvolumes. They
can be directly mapped to OpenStack Manila share groups.

Signed-off-by: Ramana Raja <rraja@redhat.com>

mgr/volumes: allow creation/deletion of snapshots of FS subvolumes

... using `ceph fs subvolume snapshot create/rm` commands.

Signed-off-by: Ramana Raja <rraja@redhat.com>

mgr/volumes: allow fetching path of FS subvolumes

... using `ceph fs subvolume getpath` command.

Signed-off-by: Ramana Raja <rraja@redhat.com>

mgr/volumes: use the fs_subvolume module

... instead of src/pybind/ceph_volume_client.py to manage CephFS
subvolumes using `ceph fs subvolume` set of commands.

Signed-off-by: Ramana Raja <rraja@redhat.com>

mgr/volumes: add fs_subvolume module

fs_subvolume module provisions and manages CephFS subvolumes, which are
CephFS subdirectories with a desired layout and quota. Its code is
heavily borrowed from, src/pybind/ceph_volume_client.py

Fixes: http://tracker.ceph.com/issues/39610
Signed-off-by: Ramana Raja <rraja@redhat.com>

monitoring: update Grafana dashboards

Fix various panels that used outdated metric names, cluncky or
unnecessary label_replace calls. Also unify the style of many panels.

Fixes: http://tracker.ceph.com/issues/39652
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

mgr/dashboard: Update tar to v4.4.8

This update fixes an Arbitrary File Overwrite issue (CVE-2018-20834).
see https://nvd.nist.gov/vuln/detail/CVE-2018-20834

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28085 from rzarzynski/wip-crimson-no-atomics-in-refcntobj

crimson, common: RefCountedObj doesn't use atomics in Seastar builds.

Reviewed-by: Kefu Chai <kchai@redhat.com>

librbd: avoid bl manipulation on single object read/writes

If a read or write only spans a single object and buffer extent,
there isn't any need to build a new bufferlist from a full sub-bl.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: switch to lightweight striper for hot IO path

Optimize non-complex IO requests using the new lightweight object
striper that avoids heap allocations for small requests.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: drop data oid parameter from IO path

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: helper method to generate data object OIDs on-demand

The new generator avoids multiple string length and copy operations
that are currently required in the Striper.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

osdc/Striper: implemented a lightweight version of StripedReadResult

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

osdc/Striper: implemented a lightweight version of file_to_extents

The previous versions of file_to_extents resulted in numerous heap
allocations for maps, vectors, and strings -- in addition to using
a CPU inefficient oid name formater.

librbd doesn't require the use of object_t and it would be better to
avoid any heap allocations on the IO path for the common case.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

common/Finisher: avoid memory re-allocations for finisher queue

Since there is only a single thread associated with the Finisher,
swap between two queues to avoid the need to re-allocate a new
vector for each iteration through the loop. Also replace the
condition broadcast with a signal since there is only a single
thread to wake up.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #27999 from trociny/wip-rbd-sparse_copyup

librbd: clone copy-on-write operations should preserve sparseness

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

crimson/osd: add crimson_debug_pg_always_active option

before crimson is able to peer, we should leave this option ON.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: add preliminary delete support

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: add pgnls support

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28071 from tchaikov/wip-crimson-perf-regres

crimson,osd: performance fixes

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson, common: RefCountedObj doesn't use atomics in SeaStar builds.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mgr/dashboard: Improve NFS Pseudo pattern message

Fixes: http://tracker.ceph.com/issues/39327
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #27270 from theanalyst/beast-fe-ipv6

rgw: normalize v6 endpoint behaviour for the beast frontend

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #26281 from smithfarm/wip-git-describe-long

make-dist: set version number only once

Reviewed-by: Tim Serong <tserong@suse.com>

Merge pull request #28045 from cbodley/wip-39635

cls/user: cls_user_set_buckets_info overwrites creation_time

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #27980 from cbodley/wip-24991

rgw: delete_obj_index() takes mtime for bilog

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #28031 from ivancich/wip-rgw-admin-unordered

rgw: allow radosgw-admin to list bucket w --allow-unordered

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #27676 from mkogan1/wip-rgw-39380

rgw: fix default_placement containing "/" when storage_class is standard

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27984 from cbodley/wip-test-rgw-multisite-sh

test/rgw: fix test-rgw-multisite.sh script for creating multisite clusters

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #27807 from cyb70289/utf8-encode

common: optimize encode_utf8

Reviewed-by: Kefu Chai <kchai@redhat.com>

cephfs-shell: Remove whitespace after '('

This patch fixes flake8 whitespace error.

Fixes: https://tracker.ceph.com/issues/39717
Signed-off-by: Varsha Rao <varao@redhat.com>

cephfs-shell: Remove unused 'flag' variable

This patch fixes flake8 unused local variable error.

Fixes: https://tracker.ceph.com/issues/39717
Signed-off-by: Varsha Rao <varao@redhat.com>

cephfs-shell: Fix flake8 invalid escape sequence warning

Backslash-character pair is not a valid escape sequence, since Python 3.6
version. Prefixing character 'r' to the escape sequence, fixes the warning.

Fixes: https://tracker.ceph.com/issues/39717
Signed-off-by: Varsha Rao <varao@redhat.com>

cephfs-shell: Remove unnecessary empty lines

This patch fixes flake8 too many blank lines error.

Fixes: https://tracker.ceph.com/issues/39717
Signed-off-by: Varsha Rao <varao@redhat.com>

cephfs-shell: Fix flake8 bare 'except' warnings

Instead of bare 'except', catch just libcephfs exceptions.

Fixes: https://tracker.ceph.com/issues/39717
Signed-off-by: Varsha Rao <varao@redhat.com>

Merge pull request #28051 from dillaman/wip-39521

librbd: fix issues with object-map/fast-diff feature interlock

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #28063 from dillaman/wip-39656

qa/workunits/rbd: use https protocol for devstack git operations

Reviewed-by: Mykola Golub <mgolub@suse.com>

scripts/kubejacker: Fix mgr_plugins target for centos

Also use the latest master as default FROM

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

cmake: define BOOST_SP_DISABLE_THREADS for crimson

because we are single-threaded

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/osd_internal_types: bring moving semantics to ObjectState.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: simplify write handling in PG::do_osd_ops.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

librbd: clone copy-on-write operations should preserve sparseness

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #27870 from theanalyst/rgw-objexp-fixes-cli

rgw: object expirer fixes

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #27926 from tchaikov/wip-38627

pybind/mgr: convert str to int using "int()"

Reviewed-by: Noah Watkins <noahwatkins@gmail.com>

Merge pull request #27914 from dzafman/wip-39304

osd: Include dups in copy_after() and copy_up_to()

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #27961 from dzafman/wip-39582

osd: Output Base64 encoding of CRC header if binary data present

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd: Fix merge_log_dups() not remove dup corresponding to log.tail

The log tail is not inclusive of the log, so not an overlap
Fix unittest_pglog to verify proper operation

Signed-off-by: David Zafman <dzafman@redhat.com>

test: unittest_pglog: Additional test cases for trim, copy_up_to and copy_after

Signed-off-by: David Zafman <dzafman@redhat.com>

test: unittest_pglog: Remove irrelevant config values

osd_min_pg_log_entries and osd_max_pg_log_entries are not involved
in the log handling code being tested.

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Adding standalone test of log copy handling

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge PR #27782 into master

* refs/pull/27782/head:
os/bluestore: protect BlueFS::FileReader::buf
os/bluestore: log 'buffered' in KernelDevice::read_random
os/bluestore: support RocksDB prefetching at BlueFS
os/bluestore: call BlueFS::_init_logger earlier.
tools/ceph-objectstore-tool: dump perf counters on competion.
os/bluestore: optimize mapping offset to extent in BlueFS.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #27928 into master

* refs/pull/27928/head:
mon/OSDMonitor: require nautilus to set pg_autoscale_mode
mon/OSDMonitor: allow pg_num to increase when require_osd_release < N

Reviewed-by: Neha Ojha <nojha@redhat.com>