qa/tests: update links for centos latest to point to 7.5

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #24406 from ivancich/wip-rgw-renew-reshard-lock

rgw: renew resharding locks to prevent expiration

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20598 from rishabh-d-dave/fix-rbdfuse-look-for-ceph-conf

rbd-fuse: look for ceph.conf in standard locations

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #24829 from p-na/pna-fix-docstring

mgr/dashboard: Fix links to external documentation

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #21767 from cfanz/wip-fix-rgw-race-condition

rgw: avoid race condition in RGWHTTPClient::wait()

Merge pull request #24810 from rhcs-dashboard/35691-landing-page-chart-improvements

mgr/dashboard: Landing Page: chart improvements

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #20794 from ghost/wip-nfs-doc-err

doc: replace rgw_namespace_expire_secs with rgw_nfs_namespace_expire_secs

Merge pull request #21599 from cfanz/wip-fix-rgw-build-error

rgw: fix radosgw-admin build error

Merge pull request #19983 from gohighsec/validate_hitset

mon:validate hit_set values before set

Merge pull request #24848 from joke-lee/wip-doc-radosgw-admin

rgw: add --op-mask in radosgw-admin help info

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24840 from ceph/wip-rm36648

ceph-volume systemd import main so console_scripts work for executable

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

rgw: add --op-mask in radosgw-admin help info

Signed-off-by: yuliyang <yuliyang@cmss.chinamobile.com>

Merge PR #19411 into master

* refs/pull/19411/head:
osd: Step down RecoveryCtx from class to struct

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #19288 into master

* refs/pull/19288/head:
bluestore/BlueRocksEnv: Object initialization after deletion

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Amit Kumar <amitkuma@redhat.com>

Merge PR #19142 into master

* refs/pull/19142/head:
doc/dev/kernel-client-troubleshooting: Add kernel dynamic debuggin

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Amit Kumar <amitkuma@redhat.com>

Merge PR #19146 into master

* refs/pull/19146/head:
doc/dev/logs: Removes non-existence file path

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #19726 into master

* refs/pull/19726/head:
osd: Sanity check, if too full or not

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #24809 into master

* refs/pull/24809/head:
os/bluestore: omit redundant '/' in OSD path for ceph-bluestore-tool if
os/bluestore: improve error handling for migrate ops in
qa/standtalone/osd-bluefs-volume-ops: remove redundant code.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #21009 into master

* refs/pull/21009/head:
rados: fix error message for cache(-try)-flush-evict-all commands

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #24534 into master

* refs/pull/24534/head:
librados: Copy buffer on AIO read

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #24545 into master

* refs/pull/24545/head:
os/bluestore: debug_omit_block_device_write isn't always respected.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #24548 into master

* refs/pull/24548/head:
Objecter: add ignore cache flag if got redirect reply

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

Merge PR #24599 into master

* refs/pull/24599/head:
os/bluestore/KernelDevice: Honor discard_granularity

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #24742 into master

* refs/pull/24742/head:
osd: slightly refactor PrimaryLogPG::do_op.
osd: avoid unnecessary map lookups in PrimaryLogPG::do_op.

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

ceph-volume systemd import main so console_scripts work for executable

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #22132 from yehudasa/wip-menv

ceph-menv: mrun shell environment

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #24755 into master

* refs/pull/24755/head:
mgr/diskprediction: use global device_failure_prediction_mode setting

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #24764 from tspmelo/wip-jest-ci

mgr/dashboard: Reduce Jest logs in CI

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #24633 from Devp00l/wip-issue-36467

mgr/dashboard: Add a unit test form helper class

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Cleanup imports

Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

mgr/dashboard: Fix links to documentations

- Added anchor to Grafana in URL of OSD, Overall Performance
- Fixed URL in Object Gateway, Buckets

Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

Merge PR #24668 into master

* refs/pull/24668/head:
mgr/DaemonState: fix get_config_defaults locking
mgr: adjust DaemonState device index when daemon metadata is updated
mgr/DaemonState: clear devices in set_metadata

Reviewed-by: Tim Serong <tserong@suse.com>

Merge pull request #24773 from sileht/no-scripts

ceph-volume: use console_scripts

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Andrew Schoen <aschoen@redhat.com>

rbd-fuse: look for ceph.conf in standard locations

Fixes: http://tracker.ceph.com/issues/12219
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: Renamed CdFormGroups inital test description

Fixes: https://tracker.ceph.com/issues/36467
Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Use form helper in dashboard tests

Using form helper in pool, RGW user, user and role form tests and
although in the cd-validator tests.

Fixes: https://tracker.ceph.com/issues/36467
Signed-off-by: Stephan Müller <smueller@suse.com>

Merge pull request #24715 from ricardoasmarques/fix-36564

mgr/dashboard: Fix ts error on iSCSI page

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Add a unit test form helper class

This class helps testing forms, most methods were outsourced from the
pool form test suite.

Fixes: https://tracker.ceph.com/issues/36467
Signed-off-by: Stephan Müller <smueller@suse.com>

Merge pull request #24735 from b-ranto/wip-rpm-dashboards

build/ops: rpm: Package grafana dashboards

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23569 from jcsp/wip-kubejacker

script/kubejacker: include cls libs

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #22833 from rjfd/wip-dashboard-jwt

mgr/dashboard: JWT authentication

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Landing Page: chart improvements
- Updated chart colors.
- Added border for chart slices (visual impairment help).
- Adjusted chart vertical centering.

Fixes: https://tracker.ceph.com/issues/35691
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #21243 from majianpeng/osd-misc-cleanup

osd/ReplicatedBackend: remove useless assert

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

osd/ReplicatedBackend: remove useless assert.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #13925 from nvedant07/bug_fix_1

osdmaptool/cleanup: Completed osdmaptool's usage

Merge pull request #24791 from falcon78921/wip-docs-demodocs

doc: added demo document changes section

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

os/bluestore: omit redundant '/' in OSD path for ceph-bluestore-tool if
needed.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

mgr/dashboard: run-frontend-unittests: fix CEPH_ROOT initialization

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: frontend: JWT authentication implementation

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: run-backend-api-request.sh: support for JTW tokens

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: backend: JWT based authentication

Signed-off-by: Ricardo Dias <rdias@suse.com>

os/bluestore: improve error handling for migrate ops in
ceph_bluestore-tool.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

mgr/dashboard: Fix ts error on iSCSI page

This error only happens until initiator is connected to the target.

Fixes: https://tracker.ceph.com/issues/36564
Signed-off-by: Ricardo Marques <rimarques@suse.com>

Merge PR #24686 into master

* refs/pull/24686/head:
os/bluestore: show compress and buffered from WriteContext
os/bluestore: fix rename race with trim on replacement onode at old name

Reviewed-by: Jianpeng Ma <jianpeng.ma@intel.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>

mgr/diskprediction: use global device_failure_prediction_mode setting

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #24229 into master

* refs/pull/24229/head:
common: drop BL_BACKWARD_COMPAT in bufferlist.
osd: OSDMap encoding uses bufferlist::contiguous_filler.
mds: encode_xattrs() uses buffer::list::contiguous_filler.
common: introduce contiguous_filler to optimize ENCODE_START.
common: optimize hole appending in bufferlist.
common: duplicate an encoding macro to suppress warnings.
common: drop backward iteration from bufferlist.

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #24787 into master

* refs/pull/24787/head:
Merge PR #24796 into nautilus
osd: fix heartbeat_reset unlock
Merge PR #24780 into nautilus
Merge PR #24761 into nautilus
Merge PR #24651 into nautilus
osd: fix race between op_wq and context_queue
test: Make sure kill_daemons failure will be easy to find
test: Add flush_pg_stats to make test more deterministic

qa/standtalone/osd-bluefs-volume-ops: remove redundant code.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #24763 from zmc/wip-36416

mgr/dashboard: Map dev 'releases' to master

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge PR #24796 into nautilus

* refs/pull/24796/head:
osd: fix heartbeat_reset unlock

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

osd: fix heartbeat_reset unlock

Fixes 51d8e2457d73c709bfa4f706793696b3ce704ff9, which moved to lock_guard
but didn't remove the unlock call on this exit path.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #24774 into master

* refs/pull/24774/head:
cmake: update dpdk drivers/modules to accomodate dpdk submodule
spdk: update to latest v18.07

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #24666 into master

* refs/pull/24666/head:
include/types: fixed compile warning for signed/unsigned comparison
osd/PrimaryLogPG: uncommitted dup ops should respond with logged return code
osd/PrimaryLogPG: propagate error return codes on object copy_get ops
osd/PGLog: optionally record error return codes for extra_reqids
osd/osd_types: include PG log return codes in object copy data

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #24688 into master

* refs/pull/24688/head:
common: make ceph_abort store same crash info as ceph_assert
global: store assert msg in global and dump to crash meta
pybind/mgr: make 'ceph crash ls' output sorted list
log: don't clear ring when dump_recent is called
ceph-crash: make clear to user that 'posted' should be directory

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #24780 into nautilus

* refs/pull/24780/head:
osd: take heartbeat_lock before checking for session

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

Merge PR #24780 into master

* refs/pull/24780/head:
osd: take heartbeat_lock before checking for session
Merge PR #24725 into nautilus
qa/tasks/qemu: use unique clone directory to avoid race with workunit
mds: add missing mds_lock

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

Merge PR #24778 into master

* refs/pull/24778/head:
rpm: use %license macro for packaging license file

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #24792 from falcon78921/wip-doc-grammar1

doc: fixed typo in man page

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24702 from wjwithagen/wjw-fix-blkdev-serial-const

common: mark BlkDev::serial() const to match with its declaration

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: fixed minor grammar error

Changed buchket to bucket

Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>

doc: added demo document changes section

Added a brief section about how to demo Ceph documentation
changes.

Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>

osd: slightly refactor PrimaryLogPG::do_op.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #24785 from falcon78921/wip-docs-36605

doc: purge subcommand link broken

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: used ceph osd command ref label

Referenced purge subcommand info via ceph osd command label.
Fixes: https://tracker.ceph.com/issues/36605
Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>

Merge PR #24761 into nautilus

* refs/pull/24761/head:
osd: fix race between op_wq and context_queue

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge PR #24651 into nautilus

* refs/pull/24651/head:
test: Make sure kill_daemons failure will be easy to find
test: Add flush_pg_stats to make test more deterministic

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #24743 from rzarzynski/wip-osd-avoid-osdmap-refcounting

 core: avoid unnecessary refcounting of OSDMap on OSD's hot paths

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #24657 from xiexingguo/wip-rm-device-class-fix

mon/OSDMonitor: two "ceph osd crush class rm" fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #24585 into master

* refs/pull/24585/head:
doc: add developer documentation on new cephfs reclaim interfaces

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #21094 from guzhongyan/dne-cleanup

src: no 'dne' acronym in user cmd output

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #22633 from dongbula/add-dbstatistics-for-filestore

OSD: add impl for filestore to get dbstatistics

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #23528 from jcsp/wip-osdmon-application

mon: don't commit osdmap on no-op application ops

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

mgr/dashboard: Map dev 'releases' to master

In CephReleaseNamePipe, we used to blindly return the "release name" portion of
the version string. This ends up e.g. returning 'nautilus' for master right
now, which causes us to link to nonexistent documentation on ceph.com.  This
change causes builds marked as 'dev' (as opposed to 'stable') to report
'master' as their release name.

Fixes: https://tracker.ceph.com/issues/36416
Signed-off-by: Zack Cerza <zack@redhat.com>

rgw: recover from incomplete reshard attempt

In case a reshard attempt is left in an incomplete state, i.e., flags
still show resharding even though the bucket reshard lock isn't being
held, try to recover by taking the bucket reshard lock and clearing
flags associated with resharding.

This change requires access to an RGWBucketInfo object. So call stack
into this function should provide that to prevent unnecessary
work. Changes were made to provide this object.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

osd: take heartbeat_lock before checking for session

When we open a connection, there is a short window before we attach
the session.  If a fault happens quickly, we won't get the reset, and
will persistently fail to send osd pings.

Move the lock up to avoid this.  Note that we should rarely really see
connections without sessions here anyway (except when this specific
race happens), so this should have no negative impact (by taking the lock
when we weren't before).

Fixes: http://tracker.ceph.com/issues/36602
Signed-off-by: Sage Weil <sage@redhat.com>

rgw: move RGWReshardBucket lock to its own separate class

There are other processes beyond resharding that would need to take a
bucket reshard lock (e.g., correcting bucet resharding flags in event
of crash, tools to remove bucket shard information from earlier
versions of ceph). Pulling this logic outside of RGWReshardBucket
allows this code to be re-used.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #24504 from ErwanAliasr1/evelu-ceph-volume-choose_disk

Additional work on ceph-volume to add some choose_disk capabilities

Reviewed-by: Alfredo Deza <adeza@redhat.com>

osd: fix race between op_wq and context_queue

        ThreadA                                                 ThreadB
  sdata->shard_lock.Lock();
  if (sdata->pqueue->empty() &&
     !(is_smallest_thread_index && !sdata->context_queue.empty())) {

    void queue(list<Context *>& ls) {
        bool empty = false;
                                                                       {
                                                                         std::scoped_lock l(q_mutex);
                                                                         if (q.empty()) {
                                                                           q.swap(ls);
                                                                           empty = true;
                                                                         } else {
                                                                           q.insert(q.end(), ls.begin(), ls.end());
                                                                         }
                                                                       }

                                                                       if (empty) {
                                                                         mutex.Lock();
                                                                         cond.Signal();
                                                                         mutex.Unlock();
                                                                       }
                                                                    }

     sdata->sdata_wait_lock.Lock();
    if (!sdata->stop_waiting) {

Fix by simply rechecking that context_queue is empty after taking the
wait lock.  We still check it without taking that lock to keep the hot/busy
path fast (we avoid the wait lock in general) at the expense of taking
the context_queue qlock twice in the idle/wait path (where we don't care
so much about additional latency/cycles).

Fixes: http://tracker.ceph.com/issues/36473
Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>
Signed-off-by: Sage Weil <sage@redhat.com>

rgw: failed resharding clears resharding status from shard heads

Previously, when resharding failed, we restored the shard status on
the bucket info object. However the status on each of the shards was
left indicating a reshard was underway. This prevented some write
operations from taking place, as they would wait for resharding to
complete. This adds the missing functionality. It also makes the
functionality available to other classes via static functions in
RGWBucketReshard.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

rgw: change the bucket reshard lock to exclusive-ephemeral

The bucket reshard lock was simply an exclusive lock that existed on
an object solely for the purpose of representing the lock. This is now
changed to exclusvie-ephemeral lock, so as not to leave these objects
behind.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

cls: add exclusive ephemeral locks that auto-clean

Add a new type of cls lock -- exclusive ephemeral for which the
object only exists to represent the lock and for which the object
should be deleted at unlock. This is to prevent the accumulation of
unneeded objects in the cluster by automatically cleaning them up.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge PR #24703 into master

* refs/pull/24703/head:
common: prefer std::size to ARRAY_SIZE macro

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #24712 into master

* refs/pull/24712/head:
mgr/DaemonServer: "osd safe-to-destroy" - more verbose output

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #24713 into master

* refs/pull/24713/head:
mon: drop repeated 'goodchars' and add osd crush ls testcase

Reviewed-by: João Eduardo Luís <joao@suse.de>

Merge PR #24724 into master

* refs/pull/24724/head:
test/librados/aio: remove unused callbacks
test/librados/aio: wait for all completions properly

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #24736 into master

* refs/pull/24736/head:
os/tests: should read size_t options using get_val<Option::size_t>()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #24767 from votdev/issue_36581

mgr/dashboard/qa: Fix various vstart_runner.py issues

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: John Spray <john.spray@redhat.com>

rpm: use %license macro for packaging license file

see https://rpm-guide.readthedocs.io/en/latest/rpm-guide.html#rpm-macros

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: update dpdk drivers/modules to accomodate dpdk submodule

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24584 from wido/docs-zabbix-template

doc: Update mgr/zabbix plugin documentation with link to Zabbix template

Reviewed-by: John Spray <john.spray@redhat.com>