ceph-volume: add inventory command

The inventory command provides information about a nodes disk inventory.
Existing logical volumes on a disk or one of its partitions are scanned
and reported.
The output can be formatted as plain text or json.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge PR #24668 into master

* refs/pull/24668/head:
mgr/DaemonState: fix get_config_defaults locking
mgr: adjust DaemonState device index when daemon metadata is updated
mgr/DaemonState: clear devices in set_metadata

Reviewed-by: Tim Serong <tserong@suse.com>

Merge pull request #24773 from sileht/no-scripts

ceph-volume: use console_scripts

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #24715 from ricardoasmarques/fix-36564

mgr/dashboard: Fix ts error on iSCSI page

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #24735 from b-ranto/wip-rpm-dashboards

build/ops: rpm: Package grafana dashboards

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23569 from jcsp/wip-kubejacker

script/kubejacker: include cls libs

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #22833 from rjfd/wip-dashboard-jwt

mgr/dashboard: JWT authentication

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #21243 from majianpeng/osd-misc-cleanup

osd/ReplicatedBackend: remove useless assert

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

osd/ReplicatedBackend: remove useless assert.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #13925 from nvedant07/bug_fix_1

osdmaptool/cleanup: Completed osdmaptool's usage

Merge pull request #24791 from falcon78921/wip-docs-demodocs

doc: added demo document changes section

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

mgr/dashboard: run-frontend-unittests: fix CEPH_ROOT initialization

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: frontend: JWT authentication implementation

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: run-backend-api-request.sh: support for JTW tokens

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: backend: JWT based authentication

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: Fix ts error on iSCSI page

This error only happens until initiator is connected to the target.

Fixes: https://tracker.ceph.com/issues/36564
Signed-off-by: Ricardo Marques <rimarques@suse.com>

Merge PR #24686 into master

* refs/pull/24686/head:
os/bluestore: show compress and buffered from WriteContext
os/bluestore: fix rename race with trim on replacement onode at old name

Reviewed-by: Jianpeng Ma <jianpeng.ma@intel.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #24229 into master

* refs/pull/24229/head:
common: drop BL_BACKWARD_COMPAT in bufferlist.
osd: OSDMap encoding uses bufferlist::contiguous_filler.
mds: encode_xattrs() uses buffer::list::contiguous_filler.
common: introduce contiguous_filler to optimize ENCODE_START.
common: optimize hole appending in bufferlist.
common: duplicate an encoding macro to suppress warnings.
common: drop backward iteration from bufferlist.

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #24787 into master

* refs/pull/24787/head:
Merge PR #24796 into nautilus
osd: fix heartbeat_reset unlock
Merge PR #24780 into nautilus
Merge PR #24761 into nautilus
Merge PR #24651 into nautilus
osd: fix race between op_wq and context_queue
test: Make sure kill_daemons failure will be easy to find
test: Add flush_pg_stats to make test more deterministic

Merge pull request #24763 from zmc/wip-36416

mgr/dashboard: Map dev 'releases' to master

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge PR #24796 into nautilus

* refs/pull/24796/head:
osd: fix heartbeat_reset unlock

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

osd: fix heartbeat_reset unlock

Fixes 51d8e2457d73c709bfa4f706793696b3ce704ff9, which moved to lock_guard
but didn't remove the unlock call on this exit path.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #24774 into master

* refs/pull/24774/head:
cmake: update dpdk drivers/modules to accomodate dpdk submodule
spdk: update to latest v18.07

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #24666 into master

* refs/pull/24666/head:
include/types: fixed compile warning for signed/unsigned comparison
osd/PrimaryLogPG: uncommitted dup ops should respond with logged return code
osd/PrimaryLogPG: propagate error return codes on object copy_get ops
osd/PGLog: optionally record error return codes for extra_reqids
osd/osd_types: include PG log return codes in object copy data

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #24688 into master

* refs/pull/24688/head:
common: make ceph_abort store same crash info as ceph_assert
global: store assert msg in global and dump to crash meta
pybind/mgr: make 'ceph crash ls' output sorted list
log: don't clear ring when dump_recent is called
ceph-crash: make clear to user that 'posted' should be directory

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #24780 into nautilus

* refs/pull/24780/head:
osd: take heartbeat_lock before checking for session

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

Merge PR #24780 into master

* refs/pull/24780/head:
osd: take heartbeat_lock before checking for session
Merge PR #24725 into nautilus
qa/tasks/qemu: use unique clone directory to avoid race with workunit
mds: add missing mds_lock

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

Merge PR #24778 into master

* refs/pull/24778/head:
rpm: use %license macro for packaging license file

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #24792 from falcon78921/wip-doc-grammar1

doc: fixed typo in man page

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24702 from wjwithagen/wjw-fix-blkdev-serial-const

common: mark BlkDev::serial() const to match with its declaration

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: fixed minor grammar error

Changed buchket to bucket

Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>

doc: added demo document changes section

Added a brief section about how to demo Ceph documentation
changes.

Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>

Merge pull request #24785 from falcon78921/wip-docs-36605

doc: purge subcommand link broken

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: used ceph osd command ref label

Referenced purge subcommand info via ceph osd command label.
Fixes: https://tracker.ceph.com/issues/36605
Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>

Merge PR #24761 into nautilus

* refs/pull/24761/head:
osd: fix race between op_wq and context_queue

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge PR #24651 into nautilus

* refs/pull/24651/head:
test: Make sure kill_daemons failure will be easy to find
test: Add flush_pg_stats to make test more deterministic

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #24743 from rzarzynski/wip-osd-avoid-osdmap-refcounting

 core: avoid unnecessary refcounting of OSDMap on OSD's hot paths

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #24657 from xiexingguo/wip-rm-device-class-fix

mon/OSDMonitor: two "ceph osd crush class rm" fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #24585 into master

* refs/pull/24585/head:
doc: add developer documentation on new cephfs reclaim interfaces

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #21094 from guzhongyan/dne-cleanup

src: no 'dne' acronym in user cmd output

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #22633 from dongbula/add-dbstatistics-for-filestore

OSD: add impl for filestore to get dbstatistics

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #23528 from jcsp/wip-osdmon-application

mon: don't commit osdmap on no-op application ops

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

mgr/dashboard: Map dev 'releases' to master

In CephReleaseNamePipe, we used to blindly return the "release name" portion of
the version string. This ends up e.g. returning 'nautilus' for master right
now, which causes us to link to nonexistent documentation on ceph.com.  This
change causes builds marked as 'dev' (as opposed to 'stable') to report
'master' as their release name.

Fixes: https://tracker.ceph.com/issues/36416
Signed-off-by: Zack Cerza <zack@redhat.com>

osd: take heartbeat_lock before checking for session

When we open a connection, there is a short window before we attach
the session.  If a fault happens quickly, we won't get the reset, and
will persistently fail to send osd pings.

Move the lock up to avoid this.  Note that we should rarely really see
connections without sessions here anyway (except when this specific
race happens), so this should have no negative impact (by taking the lock
when we weren't before).

Fixes: http://tracker.ceph.com/issues/36602
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #24504 from ErwanAliasr1/evelu-ceph-volume-choose_disk

Additional work on ceph-volume to add some choose_disk capabilities

Reviewed-by: Alfredo Deza <adeza@redhat.com>

osd: fix race between op_wq and context_queue

        ThreadA                                                 ThreadB
  sdata->shard_lock.Lock();
  if (sdata->pqueue->empty() &&
     !(is_smallest_thread_index && !sdata->context_queue.empty())) {

    void queue(list<Context *>& ls) {
        bool empty = false;
                                                                       {
                                                                         std::scoped_lock l(q_mutex);
                                                                         if (q.empty()) {
                                                                           q.swap(ls);
                                                                           empty = true;
                                                                         } else {
                                                                           q.insert(q.end(), ls.begin(), ls.end());
                                                                         }
                                                                       }

                                                                       if (empty) {
                                                                         mutex.Lock();
                                                                         cond.Signal();
                                                                         mutex.Unlock();
                                                                       }
                                                                    }

     sdata->sdata_wait_lock.Lock();
    if (!sdata->stop_waiting) {

Fix by simply rechecking that context_queue is empty after taking the
wait lock.  We still check it without taking that lock to keep the hot/busy
path fast (we avoid the wait lock in general) at the expense of taking
the context_queue qlock twice in the idle/wait path (where we don't care
so much about additional latency/cycles).

Fixes: http://tracker.ceph.com/issues/36473
Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #24703 into master

* refs/pull/24703/head:
common: prefer std::size to ARRAY_SIZE macro

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #24712 into master

* refs/pull/24712/head:
mgr/DaemonServer: "osd safe-to-destroy" - more verbose output

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #24713 into master

* refs/pull/24713/head:
mon: drop repeated 'goodchars' and add osd crush ls testcase

Reviewed-by: João Eduardo Luís <joao@suse.de>

Merge PR #24724 into master

* refs/pull/24724/head:
test/librados/aio: remove unused callbacks
test/librados/aio: wait for all completions properly

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #24736 into master

* refs/pull/24736/head:
os/tests: should read size_t options using get_val<Option::size_t>()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #24767 from votdev/issue_36581

mgr/dashboard/qa: Fix various vstart_runner.py issues

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: John Spray <john.spray@redhat.com>

rpm: use %license macro for packaging license file

see https://rpm-guide.readthedocs.io/en/latest/rpm-guide.html#rpm-macros

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: update dpdk drivers/modules to accomodate dpdk submodule

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24584 from wido/docs-zabbix-template

doc: Update mgr/zabbix plugin documentation with link to Zabbix template

Reviewed-by: John Spray <john.spray@redhat.com>

ceph-volume: use console_scripts

Using "#!/usr/bin/env python" is not recommended as it's not portable.

setuptools provides an console_scripts entry_point that generates
scripts that always have the good sheban whatever the target operating
system and python version/distribution.

http://tracker.ceph.com/issues/36601

Signed-off-by: Mehdi Abaakouk <sileht@sileht.net>

Merge pull request #24606 from p-na/osd-actions-fe

mgr/dashboard: Add support for managing individual OSD settings/characteristics in the frontend

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

doc: Update mgr/zabbix plugin documentation with link to Zabbix template

Signed-off-by: Wido den Hollander <wido@42on.com>

Merge pull request #24297 from ukernel/wip-36192

osdc: reduce ObjectCacher's memory fragments

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

spdk: update to latest v18.07

Fixes: http://tracker.ceph.com/issues/36550
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24756 from tspmelo/wip-contribs

.githubmap, .mailmap, .organizationmap: update contributors

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #24727 from zmc/wip-dashboard-gzip

mgr/dashboard: Enable gzip compression

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

.githubmap, .mailmap, .organizationmap: update contributors

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard/qa: CephfsTest - admin_socket() got an unexpected keyword argument 'timeout'

Adapt method arguments of LocalRemote::run() according to teuthology.orchestra.run.run() (see https://github.com/ceph/teuthology/blob/master/teuthology/orchestra/run.py#L364) to be able to run QA tests locally in a vstart cluster.

Fixes: http://tracker.ceph.com/issues/36581
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #24720 from gauravsitlani/master

doc: added rbd default features

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc: add developer documentation on new cephfs reclaim interfaces

Tracker: http://tracker.ceph.com/issues/36395
Signed-off-by: Jeff Layton <jlayton@redhat.com>

doc: added rbd default features

Fixes: https://tracker.ceph.com/issues/15000
Added RBD default features description

Signed-off-by: Gaurav Sitlani <gauravsitlani@riseup.net>

Merge PR #24716 into master

* refs/pull/24716/head:
test/log: drop redundant test case
common/StackStringStream: don't reserve before every insert

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #24283 into master

* refs/pull/24283/head:
client: support for exporting multiple subdirectories in faked mode

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #24050 into master

* refs/pull/24050/head:
Revert "mds: adjust export size after renaming directory into freezing subtree"
mds: cleanup nested auth pins tracking

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

rpm: Create ceph-grafana-dashboards package

Signed-off-by: Boris Ranto <branto@redhat.com>

Merge pull request #24700 from cbodley/wip-27221

rgw: support server-side encryption when SSL is terminated in a proxy

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #24721 from cbodley/wip-librados-mono-timeout

librados: fix unitialized timeout in wait_for_osdmap

Reviewed-by: Mohamad Gebai <mgebai@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24680 from cbodley/wip-36537

rgw: fix vector index out of range in RGWReadDataSyncRecoveringShardsCR

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #24677 from cbodley/wip-rgw-beast-derr

rgw: beast frontend logs socket errors at level 4

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge PR #24731 into master

* refs/pull/24731/head:
rados: fix admin/build-doc warnings

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #24641 from ricardoasmarques/fix-cloning-message

mgr/dashboard: Fix long running RBD cloning / copying message

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #24587 from ceph/wip-rm36363

ceph-volume: adds a --prepare flag to `lvm batch`

Reviewed-by: Alfredo Deza <adeza@redhat.com>

cmake: Support grafana dashboard installation

We are currently hosting the grafana dashboards in our repo but we do
not install them. This patch adds the cmake support.

Signed-off-by: Boris Ranto <branto@redhat.com>

Merge pull request #24750 from iotcg/master

doc: give pool name if default pool rbd is not created

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc: give pool name if default pool rbd is not created

unless creating the default pool named to be "rbd",
it will fail without pointing the pool name when
using rbd command.

Signed-off-by: Changcheng Liu <changcheng.liu@intel.com>

ceph-volume: update man page for batch --prepare

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

mgr/dashboard: Fix long running RBD cloning / copying message

Fixes: https://tracker.ceph.com/issues/36484
Signed-off-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #24693 from idryomov/wip-is-queued

msg/async: fix is_queued() semantics

Reviewed-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: Add support for managing individual OSD settings/characteristics in the frontend

Fixes: http://tracker.ceph.com/issues/35448
Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

Merge pull request #24739 from tspmelo/wip-py3-itervalues

ceph.in: Add support for python 3

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23962 from aclamk/fio-fix-compilation

objectstore/test/fio: Fixed fio compilation when tcmalloc is used.

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #24725 into nautilus

* refs/pull/24725/head:
mds: add missing mds_lock

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Enable gzip compression

This is related to http://tracker.ceph.com/issues/36453. It is far from
a complete solution, but seems like a positive move.

I tested this change by first disabling my browser cache, and then used
the /docs endpoint to query /api/dashboard/health. Before compression:
  Content-Length: 60748
  Time: 615ms
After:
  Content-Length: 7505
  Time: 92ms

Then, I logged into the dashboard as normal and reloaded the page once I
was in. Some values for the reload operation before compression:
  Total page load time: 58.48s
  vendor.js Content-Length: 6486025
  vendor.js time: 48.09s
After:
  Total page load time: 14.55s
  vendor.js Content-Length: 1143178
  vendor.js time: 4.50s

Signed-off-by: Zack Cerza <zack@redhat.com>

DashboardTestCase: add assertHeaders()

Signed-off-by: Zack Cerza <zack@redhat.com>

LocalCephManager.admin_socket: add timeout kwarg

This fixes "TypeError: admin_socket() got an unexpected keyword argument
'timeout'". The value is never used.

Signed-off-by: Zack Cerza <zack@redhat.com>

core: avoid get_epoch()-driven refcounting on OSDMap in ReplicatedBackend.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

core: avoid get_epoch()-driven refcounting on OSDMap in PrimaryLogPG.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mgr/DaemonState: fix get_config_defaults locking

One of the callers didn't take the daemon->lock first; fix it.  Also
rename the method so it's more clear the caller does need to lock.

Fixes: http://tracker.ceph.com/issues/36590
Signed-off-by: Sage Weil <sage@redhat.com>

mgr: adjust DaemonState device index when daemon metadata is updated

We need to unregister the old metadata (in particular, its devices
in the DaemonState device indexes) before adjusting and reregistering
the new metadata.

Do the whole thing under the DaemonState lock to that there isn't a window
where the osd metadata isn't present at all.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/DaemonState: clear devices in set_metadata

We are adding new fields in the new data provided; clear out any from the
previous values.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph.in: Add support for python 3

This will allow the condition to work in both python 2 and 3 environments.

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #24738 from alfredodeza/wip-rm36492

ceph-volume: do not send (lvm) stderr/stdout to the terminal, use the logfile

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge PR #24665 into master

* refs/pull/24665/head:
common: remove unused ioprio thread/wq infrastructure

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge PR #24701 into master

* refs/pull/24701/head:
os/bluestore: fix race between SharedBlobSet::lookup and SharedBlob::put

Reviewed-by: Igor Fedotov <ifedotov@suse.com>