bluestore/tools: Add option 'dump' to ceph-kvstore-tool to print both key and values.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

bluestore/tools: fix arguments after global_init accepts and removes some

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

Merge PR #24686 into master

* refs/pull/24686/head:
os/bluestore: show compress and buffered from WriteContext
os/bluestore: fix rename race with trim on replacement onode at old name

Reviewed-by: Jianpeng Ma <jianpeng.ma@intel.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #24229 into master

* refs/pull/24229/head:
common: drop BL_BACKWARD_COMPAT in bufferlist.
osd: OSDMap encoding uses bufferlist::contiguous_filler.
mds: encode_xattrs() uses buffer::list::contiguous_filler.
common: introduce contiguous_filler to optimize ENCODE_START.
common: optimize hole appending in bufferlist.
common: duplicate an encoding macro to suppress warnings.
common: drop backward iteration from bufferlist.

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #24787 into master

* refs/pull/24787/head:
Merge PR #24796 into nautilus
osd: fix heartbeat_reset unlock
Merge PR #24780 into nautilus
Merge PR #24761 into nautilus
Merge PR #24651 into nautilus
osd: fix race between op_wq and context_queue
test: Make sure kill_daemons failure will be easy to find
test: Add flush_pg_stats to make test more deterministic

Merge pull request #24763 from zmc/wip-36416

mgr/dashboard: Map dev 'releases' to master

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge PR #24796 into nautilus

* refs/pull/24796/head:
osd: fix heartbeat_reset unlock

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

osd: fix heartbeat_reset unlock

Fixes 51d8e2457d73c709bfa4f706793696b3ce704ff9, which moved to lock_guard
but didn't remove the unlock call on this exit path.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #24774 into master

* refs/pull/24774/head:
cmake: update dpdk drivers/modules to accomodate dpdk submodule
spdk: update to latest v18.07

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #24666 into master

* refs/pull/24666/head:
include/types: fixed compile warning for signed/unsigned comparison
osd/PrimaryLogPG: uncommitted dup ops should respond with logged return code
osd/PrimaryLogPG: propagate error return codes on object copy_get ops
osd/PGLog: optionally record error return codes for extra_reqids
osd/osd_types: include PG log return codes in object copy data

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #24688 into master

* refs/pull/24688/head:
common: make ceph_abort store same crash info as ceph_assert
global: store assert msg in global and dump to crash meta
pybind/mgr: make 'ceph crash ls' output sorted list
log: don't clear ring when dump_recent is called
ceph-crash: make clear to user that 'posted' should be directory

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #24780 into nautilus

* refs/pull/24780/head:
osd: take heartbeat_lock before checking for session

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

Merge PR #24780 into master

* refs/pull/24780/head:
osd: take heartbeat_lock before checking for session
Merge PR #24725 into nautilus
qa/tasks/qemu: use unique clone directory to avoid race with workunit
mds: add missing mds_lock

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

Merge PR #24778 into master

* refs/pull/24778/head:
rpm: use %license macro for packaging license file

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #24792 from falcon78921/wip-doc-grammar1

doc: fixed typo in man page

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24702 from wjwithagen/wjw-fix-blkdev-serial-const

common: mark BlkDev::serial() const to match with its declaration

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: fixed minor grammar error

Changed buchket to bucket

Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>

Merge pull request #24785 from falcon78921/wip-docs-36605

doc: purge subcommand link broken

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: used ceph osd command ref label

Referenced purge subcommand info via ceph osd command label.
Fixes: https://tracker.ceph.com/issues/36605
Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>

Merge PR #24761 into nautilus

* refs/pull/24761/head:
osd: fix race between op_wq and context_queue

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge PR #24651 into nautilus

* refs/pull/24651/head:
test: Make sure kill_daemons failure will be easy to find
test: Add flush_pg_stats to make test more deterministic

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #24743 from rzarzynski/wip-osd-avoid-osdmap-refcounting

 core: avoid unnecessary refcounting of OSDMap on OSD's hot paths

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #24657 from xiexingguo/wip-rm-device-class-fix

mon/OSDMonitor: two "ceph osd crush class rm" fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #24585 into master

* refs/pull/24585/head:
doc: add developer documentation on new cephfs reclaim interfaces

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #21094 from guzhongyan/dne-cleanup

src: no 'dne' acronym in user cmd output

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #22633 from dongbula/add-dbstatistics-for-filestore

OSD: add impl for filestore to get dbstatistics

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #23528 from jcsp/wip-osdmon-application

mon: don't commit osdmap on no-op application ops

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

mgr/dashboard: Map dev 'releases' to master

In CephReleaseNamePipe, we used to blindly return the "release name" portion of
the version string. This ends up e.g. returning 'nautilus' for master right
now, which causes us to link to nonexistent documentation on ceph.com.  This
change causes builds marked as 'dev' (as opposed to 'stable') to report
'master' as their release name.

Fixes: https://tracker.ceph.com/issues/36416
Signed-off-by: Zack Cerza <zack@redhat.com>

osd: take heartbeat_lock before checking for session

When we open a connection, there is a short window before we attach
the session.  If a fault happens quickly, we won't get the reset, and
will persistently fail to send osd pings.

Move the lock up to avoid this.  Note that we should rarely really see
connections without sessions here anyway (except when this specific
race happens), so this should have no negative impact (by taking the lock
when we weren't before).

Fixes: http://tracker.ceph.com/issues/36602
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #24504 from ErwanAliasr1/evelu-ceph-volume-choose_disk

Additional work on ceph-volume to add some choose_disk capabilities

Reviewed-by: Alfredo Deza <adeza@redhat.com>

osd: fix race between op_wq and context_queue

        ThreadA                                                 ThreadB
  sdata->shard_lock.Lock();
  if (sdata->pqueue->empty() &&
     !(is_smallest_thread_index && !sdata->context_queue.empty())) {

    void queue(list<Context *>& ls) {
        bool empty = false;
                                                                       {
                                                                         std::scoped_lock l(q_mutex);
                                                                         if (q.empty()) {
                                                                           q.swap(ls);
                                                                           empty = true;
                                                                         } else {
                                                                           q.insert(q.end(), ls.begin(), ls.end());
                                                                         }
                                                                       }

                                                                       if (empty) {
                                                                         mutex.Lock();
                                                                         cond.Signal();
                                                                         mutex.Unlock();
                                                                       }
                                                                    }

     sdata->sdata_wait_lock.Lock();
    if (!sdata->stop_waiting) {

Fix by simply rechecking that context_queue is empty after taking the
wait lock.  We still check it without taking that lock to keep the hot/busy
path fast (we avoid the wait lock in general) at the expense of taking
the context_queue qlock twice in the idle/wait path (where we don't care
so much about additional latency/cycles).

Fixes: http://tracker.ceph.com/issues/36473
Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #24703 into master

* refs/pull/24703/head:
common: prefer std::size to ARRAY_SIZE macro

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #24712 into master

* refs/pull/24712/head:
mgr/DaemonServer: "osd safe-to-destroy" - more verbose output

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #24713 into master

* refs/pull/24713/head:
mon: drop repeated 'goodchars' and add osd crush ls testcase

Reviewed-by: João Eduardo Luís <joao@suse.de>

Merge PR #24724 into master

* refs/pull/24724/head:
test/librados/aio: remove unused callbacks
test/librados/aio: wait for all completions properly

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #24736 into master

* refs/pull/24736/head:
os/tests: should read size_t options using get_val<Option::size_t>()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #24767 from votdev/issue_36581

mgr/dashboard/qa: Fix various vstart_runner.py issues

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: John Spray <john.spray@redhat.com>

rpm: use %license macro for packaging license file

see https://rpm-guide.readthedocs.io/en/latest/rpm-guide.html#rpm-macros

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: update dpdk drivers/modules to accomodate dpdk submodule

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24584 from wido/docs-zabbix-template

doc: Update mgr/zabbix plugin documentation with link to Zabbix template

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #24606 from p-na/osd-actions-fe

mgr/dashboard: Add support for managing individual OSD settings/characteristics in the frontend

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

doc: Update mgr/zabbix plugin documentation with link to Zabbix template

Signed-off-by: Wido den Hollander <wido@42on.com>

Merge pull request #24297 from ukernel/wip-36192

osdc: reduce ObjectCacher's memory fragments

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

spdk: update to latest v18.07

Fixes: http://tracker.ceph.com/issues/36550
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24756 from tspmelo/wip-contribs

.githubmap, .mailmap, .organizationmap: update contributors

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #24727 from zmc/wip-dashboard-gzip

mgr/dashboard: Enable gzip compression

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

.githubmap, .mailmap, .organizationmap: update contributors

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard/qa: CephfsTest - admin_socket() got an unexpected keyword argument 'timeout'

Adapt method arguments of LocalRemote::run() according to teuthology.orchestra.run.run() (see https://github.com/ceph/teuthology/blob/master/teuthology/orchestra/run.py#L364) to be able to run QA tests locally in a vstart cluster.

Fixes: http://tracker.ceph.com/issues/36581
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #24720 from gauravsitlani/master

doc: added rbd default features

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc: add developer documentation on new cephfs reclaim interfaces

Tracker: http://tracker.ceph.com/issues/36395
Signed-off-by: Jeff Layton <jlayton@redhat.com>

doc: added rbd default features

Fixes: https://tracker.ceph.com/issues/15000
Added RBD default features description

Signed-off-by: Gaurav Sitlani <gauravsitlani@riseup.net>

Merge PR #24716 into master

* refs/pull/24716/head:
test/log: drop redundant test case
common/StackStringStream: don't reserve before every insert

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #24283 into master

* refs/pull/24283/head:
client: support for exporting multiple subdirectories in faked mode

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #24050 into master

* refs/pull/24050/head:
Revert "mds: adjust export size after renaming directory into freezing subtree"
mds: cleanup nested auth pins tracking

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #24700 from cbodley/wip-27221

rgw: support server-side encryption when SSL is terminated in a proxy

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #24721 from cbodley/wip-librados-mono-timeout

librados: fix unitialized timeout in wait_for_osdmap

Reviewed-by: Mohamad Gebai <mgebai@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24680 from cbodley/wip-36537

rgw: fix vector index out of range in RGWReadDataSyncRecoveringShardsCR

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #24677 from cbodley/wip-rgw-beast-derr

rgw: beast frontend logs socket errors at level 4

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge PR #24731 into master

* refs/pull/24731/head:
rados: fix admin/build-doc warnings

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #24641 from ricardoasmarques/fix-cloning-message

mgr/dashboard: Fix long running RBD cloning / copying message

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #24587 from ceph/wip-rm36363

ceph-volume: adds a --prepare flag to `lvm batch`

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #24750 from iotcg/master

doc: give pool name if default pool rbd is not created

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc: give pool name if default pool rbd is not created

unless creating the default pool named to be "rbd",
it will fail without pointing the pool name when
using rbd command.

Signed-off-by: Changcheng Liu <changcheng.liu@intel.com>

ceph-volume: update man page for batch --prepare

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

mgr/dashboard: Fix long running RBD cloning / copying message

Fixes: https://tracker.ceph.com/issues/36484
Signed-off-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #24693 from idryomov/wip-is-queued

msg/async: fix is_queued() semantics

Reviewed-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: Add support for managing individual OSD settings/characteristics in the frontend

Fixes: http://tracker.ceph.com/issues/35448
Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

Merge pull request #24739 from tspmelo/wip-py3-itervalues

ceph.in: Add support for python 3

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23962 from aclamk/fio-fix-compilation

objectstore/test/fio: Fixed fio compilation when tcmalloc is used.

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #24725 into nautilus

* refs/pull/24725/head:
mds: add missing mds_lock

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Enable gzip compression

This is related to http://tracker.ceph.com/issues/36453. It is far from
a complete solution, but seems like a positive move.

I tested this change by first disabling my browser cache, and then used
the /docs endpoint to query /api/dashboard/health. Before compression:
  Content-Length: 60748
  Time: 615ms
After:
  Content-Length: 7505
  Time: 92ms

Then, I logged into the dashboard as normal and reloaded the page once I
was in. Some values for the reload operation before compression:
  Total page load time: 58.48s
  vendor.js Content-Length: 6486025
  vendor.js time: 48.09s
After:
  Total page load time: 14.55s
  vendor.js Content-Length: 1143178
  vendor.js time: 4.50s

Signed-off-by: Zack Cerza <zack@redhat.com>

DashboardTestCase: add assertHeaders()

Signed-off-by: Zack Cerza <zack@redhat.com>

LocalCephManager.admin_socket: add timeout kwarg

This fixes "TypeError: admin_socket() got an unexpected keyword argument
'timeout'". The value is never used.

Signed-off-by: Zack Cerza <zack@redhat.com>

core: avoid get_epoch()-driven refcounting on OSDMap in ReplicatedBackend.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

core: avoid get_epoch()-driven refcounting on OSDMap in PrimaryLogPG.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

ceph.in: Add support for python 3

This will allow the condition to work in both python 2 and 3 environments.

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #24738 from alfredodeza/wip-rm36492

ceph-volume: do not send (lvm) stderr/stdout to the terminal, use the logfile

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge PR #24665 into master

* refs/pull/24665/head:
common: remove unused ioprio thread/wq infrastructure

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge PR #24701 into master

* refs/pull/24701/head:
os/bluestore: fix race between SharedBlobSet::lookup and SharedBlob::put

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #24707 into master

* refs/pull/24707/head:
build/ops: drop sgdisk build/runtime dependencies

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge PR #24687 into master

* refs/pull/24687/head:
src/test: fix unordered manifest-unset op

Reviewed-by: Sage Weil <sage@redhat.com>

rados: fix admin/build-doc warnings

Fixed admin/build-doc warnings in librados.h
and rados_types.h

Signed-off-by: Jos Collin <jcollin@redhat.com>

ceph-volume tests.api update monkeypatching to use **kw

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests allow **kw when monkeypatching

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume api.lvm: do not spit out errors on the terminal from LVM

Signed-off-by: Alfredo Deza <adeza@redhat.com>

qa/tasks/qemu: use unique clone directory to avoid race with workunit

If there is a workunit task associated with the same client, the two
tasks will attempt to clone the suite repo to the same directory.
Worse, if it's parallel tasks, the two clones will clobber each
other.

Fixes: http://tracker.ceph.com/issues/36542
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 5d56014c61b107dcb5d05c2221c2e844324f304c)

os/tests: should read size_t options using get_val<Option::size_t>()

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24722 from zmc/rbd-redirect

mgr/dashboard: Redirect /block to /block/rbd

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Simplify frontend test helper

Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

mgr/dashboard: TableActionsComponent cleanup

Rename `buttonCondition` to `canBePrimary`

Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

mgr/dashboard: Rename `DeletionModalComponent` to `CriticalConfirmationModalComponent`

Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

mgr/dashboard: Cleanup OSD components

Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

common: prefer std::size to ARRAY_SIZE macro

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

Merge pull request #19964 from runsisi/wip-build

cmake: move ceph-osdomap-tool, ceph-monstore-tool out of ceph-test

Reviewed-by: Kefu Chai <kchai@redhat.com>

librados: fix unitialized timeout in wait_for_osdmap

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge PR #24540 into master

* refs/pull/24540/head:
mgr/status: fix fs status subcommand did not show standby-replay MDS' perf info

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mds: add missing mds_lock

Fixes: http://tracker.ceph.com/issues/36573
Introduced-by: ecbd4a8aa8e6c1c72af4e0be15e0340629bfdc3a
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #24709 into master

* refs/pull/24709/head:
PendingReleaseNotes: note about cephfs client state reclaim

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #24466 into master

* refs/pull/24466/head:
common: assert if buffer advance length overflow

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #24529 into master

* refs/pull/24529/head:
mds: cleanup command descriptions

Reviewed-by: Zheng Yan <zyan@redhat.com>