librbd: fix crash when opening nonexistent snapshot

Fixes: http://tracker.ceph.com/issues/24637
Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #22665 from idryomov/wip-krbd-lvcreate-args

qa: krbd_msgr_segments.t: filter lvcreate output

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #22217 from tianshan/fix_dir_suggest_changes

rgw: fix index update in dir_suggest_changes

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #22344 from neurodrone/add_usage_log_entry_dencoder

cls/rgw: ready rgw_usage_log_entry for extraction via ceph-dencoder

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #22410 from mkogan1/wip-rgw-t23379

rgw: remove expired entries from the cache

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #22440 from yehudasa/wip-24117

cls/rgw: don't assert in decode_list_index_key()

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #22249 from cbodley/wip-qa-rgw-verify-tasks

qa/rgw: reorganize verify tasks

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #22601 from tspmelo/wip-confirmation-modal

mgr/dashboard: Add shared Confirmation Modal

Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #22639 from tspmelo/wip-node-version

mgr/dashboard: Update node version

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #22654 from dmick/wip-mgr-store-doc

mgr/mgr_module.py: fix doc for set_store/set_store_json

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #22666 from yuriw/wip-yuriw-crontab

qa/tests - added luminous-p2p suite to the schedule

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/tests - added luminous-p2p suite to the schedule
(Merge after https://github.com/ceph/ceph/pull/22474 is merged)

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #22536 from yuriw/wip-yuriw-distros-master

qa/tests: changed disto symlink to point to new way using supported OS'es

mgr/mgr_module.py: add 'delete a key' doc comment for set_store

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #22664 from neha-ojha/wip-fix-choose-acting

osd/PG: restrict async_recovery_targets to up osds

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #22432 from tianshan/fix_RGWSyncTraceNode_crash

rgw: fix RGWSyncTraceNode crash in reload

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

qa: krbd_msgr_segments.t: filter lvcreate output

Some versions of lvm emit a log message

  Using default stripesize 64.00 KiB.

which fails the test.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

osd/PG: restrict async_recovery_targets to up osds

When an osd that is part of the acting set and not the up set, gets chosen
as an async_recovery_target, it gets removed from the acting set. Since this
osd is no longer in the up or acting set, it is classified as a stray in
the next peering cycle. This results in choose_acting() looping between two
proposed acting sets.

To avoid this, we will only choose up osds as async_recovery_targets.

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #22054 from theanalyst/wip-civetweb-url-fixes

civetweb: update for url validation fixes

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #22470 from votdev/feature_24436

mgr/dashboard: Replace RGW proxy controller

Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #22658 from joscollin/wip-doc-dashboard-numbers

doc: Fixed the wrong numbers in mgr/dashboard.rst

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

Merge pull request #22657 from ceph/wip-doc-grafana-formatting

doc/dashboard: fix formatting on Grafana instructions

Reviewed-by: Jos Collin <jcollin@redhat.com>

mgr/dashboard: Replace RGW proxy controller

Fixes: http://tracker.ceph.com/issues/24436
To fully support the role based authentication/authorization system it is necessary to replace the RGW proxy controller by separate controllers for RGW user and bucket.

Signed-off-by: Volker Theile <vtheile@suse.com>

doc: Fixed the wrong numbers in mgr/dashboard.rst

Signed-off-by: Jos Collin <jcollin@redhat.com>

doc/dashboard: fix formatting on Grafana instructions

The whole list was in a code block, instead of just the snippets.

Signed-off-by: John Spray <john.spray@redhat.com>

civetweb: update for url validation fixes

The new civetweb 1.10 version in mimic and later is strict on control characters
being url encoded, making url validation more relaxed and passing these through
to rgw where the requisite url validation is done.

Fixes: http://tracker.ceph.com/issues/24158
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #22608 from dillaman/wip-24410

librbd: namespace create/remove/list support

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #22628 from dmick/wip-cli-path

src/ceph.in: dev mode: add build path to beginning of PATH, not end

Merge PR #22585 into master

* refs/pull/22585/head:
osd: fix getting osd maps on initial osd startup

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #22647 from yuriw/wip-yuriw-teuthology-master

qa/tests - added all supported distro

Merge pull request #22648 from idryomov/wip-krbd-bump-lock-timeout

qa: krbd_exclusive_option.sh: bump lock_timeout to 60 seconds

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc/rbd: update man page

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20094 from rjfd/wip-msgr2-docs

doc/dev/msgr2.rst: update of the banner and authentication phases

qa: krbd_exclusive_option.sh: bump lock_timeout to 60 seconds

Avoid sporadic failures in combination with msgr-failures/many.yaml,
where assert_locked() might take over 10 seconds.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/tests - added all supported distro

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

mgr/dashboard: Add shared Confirmation Modal

Replaced 'FlattenConfirmationModalComponent' and
'RollbackConfirmationModalComponent' with the new generic modal.

Any new confirmation modals should use the new component.

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #22629 from dmick/master

doc/mgr/plugins.rst: explain more about the plugin command protocol

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #22528 from rjfd/wip-dashboard-fix-rest-controller

mgr/dashboard: restcontroller: minor improvements and bug fixes

Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Update node version

Updated node version to the latest LTS.

Updated 'HACKING.rst' to reflect the current node/npm version requirements.

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #22609 from qwren/qat-decompress

compressor: building error for QAT decompress

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/mgr/plugins.rst: explain more about the plugin command protocol

Signed-off-by: Dan Mick <dan.mick@redhat.com>

src/ceph.in: dev mode: add build path to beginning of PATH, not end

The build/ executables go with the LD_LIBRARY_PATH and PYTHONPATH

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #21635 from yangdongsheng/qos_read_write

librbd: support bps throttle and throttle read and write seperately.

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #22587 from trociny/wip-24545

librbd: deep_copy: don't hide parent if zero overlap for snapshot

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #22627 from ceph/rm-24456

lvm: when osd creation fails log the exception

Reviewed-by: Alfredo Deza <adeza@redhat.com>

lvm: when osd creation fails log the exception

Fixes: http://tracker.ceph.com/issues/24456
Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #22509 from taodd/osdc-optimize

osdc: optimize the code doing the BufferHead mapping

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #22596 into master

* refs/pull/22596/head:
os/bluestore: use vector instead of set for zombies
os/bluestore: reuse zombie OpSequencers by collection id
qa/suites/rados/objecstore/backends/objectstore: capture coredumps
os/bluestore: more debug output
os/bluestore: print cnode from _open_collections
os/bluestore: print cnode on fsck
qa/suites/rados/objecstore: preserve data dir for ceph_test_objecstore

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: use vector instead of set for zombies

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: reuse zombie OpSequencers by collection id

We can get a sequence that deletes and then recreates a collection where
the transaction removing the collection is delayed (due to pending IO on
its sequencer) but colleciton create is not (new sequencer).

Avoid any such reordering by recycling the old collection's sequencer if
the zombie_osr has not been reaped yet.

Fixes: http://tracker.ceph.com/issues/24550
Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/objecstore/backends/objectstore: capture coredumps

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #22606 from yuriw/wip-yuriw-crontab

qa/tests - added powercycle suite to run on weekly basis on master and mimic

Merge pull request #22495 from taodd/osdc-bugfix

osdc: Fix the wrong BufferHead offset

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #22570 from zy751713126/test_for_set_snap

pybind/rbd: add assert_raise in test set_snap

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #22566 from liewegas/wip-rbd-table

tools/rbd/action: align column headers left

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #22569 into master

* refs/pull/22569/head:
qa/cephfs: add test for "dump inode"
mds: add dump inode command

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

librbd: remove unused header

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: fix gmock warnings introduced after 9e5b87c

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: deep_copy: update end_size only if zero interval caused truncate

The problem shown up when stripping was used, and assembling a
nonexistent destination object from source objects, and one of these
objects existed and snap diff returned a zero interval. The non-zero
end_size in that case triggered invalid object map update.

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: deep_copy: don't hide parent if zero overlap for snapshot

Fixes: http://tracker.ceph.com/issues/24545
Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #17414 from fangyuxiangGL/system

rgw: admin/user ops dump user 'system' flag

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #22604 from alfredodeza/wip-bluestore-config-ref

doc/bluestore: data doesn't use two partitions (ceph-disk era)

Reviewed-by: Sage Weil sage@redhat.com

Merge PR #22595 into master

* refs/pull/22595/head:
common/options: dump flags

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge PR #22582 into master

* refs/pull/22582/head:
mon: destroy-new -> purge-new
mon/OSDMonitor: fix NEW flag adjustment in destroy[-new]

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge PR #22602 into master

* refs/pull/22602/head:
doc/releases/mimic: note lack of debian builds for mimic

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

githubmap: update contributors

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #22489 into master

* refs/pull/22489/head:
ceph-volume lvm.common use destroy-new, doesn't need admin keyring

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #22343 into master

* refs/pull/22343/head:
qa/standalone remove ceph-disk from activate_osd helper
cmake: remove subman.sh tests
test remove ceph-disk directory
debian: remove ceph_detect_init python files from base
qa/standalone remove virtualenv paths for ceph-disk and ceph-detect-init
debian: remove ceph-disk ceph-detect-init python files
rpm: remove ceph-disk ceph-detect-init python files
alpine: remove ceph-disk ceph-detect-init python files
alpine: remove ceph-osd and parttypeuuid udev rules
debian: remove ceph-osd and parttypeuuid udev rules
rpm: remove ceph-osd and parttypeuuid udev rules
ceph-helpers.sh: remove ceph-disk, set up osds directly
CMakeLists.txt: add back CEPH_BUILD_VIRTUALENV
alpine: remove ceph-disk, add ceph-volume in APKBUILD.in
upstart: remove ceph-disk activation call
doc/install add anchor for manual osd deployment in freebsd guide
doc/dev remove ceph-disk from freebsd guide, link to manual reference
doc/dev/config-key remove ceph-disk references
doc/dev remove ceph-disk.rst
doc/dev: change ceph-disk suite examples for ceph-deploy
doc/man_index: remove ceph-disk, ceph-detect-init refs
doc/install: remove ceph-disk from freebsd examples
doc/rados remove ceph-disk from man references
doc/man remove ceph-disk ref from ceph-volume-systemd
doc/man: update reference from ceph-disk to ceph-volume
doc/man: remove ceph-disk, ceph-detect-init from cmake
doc/man/ceph-volume remove doc reference to ceph-disk
doc/man: remove ceph-disk, ceph-detect-init
qa/suites: remove ceph-disk
qa/run-standalone.sh: remove requirement for ceph-detect-init virtualenv
qa/workunits: remove ceph-detect-init from rbdmapfile test
qa/workunits: remove ceph-detect-init from ceph-helpers-root.sh
qa/workunits: remove ceph-disk
build: remove ceph-disk from freebsd script
cmake: remove ceph-disk, ceph-detect-init tox tests
init-ceph: remove ceph-disk
cmake: remove top-level entries for ceph-disk, ceph-detect-init
debian: remove ceph-detect-init references
debian: remove ceph-disk references
src: remove ceph-detect-init tool
rpm: remove ceph-disk, ceph-detect-init from spec file
test: remove subman script
script: remove subman script
udev: remove parttypeuuid rules for ceph-disk
tool remove ceph-disk from ps-ceph.pl
upstart: remove ceph-disk conf file
systemd: remove ceph-disk from CMakeLists
systemd: remove ceph-disk service
udev: remove ceph-disk rules
src: remove ceph-disk tool

Merge pull request #22622 from jcsp/wip-doc-mgr

doc/mgr: note need for module.py file in plugins

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

Merge pull request #22600 from votdev/issue_24525_improvements

mgr/dashboard: Fix more layout issues in UI

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

doc/mgr: note need for module.py file in plugins

This is kind of a kludgy thing but let's at least
document it!

Signed-off-by: John Spray <john.spray@redhat.com>

mgr/dashboard: Fix layout issues in UI

Improve the CSS3 selectors and use the margin-left values from 'btn-toolbar' Bootstrap.

Signed-off-by: Volker Theile <vtheile@suse.com>

compressor: building error for QAT decompress

The parameter of decompress changes from 'bufferlist::iterator' to
'bufferlist::const_iterator', but chis change miss class QatAccel,
and so the building could not be successful for QAT environment.

This patch fix this error with '-DWITH_QATZIP=ON'.

Signed-off-by: Qiaowei Ren <qiaowei.ren@intel.com>

rbd: add namespace create/remove/ls actions

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: add namespace create/remove/list API methods

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #22480 from dzafman/wip-standalone

Fixes for standalone tests

Reviewed-by: Kefu Chai <kchai@redhat.com>

test: run-standalone move aside cores found before testing starts

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Fail immediately if some operations fail

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Fix standalone main usage

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Fix some function desciptions

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Put files in private test directory

Signed-off-by: David Zafman <dzafman@redhat.com>

doc/releases/mimic: note lack of debian builds for mimic

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tests - added powercycle suite to run on weekly basis on master and mimic

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

doc/bluestore: data doesn't use two partitions (ceph-disk era)

Signed-off-by: Alfredo Deza <adeza@redhat.com>

osd: fix getting osd maps on initial osd startup

Commit 89d0c357964ff28b83ac541c4d946f712a7bee47 introduced a check for
deleted pools that relies on getting an older osd map that might not be
available in some situations on OSDs starting up for the very first
time.

fixes https://tracker.ceph.com/issues/24423

Signed-off-by: Paul Emmerich <paul.emmerich@croit.io>

cls/rbd: add namespace_create/remove/list API methods

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls/rbd: add dir_state_set/asset methods

These new methods will be utilized to add protection for namespace
creation/removal

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test/librados_test_stub: implement namespace support

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #22597 from votdev/bug_24525

mgr/dashboard: Fix layout issues in UI

Reviewed-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #22416 from votdev/get_rgw_api_user_via_rest

mgr/dashboard: Get user ID via RGW Admin Ops API

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

ceph-volume lvm.common use destroy-new, doesn't need admin keyring

Signed-off-by: Alfredo Deza <adeza@redhat.com>

common/options: dump flags

Also document.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Add missing blanks to fix layout issue of various texts.

This behaviour starts to occur after switching to Angular 6.

Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #22473 from Devp00l/wip-static-testbed-changes

mgr/dashboard: Removed unnecessary fake services from unit tests

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #21850 from jan--f/add-monitoring-grafana-dash

add monitoring subdir and Grafana cluster dashboard

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

Merge pull request #22521 from tchaikov/wip-cmake-seastar-cleanup

cmake: only build extra boost libraries only if WITH_SEASTAR

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

qa/cephfs: add test for "dump inode"

Signed-off-by: dongdong tao <tdd21151186@gmail.com>

Merge PR #22583 into master

* refs/pull/22583/head:
test/objectstore/CMakefiles: fix unittest_alloc_bench build

Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #21494 from theanalyst/rgw-reorder-auth

rgw: have a configurable authentication order

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

test/objectstore/CMakefiles: fix unittest_alloc_bench build

Broken by 434589a3206aafe94de5a3b95b67eddb2cfc3bdb.  The add_ceph_unittest
helper does more than just add this to the list of tests--it also adjusts
linking and build options.

Signed-off-by: Sage Weil <sage@redhat.com>