rgw: The delete markers generated by object expiration should have owner
attribute.

Fixes: http://tracker.ceph.com/issues/24568
Signed-off-by: Zhang Shaowen <zhangshaowen@cmss.chinamobile.com>

Merge pull request #22497 from liewegas/wip-24486

osd/Session: fix invalid iterator dereference in Session::have_backoff()

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #22501 into master

* refs/pull/22501/head:
messages/MOSDPGLog: encode epoch for query_epoch for pre-nautilus OSDs

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #22499 into master

* refs/pull/22499/head:
os/bluestore: fix SharedBlobSet deregister race

Reviewed-by: git push gh masterVarada Kari <varadaraja.kari@flipkart.com>

githubmap: update contributors

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #22496 from liewegas/wip-localpool-config

qa/workunits/mgr/test_localpool.sh: use new config syntax

Merge pull request #22467 from zhongyimao/update-the-vstart-options

doc: document vstart options

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22009 from markhpc/wip-bs-cache-autotune

common/PriorityCache: First Step toward priority based caching

Reviewed-by: Sage Weil <sage@redhat.com>

mstop.sh: Fix print information

The print information of mstop.sh is as following:

entity=osd pid=16347 name=osd.1
kill  16347...kill  16347...16462
pid=16462
entity=osd pid=16462 name=osd.2
kill  16462...kill  16462...[root@maozy build]#

It's not clear enough, so fix it.

Signed-off-by: Mao Zhongyi <maozy.fnst@cn.fujitsu.com>

doc: vstart update a few options

Add missing several options in vstart and make
these options arranged in alphabetical order.

Signed-off-by: Mao Zhongyi <maozy.fnst@cn.fujitsu.com>

Merge pull request #22493 from wjwithagen/wjw-include-types-buffer

common: include include/types.h early, otherwise Clang will error.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22494 from wjwithagen/wjw-fix-oath2

cmake: oath lives in liboath

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22298 from p-na/settings-unset

mgr/dashboard: Reset settings to their default values

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #21783 from Devp00l/wip-static-testbed

mgr/dashboard: Resolve TestBed performance issue

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

doc: drop the unsupported option -r from vstart

vstart no longer supports -r option after commit 8dcf8d6e,
but it is not cleaned from mstart, so drop it.

Signed-off-by: Mao Zhongyi <maozy.fnst@cn.fujitsu.com>

messages/MOSDPGLog: encode epoch for query_epoch for pre-nautilus OSDs

This fix is analogous to d5a3ae6008e4399b336f3a72d8dee328f97a0002.  As of
84a1984b7cb70c74eaf517ffc424228fa954ed37 we set query_epoch to a lower
value reliably, but we need to encode the latest epoch for pre-nautilus
OSDs because they do not set last_peering_epoch reliably and havin a lower
query_epoch will cause them to discard this message.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix SharedBlobSet deregister race

In commit 8c8944b2c45ca9dc5b8fd4db1590e1d24206c0b3 we fixed one narrow race
and introduced a new (probably less narrow) one:

A: put shared blob foo, nref = 0, start removing self from set
B: open_shared_blob -> lookup gets nullptr (nref==null), creates a new shared blob
  B: takes lock, sets sb_map[sbid] = newblob
A: gets lock, erases sb_map[sbid]
B: open_shared_blob -> lookup gets null, creates another new shared blob

Fix by only removing the sb_map entry for the nref=0 sb if it still points
to us.  If it doesn't, that means some new blob has shown up in its place.

Fixes: http://tracker.ceph.com/issues/24319
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #22460 from liewegas/wip-upgrade-distro

qa/suites/rados/upgrade: remove stray link

osd/Session: fix invalid iterator dereference in Sessoin::have_backoff()

If p is i->second.end(), we do want to back up a position, but we
shouldn't dereference p for p->first.

Fixes: http://tracker.ceph.com/issues/24486
Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/mgr/test_localpool.sh: use new config syntax

Signed-off-by: Sage Weil <sage@redhat.com>

cmake: cmake: oath lives in liboath

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

common: use include/types.h early on, otherwise Clang will error.

This is due that FreeBSD does not have loff_t as type.
And with the wrong order of includes the standard linux typedef
is included and used.

In file included from /home/wjw/wip/src/common/buffer.cc:21:
/home/wjw/wip/src/include/buffer_raw.h:92:41: error: unknown type name 'loff_t'; did you mean 'off_t'?
    virtual int zero_copy_to_fd(int fd, loff_t *offset) {
                                        ^~~~~~
                                        off_t
/usr/include/sys/uio.h:49:17: note: 'off_t' declared here
typedef __off_t off_t;
                ^
1 error generated.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #22487 from tchaikov/wip-24452

osd: always set query_epoch explicitly for MOSDPGLog

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #22483 from tchaikov/wip-osd-cleanup

osd: do not include Messenger.h if not necessary

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #22488 from ai-traders/fix-gcc-install

install-deps.sh: fix installing gcc on ubuntu when no old compiler

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #22485 into master

* refs/pull/22485/head:
qa: wait longer for osd to flush pg stats

Reviewed-by: Sage Weil <sage@redhat.com>

install-deps: fix installing gcc on ubuntu when no old compiler

Signed-off-by: Tomasz Setkowski <tom@ai-traders.com>

osd: do not include Messenger.h if not necessary

to speed up the compilation, do not include Messenger.h, and use forward
declaration instead.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22415 from trociny/wip-24399

librbd: deep_copy: resize head object map if needed

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #22444 from trociny/wip-24434

librbd: force 'invalid object map' flag on-disk update

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #22456 from liewegas/wip-24373

osd/PG: reset PG on osd down->up; normalize query processing

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #22429 from liewegas/wip-osd-destroy-new

mon: add 'osd destroy-new' command that only destroys NEW osd slots

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: João Eduardo Luís <joao@suse.de>

Merge pull request #22463 from liewegas/wip-bluefs-types

ceph-dencoder: add bluefs types

Reviewed-by: Erwan Velu <erwan@redhat.com>

Merge pull request #22454 from tchaikov/wip-seastar-denc

common: add adaptor for seastar::temporary_buffer

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #22471 from tchaikov/wip-debian/rules-cleanup

debian: drop '-DUSE_CRYPTOPP=OFF' from cmake options

Reviewed-by: Casey Bodley <cbodley@redhat.com>

osd: always set query_epoch explicitly for MOSDPGLog

it's a follow-up change of 339ae18b. also remove the MOSDPGLog
contructor where query_epoch is optional. it's less error-prone if we
make this parameter mandatory.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22478 from liewegas/wip-24452

osd/PG: create new PGs from activate in last_peering_reset epoch

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa: wait longer for osd to flush pg stats

it's the test_create_from_mon()'s counterpart of 38074726.

Fixes: http://tracker.ceph.com/issues/24321
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22428 from dzafman/wip-24396-fix

test: Use a file that should be on all OSes

Reviewed-by: Kefu Chai <kchai@redhat.com>

messages/MOSDPGScan: encode map_epcoh for query_epoch for pre-nautilus peers

Pre-nautilus OSDs do not create last_peering_reset reliably (due to not
having the previous commit's fix in place).  Avoid breaking them during
an upgrade by encoding the map_epcoh in place of the query_epoch.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: create new PGs from activate in last_peering_reset epoch

If we create a new PG (e.g., a backfill target) in the current epoch, it
might be > last_peering_reset.  That can lead to last_peering_reset on
the replica having a higher last_peering_reset than the primary's, which
can then lead to future messages, like pg_scan during backfill, being
ignored.

Fixes: http://tracker.ceph.com/issues/24452
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #22462 from liewegas/wip-bluestore-no-meta-paths

os/bluestore: don't store/use path_block.{db,wal} from meta

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #22426 from alfredodeza/wip-rm24020

ceph-volume expand on the LVM API to create multiple LVs at different sizes

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

mgr/dashboard: Swap configuration without changes

Swap the TestBed configuration in suites.

The touched suites were prettified at the end.

Signed-off-by: Stephan Müller <smueller@suse.com>

ceph-volume util make chown links affect links as well as paths

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.activate chown bluestore symlinks

Signed-off-by: Alfredo Deza <adeza@redhat.com>

os/bluestore: make bad symlink target error messages more informative

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #22439 from shaba/devel/fix-rocksdb

cmake: fix find system rockdb

Reviewed-by: Kefu Chai <kchai@redhat.com>

debian: drop '-DUSE_CRYPTOPP=OFF' from cmake options

cryptopp dependency was dropped in a18f0589

Signed-off-by: Kefu Chai <kchai@redhat.com>

test: add test for seastar denc

Signed-off-by: Kefu Chai <kchai@redhat.com>

denc: genenalize denc interface for seastar::temporary_buffer

Signed-off-by: Kefu Chai <kchai@redhat.com>

denc: specialize for denc(const T&, size_t&, uint64_t)

otherwise GCC complains that 'unsigned long int' is not a class or
namespace when trying to materialize is_const_iterator_v<>. not sure why
SFINAE does not work here, though.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test: add unit test for seastar buffer

Signed-off-by: Casey Bodley <cbodley@redhat.com>

buffer: add conversion operators for seastar types

Signed-off-by: Casey Bodley <cbodley@redhat.com>

buffer: add raw type for seastar buffers

new buffer_seastar.cc implements the buffer::raw types for seastar,
built in an object library outside of libcommon

Signed-off-by: Casey Bodley <cbodley@redhat.com>

buffer: move buffer::raw to new header

this allows new raw buffer types to live outside of buffer.cc, and
potentially outside of libcommon entirely

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cmake: update for accomodate seastar

* add unit_test_framework for appease seastar's find_package() call,
  even we don't build seastar's tests
* some seastar functions declare their return value like:
  const size_t str_len(...). and GCC does not like the "const" in it.
  so silence it

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add WITH_SEASTAR option

Signed-off-by: Casey Bodley <cbodley@redhat.com>

seastar: add seastar as a submodule

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22430 from tchaikov/wip-find_package-oath

cmake: find liboath using the correct name

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Erwan Velu <erwan@redhat.com>

mgr/dashboard: Resolve TestBed performance issue

With this helper function you can easily resolve the TestBed resetting
performance issue. If more tests exists in a test suite, it makes sense
to configure TestBed only once if you are not doing a lot of TestBed
specific stuff (haven't hit the limitation). It will reduce the test
run time by around $tests * 50 %. In my case it was a test suite with
47 tests with a run time of over 30s after using the static test bed
method it ran in 1.2s. The run time was reduced to 0.04 %! This is
equivalent to a speed increase of 2500% (100/0.04)!

For our own security the normal way will be taken if you not
set the _DEV_ configuration variable to true. It will be false when
"run-frontend-unittests.sh" is run.

Signed-off-by: Stephan Müller <smueller@suse.com>

librbd: deep_copy: resize head object map if needed

Fixes: http://tracker.ceph.com/issues/24399
Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #22453 from tspmelo/fix-pool-usage

mgr/dashboard: Fix pool usage not displaying on filesystem page

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #22443 from dillaman/wip-doc-rbd-openstack

doc/rbd: corrected OpenStack Cinder permissions for Glance pool

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #22447 into master

* refs/pull/22447/head:
osd/OSDMap: fix pre-addrvec compat
mon/MgrMap: fix pre-addrvec compat
messages/MMgrBeacon: fix pre-addrvec compat

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd/PG: normalize query processing in Stray and ReplicaActive

A stray PG may end up in ReplicaActive if it is participating in backfill.
However, whether it is or isn't, we should treat queries the same.
Otherwise we end up with weird behaviors like:

 - osd's stray pg moves to ReplicaActive (gets info+log from primary)
 - osd goes down and back up
 - primary restarts peering, request FULLLOG to find missing objects
 - osd ignores FULLLOG because it is ReplicaActive and not Stray

Fixes: http://tracker.ceph.com/issues/24373
Reported-by: Kouya Shimura <kouya@jp.fujitsu.com>
Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: move send_notify() into RecoveryCtx

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: reset PG peering if osd transitions from down -> up

Consider a PG that is stray and ends up in ReplicaActive (because it is
participating as a recovery source).  If it is marked down wrongly and
then comes back up, then the PG will not reset, because there was not
an interval change (the PG is not part of the up or acting sets).
This can leave the PG in an odd state, leading to questionable behavior.
(For example, a stray might be in ReplicaActive and then ignore some
types of query messages.)

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-dencoder: add bluefs types

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: don't store/use path_block.{db,wal} from meta

This reverts most of 8d07fa5ad0772b4d979dc10695d230074d1f1e69.

ceph-volume goes to great lengths to ensure that the symlinks in the
osd dir are accurate.  Having these values here is an opportunity to
get them out of sync.  And that can happen very easily if the initial
mkfs was performed using a /dev/sdX device name (which is unstable
across reboots).  Even after ceph-volume corrects the symlink, bluestore
will continue to use the stale device path.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/upgrade: remove stray link

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #22403 from ricardoasmarques/wip-http-401

mgr/dashboard: Login failure should return HTTP 400

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #22437 from alfredodeza/wip-cv-osd-destroy

ceph-volume tests destroy osds on monitor hosts

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge PR #22445 into master

* refs/pull/22445/head:
doc: fix release notes on snapshot upgrades

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

cmake: fix find system rockdb

- fix path for version.h
- fix typo ROCKDB -> ROCKSDB

Signed-off-by: Alexey Shabalin <shaba@altlinux.org>

mgr/dashboard: Fix pool usage not displaying on filesystem page

Reference to the template was undefined during the init of the component.

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #22451 from wwformat/wip-gcc7-aarch64

install-deps.sh: support install gcc7 in xenial aarch64

Reviewed-by: Kefu Chai <kchai@redhat.com>

install-deps.sh: support install gcc7 in xenial aarch64

Signed-off-by: Yunchuan Wen <yunchuan.wen@kylin-cloud.com>

osd/OSDMap: fix pre-addrvec compat

This was broken by b47d9135d5345816037899cd1d12a5a20b54e6c1.  Old code
does not decode an addrvec as an addr.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MgrMap: fix pre-addrvec compat

This was broken by 7f787704cdcd55df18d204c5251cdd6a3cedd604.  Old code
does not decode an addrvec as an addr.

Take the opportunity to remove the old_available_modules field.

Signed-off-by: Sage Weil <sage@redhat.com>

messages/MMgrBeacon: fix pre-addrvec compat

This was broken by 7f787704cdcd55df18d204c5251cdd6a3cedd604.  Old code
doesn't not decode an addrvec as an addr.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #22388 into master

* refs/pull/22388/head:
ceph-mds: Fix usage
doc: ceph-mds option updates

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22338 into master

* refs/pull/22338/head:
doc: update toctree for add-remove-mds.rst
doc: create add/remove metadata server

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc: fix release notes on snapshot upgrades

Fixes: https://tracker.ceph.com/issues/24435
Introduced-by: d78dda7b812c63b5504f69e9df78bea4fbb37394.
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

librbd: force 'invalid object map' flag on-disk update

when called by object_map::RefreshRequest. Since we really only open
the object map when we own the exclusive lock.

Fixes: http://tracker.ceph.com/issues/24434
Signed-off-by: Mykola Golub <mgolub@suse.com>

ceph-volume tests functional osd destroy on mons only for dmcrypt

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/rbd: corrected OpenStack Cinder permissions for Glance pool

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #22282 from rjfd/wip-dashboard-api-docs

mgr/dashboard: Swagger-UI based Dashboard REST API page

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #22387 from votdev/missing_space_value_unit

mgr/dashboard: Print a blank space between value and unit

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Sebastian Krah <skrah@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

ceph-volume tests/functional destroy filestore osds on the monitors

Signed-off-by: Alfredo Deza <adeza@redhat.com>

mgr/dashboard: Swagger-UI based Dashboard REST API page

Fixes: http://tracker.ceph.com/issues/23898
Signed-off-by: Ricardo Dias <rdias@suse.com>

Merge pull request #21279 from sungjunyoung/master

doc: erasure-code-jerasure: removed default section of crush-device-class

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: find liboath using the correct name

we should reference liboath by the $name in Find${name}.cmake, also the
$name should be consistent when calling find_package_handle_standard_args().
in this change
* rename Findliboath.cmake to FindOATH.cmake to be consistent with other
  find_package() moduless.
* use "OATH" in find_package_handle_standard_args() instead of "oath"
* set the interface properties for OATH::OATH, so the target linking
  against it can reference its header directories and libraries automatically.
* remove the stale comment for find_package_handle_standard_args()
* set OATH_INCLUDE_DIRS and OATH_LIBRARIES to follow the convention of
  find_package(), even they are not used directly in this project.

Reported-by: Erwan Velu <erwan@redhat.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-volume tests/functional destroy bluestore osds on the monitors

Signed-off-by: Alfredo Deza <adeza@redhat.com>

mgr/dashboard: Print a blank space between value and unit.

When displaying values with their unit a blank space must be shown between them, see #15 at https://physics.nist.gov/cuu/Units/checklist.html.

Signed-off-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Login failure should return HTTP 400

Signed-off-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #21644 from p-na/grafana-proxy

mgr/dashboard: Grafana proxy backend

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #21880 from votdev/upgrade_swimlane_datatable_angular_6

mgr/dashboard: Upgrade Swimlane's data-table

Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #22386 from tspmelo/fix-24281

mgr/dashboard: Changed default value of decimal point to 1

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #22404 from ricardoasmarques/wip-optimize-rbd-edit

mgr/dashboard: Do not fetch pool list on RBD edit

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>