mgr/dashboard: Prettier KV-table

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Stringify object[] in KV-table

The problem was that object[] weren't handled in the key value table
before. Now they will be stringified to prevent an output like
'[Object object]'.

Signed-off-by: Stephan Müller <smueller@suse.com>

Merge pull request #22082 from tspmelo/wip-angular-6

mgr/dashboard: Update Angular to version 6

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge PR #21825 into master

* refs/pull/21825/head:
os/bluestore: rename new bitmap allocator class to BitmapAllocator.
os/bluestore: perform allocations aligned with min_length  in new bitmap allocator
test/objectstore/unitetest_fastbmap_allocator: replace ASSERT_TRUE with
os/bluestore: respect min_length as allocation granularity for new
os/bluestore: cosmetic new allocator internal method rename.
os/bluestore: properly respect min_length when analysing partially free
os/bluestore: cosmetic cleanup in new bitmap allocator.
os/bluestore: more verbose logging in new bitmap allocator
os/bluestore: default to bitmap allocator for bluestore/bluefs
os/bluestore: remove original bitmap allocator
os/bluestore: align BitMap allocator's init_rm_free/init_add_free parameters with min_alloc_size
os/bluestore: fix improper access to a BitmapFastAllocator::last_pos
test/allocator: move bluestore allocator's benchmarks to a standalone UT
os/bluestore: add new bitmap allocator
test/allocator: get rid off bitmap allocator specifics and introduce new
test/fio: dump mempool on job completion

Reviewed-by: Varada Kari <varadaraja.kari@flipkart.com>

Merge pull request #22504 from tspmelo/wip-table-filtered-numbers

mgr/dashboard: Add filtered rows number in table footer

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge PR #22423 into master

* refs/pull/22423/head:
common/blkdev: fix non-linux build for get_device_id
common/blkdev: do not include vendor in fallback mode
mgr: allow open-ended life expectancy
mgr: use prettified durations for life expectancy
common/ceph_time: very lame timespan_str() helper
mgr: "predicted failure" -> "life expectancy", and as a range
mgr: detailed output for 'ceph device ...'
mgr/DaemonState: skip blank device_ids
osd: skip blank device ids
mgr: track and report device names too
mgr/ActivePyModule: expose 'devices' to python modules
mgr: move dump/print to DeviceState
mgr: implement 'device {set,rm}-predicted-failure' commands
mgr: load persistent device metadata on mgr stat
include/utime: add parse() method
mgr: tolerate racing config-key change during load_store()
mgr: implement 'device {ls,ls-by-daemon,ls-by-host,info}' commands
mgr/DaemonState: populate DeviceState structures
mgr: parse out devids from daemon metadata
common/blkdev: add modelines
osd: include device_ids in metadata
common/blkdev: add get_device_id() helper.

Reviewed-by: Erwan Velu <erwan@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

githubmap: update contributors

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #22522 from tchaikov/wip-githubmap

githubmap: update contributors

os/bluestore: rename new bitmap allocator class to BitmapAllocator.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: perform allocations aligned with min_length  in new bitmap allocator

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

test/objectstore/unitetest_fastbmap_allocator: replace ASSERT_TRUE with
ASSERT_EQ

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: respect min_length as allocation granularity for new
bitmap allocator.
It was used a real minimum threshold before this fix which allowed e.g.
allocated extent length to be equal to min_length + 1.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: cosmetic new allocator internal method rename.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: properly respect min_length when analysing partially free
slots in bitmap allocator.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: cosmetic cleanup in new bitmap allocator.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: more verbose logging in new bitmap allocator

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: default to bitmap allocator for bluestore/bluefs

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: remove original bitmap allocator

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: align BitMap allocator's init_rm_free/init_add_free parameters with min_alloc_size

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: fix improper access to a BitmapFastAllocator::last_pos
from multiple threads.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

test/allocator: move bluestore allocator's benchmarks to a standalone UT

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: add new bitmap allocator

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

test/allocator: get rid off bitmap allocator specifics and introduce new
performance test cases

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

test/fio: dump mempool on job completion

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

mgr/dashboard: Add filtered rows number in table footer

Signed-off-by: Tiago Melo <tmelo@suse.com>

githubmap: update contributors

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22497 from liewegas/wip-24486

osd/Session: fix invalid iterator dereference in Session::have_backoff()

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #22501 into master

* refs/pull/22501/head:
messages/MOSDPGLog: encode epoch for query_epoch for pre-nautilus OSDs

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #22499 into master

* refs/pull/22499/head:
os/bluestore: fix SharedBlobSet deregister race

Reviewed-by: git push gh masterVarada Kari <varadaraja.kari@flipkart.com>

githubmap: update contributors

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #22496 from liewegas/wip-localpool-config

qa/workunits/mgr/test_localpool.sh: use new config syntax

common/blkdev: fix non-linux build for get_device_id

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #22467 from zhongyimao/update-the-vstart-options

doc: document vstart options

Reviewed-by: Kefu Chai <kchai@redhat.com>

common/blkdev: do not include vendor in fallback mode

The 'vendor' file usually contains a string like "ATA", not an actual
vendor.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr: allow open-ended life expectancy

If you leave off the second part of the life expectancy interval, it means
"more than".  e.g., Setting only the first time to today + one month would
mean "more than one month".

Signed-off-by: Sage Weil <sage@redhat.com>

mgr: use prettified durations for life expectancy

Signed-off-by: Sage Weil <sage@redhat.com>

common/ceph_time: very lame timespan_str() helper

Signed-off-by: Sage Weil <sage@redhat.com>

mgr: "predicted failure" -> "life expectancy", and as a range

1- Change terminology from "predicted failure" to "life expectancy", which
has a more intuitive meaning.

2- Change from an expected time stamp to a range.  Any expectancy has an
inherent uncertainty, so a single time is not meaningful.  Instead, express
as a range, e.g. "4-6 weeks from now", where we record the min and max
date as well as when the prediction was made.

This is still a bit awkward, but I'm not sure what is better.  It will
always be a bit awkward to express uncertainty since in precise terms it
is probably a 90% confidence interval or something.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr: detailed output for 'ceph device ...'

For JSON ls* commands, dump the full device metadata.

For plain commands, dump tables with useful information.

This is nicer for humans and jq-users, less friendly for those who want
to write a simple bash loop.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/DaemonState: skip blank device_ids

Signed-off-by: Sage Weil <sage@redhat.com>

osd: skip blank device ids

Signed-off-by: Sage Weil <sage@redhat.com>

mgr: track and report device names too

If you're looking at a specific device by its unique id, it's also nice
to know what device name (sdb etc) it is and on which host.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ActivePyModule: expose 'devices' to python modules

Signed-off-by: Sage Weil <sage@redhat.com>

mgr: move dump/print to DeviceState

Signed-off-by: Sage Weil <sage@redhat.com>

mgr: implement 'device {set,rm}-predicted-failure' commands

Record predicted device failure time.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr: load persistent device metadata on mgr stat

Prune DeviceState when there is no persistent metadata and no daemon
references.  Load persistent metadata from config-key.

Signed-off-by: Sage Weil <sage@redhat.com>

include/utime: add parse() method

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #22009 from markhpc/wip-bs-cache-autotune

common/PriorityCache: First Step toward priority based caching

Reviewed-by: Sage Weil <sage@redhat.com>

mstop.sh: Fix print information

The print information of mstop.sh is as following:

entity=osd pid=16347 name=osd.1
kill  16347...kill  16347...16462
pid=16462
entity=osd pid=16462 name=osd.2
kill  16462...kill  16462...[root@maozy build]#

It's not clear enough, so fix it.

Signed-off-by: Mao Zhongyi <maozy.fnst@cn.fujitsu.com>

doc: vstart update a few options

Add missing several options in vstart and make
these options arranged in alphabetical order.

Signed-off-by: Mao Zhongyi <maozy.fnst@cn.fujitsu.com>

Merge pull request #22493 from wjwithagen/wjw-include-types-buffer

common: include include/types.h early, otherwise Clang will error.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22494 from wjwithagen/wjw-fix-oath2

cmake: oath lives in liboath

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Remove unused code

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Update services to use the new 'provideIn' configuration

Angular 6 introduced a new way to define where a service should be provided,
which helps during the build of the project.

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Update Angular to version 6

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #22298 from p-na/settings-unset

mgr/dashboard: Reset settings to their default values

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #21783 from Devp00l/wip-static-testbed

mgr/dashboard: Resolve TestBed performance issue

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

doc: drop the unsupported option -r from vstart

vstart no longer supports -r option after commit 8dcf8d6e,
but it is not cleaned from mstart, so drop it.

Signed-off-by: Mao Zhongyi <maozy.fnst@cn.fujitsu.com>

messages/MOSDPGLog: encode epoch for query_epoch for pre-nautilus OSDs

This fix is analogous to d5a3ae6008e4399b336f3a72d8dee328f97a0002.  As of
84a1984b7cb70c74eaf517ffc424228fa954ed37 we set query_epoch to a lower
value reliably, but we need to encode the latest epoch for pre-nautilus
OSDs because they do not set last_peering_epoch reliably and havin a lower
query_epoch will cause them to discard this message.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix SharedBlobSet deregister race

In commit 8c8944b2c45ca9dc5b8fd4db1590e1d24206c0b3 we fixed one narrow race
and introduced a new (probably less narrow) one:

A: put shared blob foo, nref = 0, start removing self from set
B: open_shared_blob -> lookup gets nullptr (nref==null), creates a new shared blob
  B: takes lock, sets sb_map[sbid] = newblob
A: gets lock, erases sb_map[sbid]
B: open_shared_blob -> lookup gets null, creates another new shared blob

Fix by only removing the sb_map entry for the nref=0 sb if it still points
to us.  If it doesn't, that means some new blob has shown up in its place.

Fixes: http://tracker.ceph.com/issues/24319
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #22460 from liewegas/wip-upgrade-distro

qa/suites/rados/upgrade: remove stray link

osd/Session: fix invalid iterator dereference in Sessoin::have_backoff()

If p is i->second.end(), we do want to back up a position, but we
shouldn't dereference p for p->first.

Fixes: http://tracker.ceph.com/issues/24486
Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/mgr/test_localpool.sh: use new config syntax

Signed-off-by: Sage Weil <sage@redhat.com>

cmake: cmake: oath lives in liboath

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

common: use include/types.h early on, otherwise Clang will error.

This is due that FreeBSD does not have loff_t as type.
And with the wrong order of includes the standard linux typedef
is included and used.

In file included from /home/wjw/wip/src/common/buffer.cc:21:
/home/wjw/wip/src/include/buffer_raw.h:92:41: error: unknown type name 'loff_t'; did you mean 'off_t'?
    virtual int zero_copy_to_fd(int fd, loff_t *offset) {
                                        ^~~~~~
                                        off_t
/usr/include/sys/uio.h:49:17: note: 'off_t' declared here
typedef __off_t off_t;
                ^
1 error generated.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #22487 from tchaikov/wip-24452

osd: always set query_epoch explicitly for MOSDPGLog

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #22483 from tchaikov/wip-osd-cleanup

osd: do not include Messenger.h if not necessary

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #22488 from ai-traders/fix-gcc-install

install-deps.sh: fix installing gcc on ubuntu when no old compiler

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr: tolerate racing config-key change during load_store()

Signed-off-by: Sage Weil <sage@redhat.com>

mgr: implement 'device {ls,ls-by-daemon,ls-by-host,info}' commands

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/DaemonState: populate DeviceState structures

Track device <-> daemon (and server) mappings.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr: parse out devids from daemon metadata

Look for the "device_ids" key in the metadata and parse out a device
list.

Signed-off-by: Sage Weil <sage@redhat.com>

common/blkdev: add modelines

Signed-off-by: Sage Weil <sage@redhat.com>

osd: include device_ids in metadata

Signed-off-by: Sage Weil <sage@redhat.com>

common/blkdev: add get_device_id() helper.

Signed-off-by: Yaarit Hatuka yaarithatuka@gmail.com

Merge PR #22485 into master

* refs/pull/22485/head:
qa: wait longer for osd to flush pg stats

Reviewed-by: Sage Weil <sage@redhat.com>

install-deps: fix installing gcc on ubuntu when no old compiler

Signed-off-by: Tomasz Setkowski <tom@ai-traders.com>

osd: do not include Messenger.h if not necessary

to speed up the compilation, do not include Messenger.h, and use forward
declaration instead.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22415 from trociny/wip-24399

librbd: deep_copy: resize head object map if needed

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #22444 from trociny/wip-24434

librbd: force 'invalid object map' flag on-disk update

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #22456 from liewegas/wip-24373

osd/PG: reset PG on osd down->up; normalize query processing

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #22429 from liewegas/wip-osd-destroy-new

mon: add 'osd destroy-new' command that only destroys NEW osd slots

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: João Eduardo Luís <joao@suse.de>

Merge pull request #22463 from liewegas/wip-bluefs-types

ceph-dencoder: add bluefs types

Reviewed-by: Erwan Velu <erwan@redhat.com>

Merge pull request #22454 from tchaikov/wip-seastar-denc

common: add adaptor for seastar::temporary_buffer

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #22471 from tchaikov/wip-debian/rules-cleanup

debian: drop '-DUSE_CRYPTOPP=OFF' from cmake options

Reviewed-by: Casey Bodley <cbodley@redhat.com>

osd: always set query_epoch explicitly for MOSDPGLog

it's a follow-up change of 339ae18b. also remove the MOSDPGLog
contructor where query_epoch is optional. it's less error-prone if we
make this parameter mandatory.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22478 from liewegas/wip-24452

osd/PG: create new PGs from activate in last_peering_reset epoch

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa: wait longer for osd to flush pg stats

it's the test_create_from_mon()'s counterpart of 38074726.

Fixes: http://tracker.ceph.com/issues/24321
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22428 from dzafman/wip-24396-fix

test: Use a file that should be on all OSes

Reviewed-by: Kefu Chai <kchai@redhat.com>

messages/MOSDPGScan: encode map_epcoh for query_epoch for pre-nautilus peers

Pre-nautilus OSDs do not create last_peering_reset reliably (due to not
having the previous commit's fix in place).  Avoid breaking them during
an upgrade by encoding the map_epcoh in place of the query_epoch.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: create new PGs from activate in last_peering_reset epoch

If we create a new PG (e.g., a backfill target) in the current epoch, it
might be > last_peering_reset.  That can lead to last_peering_reset on
the replica having a higher last_peering_reset than the primary's, which
can then lead to future messages, like pg_scan during backfill, being
ignored.

Fixes: http://tracker.ceph.com/issues/24452
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #22462 from liewegas/wip-bluestore-no-meta-paths

os/bluestore: don't store/use path_block.{db,wal} from meta

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #22426 from alfredodeza/wip-rm24020

ceph-volume expand on the LVM API to create multiple LVs at different sizes

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

mgr/dashboard: Swap configuration without changes

Swap the TestBed configuration in suites.

The touched suites were prettified at the end.

Signed-off-by: Stephan Müller <smueller@suse.com>

ceph-volume util make chown links affect links as well as paths

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.activate chown bluestore symlinks

Signed-off-by: Alfredo Deza <adeza@redhat.com>

os/bluestore: make bad symlink target error messages more informative

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #22439 from shaba/devel/fix-rocksdb

cmake: fix find system rockdb

Reviewed-by: Kefu Chai <kchai@redhat.com>

debian: drop '-DUSE_CRYPTOPP=OFF' from cmake options

cryptopp dependency was dropped in a18f0589

Signed-off-by: Kefu Chai <kchai@redhat.com>

test: add test for seastar denc

Signed-off-by: Kefu Chai <kchai@redhat.com>