qa/tasks/ceph: stop any split/merge activity before scrubbing

If there are leftover merges at the end of the run they can take a long
time to get through, blowing our timeout for (waiting for pgs to become
active and to stop splitting/merge) and scrubbing pgs.  Stop all of that
at the end of the run so that we don't have to wait so long.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #25755 into master

* refs/pull/25755/head:
msg/async/Protocol*: bump global_seq when retrying connection
msg/async/Protocol*: unregister con from accept vs mark_down race

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge PR #25840 into master

* refs/pull/25840/head:
qa/msgr: add async-v1only case

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #25792 into master

* refs/pull/25792/head:
common/numa: remove unused _mask variants
osd: support osd_numa_node, osd_numa_auto_affinity, osd_numa_prefer_iface
mon/OSDMonitor: add 'osd numa-status' command
osd: report numa node for network interface(s)
common/pick_address: get numa node info for a interface
os/bluestore: report numa node info for storage
os/bluestore/KernelDevice: report numa info for each device
os/bluestore: allow get_devices() to run without bluestore started
common/blkdev: query numa info
common/numa: cpu_set helpers and numa helpers
common/blkdev: drop debug statements

Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge PR #25841 into master

* refs/pull/25841/head:
mon/MonClient: make mon hunt more aggressive

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #25827 into master

* refs/pull/25827/head:
test/cls_rgw: fix signed/unsigned warning
test/msgr/test_msgr: fix signed/unsigned warning
msg/Connection: protect peer_addrs with safe_item_history<>

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #25704 from tchaikov/wip-36737-unittest-log

test: write log file to current directory

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #25898 into master

* refs/pull/25898/head:
qa/tests: reduced nubmer of runs on 'ovh'

Reviewed-by: Sage Weil <sage@redhat.com>

qa/tests: reduced nubmer of runs on 'ovh'

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge PR #25544 into master

* refs/pull/25544/head:
common/PriorityCache: Automatic chunk sizing

Reviewed-by: Sage Weil <sage@redhat.com>

msg/async/Protocol*: bump global_seq when retrying connection

In the top connect retry in particular we do not bump connect_seq, so it
is critical that we bump global_seq so that the two connection attempts
are distinguishable at the peer.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #25744 from cbodley/wip-test-rgw-reshard-wait

test/rgw: add ifdef for HAVE_BOOST_CONTEXT

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25892 from theanalyst/doc-rgw-usage-fix

doc: rgw: fix the default value of usage log setting

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc: rgw: fix the default value of usage log setting

Fixes: http://tracker.ceph.com/issues/37856
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #25769 from votdev/navbar_focus_color

mgr/dashboard: Fix navbar focused color

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Neha Gupta <gnehapk@gmail.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #25791 from rhcs-dashboard/36740-pool-stats-read-write-columns

mgr/dashboard: add columns to Pools table

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #25839 from tchaikov/wip-37773

mgr/test_orchestrator: correct ceph-volume path

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #25816 from neha-ojha/wip-36686

osd/mon: fix upgrades for pg log hard limit

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #25321 from votdev/fix_dict_changed_size

ceph-volume: Add unit test

Reviewed-by: Alfredo Deza <adeza@redhat.com>

pybind/mgr/dashboard: add modified messages.xlf

This is to fix the make check failure in
mgr-dashboard-frontend-unittests.

Signed-off-by: Neha Ojha <nojha@redhat.com>

common/numa: remove unused _mask variants

These parse the 'local_cpus' file that has a hex bitmask, but I didn't end
up using them.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #25863 from trociny/wip-rbd-mirror-mock-warnings

 test/rbd_mirror: fix gmock warnings

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/test_orchestrator: correct ceph-volume path

s/ceph_volume/ceph-volume/

Fixes: http://tracker.ceph.com/issues/37773
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25860 from trociny/wip-prometheus-rbd_stats_pools

pybind/mgr/prometheus: improve 'rbd_stats_pools' param parsing

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #25300 from votdev/improve_rgw_user_list

mgr/dashboard: Improve RgwUser controller

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #25634 from sebastian-philipp/doc-orchestrator-wal

doc/mgr/orchestrator: add `wal` to blink lights

Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #25813 from rhcs-dashboard/updated-test-full-health

mgr/dashboard: updated health API test

Merge pull request #25743 from trociny/wip-37541

 journal: set max journal order to 26

Merge pull request #25789 from trociny/wip-36038

rbd_mirror: don't report error if image replay canceled

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #25832 from trociny/wip-37729-1

test/pybind/test_rbd: filter out unknown list_children2 keys

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #25861 from tchaikov/wip-seastar-connect-fix

seastar: pickup fix for segfault in POSIX stack

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

test/rbd_mirror: expect service daemon update instance_id attribute

Signed-off-by: Mykola Golub <mgolub@suse.com>

test/rbd_mirror: fix gmock warnings when running with debug

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #25572 from nathan-weinberg/pools-tests

mgr/dashboard: Added breadcrumb and tab tests to Pools menu

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

seastar: pickup fix for segfault in POSIX stack

see
https://github.com/scylladb/seastar/commit/5e399906d1efa9e5ca9d121d09c6d2b1cdc41404

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/prometheus: improve 'rbd_stats_pools' param parsing

re.split can still produce a list with empty string elements (e.g. if
'rbd_stats_pools' is just an empty string) and we want to filter out
those.

Signed-off-by: Mykola Golub <mgolub@suse.com>

mgr/dashboard: add columns to Pools table

* Added columns:
Usage, Read bytes, Write bytes, Read ops, Write ops.

* TableKeyValueComponent:
Bugfix in method: _insertFlattenObjects

* Updated jest global mock:
Now window.getComputedStyle returns an object
that has getPropertyValue defined
(otherwise jest pool tests involving chart.js threw error). See:
https://developer.mozilla.org/en-US/docs/Web/API/Window/getComputedStyle

Fixes: https://tracker.ceph.com/issues/37717
Fixes: https://tracker.ceph.com/issues/34320
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge PR #25621 into master

* refs/pull/25621/head:
mds: allow boot on read-only
mds: setup readonly mode for PurgeQueue
mds: return string_view for type str
mds: add missing locks for PurgeQueue methods
mds: delete on_error context on des

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #24556 into master

* refs/pull/24556/head:
tools/cephfs: make 'cephfs-data-scan scan_links' update snaptable
tools/cephfs: make 'cephfs-data-scan scan_links' update inotable

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #25818 into master

* refs/pull/25818/head:
mon: adjust mon sync, extra_probe_peers to use addrvec

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mon/MonClient: make mon hunt more aggressive

With v2 and v1 addresses, it helps to be a bit more aggressive
searching for mons since the v1 or v2 ports may not be in use.

- try 3 parallel connection attempts, not 2
- increase the timeout interval more slowly

Signed-off-by: Sage Weil <sage@redhat.com>

qa/msgr: add async-v1only case

Signed-off-by: Sage Weil <sage@redhat.com>

mon: adjust mon sync, extra_probe_peers to use addrvec

The peer addr stuff via asok is a bit fragile because the user must
provide an exact addrvec matching the mon to avoid some weirdness, but
it's rarely used, and the fix would be some robustness/tolerance in the
messenger that is a bigger project than this.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #25823 into master

* refs/pull/25823/head:
msg/async/Protocol*: fix mark_down vs accept race

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

osd: support osd_numa_node, osd_numa_auto_affinity, osd_numa_prefer_iface

- osd_numa_node manually specifies a numa node.
- osd_numa_prefer_iface makes us prefer IPs in public_network that are
  on the same numa node as the storage
- osd_numa_auto_affinity will set affinity to a numa node when both the
  store and network(s) are on the same numa node.

These options are all flagged as 'startup', although osd_numa_node and
osd_numa_auto_affinity takes effect when the OSD is marked up, so doing
'ceph osd down ...' is sufficient to induce an update.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: add 'osd numa-status' command

Signed-off-by: Sage Weil <sage@redhat.com>

osd: report numa node for network interface(s)

Signed-off-by: Sage Weil <sage@redhat.com>

common/pick_address: get numa node info for a interface

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #25381 from cbodley/wip-qa-rgw-cls

qa/rgw: add cls_lock/log/refcount/version tests to verify suite

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #24027 from theanalyst/wip-civetweb-poll

rgw: civetweb: use poll instead of select while waiting on sockets

Merge PR #25835 into master

* refs/pull/25835/head:
mgr/ServiceMap: print daemon addr in legacy format

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge PR #24172 into master

* refs/pull/24172/head:
client: fix fuse client hang because its pipe to mds is not ok

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/dashboard: Added breadcrumb and tab tests to Pools menu

Fixes: https://tracker.ceph.com/issues/37667
Signed-off-by: Nathan Weinberg <nweinber@redhat.com>

mgr/ServiceMap: print daemon addr in legacy format

The v1/v2 prefix isn't useful anyway since the client can connect via
any protocol.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #25741 into master

* refs/pull/25741/head:
mds: remove wrong assertion in Locker::snapflush_nudge

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

test/cls_rgw: fix signed/unsigned warning

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #25479 into master

* refs/pull/25479/head:
client: fix fuse client can't read or write data due its caps is invalid
Revert "client: fix fuse client can't read or write data due its caps is invalid"

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

test/msgr/test_msgr: fix signed/unsigned warning

Signed-off-by: Sage Weil <sage@redhat.com>

msg/Connection: protect peer_addrs with safe_item_history<>

The peer_addrs can be updated during the initial connection handshake,
but we don't want users (e.g., dout()) to race with an update and
wander off into bad memory.

We use the same strategy for Messenger's my_addrs.

Fixes: http://tracker.ceph.com/issues/37807
Signed-off-by: Sage Weil <sage@redhat.com>

test/pybind/test_rbd: filter out unknown list_children2 keys

Fixes: http://tracker.ceph.com/issues/37729
Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #25812 from Exotelis/toc

mgr/dashboard: Add table of contents to HACKING.rst

Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #25830 from rhcs-dashboard/grafana-500

mgr/dashboard: Fixes Grafana 500 error

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge PR #25801 into master

* refs/pull/25801/head:
qa/tasks/radosbench: default to 64k writes

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24714 from pritha-srivastava/wip-rgw-role-doc

rgw: Adding documentation for Roles.

mgr/dashboard: Improve RgwUser controller

Use the new 'user?list' RGW Admin OPS API endpoint.

Signed-off-by: Volker Theile <vtheile@suse.com>

Merge PR #25821 into master

* refs/pull/25821/head:
doc/dev/msgr2: checksum frame header (tag + length)

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

msg/async/Protocol*: fix mark_down vs accept race

If we mark_down the connection while we have the lock dropped the state
maybe CLOSED when we retake it.  Bail out if that happens instead of
continuing to process the connection and eventually running into an
assertion on can_write.

Fixes: http://tracker.ceph.com/issues/36497
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #25820 from alfredodeza/wip-rm37805

ceph-volume tests/functional declare ceph-ansible roles instead of importing them

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

mgr/dashboard: Fix navbar focused color

The previous gray does not match the new branding color. Now use the green from https://ceph.com.

Signed-off-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Fixes Grafana 500 error

Fixes: http://tracker.ceph.com/issues/37809
Signed-off-by: Kanika Murarka <kmurarka@redhat.com>

client: fix fuse client can't read or write data due its caps is invalid

We can't do read or write if we don't have Fr or Fw. Similarly, if the
caps is invalid, we can't read or wirte either.

What's worse, the mds don't konw our cap gen is invalid and think client
inode's caps is right. Thus will lead client application hung all the time.

Fixes: http://tracker.ceph.com/issues/36189
Signed-off-by: Guan yunfei <yunfei.guan@xtaotech.com>
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Revert "client: fix fuse client can't read or write data due its caps is invalid"

This reverts commit 1d742aa51be07cebf1b0d19b6640684b263e6bfc.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mon/OSDMonitor.cc: make a note about reusing jewel feature bit

For OSD_PGLOG_HARDLIMIT, we have reused a jewel feature bit that was retired
in luminous. Therefore, we need to check the release version for
>= CEPH_RELEASE_LUMINOUS, before using it.

Signed-off-by: Neha Ojha <nojha@redhat.com>

doc/dev/msgr2: checksum frame header (tag + length)

Ensure we catch transport errors early (before we, say, try to allocate
a missized payload buffer).

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-volume tests/functional declare ceph-ansible roles instead of importing them

Signed-off-by: Alfredo Deza <adeza@redhat.com>

qa/suites/upgrade/*-x/stress-split: set pglog_hardlimit flag

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/suites/upgrade/luminous-x: add pg log settings

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/tests: added pg log settings to mimic-x

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge PR #25795 into master

* refs/pull/25795/head:
ceph-objectstore-tool: fix split check on import

Reviewed-by: Neha Ojha <nojha@redhat.com>

qa/tasks/radosbench: default to 64k writes

The default rados bench is 4MB writes, but this tends to fill up our
tiny test clusters.  Use 64KB instead.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: updated health API test

Updated test_full_health to reflect changes in
mgr_map / mon_status payloads.

Signed-off-by: Alfonso Martínez <almartin@redhat.com>

mgr/dashboard: Add table of contents

Adds a table of contents to the HACKING.rst to make the navigation easier

Signed-off-by: Sebastian Krah <skrah@suse.com>

tools/cephfs: make 'cephfs-data-scan scan_links' update snaptable

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

tools/cephfs: make 'cephfs-data-scan scan_links' update inotable

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #25646 from tone-zhang/nvme-hang

bluestore/NVMEDevice.cc: fix NVMEManager thread hang

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #25797 from falcon78921/wip-doc-37793

doc: updated Ceph documentation links

Reviewed-by: Kefu Chai <kchai@redhat.com>

bluestore/NVMEDevice.cc: fix NVMEManager thread halt

When enable SPDK in Ceph and start up Ceph development cluster, met
NVMEManager thread halt.

On aarch64 platform, the log as below:

Starting SPDK v18.04.1 / DPDK 18.05.0 initialization...
[ DPDK EAL parameters: nvme-device-manager -c 0x1 -m 2048 --file-prefix=spdk_pid16987 ]
EAL: Detected 46 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/spdk_pid16987/mp_socket
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:01:00.0 on NUMA socket 0
EAL:   probe driver: 8086:953 spdk_nvme
EAL:   using IOMMU type 1 (Type 1)
^C

The reason is that pthread_cond_destroy() cannot destroy the active
condition_variable parameter.

Also on x86 debug builds we get the following error messages due to
probe_queue_lock still being active during ~NVMEManager:

/home/ubuntu/ceph/src/common/mutex_debug.h: 114: FAILED ceph_assert(r == 0)
 ceph version 14.0.1-1862-g403622b (403622be721a460f3dff2d84f6bfc628f5026704) nautilus (dev)

The change fixes the issue.

Fixes: http://tracker.ceph.com/issues/37720
Signed-off-by: tone.zhang <tone.zhang@arm.com>
Signed-off-by: Steve Capper <steve.capper@arm.com>

Merge PR #25799 into master

* refs/pull/25799/head:
msg: fix includes for ostringstream

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #25802 into master

* refs/pull/25802/head:
qa/suites/rados/thrash-old-clients: no async-v2only

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/suites/rados/thrash-old-clients: no async-v2only

Old clients don't support the v2 protocol.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #25780 into master

* refs/pull/25780/head:
mon/OSDMonitor: do not populate void pg_temp into nextmap

Reviewed-by: Sage Weil <sage@redhat.com>

msg: fix includes for ostringstream

Clang complains...

```
/home/jenkins/workspace/ceph-master/src/msg/msg_types.h:420:19: error: implicit instantiation of undefined template 'std::__1::basic_ostringstream<char, std::__1::char_traits<char>, std::__1::allocator<char> >'
    ostringstream ss;
                  ^
/usr/include/c++/v1/iosfwd:123:32: note: template is declared here
    class _LIBCPP_TEMPLATE_VIS basic_ostringstream;
                               ^
1 error generated.
```

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #25716 from cyx1231st/wip-crimson-msgr-errorleak

crimson/net: fix crimson msgr error leaks to caller

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #25750 into master

* refs/pull/25750/head:
common: expose internal StackStringStream via deref

Reviewed-by: Sage Weil <sage@redhat.com>

pybind/mgr/dashboard: add pglog_hardlimit flags

Note: Tested using vstart

Signed-off-by: Neha Ojha <nojha@redhat.com>

mon: add and use OSD_PGLOG_HARDLIMIT feature bit

Signed-off-by: Neha Ojha <nojha@redhat.com>

mon: set CEPH_OSDMAP_PGLOG_HARDLIMIT for new clusters

Signed-off-by: Neha Ojha <nojha@redhat.com>

osd/mon: fix upgrades for pg log hard limit

Signed-off-by: Neha Ojha <nojha@redhat.com>

osd: bring back old calc_trim_to and rename new method

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge PR #25793 into master

* refs/pull/25793/head:
Merge remote-tracking branch 'private/wip-mon-kv-fix' into wip-mimic-4
Merge remote-tracking branch 'private/wip-rgw-max-keys' into wip-mimic-4
doc/releases/mimic: v13.2.4
rgw: fix issues with 'enforce bounds' patch
rgw: enforce bounds on max-keys/max-uploads/max-parts
mon/config-key: limit caps allowed to access the store

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: David Galloway <dgallowa@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>