common: fix for broken rbdmap parameter parsing

This improves the broken parameter parsing discussed in #36327

Test expression for the parsing function:

PARAMS="options='lock_on_read,queue_depth=1024',id=archiv,keyring=/etc/ceph/ceph.client.archiv.keyring"
echo -e " ORIGINAL PARAM: >>>$PARAMS<<<\nCONVERTED PARAM: >>>$(create_cmd_params "$PARAMS")<<<"

PARAMS="id=archiv,keyring=/etc/ceph/ceph.client.archiv.keyring,options='lock_on_read,queue_depth=1024'"
echo -e " ORIGINAL PARAM: >>>$PARAMS<<<\nCONVERTED PARAM: >>>$(create_cmd_params "$PARAMS")<<<"

PARAMS=""
echo -e " ORIGINAL PARAM: >>>$PARAMS<<<\nCONVERTED PARAM: >>>$(create_cmd_params "$PARAMS")<<<"

PARAMS=",keyring=/etc/ceph/ceph.client.archiv.keyring,options='lock_on_read,queue_depth=1024',id=archiv,"
echo -e " ORIGINAL PARAM: >>>$PARAMS<<<\nCONVERTED PARAM: >>>$(create_cmd_params "$PARAMS")<<<"

PARAMS="'keyring'=/etc/ceph/ceph.client.archiv.keyring,options='lock_on_read,queue_depth=1024',id=archiv,"
echo -e " ORIGINAL PARAM: >>>$PARAMS<<<\nCONVERTED PARAM: >>>$(create_cmd_params "$PARAMS")<<<"

PARAMS="--keyring=/etc/ceph/ceph.client.archiv.keyring,options='lock_on_read,queue_depth=1024',id=archiv, # a comment"
echo -e " ORIGINAL PARAM: >>>$PARAMS<<<\nCONVERTED PARAM: >>>$(create_cmd_params "$PARAMS")<<<"

Fixes: https://tracker.ceph.com/issues/36327
Signed-off-by: Marc Schoechlin <ms@256bit.org]>

Merge pull request #25935 from trociny/wip-37883

test/rbd_mirror: wait for release leader lock fully complete

Merge pull request #25925 from markhpc/wip-rocksdb-pricache

kv/KeyValueDB: Move PriCache implementation to ShardedCache.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #25876 from s0nea/wip-dashboard-rename-config-option-types

mgr/dashboard: config option type names update

Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #25943 from rhcs-dashboard/update-sso-doc

mgr/dashboard: updated SSO documentation

Reviewed-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: updated SSO documentation

Added info about SAML request issuer value.

Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #25927 from dillaman/wip-37867

mgr/dashboard: incorporate RBD overall performance grafana dashboard

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Kanika Murarka <kmurarka@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #25492 from sebastian-philipp/orchestrator-common-metaclass

pybind/mgr: Unified bits of volumes and orchestrator

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Tim Serong <tserong@suse.com>

Merge pull request #25909 from votdev/issue_37862

mgr/dashboard: Confusing tilted time stamps in the CephFS performance graph

Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #23729 from daikk115/master

docs: Fixed swift client authentication fail

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #25913 from s0nea/wip-dashboard-test_full_health-fix

mgr/dashboard: Fix test_full_health test

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #25905 from s0nea/wip-dashboard-osd-add-flag-fix

mgr/dashboard: Provide all four 'mandatory' OSD flags

Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #25859 from xiexingguo/wip-async-no-force-renew-auth

msg/async: do not force updating rotating keys inline

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #25930 from xiexingguo/wip-invalid-upmap-access

osd/OSDMap: potential access violation fix

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #25933 into master

* refs/pull/25933/head:
common/numa: remove stray debug output

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd/OSDMap: potential access violation fix

Seems we'll continue to access the iterator after it is invalidated
by the __erase__ method.
Also this is more efficient considering there could be some extreme
ec-pool (e.g., 8 + 2) consumers..

Fixes: http://tracker.ceph.com/issues/37881
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #25931 from yangdongsheng/journal_import

rbd: fix a typo in error output

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

common/numa: remove stray debug output

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #25902 into master

* refs/pull/25902/head:
qa/standalone/mon/mon-handle-forward: fix grep path and check return results
mon/MonClient: be less aggressive/paranoid about reopening mon connections

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #25884 into master

* refs/pull/25884/head:
msg/async: better debug output

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge PR #25917 into master

* refs/pull/25917/head:
qa/suites/rados/multimon/tasks/mon_recovery: whitelist PG_AVAILABILITY

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

test/rbd_mirror: wait for release leader lock fully complete

Previously the wait context was completed at post_release_lock
stage, which still was not the end of the lock releasing process.

Fixes: https://tracker.ceph.com/issues/37883
Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge PR #25273 into master

* refs/pull/25273/head:
doc/rados/operations/health-checks: Add LARGE_OMAP_OBJECTS

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #25828 into master

* refs/pull/25828/head:
osd, mgr, mds: make timeout of updating rotating keys configurable

Reviewed-by: yanjun <yan.jun8@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #25886 into master

* refs/pull/25886/head:
qa/tasks/rebuild_mondb: use monmap to properly name the mons
ceph-monstore-tool: add --monmap <path> argument to rebuild command
ceph-mon: dump monmap to debug logs
Revert "mon/MonMap: no noname- mon name prefix when for_mkfs"
mon/MonMap: improve/fix initial monmap generation for mkfs
ceph-mon: make membership check with public_addr more robust
ceph-mon: make mkfs check whether I am in monmap more robust

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #25806 into master

* refs/pull/25806/head:
mgr: diskprediction_cloud Add nvme devcie support

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #25662 from yangdongsheng/abort_on_full

rbd: introduce abort_on_full option for rbd map

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

rbd: introduce abort_on_full option for rbd map

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: fix a typo in error output

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

doc/rados/operations/health-checks: Add LARGE_OMAP_OBJECTS

Document LARGE_OMAP_OBJECTS health check

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #25924 from neha-ojha/wip-fix-scrub-doc

doc/rados/configuration: add osd scrub {begin,end} week day

Reviewed-by: Sage Weil <sage@redhat.com>

qa/tasks/rebuild_mondb: use monmap to properly name the mons

We used to rely on the monmap bootstrap code to magically create a valid
monmap with named mons because our old-style ceph.conf had mon_addr
values in each mon.foo section.  Instead, just feed it a real monmap
from pre-destruction.

In practice, a user can manually generate this monmap, or rename the
mons after the fact with --inject-monmap, or whatever.  Out of scope
for this test, so we just do the simplest thing to make the rebuild test
work.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-monstore-tool: add --monmap <path> argument to rebuild command

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-mon: dump monmap to debug logs

Verbosely during mkfs, at level 10 on normal startup.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: incorporate RBD overall performance grafana dashboard

Fixes: http://tracker.ceph.com/issues/37867
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

monitoring/grafana: new RBD overview dashboard page

This page pulls RBD stats from the Natuatilus prometheus exporter.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

doc/rados/configuration: add osd scrub {begin,end} week day

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #24427 from thmour/thmour_api_purge

librbd: add trash purge api calls

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #25908 from trociny/wip-dperf-bytes

mgr: change 'bytes' dynamic perf counters to COUNTER type

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

kv/KeyValueDB: Move PriCache implementation to ShardedCache.

Signed-off-by: Mark Nelson <mnelson@redhat.com>

msg/async: better debug output

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #25887 from neha-ojha/wip-hide-flag

include/rados.h: hide CEPH_OSDMAP_PGLOG_HARDLIMIT from ceph -s

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Xie Xingguo <xie.xingguo@zte.com.cn>

qa/suites/rados/multimon/tasks/mon_recovery: whitelist PG_AVAILABILITY

The mgr creates a pool for device health, and mons may be thrashing and
make peering slow.

Signed-off-by: Sage Weil <sage@redhat.com>

librbd: tweaked trash purge API

Removed the internal usage of the public librbd API, avoided double-opening
image to determine data pool, and fixed issues with the unit test case.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: Fix test_full_health test

Fixes: http://tracker.ceph.com/issues/37872
Signed-off-by: Tatjana Dehler <tdehler@suse.com>

cls/rbd: reduce erroneous error messages when omap keys don't exist

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test/librbd: fix librbrd test relative to trash purge

Signed-off-by: Julien Collet <julien.collet@cern.ch>

librbd: add trash purge api calls

Add trash purge api calls and fix the docs of some other rbd trash functions

Signed-off-by: Theofilos Mouratidis <mtheofilos@gmail.com>

Merge pull request #25529 from Songweibin/wip-fix-rbd-migration

librbd: disallow trash restoring when image being migrated

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #25765 from Songweibin/wip-fix-migrate-parent

librbd/api: misc fix migration

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #25834 from trociny/wip-21392

rbd-mirror: create and export replication perf counters to mgr

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #25896 from trociny/wip-37861

rbd: fix delay time calculation for trash move

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #25656 from tspmelo/wip-npm-update

mgr/dashboard: Update npm packages

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #25883 from dillaman/wip-37745

librbd: keep access/modified timestamp updates out of IO path

Reviewed-by: Mykola Golub <mgolub@suse.com>

mgr/dashboard: Provide all four 'mandatory' flags

Fixes: https://tracker.ceph.com/issues/37857
Signed-off-by: Tatjana Dehler <tdehler@suse.com>

mgr/dashboard: Confusing tilted time stamps in the CephFS performance graph

Fixes: https://tracker.ceph.com/issues/37862
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #25894 from Devp00l/wip-kv-table-update

mgr/dashboard: Hide empty fields and render all objects in KV-table

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr: change 'bytes' dynamic perf counters to COUNTER type

(so they are of the same type as in static perf)

Signed-off-by: Mykola Golub <mgolub@suse.com>

mgr/dashboard: Render all objects in KV-table

The problem was that the key value table didn't render all objects in
the given object, even if 'renderObjects' was set to true.

For example you give it an object with 2 attributes each of them holds a
non empty object, this will result in a table data that is not fully
converted to key value objects, which will result as text '[Object
object]' in the table view.

Now every object will be rendered and the key value array will be sorted
by key.

Fixes: https://tracker.ceph.com/issues/37859
Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Hide empty fields in KV-table

It exends the key value table by the option to hide empty fields.

Fixes: https://tracker.ceph.com/issues/37860
Signed-off-by: Stephan Müller <smueller@suse.com>

Revert "mon/MonMap: no noname- mon name prefix when for_mkfs"

This reverts commit c078c810313ece647db6d9056c0eca5256a91a34.

# Conflicts:
# src/mon/MonMap.cc

The ceph_mon.cc bootstrap code expects a noname- prefix to identify which
unamed mons (i.e., generated from mon_host) in the initial monmap can be
taken over as itself.

At the same time, the mon recovery code needs a way to know how to name
the monitors.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonMap: improve/fix initial monmap generation for mkfs

The monmap generation from mon_host for mon mkfs was pretty much completely
broken.  Restructure the code.

Generally, prefer v2+v1 addrvecs, unless a specific addr type and/or
port is specified.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-mon: make membership check with public_addr more robust

If the public_addr has no port, we will check v1 and v2 addresses.
MonMap::contains() checks for *any* overlap/match, so we should find
ourselves.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-mon: make mkfs check whether I am in monmap more robust

If we have a local address, check for both v1 and v2 variants in the
monmap.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #25755 into master

* refs/pull/25755/head:
msg/async/Protocol*: bump global_seq when retrying connection
msg/async/Protocol*: unregister con from accept vs mark_down race

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge PR #25840 into master

* refs/pull/25840/head:
qa/msgr: add async-v1only case

Reviewed-by: Neha Ojha <nojha@redhat.com>

qa/standalone/mon/mon-handle-forward: fix grep path and check return results

This makes the test more strict and less confusing.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonClient: be less aggressive/paranoid about reopening mon connections

Previously we were reopening any time the name of our mon changed.  That
was pretty much every time, since we started out with names from the
seed monmap like 'noname-[abc]' that never match the real mons.

Instead, only reopen if the addr we are connecting to is no longer in the
monmap.  We don't actually care about the mon's name.

Fixes: http://tracker.ceph.com/issues/37868
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #25792 into master

* refs/pull/25792/head:
common/numa: remove unused _mask variants
osd: support osd_numa_node, osd_numa_auto_affinity, osd_numa_prefer_iface
mon/OSDMonitor: add 'osd numa-status' command
osd: report numa node for network interface(s)
common/pick_address: get numa node info for a interface
os/bluestore: report numa node info for storage
os/bluestore/KernelDevice: report numa info for each device
os/bluestore: allow get_devices() to run without bluestore started
common/blkdev: query numa info
common/numa: cpu_set helpers and numa helpers
common/blkdev: drop debug statements

Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge PR #25841 into master

* refs/pull/25841/head:
mon/MonClient: make mon hunt more aggressive

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #25827 into master

* refs/pull/25827/head:
test/cls_rgw: fix signed/unsigned warning
test/msgr/test_msgr: fix signed/unsigned warning
msg/Connection: protect peer_addrs with safe_item_history<>

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #25704 from tchaikov/wip-36737-unittest-log

test: write log file to current directory

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #25898 into master

* refs/pull/25898/head:
qa/tests: reduced nubmer of runs on 'ovh'

Reviewed-by: Sage Weil <sage@redhat.com>

qa/tests: reduced nubmer of runs on 'ovh'

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge PR #25544 into master

* refs/pull/25544/head:
common/PriorityCache: Automatic chunk sizing

Reviewed-by: Sage Weil <sage@redhat.com>

msg/async/Protocol*: bump global_seq when retrying connection

In the top connect retry in particular we do not bump connect_seq, so it
is critical that we bump global_seq so that the two connection attempts
are distinguishable at the peer.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #25744 from cbodley/wip-test-rgw-reshard-wait

test/rgw: add ifdef for HAVE_BOOST_CONTEXT

Reviewed-by: Kefu Chai <kchai@redhat.com>

rbd: fix delay time calculation for trash move

Fixes: https://tracker.ceph.com/issues/37861
Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: keep access/modified timestamp updates out of IO path

If the cache is enabled, it was possible for the timestamp updates
to cause race conditions with librbd clients (e.g. rbd export)
that had data corrupting effects since callbacks were expected to
be serialized.

Fixes: http://tracker.ceph.com/issues/37745
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #25892 from theanalyst/doc-rgw-usage-fix

doc: rgw: fix the default value of usage log setting

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rbd-mirror: add aggregated replication perf counters

Signed-off-by: Mykola Golub <mgolub@suse.com>

pybind/mgr/prometheus: export rbd-mirror metadata

Signed-off-by: Mykola Golub <mgolub@suse.com>

pybind/mgr: add rbd-mirror to get_all_perf_counters by default

This effectively enables rbd-mirror stats for modules like prometheus or
influx.

Signed-off-by: Mykola Golub <mgolub@suse.com>

pybind/mgr/diskprediction_cloud: only perfs for mon and osd needed

Signed-off-by: Mykola Golub <mgolub@suse.com>

pybind/mgr: allow to specify services for get_all_perf_counters

Signed-off-by: Mykola Golub <mgolub@suse.com>

rbd-mirror: create and export replication perf counters to mgr

Fixes: https://tracker.ceph.com/issues/21392
Signed-off-by: Mykola Golub <mgolub@suse.com>

doc: rgw: fix the default value of usage log setting

Fixes: http://tracker.ceph.com/issues/37856
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #25769 from votdev/navbar_focus_color

mgr/dashboard: Fix navbar focused color

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Neha Gupta <gnehapk@gmail.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #25791 from rhcs-dashboard/36740-pool-stats-read-write-columns

mgr/dashboard: add columns to Pools table

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #25839 from tchaikov/wip-37773

mgr/test_orchestrator: correct ceph-volume path

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: use new config options type names

Use the new config option type names (given by the cluster) in the
dashboard.

Fixes: http://tracker.ceph.com/issues/37843
Signed-off-by: Tatjana Dehler <tdehler@suse.com>

include/rados.h: hide CEPH_OSDMAP_PGLOG_HARDLIMIT from ceph -s

Signed-off-by: Neha Ojha <nojha@redhat.com>

mgr/dashboard: Update bootstrap to 3.4.0

Fixes: https://tracker.ceph.com/issues/37834
Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Update npm packages

Signed-off-by: Tiago Melo <tmelo@suse.com>

test/journal: ReplayEntry::get_data is const

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #25816 from neha-ojha/wip-36686

osd/mon: fix upgrades for pg log hard limit

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #25321 from votdev/fix_dict_changed_size

ceph-volume: Add unit test

Reviewed-by: Alfredo Deza <adeza@redhat.com>

pybind/mgr/dashboard: add modified messages.xlf

This is to fix the make check failure in
mgr-dashboard-frontend-unittests.

Signed-off-by: Neha Ojha <nojha@redhat.com>

common/numa: remove unused _mask variants

These parse the 'local_cpus' file that has a hex bitmask, but I didn't end
up using them.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #25863 from trociny/wip-rbd-mirror-mock-warnings

 test/rbd_mirror: fix gmock warnings

Reviewed-by: Jason Dillaman <dillaman@redhat.com>