Merge pull request #22617 from tianshan/fix_http_err_return

rgw: set cr state if aio_read err return in RGWCloneMetaLogCoroutine

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #22793 from kshtsk/wip-add-opensuse-leap

Add new OpenSUSE Leap id for install-deps.sh

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #22829 from tspmelo/fix-24757

mgr/dashboard: Fix regression on rbd form component

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #22762 from ricardoasmarques/wip-toolbar-text

mgr/dashboard: Remove top-right actions text and add "About" page

Reviewed-by: Kanika Murarka <murarkakanika@gmail.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #20133 from smithfarm/wip-redundant-recommends

build/ops: deb: drop redundant ceph-common recommends

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22847 from ErwanAliasr1/evelu-ccache2

run-make-check.sh: Adding ccache tuning for the CI

Reviewed-by: Sébastien Han <seb@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #22703 into master

* refs/pull/22703/head:
doc/mgr: add doc for crash mgr module
qa/suites/rados/mgr: Add test_crash
qa/tasks/mgr: add test_crash, call from test_module_selftest
qa/tasks/{ceph_manager.py,vstart_runner.py}: allow kwargs in raw_*
mgr/pybind/crash: handle crashdumps

Reviewed-by: John Spray <john.spray@redhat.com>

Merge PR #22704 into master

* refs/pull/22704/head:
osd/PG: do not send notify to empty peer
osd/PG: do not assume delete event means found_missing

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #22733 into master

* refs/pull/22733/head:
os/bluestore: set correctly shard for existed Collection.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #22738 into master

* refs/pull/22738/head:
mgr/status: add exabytes unit

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #22534 into master

* refs/pull/22534/head:
msg/Messenger: use safe_item_history<> for my_addrs
msg/simple: fix set_myaddrs()
msg/Messenger: use mutable_item_history<> for my_addrs
common/item_history: container to support lockless reads, infrequent updates
msg/simple/Accepter: fix my_addr == my_addrs.v[0] constraint
msg/async: track target_addr for each connection
msg/async: mark accepted connections with addr type (legacy or msgr2)
msg/async: mark AsyncConnection with msgr2 flag
msg/async: track connections by addrvec
mon/Session: inst -> name and addrs
osd/OSDMap: don't print hb addrs
msg/DispatchQueue: myaddr -> myaddrs
mgr: myaddr -> myaddrs
msg: make set_addr_unknowns take an addrvec
mon/LogMonitor: myaddr -> myaddrs
librados: myaddr -> myaddrs
common/LogClient: myaddr -> myaddrs
client: myaddr -> myaddrs
osd/OSDMap: is_blacklisted() for addrvecs
osd: populate metadata with all addrs
mds: addr -> addrvec
mon/MonClient: get_myaddrs()
mon/OSDMonitor: addrvec blacklist helper
mds: use new pick_addresses
ceph-osd: use new bindv()
msg/async: bind to multiple addresses
msg/async: (legacy) handshake using legacy addr
msg/async: fix some debug prefixes
msg/async: multiple listening ServerSockets
osd/OSDMap: make cluster addrs addrvecs too
msg/Messenger: my_addr -> my_addrs
msg/Connection: peer_addr -> peer_addrs
msg/msg_types: hash<> for entity_addrvec_t
mgr/DaemonServer: use new bindv() and pick_addresses()
msg/Messenger: new bindv() that takes an addrvec
common/pick_address: fill in ipv4/6 and msgr1/msgr2 via config options
common/pick_addresses: new addrvec-based pick_addresses()
common/pick_addresses: add filtering by ipv4 and ipv6
CEPH_MON_PORT -> CEPH_MON_PORT_LEGACY; define CEPH_MON_PORT_IANA
osd: kill osd_heartbeat_addr option
common/options: add addrvec option type

Reviewed-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: Fix regression on rbd form component

'decodeURIComponent' method was decoding an undefined variable to the string
'undefined', which would later cause problems.

This regression was introduced in057d6025894800c1b4a422b081fe02ab82b044fe.

Fixes: http://tracker.ceph.com/issues/24757
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #22852 from ceph/jcsp-cephfs-exp-doc

doc/cephfs: remove lingering "experimental" note about multimds

Reviewed-by: Jos Collin <jcollin@redhat.com>

doc/cephfs: remove lingering "experimental" note about multimds

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #22830 from ricardoasmarques/fix-24756

mgr/dashboard: Fix RBD object size dropdown options

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

run-make-check.sh: Adding ccache tuning for the CI

When run-make-check is run by the CI, some tuning can be performed to
speedup the build.

This commit :
- Detect jenkins by searching JENKINS_HOME env variable
- Defines the SOURCE_DATE_EPOCH to enforce a stable date across builds
- Ask cmake not to use the git versioning which adds useless entropy for a temporary build (ENABLE_GIT_VERSION=OFF)
- Define the ccache slopiness to increase efficiency
- Increase the ccache size to save multiple builds to maximise cache hit between PRs
- Print ccache statistics to evaluate ccache efficiency

Signed-off-by: Erwan Velu <erwan@redhat.com>

Merge pull request #22644 from Devp00l/wip-forms-helper-service

 mgr/dashboard: CdFormGroup

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Fix RBD object size dropdown options

Fixes: https://tracker.ceph.com/issues/24756
Signed-off-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #22461 from tspmelo/wip-osd-flags-ui

mgr/dashboard: Add UI for Cluster-wide OSD Flags configuration

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Add "About" modal

Fixes: https://tracker.ceph.com/issues/24646
Signed-off-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: Usage of 'BehaviorSubject' in summary

Signed-off-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: Remove text from top-right actions

Fixes: https://tracker.ceph.com/issues/24624
Signed-off-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #22757 from votdev/improve_str_to_bool

mgr/dashboard: Improve str_to_bool

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

os/bluestore: set correctly shard for existed Collection.

For existed Collection, the constructor of Collection will be called in _open_collections.
But m_finisher_num can't setup when enable bluestore_shard_finishers.

So move m_finisher_num setup before _open_collections && _kv_start.

Fixes: http://tracker.ceph.com/issues/24761
Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #22837 from dzafman/wip-fix-timeout

test: test_get_timeout_delays() fix

Reviewed-by: Erwan Velu <erwan@redhat.com>

test: test_get_timeout_delays() fix

Caused by: 7b0d1c8b8acff2a7010bfb0400df09786033ac63

Signed-off-by: David Zafman <dzafman@redhat.com>

msg/Messenger: use safe_item_history<> for my_addrs

Mutating the item in place is not safe.

Signed-off-by: Sage Weil <sage@redhat.com>

msg/simple: fix set_myaddrs()

Signed-off-by: Sage Weil <sage@redhat.com>

msg/Messenger: use mutable_item_history<> for my_addrs

We read this without a lock.

Signed-off-by: Sage Weil <sage@redhat.com>

common/item_history: container to support lockless reads, infrequent updates

Signed-off-by: Sage Weil <sage@redhat.com>

msg/simple/Accepter: fix my_addr == my_addrs.v[0] constraint

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async: track target_addr for each connection

The target_addr is item from peer_addrs that we are using for the
connection.  On accept, we learn it during the handshake.  On connect,
we choose it early on (first item in addrvec that we understand).

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async: mark accepted connections with addr type (legacy or msgr2)

We have to note the addr type we are listening for in the ServerSocketImpl
and pass that on to any AsyncConnections that result.

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async: mark AsyncConnection with msgr2 flag

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async: track connections by addrvec

Signed-off-by: Sage Weil <sage@redhat.com>

mon/Session: inst -> name and addrs

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMap: don't print hb addrs

Nobody cares!

Signed-off-by: Sage Weil <sage@redhat.com>

msg/DispatchQueue: myaddr -> myaddrs

Signed-off-by: Sage Weil <sage@redhat.com>

mgr: myaddr -> myaddrs

Signed-off-by: Sage Weil <sage@redhat.com>

msg: make set_addr_unknowns take an addrvec

Nicely collapse the wonky logic in OSD into the helper!

Signed-off-by: Sage Weil <sage@redhat.com>

mon/LogMonitor: myaddr -> myaddrs

Signed-off-by: Sage Weil <sage@redhat.com>

librados: myaddr -> myaddrs

Signed-off-by: Sage Weil <sage@redhat.com>

common/LogClient: myaddr -> myaddrs

Signed-off-by: Sage Weil <sage@redhat.com>

client: myaddr -> myaddrs

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMap: is_blacklisted() for addrvecs

Signed-off-by: Sage Weil <sage@redhat.com>

osd: populate metadata with all addrs

Signed-off-by: Sage Weil <sage@redhat.com>

mds: addr -> addrvec

Update MDSMap to use entity_addrvec_t instead of entity_addr_t.  Adjust
the various mon and mgr pieces that touch it accordingly.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonClient: get_myaddrs()

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: addrvec blacklist helper

Signed-off-by: Sage Weil <sage@redhat.com>

mds: use new pick_addresses

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-osd: use new bindv()

Rely on the new and improved pick_addresses() to produce an addrvec of
addresses we should bind to.

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async: bind to multiple addresses

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async: (legacy) handshake using legacy addr

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async: fix some debug prefixes

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async: multiple listening ServerSockets

No functional change yet.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMap: make cluster addrs addrvecs too

This will allow a single-step upgrade from msgr1 to msgr2: as each
nautilus OSD starts up, it will bind to both msgr1 and msgr2 addrs
(assuming it starts up after the mons have been upgraded).  Peer OSDs will
use the msgr2 address if they can, or the legacy one if they can't.
Once require_osd_release >= nautilus OSDs will stop binding to msgr1
ports at all.

Signed-off-by: Sage Weil <sage@redhat.com>

msg/Messenger: my_addr -> my_addrs

Minimal changes to SimpleMessenger and AsyncMessenger to keep things
working, assuming we ony have a single addr in the addrvec.

Signed-off-by: Sage Weil <sage@redhat.com>

msg/Connection: peer_addr -> peer_addrs

Signed-off-by: Sage Weil <sage@redhat.com>

msg/msg_types: hash<> for entity_addrvec_t

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/DaemonServer: use new bindv() and pick_addresses()

Signed-off-by: Sage Weil <sage@redhat.com>

msg/Messenger: new bindv() that takes an addrvec

Signed-off-by: Sage Weil <sage@redhat.com>

common/pick_address: fill in ipv4/6 and msgr1/msgr2 via config options

This allows users to simply pass in CLUSTER or PUBLIC and the function
fills in the rest.  (Or, if the caller is particular, they can pass in
the other flags too.)

Signed-off-by: Sage Weil <sage@redhat.com>

common/pick_addresses: new addrvec-based pick_addresses()

Flags for everything we would want.  The intent is to return an addrvec
of addrs that we can directly bind to.

Signed-off-by: Sage Weil <sage@redhat.com>

common/pick_addresses: add filtering by ipv4 and ipv6

Signed-off-by: Sage Weil <sage@redhat.com>

CEPH_MON_PORT -> CEPH_MON_PORT_LEGACY; define CEPH_MON_PORT_IANA

Signed-off-by: Sage Weil <sage@redhat.com>

osd: kill osd_heartbeat_addr option

This serves no (good) purpose: we should always use the public addr/
network.

Signed-off-by: Sage Weil <sage@redhat.com>

common/options: add addrvec option type

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #22165 into master

* refs/pull/22165/head:
qa: add one-off clusters to qa/cephfs/clusters
qa: allocate more space for VM disk
qa/cephfs/clusters/*: bigger cinder volumses

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/dashboard: Move validators into forms directory

Now that 'forms' directory exists, the validators are moved to
it. Angular also does this with it's validators ('@angular/forms').

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Use forms in deletion modal

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Use forms in RGW user capability

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Use forms in RGW subuser modal

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Use forms in RGW user s3 key modal

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Use forms in RGW user form

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Use forms in RGW bucket form

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Use forms in RBD snapshot form

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Use forms in RBD form

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: CdFormBuilder

Now the form builder can be used to build CdFormGroups.

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: CdFormGroup

CdFormGroup extends 'FormGroup' with a few new methods that will help
form development.

Signed-off-by: Stephan Müller <smueller@suse.com>

Merge pull request #22828 from ceph/wip-conf-error-message

ceph-volume: provide a nice errror message when missing ceph.conf

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #22548 from ovh/bp-cls-otp-build

cls: build cls_otp only WITH_RADOSGW

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #22764 from ceph/wip-rm24504

ceph-volume: refuse to zap mapper devices

Reviewed-by: Alfredo Deza

ceph-volume: always ignore a missing ceph conf in main.py

Now that we have a nice error message when a ceph.conf is missing
and we try to use values from it, maintaining a list of commands that
don't need ceph.conf isn't as helpful. We had actually missed 'simple
trigger' when we first implemented this causing all our luminous tests
for simple to fail when we backported.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #22808 from jcsp/wip-mgr-doc-initial-modules

doc/dashboard: don't advise mgr_initial_modules

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

doc/dashboard: don't advise mgr_initial_modules

This setting tends to confuse people, as it's only respected
on the very first startup of the cluster.  Instead, mention
it (with appropriate caveats) on the general mgr admin
page.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #22802 from bradfitz/its

doc: fix some it's -> its typos

Reviewed-by: Jos Collin <jcollin@redhat.com>

doc: fix some it's -> its typos

Signed-off-by: Brad Fitzpatrick <brad@danga.com>

Merge pull request #22761 from fullerdj/wip-djf-24686

osd/filestore: Change default filestore_merge_threshold to -10

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

ceph-volume: show a nice error message when ceph.conf is not loaded

If the ceph configuration file is not loaded correctly and then values
from it are used then an undescript error message is shown, e.g.

AttributeError: 'property' object has no attribute 'get'

With this change that same error condition shows the message:

RuntimeError: No valid ceph configuration file was loaded.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Add new OpenSUSE Leap id for install-deps.sh

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>

Merge pull request #21772 from oritwas/wip-rgw-test-bi-list

rgw: test bi list

Merge pull request #22769 from idryomov/wip-rbdmap-remote-fs-pre

systemd/rbdmap.service: order us before remote-fs-pre.target

Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #22760 from s0nea/wip-dashboard-config-option-flags

mgr/dashboard: add supported flag information to config options documentation

Reviewed-by: Sebastian Krah <skrah@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Add UI for Cluster-wide OSD Flags configuration

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #22772 from joscollin/wip-doc-dashboard-ref

doc: s/doc/ref for dashboard urls

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #22776 from tchaikov/wip-thread-setaffinity

crimson/thread: pin thread pool to given CPU

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc/mgr: add doc for crash mgr module

Signed-off-by: Dan Mick <dan.mick@redhat.com>

qa/suites/rados/mgr: Add test_crash

Signed-off-by: Dan Mick <dan.mick@redhat.com>

qa/tasks/mgr: add test_crash, call from test_module_selftest

Signed-off-by: Dan Mick <dan.mick@redhat.com>

qa/tasks/{ceph_manager.py,vstart_runner.py}: allow kwargs in raw_*

Allow passing kwargs (like stdin=) to the local and teuthology
clusters when running tests

Signed-off-by: Dan Mick <dan.mick@redhat.com>