crimson/net: apply strict checks when handle addresses

* Apply strict checks at each step of addresses exchange, make sure we
  fail early when notice inconsistent addresses sent from peer;
* Throw error::bad_peer_address for address validation from peer;
* Stop learning my address before sending it out for peer as identity;
* Fix potential racing when try to learn my address by submit to core 0;

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: fix crimson-osd msgr policies

Also add checks to make sure we have a valid target address with
lossless policy.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: use target_addr to store the address to reconnect

With lossless policy, we will use target_addr to reconnect to peer,
which would be more useful when we switch to entity_addrvec_t.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: remove duplicated logging when connected

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: set connection policy for v1 protocol

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: use set_ephemeral_port() for non-bound port

Ephemeral port is different from the bound port when connecting, and
different from the peer_addr port when accepting.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: printable accept address

entity_addr_t::TYPE_NONE is not printable, change to TYPE_ANY.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: unify loggings for message exchange

* remove duplicated information in message logs,
  because connection prefix already contains information about:
  "[my_name(my_logic_name) my_addr >> peer_name peer_addr]"
* unify message logs for both v1 and v2;
* continue to use familiar identifiers to highlight message logs:
  - received messages identified by "<== #seq === msg (type)";
  - sent messages identified by "--> #seq === msg (type)";

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: clean up loggings in v2 protocol

Defind log levels in V2 Protocol:
* error level, something error that cause connection to terminate:
  - fatal errors;
  - bugs;
* warn level: something unusual that identifies connection fault or replacement:
  - unstable network;
  - incompatible peer;
  - auth failure;
  - connection race;
  - connection reset;
* info level, something very important to show connection lifecycle,
  which doesn't happen very frequently;
* debug level, important logs for debugging, including:
  - all the messages sent/received (-->/<==);
  - all the frames exchanged (WRITE/GOT);
  - important fields updated (UPDATE);
  - connection state transitions (TRIGGER);
* trace level, trivial logs showing:
  - the exact bytes being sent/received (SEND/RECV(bytes));
  - detailed information of sub-frames;
  - integrity checks;
  - etc.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: cleanup, rename _peer_addr to _my_addr_from_peer

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: abort with unhandled exceptions from dispatcher

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: cleanup, fix spelling error

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: cleanup, return void from unexpected_tag()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: unify the usage of entity_type_t and entity_name_t

* use standard entity_type_t(uint8_t) in favor of peer_type_t(int);
* use standard ceph_entity_type_name() to print where possible;
* use peer_name to encapsulate peer_type and peer_id in Connection.h;
* clean related interfaces in Messenger and Connection;
* print peer_name in Connection logging prefix;

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #29033 from neha-ojha/wip-add-yt

doc/dev/developer_guide/index.rst: add youtube references

Reviewed-by: Noah Watkins <nwatkins@redhat.com>

Merge pull request #28649 from votdev/issue_40427

mgr/dashboard: Pool form uses different loading spinner

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Pool form uses different loading spinner

Use the 'cd-loading-panel' element to display the loading progress.

Fixes: https://tracker.ceph.com/issues/40427
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #29043 from votdev/issue_40778

mgr/dashboard: Form fields do not show error messages/hints.

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge PR #28951 into master

* refs/pull/28951/head:
mds: apply configuration changes through MDSRank
common/TrackedOp: make settings atomic

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28964 from cbodley/wip-qa-workunit-s3tests

qa/rgw: update run-s3tests.sh

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

mgr/dashboard: Form fields do not show error messages/hints.

Fixes: https://tracker.ceph.com/issues/40778
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #28682 from cyx1231st/wip-seastar-msgr-socket2

crimson/net: implement preemptive shutdown/close

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #26104 into master

* refs/pull/26104/head:
client/Client: change dir's size delta to current write file size delta

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #28683 into master

* refs/pull/28683/head:
mon/MDSMonitor: use stringstream instead of dout for mds repaired

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #28710 into master

* refs/pull/28710/head:
cephfs-shell: Remove undefined variable files in do_rm()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #28831 into master

* refs/pull/28831/head:
client: support the fallocate() when fuse version >= 2.9

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #28906 into master

* refs/pull/28906/head:
cephfs-shell: Fix TypeError in poutput()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mds: apply configuration changes through MDSRank

This avoids the need to acquire the mds_lock prior to responding to config
changes. The MDSRank can create a finisher context that acquires the lock
later.

Fixes: https://tracker.ceph.com/issues/40694
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

common/TrackedOp: make settings atomic

To avoid locks on configuration changes.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc/dev/developer_guide/index.rst: add youtube references

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge PR #28978 into master

* refs/pull/28978/head:
qa/tasks/mgr/dashboard/test_health: fix test_full_health test
mon: use per-pool stats only when all OSDs are reporting
osd: report whether we have per-pool stats
osd/osd_types: osd_stat_t: include num_per_pool_osds member

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #29003 into master

* refs/pull/29003/head:
qa/standalone/scrub/osd-scrub-snaps.sh: adjust expected output
osd/SnapMapper: to_raw_key: use hobject pool member, not SnapMapper:pool
ceph-objectstore-tool: fix use of SnapSet::snaps
qa/standalone/scrub/osd-scrub-snaps: fix kv grep

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #29013 into master

* refs/pull/29013/head:
qa/tasks/ceph_manager: 5s -> 15s for 'osd out' to be visible

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #28998 from alfredodeza/wip-rm40664

ceph-volume  zap always skips block.db, leaves them around

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

mgr/dashboard: controllers/grafana is not Python3 compatible (#28956)

mgr/dashboard: controllers/grafana is not Python3 compatible

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #28979 from falcon78921/wip-doc-18197

doc: updated OpenStack rbd documentation

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/standalone/scrub/osd-scrub-snaps.sh: adjust expected output

SnapSet now dumps just seq, not a (fake) SnapContext.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/SnapMapper: to_raw_key: use hobject pool member, not SnapMapper:pool

Some callers (ceph-objectstore-tool) instantiate SnapMapper with a bogus
pool id (0) and want to use it to adjust mappings.  We have the hobj
available, so use its pool member instead.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-objectstore-tool: fix use of SnapSet::snaps

Instead, use clone_snaps to identify clones.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Minor fixes (#29009)

mgr/dashboard: Minor fixes

- Fix radio input display
- Fix the close button on initiators card
- Fix missing 'fa-' prefix in spinner icon
- Fix label font-weight

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge PR #28945 into master

* refs/pull/28945/head:
os/bluestore: StupidAllocator stucks on 4GB allocations
os/bluestore: avoid length overflow in extents returned by Stupid

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #28990 into master

* refs/pull/28990/head:
qa: vstart_runner fails because of string index out of range

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

qa/tasks/ceph_manager: 5s -> 15s for 'osd out' to be visible

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-volume tests: verify that wipefs tries several times

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.zap: retry wipefs several times to prevent race condition failures

Signed-off-by: Alfredo Deza <adeza@redhat.com>

qa/tasks/mgr/dashboard/test_health: fix test_full_health test

New fields!

Signed-off-by: Sage Weil <sage@redhat.com>

qa/standalone/scrub/osd-scrub-snaps: fix kv grep

SnapMapper keys are now SNA_, not MAP_.

Fixes: http://tracker.ceph.com/issues/40725
Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Remove button and badge radius (#29001)

mgr/dashboard: Remove button and badge radius

Reviewed-by: Patrick Seidensal <pnawracay@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

mgr/dashboard: Fix form validation problems in RGW user form (#29000)

mgr/dashboard: Fix form validation problems in RGW user form

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #28267 from flumm/ceph-volume-zap-fix

ceph-volume lvm.zap fix cleanup for db partitions

Reviewed-by: Alfredo Deza <adeza@redhat.com>

mgr/dashboard: Fix label font-weight

Some labels were not bold, because the css rule didn't apply to them.
I have improved the css rule and also add missing classes in some labels.

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Fix missing 'fa-' prefix in spinner icon

This was causing the submit button to not show the spinner.

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Add nolockdep option to e2e-script (#28981)

mgr/dashboard: Add nolockdep option to e2e-script

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

mgr/dashboard: Fix the close button on initiators card

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Fix radio input display

We were using the wrong class for the labels.

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #28957 from mgerstner/sfw2

rpm,etc/sysconfig: remove SuSEfirewall2 support

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

packaging: remove SuSEfirewall2 support

removal of SuSEfirewall2 service, since SuSEfirewall2 has been replaced
by firewalld, see [1].

[1]: https://lists.opensuse.org/opensuse-factory/2019-01/msg00490.html

Fixes: http://tracker.ceph.com/issues/40738
Signed-off-by: Matthias Gerstner <matthias.gerstner@suse.de>

Merge pull request #28802 from lixiaoy1/cmake_pmdk

cmake: pmem/pmdk changes to cmake

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28984 from tchaikov/wip-cmake-rebuild-cleanup

cmake: use BUILD_ALWAYS for rebuilding external project

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #29002 into master

* refs/pull/29002/head:
qa/tasks/ceph_deploy: do not rely on ceph-create-keys

Reviewed-by: Vasu Kulkarni <vasu@redhat.com>

Merge PR #28643 into master

* refs/pull/28643/head:
doc: removed old EOL Ceph versions from the list
doc: added openSUSE distros to the list and updated the releases
doc: updated the OS recommendations to reflect the Nautilus release

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc: updated OpenStack rbd documentation

Removed references to older OpenStack releases. Updated
documentation in regards to using QCOW2 as a virtual machine
disk type.

Fixes: https://tracker.ceph.com/issues/18197
Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>

Merge PR #28944 into master

* refs/pull/28944/head:
qa/workunits/rados/test_cache_pool.sh: s/forward/proxy/
qa/workunits/cephtool/test.sh: cache forward mode is no longer allowed
mon/OSDMonitor: forward -> proxy, readforward -> readproxy on octopus
mon/OSDMonitor: disallow FORWARD and READFORWARD cache modes
osd: do proxy when in forward/readforward cache modes

Reviewed-by: Neha Ojha <nojha@redhat.com>

mgr/dashboard: Fix form validation problems in RGW user form

ng-bootstrap-form-validation requires that every "form-control" has
an associated "formControlName".

Since these didn't had one, we had to either removed the "form-control" class
or rename it to "cd-form-control" (which will apply the same style without
triggering the validation).

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Remove button and badge radius

Fixes: http://tracker.ceph.com/issues/38953
Signed-off-by: Tiago Melo <tmelo@suse.com>

qa/tasks/ceph_deploy: do not rely on ceph-create-keys

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #28426 from mkogan1/wip-rgw-40183

rgw: save an unnecessary copy of RGWEnv

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge PR #28969 into master

* refs/pull/28969/head:
qa/tasks/ceph_manager: make is_{clean,recovered,active_or_down} less racy

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #28943 into master

* refs/pull/28943/head:
qa: stop testing on 16.04 xenial

Reviewed-by: Kefu Chai <kchai@redhat.com>

mon: use per-pool stats only when all OSDs are reporting

Previously, we would start using the per-pool stat sums as soon as *any*
OSDs were reporting.  For a legacy cluster, that meant that as soon as
one bluestore instance is updated or one new bluestore OSD is created, the
usage stats per pool would become useless.

Instead, only use the per-pool stats once *all* OSDs are reporting the new
values.  This mostly aligns with the health warning when one more bluestore
OSDs are not reporting; once they are update the warning goes away.  (That
does not factor in fielstore OSDs, though; all OSDs need to be new *and*
bluestore.)

Signed-off-by: Sage Weil <sage@redhat.com>

qa: vstart_runner fails because of string index out of range

Fix problem in LocalRemote::_perform_checks_and_return_list_of_args() when LocalCephManager::raw_cluster_cmd() is called with a empty command parameter.

Fixes: https://tracker.ceph.com/issues/40729
Signed-off-by: Volker Theile <vtheile@suse.com>

cmake: do not always build brotli or zstd

these two libraries are relatively stable, and Ceph developers are not
likely to hack them. so no need to rebuild them every time.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake/modules/BuildRocksDB.cmake: use BUILD_ALWAYS

this option was introduced by cmake 3.1, and we require cmake 3.5.1, so
we can use it without using the homebrew solution.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Add nolockdep option to e2e-script

Adding the --nolockdep option to the vstart command
seems to fix the current issue where vstart just
hangs at the dashboard ac-user-create command.
Hopefully we can get the e2e tests to run again
in Jenkins by applying this change.
Note that this might be a workaround only.

Fixes: https://tracker.ceph.com/issues/40688
Signed-off-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #28766 from liewegas/wip-clean-out-nibblewise

osd/osd_types: drop last_backfill_bitwise member

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28801 from ideepika/wip-update-dump-historic-ops

osd: add duration field to dump_historic_ops method

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>

mgr/dashboard: fix icons variable assignment (#28961)

mgr/dashboard: fix icons variable assignment

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Update bootstrap (#25188)

mgr/dashboard: Update bootstrap

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #28797 from simon-rock/simon_bug_fix

mgr/status: fix 'ceph osd status' ZeroDivisionError

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28687 from hsiang41/mgr/diskprediction_cloud_service_unavailable

mgr/diskprediction_cloud: Service unavailable

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #28950 from tchaikov/wip-corpus-gen

script,doc: add gen-corpus.sh

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc/dev/corpus.rst: updates to match gen-corpus.sh

* to reference the script to generate corpus
* use qa/workunits/rgw/run-s3tests.sh for producing rgw corpus

Signed-off-by: Kefu Chai <kchai@redhat.com>

script: add gen-corpus.sh

to automate generation of ceph-object-corpus objects. it facilitate the
process of cutting a new release

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28070 from badone/wip-lazy-omap-test

Wip lazy omap test

Reviewed-by: Greg Farnum gfarnum@redhat.com

qa/workunits/rados/test_cache_pool.sh: s/forward/proxy/

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/cephtool/test.sh: cache forward mode is no longer allowed

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: forward -> proxy, readforward -> readproxy on octopus

Adjust cache modes on upgrade.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: disallow FORWARD and READFORWARD cache modes

Signed-off-by: Sage Weil <sage@redhat.com>

osd: report whether we have per-pool stats

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: osd_stat_t: include num_per_pool_osds member

Count OSDs that are reporting per-pool stats.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #28949 from adamemerson/wip-40416

Don't crash on copy when metadata directive not supplied

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/tasks/ceph_manager: make is_{clean,recovered,active_or_down} less racy

Currently these can be thrown off if the cluster is creating or removing
pools at the same time.  Fix by taking a single snapshot of the pg stats
and based our judgement on that.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #28940 into master

* refs/pull/28940/head:
common/config: Remove unused code.

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/rgw: run-s3tests.sh runs in s3tests directory

something looks for request_decision_graph.yml in the working directory

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cmake: Add the support to use system PMDK library

Signed-off-by: Scott Peterson <scott.d.peterson@intel.com>
Signed-off-by: Xiaoyan Li <xiaoyan.li@intel.com>

cmake: move pmdk build script to Buildpmem.cmake

Signed-off-by: Scott Peterson <scott.d.peterson@intel.com>
Signed-off-by: Xiaoyan Li <xiaoyan.li@intel.com>

cmake: update HAVE_PMEM to HAVE_BLUESTORE_PMEM

Signed-off-by: Scott Peterson <scott.d.peterson@intel.com>
Signed-off-by: Xiaoyan Li <xiaoyan.li@intel.com>

mgr/dashboard: Use bootstrap custom-select

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Use ng-bootstrap-form-validation

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Replace awesome-boostrap-checkbox with bootstrap

Fixes: http://tracker.ceph.com/issues/36400
Signed-off-by: Tiago Melo <tmelo@suse.com>