crimson/onode-staged-tree: add asserts

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: tolerate eagain during fix_index()

Fix the one-directional link during fix_index() which causes errors
during node destruction under interruptive eagain.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: cleanup Node tracking logic for eagain

Introduce deref_super/parent() to make sure the bi-directional links
are reset together to survive eagain.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: reduce unit test efforts

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: validate insert/lookup/erase with eagain

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: fix initialization in perf tool

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: free resources when call submit_transaction()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: tolerate eagain between extent allocation and node initialization

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: distinguish extent state between retired and invalid

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: extend SeastoreNodeExtentManager to test eagain

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: cleanup SeastoreSuper::write_root_laddr()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: change FLTreeOnodeManager to use create_seastore()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/seastore: simplify errors in OnodeManager

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: use eagain_future/ertr internally

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: handle fatal errors when call NodeExtentManager

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: use fine-grained errorator in NodeExtentManager

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #41392 from tchaikov/wip-librados

include/rados/librados.h: use UINT64_C to define constants

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #41409 from zdover23/wip-doc-security-2021-05-19-fifth-item

doc/security: updating fifth listitem

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #41410 from zdover23/wip-doc-security-2021-05-19-seventh-item

doc/security: enrich seventh listitem

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #41424 from athanatos/sjust/wip-interruptor-unittest-name

test/crimson/CMakeLists.txt: fix name of unittest-interruptible-future

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Xuehan Xu <xuxuehan@360.cn>

Merge pull request #41425 from athanatos/sjust/wip-seastore-build

crimson: fix misc build issues

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/tools/store_nbd/tm_driver: remove perfcounter argument

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/logging: fix log() argument

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crimson/CMakeLists.txt: fix name of unittest-interruptible-future

unittest-interruptible-future matches the other unit tests here.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #40836 from kamoltat/wip-autoscaler-new-alg-turn-on-off

pybind/mgr/pg_autoscaler: Added autoscale-profile feature

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

pybind/mgr/pg_autoscaler: Added autoscale-profile feature

The autoscaler by default will start out each pool with minimal
pgs and `scale-up` the pgs when there is more usage in each pool.

Users can now use the commands:

`osd pool set autoscale-profile scale-down` to make the pools
start out with a full complement of pgs and only `scale-down`
when usage ratio across the pools are not even.

`osd pool set autoscale-profile scale-up` (by default) to make the pools
start out with minimal pgs and `scale-up` the pgs when there
is more usage in each pool.

Edited KVMonitor.cc file to make the `autoscale_profile` variable
persistent.

Edited tests/test_cal_final_pg_target.py so that it takes into account
the new `profile` argument when calling cal_final_pg_target(). Also,
added some new test cases for when profile is `scale-up`

Renamed tests/test_autoscaler.py to a more appropriate name:
tests/test_cal_ratio.py

Signed-off-by: Kamoltat <ksirivad@redhat.com>

doc/security: enrich seventh listitem

This PR improves the language of the seventh listitem
in the Vulnerability Management Process in the security
documentation.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #41406 from zdover23/wip-doc-security-2021-05-19-first-item

doc/security: removing pleonasm

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/security: updating fifth listitem

This PR enriches the text in the fifth listitem
of the Vulnerability Management Process.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #41391 from tchaikov/wip-libcephfs

libcephfs: include <sys/time.h>

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #41359 from tchaikov/wip-cmake-cleanup

cmake: cleanups related to file(GLOB_RECURSE..) call

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

doc/security: removing pleonasm

s/three business days or less/three business days/

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #41355 from tchaikov/wip-doc-rgw-conf

doc/radosgw: use confval directive to define options

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #41399 from idryomov/wip-mrun-if-posix

mrun: avoid == in [ command

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41393 from rosinL/fix-50835

common/crc32c_aarch64: fix crc32c unittest failed on aarch64

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/radosgw: use confval directive to define options

less repeating this way

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #41388 into master

* refs/pull/41388/head:
cephadm: write config files as utf-8

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #41286 into master

* refs/pull/41286/head:
qa/suites/orch/rook: disable centos for now
qa/suites/orch/rook/smoke: initial smoke suite
qa/tasks/rook: ROOK_HOSTPATH_REQUIRES_PRIVILEGED=true on centos
qa/tasks/rook: simplify shutdown
qa/tasks/rook: archive logs
qa/tasks/rook: more orderly cluster teardown
qa/tasks/rook: deploy ceph via rook on top of kubernetes
qa/tasks/kubeadm: install kubernetes with kubeadm
qa/suites: move rados/cephadm -> orch/cephadm; symlink
qa/tasks/cephadm: add whitespace between functions
qa/tasks/cephadm: clean up ctx.manager setup

Reviewed-by: Sébastien Han <seb@redhat.com>

Merge PR #41370 into master

* refs/pull/41370/head:
mgr/cephadm: pin haproxy image to 2.3.z

Reviewed-by: Daniel Pivonka <dpivonka@redhat.com>
Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #41345 from rhcs-dashboard/fix-CVE-2021-3509-master

mgr/dashboard: fix cookie injection issue (CVE-2021-3509)

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

mrun: avoid == in [ command

== operator is not supported in POSIX sh (e.g. dash):

  ./mrun: 24: [: unexpected operator

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #41104 from rhcs-dashboard/fix-ESOCKETTIMEDOUT-e2e-failure

mgr/dashboard: fix ESOCKETTIMEDOUT E2E failure

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #41273 from rhcs-dashboard/update-frontend-deps

mgr/dashboard: update frontend deps due to security vulnerabilities

Reviewed-by: Waad Alkhoury <walkhour@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

common/crc32c_aarch64: fix crc32c unittest failed on aarch64

On centos 8.2 for aarch64 with gcc 8.3, the complier will use
register v0 conflicting with the register v0 be usded in inline
asm code. Adding the related registers into clobber list to inform
complier avoiding the confict.

Fixes: https://tracker.ceph.com/issues/50835
Signed-off-by: luo rixin <luorixin@huawei.com>

cephadm: write config files as utf-8

If we don't specify the encoding, python may use something like 'latin-1',
which cannot handle unicode characters.

Signed-off-by: Sage Weil <sage@newdream.net>

include/rados/librados.h: use UINT64_C to define constants

UINT64_C add the appropriate suffix to literals, this is more readable
than casting the literal to the expected type.

Signed-off-by: Kefu Chai <kchai@redhat.com>

libcephfs: include <sys/time.h>

header files should be self-contained. we are using `timeval` in the header,
but failed to include the header defining it.

this change addresses the build failures on Darwin like:

int ceph_lutimes(struct ceph_mount_info *cmount, const char *path, struct timeval times[2]);
                                                                          ^
src/include/cephfs/libcephfs.h:1080:78: error: array has incomplete element type 'struct timeval'
int ceph_futimes(struct ceph_mount_info *cmount, int fd, struct timeval times[2]);
                                                                             ^
src/include/cephfs/libcephfs.h:1080:65: note: forward declaration of 'struct timeval'
int ceph_futimes(struct ceph_mount_info *cmount, int fd, struct timeval times[2]);
                                                                ^
3 errors generated.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41390 from batrick/i50865

doc: use sphinx .dot include

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #41357 into master

* refs/pull/41357/head:
mds: do not assert when receiving a unknow metric type

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #41347 into master

* refs/pull/41347/head:
mon,doc: deprecate CephFS min_compat_client
doc: add alternate_name cephfs feature to table

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge PR #41332 into master

* refs/pull/41332/head:
mds: place the journaler pointer under the mds_lock

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #41171 into master

* refs/pull/41171/head:
test: disable mirroring module for certain tests

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #41097 into master

* refs/pull/41097/head:
cephfs-mirror: allow connecting to local cluster using mon address

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sébastien Han <seb@redhat.com>

Merge PR #40885 into master

* refs/pull/40885/head:
doc: document cephfs-mirror configuration options
cephfs-mirror: use sensible mount timeout when mounting local/remote fs
test: add tests for settting mount timeout
pybind/cephfs: add interface to set mount timeout
libcephfs: add interface to set mount timeout

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #41371 into master

* refs/pull/41371/head:
mds/Mutation: initialize MutationImpl object's initiated_at

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/suites/orch/rook: disable centos for now

We are hitting an selinux denial:

type=AVC msg=audit(1621353663.595:6016): avc: denied { write } for pid=28812 comm="cephcsi" name="pids.max" dev="cgroup" ino=2933 scontext=system_u:system_r:container_t:s0:c43,c177 tcontext=system_u:object_r:cgroup_t:s0 tclass=file permissive=1

Signed-off-by: Sage Weil <sage@newdream.net>

qa/suites/orch/rook/smoke: initial smoke suite

- os: 18.04, 20.04, centos, centos+kubic
- nodes: 1 or 3
- rook: 1.6.2 or master
- k8s: 1.21 always (for now)
- net: calico always (flannel + centos is wonky)
- workload: none, rados bench

Signed-off-by: Sage Weil <sage@newdream.net>

qa/tasks/rook: ROOK_HOSTPATH_REQUIRES_PRIVILEGED=true on centos

This fixes OSD creation on centos.

Signed-off-by: Sage Weil <sage@newdream.net>

qa/tasks/rook: simplify shutdown

For some reason deleting common.yaml sometimes fails.  Not really
sure why, but since we will tear down kubernetes anyway this
cleanup isn't really needed.

Signed-off-by: Sage Weil <sage@newdream.net>

qa/tasks/rook: archive logs

Signed-off-by: Sage Weil <sage@newdream.net>

qa/tasks/rook: more orderly cluster teardown

Signed-off-by: Sage Weil <sage@newdream.net>

qa/tasks/rook: deploy ceph via rook on top of kubernetes

This assumes that k8s is installed and kubectl works.

The ceph container to use is selected the same way the cephadm
task does it.

All scratch devices are consumed as OSDs.

A ceph.conf and client.admin keyring are deployed on all test
nodes, so normal tasks should work (if/when packages are installed).

Fixes: https://tracker.ceph.com/issues/47507
Signed-off-by: Sage Weil <sage@newdream.net>

doc: use sphinx .dot include

To compile svg automatically into docs.

Fixed: https://tracker.ceph.com/issues/50865
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #40921 from kamoltat/wip-pool-start-out-with-1-pg

mon/OSDMonitor: prepare_new_pool() starts out with 1 pg

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #41383 into master

* refs/pull/41383/head:
doc/security: Add single GPG key in lieu of existing three GPG keys

Reviewed-by: Sage Weil <sage@redhat.com>

qa/tasks/kubeadm: install kubernetes with kubeadm

- install k8s with kubeadm
- initial support for flannel only
- remove taint from bootstrap/master node
- create PVs for all scratch_devs + a 'scratch' SC
- kubeadm.kubectl task

Signed-off-by: Sage Weil <sage@newdream.net>

qa/suites: move rados/cephadm -> orch/cephadm; symlink

Move cephadm under orch/ top-level suite.  Symlink so that we
still include it in a rados run.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #41352 from tchaikov/wip-crimson-metrics

crimson/os/seastore: use seastar::metrics for metrics reporting

Reviewed-by: Amnon Hanuhov <ahanukov@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #41377 from rhcs-dashboard/change-password-preserve-url-fix

mgr/dashboard: Fix for query params resetting on change-password

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

qa/tasks/cephadm: add whitespace between functions

Signed-off-by: Sage Weil <sage@newdream.net>

doc/security: Add single GPG key in lieu of existing three GPG keys

Replace existing three GPG keys with new Ceph Security Team GPG key

Signed-off-by: Hardik Vyas <hvyas@redhat.com>

Merge PR #40640 into master

* refs/pull/40640/head:
common: send SYSLOG_IDENTIFIER to journald
cephadm: enable log to journald by default

Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>

Merge pull request #39580 from ifed01/wip-ifed-migrate

ceph-volume: implement bluefs volume migration.

mgr/cephadm: pin haproxy image to 2.3.z

The 2.4 version fails for reasons that aren't immediate clear:

May 17 15:28:42 dael conmon[3007013]: [NOTICE]   (8) : haproxy version is 2.4.0-6cbbecf
May 17 15:28:42 dael conmon[3007013]: [NOTICE]   (8) : path to executable is /usr/local/sbin/haproxy
May 17 15:28:42 dael conmon[3007013]: [ALERT]    (8) : Cannot open configuration file/directory /var/lib/haproxy/haproxy.cfg : Permission denied

Even with mode 777 on those files, there is an error about the chroot to
/var/lib/haproxy.

For now, just stick with a working version.

Fixes: https://tracker.ceph.com/issues/50830
Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #41282 from cbodley/wip-rgw-rm-civetweb

rgw: remove the civetweb and fcgi frontends

crimson/admin: reindent SeastarMetricsHook

we don't use 1-space indent.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/admin: add optional prefix arg to "perf dump_seastar"

so we can filter the dumped metrics by their prefix

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/admin: do not construct AdminSocketHook with osd

osd is not used at all.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/admin: add desc for DumpPerfCountersHook

logger and counter are used when executing the hook

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os/seastore: use seastar::metrics for metrics reporting

to be better prepeare for the m:n mapping and to attach more labels
to a metric for better off-line data analysis using ETL tools in future.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/admin: print different metric type in different way

and use registered_metric::operator() for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/admin: use better variable names

for better readability.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: define SeastoreCollection as a class

to be consistent with the forward declaration. C++ standard does not
differentiate class from struct in this perspective. but Clang warngs at
seeing it. so silence the warning.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41364 from rzarzynski/wip-crimson-monc-pending_messages-assert

crimson/monc: fix send_message() racing with reopen_session().

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41366 from tchaikov/wip-crimson-os-debug

crimson/os: use compile-time validation

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>

mgr/dashboard: Fix for query params resetting on change-password

Fixes: https://tracker.ceph.com/issues/50857
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #41354 from idryomov/wip-rbd-pwl-ssd-recovery

librbd/cache/pwl/ssd: fix some crash recovery issues

Reviewed-by: Yin Congmin <congmin.yin@intel.com>
Reviewed-by: Mahati Chamarthy <mahati.chamarthy@intel.com>

Merge pull request #39772 from xxhdx1985126/wip-crimson-client-req-pipeline-parallelism

crimson/osd: optimize crimson-osd's client requests process parallelism

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #41362 from Aran85/crimson-diagrams

crimson/seastore: add string_kv_node_layout diagrams

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge PR #40234 into master

* refs/pull/40234/head:
client: always register callbacks before mount()
client: move SnapRealm methods to ClientSnapRealm.cc

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #40842 into master

* refs/pull/40842/head:
qa: update the ffsb.sh to clone it from git://git.ceph.com/ffsb.git

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #41235 into master

* refs/pull/41235/head:
mds: PurgeQueue.cc fix for 32bit compilation

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #41239 into master

* refs/pull/41239/head:
librbd: use uint64_t instead of size_t for SparseExtent::length
mgr/PyModule: use Py_ssize_t for the PyList index
os/bluestore: print size_t using %xz
client: print int64_t using PRId64

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge PR #41254 into master

* refs/pull/41254/head:
mds: save the metadata pool id MDSRank class's private member

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #41267 into master

* refs/pull/41267/head:
mds: defer the journal recovered success log

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #41268 into master

* refs/pull/41268/head:
mds: fix possible mds_lock not locked assert

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #41341 from tchaikov/wip-dmclock

dmclock: pick up change to fix run_sched_ahead() scheduling issue

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sridhar Seshasayee <sseshasa@redhat.com>

mds/Mutation: initialize MutationImpl object's initiated_at

... to current time, ceph_clock_now(). This is a more useful
initialized value than utime_t().

Fixes: https://tracker.ceph.com/issues/42516
Signed-off-by: Ramana Raja <rraja@redhat.com>
Co-Authored-By: Patrick Donnelly <pdonnell@redhat.com>

mon,doc: deprecate CephFS min_compat_client

Instead admins should specify specific features to require.

Fixes: https://tracker.ceph.com/issues/50819
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

rgw: building the beast frontend is no longer optional

Signed-off-by: Casey Bodley <cbodley@redhat.com>