test: add librbd/cache/ParentImageCache UT

Signed-off-by: Dehao Shang <dehao.shang@intel.com>

test: add domain socket mock

Signed-off-by: Dehao Shang <dehao.shang@intel.com>

librbd: template CacheClient at RO hook

Signed-off-by: Dehao Shang <dehao.shang@intel.com>

librbd: enable librbd hook for shared RO cache

Signed-off-by: Dehao Shang <dehao.shang@intel.com>
Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

Merge pull request #28665 from mikechristie/wip-doc-mikechristie-update-igw-ansible

doc: update ceph ansible iscsi info

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #28721 from yuriw/wip-yuriw-crontab-master

qa/tests: changed the TO email to 'ceph-qa@ceph.io'

doc: Add rbd-target-api ceph iscsi ansible vars

Signed-off-by: Mike Christie <mchristi@redhat.com>

doc: Update rbd-target-gw behavior info

The lio setup and teardown functionality was moved from rbd-target-gw to
rbd-target-api in ceph-iscsi.

Signed-off-by: Mike Christie <mchristi@redhat.com>

doc: Update ceph-ansible iscsigws settings

The ceph-ansible ceph-iscsi support does not allow setting up iscsi
objects. It must be done through gwcli like is done in the container
case. This drops the iscsi object related settings and admin info for
it.

Signed-off-by: Mike Christie <mchristi@redhat.com>

doc: Fix ceph-ansible iscsigws command

Use the more standard --limit iscsigws argument instead of running the
playbook command on the iscsigws.yml file directly.

Signed-off-by: Mike Christie <mchristi@redhat.com>

doc: Fix igw ceph ansible host group name

The ceph-ansible iscsi group name is iscsigws and not
ceph-iscsi-gws.

Signed-off-by: Mike Christie <mchristi@redhat.com>

Merge PR #28624 into master

* refs/pull/28624/head:
mgr/zabbix: encode string for Python 3 compatibility

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #28658 into master

* refs/pull/28658/head:
qa/suites/rados/thrash: force normal pg log length with cache tiering

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #28662 into master

* refs/pull/28662/head:
osd: share curmap in handle_osd_ping

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #28430 from cbodley/wip-40187

cls/rgw: keep issuing bilog trim ops after reset

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #28667 from majianpeng/msg-async-comments

msg/async: add comments for commit 294c41f18adada6ab.

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/tests: changed the TO email to 'ceph-qa@ceph.io'

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #28704 from joke-lee/sts_assume_policy

rgw: Policy should be url_decode when assume_role

Reviewed-by: Pritha Srivastava <prsrivas@redhat.com>

msg/async: add comments for commit 294c41f18adada6ab.

Consider this case:
    send-thread                                 msg-work
                                            write_event()
                                             r = write_message()
connection->write_lock.lock()
if (.. && !write_in_progress) {
  write_in_progress = true;
  add external_event
}
connection->write_lock.unlock()
                                          connection->write_lock.lock()
                                          if (r > 0)
                                            break;
                                          } while ();
                                          write_in_progress = false;

For this case, we don't add external_event and in write_event we don't
check out_q whether empty rather than break.
This make msg-work never wake up again.
Fortunately, if write_message() > 0, in AsyncConnection::_try_send will
add an EVENT_WRITEABLE to wake up msg-work. So bug can't occur.

I add comment to descript this and hope get the better method to fix
this.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #28709 from tchaikov/wip-seastar

seasetar: pick up the latest seastar

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

seasetar: pick up the latest seastar

couple noticeable changes

* use dpdk v19.05, please note the older version of dpdk might not
  work, for instance dpdk v18.11 does not compile with the latest
  seastar master HEAD
* future cleanups

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28707 from majianpeng/perf-async-msgr-remove-unsued-header

test/crimson/perf_async_msgr: remove unsued header file.

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/crimson/perf_async_msgr: remove unsued header file.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #28684 from tchaikov/wip-silence-warnings

src/: silence GCC warnings

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #28677 from liewegas/wip-recovery-nvme

osd: add hdd and ssd variants for osd_recovery_max_active

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>

rgw, Policy should be url_decode when assume_role

Signed-off-by: yuliyang <yuliyang@cmss.chinamobile.com>

Merge pull request #17619 from liuchang0812/wip-ec-below-min-size

osd: allow EC PGs to do recovery below min_size

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28697 from cbodley/wip-qa-rgw-swift-skip-client

qa/rgw: fix swift warning message

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

qa/rgw: fix swift warning message

  File "/home/teuthworker/src/github.com_ceph_ceph_master/qa/tasks/swift.py", line 245, in task
    log.warning('Swift tests cannot run on rhel 7.6+, skipping client {client}'.format(client))
KeyError: 'client'

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28657 from dillaman/wip-40435

rbd: use the ordered throttle for the export action

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #28567 into master

* refs/pull/28567/head:
Revert "mon/OSDMonitor: report pg[pgp]_num_target instead of pg[pgp]_num"

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #28674 from tchaikov/wip-doc/crimson

doc/dev: add crimson.rst

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #28462 from myoungwon/wip-doc-dedup

doc: add a deduplication document

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26251 from majianpeng/add-new-op-create

os/objectstore: add new op OP_CREATE for create a new object.

Reviewed-by: Sage Weil <sage@redhat.com>

doc: add a deduplication document

Signed-off-by: Myoungwon Oh <omwmw@sk.com>

doc/dev: add crimson.rst

Signed-off-by: Kefu Chai <kchai@redhat.com>

test: Replace deprecated TYPED_TEST_CASE with TYPED_TEST_SUITE

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Fix pessimizing move

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

client,rbd,mon: Fix signed compare warnings

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

test/simple_spin: Explicitly wait for futures from async

Rather than count on implicitly blocking destruct.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

OSD: Initialize pool and recovery_state in the order they are defined

Currently we call get_pool on the PeeringState before the PeeringState
has been initialized.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rbd: Remove unused capture

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

immutable_object_cache: Remove unused capture

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

immutable_object_cache: Remove pessimizing moves

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

osd: add hdd and ssd variants for osd_recovery_max_active

Semi-arbitrarily set the SSD max to 10 (instead of 3).  This should be
tuned based on some real data.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #28676 from tchaikov/wip-stop.sh-with-crimson

stop.sh: add --crimson option

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #28204 from dzafman/wip-39555

mon: Improve health status for backfill_toofull and recovery_toofull

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #28663 into master

* refs/pull/28663/head:
cephfs-shell: use consistent naming for paths
cephfs-shell: remove unnecessary instances of to_bytes
cephfs-shell: refactor list_items
pybind/cephfs: improve user-friendliness of DirResult
pybind/cephfs: avoid unicode check

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #28561 into master

* refs/pull/28561/head:
vstart_runner: upgrade the check for commands to be run as another user
vstart_runner: split unicode arguments into lists

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #28049 from theanalyst/rgw-lc-bl-fix

rgw_lc: use a new bl while encoding RGW_ATTR_LC

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #28276 from liuchang0812/multi-site-request-forward

rgw: forwarded some requests to master zone

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #26167 from yaozongyou/rgw-lc-add-more-check

rgw: check lc objs not empty after fetching

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #26538 from zhangsw/feature-rgw-object-lock

rgw: add S3 object lock feature to support object worm

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #26351 from linuxbox2/wip-rgw-gcretire

rgw: perfcounters: add gc retire counter

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #28527 from tianshan/fix_ret

rgw: fix miss get ret in STSService::storeARN

Reviewed-by: Pritha Srivastava <prsrivas@redhat.com>

Merge pull request #28327 from linuxbox2/wip-rgw-email

rgw: conditionally allow non-unique email addresses

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28308 from joke-lee/usage_dump

rgw: usage dump_unsigned instead dump_int

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28172 from joke-lee/rgw-sts-bad-token

rgw: fix rgw crash and set correct error code

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Pritha Srivastava <prsrivas@redhat.com>

vstart_runner: upgrade the check for commands to be run as another user

Rectify the condition that checks if command to be issued as another
user using sudo is issued as a single argument after "-c".

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #28539 from dillaman/wip-40072

librbd: improve journal performance to match expected degradation

Reviewed-by: Mykola Golub <mgolub@suse.com>

stop.sh: add --crimson option

so we can pkill crimson-osd if this option is specified

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28655 from dengchj/multisite_doc

doc: fixed --read-only argument value in multisite doc

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #28194 into master

* refs/pull/28194/head:
test_volume_client: declare only one default for python version
test_volume_client: don't shadow class variable

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

cephfs-shell: use consistent naming for paths

And some small changes to make some commands take variable arguments and fix
defaults.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

cephfs-shell: remove unnecessary instances of to_bytes

cephfs binding already automatically converts paths and some arguments to
bytes.

Fixes: http://tracker.ceph.com/issues/40455
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

cephfs-shell: refactor list_items

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

pybind/cephfs: improve user-friendliness of DirResult

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

pybind/cephfs: avoid unicode check

py3 does not have the unicode built-in. Instead, simply try to do the encoding
and catch failures appropriately. If it quacks like a duck...

(Note that cython apparently cheats and allows the unicode check but this is
simpler.)

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #28638 into master

* refs/pull/28638/head:
qa: elide python version config
cephfs-shell: assert python3

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #28498 from kamoltat/wip-mgr-progress-add-event-osd-marked-in

mgr/progress: Add recovery event when OSD marked in

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #28564 from ceph/store_statfs-log

log: fix store_statfs log line

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #28621 from Aran85/fix-ec-rf

osd: accident of rollforward may need to mark pglog dirty

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #28656 from dvanders/zstd-upgrade

zstd: upgrade to v1.4.0

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/zabbix: encode string for Python 3 compatibility

Credits to Jochem Kuijpers for the fix.

Fixes: https://tracker.ceph.com/issues/36318
Signed-off-by: Nathan Cutler <ncutler@suse.com>

test_volume_client: declare only one default for python version

Fixes: http://tracker.ceph.com/issues/40460
Signed-off-by: Rishabh Dave <ridave@redhat.com>

vstart_runner: split unicode arguments into lists

Split not just string arguments but also unicode arguments into lists.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #28570 from xiexingguo/wip-objecter-direct-spgid

osdc/Objecter: _calc_target - inline spgid

Reviewed-by: Sage Weil <sage@redhat.com>

doc: fixed --read-only argument value in multisite doc

Fixes: https://tracker.ceph.com/issues/40458
Signed-off-by: Chenjiong Deng <dengchenjiong@umcloud.com>

Merge pull request #28568 from xiexingguo/wip-off-by-one-max-osd

mon/OSDMonitor: fix off-by-one when updating new_last_in_change

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #28629 from xiexingguo/wip-osd-df-by-pool-name

mgr, osd: 'osd df' by pool

Reviewed-by: Sage Weil <sage@redhat.com>

mon: Improve health status for backfill_toofull and recovery_toofull

Treat backfull_toofull as a warning condition because it can resolve itself.
Includes test case for PG_BACKFILL_FULL
Includes test case for recovery_toofull / PG_RECOVERY_FULL

Fixes: https://tracker.ceph.com/issues/39555
Signed-off-by: David Zafman <dzafman@redhat.com>

qa: elide python version config

This test doesn't actually use the config and cephfs-shell is py3 only.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

cephfs-shell: assert python3

There's little reason to support this new tool with python2.

Fixes: http://tracker.ceph.com/issues/40418
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #28553 from trociny/wip-40071

librbd: mirror image enable/disable should enable/disable journaling

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #28627 from trociny/wip-40409

journal: properly advance read offset after skipping invalid range

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

osd: share curmap in handle_osd_ping

We just took the curmap ref above; do not call get_osdmap() again.

I think it may explain a weird segv I saw here in ~shared_ptr, although
I'm not quite certain.  Regardless, this change is correct and better.

(gdb) bt
#0  raise (sig=sig@entry=11) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x00005596e5a98261 in reraise_fatal (signum=11) at ./src/global/signal_handler.cc:326
#2  handle_fatal_signal(int) () at ./src/global/signal_handler.cc:326
#3  <signal handler called>
#4  0x00005596f4fe80e0 in ?? ()
#5  0x00005596e5464068 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x5596f4b7cf60) at /usr/include/c++/9/bits/shared_ptr_base.h:148
#6  std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x5596f4b7cf60) at /usr/include/c++/9/bits/shared_ptr_base.h:148
#7  0x00005596e543377f in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=0x7f2b25044e28, __in_chrg=<optimized out>) at /usr/include/c++/9/bits/shared_ptr_base.h:1169
#8  std::__shared_ptr<OSDMap const, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (this=0x7f2b25044e20, __in_chrg=<optimized out>) at /usr/include/c++/9/bits/shared_ptr_base.h:1169
#9  std::shared_ptr<OSDMap const>::~shared_ptr (this=0x7f2b25044e20, __in_chrg=<optimized out>) at /usr/include/c++/9/bits/shared_ptr.h:103
#10 OSD::handle_osd_ping(MOSDPing*) () at ./src/osd/OSD.cc:4662

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #28460 from tchaikov/wip-cmake-detect-libnl

cmake: rewrite Findgenl to support components argument

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/suites/rados/thrash: force normal pg log length with cache tiering

When we are doing cache tiering, we are more sensitive to short PG logs
because the dup op entries are not perfectly promoted from the base to
the cache.

See:
 http://tracker.ceph.com/issues/38358
 http://tracker.ceph.com/issues/24320

This works around the problem by not testing short pg logs in combination
with cache tiering.  This works because the short_pg_log.yaml fragment
sets the short log in the [global] section but the cache workloads overload
it (back to a large/default value) in the [osd] section.

Signed-off-by: Sage Weil <sage@redhat.com>

rbd: use the ordered throttle for the export action

This allows exports to STDOUT to use multiple concurrent operations
and also fixes a potential race condition with concurrent callbacks
and file seeking.

Fixes: http://tracker.ceph.com/issues/40435
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

zstd: compat with v1.4.0

In zstd d8e215cbee03b038fffe74aebad63b625c42f23c
ZSTD_compress_generic() is renamed to ZSTD_compressStream2().

Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>

librbd: tweaks to improve throughput for journaled IO

Increase the per-object append flush bytes to 1MiB to mimic the 16MiB
in-memory writeback cache. Finally, default enable the new journal
writethrough until flush option.

Performance results comparing no journal, pre-change, and post-change:

no journal original modified
4K 40310.84 7223.73 27427.27
32K 21530.37 3256.39 10284.05
256K 3159.67 564.73 1627.16
4M 223.03 48.03 91.91

Fixes: http://tracker.ceph.com/issues/40072
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: new rbd_journal_object_writethrough_until_flush option

When set to true, the journal will not attempt to batch appends until
after it receives the the first flush request from the user.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: support dynamically updating recorder flush options

Default to disabling writeback-style append flushes unless overridden
by a call to 'set_append_batch_options'.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: fix broken append batching implementation

The original flush_interval/bytes/age batching options no longer
actually do anything. Integrate these better with the max in-flight
AIO handling.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

zstd: upgrade to v1.4.0

Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>

Merge pull request #28631 from cbodley/wip-17882

qa/rgw: extra s3tests tasks use rgw endpoint configuration

Reviewed-by: Kefu Chai <kchai@redhat.com>

debian/control: remove "libnl-3-dev" from build dependencies

what we need is "libnl-genl-3-dev", libnl-3-dev should be a dependency
of "libnl-genl-3-dev". the only reason we would have added it, is if
the package maintainer of "libnl-genl-3-dev" fails to do so. and
the packages of "libnl-genl-3-dev" on both debian and ubuntu just look
correct in this perspective.

so let's remove this.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: rewrite Findgenl to support components argument

* rename genl to nl:
  "genl" is not very specific without more context. and libnl is actually
  a library suite. so it would be better if we can rename the library to
  be found to "libnl", and by following the naming convention of CMake,
  in this change, Findgenl.cmake is renamed to Findnl.cmake
* Findnl.cmake: support the component argument, in our specific case,
  what we want is "libnl-genl" library, which is one of the libraries
  offered by libnl. so let's just make it a component.
* Findnl.cmake: should pass the names of the required variables instead of
  their values to `find_package_handle_standard_args()`. before this
  change, we pass `GENL_LIBRARIES` to this function. it was correct at the
  first glance. but it is not able to handle the case where case where
  libnl-genl is not installed. so the fix is to pass all the names of
  required library paths to this function. in this change, their name
  are concatenated to a single variable -- `nl_LIBRARIES`, and the
  value of this variable is passed to
  `find_package_handle_standard_args()`. and the error message would
  be more specific this way if libnl-genl is not found:
    Could NOT find nl (missing: nl_genl_LIBRARY)
* Findnl.cmake: add nl::<component> as imported library, it helps the
  consumer of these libraries to import them -- no need to
  specify the `target_include_directories()` separately anymore.
* move the find_package() code to where it is used. it helps to improve
  the readability.
* remove `HAVE_GENL` variable: it's not used anywhere.
* drop the messages of "Not using foobar", they do not help.

Signed-off-by: Kefu Chai <kchai@redhat.com>

journal: improve logging on recorder append path

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: Ensure E2E tests can be run independently (#28592)

mgr/dashboard: Ensure E2E tests can be run independently

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Nathan Weinberg <nweinber@redhat.com>
Reviewed-by: Rafael Quintero <rquinter@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>