os/bluestore: load OSD all compression settings unconditionally.

Inititally we don't load them if OSD compression mode is set to none.
But we might need them when pool's compression is enabled but some
settings aren't set.

Fixes: http://tracker.ceph.com/issues/40480
Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #26251 from majianpeng/add-new-op-create

os/objectstore: add new op OP_CREATE for create a new object.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #28676 from tchaikov/wip-stop.sh-with-crimson

stop.sh: add --crimson option

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #28204 from dzafman/wip-39555

mon: Improve health status for backfill_toofull and recovery_toofull

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #28663 into master

* refs/pull/28663/head:
cephfs-shell: use consistent naming for paths
cephfs-shell: remove unnecessary instances of to_bytes
cephfs-shell: refactor list_items
pybind/cephfs: improve user-friendliness of DirResult
pybind/cephfs: avoid unicode check

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #28561 into master

* refs/pull/28561/head:
vstart_runner: upgrade the check for commands to be run as another user
vstart_runner: split unicode arguments into lists

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #28049 from theanalyst/rgw-lc-bl-fix

rgw_lc: use a new bl while encoding RGW_ATTR_LC

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #28276 from liuchang0812/multi-site-request-forward

rgw: forwarded some requests to master zone

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #26167 from yaozongyou/rgw-lc-add-more-check

rgw: check lc objs not empty after fetching

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #26538 from zhangsw/feature-rgw-object-lock

rgw: add S3 object lock feature to support object worm

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #26351 from linuxbox2/wip-rgw-gcretire

rgw: perfcounters: add gc retire counter

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #28527 from tianshan/fix_ret

rgw: fix miss get ret in STSService::storeARN

Reviewed-by: Pritha Srivastava <prsrivas@redhat.com>

Merge pull request #28327 from linuxbox2/wip-rgw-email

rgw: conditionally allow non-unique email addresses

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28308 from joke-lee/usage_dump

rgw: usage dump_unsigned instead dump_int

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28172 from joke-lee/rgw-sts-bad-token

rgw: fix rgw crash and set correct error code

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Pritha Srivastava <prsrivas@redhat.com>

vstart_runner: upgrade the check for commands to be run as another user

Rectify the condition that checks if command to be issued as another
user using sudo is issued as a single argument after "-c".

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #28539 from dillaman/wip-40072

librbd: improve journal performance to match expected degradation

Reviewed-by: Mykola Golub <mgolub@suse.com>

stop.sh: add --crimson option

so we can pkill crimson-osd if this option is specified

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28655 from dengchj/multisite_doc

doc: fixed --read-only argument value in multisite doc

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #28194 into master

* refs/pull/28194/head:
test_volume_client: declare only one default for python version
test_volume_client: don't shadow class variable

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

cephfs-shell: use consistent naming for paths

And some small changes to make some commands take variable arguments and fix
defaults.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

cephfs-shell: remove unnecessary instances of to_bytes

cephfs binding already automatically converts paths and some arguments to
bytes.

Fixes: http://tracker.ceph.com/issues/40455
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

cephfs-shell: refactor list_items

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

pybind/cephfs: improve user-friendliness of DirResult

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

pybind/cephfs: avoid unicode check

py3 does not have the unicode built-in. Instead, simply try to do the encoding
and catch failures appropriately. If it quacks like a duck...

(Note that cython apparently cheats and allows the unicode check but this is
simpler.)

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #28638 into master

* refs/pull/28638/head:
qa: elide python version config
cephfs-shell: assert python3

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #28498 from kamoltat/wip-mgr-progress-add-event-osd-marked-in

mgr/progress: Add recovery event when OSD marked in

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #28564 from ceph/store_statfs-log

log: fix store_statfs log line

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #28621 from Aran85/fix-ec-rf

osd: accident of rollforward may need to mark pglog dirty

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #28656 from dvanders/zstd-upgrade

zstd: upgrade to v1.4.0

Reviewed-by: Kefu Chai <kchai@redhat.com>

test_volume_client: declare only one default for python version

Fixes: http://tracker.ceph.com/issues/40460
Signed-off-by: Rishabh Dave <ridave@redhat.com>

vstart_runner: split unicode arguments into lists

Split not just string arguments but also unicode arguments into lists.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #28570 from xiexingguo/wip-objecter-direct-spgid

osdc/Objecter: _calc_target - inline spgid

Reviewed-by: Sage Weil <sage@redhat.com>

doc: fixed --read-only argument value in multisite doc

Fixes: https://tracker.ceph.com/issues/40458
Signed-off-by: Chenjiong Deng <dengchenjiong@umcloud.com>

Merge pull request #28568 from xiexingguo/wip-off-by-one-max-osd

mon/OSDMonitor: fix off-by-one when updating new_last_in_change

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #28629 from xiexingguo/wip-osd-df-by-pool-name

mgr, osd: 'osd df' by pool

Reviewed-by: Sage Weil <sage@redhat.com>

mon: Improve health status for backfill_toofull and recovery_toofull

Treat backfull_toofull as a warning condition because it can resolve itself.
Includes test case for PG_BACKFILL_FULL
Includes test case for recovery_toofull / PG_RECOVERY_FULL

Fixes: https://tracker.ceph.com/issues/39555
Signed-off-by: David Zafman <dzafman@redhat.com>

qa: elide python version config

This test doesn't actually use the config and cephfs-shell is py3 only.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

cephfs-shell: assert python3

There's little reason to support this new tool with python2.

Fixes: http://tracker.ceph.com/issues/40418
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #28553 from trociny/wip-40071

librbd: mirror image enable/disable should enable/disable journaling

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #28627 from trociny/wip-40409

journal: properly advance read offset after skipping invalid range

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #28460 from tchaikov/wip-cmake-detect-libnl

cmake: rewrite Findgenl to support components argument

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

zstd: compat with v1.4.0

In zstd d8e215cbee03b038fffe74aebad63b625c42f23c
ZSTD_compress_generic() is renamed to ZSTD_compressStream2().

Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>

librbd: tweaks to improve throughput for journaled IO

Increase the per-object append flush bytes to 1MiB to mimic the 16MiB
in-memory writeback cache. Finally, default enable the new journal
writethrough until flush option.

Performance results comparing no journal, pre-change, and post-change:

no journal original modified
4K 40310.84 7223.73 27427.27
32K 21530.37 3256.39 10284.05
256K 3159.67 564.73 1627.16
4M 223.03 48.03 91.91

Fixes: http://tracker.ceph.com/issues/40072
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: new rbd_journal_object_writethrough_until_flush option

When set to true, the journal will not attempt to batch appends until
after it receives the the first flush request from the user.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: support dynamically updating recorder flush options

Default to disabling writeback-style append flushes unless overridden
by a call to 'set_append_batch_options'.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: fix broken append batching implementation

The original flush_interval/bytes/age batching options no longer
actually do anything. Integrate these better with the max in-flight
AIO handling.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

zstd: upgrade to v1.4.0

Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>

Merge pull request #28631 from cbodley/wip-17882

qa/rgw: extra s3tests tasks use rgw endpoint configuration

Reviewed-by: Kefu Chai <kchai@redhat.com>

debian/control: remove "libnl-3-dev" from build dependencies

what we need is "libnl-genl-3-dev", libnl-3-dev should be a dependency
of "libnl-genl-3-dev". the only reason we would have added it, is if
the package maintainer of "libnl-genl-3-dev" fails to do so. and
the packages of "libnl-genl-3-dev" on both debian and ubuntu just look
correct in this perspective.

so let's remove this.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: rewrite Findgenl to support components argument

* rename genl to nl:
  "genl" is not very specific without more context. and libnl is actually
  a library suite. so it would be better if we can rename the library to
  be found to "libnl", and by following the naming convention of CMake,
  in this change, Findgenl.cmake is renamed to Findnl.cmake
* Findnl.cmake: support the component argument, in our specific case,
  what we want is "libnl-genl" library, which is one of the libraries
  offered by libnl. so let's just make it a component.
* Findnl.cmake: should pass the names of the required variables instead of
  their values to `find_package_handle_standard_args()`. before this
  change, we pass `GENL_LIBRARIES` to this function. it was correct at the
  first glance. but it is not able to handle the case where case where
  libnl-genl is not installed. so the fix is to pass all the names of
  required library paths to this function. in this change, their name
  are concatenated to a single variable -- `nl_LIBRARIES`, and the
  value of this variable is passed to
  `find_package_handle_standard_args()`. and the error message would
  be more specific this way if libnl-genl is not found:
    Could NOT find nl (missing: nl_genl_LIBRARY)
* Findnl.cmake: add nl::<component> as imported library, it helps the
  consumer of these libraries to import them -- no need to
  specify the `target_include_directories()` separately anymore.
* move the find_package() code to where it is used. it helps to improve
  the readability.
* remove `HAVE_GENL` variable: it's not used anywhere.
* drop the messages of "Not using foobar", they do not help.

Signed-off-by: Kefu Chai <kchai@redhat.com>

journal: improve logging on recorder append path

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: Ensure E2E tests can be run independently (#28592)

mgr/dashboard: Ensure E2E tests can be run independently

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Nathan Weinberg <nweinber@redhat.com>
Reviewed-by: Rafael Quintero <rquinter@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge PR #28453 into master

* refs/pull/28453/head:
qa/valgrind.supp: be slightly less specific on suppression
msg/async, v2: make the reset_recv_state() unconditional.

Reviewed-by: Sage Weil <sage@redhat.com>

test_volume_client: don't shadow class variable

Name class and instance variables differently so that the latter doesn't
shadow the former.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #28566 from liewegas/wip-40377

osd: copy (don't move) pg list when sending beacon

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/rgw: update default port in perl workunits

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: extra s3tests tasks use rgw endpoint configuration

Signed-off-by: Casey Bodley <cbodley@redhat.com>

journal: properly advance read offset after skipping invalid range

Fixes: https://tracker.ceph.com/issues/40409
Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge PR #28212 into master

* refs/pull/28212/head:
mds: trim cache during standby replay

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #28551 into master

* refs/pull/28551/head:
mds: don't mark cap NEEDSNAPFLUSH if client has no pending capsnap

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #28545 into master

* refs/pull/28545/head:
client: set snapdir's link count to 1

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #28496 from ifed01/wip-ifed-fix-40080

os/bluestore: fix duplicate allocations in bmap allocator

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #28509 from tchaikov/wip-mon_clean_pg_upmaps_per_chunk

common/options: make mon_clean_pg_upmaps_per_chunk unsigned

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #28418 from xiexingguo/wip-kick-recovery-priority

osd: give recovery ops initialized by client op a higher priority

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #25395 from rzarzynski/wip-build-compiletest_cxx11_client

tests: introduce compiletest_cxx11_client for C++11 conformity.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #28229 into master

* refs/pull/28229/head:
os/bluestore: do garbage collection if blob count is too high.
common/perf_conters: make dump_formatted_xxx funcs as const.
os/bluestore: store extents for GC within WriteContext.
os/bluestore: GC class, make some members local.
os/bluestore: vector -> interval set in GC to track extents to collect.
tests/store_test: many-many spanning blobs test case

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #28630 into master

* refs/pull/28630/head:
os/bluestore: fix duplicative and misleading debug in KernelDevice::open().

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

mgr/dashboard: Ensure E2E tests can be run independently

Introduces the PageHelper.

- Reads class attributes from Helpers to reduce boilerplace code for
  navigation.
- The PageHelper is supposed to be the new class for code that's reused
  across all Helpers for page related code.
- The Helper class is by now meant to be used for non-page specific
  helper code, like used in the `checkConsole` method.
- The Helper class will act as central registry to enable all other
  tests to use helper functions of all derived PageHelper tests.

  Example:

  `Helper.pools.create('foobar', ...);`

Fixes: http://tracker.ceph.com/issues/40397
Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

os/bluestore: fix duplicative and misleading debug in KernelDevice::open().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mgr, osd: 'ceph osd df' by pool

Our test admin has been asking for this for the past few years:-)
Besides, this is also useful for operating on large Ceph clusters with
mutliple storage pools possibly spanning over all osds.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mgr/dashboard: Update translations (#28059)

mgr/dashboard: Update translations

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

qa/workunits/rbd: tweak mirror tests

(after "image mirror disable" started to disable journaling)

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: mirror image enable/disable should enable/disable journaling

Fixes: https://tracker.ceph.com/issues/40071
Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #28377 from p-na/wip-pna-message-xlf-opt

mgr/dashboard: Improve hints shown when message.xlf is invalid

Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Log's time from UTC to local (#27777)

mgr/dashboard: Log's time from UTC to local

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

osd/OSDMap: pass filter directly to the OSDUtilizationDumper

This makes the code easy to maintain and especially to extend,
e.g., adding per pool filters.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #28264 from rhcs-dashboard/mypy-config-to-file

src/script: extract mypy config to mypy.ini

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

osd: accident of rollforward may need to mark pglog dirty

refers: https://github.com/ceph/ceph/pull/27015/files#r294114392

Fixes: http://tracker.ceph.com/issues/40403
Signed-off-by: Zengran Zhang <zhangzengran@sangfor.com.cn>

rgw: fix doc compile warning caused by objectops.rst

Signed-off-by: zhang Shaowen <zhangshaowen@cmss.chinamobile.com>

Merge pull request #28605 from cbodley/wip-40304

qa/rgw: fix import error in tasks/swift.py

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

qa/rgw: fix import error in tasks/swift.py

> ImportError: No module named packaging

replaces packaging.version.Version with distutils.version.LooseVersion

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28095 from Devp00l/wip-39702

mgr/dashboard: OSD custom action button removal

Reviewed-by: Kanika Murarka <kmurarka@redhat.com>
Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #27670 from trociny/wip-rbd-mirror-fetch-autotune

 rbd-mirror: adjust journal fetch properties based on memory target

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #28524 from trociny/wip-qa-rbd-mirror-compare

qa/workunits/rbd: fix compare_images and compare_image_snapshots

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #28529 from trociny/wip-journal-in_flight_advance_sets

journal: wait for in flight advance sets on stopping recorder

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: Add QA tests for RBD configuration (#27774)

mgr/dashboard: Add QA tests for RBD configuration

Reviewed-by: Tatjana Dehler <tdehler@suse.com>

mgr/dashboard: Cleanup HACKING.rst

Rephrase explanations on how to write E2E tests with Protractor.

Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

Merge pull request #28586 from aclamk/fix-vstart-extra-conf

vstart.sh: Fix problem that all extra_conf got merged into single line.

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc:fix radosgw bucketops doc compile error

Signed-off-by: zhang Shaowen <zhangshaowen@cmss.chinamobile.com>

vstart.sh: Fix problem that all extra_conf got merged into single line.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

rgw: fix compile error in unittest:iam_policy

Signed-off-by: zhang Shaowen <zhangshaowen@cmss.chinamobile.com>

log: fix store_statfs log line

Signed-off-by: Mohamad Gebai <mgebai@suse.com>

rgw: fix miss get ret in STSService::storeARN

Fixes: http://tracker.ceph.com/issues/40386
Signed-off-by: Tianshan Qu <tianshan@xsky.com>

Merge pull request #28559 from dillaman/wip-40368

librbd: avoid dereferencing an empty container during deep-copy

Reviewed-by: Mykola Golub <mgolub@suse.com>

osdc/Objecter: _calc_target - inline spgid

A follow-up of https://github.com/ceph/ceph/pull/28487.
For erasure pools this can save us an obviously redundant call
of do_crush and other things.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #28487 from xiexingguo/wip-objecter-pg-mapping-cache

osdc/Objecter: pg-mapping cache

Reviewed-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: fix off-by-one when updating new_last_in_change

E.g., osdmap.osd_weight[osdmap.max_osd] does not exist.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #28082 into master

* refs/pull/28082/head:
mgr / volumes: improve error handling
py / cephfs: invoke base class Error::__init__() from OSError::__init__()
mgr / volumes: carve out subvolume operations as a separate class
mgr / volumes: introduce subvolume specification class
mgr / volumes: carve of volume as a separate class

osd: copy (don't move) pg list when sending beacon

If we move the list, then a second beacon sent before the pg list refreshes
will be empty.

Fixes: http://tracker.ceph.com/issues/40377
Signed-off-by: Sage Weil <sage@redhat.com>