qa: add new test case for pulling error

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Add checking of num_shards_repaired in osd stats

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: automatically repair replicated replica on pulling error

However this is not a very complete solution since the broken object
info may still get lost if we switch primaries or simply power off nodes.
I think a better idea would be also adding these kind of broken objects
back into replica's own missing set simultaneously, e.g., like we handling
primary reading errors.

But for now I am not sure if that should be a concern?

Fixes: http://tracker.ceph.com/issues/39101
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #26970 into master

* refs/pull/26970/head:
mds: initialize cap_revoke_eviction_timeout with conf

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #26992 into master

* refs/pull/26992/head:
mount.ceph.c: do not pass nofail to the kernel

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #27087 into master

* refs/pull/27087/head:
mds: remove duplicated check on balance amount

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #27063 into master

* refs/pull/27063/head:
mds: there is an assertion when calling Beacon::shutdown()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #27294 from tspmelo/wip-nfs-squash

mgr/dashboard: Fix NFS squash default value

Reviewed-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #25353 from cbodley/wip-rgw-sysobj-yield

rgw: add optional_yield to SysObj service interfaces

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #25355 from cbodley/wip-rgw-http-yield

rgw: add optional_yield to http client interface

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #27276 from cbodley/wip-39033

rgw multisite: data sync checks empty next_marker for datalog

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #27244 from tspmelo/wip-nfsv3-prop

mgr/dashboard: Fix deletion of NFS protocol properties

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #27130 from cbodley/wip-38700

rgw: decrypt filter does not cross multipart boundaries

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

mgr/dashboard: Fix NFS squash default value

Fixes: http://tracker.ceph.com/issues/39064
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge PR #27206 into master

* refs/pull/27206/head:
osd: shutdown recovery_request_timer earlier

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27074 from LenzGr/master-documentation

doc: Updated dashboard iSCSI configuration, added labels

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #27164 from Devp00l/issue-38937

mgr/dashboard: Back button component

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #27269 from rhcs-dashboard/wip-39030-master

mgr/dashboard: update vstart to use new ssl port

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #27203 from falcon78921/wip-doc-38721

doc: removed OpenStack Kilo references in Keystone docs

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #27264 from rouming/fio-ceph-messenger--dummy-auth-for-v2

fio_ceph_messenger: catch up v2 proto changes by using dummy auth

Reviewed-by: Ricardo Dias <rdias@suse.com>

doc: Updated dashboard iSCSI configuration, added labels

Added note about the requirement for the latest ceph-iscsi version
3 to the dashboard documentation. Added some doc references
and replaced some URLs in the iSCSI docs with reST labels instead.

Signed-off-by: Lenz Grimmer <lgrimmer@suse.com>

Merge pull request #27205 from xiexingguo/wip-38784

osd/PG: introduce all_missing_unfound helper

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #27182 from xiexingguo/wip-38931

osd/PG: move down peers out from peer_purged

Reviewed-by: Sage Weil <sage@redhat.com>

osd/PG: move down peers out from peer_purged

In purge_strays(), we'll aggressively clear stray_set and
add all related peers into peer_purged.

However, if the corrsponding peer is down and becomes
up again, (unconditionally) adding it to peer_purged
will prevent primary from re-purging it.
(See Active::react(const MNotifyRec& notevt))

On consuming a new osdmap, let's move any down peers out from
peer_purged simutaneously. This way we can lower the risk
of leaving any leftover PGs behind.

Related-to: http://tracker.ceph.com/issues/38931
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #27249 into master

* refs/pull/27249/head:
qa/suites/rados/thrash-old-clients: fix up v1/v2 options

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #27133 into master

* refs/pull/27133/head:
log,global: do not start flusher thread until after we have our mon config
log: buffer log entries until flusher thread starts
log: open log file from flusher thread
common/ceph_context: fix log_to_file observer

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #26976 into master

* refs/pull/26976/head:
doc: add doc for cephfs lazyio

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #27199 into master

* refs/pull/27199/head:
mds: fix dump routine for session_info_t.used_inos
mds: fix some misleading log messages
client: comment fix in _lookup
mds: fix comment over MDRequestImpl
mds: use stdbool.h instead of hand-rolling our own bool type

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

mgr/dashboard: udpate vstart to use new ssl port

Fixes: https://tracker.ceph.com/issues/39030
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #27242 from theanalyst/beast-fe-ports

rgw: beast port parsing

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: beast: set a default port for endpoints

if no port is specified we don't set one(or set to 0), and don't exactly error
out, so rgw keeps running without being bound to a port. Set the port to default
in these cases

Fixes: http://tracker.ceph.com/issues/39000
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: callers pass null_yield to RGWSI_SysObj

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add optional_yield to RGWSI_Notify

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add optional_yield to RGWSI_SysObj

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add optional_yield to RGWSI_SysObj_Cache

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add optional_yield to RGWSI_SysObj_Core interface

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWSI_RADOS::Obj::notify() takes optional_yield

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add rgw_rados_notify() with optional_yield

Signed-off-by: Casey Bodley <cbodley@redhat.com>

librados/asio: add async_notify() to wrap aio_notify()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: Fix deletion of NFS protocol properties

Fixes: http://tracker.ceph.com/issues/38997
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #27213 from ajarr/fix-38946

pybind: fix use of WriteOpCtx and ReadOpCtx

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw: data sync checks empty next_marker for datalog

RGWReadRemoteDataLogShardCR tracks the marker and next_marker
separately, because next_marker will be empty when it reaches the end

this allows RGWDataSyncShardCR to avoid clearing its sync_marker and
restarting datalog listing from the beginning

Fixes: http://tracker.ceph.com/issues/39033
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27157 from Devp00l/issue-38933

mgr/dashboard: FixtureHelper

Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #27247 from Exotelis/polish-translation

mgr/dashboard: Add polish translation

Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: FixtureHelper

Fixes: https://tracker.ceph.com/issues/38933
Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Update messages.xlf

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Back button component

Fixes: https://tracker.ceph.com/issues/38937
Signed-off-by: Stephan Müller <smueller@suse.com>

test/fio/fio_ceph_messenger: catch up v2 proto changes by using dummy auth

This change makes v2 protocol work.

Signed-off-by: Roman Penyaev <rpenyaev@suse.de>

rgw: RGWHTTPClient::wait() warns if it blocks in an asio thread

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: http interfaces take optional_yield

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27148 from tchaikov/ceph-volume-terminal-unicode

ceph-volume: terminal: encode unicode when writing to stdout

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #27252 from yuriw/wip-yuriw-crontab

qa/tests: added nautilus-x run

Merge pull request #27250 from ivancich/wip-update-resharding-docs

rgw: updates to resharding documentation

Reviewed-by: Adam Emerson <aemerson@redhat.com>

rgw: updates to resharding documentation

Some minor updates to the resharding documentation with the aim of
clarifying the language and using terms consistently.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

qa/tests: added nautilus-x run

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

qa/suites/rados/thrash-old-clients: fix up v1/v2 options

- don't specify ceph.py options in the ceph.py
- instead, specify them in the per-version facet

Note that we don't currently have a way to do v2 only for the mon IPs, so
in the v2only cases, we are still binding the mons to v1.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #27243 from theanalyst/doc-scheduler

config-ref: add a note on current scheduler settings.

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge PR #27202 into master

* refs/pull/27202/head:
doc: Miscellaneous spelling fixes

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #27154 into master

* refs/pull/27154/head:
mgr/pg_autoscaler: apply bias to pg_num selection
mgr/pg_autoscaler: include pg_autoscale_bias in autoscale-status table
osd/osd_types,mon: add pg_autoscale_bias pool property

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #27012 into master

* refs/pull/27012/head:
common/options: allow (but to not prefer or require) secure mode
common/options: make clients prefer to connect to mons via secure mode
common/options: allow connections to mons in secure mode
common/options: prefer secure mode between monitors
auth/AuthRegistry: remove experimental flag for 'secure' mode

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

docs: rgw: add a x-ref to rados dmclock docs

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

mgr/dashboard: Add polish translation

Add polish translation.
Removed 5 unused trans-units.

Fixes: https://tracker.ceph.com/issues/38236
Signed-off-by: Sebastian Krah <skrah@suse.com>

config-ref: add a note on current scheduler settings.

Adding a note on configuirables for max concurrent requests and the rest of
experimental options for tuning dmclock scheduler

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

mds: fix dump routine for session_info_t.used_inos

It says it's dumping used_inos, but it's really dumping prealloc_inos.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #26887 from theanalyst/reshard-tenant

rgw admin: add tenant argument to reshard cancel

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27222 from ceph/wip-doc-dahsboard

mgr/dashboard: fix hacking.rst

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #27233 from tchaikov/script/ceph-backport.sh

script/ceph-backport.sh: allow user to specify remote repo

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #27135 from ricardoasmarques/iscsi-filter-images-based-on-features

mgr/dashboard: Validate iSCSI images features

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

script/ceph-backport.sh: allow user to specify remote repo

before this change, the script always push to "origin", in my case,
"origin" points to "ceph/ceph", while we need to create the branch in
private repo. and the name of private repo might vary. so would be great
if we can customize this setting.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26914 from votdev/issue_38331

mgr/dashboard: Add separate option to config SSL port

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #27166 from rhcs-dashboard/add-protractor-screenshoter-plugin

mgr/dashboard: Add protractor-screenshoter-plugin

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #24678 from rouming/master

test/fio: introduce fio ioengine: fio_ceph_messenger

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>

script/ceph-backport.sh: fix sanity test of issue number

* no need for checking negative numbers
* use regex for matching the number instead of using bash's pattern
  matching, otherwise i have
./ceph-backport.sh: line 79: syntax error in conditional expression:
unexpected token `('
  the pattern matching works fine if i use it in command line, but
  it fails to work in shell script.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27200 from neha-ojha/wip-21174-2

osd/PGLog: preserve original_crt to check rollbackability

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #27218 from yuriw/wip-yuriw-crontab

qa/tests: added nautilus-p2p to cron

mgr/dashboard: fix hacking.rst

Add missing dash to `-prod` and specify full command.

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

rgw: fixup_range() clamps offsets to valid part range

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/tests: added nautilus-p2p to cron

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #27165 from yuriw/wip-yuriw-crontab-master_2

qa/tests: reduced number of jobs for `kcephfs`

Merge PR #27169 into master

* refs/pull/27169/head:
common/config: parse --default-$option as a default value

Reviewed-by: Sébastien Han <seb@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #27022 into master

* refs/pull/27022/head:
msg/async v2: make v2 work on rdma.

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge PR #27184 into master

* refs/pull/27184/head:
mgr/DaemonServer: handle_conf_change - fix broken locking

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #27187 into master

* refs/pull/27187/head:
mgr/dashboard/qa: Increase timeout for test_disable (tasks.mgr.dashboard.test_mgr_module.MgrModuleTelemetryTest)

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

ceph-volume: terminal: encode unicode when writing to stdout

python determins the encoding of stdout and stderr based on the LC_CTYPE
and PYTHONIOENCODING env variable, by default, python3's sys.stdout uses
'utf-8' as its encoding, so it will be able to write unicode string even
the stdout is not attached to a tty device. but when it comes to
python2, it will default to ascii if neither of these variabls is set.
so, if we are writing unicode using `_Write` in an environment where
LC_CTYPE and/or PYTHONIOENCODING are using non UTF-8 encoding, it chokes
by raising `UnicodeEncodeError` exception.

in this change, we add a wrapper around `_Write._writer` so it is able
to write unicode string in such a non-unicode-friendly environment.

for more info related the encoding of stdout and stderr, see
https://docs.python.org/3/using/cmdline.html#envvar-PYTHONIOENCODING .

Signed-off-by: Alfredo Deza <adeza@redhat.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #27196 into master

* refs/pull/27196/head:
mon/MonClient: do not dereference auth_supported.end()

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Add protractor-screenshoter-plugin

A plugin to save screenshots on E2E tests failures as well as
console logs, raw HTML, ...
All info visualized in an HTML report (screenshots are clickable to see full image).
Useful for debugging/fixing E2E tests.

Signed-off-by: Alfonso Martínez <almartin@redhat.com>

test_rgw_crypto: add invalid ranges beyond obj boundary

This is just to assert we're not doing an invalid memory access

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #27211 from tchaikov/wip-crimson-partition-args

crimson/osd: partition args the right way

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #27153 from Devp00l/issue-38932

mgr/dashboard: Fixes tooltip behavior

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

test/pybind: fix use of rados.ReadOpCtx and rados.WriteOpCtx

classes WriteOpCtx and ReadOpCtx don't take in any arguments
during instantiation

Signed-off-by: Ramana Raja <rraja@redhat.com>

ceph_volume_client: fix use of rados.WriteOpCtx

Class WriteOpCtx doesn't take in any arguments during instantiation.

Fixes: http://tracker.ceph.com/issues/38946
Signed-off-by: Ramana Raja <rraja@redhat.com>

Merge pull request #27159 from smithfarm/wip-38934

doc: move Developer Guide to its own subdirectory

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #26561 from Devp00l/issue-38418

mgr/dashboard: Make preventDefault work with 400 errors

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

crimson/osd: partition args the right way

before this change, all "unknown" args are considered `ceph_args`, but
there is chance we have command line like

crimson-osd -i 1 -c ceph.conf --smp 1 --cpuset 00000001

in this case, "1" is listed in `unknown_args`, but it is also in
`parsed.options`. so we will put the second "1" into `ceph_args` even it
should be appended to `app_args`.

so, in this change, `std::partition_copy()` is replaced with a
hand-written loop which advances through the `argv` and `unknown_args`,
try to match as many consecutive options are possible. it assumes that
the order of args in `unknown_args` is the same as their order in
`argv[]`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

tests: rgw crypto tests with partlen vector

4 tests are added when either obj_size for an aligned object, and when an object
is not aligned to a boundary (1 B last part), or part_size is not aligned to a
boundary. Comments on boundary case explains the logic on choosing whether to
align to a 4095B boundary or not

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #27190 from tchaikov/wip-ceph.in-asan-only-for-debug

 ceph.in: only preload asan library for Debug build

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #26813 from DengkeDu/master

cmake/FindRocksDB: fix IMPORTED_LOCATION for ROCKSDB_LIBRARIES

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27204 from xiexingguo/wip-clean-weird-brace

osd/PG: move '}' to the proper place

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

doc: add doc for cephfs lazyio

Fixes: http://tracker.ceph.com/issues/38729
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

cmake/FindRocksDB: fix IMPORTED_LOCATION for ROCKSDB_LIBRARIES

Signed-off-by: dudengke <pinganddu90@gmail.com>