Merge PR #35690 into master

* refs/pull/35690/head:
mds: fix hang issue when accessing a file under a lost parent directory

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #35848 into master

* refs/pull/35848/head:
mds: do not submit omap_rm_keys if the dir is the basedir of merge.

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #35951 into master

* refs/pull/35951/head:
qa/cephfs: set omit_sudo to False for wait_until_mounted() in...

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #34920 from ofriedma/ofriedma-civetweb-timeout

rgw: increase civetweb request timeout to 65 seconds

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #36055 from zli091/master

 rgw: lc: fix Segmentation Fault when the tag of the object was not f…

Reviewed-by: Or Friedmann <ofriedma@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

rgwlc: fix Segmentation Fault related to tags.

It is found that rgw crashes when putting a tag beyond the rule to the object, which can be solved by adding the return value judgment.

Fixes: https://tracker.ceph.com/issues/46485
Signed-off-by: zhuo li <lizhuo@chinatelecom.cn>
Signed-off-by: yupeng chen <chenyupeng@chinatelecom.cn>

Merge pull request #36008 from idryomov/wip-krbd-58-options

rbd: recognize crush_location, read_from_replica and compression_hint map options

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #36041 from tchaikov/wip-mds-EMetaBlob

ceph-object-corpus: pick up changes to drop backward compatibility of…

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-object-corpus: pick up changes to drop backward compatibility

to drop backward compatibility of EMetaBlob

see also https://github.com/ceph/ceph-object-corpus/pull/12

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35747 from adk3798/cephadm-44877

mgr/cephadm: allow custom dashboard grafana url

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge pull request #32852 from ifed01/wip-ifed-simplify-pin

os/bluestore: simplify Onode pin/unpin logic.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #35973 from mlausch/mgr-progress-perf

mgr/progress: Skip pg_summary update if _events dict is empty.

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #35915 from jmolmo/fix_rook_create_osd

mgr/rook: Fix error creating OSDs

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #35968 from optimistyzy/73_debug

NVMEDevice: Fix the issue of multiple OSDs when using SPDK

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32707 from petrutlucian94/windows.4

Windows support [part 4]

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35971 from tchaikov/wip-addr-parsing-strncpy

common/addr_parsing: refactor safe_cat()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #35990 from votdev/fix_cephadm_py_code

cephadm: Fix Python formating issues

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35717 from dmick/wip-cephadm-doc

doc/dev/cephadm.rst: clarify, fix spelling nits

Merge PR #35813 into master

* refs/pull/35813/head:
vstart.sh: use output of hostname for cephadm

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #36009 into master

* refs/pull/36009/head:
doc: Update nfs-ganesha package requirements

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #35930 from smithfarm/wip-luminous-eol

doc: remove luminous from active releases

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

qa/suites/krbd: turn on balanced reads for the fsx subsuite

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

test/librbd/fsx: respect rbd_default_map_options in krbd_open()

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

rbd: recognize compression_hint map option

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

rbd: recognize crush_location and read_from_replica map options

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #36010 from votdev/issue_46450_fix_stateful_tab_unit_test

mgr/dashboard: StatefulTabDirective unit test fails

Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #36003 from tserlin/private-tserlin-selinux-add-amqp-soundd-types

selinux: add amqp and soundd types to ceph.te

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: StatefulTabDirective unit test fails

Fixes: https://tracker.ceph.com/issues/46450
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #35967 from tspmelo/wip-language-selector

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Sebastian Krah <skrah@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

doc: Update nfs-ganesha package requirements

Fixes: https://tracker.ceph.com/issues/46449
Signed-off-by: Varsha Rao <varao@redhat.com>

Merge pull request #35939 from votdev/issue_46370_fix_edit_host_label

mgr/dashboard: Edit host label dialog is crashing

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #34946 from votdev/issue_43120_stateful_tabs

mgr/dashboard: stay on active tab

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mds: fix hang issue when accessing a file under a lost parent directory

When a directory metadata is lost, if we looked up a file under this lost directory,
this request will hang forever.

Fixes: https://tracker.ceph.com/issues/46129
Signed-off-by: Zhi Zhang <zhangz.david@outlook.com>

cephadm: Fix Python formating issues

Fix obvious Python formating errors reported by my IDE.

Signed-off-by: Volker Theile <vtheile@suse.com>

selinux: add amqp and soundd types to ceph.te

Signed-off-by: Thomas Serlin <tserlin@redhat.com>

Merge pull request #35904 from mkogan1/wip-fix-endpoint-not-connected-r2

rgw: fix boost::asio::async_write() does not return error...

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #35853 from wjwithagen/wjw-fix-rgw-to-base64

rgw: include string header

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Prateek Sachan <psachan@cs.iitr.ac.in>

Merge pull request #35735 from ivancich/wip-radoslist-incomplete-multiparts-marker-maintenance

rgw: radoslist incomplete multipart uploads fix marker progression

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #35525 from cbodley/wip-45969

rgw: stop realm reloader before store shutdown

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #35409 from pritha-srivastava/wip-iam-get-role-policy

rgw/iam: correcting the result of get role policy.

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #35189 from linuxbox2/wip-xxh-gc

rgw:gc: use XXHash for gc queue selection

Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

Merge PR #35999 into master

* refs/pull/35999/head:
doc/governance: add Patrick to CLTW
doc/governance: correct João's name

Reviewed-by: Neha Ojha <nojha@redhat.com>

doc/governance: add Patrick to CLTW

I guess I was missed!

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc/governance: correct João's name

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #35963 from ricardoasmarques/enable-iscsi-edit

mgr/dashboard: Allow to edit iSCSI target with active session

Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #35983 from ceph/kalebskeithley-1854083-1

selinux: allow ceph_t amqp_port_t:tcp_socket

Merge PR #34861 into master

* refs/pull/34861/head:
test: adjust scrub control tests for optional scrub status
mgr: set `task_dirty_status` on reconnect
mds: send scrub status to ceph-mgr only when scrub is running

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #35927 from gekios/wip-cephadm-image-fix

qa/tasks: fix cephadm config image check

Merge pull request #35919 from rzarzynski/wip-ec-clean-endecode_chunks

ec: drop stubs of {encode,decode}_chunks() from ErasureCode.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35465 from pritha-srivastava/wip-sts-token-fields-op-logs

rgw: adding user related web token claims to ops log

Merge pull request #35481 from pritha-srivastava/wip-rgw-keycloak-doc-correction

rgw/sts: Removing references to token introspection url,

Merge pull request #35858 from pritha-srivastava/wip-rgw-query-string

rgw: Normalizing X-Amz- headers for case in RGWHTTPArgs

test/store_test: add missing collection reopen

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: modify cache shutdown

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

mds: do not submit omap_rm_keys if the dir is the basedir of merge.

Fixes: https://tracker.ceph.com/issues/46273
Signed-off-by: Chencan <chen.can2@zte.com.cn>

qa/tasks: fix cephadm config image check

Check container_image_name only if ceph cluster image is not pre-defined in config.
We shouldn't care about container_image_name if there cephadm or ceph already have image defined.

Signed-off-by: Georgios Kyratsas <gkyratsas@suse.com>

rgw: fix boost::asio::async_write() does not return error...

although remote has closed the connection

Fixes: https://tracker.ceph.com/issues/46332
Signed-off-by: Mark Kogan <mkogan@redhat.com>

Merge pull request #34908 from tspmelo/wip-e2e-shared

mgr/dashboard: Add E2E tests for navigation, language and login

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

mgr/progress: Skip pg_summary update if _events dict is empty.

The get call of pg_stats and pg_ready are expensive and are not necessary in this case. This avoids blocking other modules like the prometheus module

Fixes: https://tracker.ceph.com/issues/46416
Signed-off-by: Manuel Lausch <manuel.lausch@1und1.de>

test: adjust scrub control tests for optional scrub status

Fixes: http://tracker.ceph.com/issues/45349
Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr: set `task_dirty_status` on reconnect

... so that upon reconnect the manager client updates manager
with whatever is available in ->task_status field.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mds: send scrub status to ceph-mgr only when scrub is running

Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge PR #35857 into master

* refs/pull/35857/head:
pybind/mgr/volumes: fix volume search for pools

Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>

Merge PR #35415 into master

* refs/pull/35415/head:
mds: make threshold for MDS_TRIM configurable

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

pybind/mgr/volumes: fix volume search for pools

Loop logic would bail out if it first sees any file system that does not
match the volume it's looking for.

Fixes: https://tracker.ceph.com/issues/46277
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #35868 into master

* refs/pull/35868/head:
mds: update codes of MSG_MDS_METRICS and MSG_MDS_PING

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #35410 into master

* refs/pull/35410/head:
client: release the client_lock before copying data in read

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #33827 into master

* refs/pull/33827/head:
mon: add 'ceph fs required_client_feature <fs_name> add/rm' command

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #33912 into master

* refs/pull/33912/head:
pybind: add special values for not reading conffile

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #35934 into master

* refs/pull/35934/head:
pybind/cephfs: raise custom Exception with same arguments
pybind/cephfs: add DiskQuotaExceeded exception

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Varsha Rao <varao@redhat.com>

Merge PR #35810 into master

* refs/pull/35810/head:
tool: convert str to num using strtoull

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #35361 into master

* refs/pull/35361/head:
mgr/volumes: Validate mon_allow_pool_delete before volume deletion

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #35979 from ivancich/wip-fix-orphan-list-teuthology

rgw: orphan list teuthology test & fully-qualified domain issue

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #35976 from idryomov/wip-make-check-msg

run-make-check.sh: fix the log message

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35964 from dillaman/wip-neorados-compat

neorados: bridge support to librados cluster connections

Reviewed-by: Adam Emersen <aemerson@redhat.com>

selinux: allow ceph_t amqp_port_t:tcp_socket

allow ceph_t amqp_port_t:tcp_socket name_connect;
allow ceph_t soundd_port_t:tcp_socket name_connect;

Required for running RabbitMQ

(soundd_port_t) for running RabbitMQ on port 8000

Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1854083
Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>

Merge pull request #35949 from zdover23/wip-doc-dev-guide-essentials-2020-07-07

doc/devel: repair sentences in "Essentials" page

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

common/addr_parsing: refactor safe_cat()

* add a ROUND_UP_128() macro for calculating the round of to a multiple of
  128.
* instead of using a loop, use ROUND_UP_128() for the new size of *pstr
* rename str2 to src for better readability
* use memcpy() instead of strncpy(), as the size of src string is already known

this change also silences the warning of

ceph/src/common/addr_parsing.c: In function ‘safe_cat’:
ceph/src/common/addr_parsing.c:45:3: warning: ‘strncpy’ output truncated before terminating nul copying as many bytes from a string as its length [-Wstringop-truncation]
strncpy((*pstr)+pos, str2, len2);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ceph/src/common/addr_parsing.c:28:14: note: length computed here
int len2 = strlen(str2);
^~~~~~~~~~~~

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35966 from ivancich/wip-fix-orphan-list-quoting

rgw: rgw-orphan-list -- fix interaction, quoting, and percentage calc

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

rgw: orphan list teuthology test & fully-qualified domain issue

Sometimes when teuthology machines are provisioned, the command
`hostname --fqdn` does not provide a fully qualified domain name but
instead just the hostname (e.g., smithi149 instead of
smithi149.front.sepia.ceph.com). This prevents the teuthology test for
rgw-orphan-list from running successfully [for example, the hostname
was for some reason mis-interpreted as the bucket name in the
request].

This commit checks whether the hostname derived from `hostname --fqdn`
contains any '.'s and if it does not, it will append
".front.sepia.ceph.com" to the hostname. This is a hack, but until
teuthology machines are configured appropriately it seems to be a
reasonable work-around.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

NVMEDevice: Fix the issue of multiple OSDs when using SPDK

For each OSD when using SPDK, it should be the type of primary
process. So when there are multiple NVMe SSDs, each OSD will use
one. So we do not want an OSD process claim all the NVMe SSDs,
so we use the allowed list to restrict the probe.

And with this patch, we can support multiple OSDs while using
SPDK.

Signed-off-by: Ziye Yang <ziye.yang@intel.com>

Merge pull request #35910 from galsalomon66/add_s3select_filter

adding filter for s3select

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #35286 from theanalyst/rgw/cleanup/override-umdhandler

rgw: user: RGWUserMetadataHandler mark remove method override

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #35911 from majianpeng/bluestore-check-written_slow-bytes

os/bluestore/bluefs: only bytes_written_slow > 0, we update perfcounter.

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #35908 from bk201/wip-46329

mgr/cephadm: fix Dashboard ganesha setting duplication error

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

run-make-check.sh: fix the log message

With 5e9a1d95c9f7 ("run-make-check.sh: extract run-make.sh"), this log
message is actively confusing.  At this point we just have the binaries,
no tests have run yet (and if -N is passed through CHECK_MAKEOPTS, none
will).

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #35907 from agayev/fix-zoned-device-detection

os/bluestore: Fix handling the return value of zbc_device_is_zoned() call.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #35825 from changchengx/watch_stress

test/stress_watch: remove unused variable

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35943 from tchaikov/wip-mgr-balancer

pybind/mgr/balancer: use "==" and "!=" for comparing str

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #35948 from tspmelo/wip-clean-jest

mgr/dashboard: Clean unit tests configuration and output

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #35922 from tspmelo/wip-clean-lint

mgr/dashboard: Clean the output of lint and fix scripts

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #35338 from myoungwon/wip-doc-manifet-snap

doc/dev/osd_internals/manifest.rst: add information about clone snap refcounting

Reviewed-by: Myoungwon Oh <myoungwon.oh@samsung.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

mgr/dashboard: Allow to edit iSCSI target with active session

Fixes: https://tracker.ceph.com/issues/46383
Signed-off-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #35874 from tspmelo/wip-fix-pwd-notification

mgr/dashboard: Hide password notification when expiration date is far

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #35501 from SchoolGuy/streamline-bootstrap-usage

mgr/dashboard: Introduce SASS modules

Reviewed-by: Sebastian Krah <skrah@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

common,msg: define SOCKOPT_VAL_TYPE

On Windows, the "optval" argument of the setsockopt function is
a char pointer, as opposed to a void pointer on other platforms.

In order to avoid compilation issues, we'll do an explicit cast
using a conditionally defined type.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

cls,rados,rbd,mds,common: Avoid name collision with Windows headers

There are a few name collisions between Windows headers and ceph.
Where possible, we'll update Ceph code to avoid redefinitions.

The following names overlap with Windows headers:
* ERROR, NO_ERROR
* DELETE
* LocalLock
* LOCK_SHARED, LOCK_EXCLUSIVE

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

msg: fix encode function

This commit [1] refactored the "encode" function but didn't handle
the variable names properly while moving code around, using the
undefined "wireaddr" variable.

[1] cf778097d6d4fdf4c178d0f6d1e1089d347ffc6d

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

common,msg,kv: Use Windows or portable function alternatives

Some functions are unavailable on Windows. We'll add some platform
checks, using either portable functions, either the Windows specific
ones:

* utimes -> utime
* strerror_r -> strerror_s
* poll -> WSAPoll
* sendmsg -> WSASend
* switch to portable time format specifiers
* fcntl -> ioctlsocket, when setting up non-blocking sockets
* /dev/urandom -> BCryptGenRandom
* sysconf(_SC_PAGESIZE) -> GetSystemInfo()
* define compat_mkdir, handling the fact that mkdir doesn't accept
  the mode argument on Windows

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

include,common: Add functions missing on Windows

The following functions are not avaialble on Windows, so we'll
need to define them.

* strptime
* pipe (defined as _pipe)
* lrand48
* posix_memalign
* pread, pwrite, fsync
* readv, writev

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>