rgw/coroutine: check for null stack on wakeup

Fixes: https://tracker.ceph.com/issues/56920
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #47690 from mty22/update-inactive-ceph-mirrors

doc: Remove inactive Ceph mirrors.

Merge pull request #45552 from wxypro/man-cephfs-mirror

doc/man/cephfs-mirror.rst: Update missing options for cephfs-mirror

Merge pull request #47727 from snosratiershad/docs/correct-radosgw-admin-doc-bucket-name

doc/man/8: Remove slash behind bucket name in radosgw-admin.rst

doc/man/8: Remove slash behind bucket name in radosgw-admin.rst

Signed-off-by: Salar Nosrati-Ershad <snosratiershad@gmail.com>

Merge pull request #47371 from rhcs-dashboard/mstart-configurable

mstart: make some mstart env variables configurable

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #47207 from oOHenry/rgw-dashboard-ssl-error

mgr/dashboard: fix rgw connect when using ssl

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: David Galloway <dgallowa@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #47720 from tchaikov/wip-mgr-dashboard-test-with-headers

qa/tasks/mgr/dashboard: accept headers as a param

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #47716 from MalteJ/patch-1

doc: virtio discard is supported since kernel 5.0

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

doc: virtio discard is supported since kernel 5.0

Signed-off-by: Malte Janduda <mail@janduda.net>

Merge pull request #46702 from rhcs-dashboard/expand-cluster-hide-maintenance

mgr/dashboard: Hide maintenance option on expand cluster

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
Reviewed-by: sunilangadi2 <NOT@FOUND>

qa/tasks/mgr/dashboard: accept headers as an option param

* add optional "headers" parameter to _request() and _get()
* correct the test of test_force_no_gzip, as the header in response
  should be "application/vnd.ceph.api.v1.0+json", not "application/json"

with this change, we are able to pass headers to _get() in dashboard api
tests.

without this change, we'd have following error when testing
`test_force_no_gzip` defined by `tasks.mgr.dashboard.test_requests.RequestsTest`:

raceback (most recent call last):
  File "/home/jenkins-build/build/workspace/ceph-api/src/pybind/mgr/dashboard/services/exception.py", line 47, in dashboard_exception_handler
    return handler(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/cherrypy/_cpdispatch.py", line 54, in __call__
    return self.callable(*self.args, **self.kwargs)
  File "/home/jenkins-build/build/workspace/ceph-api/src/pybind/mgr/dashboard/controllers/_base_controller.py", line 263, in inner
    ret = func(*args, **kwargs)
TypeError: Summary.__call__() got an unexpected keyword argument 'headers'

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

mgr/dashboard: add out_client dir to api and e2e checks

Signed-off-by: Nizamudeen A <nia@redhat.com>

vstart: isolate the client logs

Signed-off-by: Nizamudeen A <nia@redhat.com>

mstart: make some mstart env variables configurable

Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #47711 from tchaikov/wip-crimson-os-cleanups

crimson/os: cleanups

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #47715 from maryfrances01/wip-doc-punctuation-error

doc/radosgw: Fix punctuation and grammar in encryption.rst

doc: Fixes a punctuation and grammar issue

Merge pull request #47639 from ronen-fr/wip-rf-scrubmap-swap

osd: fix ScrubMap::swap() to include all ScrubMap data members
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc: Fixes a punctuation and grammar issue

Signed-off-by: Mary Frances Hull mhull@redhat.com

Merge pull request #44356 from SMIL-Infra/clog

common,mon: use fmt::print(cerr, ...) for stderr logging

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

crimson/os: use IsFullKey in *::estimate_insert()

to fade out KeyT, so we can have more straightforward definitions.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

crimson/os: use use IsFullKey in node_fields_2_t::insert_at()

to fade out KeyT, so we can have more straightforward definitions.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #47325 from CongMinYin/fix-owner-lock-ordering

librbd/cache/pwl: narrow the scope of m_lock in write_image_cache_state()

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

crimson/os: use IsFullKey in _node_fields_013_t::append_key()

to fade out KeyT, so we can have more straightforward definitions.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

crimson/os: avoid using KeyT in is_valid_key()

and we don't need to specify KeyT when instantiating this template
function.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

crimson/os: avoid using KeyT in ns_oid_view_t

so we don't need to use it when instantiating the template member
functions of this class.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #47628 from mkogan1/wip-fix-tcmalloc-link

rgw: fix linking with tcmalloc when specified by -DALLOCATOR

Reviewed-by: Casey Bodley <cbodley@redhat.com>

osd: modify two ScrubMap data members from fields to full variables

as there is no size advantage in this specific case.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd: fix ScrubMap::swap() to include all ScrubMap data members

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #47632 from linuxbox2/wip-rgw-53455

rgw/s3: add conditional json output for s3:ListBucket

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #46443 from ybwang0211/head_obj

rgw: The error reported in the "rgw_s3_prepare_decrypt" function is unreasonable

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #46547 from Huber-ming/check_appendable

rgw: check object appendable when check_disk_state

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #47228 from Huber-ming/multi_meta

rgw: 'bucket check' deletes index of multipart meta when its pending_map is nonempty

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #47547 from mdw-at-linuxbox/wip-master-xamzdate-1

rgw: x-amz-date change breaks certain cases of aws sig v4.

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #45036 from rishabh-d-dave/qa-fix-is_mounted

qa/cephfs: upgrade is_mounted() in mount.py

Reviewed-by: Kotresh HR <khiremat@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Nikhilkumar Shelke <nshelke@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #45869 from neesingh-rh/fix_55234

mgr/snap_schedule: replace .snap with the client configured snap dir name

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Nikhilkumar Shelke <nshelke@redhat.com>

Merge pull request #46749 from ibm-s390-cloud/fsmap_switch_to_iterator

fsmap: switch to using iterator based loop

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

Merge pull request #47691 from myoungwon/wip-cbjournal-integration

crimson/os/seastore: CBJournal integration works

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #47597 from NitzanMordhai/wip-nitzan-crimson-watch-list-empty-return-code

crimson/osd: watch list empty return incorrect

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Sam Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #47670 from rzarzynski/wip-common-ftbfs-systemd

common: fix a FTBFS in Journald.cc when building without systemd

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #47680 from tchaikov/wip-crimson-cleanups

crimson: cleanups to use `IsFullKey` concept

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: introduce finish_commit to update tails after trim transaction

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore/cbjournal: rename journal_tail to dirty_tail

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore/cbjournal: make cbjournal to store alloc_tail

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore/cbjournal: make journal_tail and written_to to use journal_seq_t instead of rbm_abs_addr

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore: divide the role of open_device_read_header into device.mount()
and open_for_mount()

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

doc: Remove inactive Ceph mirrors.

hk.ceph.com - notified mirror maintainer list of mirror decommission on 2020-08-31
us-east.ceph.com - mirror has been offline for years.

Signed-off-by: Matthew Taylor <matthew.taylor@digitalpacific.com.au>

Merge pull request #47615 from adamemerson/wip-objv-doc

rgw: Document `RGWOvjVersionTracker`

Reviewed-by: Casey Bodley <cbodley@redhat.com>

librbd/cache/pwl: generate image cache state json under m_lock

The previous commit moved the entirety of write_image_cache_state()
from under m_lock.  This was a step too far because the generated image
cache state json is no longer guaranteed to be consistent.

Arrange for m_lock to still be held during image cache json generation
but released before owner_lock is grabbed.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

crimson/os: make _node_fields_013_t::insert_at() use IsFullKey

less repeatings this way

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #47288 from batrick/i56274

debian: add libsqlite3-mod-ceph dependency for ceph-mgr

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

crimson/os: define *::from_key() with IsFullKey

more compact this way.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

crimson: move helper classes down

so we can use the concept of `IsFullKey` for defining accessors in the
incoming commit.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

crimson: define encode_key() with IsFullKey constrait

also, let the compiler to deduce the type of the template parameter,
less repeating and less error-prone this way.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

common: fix a FTBFS in Journald.cc when building without systemd

This came from Josh Salomon:

```
/home/jsalomon/src/prim_score/src/common/Journald.h:93:24: error: ‘SubsystemMap’ does not name a type
   93 |   JournaldLogger(const SubsystemMap *) {}
      |                        ^~~~~~~~~~~~
/home/jsalomon/src/prim_score/src/common/Journald.h:94:23: error: ‘Entry’ does not name a type
   94 |   int log_entry(const Entry &) {
      |
```

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

Merge pull request #47633 from bosc0/alerts_multicluster

prometheus: add multicluster support to alerts

Merge pull request #47554 from Matan-B/wip-matanb-c-rados-python

qa/suite/crimson-rados: Fix rados_python test

Reviewed by: Samuel Just <sjust@redhat.com>
Reviewed by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed by: Nitzan Mordechai <nmordech@redhat.com>

Merge pull request #47653 from Matan-B/wip-matanb-snapmapper-pending-typo

PendingReleaseNotes: fix typo in 15.2.17

Reviewed-by: Nitzan Mordechai <nmordech@redhat.com>

Merge pull request #47392 from Matan-B/wip-matanb-memstore-subsys

common: MemStore debug subsystem

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #47643 from aisakaki/wip-omap-merge

crimson/os/seastore/omap: optimize merge policy

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #47558 from rzarzynski/wip-crimson-extradbgs

crimson/{net,osd}: add debugs after bughunt's investigations

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge PR #47600 into main

* refs/pull/47600/head:
doc: document debugging for libcephsqlite

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #47523 from adk3798/oracle-linux-support

cephadm: support for Oracle Linux 8

Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Redouane Kachach <rkachach@redhat.com>

PendingReleaseNotes: fix typo in 15.2.17

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #47422 from adamemerson/wip-coroutine-cleaver

rgw: Break Full and Incremental sync out into their own classes

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/suites: Reduce rados_python time out

This test runs for few minutes,
reducing timeout from 3h to 1h to avoid hanging jobs.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

qa/crimson-rados: add 'wait' attribute

wait_for_complete tests currently hanging with crimson-osd
and are temporary disabled in Crimson suite.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

ceph-mixin: Remove jsonnet building

Signed-off-by: Aswin Toni <aswin.toni@cern.ch>

prometheus: add multicluster support to alerts

Signed-off-by: Aswin Toni <aswin.toni@cern.ch>

crimson/os/seastore/omap: optimize merge policy

in some corner cases on single key erase sence,
for e.g. we have three nodes which are brothers:
1 2 3 4 5 | 6 7 8 | 9 10 11, node min size is 3.
When erasing 11 will trigger balance instead of
merge, the middle nodes will adjust to 6 7, this
will not ensure the all nodes above min size.
This corner case might not be harmful but if we
want to erase 10,11 at one time (range remove wip
need this), this will trigger balance in past
policy which will make both two nodes below min
size and we might need at least one more merge.
This is actually because we cannot ensure a node
to merge in tree is above min size.

Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>

crimson/os/seastore/omap: code cleanup

Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>

Merge pull request #47491 from cyx1231st/wip-seastore-fix-return-future

crimson/os/seastore/transaction_manager: return future when enoent

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #46855 from ronen-fr/wip-rf-pg-updates

osd/scrub: Scrubbing PGs now periodically update their stats

Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Aishwarya Mathuria <amathuri@redhat.com>

Merge pull request #47553 from cyx1231st/wip-seastore-cleanup-reclaim

crimson/os/seastore: cleanups to gc reclaim

Reviewed-by: Myoungwon Oh <myoungwon.oh@samsung.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>
Reviewed-by: Zhang Song <zhangsong325@gmail.com>

Merge pull request #47477 from mchangir/doc-clarify-start-time-in-snap-schedule

doc/cephfs: note regarding start time time zone

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

Merge pull request #47099 from joscollin/wip-B56537-wrong-rsp-wsp-values

cephfs-top: fix the rsp/wsp display

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Neeraj Pratap Singh <neesingh@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

rgw/s3: add conditional json output for s3:ListBucket

Adapt the XML ListBucket response to contain well-formed JSON,
matching the implied schema of awscli:

$ aws --endpoint-url http://localhost:8000 s3api list-objects --bucket testbucket

suggested by Casey in review.

Fixes: https://tracker.ceph.com/issues/53455
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #47365 from adk3798/explicit-host-no-schedule

mgr/cephadm: fix handling of draining hosts with explicit placement specs

Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #47529 from hookak/container-args

cephadm: Remove duplicated process args in promtail and loki

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #47545 from adk3798/grafana-password

mgr/cephadm: set dashboard grafana-api-password when user provides one

Reviewed-by: Michael Fritch <mfritch@suse.com>

rgw/s3: align output format with client request

In general, e.g., in the S3 ListBucket response, the
output Formatter is already of the type requested in the HTTP
Accept header.

Therefore, while there are still many instances where the of
Formatter is assumed to be XML, it appears necessarily safe
and seemingly correct to extend this assumption to end_header(...),
which sends the response content-type.

Fixes: https://tracker.ceph.com/issues/55680
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw/rest: replace RGW_FORMAT_ macros with an enum class

This is a pure cleanup.  The method to print an RGWFormat object
as a MIME type is now called to_mime_type().

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw/iam: thread DoutPrefixProvider into policy evaluation

Also conditionally prints policy and resource in verify_bucket_permission.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw: fix link dependencies when not building dbstore

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

osd/scrub: PGs that are scrubbing now update stats periodically

Added periodic calls to all PGs for which the OSD is the primary,
asking for a scrub statistics update.

This allows operator queries (e.g. 'pg dump pgs') to present
up-to-date scrub duration, "scrub is blocked" duration, etc'.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #47565 from adk3798/unit-test-runtime

mgr/cephadm: mock CephadmHttpServer run in unit tests

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Redouane Kachach <rkachach@redhat.com>

doc/cephfs: note regarding start time time zone

Fixes: https://tracker.ceph.com/issues/56730
Signed-off-by: Milind Changire <mchangir@redhat.com>

Merge pull request #47119 from rhcs-dashboard/image-refs-bug

mgr/dashboard: fix _rbd_image_refs caching

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #47609 from cbodley/wip-install-deps-rhel

install-deps: remove outdated rhel8 devtool packages

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

rgw: fix linking with tcmalloc when specified by -DALLOCATOR

Fixes: https://tracker.ceph.com/issues/57140
Signed-off-by: Mark Kogan <mkogan@redhat.com>

Merge pull request #45628 from PepperJo/comparev_and_writev_v7

librbd: add compare-and-write compare length limit, remove sector size restriction

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #47617 from tchaikov/wip-cmake-jsonnet

cmake: set $PATH for tests using jsonnet tools

Reviewed-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@cern.ch>

Merge pull request #47618 from xxhdx1985126/wip-journal-move

crimson/os/seastore/journal: move bufferlists to reduce the amount of memory copies

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/journal: move bufferlists to reduce the amount of memory copies

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

cmake: set $PATH for tests using jsonnet tools

otherwise they would not able to find executables installed into
${CMAKE_CURRENT_BINARY_DIR}.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #47594 from tchaikov/wip-cmake-win32

cmake: enforce gcc-10 for win32 port

Reviewed-by: Casey Bodley <cbodley@redhat.com>

cephadm: Remove duplicated process args in promtail and loki(config.file)

Signed-off-by: jinhong.kim <jinhong.kim0@navercorp.com>

rgw: Document `RGWOvjVersionTracker`

Give some guidelines as to what it's for and how to use it.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #47612 from cbodley/wip-rgw-revert-lua-filter

Revert "rgw/lua: allow read access to object data"

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>