rgw: beast frontend closes connections on stop

the strategy for stop relies on the fact that process_request() is
completely synchronous, so that io_context.stop() would still complete
each request and clean up properly

to tolerate an asynchronous process_request(), we instead need to drain
all outstanding work on the io_context so that io_context.run() can
return control natually to all of the worker threads. that would allow
us to suspend our coroutine in the middle of process_request(), and
still guarantee that process_request() will resume and run to completion
before the worker threads exit

each connected socket also counts as outstanding work, and needs to be
closed in order to drain the io_context. each connection now adds itself
to a connection list so that stop() can close its socket

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: beast frontend uses async SharedMutex for pause

the strategy for pause relied on stopping the io_context and waiting for
io_context.run() to return control to all of the worker threads. this
relies on the fact that process_request() is completely synchronous (so
considered a single unit of work in the io_context) - otherwise, pause
could complete in the middle of a call to process_request(), and destroy
the RGWRados instance while it's still in use

calling io_context.stop() to pause the worker threads also assumes that
no other work will be scheduled on these threads

to decouple pause from worker threads, handle_connection() now uses an
async shared mutex to synchronize with pause/unpause

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23634 from cbodley/wip-21154

rgw: RGWRadosGetOmapKeysCR takes result by shared_ptr

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: RGWRadosGetOmapKeysCR takes result by shared_ptr

Fixes: http://tracker.ceph.com/issues/21154
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23920 from cbodley/wip-rgw-cr-rados-fixes

rgw multisite: async rados requests don't access coroutine memory

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #23959 from rubenk/doc-remove-unknown-option-from-manpage

doc: remove deprecated 'scrubq' from ceph(8)

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23931 from cyx1231st/wip-msgr-test

tests: fix to check server_conn in MessengerTest.NameAddrTest

Reviewed-by: Kefu Chai <kchai@redhat.com>

tests: fix to check server_conn in MessengerTest.NameAddrTest

Signed-off-by: Yingxin <yingxin.cheng@intel.com>

Merge pull request #23944 from ceph/wip-s3a-update-mirror

qa/tasks: update mirror link for maven

Merge pull request #23963 from alfredodeza/wip-rm35535

ceph-volume:  batch tests for mixed-type of devices

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume lvm.batch use 'ceph' as the cluster name with filestore

Custom cluster names are currently broken on ceph-volume, should get
addressed with http://tracker.ceph.com/issues/27210 which is out of
scope for these changes

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests/functional update filestore xenial test vars

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests/functional update bluestore xenial test vars

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests/functional update filestore centos7 test vars

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests/functional update bluestore centos7 test vars

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.api use double -f flags when calling pvremove

Fairly destructive, just like everything else when zapping a device.
This is required in the case of double UUIDs detected, something that
surfaced when testing with a loop device to create an nvme (the loop
device ends up with the same UUID as the nvme).

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests/functional remove duplicate test playbooks

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests/functional setup playbook for nvme devices

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests/functional single (test) playbook to zap and redeploy batch operations

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests/functional create a noop playbook to allow symlinking always to setup.yml

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests/functional add mixed-type scenarios to tox.ini

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge PR #23673 into master

* refs/pull/23673/head:
qa: automate distro/kernel matrix for kclient

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Tested-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-volume tests/functional add CentOS7 bluestore mixed-type scenarios

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests/functional add CentOS7 filestore mixed-type scenarios

Signed-off-by: Alfredo Deza <adeza@redhat.com>

qa: automate distro/kernel matrix for kclient

It's no longer necessary to pass `-k testing` to teuthology-suite. We're also
now regularly testing RHEL 7.5 kernel in upstream testing.

This work is prep for eventually integrating kclient into fs.

Fixes: http://tracker.ceph.com/issues/26995
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc: remove deprecated 'scrubq' from ceph(8)

The option was removed in a30cbe9adea34782be8699c010d4281d245feb6b

Fixes: http://tracker.ceph.com/issues/35813
Signed-off-by: Ruben Kerkhof <ruben@rubenkerkhof.com>

Merge pull request #23846 from liewegas/wip-mgr-config-fixes

mon: fix mgr module config option handling

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23836 from sepia-liu/wip-cephfuse-doc

doc/cephfs: fixup add/remove mds docs

Merge pull request #23911 from votdev/cleanup

mgr/dashboard: Remove useless code

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #23948 from libingyang-zte/master

doc: Fix Spelling Error of Radosgw

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #23947 from tchaikov/wip-test-msgr-include

test/msgr: add missing #include

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #23926 from Gangbiao/lgb-devel

doc: Document correction

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

doc: Fix Spelling Error of Radosgw

Signed-off-by: Li Bingyang <li.bingyang1@zte.com.cn>

Merge pull request #23850 from liewegas/wip-22544

osdc/Objecter: fix split vs reconnect race

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #23557 from liewegas/wip-cmd-getval

common/cmdparse: cmd_getval_throws -> cmd_getval

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/msgr: add missing #include

it's a regression introduced by #23930

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks: update mirror link for maven, the original mirror no longer exists

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge PR #23851 into master

* refs/pull/23851/head:
mds: print is_laggy message once

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #23890 into master

* refs/pull/23890/head:
msg: lower verbosity on normal event

Reviewed-by: Sage Weil <sage@redhat.com>

mds: print is_laggy message once

Beacon::is_laggy gets called frequently which causes the debug log to get
spammed with messages.

Steps to reproduce:
- setup vstart cluster
- start IO from client (e.g. `cp -av /usr /mnt`)
- find connection to mon by the mds
  $ ss -tnp | grep ceph-mds
    ...
    ESTAB      0      0      127.0.0.1:58988              127.0.0.1:40928               users:(("ceph-mds",pid=13095,fd=37))
- block the connection using iptables on the MDS node:
  $ sudo iptables -I INPUT -p tcp --sport 58988 --dport 40928 -j DROP
- verify MDS log using:
  $ tail -f mds.a.log | grep beacon
    ...
    2018-09-05 19:21:05.672 7f2908a9d700  1 mds.beacon.a is_laggy 19.8876 > 15 since last acked beacon
    2018-09-05 19:21:05.672 7f2908a9d700  1 mds.beacon.a is_laggy 19.8876 > 15 since last acked beacon
    2018-09-05 19:21:05.672 7f2908a9d700  1 mds.beacon.a is_laggy 19.8876 > 15 since last acked beacon
    2018-09-05 19:21:05.672 7f2908a9d700  1 mds.beacon.a is_laggy 19.8876 > 15 since last acked beacon
    2018-09-05 19:21:05.784 7f2905a97700  5 mds.beacon.a Sending beacon up:active seq 114
    2018-09-05 19:21:05.784 7f2905a97700  1 -- 127.0.0.1:6813/2277776624 --> 127.0.0.1:40929/0 -- mdsbeacon(14162/a up:active seq 114 v7) v7 -- 0x7d1e7e2a80 con 0
    2018-09-05 19:21:05.784 7f2905a97700 20 mds.beacon.a sender thread waiting interval 4s

Fixes: http://tracker.ceph.com/issues/35250
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23645 from jcsp/wip-mgr-unknown-error

mgr: replace "Unknown error" string on always_on

Reviewed-by: Noah Watkins <nwatkins@redhat.com>

Merge pull request #23942 from trociny/wip-rados-rmpool

qa/workunits/rbd: replace usage of 'rados rmpool'

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits/rbd: replace usage of 'rados rmpool'

This command was dropped.

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge PR #23907 into master

* refs/pull/23907/head:
doc: Fix Spelling Error of Cephfs Shell

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #23930 from tchaikov/wip-include-cleanup

common,mon: add implicit `#include` headers

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23909 from theanalyst/doc/releases/12.2.8

doc: release notes for 12.2.8 luminous

doc: release notes for 12.2.8 luminous

- Add a special note for upgrades from earlier releases
- Add a notable section mentioning the scrub and rgw options

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #23852 from tchaikov/wip-cmake-cleanup

cmake: link compressor plugins against lib the modern way

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23925 from cfanz/wip-fix-data-sync-report

rgw-admin: fix data sync report for master zone

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23626 from Liuchang0812/rgw-put-cors-compatible

rgw: return err_malformed_xml when MaxAgeSeconds is an invalid integer

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23835 from Songweibin/wip-import-34536

rbd: fix error import when the input is a pipe

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #23815 from shun-s/fix-acquire-lock-latency

librbd: fix possible unnecessary latency when requeue request

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #23935 from LenzGr/bug-35686-missing-tooltip

mgr/dashboard: Added missing tooltip to settings icon

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #23409 from ricardoasmarques/wip-role-management

mgr/dashboard: Role management from the UI

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #23938 from dillaman/wip-qa-rbd-mkpool

qa/workunits/rbd: replace usage of 'rados mkpool'

Reviewed-by: Mykola Golub <mgolub@suse.com>

qa/workunits/rbd: replace usage of 'rados mkpool'

This command was dropped under commit 2c26fb0fe1, so use
'ceph osd pool create' instead.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: Role management from the UI

Fixes: https://tracker.ceph.com/issues/24447
Signed-off-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #23937 from tchaikov/wip-crimson-test-denc

test/crimson: do not use unit.cc as the driver of unittest_seastar_denc

Reviewed-by: Casey Bodley <cbodley@redhat.com>

test/crimson: do not use unit.cc as the driver of unittest_seastar_denc

as unit.cc initializes the CephContext and all of Ceph's infratructure,
which is not necessary for the denc test. also, seastar's builtin allocator
only pre-allocates 32 << 20 bytes. it's enough for the denc test, but
not necessarily enough for create CephContext and its friends. an option is
to use seastar's app template to initialize the memory allocator properly.
another option is to avoid initializing CephContext in this test.

the latter is simpler.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Added missing tooltip to settings icon

Fixes: https://tracker.ceph.com/issues/35686
Signed-off-by: Lenz Grimmer <lgrimmer@suse.com>

Merge pull request #23797 from votdev/fix_user_notifications

mgr/dashboard: Fix notifications in user list and form

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

mon/MonClient.h: include necessary header

MonClient references LockPolicy, so it should include
common/lock_policy.h .

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/TrackedOp.h: include spinlock.h as this header uses it

Signed-off-by: Kefu Chai <kchai@redhat.com>

common,crush,mds,messages,osd: add missing #include

they rely on dout.h to do this, but they should not.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23816 from cyx1231st/wip-seastar-msgr-fix

crimson/net: miscellaneous fixes to seastar-msgr

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: Fix notifications in user list

The notification message title and text were inverted.

Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #23912 from tchaikov/wip-install-deps-openldap

install-deps: s/openldap-client/openldap24-client/

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

doc: Document correction
    "doc/ceph-volume/lvm/prepare.rst":  Remove duplicated words
    "doc/radosgw/cloud-sync-module.rst": Remove duplicated words

Signed-off-by: Gangbiao Liu <liugangbiao@cmss.chinamobile.com>

rgw-admin: fix data sync report for master zone

Signed-off-by: cfanz <songxinying@sensetime.com>

Merge pull request #23918 from dzafman/wip-fix-assert

osd: Change assert() to ceph_assert() missed in the transition

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #23759 from cbodley/wip-rgw-curl-timeout

rgw: raise default rgw_curl_low_speed_time to 300 seconds

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23751 from joke-lee/wip-copy-response

rgw: etag in rgw copy result response body rather in header

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23749 from joke-lee/wip-etag-copy-part

rgw: fix Etag error in multipart copy response

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23646 from ashitakasam/get_cors_err_code

rgw: the error code returned by rgw is different from amz s3 when getting cors

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23590 from yaozongyou/fix-rgw-deadlock

rgw: fix deadlock on RGWIndexCompletionManager::stop

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWAsyncPutSystemObjAttrs does not access coroutine memory

also exposes optional objv_tracker to RGWSimpleRadosWriteAttrsCR

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWAsyncPutSystemObj does not access coroutine memory

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWAsyncGetSystemObj does not access coroutine memory

now reads the buffer/attrs into local memory, and calling coroutines
copy out the result during request_complete()

Fixes: http://tracker.ceph.com/issues/35543
Signed-off-by: Casey Bodley <cbodley@redhat.com>

osd: Change assert() to ceph_assert() missed in the transition

Caused by: c789fd1899f697a994a28e45a6f9262c6721d77d

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #23917 from Gangbiao/master

doc: Fix Spelling Error In File "ceph.rst"

msg: lower verbosity on normal event

Fixes: http://tracker.ceph.com/issues/35251
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23827 from trociny/wip-mock-valgrind

 test/librbd: fix valgrind warnings

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc: Fix Spelling Error In File "ceph.rst" and "rbd.rst"

Signed-off-by: Gangbiao Liu <liugangbiao@cmss.chinamobile.com>

Merge pull request #23898 from tchaikov/wip-crimson-net-cleanup

crimson/net: s/repeat/keep_doing/

Reviewed-by: Casey Bodley <cbodley@redhat.com>

crimson/net: s/repeat/keep_doing/

repeat ~= while() {}
keep_doing ~= for (;;) {}

so it's simpler this way.

Signed-off-by: Kefu Chai <kchai@redhat.com>

install-deps: s/openldap-client/openldap24-client/

please refer:

https://svnweb.freebsd.org/ports/head/net/openldap24-client/
https://svnweb.freebsd.org/ports/head/net/openldap-client/

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23842 from votdev/issue_34315

mgr/dashboard: Display RGW user/bucket quota max size in human readable form

Reviewed-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: Remove useless code

Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #23905 from idryomov/wip-cram-suite-repo

qa/tasks/cram: use suite_repo repository for all cram jobs

Reviewed-by: Nathan Cutler <ncutler@suse.com>

crimson/net: only do zero-copy writes to out_stream

seastar doesn't support mixed buffered writes and zero-copy writes.

Signed-off-by: Yingxin <yingxin.cheng@intel.com>

crimson/net: init h.global_seq during handshake

Signed-off-by: Yingxin <yingxin.cheng@intel.com>

crimson/net: remove unused and dup global_seq

Signed-off-by: Yingxin <yingxin.cheng@intel.com>

crimson/net: use parameter tag in handle_connect_reply()

Signed-off-by: Yingxin <yingxin.cheng@intel.com>

crimson/net: support multiple call to conn.close()

It is possible while closing a connectin during a msgr shutdown,
a second close() can be called in `read_tags_until_next_message()`

Signed-off-by: Yingxin <yingxin.cheng@intel.com>

Merge pull request #23491 from p-na/per-osd-settings

mgr/dashboard: Add support for managing individual OSD settings in the backend

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

doc: Fix Spelling Error of Cephfs Shell

Signed-off-by: Li Bingyang <li.bingyang1@zte.com.cn>

Merge pull request #23735 from votdev/suggest_snapshot_name

mgr/dashboard: Auto-create a name for RBD image snapshots

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #23627 from Devp00l/wip-hints-for-jest-failures

mgr/dashboard: Add hints to resolve unit test failures

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #23903 from libingyang-zte/master

doc: Fix Spelling Error of Rados Operations

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

doc: Fix Spelling Error of Rados Operations

Signed-off-by: Li Bingyang <li.bingyang1@zte.com.cn>