rgw: rest_client: work towards throttling of http read requests

Adjust the interfaces to provide the ability for the read callback
to pause the reads. While doing that, define a new class interface
for this instead of RGWGetDataCB. This had a butterfly effect that
required modifications to the obj read filters, but the end result
is a bit cleaner.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: cr rest splice, work towards write throttling

Need to throttle writes, so that we don't just accumulate all data read
from source endpoint in memory, in the case where the write endpoint is
too slow.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: cr: introduce io channels

ios can have multiple channels, so that we can differentiate between
waiting on read/write/control events. We can then block on a specific,
any, or multiple events.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: aws sync: configurable multipart threshold, part size

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: aws sync conf, consolidate configurables

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: http req, guard out_headers

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: aws sync: check that source object doesn't change

Make suret that while syncing the object it doesn't change, which can
be a problem when uploading the object piecemeal.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: aws sync, store temp per-object sync info

When doing a multipart object sync, need to store the object's info
so that we can either continue if upload is interrupted, and abort
older upload_id in case object changed.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: aws sync: multipart upload complete

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: don't schedule a cr if already scheduled

This can be triggered when multiple IOs return for the same cr. We
don't want to allow it to be scheduled multiple times, as this
might wake up blocking IOs that shouldn't be woken up. If we try to
block on an IO that we already complete, we won't block anyway.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: aws sync, initial work on multipart data parts creation

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix io blocked accounting

Now that we can have multiple IOs per coroutine, need to only decrease
num_blocked one time when switching from io blocked to unblocked. Also,
is_blocked() should not return true if is_done.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: aws sync, more work on large object sync via multipart upload

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: aws sync module, fix target object naming

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: aws sync module, fix target object naming

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix rest read crf handling of extra data

also fix splice coroutine

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: recover from empty meta.history object

Fixes: http://tracker.ceph.com/issues/21802
Empty meta.history object is problematic, as when reading it when doing
meta log trimming, we'd just initialize it to an empty structure, and
end up asserting because cursor is empty. Cannot just return -ENOENT
here, because it wouldn't solve the issue.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: aws sync, add hooks for decoding/encoding rest obj

Object sync is now functional. Create abstraction that will later help
with different cloud providers.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: aws sync, in_crf init abstraction

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: aws sync: generate target bucket

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: a bit cleaner interfaces for sending put crf

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: first go at sync to aws splice

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: RGWRESTConn() don't override provided key

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more streaming crf abstraction

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: separte stream crfs for read and write

also add a base class

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: splice cr drains writes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rest cr drains writes, returns errors

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: groundwork for supporting concurrent IOs in single cr

Add io_id field to io operation, allow blocking on a specific IO.
Locking changes in RGWCoroutinesManager::run().

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: initial work for integrating streaming read/write with cr

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: amend http client manager interface

s/set_threaded/start
s/is_threaded/is_started

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: http client, simplify interfaces

work towards removal of duplicate synchronous api

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: create global http manager

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: restructure http/rest client class hierarchy

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: tweak http client interface

RGWHTTPClient now holds the url and method.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add api to finish streaming writes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: http client, streaming writes api

Add a new class that enables streaming writes, that is --
doesn't require having all the data before the start of the
send.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fixes following rebase

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: typo fix during rebase (squash)

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw_rest_client: clarify comment on set content length

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

aws: remove comment

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

aws: use slashes in obj name

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw_sync_module_aws: make conn a unique ptr

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw_sync_module_aws: use the readrawrestresource cr

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw_cr_rest: have a raw variant of read resource

Useful when we don't have a json formatted input to send and just
sending raw data

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

sync_module_aws: minor logging improvements

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

set_content_length for streamrw requests

Also remove the earlier content length hack
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: aws sync: make bucket name more url friendly

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw_aws_sync: untested code for bucket create

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: sync_module_aws implement rm object

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

aws: ws fix

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

delete-me: log path for requests

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw_cr_rest: send content length on raw requests

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw_aws_module: lets call put object and see what happens

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw_cr_rest: allow sending of raw put requests

Modify SendRESTResourceCR to handle when we don't have a json formatted
input class and send raw bufferlists instead.

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: aws sync module: set keys for rest client

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: RGWRestConn allow keys to be initialized

This allows for other s3 clients to be created, we previously only set
the value of S3 system key from rados.

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

well more hacks to get my previous hack working

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

allow cr http client to deal with non json types

I'm quite crossed about this!

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

non working download client

basically readrestresourcecr has enforced a rule where only Json types
can be parsed :/

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: sync_module: register aws module

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: make RGWSyncModule classes for AWS

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: sync aws module compiles

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: sync plugin for aws (the humble beginnings!)

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #19283 from yehudasa/wip-rgw-mfa

rgw: mfa support

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #21327 from LenzGr/dashboard-fork-awesome

mgr/dashboard: Replace font-awesome with fork-awesome

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile vtheile@suse.com

Merge PR #21080 into master

* refs/pull/21080/head:
src: Added const references to various function parameters

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #21126 from tspmelo/wip-services

mgr/dashboard: Add Api module

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Replace font-awesome with fork-awesome

Replaces the dependency on Font Awesome with Fork Awesome
(https://forkawesome.github.io/), which is a plug-in replacement
for font-awesome 4.7.0 and adds a few more icons (see
https://forkawesome.github.io/Fork-Awesome/whats-new/ for
background information about this fork).

It is licensed under the following licenses - see
https://forkawesome.github.io/Fork-Awesome/license/ for details:

Fonts: SIL OFL 1.1
Code: MIT License
Documentation: CC BY 3.0

Signed-off-by: Lenz Grimmer <lgrimmer@suse.com>

Merge pull request #21319 from majianpeng/doc-remove-osd-op-threads

doc: remove unused config: "osd op threads"

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21112 from guzhongyan/fix-vlog

src: fix various log messages

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21028 from rzarzynski/wip-osd-get_rw_locks-norefcnt

osd: avoid unnecessary ref-counting across PrimaryLogPG::get_rw_locks.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21235 from xiexingguo/wip-fix-obj-start

osd/osd_types: fix ideal lower bound object-id of pg

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21313 from ceph/wip-pg-clean-timeout

qa/tasks: Add default timeout for wait for pg clean task

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #21284 from tchaikov/wip-cmake-libstdcxx

cmake: define HAVE_STDLIB_MAP_SPLICING for both libstdc++ and libc++

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #21290 from rjfd/wip-dashboard-fix-rest-controller

mgr/dashboard: restcontroller: fix detection of id args in element requests

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #21039 into master

* refs/pull/21039/head:
qa/workunits/fs: test for cephfs rstats
mds: make rstat.rctime follow inodes' ctime

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #21040 into master

* refs/pull/21040/head:
mds: mark damaged if sessions' preallocated inos don't match inotable

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #21132 into master

* refs/pull/21132/head:
client: remove getgroups_cb
client: remove _getgrouplist
client: have init_gids just set alloced_gids to true
client: remove init_groups
fuse: handle errors appropriately when getting group list

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #21144 into master

* refs/pull/21144/head:
mds: fix unhealth heartbeat during rejoin

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #21167 into master

* refs/pull/21167/head:
mds: make stopping mds export strays and subtree at the same time
mds: handle client requests while mds is stopping
mds: check projected linkage before migrating stray dentry

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #21191 into master

* refs/pull/21191/head:
client: fix request send_to_auth was never really used

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #21255 into master

* refs/pull/21255/head:
mds: automatically create snaprealm for old format quota
client: check if mds create snaprealm when setting quota
mds: remove unused MDCache::snaprealm_create()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #21295 from badone/wip-large-omap-scrub-big-random-loser-2

qa/suites/rados: Disable scrub backoff

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/tasks: wait_for_clean is called after ceph task as well after osd's are up,
the default timeout is none in that case, there are cases where it can hang  forever
due to error cases, since this dumps quite a lot of info the logs grow in GB's, with
default timeout of 1200 we can avoid such huge logs and fail sooner. Any tests needing
higher timeout can pass the required value.

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

doc: remove unused config: "osd op threads"

This option already removed.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #21309 from ceph/wip-yuriw-crontab

qa/tests - minor clean ups and made perf-suite run 3 times, so we can…

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #21282 from liewegas/wip-pgpool-enc

osd/osd_types: fix pg_pool_t encoding for hammer

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ilya Drymov <idryomov@redhat.com>

qa/tests - minor clean ups and made perf-suite run 3 times, so we can have more data points

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge PR #21263 into master

* refs/pull/21263/head:
doc: outline upgrade procedure for mds cluster

Reviewed-by: Douglas Fuller <dfuller@redhat.com>

mgr/dashboard: Move services provision to service module

Few services were being declared in the shared module.

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Add Api module

Moved all the api related services into the new module.

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #20807 from Devp00l/improve-table-search

mgr/dashboard: Improve table search

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #20943 from tspmelo/fix-e2e

mgr/dashboard: fix frontend e2e tests

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

common/options: add otp to default osd class load list

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: mfa documentation

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

qa/tasks/s3tests: configure mfa tests

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: require mfa only if mfa status actually changes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add distro dependencies for liboath

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fixes following rebase

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/otp: fixes following review

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>