rgw: fix import

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test/rgw: drop use of urllib

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: get_rest_filter() delete original rest manager

When overriding rest manager, delete original.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: check init_sync return code

fix following review

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: minor fixes following review

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: es: fix system user check

check was inverted

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: package radosgw-es

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: system users override elasticsearch permission filter

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test_multi: don't pass array as default param to constructor

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test/rgw/test_multi: cleanup

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: minor cleanup

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: adjust log levels

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: use RGW_AMZ_META_PREFIX

instead of defining X_AMZ_META_PREFIX

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test_multi: realm checkpoint after init

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw/test_multi: add test_es_bucket_conf test

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw/test-multi: test more complicated queries

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: don't send raw date header to elasticsearch

parse the header, and encode it in the json doc using
a format that ES can understand. Skip header if fails to
parse.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw/test_multi: add tests for different key types

add int and date tests

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw/test_multi: zone_conn can hold more than one bucket per zone

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw/test_multi/es: extend test

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test/rgw/test_multi: initial es functional tests

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test/rgw/test_multi: differentiate between zone and zone connection

Instead of having a Zone type used for the connection, create a new
ZoneConn type that represents the connection. This frees us from the
need to pass in credentials all around.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw/test_multi: add support for elasticsearch testing

Add support for different zone types, and create an elasticsearch
zone type that deals with es testing.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: es: index and return versioned epoch

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move data sync instance_id initialization to caller

so that caller can easily know the instance id.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: don't pass sync module to rest filter in creation

sync module instance might change due to reconfiguration. Pass it into the
handler when executing op.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: zone create handles --tier-config

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: es: use "null" as instance in case key.instance is empty

when indexing objects. We can get either.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: es: generic keys are not case sensitive

make map case insensitive, and explicitly define all generic keys
so that they can be forced to be lower case.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix time format that is sent to elasticsearch

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add missing custom meta map initialization

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: es: configurable number of replicas and shards

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: es: add a configurable to override index path

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: es: elasticsearch index path unique per sync instance

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: es: configurable list of buckets and owners to index

and a tier configuration that allows to select both list of buckets
and list of bucket owners (if both configured then will intersect
the selection).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: create helper for configurable item selection

with optional prefix and suffix.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add "explicit_custom_meta" configurable to es sync module

Modify all the map<string, string> that is used to pass in module config
to map<string, string, ltstr> so that it can be used with the conf get
vals util. Also, switch to using shared_ptr to hold the ElasticConfig,
so that later when we hold maps and such it doesn't need to be copied
on every call.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: expose conf_get(), conf_get_int(), conf_get_bool()

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

common/ceph_json: map type can handle 2 or 3 template arguments

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: es: handle get config api in es zone

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: es: api to delete bucket mdsearch config

DELETE /bucket?mdsearch

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: es: api to retrieve bucket mdsearch config

GET /bucket?mdsearch

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: es: also return custom int and date results

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: es: limit indexing to only configured user meta

only metadata keys that are configured on user's bucket are allowed
to be indexed. Also, use the type of meta that was configured.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: new api to configure bucket's custom keys for mdsearch

POST /bucket?mdsearch
x-rgw-meta-search: <x-amz-meta-key>[;<str|int|date>][,...]

note that x-amz-meta-search header can also be used.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: guard restricted fields in es query

don't allow users to use the 'permissions' field.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add dout defs

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: support field aliases in es compiler

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: support more meaningful compilation error string

in es module

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: support for generic entities types in es

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: initial support for multiple types in es requests

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: metadata search pagination

Leveraging elasticsearch "size" and "from" for doing pagination. This
is not optimal, but sadly it's currently the way to go as elasticsearch
does not have an efficient way to retrieve ordered entries (other than
using stateful api). This should be revisited in the future. A preferable
way would be to hold a unique identifier for each object doc, and have
the query sort by that unique identifier (that could be used as marker).
At the moment sorting by anything would require elasticseatch to load
all data into memory.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add permissions and bucket to es query

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: also dump custom meta entries

when doing metadata search

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: send request to elasticsearch and parse response

and send back response to user

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: extend interfaces to allow sending data

when retrieving REST resource

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move code around

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: can send data in RGWRESTStreamRWRequest::send_request()

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: make key param in RGWRESTStreamRWRequest::send_request() optional

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rename a few methods

just rename calls

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: meta search rest handler can access es module

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: work on REST handler for es module

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rename rgw_rest_es.cc to rgw_es_query.cc

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: simplify es compile interface

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move code into class

just cleaning up

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: handle nested fields in es queries

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: implement init_sync() callback in es module

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: define es index mapping

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add init callback to sync modules

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: initial implementation of mdsearch query compiler

convert infix queries that look as follows:

[(]<name> <operator> <value> [)] [<and|or> ...]

into a prefix structure that is understood by elasticsearch.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #15184 from cbodley/wip-qa-rgw-cleanup

qa/rgw: remove apache/fastcgi and radosgw-agent tests

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #13851 from zhangsw/optimize-datasync

rgw: optimize data sync. Add zones_trace in log to avoid needless sync.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15368 from dzafman/wip-20089

osd: When scrub finds an attr error mark shard inconsistent

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15350 from mrbojangles3/master

libradosstriper: Add example code

Merge pull request #15356 from joscollin/wip-test-msgr-warning

test/msgr: silence warnings from -Wsign-compare

Merge pull request #15362 from andihit/doc-rados-xattr-commands

doc: add rados xattr commands to manpage

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15366 from joscollin/wip-cleanup-test-remove-commented-code

test/old: Removed commented code

Reviewed-by: Sage Weil <sage@redhat.com>

osd: When scrub finds an attr error mark shard inconsistent

Fixes: http://tracker.ceph.com/issues/20089
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #14971 from SUSE/wip-mgr-py-sub-interpreter

mgr: load modules in separate python sub-interpreters

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15259 from zealoussnow/wip-0524

build: remove ceph-disk-udev entirely

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Loic Dachary <loic@dachary.org>

Merge pull request #15285 from smithfarm/wip-kill-fcgi

build/ops: drop libfcgi build dependency

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/old: Removed commented code

There were lot of commented code inside src/test/old/testcrush.cc. This was found while doing
the PR#15364. Removed them.

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #15359 from liewegas/wip-kraken-x

qa/suites/upgrade/kraken-x: enable experimental for bluestore

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: add rados xattr commands to manpage

Signed-off-by: Andreas Gerstmayr <andreas.gerstmayr@catalysts.cc>

Merge pull request #15306 from mikulely/fix-test_multi

rgw: fix test_multi.py default config file path

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/suites/upgrade/kraken-x: enable experimental for bluestore

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15294 from liewegas/wip-osd-tree

osd: 'osd tree in|out|up|down' to filter tree results

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15225 from smithfarm/wip-s390-rgw

build/ops: conditionalize rgw Beast frontend so it isn't built on s390x architecture

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mon: add up|down|in|out filters to 'osd tree'

For example, 'ceph osd tree down' will show *just* down OSDs and their
ancestors.  \o/

Signed-off-by: Sage Weil <sage@redhat.com>

cmake: link radosgw with boost libraries

Signed-off-by: Nathan Cutler <ncutler@suse.com>

test/msgr: silence warnings from -Wsign-compare

The following warning appears during make:

ceph/src/test/msgr/test_msgr.cc: In member function ‘virtual void MessengerTest_MarkdownTest_Test::TestBody()’:
ceph/src/test/msgr/test_msgr.cc:1474:46: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
     CHECK_AND_WAIT_TRUE(srv_dispatcher.count > last + 1);
                         ~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~
ceph/src/test/msgr/test_msgr.cc:54:9: note: in definition of macro ‘CHECK_AND_WAIT_TRUE’
     if (expr)                           \
         ^~~~
ceph/src/test/msgr/test_msgr.cc:1475:30: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
     if (srv_dispatcher.count == last) {
         ~~~~~~~~~~~~~~~~~~~~~^~~~~~~

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #15355 from joscollin/wip-cleanup-testdenc-warning

test: test_denc.cc: silence warning from -Wsign-compare

Reviewed-by: Kefu Chai <kchai@redhat.com>

test: test_denc.cc: silence warning from -Wsign-compare

The following warning appears during make:
ceph/src/test/test_denc.cc:669:5:   required from here
ceph/src/googletest/googletest/include/gtest/gtest.h:1530:28: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
ceph/src/googletest/googletest/include/gtest/gtest.h:1510:7:
   if (val1 op val2) {\

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #15058 from yuyuyu101/wip-rocksdb-opt

kv/rocksdb: supports SliceParts interface

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15068 from liewegas/wip-osdmap-require

osd/OSDMap: replace require_*_osds flags with a single require_osd_release field

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15313 from Linbing/wip-pg-scrub-code-optimize

mon: it's no need to get pg action_primary osd twice in pg scrub

Reviewed-by: Kefu Chai <kchai@redhat.com>

mon/OSDMonitor: fix osd metadata update ordering

Do not do both.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: replace all require_*_osds flag checks with a require_osd_release comparison

Signed-off-by: Sage Weil <sage@redhat.com>

osd: replace require_*_osds flags with require_osd_release field

- OSDMap encode and decode translate between the flags and int
representations.
- OSDMap::Incremental only does decode; we do not expect to ever encode
an incremental osdmap for an old osd that sets any of these flags.
- the 'osd set' command still lets you set the jewel and kraken flags,
but not luminous.
- OSDMap::apply_incremental handles the conversion of legacy require flags
to the new field if the jewel or kraken flags have to be set before
starting the osd upgrade.
- clear out the legacy flags when we make the luminous transition only;
until then we keep using the old flag in the encoded and decoded version
(although the require_osd_release field will be accurate in memory in all
cases).

Signed-off-by: Sage Weil <sage@redhat.com>

mon: new 'osd require-osd-release' command

Note that we only allow setting this to 'luminous'.  If the 'jewel' or 'kraken'
flags need to be set that can be done through the legacy interface.

Signed-off-by: Sage Weil <sage@redhat.com>