rgw: implement s3 encoding-type for get bucket

This change introduces handling for the encoding-type request
parameter on the get bucket operation. An object key may contain
characters which are not supported in XML. Passing the value "url" for
the encoding-type parameter will cause the key to be urlencoded in the
response.

Fixes: #12735
Signed-off-by: Jeff Weber <jweber@cofront.net>

Merge pull request #4744 from ceph/wip-11625

librbd: diff_iterate should issue concurrent ops

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #5555 from dachary/wip-mon-test-timeouts

tests: be more generous with mon tests timeouts

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5576 from liewegas/wip-dencoder

simplify handling for objects w/ nondeterministic encoding

Reviewed-by: John Spray <john.spray@redhat.com>

test/encoding/check-generated: test sorted json dumps for nondeterministic objects

Nondeterministic objects dump nondeterministically (usually due
to unordered_map or _set).  Compare their sorted output.  This
is a weaker test but is better than nothing.

Signed-off-by: Sage Weil <sage@redhat.com>

Revert "osd/HitSet: make subclasses dump deterministically"

This reverts commit 34c048b3cb2b16658e692ac5110115612031510d.

Revert "mon/PGMap: dump osd_epochs in deterministic order"

This reverts commit 3df6438cd2c9a77a06b7a8d011bafbed00f1fc5e.

Merge pull request #4864 from stevecapperlinaro/for-ceph/journal-page-size-fix

FileJournal: Remove CEPH_PAGE_SIZE assumptions

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5100 from stevecapperlinaro/for-ceph/tweak-buffer-append-size

Common: Do not use CEPH_PAGE_SIZE when appending buffers in Ceph

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5339 from xinxinsh/wip-use-throttle

use throttle framework to throttle objectstore ops/bytes

Reviewed-by: Li Wang <liwang@ubuntukylin.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5573 from SUSE/wip-restart-on-rpm-removal-only

ceph.spec.in: test %preun argument is zero for removal-only operations

ceph.spec.in: test %preun argument is zero for removal-only operations

The %preun section now contains logic for disabling and stopping all the
Ceph systemd units when the ceph package is removed. However, there is no
conditional around it, so the units are disabled and stopped on RPM upgrade
as well as removal.

http://tracker.ceph.com/issues/12685 Fixes: #12685

Signed-off-by: Nathan Cutler <ncutler@suse.com>
Signed-off-by: Tim Serong <tserong@suse.com>

Merge pull request #5562 from varadakari/wip-kvstore-statfs

Adding statfs api to KeyValueDB

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5565 from varadakari/wip-kvstore-makefile

Remove redundant line from Makefile

Reviewed-by: Loic Dachary <ldachary@redhat.com>

tests: do not test timeout mon add

Strictly speaking it's not a feature, but an expected hahaviour. After
discussion it does not seem useful to have that test.

Signed-off-by: Kefu Chai <kchai@redhat.com>
Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge remote-tracking branch 'origin/next'

Merge pull request #5564 from wonzhq/test-inode

mds: initialize 'divergent' to false when comparing inode_t

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #5553 from majianpeng/mds

mds (fix to flags_to_mode and cleanup in handle_client_open)

Reviewed-by: John Spray <john.spray@redhat.com>

mds/Server: clean up code for handle_client_open.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

common/ceph_fs: set mode default is -1 for func ceph_flags_to_mode.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Remove redundant line from Makefile

Signed-off-by: Varada Kari <varada.kari@sandisk.com>

mds: initialize 'divergent' to false when comparing inode_t

'divergent' is not set in the case when all the fields of 'inode_t' are
equal.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

Adding statfs api to KeyValueDB

If any backend supports and implements statfs will extract the
stats from backend to show correct utilization in the status.

Signed-off-by: Varada Kari <varada.kari@sandisk.com>

Merge pull request #5544 from ceph/wip-12664-next

librbd: prevent race condition between resize requests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

tests: be more generous with mon tests timeouts

Change the timeouts of the TEST_mon_add_to_single_mon tests to be 120
seconds. They have value to guard against blocking forever. The exact
timing does not matter as long as the operation completes.

When a timeout is too short, it will create false negatives when running
on slow machines. It could be argued that being too generous with
timeout in general may hide problems. But not in this specific case.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5550 from schuzhoy/schuzhoy-patch-1

doc: Correcting the default number of copies.

Reviewed-by: Travis Rhoden <trhoden@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5072 from zhouyuan/doc_rgw_rados_handlers

doc: update rgw configuration on multiple rgw rados handlers feature

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5495 from tchaikov/wip-12620

doc/rados/operations/add-or-rm-mons: simplify the steps to add a mon

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5528 from Be-El/master

Correct handling of supplementary groups in ceph-fuse (Bug 12617)

doc: update rgw configuration on multiple rgw rados handlers feature

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

Doc: Correcting the default number of copies.

According to the "quick ceph deploy" page, the default number of replicas in the Ceph configuration file is 3

Signed-off-by: Sasha Chuzhoy <sashac88@hotmail.com>

Merge pull request #5047 from rzarzynski/wip-12108

rgw: rework X-Trans-Id header to be conform with Swift API.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #5449 from yangruifeng/wip-rgw-bugfix1

rgw:segmentation fault when rgw_gc_max_objs > HASH_PRIME

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #5447 from yangruifeng/master

rgw:the arguments 'domain' should not be assigned when return false

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #5543 from ceph/wip-cmake-nlist

CMake: add missing librados nlist test

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5547 from ritz303/wip-doc-11082

doc: Removed reference to RAID-4

Reviewed-by: John Wilkins <jowilkin@redhat.com>

doc: Removed reference to RAID-4

Fixes: #11082
Signed-off-by: ritz303 <ritz_303@yahoo.com>

Merge pull request #5546 from ritz303/wip-doc-11897

doc: Clarify how 'ceph auth caps' works.

Reviewed-by: John Wilkins <jowilkin@redhat.com>

CMake: cut down unnecessary linkage on rados tests

Signed-off-by: John Spray <john.spray@redhat.com>

CMake: add missing librados nlist test

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #5354 from ceph/wip-pgls-filter-librados

librados: partially expose PGLS filter interface

Reviewed-by: David Zafman <dzafman@redhat.com>

doc/rados/operations/add-or-rm-mons: simplify the steps to add a mon

this change removes the step to "ceph mon add" before starting a new
monitor. because the existing leader will start an election at seeing
the MMonJoin message sent by the new joiner, after the quorum is
archieved, the monmap will be updated with the new monitor.
so, "ceph mon add" is not necessary to add a new monitor.
moreover, this command will be blocked until a new quorum is formed,
and the proposed monmap is accepted. but in case of adding a monitor
to a single monitor cluster, the leader will wait until at least two
of the monitors reply to it. apparently, this does not happen unless
the new monitor starts. so from the user's point of view, this
command hangs until timesout, if he/she does not start the mon.b
beforehand. but this is an expected behaviour.

so, to avoid this confusion and simplify the steps to add a new
monitor. we'd better simply remove this "ceph mon add" step.

Fixes: #12620
Signed-off-by: Kefu Chai <kchai@redhat.com>

librbd: prevent race condition between resize requests

It was possible that the same resize request could be sent twice
if a completed resize op started a newly created resize op while
it was also being concurrently started by another thread.

Fixes: #12664
Backport: hammer
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #5535 from ceph/wip-gentest

mon: Improve PGMap::generate_test_instances(), build map per incremental

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5457 from ceph/wip-cmake-vstart

Make vstart work with cmake out of tree builds

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge remote-tracking branch 'upstream/master'

test: add case for pgls filter arg in librados

Signed-off-by: John Spray <john.spray@redhat.com>

librados: partially expose PGLS filter interface

Not bothering with this for non-namespaced object iterator
or the C interface.  Expect to replace this with
the new-style pgls, so just exposing it enough for
cephfs-data-scan to touch the new osd-side pgls filtering
on forward scrub tags.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #5542 from l11625/master

The listen_sd of thread also can be zero when we call the processor::start function..

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #5533 from ceph/wip-fix-mon-join.joao

mon: Monitor: set MMonJoin's op type as service

Reviewed-by: Kefu Chai <kchai@redhat.com>

when the processor start ,listen_sd also can be zero.

Merge pull request #5480 from ceph/wip-12617

client: ignore permission check when fuse_default_permissions is on

test/mon: add test for "mon add"

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5536 from ceph/wip-12662-next

WorkQueue: add/remove_work_queue methods now thread safe

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

WorkQueue: add/remove_work_queue methods now thread safe

These methods were not acquiring the ThreadPool lock when
manipulating the work_queue collection.  This was causing
occasional crashes within librbd when opening and closing
images.

Fixes: #12662
Backport: hammer, firefly
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #5344 from trhoden/update_debain_maintainer

debian: Update maintainers and uploaders

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #5525 from ceph/wip-12611-next

bit_vector extent calculation incorrect for last page

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Fixed mistaken reference to mon in osd section. Changed release name to hammer.

Fixes: #7807
Signed-off-by: John Wilkins <jowilkin@redhat.com>

doc: Clarify how 'ceph auth caps' works.

Fixes: #11897
Signed-off-by: ritz303 <ritz_303@yahoo.com>

mon: Improve PGMap::generate_test_instances(), build map per incremental

Signed-off-by: David Zafman <dzafman@redhat.com>

mon: Monitor: set MMonJoin's op type as service

Otherwise, when the MonmapMonitor's PaxosService::dispatch() handles the
message, it will assert (as it should) because the op is marked as being
a monitor op. And this function should never handle monitor ops.

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

added permission check based on getgrouplist

rbd: export-diff now issues concurrent AIO read requests

Combined with the improvements to the diff_iterate API, this should
noticeably increase the speed of a large image export-diff.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: verify that diff_iterate callbacks can abort diff

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: diff_iterate should handle callback errors

If the diff_iterate callback returns an error, it should
result in diff_iterate being aborted with the error code
propagated.

Fixes: #11593
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: execute multiple object diffs in parallel

Issue up to 'concurrent_management_ops' list_snap ops to the
OSDs concurrently.  The iterate callbacks will continue to be
serialized -- clients that wish higher throughput should avoid
blocking to maximum performance.

Fixes: #11625
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: move diff_iterate logic to its own class

In preparation for parallelizing the diff_iterate logic,
move all existing logic to its own class.  No functional
changes have been made.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: increase test coverage for partial encodes/decodes

Multiple combinations of offsets/lengths are now tested when
performing partial encodes/decodes of the bit vector.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

common: bit_vector extent calculation incorrect for last page

It's highly probable that the last page in the bit vector will not
be a full page size.  As a result, the computed extents will extend
beyond the data portion of the bit vector, resulting in a end_of_buffer
exception.

Fixes: #12611
Backport: hammer
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

added autoconf check for getgrouplist

Merge pull request #5353 from ceph/wip-pgls-filter-osd

Make PGLSFilter interface more flexible (for cephfs)

Reviewed-by: Sage Weil <sage@redhat.com>

vstart: detect CEPH_BIN in stop.sh too

...for the benefit of rbd.

Signed-off-by: John Spray <john.spray@redhat.com>

ceph.in: detect paths in out of tree build

a la what we currently do for PYTHONPATH
and LD_LIBRARY_PATH, but for cmake out
of tree builds.

Signed-off-by: John Spray <john.spray@redhat.com>

vstart: detect and handle cmake environ

Signed-off-by: John Spray <john.spray@redhat.com>

vstart: enable more path customization

...for working with out-of-tree builds.

Signed-off-by: John Spray <john.spray@redhat.com>

Makefile: build ceph.in the cmake way

A little bit more of a tongue-twisting command line
to do string replacement, but saves us from having
separate ceph.in files for cmake vs autotools.

Signed-off-by: John Spray <john.spray@redhat.com>

ceph.in: use cmake-style substitution

Avoid need for separate ceph.in and ceph.in.cmake files.

Signed-off-by: John Spray <john.spray@redhat.com>

init-ceph.in: set executable bit

Scripts expect the generated init-ceph script
to be +x, and CMake does that if the file you
feed into it is +x.  This matches what we already
do with ceph.in.

Signed-off-by: John Spray <john.spray@redhat.com>

CMake: build dencoder at src/ceph-dencoder

...instead of in src/test/dencoder.

While the source lives in test/, the dencoder tool is built
and distributed as a first class binary, so it's awkward for
paths in dev/test to have it in a unique location.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #5522 from oritwas/wip-cmake-fix-mds

cmake: fix mds compilation and link

Reviewed-by: John Spray <john.spray@redhat.com>

cmake: fix mds compilation and link

Rename MDS.cc MDSDaemon.cc
Add missing files and match order to the Makefile

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

Merge branch 'wip-vumrao-12111' of https://github.com/vumrao/ceph

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rbd: creating refused, if feature is specified when the format is equal to 1

Signed-off-by: x11562 <xie.na@h3c.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #5520 from badone/master

doc: remove duplicate word in Motivation section.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4927 from wonzhq/snapset-obc

osd/ReplicatedPG: snapset is not persisted

Merge pull request #5212 from ceph/wip-mds-refactor

mds: big refactor of MDS into MDSDaemon, MDSRank, MDSRank Dispatcher

doc: remove duplicate word in Motivation section.

Remove to\nto typo in doc/dev/osd_internals/erasure_coding/pgbackend.rst

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #5517 from tchaikov/wip-cmake-remove-flat-index

cmake: remove Flat_index stuff

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5515 from ceph/wip-make-check-fixes

several fixes for make check

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #4683 from ceph/wip-siddharth-security-fixes

Enable security hardening flags globally

Testedj-by: Kefu Chai <kchai@redhat.com>

cmake: remove Flat_index stuff

the Flat_index stuff was removed in 5df688a

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5504 from tchaikov/wip-check-good-yasm

cmake: check for good yasm

Removing unwanted -fPIC in cflags

Removing unwanted -grecord-gcc-switches , relro in cflags and adding -fPIC

Enable security hardening flags globally

Merge pull request #5293 from majianpeng/oprequest-markevent

osd: Oprequest markevent

Reviewed-by: Huang Jun <hjwsm1989@gmail.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5267 from dachary/wip-12356-rados-osds-out-in

tests: test-erasure-code.sh/rados_osds_out_in must wait_for_clean

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4909 from zhouyuan/rgw_doc_bucket_index

doc: update rgw configuration doc on bucket index sharing feature

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

test/encoding/check-generated: make error msg cut and pasteable

Signed-off-by: Sage Weil <sage@redhat.com>

osd/HitSet: make subclasses dump deterministically

Signed-off-by: Sage Weil <sage@redhat.com>