rgw: make sending Content-Length in 204 and 304 controllable

This commit introduces a new configurable "rgw print prohibited
content length" to let operator decide whether RadosGW complies
to RFC 7230 (a part of the HTTP specification) or violates it
but follows the Swift's behavior.

Fixes: http://tracker.ceph.com/issues/16602
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
(cherry picked from commit d8e3e64ec97a3c222a56bb6f510e5e23d7858615)

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #13276 from rzarzynski/wip-rgw-18364-2-jewel

jewel: rgw: be aware abount tenants on cls_user_bucket -> rgw_bucket conversion

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #12729 from liupan1111/wip-fix-disable-rgw-compile-error

jewel: build/ops: fixed compilation error when --with-radowsgw=no

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #12239 from liupan1111/wip-fix-makefile-bug

jewel: build/ops: fixed the issue when --disable-server, compilation fails.

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #12380 from dachary/wip-18183-jewel

jewel: cephfs metadata pool: deep-scrub error omap_digest != best guess omap_digest

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13130 from rjfd/wip-18608-jewel

jewel: librbd: allow to open an image without opening parent image

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13233 from smithfarm/wip-18556-jewel

jewel: rbd: Potential race when removing two-way mirroring image

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13113 from jcsp/wip-18679

jewel: mds: finish clientreplay requests before requesting active state

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #13139 from jcsp/wip-18100

jewel: mds: fix MDSMap upgrade decoding

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #12490 from linuxbox2/jewel-rgw-header

jewel: rgw: add option to log custom HTTP headers (rgw_log_http_headers)

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #12079 from rzarzynski/wip-rgw-17961

jewel: rgw: TempURL properly handles accounts created with the implicit tenant

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #12044 from Abhishekvrshny/wip-17886-jewel

jewel: rgw: multisite: ECANCELED & 500 error on bucket delete

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #13422 from smithfarm/wip-fix-hammer-x-fix

tests: remove extra indentation in exec block

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #13364 from tmuthamizhan/wip-ansible-jewel

qa: ceph-ansible smoke suite modified for jewel

tests: remove extra indentation in exec block

The exec block was indented by an extra space, causing

line 439, in parse_block_mapping_key
    "expected <block end>, but found %r" % token.id, token.start_mark)
yaml.parser.ParserError: while parsing a block mapping
  in "<string>", line 111, column 3:
      sequential:
      ^
expected <block end>, but found '<block sequence start>'
  in "<string>", line 126, column 4:
       - exec:
       ^

when trying to run upgrade/hammer-x

Reported-by: Yuri Weinstein <yweins@redhat.com>
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #13404 from smithfarm/wip-fix-hammer-x

tests: add require_jewel_osds to upgrade/hammer-x/tiering

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

tests: add require_jewel_osds to upgrade/hammer-x/tiering

Without this, the test tends to fail, but sometimes passes (apparently because
the "wait-for-healthy: true" in the last ceph.restart is racy - HEALTH_OK is
received before the MONs notice that all OSDs are running jewel without
require_jewel_osds.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

librbd: async method to check journal tag owner

Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit 953561f84140efcc870ec4a8f1ecb732b8e99409)

Conflicts:
src/librbd/Journal.cc - trivial context difference (master does not
                                have "class SafeTimerSingleton"),
                                added "typedef ::journal::Journaler Journaler;" in
                                C_IsTagOwner template to fix build failure

rbd-mirror: check image mirroring state when bootstrapping

Fixes: http://tracker.ceph.com/issues/18447
Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit 5fc5a8ac895524f05eed6e7db20b0dda3a8cb60f)

rbd-mirror: async request to test if image is primary

Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit 0a1cb35caacdf85029f31a0364dc07a5d7462f5f)

Conflicts:
src/tools/rbd_mirror/CMakeLists.txt - file doesn't exist in jewel
            (ported the patch to src/tools/Makefile-client.am)
src/tools/rbd_mirror/image_replayer/BootstrapRequest.cc - Journal is
             implemented by a typedef in jewel
src/tools/rbd_mirror/image_replayer/OpenLocalImageRequest.cc -
             no compat.h include in jewel, Journal is implemented by a typedef
             in jewel

Merge pull request #13331 from smithfarm/wip-18869-jewel

jewel: tests: SUSE yaml facets in qa/distros/all are out of date

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #11410 from dachary/wip-17334-jewel

jewel: crushtool --compile is create output despite of missing item

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/cli/crushtool: fix the test of compile-decompile-recompile.t

should read the map from $TESTDIR,
it's a regression introduced by b2c0a07

Fixes: http://tracker.ceph.com/issues/17306
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 404043980d7882f320f48fbaed4b18f31bb15eb7)

crush/CrushCompiler: error out as long as parse fails

do not output compiled crush map if anything goes wrong when parsing
crush map.

Fixes: http://tracker.ceph.com/issues/17306
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit b2c0a079dc074c907e1bc429861230096534f597)

qa: ceph-ansible smoke suite modified for jewel

Signed-off-by: Tamil Muthamizhan <tmuthami@redhat.com>

Merge pull request #13358 from linuxbox2/wip-jewel-marker

jewel: rgw: fix use of marker in List::list_objects()

rgw: fix use of marker in List::list_objects()

Fixes: http://tracker.ceph.com/issues/18331
List marker is an index key, so treat it as such. This
fixes infinite loop in orphans find command.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit a5d1fa0587184f43c69d8e03114b58d43f320781)

tests: drop buildpackages.py

The buildpackages suite has been moved to teuthology. This cleans up a file
that was left behind by https://github.com/ceph/ceph/pull/13297

Fixes: http://tracker.ceph.com/issues/18846
Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit 6b7443fb50c117ee7f20d53bbc7530bb0eb7ebd5)

tests: update SUSE yaml facets in qa/distros/all

Fixes: http://tracker.ceph.com/issues/18856
Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit 0bd56e871a8549d4b0b1211f09dad2d1120fb606)

Merge pull request #13299 from dachary/wip-18848-jewel

jewel: tests: remove qa/suites/buildpackages

Reviewed-by: Nathan Cutler <ncutler@suse.cz>

buildpackages: remove because it does not belong

It should live in teuthology, not in Ceph. And it is currently broken:
there is no need to keep it around.

Fixes: http://tracker.ceph.com/issues/18846
Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 5a43f8d57925da227c95480501ceec10a29395d8)

Conflicts:
qa/tasks/buildpackages/common.sh (remove)
qa/tasks/buildpackages/make-rpm.sh (remove)

Merge pull request #13280 from athanatos/wip-revert-jewel-18581

Revert "Merge pull request #12978 from asheplyakov/jewel-18581"

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Revert "Merge pull request #12978 from asheplyakov/jewel-18581"

See: http://tracker.ceph.com/issues/18809

This reverts commit 8e69580c97622abfcbda73f92d9b6b6780be031f, reversing
changes made to c05730ceb3387fb43c35937f0506297a34a44452.

Signed-off-by: Samuel Just <sjust@redhat.com>

rgw: be aware abount tenants on cls_user_bucket -> rgw_bucket conversion.

Fixes: http://tracker.ceph.com/issues/18364
Fixes: http://tracker.ceph.com/issues/16355
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
(cherry picked from commit 871e1f51afe9d6c8b88debc07460b4316121f999)

librbd: allow to open an image without opening parent image

Fixes: http://tracker.ceph.com/issues/18325
Signed-off-by: Ricardo Dias <rdias@suse.com>
(cherry picked from commit 61af1c25015de087a2423811548d975dd7d430b4)

Conflicts:
  src/librbd/ImageState.cc - added missing arg to RefreshRequest::create
  src/librbd/exclusive_lock/PostAcquireRequest.cc -
                                        deleted, does not exist in jewel
  src/librbd/image/OpenRequest.cc - added missing arg to
                                    RefreshRequest::create
  src/librbd/internal.cc - added missing arg to ImageState::open
  src/librbd/librbd.cc - added missing arg to ImageState::open
  src/test/librbd/exclusive_lock/test_mock_PostAcquireRequest.cc -
                                        deleted, does not exist in jewel
  src/test/rbd_mirror/image_replayer/test_mock_CreateImageRequest.cc -
                                        added missing arg to ImageState::open
  src/test/rbd_mirror/test_PoolWatcher.cc - added missing arg to
                                            ImageState::open

rbd-mirror: hold owner lock when testing if lock owner

Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit 82aa89668d666f434cd19ff444223017b5512c6a)

Merge pull request #11872 from dachary/wip-17838-jewel

jewel: leak in RGWFetchAllMetaCR

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #13182 from smithfarm/wip-18498-jewel

jewel: rgw: Realm set does not create a new period

Merge pull request #13180 from smithfarm/wip-18547-jewel

jewel: rgw: multisite: segfault after changing value of rgw_data_log_num_shards

Merge pull request #13177 from smithfarm/wip-18676-jewel

jewel: rgw: librgw: objects created from s3 apis are not visible from nfs mount point

Merge pull request #13175 from smithfarm/wip-18684-jewel

jewel: rgw: multisite: sync status reports master is on a different period

Merge pull request #13173 from smithfarm/wip-18710-jewel

jewel: rgw: slave zonegroup cannot enable the bucket versioning

Merge pull request #13171 from smithfarm/wip-18712-jewel

jewel: rgw: radosgw-admin period update reverts deleted zonegroup

Merge pull request #13001 from asheplyakov/jewel-bp/18559

jewel: rgw multisite: fix ref counting of completions

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #12997 from asheplyakov/jewel-bp-18569

jewel: radosgw valgrind "invalid read size 4" RGWGetObj

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #12764 from SUSE/wip-17342-jewel

jewel: tests: assertion failure in a radosgw-admin related task

Merge pull request #12738 from SUSE/wip-18286-jewel

jewel: rgw: multisite: coroutine deadlock in RGWMetaSyncCR after ECANCELED errors

Merge pull request #12678 from Abhishekvrshny/wip-18348-jewel

jewel: rgw ldap: enforce simple_bind w/LDAPv3 redux

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #12622 from wido/jewel-15776

jewel: rgw: log name instead of id for SystemMetaObj on failure

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #12542 from dreamhost/wip-17076-jewel

jewel: rgw: Replacing '+' with "%20" in canonical uri for s3 v4 auth.

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #12428 from SUSE/wip-18216-jewel

jewel: rgw-admin: missing command to modify placement targets

Merge pull request #12426 from SUSE/wip-18214-jewel

jewel: add max_part and nbds_max options in rbd nbd map, in order to keep consistent with

http://tracker.ceph.com/issues/17851#note-59

Merge pull request #12419 from SUSE/wip-18217-jewel

jewel: rgw sends omap_getvals with (u64)-1 limit

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #12405 from jan--f/wip-18199-jewel

jewel: build/ops: install-deps.sh based on /etc/os-release

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #12397 from linuxbox2/jewel-next

jewel: rgw: do not abort when accept a CORS request with short origin

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #12316 from dachary/wip-18101-jewel

jewel: Add workaround for upgrade issues for older jewel versions

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #12890 from dillaman/wip-18453-jewel

jewel: librbd: new API method to force break a peer's exclusive lock

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #12302 from SUSE/wip-18135-jewel

jewel: build/ops: add hostname sanity check to run-{c}make-check.sh

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #12315 from dachary/wip-18102-jewel

jewel: rgw: Unable to commit period zonegroup change

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #12314 from dachary/wip-18107-jewel

jewel: multisite: failed assertion in 'radosgw-admin bucket sync status'

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #12313 from dachary/wip-18112-jewel

jewel: rgw: multisite requests failing with '400 Bad Request' with civetweb 1.8

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #12258 from rzarzynski/wip-rgw-17931-jewel

jewel: rgw: add support for the prefix parameter in account listing of Swift API

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #12156 from dachary/wip-17969-jewel

jewel: rgw: multisite upgrade from hammer -> jewel ignores rgw_region_root_pool

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #11990 from asheplyakov/jewel-17908

jewel: rgw: for the create_bucket api, if the input creation_time is zero, we …

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #11876 from dachary/wip-17839-jewel

jewel: rgw: the value of total_time is wrong in the result of 'radosgw-admin log show' opt

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #11868 from dachary/wip-17512-jewel

jewel: multisite: metadata master can get the wrong value for 'oldest_log_period'

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #11866 from dachary/wip-17709-jewel

jewel: rgw: multisite: coroutine deadlock assertion on error in FetchAllMetaCR

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #11497 from rzarzynski/wip-rgw-slashinfo-jewel

jewel: rgw: add suport for Swift-at-root dependent features of Swift API

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #11477 from dachary/wip-17119-jewel

jewel: rgw: multisite: assert(next) failed in RGWMetaSyncCR

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #11476 from dachary/wip-17162-jewel

jewel: rgw: multisite doesn't retry RGWFetchAllMetaCR on failed lease

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #11470 from dachary/wip-17514-jewel

jewel: rgw：bucket check remove _multipart_ prefix

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #10661 from dachary/wip-16871-jewel

jewel: rgw: Have a flavor of bucket deletion in radosgw-admin to bypass garbage collection

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #13187 from asheplyakov/jewel-bp-18729

jewel: cli: ceph-disk: convert none str to str before printing it

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #13153 from smithfarm/wip-no-firefly-on-centos

jewel: tests: upgrade:hammer-x: install firefly only on Ubuntu 14.04

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #13118 from smithfarm/wip-18274-jewel

jewel: Memory leaks in object_list_begin and object_list_end

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13025 from SUSE/wip-18605-jewel

jewel: ceph-disk prepare writes osd log 0 with root owner

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #13106 from ceph/wip-cd-dev-option

qa/tasks: jewel backport -  ceph-deploy use of dev option

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #13062 from asheplyakov/jewel-bp-18379

jewel: msg/simple: clear_pipe when wait() is mopping up pipes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12978 from asheplyakov/jewel-18581

jewel: ReplicatedBackend: take read locks for clone sources during recovery

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #12755 from Abhishekvrshny/wip-18284-jewel

jewel: Need CLI ability to add, edit and remove omap values with binary keys

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12677 from SUSE/wip-18104-jewel

jewel: mon: ceph osd down detection behaviour

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12291 from asheplyakov/jewel-18108

jewel: msg/simple/Pipe: error decoding addr

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11998 from Abhishekvrshny/wip-17877-jewel

jewel: FileStore: fiemap cannot be totally retrieved in xfs when the number of extents > 1364

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11991 from asheplyakov/jewel-17909

jewel: osd: limit omap data in push op

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #12045 from Abhishekvrshny/wip-17875-jewel

jewel: rgw: file: remove spurious mount entries for RGW buckets

Merge pull request #13161 from smithfarm/wip-lfn-upgrade-hammer

jewel: tests: add require_jewel_osds before upgrading last hammer node

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #12425 from SUSE/wip-18190-jewel

jewel: rbd-mirror: gmock warnings in bootstrap request unit tests

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13129 from smithfarm/wip-18558-jewel

jewel: rbd: bench-write will crash if --io-size is 4G

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13155 from smithfarm/wip-18494-jewel

jewel: rbd: [rbd-mirror] sporadic image replayer shut down failure

Merge pull request #13128 from smithfarm/wip-18633-jewel

jewel: rbd: [qa] crash in journal-enabled fsx run

http://tracker.ceph.com/issues/13512

Merge pull request #13104 from idryomov/wip-fio-unmap-devices-jewel

qa/tasks: backport rbd_fio fixes to jewel

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13156 from smithfarm/wip-18455-jewel

jewel: rbd: Attempting to remove an image w/ incompatible features results in partial removal

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13157 from smithfarm/wip-18434-jewel

jewel: rbd: Improve error reporting from rbd feature enable/disable

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13168 from trociny/wip-18550-jewel

jewel: librbd: metadata_set API operation should not change global config setting

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12739 from SUSE/wip-18323-jewel

jewel: JournalMetadata flooding with errors when being blacklisted

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12416 from SUSE/wip-18219-jewel

jewel: msg: don't truncate message sequence to 32-bits

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13115 from smithfarm/wip-18404-jewel

jewel: mon: cache tiering: base pool last_force_resend not respected (racing read got wrong version)

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13045 from SUSE/wip-18553-jewel

jewel: mon: peon wrongly delete routed pg stats op before receive pg stats ack

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13117 from smithfarm/wip-18280-jewel

jewel: mon: osd flag health message is misleading

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12320 from dachary/wip-18061-jewel

jewel: rgw:fix for deleting objects name beginning and ending with underscores of one bucket using POST method of js sdk.