rgw: fix chunked-encoding for chunks >1MiB

For HTTP responses sent with chunked-encoding, and greater than 1MiB in
size, the chunk-size field was being printed wrong.

Specifically, the chunk-size field was being sent with a mangled or
missing trailer of '\r\n'.

This bug manifested as HTTP clients being unable to read the response:
Chrome generates ERR_INCOMPLETE_CHUNKED_ENCODING
Python/boto generates httplib.LineTooLong: got more than 65536 bytes when reading chunk size

The wrong variable was being used to determine the size of the buffer
used for the chunk-size field.

Fix it by using the correct variable, and rename the variables to
clearly reflect their purpose.

Prior to PR#23940, this would only have been seen in some Swift
operations. PR#23940 changed some S3 operations to also use chunked
encoding to get responses sent faster, and made the bug easier to
detect. It was initially reported for a ListBucket call with a high
max-keys argument.

Backport: luminous, mimic
Reference: https://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.6.1
Reference: https://github.com/ceph/ceph/pull/23940
Fixes: http://tracker.ceph.com/issues/35990
Signed-off-by: Robin H. Johnson <rjohnson@digitalocean.com>

Merge pull request #23650 from tchaikov/wip-dpdk-cmake

cmake: fix the build WITH_DPDK=ON

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24097 from cyx1231st/fix-seastar-msgr-read-racing

crimson/net: Fix racing for promise on_message

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/net: Fix racing for promise on_message

read_tags_until_next_message() will forward the ready future and create
a new promise for on_message, which assumes there is already a
read_message() holding the previous promise, but it is not true.

Signed-off-by: Yingxin <yingxin.cheng@intel.com>

Merge pull request #23972 from badone/wip-documentation-add-12.2.8-release

doc/releases/schedule: Add Luminous 12.2.8

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #24099 from badone/wip-mimic-13-2-0-date-correction

doc/releases: Mark Mimic first release as June

Reviewed-by: Nathan Cutler <ncutler@suse.com>

doc/releases: Mark Mimic first release as June

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #24094 from libingyang-zte/master

doc: Fix Typos of Developer Guide

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

doc: Fix Typos of Developer Guide

Signed-off-by: Li Bingyang <li.bingyang1@zte.com.cn>

Merge pull request #23896 from Songweibin/wip-group-snap-rollback

rbd: add group snap rollback method

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #24077 from alfredodeza/wip-rm35970

ceph-volume  fix zap not working with LVs

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume tests.functional comment why not use --destroy when zapping lvs in filestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional use --destroy when zapping lvs in bluestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #24024 from p-na/fix-flaky-qa-tests

mgr/dashboard: Fix flaky QA tests

Reviewed-by: Ricardo Dias <rdias@suse.com>

ceph-volume lvm.zap use remove_pv only for physical devices

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #23660 from falcon78921/wip-multisite-docs

doc: updated multisite documentation

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23995 from tchaikov/wip-add-ceph-libboost-repo

install-deps,run-make-check: use ceph-libboost repo

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Erwan Velu <erwan@redhat.com>

Merge pull request #24067 from libingyang-zte/master

doc: Fix typos in Developer Guide

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: Fix Spelling Error in Developer Guide

Signed-off-by: Li Bingyang <li.bingyang1@zte.com.cn>

Merge pull request #23919 from adamemerson/wip-end-to-warnings

Clean up warnings

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24018 from dzafman/wip-35912

qa/standalone: Minor test improvements

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24060 from alfredodeza/wip-rm35913

ceph-volume batch: allow --osds-per-device, default it to 1

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume tests.lvm.strategies initial filestore validation

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.tests conftest fixtures for custom vgs

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.devices.lvm add bluestore strategy cases

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests verify strategy validators for minimum size

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.batch.filestore allow osds-per-device

Signed-off-by: Alfredo Deza <adeza@redhat.com>

cls_cas: Fix format string warning

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge PR #24025 into master

* refs/pull/24025/head:
mon/OSDMonitor: reject pg_num -> 0

Reviewed-by: huangjun <huangjun@xsky.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #21611 into master

* refs/pull/21611/head:
osd/ECBackend: don't get result code of subchunk-read overwritten

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge PR #23273 into master

* refs/pull/23273/head:
os/bluestore: handle spurious read errors

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #23694 into master

* refs/pull/23694/head:
mon: remove pre-luminous health output compatibility

Reviewed-by: Sage Weil <sage@redhat.com>

test: Remove unused variable

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

mds: Remove pessimizing move

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

test: Silence unused capture warnings

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

test: Fix sprintf overflow warning

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

test: Quell unused variable warnings

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

test: Fix sign comparison warnings

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge PR #24035 into master

* refs/pull/24035/head:
osd/PG: avoid choose_acting picking want with > pool size items

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #24052 into master

* refs/pull/24052/head:
doc: update mds stat output

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #24032 from liewegas/wip-pg-ls

mon/PGMap: include unknown PGs in 'pg ls'

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #24040 from rzarzynski/wip-common-bl-drop-raw_mmap_pages

common: drop the unused buffer::raw_mmap_pages.

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

rbd: sort to display group image by name

Signed-off-by: songweibin <song.weibin@zte.com.cn>

pybind/rbd: new group snap rollback Python API

Signed-off-by: songweibin <song.weibin@zte.com.cn>

tools/rbd: support group snap rollback CLI

Signed-off-by: songweibin <song.weibin@zte.com.cn>

librbd: minor cleanup in api/Group.cc

Signed-off-by: songweibin <song.weibin@zte.com.cn>

librbd: add group snap rollback method

Fixes: http://tracker.ceph.com/issues/23550
Signed-off-by: songweibin <song.weibin@zte.com.cn>

cephfs: Do not catch polymorphic exception by value

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

crush: Quell unused variable warnings

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

ceph-volume lvm.batch.validators account for osds per device, include collocation

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.batch.bluestore allow osds-per-device with MixedType

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #24042 from liewegas/wip-merge-derr

osd: remove stray derr

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23996 from mogeb/fmt-cmake

cmake: fix version matching for Findfmt

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24013 from dzafman/wip-35845

test: Use a grep pattern that works across releases

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24045 from dillaman/wip-35702

librbd: fixed assert when flattening clone with zero overlap

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #24030 from LenzGr/mgr-dashboard-documentation

doc: Updated Mgr Dashboard documentation

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #24055 from sebastian-philipp/kubejacker-opensuse-mimic

script/kubejacker: Add openSUSE based images

Reviewed-by: John Spray <john.spray@redhat.com>

script/kubejacker: Add openSUSE based images

* Added executable bit of `kubejacker.sh`.
* Reduced the number of `RUN` commands
* Added zypper commands for SUSE.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: Cleanup

Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

mgr/dashboard: Fix flaky QA tests

Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

doc: update mds stat output

Signed-off-by: Jos Collin <jcollin@redhat.com>

librbd: fixed assert when flattening clone with zero overlap

Fixes: http://tracker.ceph.com/issues/35702
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mon/OSDMonitor: reject pg_num -> 0

Otherwise it will finally crash the mgr process:
```
-2> 2018-09-11 16:23:31.272 7ff521d59700 -1 /clove/xxG/ceph/src/osd/osd_types.cc: In function 'pg_t pg_t::get_parent() const'
 thread 7ff521d59700 time 2018-09-11 16:23:31.272096
/clove/xxG/ceph/src/osd/osd_types.cc: 587: FAILED ceph_assert(bits)
```

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #23941 from trociny/wip-23445

librbd: flatten operation should use object map

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #22675 from nuyhah/fix_block_name_prefix

librbd: block_name_prefix is not created randomly

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #23790 from iridescent-rsy/myfix

librbd: remove template declaration of a non-template function

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #23853 from zy751713126/add_protect_snapshot_list

rbd: add protected in snap list

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #24009 from trociny/wip-test-migration-other-pool

test/librbd: migration tests did not delete additional pool

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

osd: remove stray derr

This was debug output that crept into the final commit.

Signed-off-by: Sage Weil <sage@redhat.com>

librbd: block_name_prefix is not created randomly

fixes: http://tracker.ceph.com/issues/24634
Signed-off-by: hyun ha hyun.ha@navercorp.com

mon/PGMap: allow 'pg ls unknown'

"unknown" is annoying because it is 0 and needs a special case.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #23702 from Rubab-Syed/rgw-orchestrator-rook

mgr/orchestrator: Add RGW service support

Reviewed-by: John Spray <john.spray@redhat.com>

mon/PGMap: include unknown PGs in 'pg ls [all]'

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #23908 into master

* refs/pull/23908/head:
client: retry remount on dcache invalidation failure

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #23691 into master

* refs/pull/23691/head:
client:two ceph-fuse client, one can not list out files created by another.

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #23323 into master

* refs/pull/23323/head:
client: set f_files to the total number of files in the filesystem

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23314 into master

* refs/pull/23314/head:
mds: include session uptime when diplaying session list
mds: track average uptime of sessions

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23954 from zy751713126/clear_snap_limit

rbd: simplified code to remove do_clear_limit function

Merge PR #23198 into master

* refs/pull/23198/head:
mds: optimize acquiring locks
mds: reduce memory fragments of EMetaBlob
mds: optimize memory allocation in CInode::encode_inodestat
mds: convert MDSCacheObject::ref_map to flat_map

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23199 into master

* refs/pull/23199/head:
script: backport-create-issue: optionally take list of issue numbers

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23728 into master

* refs/pull/23728/head:
doc: suppored features of kernel client

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

common: drop the unused buffer::raw_mmap_pages.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #23824 into master

* refs/pull/23824/head:
doc/cephfs: add notes on application best practices

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

script: backport-create-issue: optionally take list of issue numbers

Make the script optionally take a comma-separated list of issue numbers.
(Could be just one issue.)

Before this patch, backport-create-issue script insisted on looping over all
issues in Pending Backport status. This was cumbersome in cases when only
one issue (or a couple issues) needed to be processed.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

osd/PG: avoid choose_acting picking want with > pool size items

If the pool size recently changed, we might see an up that is larger than
the pool size.  Or, we might already have selected a primary that is not
part of up, and then add all up osds and end up with a want that is too
big.

Fixes: http://tracker.ceph.com/issues/35924
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #21379 from pritha-srivastava/wip-rgw-user-policy

rgw: User Policy

ceph-volume lvm.batch.bluestore allow osds-per-device with SingleType

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.batch add --osds-per-device flag, defaulting to 1

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #23497 from noahdesu/insights

mgr/insights: insights reporting module

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #23146 from jcsp/wip-progress

mgr/progress: improve+test OSD out handling

Reviewed-by: Noah Watkins <nwatkins@redhat.com>

Merge pull request #23897 from votdev/improve_delete_modal

mgr/dashboard: Make deletion dialog more touch device friendly

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #24016 from votdev/bug_35907

mgr/dashboard: Progress bar does not stop in TableKeyValueComponent

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge PR #22987 into master

* refs/pull/22987/head:
common,rgw: rename sha1_digest_t
osd: decrement old chunk's reference count if the chunk has a reference.
src/test: add a unit test
osd: using fingerprint OID if fingerprint is set
osd: add flag interfaces in chunk_info_t
common/buffer.cc: add sha1 fingerprint
osd: add fingerprint property
mon: add a command to set fingerprint algorithm

Merge PR #24006 into master

* refs/pull/24006/head:
osd/OSD: clear ping_history on heartbeat_reset
mon/OSDMonitor: share new maps with even non-active osds

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #24010 into master

* refs/pull/24010/head:
osd/OSD: kick right merge source
mgr/DaemonServer: split should respect inflight creating pgs

Reviewed-by: Sage Weil <sage@redhat.com>

run-make-check,do_cmake: set BOOST_J if !WITH_SYSTEM_BOOS

it's also more readable if we can set the number of make jobs
in a single place.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: always add ${Boost_INCLUDE_DIRS} to include dirs

we need to -I${Boost_INCLUDE_DIRS} if WITH_SYSTEM_BOOST=ON and boost is
found by FindBoost.cmake.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: support multi-arch lib directory used along with BOOST_ROOT

by default, debian packaging creates ${prefix}/lib/x86_64-linux-gnu for
multi-arch support. but FindBoost.cmake does not check in such a
directory if $BOOST_ROOT is set. in this change, this directory is added
to the search path.

Signed-off-by: Kefu Chai <kchai@redhat.com>

install-deps,run-make-check: use ceph-libboost repo

ceph-libboost1.67 repo is setup to satisfy the libboost dependencies
used when building ceph. now it offers arm64 and amd64 builds for xenial.

* install-deps.sh: add ceph-libboost1.67 repo hosted on chacra if the
  building host does not have any of the listed ceph-libboost1.67
  packages installed
* run-make-check.sh: add "-DWITH_SYSTEM_BOOST=ON
  -DBOOST_ROOT=/opt/ceph" if include/boost/config.hpp is found
  under /opt/ceph

See-also: http://tracker.ceph.com/issues/25186
Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake/modules: add FindBoost.cmake

the FindBoost.cmake shipped by cmake 3.5.1 is not new enough to identify
the dependencies of boost 1.63 and up. so we need to copy the
FindBoost.cmake from cmake upstream. it supports up to boost 1.68.

Signed-off-by: Kefu Chai <kchai@redhat.com>