radosgw: fix small sized chunked uploads with s3

The logic to compute the number of bytes to copy with a chunked file
upload was failing to take into account the amount of data that had been
consumed, but was still pending in the parse buffer.  This would cause
strange behavior ranging from "just works" to "fails".  There was also a
strange "works, one byte at a time" mode.  Using the correct stream_pos
offset eliminates the bad behavior.

This fix also adds debug statements to make the correct behavior easier to see.

Fixes: http://tracker.ceph.com/issues/21003
Signed-off-by: Marcus Watts <mwatts@redhat.com>

radosgw: fix zero sized chunked uploads with s3

s3 allows for zero sized objects, which can be
created using chunked uploads.  It is important to
call "recv_body (and ChunkMeta::create_next()) even
on zero sized objects, so that the proper chunk signature
can be presented at signature verification time.

Fixes: http://tracker.ceph.com/issues/21000
Signed-off-by: Marcus Watts <mwatts@redhat.com>

rgw: replace '+' with "%20" in canonical query string for s3 v4 auth.

( note: this patch modified: now replaces "+" with " ".  mwatts@redhat.com )

Fixes: http://tracker.ceph.com/issues/20501
Signed-off-by: Zhang Shaowen <zhangshaowen@cmss.chinamobile.com>

Merge pull request #16628 from theanalyst/rgw-policy-list-buckets

rgw: policy: support for s3 conditionals in ListBucket Op

Merge pull request #16968 from liewegas/wip-store-test-expect-death

ceph_test_objectstore: drop expect regex

ceph_test_objectstore: drop expect regex

If logging is enabled (as it now is in teuthology) this won't match the
forked output.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16991 from liewegas/wip-doc-bluestore-migration

doc/rados/operations/bluestore-migration: typos and whitespace

doc/rados/operations/bluestore-migration: typos and whitespace

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16532 from liupan1111/wip-remove-reb

kv/RocksDBStore: extract common code to a new function

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16594 from wjwithagen/wip-wjw-reduce-info-messages

common: make some message informative, instead of error

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16964 from kungf/mon_timer

mon: return directly after health_events_cleanup

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16936 from TsaiJin/wip-set-polling-thread-name

msg/async/rdma: fix the bug that rdma polling thread uses the same thread name with msg worker

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Pan Liu <liupan1111@gmail.com>

Merge pull request #16961 from xiexingguo/wip-class-rename

crush: "osd crush class rename" support

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16973 from jan--f/mgr-plugin-doc-accessor-init

doc/mgr/plugins: mgr accessor during init causes exception

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #16983 from renhwztetecs/renhw-wip-rados-tool-cleanup

rados/tool: fixup rados stat command hint

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge branch 'master' into wip-set-polling-thread-name

rados/tool: fixup rados stat command hint

Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

Merge pull request #16709 from dzafman/wip-standalone

qa/standalone: misc fixes

Merge pull request #16954 from gregsfortytwo/wip-20891-pool-metadata

mon: fix "fs new" pool metadata update, tests

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16981 from yuyuyu101/wip_rdma_mt_fix

msg/async/rdma: fixes crash for multi rados client within one process

mon: return directly after health_events_cleanup

when mon_health_to_clog was set false, all health events was cleanup,
no need to judge the change of mon_health_to_clog_interval and
mon_health_to_clog_tick_interval.

Signed-off-by: wang yang <yang.wang@easystack.cn>

Merge pull request #16664 from amitkumar50/coverity-1414508-3

rdma: Assign instead of compare

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #16965 from TsaiJin/wip-use-rdtsc-to-get-time

src/msg/async/rdma: decrease cpu usage by rdtsc instruction

Reviewed-by: Haomai Wang <haomai@xsky.com>

msg/async/rdma: add atomic and lock protection

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async/rdma: fixes crash in fio

fio creates multiple CephContext in a single process.
Crash(es) happen because rdma stack has a global resources that
are still used from one ceph context while have already been destroyed
by another context.

The commit removes global instances of RDMA dispatcher and infiniband
and makes them context (rdma stack) specific.

Signed-off-by: Adir Lev <adirl@mellanox.com>
Signed-off-by: Alex Mikheev <alexm@mellanox.com>

Merge pull request #16980 from ceph/revert-16893-wip-fix-multicct-ib

Revert "msg/async/rdma: fix multi cephcontext confllicting"

Revert "msg/async/rdma: fix multi cephcontext confllicting"

Merge pull request #16977 from liewegas/wip-no-pidfile

test/CMakeLists: disable test_pidfile.sh

Reviewed-by: Xie Xingguo <xie.xingguo@zte.com.cn>

crush: "osd crush class rename" support

In 076a6abd80cc90ebcb901f908f880ef030721b2a I killed the 'class rename' command
and thought it was totally useless but I was wrong.

Consider the following user case:
(1) randomly choose some OSDs(e.g., from different hosts) and try to make them for private use only,
    say, by grouping them into 'pool1'
(2) ceph osd crush set-device-class pool1 'OSDs from (1)'
(3) ceph osd crush rule create-replicated rule_for_pool1 default host pool1
(4) ceph osd pool rename pool1 pool2
(5) ceph osd crush class rename pool1 pool2

From the above user case, we need to safely change a pool name without worrying
any risk of data migration. That is why the 'osd crush class rename' command
is still needed here.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

test/CMakeLists: disable test_pidfile.sh

Too flaky, see http://tracker.ceph.com/issues/20975

Signed-off-by: Sage Weil <sage@redhat.com>

qa: Fix broken test_activate_osd() due to missing space

Signed-off-by: David Zafman <dzafman@redhat.com>

qa: Fix races with waiting for scrubs

The trigger_scrub sets the last_scrub_stamp backwards to
force a scheduled scrub.  In a small window this stamp could get propagated
to the mgr.  A test failure occurred because wait_for_scrub() was confused
by seeing a backward moving date.

The most critical change is having wait_for_scrub() make sure that the
date advances past the previous in value.

A test failed because the random backoff kept delayed triggered scrub, so
set osd_scrub_backoff throughout.

Signed-off-by: David Zafman <dzafman@redhat.com>

qa: Testing of ceph-helpers.sh, teardown on fail to dump logs, save cores

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Avoid confusion over legacy snaps when head_exists corrupt

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Fix Paxos shutdown handling for commit_finish race

Fixes: http://tracker.ceph.com/issues/20921
Signed-off-by: David Zafman <dzafman@redhat.com>

qa: Add support for core dumps

Save core dumps when running tests locally
Dump logs to output whenever cores seen

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #16918 from liewegas/wip-doc-bluestore-migration

doc/rados/operations/bluestore-migration: document bluestore migration process

Reviewed-by: Vasu Kulkarni <vasu@redhat.com>

doc/rados/operations/bluestore-migration: document bluestore migration process

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16975 from theanalyst/doc-12-1-3-release-notes

doc: 12.1.3 release notes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16974 from liewegas/wip-release-notes

doc/release-notes: remove mention of crush weight optimization

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

qa: test that "fs new" correctly set the application_metadata

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

doc: 12.1.3 release notes

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

doc/release-notes: remove mention of crush weight optimization

It's not there yet.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15330 from guihecheng/rgw_file-fsid-mount

rgw_file: introduce new fsid and rgw_mount

mdsmon: treat the osdmon correctly when doing plugged updates

Make sure it's writeable before invoking changes, and propose_pending()
on it when we're done.
Make the PaxosService::C_RetryMessage public so we can do this from FSCommands.

Maybe-
Fixes: http://tracker.ceph.com/issues/20959
Signed-off-by: Greg Farnum <gfarnum@redhat.com>

doc/mgr/plugins: mgr accessor during init causes exception

Currently calling a mgr accessor during a plugins __init__ method causes
a cyclic locking exception.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #16648 from ovh/bp-kill-str-conv

osdc, class_api: kill implicit string conversions

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

Merge pull request #16850 from liewegas/wip-bluestore-dirty-range

os/bluestore: fix dirty_shard off-by-one

Reviewed-by: Liu-Chunmei <chunmei.liu@intel.com>

Merge pull request #16872 from markhpc/wip-bluestore-faster_gncb

os/bluestore:  clearer comments, not slower code.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16941 from jcsp/wip-20956

osd: include front_iface+back_iface in metadata

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16944 from liewegas/wip-kraken-x

mon/Elector: force election epoch bump on start

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #16953 from neha-ojha/wip-20798

test/librados: create unique lock names

Reviewed-by: Sage Weil <sage@redhat.com>

qa: For FreeBSD skip osd-dup.sh because there is no bluestore

Signed-off-by: David Zafman <dzafman@redhat.com>

qa: Raise mon-data-avail-warn to pass tests with less space

Signed-off-by: David Zafman <dzafman@redhat.com>

qa: Add common generalized inject_eio() to ceph-helpers.sh

Retry for a while to allow pool to appear

Signed-off-by: David Zafman <dzafman@redhat.com>

qa: osd-scrub-repair.sh handle older versions of jq

Signed-off-by: David Zafman <dzafman@redhat.com>

qa: Add support for specifying sub-tests with run-standalone.sh

Fix test-ceph-helpers.sh to pass additional arguments on

Signed-off-by: David Zafman <dzafman@redhat.com>

qa: ceph-helpers.sh fixes

Add missing teardown to cleanup test directory
Fix pgid due to elimination of initial default pool
Testing could never fail because run_tests return ignored

Signed-off-by: David Zafman <dzafman@redhat.com>

qa: Skip ceph-helpers.sh which is tested through test-ceph-helpers.sh

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #16962 from ceph/wip-bz1479797

ceph-volume: tests add tests for the is_mounted utility

Reviewed-by: Andrew Schoen aschoen@redhat.com

Merge pull request #16618 from adamemerson/wip-throttle

common/Throttle: Clean up

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16702 from hjwsm1989/wip-recover-timeout-expired

 osd: do not forget pg_stat acks which failed to send

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

Merge pull request #16928 from amitkumar50/cov-messages-7

messages: Initializing members in MOSDPGUpdateLogMissing

Reviewed-by: Jos Collin <jcollin@redhat.com>

doc/release-notes: fix link typo

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16966 from idryomov/wip-krbd-unmap-tests-pool

qa: fix POOL_APP_NOT_ENABLED warning in krbd:unmap suite

Reviewed-by: Sage Weil <sage@redhat.com>

Revert "qa/suites/upgrade/jewel-x/parallel: thrash layout"

This reverts commit 435777dbffc77c93d06476caf83be141359a5778.

This test combination is not yet stable.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16950 from liewegas/wip-20959

osd/PG: force rebuild of missing set on jewel upgrade

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

ceph-volume: is_mounted should use a bytes->string util to compare strings

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: create a utf-8 string decoder for py3 compat

Signed-off-by: Alfredo Deza <adeza@redhat.com>

msg: get the time increment by rdstc inscturtion to decrease the cpu usage

     in RDMADispatcher polling thread

Signed-off-by: Jin Cai <caijin.caij@alibaba-inc.com>

Merge pull request #16945 from liupan1111/wip-fix-spdk-compile

bluestore: fixed compilation error when enable spdk with gcc 4.8.5

Reviewed-by: Haomai Wang <haomai@xsky.com>

ceph-volume: tests add tests for the is_mounted utility

Signed-off-by: Alfredo Deza <adeza@redhat.com>

qa: fix POOL_APP_NOT_ENABLED warning in krbd:unmap suite

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #16956 from theanalyst/doc-changelog-v11.2.1

doc: add changelog for v11.2.1

Reviewed-By: Nathan Cutler <ncutler@suse.com>

doc: add changelog for v11.2.1

Generated as::

git log v11.2.0..v11.2.1 --after="Thu Jan 19 13:08:40 2017 +0000" > doc/changelog/v11.2.1.txt

In order to avoid qe suite merges before this date

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #16893 from yuyuyu101/wip-fix-multicct-ib

msg/async/rdma: fix multi cephcontext confllicting

Merge pull request #16905 from liewegas/wip-upgrade-sloppy-flush

qa/tasks/ceph.py: tolerate flush pg stats exception

qa/suites/upgrade/jewel-x/parallel: thrash layout

We can't kill and restart osds because that will interfere with
the upgrade process.  We can, however, thrash the layout by
tweaking osd weights and so on.  This will exercise osd recovery
paths during the upgrade that aren't normally exercised (outside
of stress-split..which doesn't upgrade individual osds while they
are non-clean).

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: force rebuild of missing set on jewel upgrade

Previously we were detecting the need to rebuild missing based on
whether the "divergent_priors" omap key was present.  Unfortunately,
jewel does not always set this, so it is not a reliable indicator.
(It only gets set if you actually have a divergent prior at some
point in the PG's life time on that OSD.)

Fix by using the info_struct_v on the PG to detect whether we need
to do the conversion.  We didn't bump the value when we adding
the missing persistence, but the fastinfo was also added during
the same period between jewel and kraken, so it will work just as
well.

Fixes: http://tracker.ceph.com/issues/20958
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16006 from shangfufei/wip_modify_connect_logic

Optimization for connection establishment.

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #16382 from PCzhangPC/bug2

common: bench test fall into dead loop when <seconds>=0

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #16396 from amitkumar50/branch20604

core: remove startsync

Reviewed-by: Sage Weil <sweil@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #16886 from liupan1111/wip-add-size

os/bluestore: move size and block_size to the base class BlockDevice

Reviewed-by: Sage Weil <sweil@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Liu-Chunmei <chunmei.liu@intel.com>

Merge pull request #16888 from gmayyyha/osd-ms-verify-auth

osd/OSD.cc: add 'isvalid=false' when failed to parse caps

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #16829 from ivancich/wip-dup-ops-cleanup-1

osd: clean up dup index logic; maintain index flag logic in fewer places

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #16887 from xiexingguo/wip-mon-count

mon: fix wrong mon-num counting  logic of 'ceph features' command

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16892 from xiexingguo/wip-clean-pg-temp

mon/OSDMonitor: fix 'osd pg temp' unable to cleanup pg-temp

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16897 from amitkumar50/cov-MDentryLink-1

messages: Initialization of is_primary

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #16919 from ceph/wip-volume-tests

ceph-volume: adds functional CI testing

Reviewed-by: Alfredo Deza <adeza@redhat.com>

mdsmon: don't add pool application metadata until running fully-luminous

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

test/librados: create unique lock names

Signed-off-by: Neha Ojha <nojha@redhat.com>

ceph-volume: lvm activate should check if the device is mounted to prevent errors from mount

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util add a helper to check if a device is mounted

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #16834 from mdw-at-linuxbox/policy

radosgw: usage: fix bytes_sent bug.

mon/Elector: force election epoch bump on start

We are generally careful when bumping the epoch so that we can join
existing rounds.  However, if we restart in the middle of an election,
and change versions, we need to be certain that our previous ACK (as
$version - 1) isn't accepted as truth for the restarted daemon (running
$version) keeping the same epoch.

The conservatism with bumping is to avoid spurious election cycles, but
mon restarts are more rare, and we need them here.

Fixes: http://tracker.ceph.com/issues/20949
Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/upgrade/kraken-x/stress-split: more whitelisting

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16828 from liewegas/wip-18209

common/LogClient: assign seq and queue atomically

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

Merge pull request #16920 from liewegas/wip-osd-crush-ls

mon/OSDMonitor: implement 'osd crush ls <node>'

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #16923 from liewegas/wip-20738

qa/suites/rados/objectstore: logs

Merge pull request #16737 from dillaman/wip-rbd-config

rbd: switched from legacy to new-style configuration options

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>