unittest_bufferlist: test page_aligned_appender

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: replace claim_append with claim_append_piecewise to reduce memory copy.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

common/buffer: add function  bufferlist::claim_append_piecewise(list&  bl).

This only useful for bl is bufferlist::page_aligned_appender. Using
this function can remove memcopy for continue ptrs.
Because page_aligned_appender::flush will split a ptr into two or
more ptrs. For this case, rebuild_aligned_size_and_memory can't handle,
it  will rebuild.

For example
a=bl.get_page_aligned_appender(1);
a.append(3K)
a.flush();
t.claim_append(bl);
a.append(1K);
a.flush();
t.claim_append(bl);

dst.claim_append(t);
//3K and 1K ptr are continue in memory. But they are two ptrs..
dst.is_aligned_size_and_memory(4096,4096) is false.

We add new function claim_append_piecewise() to specially
handle this case.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #15651 from zealoussnow/wip-0613

config_opt: use bool instead of int for the default value of filestore_debug_omap_check

Merge pull request #15676 from Linbing/wip-mon-delete-unused-config

mon: delete unused config opts of mon_sync_fs_threshold

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15697 from gregsfortytwo/wip-mgr-eaccss-20296

mgr: print a more helpful error message for when users lack mgr ceph caps

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15744 from badone/wip-CID-1412579

ceph-dencoder: Silence coverity CID 1412579

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15637 from liewegas/wip-point-to-point

qa/upgrade/jewel-x/point-to-point: add a mgr during final upgrade

Merge pull request #15365 from ShiqiCooperation/newshiqi

test/unittest_bluefs: When fsync ret is less than 0, fsync can not be…

Reviewed-by: Varada Kari <varada.kari@gmail.com>

Merge pull request #15533 from liewegas/wip-crush-version

mon,osd: add crush_version to OSDMap, and allow crush map updates to gate on crush_version

Reviewed-by: Loic Dachary <loic@dachary.org>

Merge pull request #15685 from wjwithagen/wip-wjw-freebsd-ENODATA-2

core/common: Fix ENODATA for FreeBSD with compat.h

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #15710 from liewegas/wip-20273

osd: bail from _committed_osd_maps inside osd_lock

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15333 from mikulely/validate-bucket-location

rgw/multisite: validate bucket location during bucket creation

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15633 from cloudbau/reduce-rgw-copy-loglevel

rgw: do not log debug output at level 0

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15598 from aclamk/encryption_refactor

radosgw, crypto: simplified code in handle_data functions

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15599 from theanalyst/rgw/fix/bl-cstr

rgw: don't specify a length when converting bl -> string

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15756 from rzarzynski/wip-rgw-cleanup-swift-errmapping

rgw: clean-up error mapping in Swift's authentication strategy.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15677 from zhangsw/fix-rgw-lifecycle-bug

rgw: lifecycle thread shouldn't process the bucket which has been deleted

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #15682 from mikulely/rgw-putobj-params

rgw: abort early when s->length empty during putobj

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15747 from liewegas/wip-scrub-retry

qa/tasks/ceph: osd_scrub_pgs: reissue scrub requests in loop

qa/suites/rados/singleton-nomsgr/multi-backfill-reject: clean up

Set pool size back to 2 so we don't have to have backfill
complete (despite rejection probability) in order to get back to
healthy.  This way we scrub on cleanup.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15712 from liewegas/wip-19882

osd/ECTransaction: only read partial stripes when below *original* object size

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/suites/rados/singleton-nomsgr/full-tiering: unset quota at end

If we leave the quota set, the proxied ops will block
indefinitely, which will block scrubbing on the cache tier pgs
indefinitely.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/resolve_stuck_peering: start osd at end

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15679 from guihecheng/rgw_file-fix-setattr

rgw_file: fix assert upon setattr on bucket

Merge pull request #15660 from fangyuxiangGL/meta_sync_crash

rgw: meta sync thread crash at RGWMetaSyncShardCR

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/workunits/cephtool/test.sh: silence dev warnings

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph: osd_scrub_pgs: reissue scrub requests in loop

The scrub commands are not reliable: if the OSD doesn't happen to
be connected at the time the command is issued it may not get
delivered.  Re-request scrubs for each PG that has not yet been
scrubbed so that we don't wait forever when the original request
is dropped.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph: osd_scrub_pgs: tolerate down osd at initial scrub time

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15738 from cbodley/wip-rgw-string-constexpr

rgw: fix constexpr for string_size in clang

Reviewed By: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #15544 from taodd/master

mds:  Client syncfs is slow (waits for next MDS tick)

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #15746 from liewegas/wip-20227

os/bluestore: better debug output on unsharing blobs

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

client: signal MDS to flush log when doing a syncfs

Fixes: http://tracker.ceph.com/issues/20129
Signed-off-by: dongdong tao <tdd21151186@gmail.com>

Client: signal MDS to flush log when doing a syncfs
Fixes: http://tracker.ceph.com/issues/20129
Signed-off-by: dongdong tao <tdd21151186@gmail.com>

Merge pull request #15745 from zealoussnow/wip-0617

.gitignore: exclude rpm files

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #15481 from jcsp/wip-20178

client: avoid returning negative space available

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #15614 from jcsp/wip-jt-help

tools/cephfs: fix cephfs-journal-tool --help

Reviewed-by: Douglas Fuller <dfuller@redhat.com>

rgw: clean-up error mapping in Swift's authentication strategy.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #15749 from Adirl/wip

msg/async/rdma: handle buffers after close msg

Reviewed-by: Haomai Wang <haomai@xsky.com>

rgw: meta sync thread crash at RGWMetaSyncShardCR

Fixes: http://tracker.ceph.com/issues/20251
Signed-off-by: fang yuxiang fang.yuxiang@eisoo.com

ceph-dencoder: Silence coverity CID 1412579

141         if (i == 0)
142           i = m_list.size();
143         if ((i == 0) || (i > m_list.size()))
144           return "invalid id for generated object";
145         typename list<T*>::iterator p = m_list.begin();
146         for (i--; i > 0 && p != m_list.end(); ++p, --i) ;
>>>     CID 1412579:  API usage errors  (INVALIDATE_ITERATOR)
>>>     Dereferencing iterator "p" though it is already past the end
of its container.
147         m_object = *p;
148         return string();
149       }
150
151       bool is_deterministic() override {
152         return !nondeterministic;

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #15725 from xiexingguo/wip-noupin

mon/OSDMonitor: batch noup/noin osds support

Reviewed-by: Sage Weil <sage@redhat.com>

msg/async/rdma: remove assert from srq destroy in ~Infiniband

issue:1040242

Even though ibv_destroy_srq does not return 0, all resources
are destroyed. No need to assert.

Change-Id: I96562130d22352390f9f7f9a3e0ddbe77a82d8f4
Signed-off-by: DanielBar-On <danielbo@mellanox.com>

msg/async/rdma: handle buffers after close msg

fixing an issue of OSD’s going down during recovery process while traffic
is running in the background.
fix is making sure that messages previously pulled from the cqe and are
now stored in the socket queue “buffers” will always be handled and fd
will be notified even if cqe is empty.

issue: 1025030

Change-Id: I6ced6b8ee0152e233632700ccc9ec19970e66b6a
Signed-off-by: Alexander Mikheev <alexm@mellanox.com>
Signed-off-by: Adir Lev <adirl@mellanox.com>

Merge pull request #15748 from Liuchang0812/wip-fix-release-note

doc/release-note: update release-note

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #15736 from cbodley/wip-20297

rgw: forward RGWPutBucketPolicy to meta master
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #15132 from cbodley/wip-doc-rgw-remove-agent

doc/rgw: remove Federated Configuration, clean up multisite

doc/release-note: update release-note

Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

rgw/multisite: check location constraint existness

to match the behavior of AWS S3

Signed-off-by: Jiaying Ren <jiaying.ren@umcloud.com>

rgw/multisite: unify zg is_master judge

Signed-off-by: Jiaying Ren <jiaying.ren@umcloud.com>

rgw: fix bucket location s3 compatible issue

create bucket with location constraint doesn't exist in AWS S3
will response like the following:

{'data': '<?xml version="1.0" encoding="UTF-8"?>\n
<Error><Code>InvalidLocationConstraint</Code>
<Message>The specified location-constraint is not valid</Message>
<LocationConstraint>ap-southeast-4</LocationConstraint>
<RequestId>15D886DFEF49765A</RequestId>
<HostId>IroSlmBZq7iSluYIgea7y+72OXbOSmjX2eVYING0tUuGR8BAHLcYPOgUWUIJUuG1z/1FhfaxJUo=</HostId></Error>',
 'headers': {'connection': 'close',
             'content-type': 'application/xml',
             'date': 'Thu, 25 May 2017 03:45:49 GMT',
             'server': 'AmazonS3',
             'transfer-encoding': 'chunked',
             'x-amz-id-2': 'IroSlmBZq7iSluYIgea7y+72OXbOSmjX2eVYING0tUuGR8BAHLcYPOgUWUIJUuG1z/1FhfaxJUo=',
             'x-amz-request-id': '15D886DFEF49765A'},
 'reason': 'Bad Request',
 'status': 400}

Signed-off-by: Jiaying Ren <jiaying.ren@umcloud.com>

mon/OSDMonitor: batch noup/noin osds support

This is a follow-up change of https://github.com/ceph/ceph/pull/15381.
This patch also simplifies the original code logic a bit.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15658 from xiexingguo/wip-fix-reshard

os/bluestore: fix possible out of order shard(offset == 0); add sanity check

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15716 from jcsp/wip-mgr-fsmap

mgr: fix MgrStandby eating messages

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15732 from liewegas/wip-osd-boot-message

osd: do not try to boot until we've seen the first osdmap

Merge pull request #15740 from liewegas/wip-pgmap-debug

mon,mgr: print pgmap reports to debug (not cluster) log

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #15389 from gregsfortytwo/wip-18126

qa: do not restrict valgrind runs to centos

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: better debug output on unsharing blobs

Looking for http://tracker.ceph.com/issues/20227

Signed-off-by: Sage Weil <sage@redhat.com>

.gitignore: exclude rpm files

Signed-off-by: Leo Zhang <nguzcf@gmail.com>

Merge pull request #15618 from liupan1111/wip-fix-comment

tools/rados: fixed typo in help information

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15720 from ShiqiCooperation/rbdshiqi

test/unittest_librbd: remove unused variables

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr: print a more helpful error message for when users lack mgr cephx caps

Add some brief documentation on updating their caps and link to it.

Fixes: http://tracker.ceph.com/issues/20296
Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #15741 from wjwithagen/wip-wjw-freebsd-host-errno-2

common/freebsd_errno.cc: fix missing ;

common/freebsd_errno.cc: fix missing ;

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

mgr/DaemonServer: print PGMapDigest summary on send

Useful for debugging.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15495 from wjwithagen/wip-wjw-freebsd-host-errno

core: introduce (and fix) code to pass errno to other OSes

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15717 from liewegas/wip-20326

qa/tasks/ceph.py: tolerate active+clean+something

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15721 from majianpeng/bluestore-nodevice-nosymlink

os/bluestore/BlueStore: no device no symlink.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15559 from dzafman/wip-20186

osd: Check snapset for validity when selecting authoritative shard

Reviewed-by: Sage Weil <sage@redhat.com>

rgw: fix constexpr for string_size

string_size<const char*>() is no longer constexpr. added new constexpr
specialization for string_size<char[N]>()

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15552 from TsaiJin/wip-queue-scrub-with-higher-priority

osd/OSD.h: requeue the scrub job with higher priority to shorten the blocking time of related requests

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

mon/MgrStatMonitor: print PGMapDigest summary on receipt

Useful for debugging.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15722 from dmick/master

test/osd/osd-dup.sh: use wait_for_clean

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #15735 from theanalyst/rgw-uninit-atomic

rgw_rados: initialize cur_shard

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15524 from cbodley/wip-rgw-sync-debug-logging

rgw: raise debug level of meta sync logging

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #15711 from cbodley/wip-rgw-swift-default-auth-error

rgw: fix swift default auth error after auth strategy refactoring

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

test/rgw: add multisite test_set_bucket_policy

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: forward RGWPutBucketPolicy to meta master

Fixes: http://tracker.ceph.com/issues/20297
Signed-off-by: Casey Bodley <cbodley@redhat.com>

os/bluestore/BlueStore: no device no symlink.

If bluestore_block_db_create or bluestore_block_db_create is false
and we use non-exist blockdevice as db/wal. ceph-osd still work and it
use block as db/wal. This bug introduced by 316810d7e74.

If stat(symlink) error and symlink still exist, this is bug.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

osd/ECTransaction: only read partial stripes when below *original* object size

If we touch a partial stripe that is before the stripe-aligned object size
we need to read it in to do a read/modify/write that overalys the new data
onto the old.  However, if we have two writes, both touching the same
stripe but not completely covering it, and that stripe is past the original
end of file, there is no old data to read.  The previous code was using
the projected size to decide whether to do a stripe read, and would end
up trying to read data that wasn't there.

Fix by basing the stripe read decision on the original object size instead
of the projected_size.

Fixes: http://tracker.ceph.com/issues/19882
Signed-off-by: Sage Weil <sage@redhat.com>

rgw_rados: initialize cur_shard

valgrind reported an uninitialized value on use of cur_shard at a few
places, so explicitly declaring this to 0 as we are incrementing this
later

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

doc/rgw: update pool names, document namespaces

Fixes: http://tracker.ceph.com/issues/19504
Signed-off-by: Casey Bodley <cbodley@redhat.com>

doc/rgw: remove Upgrade to Older Versions of Jewel page

Signed-off-by: Casey Bodley <cbodley@redhat.com>

doc/rgw: remove Federated Configuration page

Fixes: http://tracker.ceph.com/issues/18082
Signed-off-by: Casey Bodley <cbodley@redhat.com>

doc/rgw: clean up multisite config options

remove region stuff, radosgw-agent stuff, and zone/pool stuff that is
duplicated in the multisite page

moved sync log config options into multisite section, added description
for rgw_run_sync_thread

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15678 from cbodley/wip-rgw-string-join

rgw: add variadic string join for s3 signature generation

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: remove debug log statements in RGWMetaSyncShardCR

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: raise debug level of meta sync logging

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15036 from joke-lee/master

rgw-multisite: fix the problem of rgw website configure 'RedirectAllRequestsTo' failed to sync to slave zone

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15731 from cbodley/wip-rgw-rest-client-signature

rgw: fix rest client's order of args in get_v2_signature
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

osd: do not try to boot until we've seen the first osdmap

This is actually not a change in behavior *except* we see this message
instead of

2017-06-15 18:05:29.760938 24a7f700 -1 osd.1 0 osdmap SORTBITWISE OSDMap flag is NOT set; please set it

because the OSDMap() epoch 0 does not have SORTBITWISE.

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: fix error mapping after auth strategy refactoring

the changes in bd81c216d0ef36a66c27180f13430c9702393fb6 inadvertently
changed swift's default auth error from EPERM to ACCESS. hacky fix to
change it back

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: fix rest client's order of args in get_v2_signature

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15672 from liewegas/wip-bluestore-cache-ratios

os/bluestore: handle rounding error in cache ratios

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #15718 from ztczll/zl

os/bluestore: remove unused variables

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15730 from liewegas/wip-make-check

ceph-object-corpus: mark MMDSSlaveRequest incompat change

ceph-object-corpus: mark MMDSSlaveRequest incompat change

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15251 from joscollin/wip-cleanup-redundant-headers-4

common: Remove redundant includes - 4

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #15310 from liewegas/wip-qa-scrubs

qa/tasks/ceph: raise exceptions if scrubbing fails or cannot proceed

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15657 from xiexingguo/wip-reshard-allocation

os/bluestore: fix potential access violation

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>