Merge pull request #14500 from zhangsw/fix-rgw-multipart-index-bug

rgw: fix the bug that part's index can't be removed after completing multipart upload

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #15771 from fangyuxiangGL/cleanup_rgw_admin

rgw: cleanup unused codes in rgw_admin.cc

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15768 from ukernel/wip-objecter-inflight-ops

osdc/Objecter: fix inflight_ops update

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15737 from wjwithagen/wip-wjw-test_rgw_string

rgw/rgw_string.h: FreeBSD would like errno.h included

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15782 from cbodley/wip-rgw-vstart-s3tests

vstart: add rgw configuration needed to pass all s3tests

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

vstart: add rgw configuration needed to pass all s3tests

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15751 from xiexingguo/wip-add-state-name

src/common/ceph_string: stringify new osd states

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14703 from zhangsw/feature-lifecycle-deletemarker

rgw: add support for delete marker expiration in s3 lifecycle.

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #15540 from joscollin/wip-cleanup-test-commented-code-2

test/osd: Removed Commented Code - 2

Merge pull request #15770 from tchaikov/wip-test-test_mon_crush

qa/workunits/cephtool/test.sh: only include last line for epoch

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15733 from wjwithagen/wip-wjw-freebsd-ceph-disk-service

ceph-disk: command invocation needs all fields separate

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/osd: Removed Commented Code - 2

Removed commented code found in test/osd/.

Fixes: http://tracker.ceph.com/issues/20207 (Partially)
Signed-off-by: Jos Collin <jcollin@redhat.com>

ceph-disk: command invocation needs all fields separate

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #15592 from david-z/wip-fix-dump-cache

mds: explicitly output error msg for dump cache asok command

Reviewed-by: John Spray <john.spray@redhat.com>

qa/workunits/cephtool/test.sh: only include last line for epoch

/home/jenkins-build/build/workspace/ceph-pull-requests-arm64/qa/workunits/cephtool/test.sh:1606: test_mon_crush:  epoch='2017-06-20 04:44:52.862459 ffffad4d0200 -1 asok(0xffffa8000b10) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: The UNIX domain socket path /home/jenkins-build/build/workspace/ceph-pull-requests-arm64/build/src/test/td/t-7202/out/client.admin.48876.asok is too long! The maximum length on this system is 107
12'
/home/jenkins-build/build/workspace/ceph-pull-requests-arm64/qa/workunits/cephtool/test.sh:1607: test_mon_crush:  '[' '2017-06-20 04:44:52.862459 ffffad4d0200 -1 asok(0xffffa8000b10) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: The UNIX domain socket path /home/jenkins-build/build/workspace/ceph-pull-requests-arm64/build/src/test/td/t-7202/out/client.admin.48876.asok is too long! The maximum length on this system is 107
12' -gt 1 ']'
/home/jenkins-build/build/workspace/ceph-pull-requests-arm64/qa/workunits/cephtool/test.sh: line 1607: [: 2017-06-20 04:44:52.862459 ffffad4d0200 -1 asok(0xffffa8000b10) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: The UNIX domain socket path /home/jenkins-build/build/workspace/ceph-pull-requests-arm64/build/src/test/td/t-7202/out/client.admin.48876.asok is too long! The maximum length on this system is 107
12: integer expression expected

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: cleanup unused codes in rgw_admin.cc

Signed-off-by: fang yuxiang <fang.yuxiang@eisoo.com>

Merge pull request #15742 from wjwithagen/wip-wjw-freebsd-host-errno-3

common/freebsd_errno.cc: fixed again a stupid typo

Reviewed-by: Kefu Chai <kchai@redhat.com>

osdc/Objecter: fix inflight_ops update

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #15651 from zealoussnow/wip-0613

config_opt: use bool instead of int for the default value of filestore_debug_omap_check

Merge pull request #15676 from Linbing/wip-mon-delete-unused-config

mon: delete unused config opts of mon_sync_fs_threshold

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15697 from gregsfortytwo/wip-mgr-eaccss-20296

mgr: print a more helpful error message for when users lack mgr ceph caps

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15744 from badone/wip-CID-1412579

ceph-dencoder: Silence coverity CID 1412579

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15637 from liewegas/wip-point-to-point

qa/upgrade/jewel-x/point-to-point: add a mgr during final upgrade

Merge pull request #15365 from ShiqiCooperation/newshiqi

test/unittest_bluefs: When fsync ret is less than 0, fsync can not be…

Reviewed-by: Varada Kari <varada.kari@gmail.com>

Merge pull request #15533 from liewegas/wip-crush-version

mon,osd: add crush_version to OSDMap, and allow crush map updates to gate on crush_version

Reviewed-by: Loic Dachary <loic@dachary.org>

Merge pull request #15685 from wjwithagen/wip-wjw-freebsd-ENODATA-2

core/common: Fix ENODATA for FreeBSD with compat.h

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #15710 from liewegas/wip-20273

osd: bail from _committed_osd_maps inside osd_lock

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rgw_string.h: FreeBSD would like errno.h included

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #15333 from mikulely/validate-bucket-location

rgw/multisite: validate bucket location during bucket creation

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15633 from cloudbau/reduce-rgw-copy-loglevel

rgw: do not log debug output at level 0

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15598 from aclamk/encryption_refactor

radosgw, crypto: simplified code in handle_data functions

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15599 from theanalyst/rgw/fix/bl-cstr

rgw: don't specify a length when converting bl -> string

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15756 from rzarzynski/wip-rgw-cleanup-swift-errmapping

rgw: clean-up error mapping in Swift's authentication strategy.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15677 from zhangsw/fix-rgw-lifecycle-bug

rgw: lifecycle thread shouldn't process the bucket which has been deleted

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #15682 from mikulely/rgw-putobj-params

rgw: abort early when s->length empty during putobj

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15747 from liewegas/wip-scrub-retry

qa/tasks/ceph: osd_scrub_pgs: reissue scrub requests in loop

qa/suites/rados/singleton-nomsgr/multi-backfill-reject: clean up

Set pool size back to 2 so we don't have to have backfill
complete (despite rejection probability) in order to get back to
healthy.  This way we scrub on cleanup.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15712 from liewegas/wip-19882

osd/ECTransaction: only read partial stripes when below *original* object size

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/suites/rados/singleton-nomsgr/full-tiering: unset quota at end

If we leave the quota set, the proxied ops will block
indefinitely, which will block scrubbing on the cache tier pgs
indefinitely.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/resolve_stuck_peering: start osd at end

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15679 from guihecheng/rgw_file-fix-setattr

rgw_file: fix assert upon setattr on bucket

Merge pull request #15660 from fangyuxiangGL/meta_sync_crash

rgw: meta sync thread crash at RGWMetaSyncShardCR

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/workunits/cephtool/test.sh: silence dev warnings

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph: osd_scrub_pgs: reissue scrub requests in loop

The scrub commands are not reliable: if the OSD doesn't happen to
be connected at the time the command is issued it may not get
delivered.  Re-request scrubs for each PG that has not yet been
scrubbed so that we don't wait forever when the original request
is dropped.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph: osd_scrub_pgs: tolerate down osd at initial scrub time

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15738 from cbodley/wip-rgw-string-constexpr

rgw: fix constexpr for string_size in clang

Reviewed By: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #15544 from taodd/master

mds:  Client syncfs is slow (waits for next MDS tick)

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #15746 from liewegas/wip-20227

os/bluestore: better debug output on unsharing blobs

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

client: signal MDS to flush log when doing a syncfs

Fixes: http://tracker.ceph.com/issues/20129
Signed-off-by: dongdong tao <tdd21151186@gmail.com>

Client: signal MDS to flush log when doing a syncfs
Fixes: http://tracker.ceph.com/issues/20129
Signed-off-by: dongdong tao <tdd21151186@gmail.com>

Merge pull request #15745 from zealoussnow/wip-0617

.gitignore: exclude rpm files

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #15481 from jcsp/wip-20178

client: avoid returning negative space available

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #15614 from jcsp/wip-jt-help

tools/cephfs: fix cephfs-journal-tool --help

Reviewed-by: Douglas Fuller <dfuller@redhat.com>

rgw: clean-up error mapping in Swift's authentication strategy.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #15749 from Adirl/wip

msg/async/rdma: handle buffers after close msg

Reviewed-by: Haomai Wang <haomai@xsky.com>

src/common/ceph_string: stringify new osd states

In https://github.com/ceph/ceph/pull/15725 I introduce
two more osd states.

This patch pulls them in.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

rgw: meta sync thread crash at RGWMetaSyncShardCR

Fixes: http://tracker.ceph.com/issues/20251
Signed-off-by: fang yuxiang fang.yuxiang@eisoo.com

rgw: add support for delete marker expiration in s3 lifecycle.

Fixes: http://tracker.ceph.com/issues/19730
Signed-off-by: Zhang Shaowen <zhangshaowen@cmss.chinamobile.com>

ceph-dencoder: Silence coverity CID 1412579

141         if (i == 0)
142           i = m_list.size();
143         if ((i == 0) || (i > m_list.size()))
144           return "invalid id for generated object";
145         typename list<T*>::iterator p = m_list.begin();
146         for (i--; i > 0 && p != m_list.end(); ++p, --i) ;
>>>     CID 1412579:  API usage errors  (INVALIDATE_ITERATOR)
>>>     Dereferencing iterator "p" though it is already past the end
of its container.
147         m_object = *p;
148         return string();
149       }
150
151       bool is_deterministic() override {
152         return !nondeterministic;

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #15725 from xiexingguo/wip-noupin

mon/OSDMonitor: batch noup/noin osds support

Reviewed-by: Sage Weil <sage@redhat.com>

msg/async/rdma: remove assert from srq destroy in ~Infiniband

issue:1040242

Even though ibv_destroy_srq does not return 0, all resources
are destroyed. No need to assert.

Change-Id: I96562130d22352390f9f7f9a3e0ddbe77a82d8f4
Signed-off-by: DanielBar-On <danielbo@mellanox.com>

msg/async/rdma: handle buffers after close msg

fixing an issue of OSD’s going down during recovery process while traffic
is running in the background.
fix is making sure that messages previously pulled from the cqe and are
now stored in the socket queue “buffers” will always be handled and fd
will be notified even if cqe is empty.

issue: 1025030

Change-Id: I6ced6b8ee0152e233632700ccc9ec19970e66b6a
Signed-off-by: Alexander Mikheev <alexm@mellanox.com>
Signed-off-by: Adir Lev <adirl@mellanox.com>

Merge pull request #15748 from Liuchang0812/wip-fix-release-note

doc/release-note: update release-note

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #15736 from cbodley/wip-20297

rgw: forward RGWPutBucketPolicy to meta master
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #15132 from cbodley/wip-doc-rgw-remove-agent

doc/rgw: remove Federated Configuration, clean up multisite

doc/release-note: update release-note

Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

rgw/multisite: check location constraint existness

to match the behavior of AWS S3

Signed-off-by: Jiaying Ren <jiaying.ren@umcloud.com>

rgw/multisite: unify zg is_master judge

Signed-off-by: Jiaying Ren <jiaying.ren@umcloud.com>

rgw: fix bucket location s3 compatible issue

create bucket with location constraint doesn't exist in AWS S3
will response like the following:

{'data': '<?xml version="1.0" encoding="UTF-8"?>\n
<Error><Code>InvalidLocationConstraint</Code>
<Message>The specified location-constraint is not valid</Message>
<LocationConstraint>ap-southeast-4</LocationConstraint>
<RequestId>15D886DFEF49765A</RequestId>
<HostId>IroSlmBZq7iSluYIgea7y+72OXbOSmjX2eVYING0tUuGR8BAHLcYPOgUWUIJUuG1z/1FhfaxJUo=</HostId></Error>',
 'headers': {'connection': 'close',
             'content-type': 'application/xml',
             'date': 'Thu, 25 May 2017 03:45:49 GMT',
             'server': 'AmazonS3',
             'transfer-encoding': 'chunked',
             'x-amz-id-2': 'IroSlmBZq7iSluYIgea7y+72OXbOSmjX2eVYING0tUuGR8BAHLcYPOgUWUIJUuG1z/1FhfaxJUo=',
             'x-amz-request-id': '15D886DFEF49765A'},
 'reason': 'Bad Request',
 'status': 400}

Signed-off-by: Jiaying Ren <jiaying.ren@umcloud.com>

mon/OSDMonitor: batch noup/noin osds support

This is a follow-up change of https://github.com/ceph/ceph/pull/15381.
This patch also simplifies the original code logic a bit.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15658 from xiexingguo/wip-fix-reshard

os/bluestore: fix possible out of order shard(offset == 0); add sanity check

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15716 from jcsp/wip-mgr-fsmap

mgr: fix MgrStandby eating messages

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15732 from liewegas/wip-osd-boot-message

osd: do not try to boot until we've seen the first osdmap

Merge pull request #15740 from liewegas/wip-pgmap-debug

mon,mgr: print pgmap reports to debug (not cluster) log

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #15389 from gregsfortytwo/wip-18126

qa: do not restrict valgrind runs to centos

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: better debug output on unsharing blobs

Looking for http://tracker.ceph.com/issues/20227

Signed-off-by: Sage Weil <sage@redhat.com>

.gitignore: exclude rpm files

Signed-off-by: Leo Zhang <nguzcf@gmail.com>

Merge pull request #15618 from liupan1111/wip-fix-comment

tools/rados: fixed typo in help information

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15720 from ShiqiCooperation/rbdshiqi

test/unittest_librbd: remove unused variables

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr: print a more helpful error message for when users lack mgr cephx caps

Add some brief documentation on updating their caps and link to it.

Fixes: http://tracker.ceph.com/issues/20296
Signed-off-by: Greg Farnum <gfarnum@redhat.com>

common/freebsd_errno.cc: fixed again a stupid typo

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #15741 from wjwithagen/wip-wjw-freebsd-host-errno-2

common/freebsd_errno.cc: fix missing ;

common/freebsd_errno.cc: fix missing ;

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

mgr/DaemonServer: print PGMapDigest summary on send

Useful for debugging.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15495 from wjwithagen/wip-wjw-freebsd-host-errno

core: introduce (and fix) code to pass errno to other OSes

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15717 from liewegas/wip-20326

qa/tasks/ceph.py: tolerate active+clean+something

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15721 from majianpeng/bluestore-nodevice-nosymlink

os/bluestore/BlueStore: no device no symlink.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15559 from dzafman/wip-20186

osd: Check snapset for validity when selecting authoritative shard

Reviewed-by: Sage Weil <sage@redhat.com>

rgw: fix constexpr for string_size

string_size<const char*>() is no longer constexpr. added new constexpr
specialization for string_size<char[N]>()

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15552 from TsaiJin/wip-queue-scrub-with-higher-priority

osd/OSD.h: requeue the scrub job with higher priority to shorten the blocking time of related requests

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

mon/MgrStatMonitor: print PGMapDigest summary on receipt

Useful for debugging.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15722 from dmick/master

test/osd/osd-dup.sh: use wait_for_clean

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #15735 from theanalyst/rgw-uninit-atomic

rgw_rados: initialize cur_shard

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15524 from cbodley/wip-rgw-sync-debug-logging

rgw: raise debug level of meta sync logging

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #15711 from cbodley/wip-rgw-swift-default-auth-error

rgw: fix swift default auth error after auth strategy refactoring

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

test/rgw: add multisite test_set_bucket_policy

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: forward RGWPutBucketPolicy to meta master

Fixes: http://tracker.ceph.com/issues/20297
Signed-off-by: Casey Bodley <cbodley@redhat.com>

os/bluestore/BlueStore: no device no symlink.

If bluestore_block_db_create or bluestore_block_db_create is false
and we use non-exist blockdevice as db/wal. ceph-osd still work and it
use block as db/wal. This bug introduced by 316810d7e74.

If stat(symlink) error and symlink still exist, this is bug.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

osd/ECTransaction: only read partial stripes when below *original* object size

If we touch a partial stripe that is before the stripe-aligned object size
we need to read it in to do a read/modify/write that overalys the new data
onto the old.  However, if we have two writes, both touching the same
stripe but not completely covering it, and that stripe is past the original
end of file, there is no old data to read.  The previous code was using
the projected size to decide whether to do a stripe read, and would end
up trying to read data that wasn't there.

Fix by basing the stripe read decision on the original object size instead
of the projected_size.

Fixes: http://tracker.ceph.com/issues/19882
Signed-off-by: Sage Weil <sage@redhat.com>

rgw_rados: initialize cur_shard

valgrind reported an uninitialized value on use of cur_shard at a few
places, so explicitly declaring this to 0 as we are incrementing this
later

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>