qa/suites/upgrade/hammer-jewel-x: upgrade all mon to luminous before osd

luminous osd requires that monmap has REQUIRE_LUMINOUS before it boots.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/suites/upgrade/hammer-jewel-x: replace kraken.yaml with luminous.yaml

* add mgr.x to roles
* to setup mgr and set the require-osd-release bit in osdmap
* do not restart an osd for waiting for healthy: the cluster is not
  healthy until the require-osd-release=luminous is set in osdmap, and a
  mgr is up and running.

Fixes: http://tracker.ceph.com/issues/20342
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15728 from majianpeng/bluefs-remove-memcopy-for-continue-memory

os/bluestore/BlueFS: Rebuild memcopy for bufferlist::page_aligned_app…

Reviewed-by: Sage Weil <sage@redhat.com>

unittest_bufferlist: test page_aligned_appender

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: replace claim_append with claim_append_piecewise to reduce memory copy.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

common/buffer: add function  bufferlist::claim_append_piecewise(list&  bl).

This only useful for bl is bufferlist::page_aligned_appender. Using
this function can remove memcopy for continue ptrs.
Because page_aligned_appender::flush will split a ptr into two or
more ptrs. For this case, rebuild_aligned_size_and_memory can't handle,
it  will rebuild.

For example
a=bl.get_page_aligned_appender(1);
a.append(3K)
a.flush();
t.claim_append(bl);
a.append(1K);
a.flush();
t.claim_append(bl);

dst.claim_append(t);
//3K and 1K ptr are continue in memory. But they are two ptrs..
dst.is_aligned_size_and_memory(4096,4096) is false.

We add new function claim_append_piecewise() to specially
handle this case.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #15260 from shashalu/fix-secondary-zonegroup-cors

rgw: update bucket cors in secondary zonegroup should forward to master

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15709 from jcsp/wip-dashboard-loglast

mgr/dashboard: load log lines on startup, split out audit log

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15750 from xiexingguo/wip-bluestore-20170619

os/bluestore: refactor BlueStore::_do_write;  kill dead ExtentMap::find_lextent() method

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #15766 from liewegas/wip-20227-fix

os/bluestore: fix unsharing blob dirty_range args

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15014 from zhangsw/fix-rgw-versionIdMarker-bug

rgw: VersionIdMarker and NextVersionIdMarker should be returned when listing object versions

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14500 from zhangsw/fix-rgw-multipart-index-bug

rgw: fix the bug that part's index can't be removed after completing multipart upload

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #15771 from fangyuxiangGL/cleanup_rgw_admin

rgw: cleanup unused codes in rgw_admin.cc

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15768 from ukernel/wip-objecter-inflight-ops

osdc/Objecter: fix inflight_ops update

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15737 from wjwithagen/wip-wjw-test_rgw_string

rgw/rgw_string.h: FreeBSD would like errno.h included

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15782 from cbodley/wip-rgw-vstart-s3tests

vstart: add rgw configuration needed to pass all s3tests

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

vstart: add rgw configuration needed to pass all s3tests

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15751 from xiexingguo/wip-add-state-name

src/common/ceph_string: stringify new osd states

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14703 from zhangsw/feature-lifecycle-deletemarker

rgw: add support for delete marker expiration in s3 lifecycle.

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #15540 from joscollin/wip-cleanup-test-commented-code-2

test/osd: Removed Commented Code - 2

Merge pull request #15770 from tchaikov/wip-test-test_mon_crush

qa/workunits/cephtool/test.sh: only include last line for epoch

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15733 from wjwithagen/wip-wjw-freebsd-ceph-disk-service

ceph-disk: command invocation needs all fields separate

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/osd: Removed Commented Code - 2

Removed commented code found in test/osd/.

Fixes: http://tracker.ceph.com/issues/20207 (Partially)
Signed-off-by: Jos Collin <jcollin@redhat.com>

ceph-disk: command invocation needs all fields separate

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #15592 from david-z/wip-fix-dump-cache

mds: explicitly output error msg for dump cache asok command

Reviewed-by: John Spray <john.spray@redhat.com>

qa/workunits/cephtool/test.sh: only include last line for epoch

/home/jenkins-build/build/workspace/ceph-pull-requests-arm64/qa/workunits/cephtool/test.sh:1606: test_mon_crush:  epoch='2017-06-20 04:44:52.862459 ffffad4d0200 -1 asok(0xffffa8000b10) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: The UNIX domain socket path /home/jenkins-build/build/workspace/ceph-pull-requests-arm64/build/src/test/td/t-7202/out/client.admin.48876.asok is too long! The maximum length on this system is 107
12'
/home/jenkins-build/build/workspace/ceph-pull-requests-arm64/qa/workunits/cephtool/test.sh:1607: test_mon_crush:  '[' '2017-06-20 04:44:52.862459 ffffad4d0200 -1 asok(0xffffa8000b10) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: The UNIX domain socket path /home/jenkins-build/build/workspace/ceph-pull-requests-arm64/build/src/test/td/t-7202/out/client.admin.48876.asok is too long! The maximum length on this system is 107
12' -gt 1 ']'
/home/jenkins-build/build/workspace/ceph-pull-requests-arm64/qa/workunits/cephtool/test.sh: line 1607: [: 2017-06-20 04:44:52.862459 ffffad4d0200 -1 asok(0xffffa8000b10) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: The UNIX domain socket path /home/jenkins-build/build/workspace/ceph-pull-requests-arm64/build/src/test/td/t-7202/out/client.admin.48876.asok is too long! The maximum length on this system is 107
12: integer expression expected

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: cleanup unused codes in rgw_admin.cc

Signed-off-by: fang yuxiang <fang.yuxiang@eisoo.com>

Merge pull request #15742 from wjwithagen/wip-wjw-freebsd-host-errno-3

common/freebsd_errno.cc: fixed again a stupid typo

Reviewed-by: Kefu Chai <kchai@redhat.com>

osdc/Objecter: fix inflight_ops update

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #15651 from zealoussnow/wip-0613

config_opt: use bool instead of int for the default value of filestore_debug_omap_check

Merge pull request #15676 from Linbing/wip-mon-delete-unused-config

mon: delete unused config opts of mon_sync_fs_threshold

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15697 from gregsfortytwo/wip-mgr-eaccss-20296

mgr: print a more helpful error message for when users lack mgr ceph caps

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15744 from badone/wip-CID-1412579

ceph-dencoder: Silence coverity CID 1412579

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15637 from liewegas/wip-point-to-point

qa/upgrade/jewel-x/point-to-point: add a mgr during final upgrade

os/bluestore: fix unsharing blob dirty_range args

Fixes: http://tracker.ceph.com/issues/20227
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15365 from ShiqiCooperation/newshiqi

test/unittest_bluefs: When fsync ret is less than 0, fsync can not be…

Reviewed-by: Varada Kari <varada.kari@gmail.com>

Merge pull request #15533 from liewegas/wip-crush-version

mon,osd: add crush_version to OSDMap, and allow crush map updates to gate on crush_version

Reviewed-by: Loic Dachary <loic@dachary.org>

Merge pull request #15685 from wjwithagen/wip-wjw-freebsd-ENODATA-2

core/common: Fix ENODATA for FreeBSD with compat.h

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #15710 from liewegas/wip-20273

osd: bail from _committed_osd_maps inside osd_lock

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rgw_string.h: FreeBSD would like errno.h included

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #15333 from mikulely/validate-bucket-location

rgw/multisite: validate bucket location during bucket creation

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15633 from cloudbau/reduce-rgw-copy-loglevel

rgw: do not log debug output at level 0

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15598 from aclamk/encryption_refactor

radosgw, crypto: simplified code in handle_data functions

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15599 from theanalyst/rgw/fix/bl-cstr

rgw: don't specify a length when converting bl -> string

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15756 from rzarzynski/wip-rgw-cleanup-swift-errmapping

rgw: clean-up error mapping in Swift's authentication strategy.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15677 from zhangsw/fix-rgw-lifecycle-bug

rgw: lifecycle thread shouldn't process the bucket which has been deleted

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #15682 from mikulely/rgw-putobj-params

rgw: abort early when s->length empty during putobj

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15747 from liewegas/wip-scrub-retry

qa/tasks/ceph: osd_scrub_pgs: reissue scrub requests in loop

qa/suites/rados/singleton-nomsgr/multi-backfill-reject: clean up

Set pool size back to 2 so we don't have to have backfill
complete (despite rejection probability) in order to get back to
healthy.  This way we scrub on cleanup.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15712 from liewegas/wip-19882

osd/ECTransaction: only read partial stripes when below *original* object size

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/suites/rados/singleton-nomsgr/full-tiering: unset quota at end

If we leave the quota set, the proxied ops will block
indefinitely, which will block scrubbing on the cache tier pgs
indefinitely.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/resolve_stuck_peering: start osd at end

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15679 from guihecheng/rgw_file-fix-setattr

rgw_file: fix assert upon setattr on bucket

Merge pull request #15660 from fangyuxiangGL/meta_sync_crash

rgw: meta sync thread crash at RGWMetaSyncShardCR

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/workunits/cephtool/test.sh: silence dev warnings

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph: osd_scrub_pgs: reissue scrub requests in loop

The scrub commands are not reliable: if the OSD doesn't happen to
be connected at the time the command is issued it may not get
delivered.  Re-request scrubs for each PG that has not yet been
scrubbed so that we don't wait forever when the original request
is dropped.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph: osd_scrub_pgs: tolerate down osd at initial scrub time

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15738 from cbodley/wip-rgw-string-constexpr

rgw: fix constexpr for string_size in clang

Reviewed By: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #15544 from taodd/master

mds:  Client syncfs is slow (waits for next MDS tick)

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #15746 from liewegas/wip-20227

os/bluestore: better debug output on unsharing blobs

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

client: signal MDS to flush log when doing a syncfs

Fixes: http://tracker.ceph.com/issues/20129
Signed-off-by: dongdong tao <tdd21151186@gmail.com>

Client: signal MDS to flush log when doing a syncfs
Fixes: http://tracker.ceph.com/issues/20129
Signed-off-by: dongdong tao <tdd21151186@gmail.com>

Merge pull request #15745 from zealoussnow/wip-0617

.gitignore: exclude rpm files

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #15481 from jcsp/wip-20178

client: avoid returning negative space available

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

os/bluestore: kill dead ExtentMap::find_lextent() method

Which currently has no consumers.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: refactor BlueStore::_do_write

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15614 from jcsp/wip-jt-help

tools/cephfs: fix cephfs-journal-tool --help

Reviewed-by: Douglas Fuller <dfuller@redhat.com>

rgw: clean-up error mapping in Swift's authentication strategy.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #15749 from Adirl/wip

msg/async/rdma: handle buffers after close msg

Reviewed-by: Haomai Wang <haomai@xsky.com>

src/common/ceph_string: stringify new osd states

In https://github.com/ceph/ceph/pull/15725 I introduce
two more osd states.

This patch pulls them in.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

rgw: meta sync thread crash at RGWMetaSyncShardCR

Fixes: http://tracker.ceph.com/issues/20251
Signed-off-by: fang yuxiang fang.yuxiang@eisoo.com

rgw: add support for delete marker expiration in s3 lifecycle.

Fixes: http://tracker.ceph.com/issues/19730
Signed-off-by: Zhang Shaowen <zhangshaowen@cmss.chinamobile.com>

rgw: update bucket cors in secondary zonegroup should forward to master

pr#14082 is incomplete, cors should be redirect to master

Fixes: http://tracker.ceph.com/issues/16888
Signed-off-by: Shasha Lu <lu.shasha@eisoo.com>

ceph-dencoder: Silence coverity CID 1412579

141         if (i == 0)
142           i = m_list.size();
143         if ((i == 0) || (i > m_list.size()))
144           return "invalid id for generated object";
145         typename list<T*>::iterator p = m_list.begin();
146         for (i--; i > 0 && p != m_list.end(); ++p, --i) ;
>>>     CID 1412579:  API usage errors  (INVALIDATE_ITERATOR)
>>>     Dereferencing iterator "p" though it is already past the end
of its container.
147         m_object = *p;
148         return string();
149       }
150
151       bool is_deterministic() override {
152         return !nondeterministic;

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #15725 from xiexingguo/wip-noupin

mon/OSDMonitor: batch noup/noin osds support

Reviewed-by: Sage Weil <sage@redhat.com>

msg/async/rdma: remove assert from srq destroy in ~Infiniband

issue:1040242

Even though ibv_destroy_srq does not return 0, all resources
are destroyed. No need to assert.

Change-Id: I96562130d22352390f9f7f9a3e0ddbe77a82d8f4
Signed-off-by: DanielBar-On <danielbo@mellanox.com>

msg/async/rdma: handle buffers after close msg

fixing an issue of OSD’s going down during recovery process while traffic
is running in the background.
fix is making sure that messages previously pulled from the cqe and are
now stored in the socket queue “buffers” will always be handled and fd
will be notified even if cqe is empty.

issue: 1025030

Change-Id: I6ced6b8ee0152e233632700ccc9ec19970e66b6a
Signed-off-by: Alexander Mikheev <alexm@mellanox.com>
Signed-off-by: Adir Lev <adirl@mellanox.com>

Merge pull request #15748 from Liuchang0812/wip-fix-release-note

doc/release-note: update release-note

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #15736 from cbodley/wip-20297

rgw: forward RGWPutBucketPolicy to meta master
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #15132 from cbodley/wip-doc-rgw-remove-agent

doc/rgw: remove Federated Configuration, clean up multisite

doc/release-note: update release-note

Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

rgw/multisite: check location constraint existness

to match the behavior of AWS S3

Signed-off-by: Jiaying Ren <jiaying.ren@umcloud.com>

rgw/multisite: unify zg is_master judge

Signed-off-by: Jiaying Ren <jiaying.ren@umcloud.com>

rgw: fix bucket location s3 compatible issue

create bucket with location constraint doesn't exist in AWS S3
will response like the following:

{'data': '<?xml version="1.0" encoding="UTF-8"?>\n
<Error><Code>InvalidLocationConstraint</Code>
<Message>The specified location-constraint is not valid</Message>
<LocationConstraint>ap-southeast-4</LocationConstraint>
<RequestId>15D886DFEF49765A</RequestId>
<HostId>IroSlmBZq7iSluYIgea7y+72OXbOSmjX2eVYING0tUuGR8BAHLcYPOgUWUIJUuG1z/1FhfaxJUo=</HostId></Error>',
 'headers': {'connection': 'close',
             'content-type': 'application/xml',
             'date': 'Thu, 25 May 2017 03:45:49 GMT',
             'server': 'AmazonS3',
             'transfer-encoding': 'chunked',
             'x-amz-id-2': 'IroSlmBZq7iSluYIgea7y+72OXbOSmjX2eVYING0tUuGR8BAHLcYPOgUWUIJUuG1z/1FhfaxJUo=',
             'x-amz-request-id': '15D886DFEF49765A'},
 'reason': 'Bad Request',
 'status': 400}

Signed-off-by: Jiaying Ren <jiaying.ren@umcloud.com>

mon/OSDMonitor: batch noup/noin osds support

This is a follow-up change of https://github.com/ceph/ceph/pull/15381.
This patch also simplifies the original code logic a bit.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15658 from xiexingguo/wip-fix-reshard

os/bluestore: fix possible out of order shard(offset == 0); add sanity check

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15716 from jcsp/wip-mgr-fsmap

mgr: fix MgrStandby eating messages

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15732 from liewegas/wip-osd-boot-message

osd: do not try to boot until we've seen the first osdmap

Merge pull request #15740 from liewegas/wip-pgmap-debug

mon,mgr: print pgmap reports to debug (not cluster) log

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #15389 from gregsfortytwo/wip-18126

qa: do not restrict valgrind runs to centos

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: better debug output on unsharing blobs

Looking for http://tracker.ceph.com/issues/20227

Signed-off-by: Sage Weil <sage@redhat.com>

.gitignore: exclude rpm files

Signed-off-by: Leo Zhang <nguzcf@gmail.com>

Merge pull request #15618 from liupan1111/wip-fix-comment

tools/rados: fixed typo in help information

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15720 from ShiqiCooperation/rbdshiqi

test/unittest_librbd: remove unused variables

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr: print a more helpful error message for when users lack mgr cephx caps

Add some brief documentation on updating their caps and link to it.

Fixes: http://tracker.ceph.com/issues/20296
Signed-off-by: Greg Farnum <gfarnum@redhat.com>

common/freebsd_errno.cc: fixed again a stupid typo

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #15741 from wjwithagen/wip-wjw-freebsd-host-errno-2

common/freebsd_errno.cc: fix missing ;

common/freebsd_errno.cc: fix missing ;

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

mgr/DaemonServer: print PGMapDigest summary on send

Useful for debugging.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15495 from wjwithagen/wip-wjw-freebsd-host-errno

core: introduce (and fix) code to pass errno to other OSes

Reviewed-by: Kefu Chai <kchai@redhat.com>