rgw: encryption add exception handling for from_base64 on bad input

If calling from_base64 with charactors that is not base64 encoded,
rgw will crash.

Signed-off-by: Enming Zhang <enming.zhang@umcloud.com>

rgw: encryption fix the issue when not provide encryption mode

Now, in RGW, if someone want to upload an object using server-side
encryption with providing customer key or kms key id, but not
specify the encryption mode in the
"x-amz-server-side-encryption-customer-algorithm" or
"x-amz-server-side-encryption", the object will be uploaded
successfully without encryption.

This is not a correct way to deal with it. It is better to
return error.

Fixes: http://tracker.ceph.com/issues/21581
Signed-off-by: Enming Zhang <enming.zhang@umcloud.com>

rgw: encryption SSE-KMS add the details of error msg in response

Signed-off-by: Enming Zhang <enming.zhang@umcloud.com>

rgw: encryption SSE-C add the details of error msg in response

Signed-off-by: Enming Zhang <enming.zhang@umcloud.com>

Merge PR #17205 into master

* refs/remotes/upstream/pull/17205/head:
client: fix locking in Client::getcwd

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #17204 into master

* refs/remotes/upstream/pull/17204/head:
fuse: use c++ allocations for group list

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #17199 into master

* refs/remotes/upstream/pull/17199/head:
client: fix compat version on MStatfs

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Douglas Fuller <dfuller@redhat.com>

Merge pull request #17121 from zmedico/PGPool-update-optimize-with-interval_set-swap

PGPool::update: optimize with interval_set.swap

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17014 from xiexingguo/wip-various-weight-set-fix

crush: various weight-set fixes

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17099 from xiexingguo/wip-stuck-op-count

mon/PGMap: fix "0 stuck requests are blocked > 4096 sec" warn

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17203 from liewegas/wip-jewel-x

qa/suites/upgarde/jewel-x/parallel: tolerate mgr warning

Merge pull request #17207 from ceph/wip-bz1484002

ceph-volume: use unique logical volumes

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #12429 from yehudasa/wip-meta-stream

rgw: stream metadata full sync init

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #16926 from fangyuxiangGL/infinitely-send-data-log-list

rgw: break sending data-log list infinitely

Merge pull request #16612 from cbodley/wip-20668

rgw: fixes for multisite replication of encrypted objects

ceph-volume tests centos7 use the new ansible syntax for lvm

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests use the new ansible syntax for lvm

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests create tests for the get_lv helper method

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests create tests for the new arg validator

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util create a validator module for argparse

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.prepare enforce usage of vg/lv when preparing lvm devices

Signed-off-by: Alfredo Deza <adeza@redhat.com>

qa/suites/upgrade/jewel-x/parallel: tolerate OBJECT_MISPLACED

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/upgarde/jewel-x/parallel: tolerate mgr warning

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #17175 into master

* refs/remotes/upstream/pull/17175/head:
client: add blank space between function name and inode number for log

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #17150 into master

* refs/remotes/upstream/pull/17150/head:
doc/cephfs: s/rados_max_object_size/osd_max_object_size

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

client: fix locking in Client::getcwd

Currently, it doesn't take the client_lock at all, which is problematic
as make_request may very well end up unlocking it. Rename the current
function to _getcwd, and add a new getcwd wrapper that takes the mutex
before calling _getcwd.

This fixes: http://tracker.ceph.com/issues/21082

Signed-off-by: Jeff Layton <jlayton@redhat.com>

ceph-volume lvm.common update help values for vg/lv usage

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.create update docstring for vg/lv usage

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.prepare update docstring for vg/lv usage

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #17162 from cbodley/wip-qa-rgw-pool-application

qa/rgw: use 'ceph osd pool application enable' on created pools

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge PR #17163 into master

* refs/remotes/upstream/pull/17163/head:
mon: get writeable osdmap for added data pool

Reviewed-by: Sage Weil <sage@redhat.com>

fuse: use c++ allocations for group list

Valgrind is unhappy about our turning on supplimentary group handling
with fuse by default. The problem is that we end up calling delete to
free the supplimentary gids list, but fuse uses malloc to allocate it.

Note that I was initially concerned that I needed to use malloc and
free there to handle the case of userland calling ceph_userperm_new,
but we leave freeing the pointer up to the caller in that case.

Convert fuse to use new/delete to allocate and free the group lists
instead.

Tracker: http://tracker.ceph.com/issues/21065
Signed-off-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #17075 from ivancich/wip-adjust-dup-ops-options

osd: adjust osd_min_pg_log_entries

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

client: fix compat version on MStatfs

Fixes: http://tracker.ceph.com/issues/21078
Signed-off-by: John Spray <john.spray@redhat.com>

Merge PR #16959 into master

* refs/remotes/upstream/pull/16959/head:
client: skip lookupname if writing to unlinked file

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #17053 into master

* refs/remotes/upstream/pull/17053/head:
qa: add chown test clearing S_ISGID and S_ISUID
ceph-fuse: load supplementary groups by default
client: clear suid/sgid bits on non-zero write
client: add missing space in debug msg
cephfs: clear suid/sgid if regular file is exe
client: refactor clear set uid/gid if -1

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #17122 into master

* refs/remotes/upstream/pull/17122/head:
mds: check cap string only if !allow_all

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #17111 into master

* refs/remotes/upstream/pull/17111/head:
qa: add health whitelist for kcephfs suite

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #17054 into master

* refs/remotes/upstream/pull/17054/head:
mds: Initializing uninitiazed members SnapServer

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #17184 from PCzhangPC/rmconv

rbd:rm unnecessary conversion from string to char* in image-meta function

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17186 from trociny/wip-librados_test_stub-zero-snap

test/librados_test_stub: pass snap context to zero op

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd:rm unnecessary conversion from string to char* in image-meta function

the APIs in librbd receive string type as parameter, so there is no need to convert string to char* in image-meta function

Signed-off-by: PCzhangPC <pengcheng.zhang@easystack.cn>

test/librados_test_stub: pass snap context to zero op

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #17154 from ktdreyer/doc-cat

doc: eliminate useless cat statement

Reviewed-by: Kefu Chai <kchai@redhat.com>

mon: get writeable osdmap for added data pool

Continuation of: 435717791ec499f71c9d1485b1e4e63239a343e2

Fixes: http://tracker.ceph.com/issues/21064
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

client: add blank space between function name and inode number for log

Fixes: http://tracker.ceph.com/issues/21069
Signed-off-by: Zhu Shangzhong <zhu.shangzhong@zte.com.cn>

Merge pull request #16127 from ZVampirEM77/wip-obj-acl-bug

rgw: fix obj copied from remote gateway acl full_control issue

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #17041 from dengxiafubi/EAS-1

rgw: exit early if rgw_bucket_set_attrs() fails

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #17065 from tchaikov/wip-20785

mon/OSDMonitor: do not send_pg_creates with stale info

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #17168 from ceph/wip-volume-man-fix

doc: include ceph-disk and ceph-disk-volume man pages in index

Reviewed-by: Dan Mick <dmick@redhat.com>

doc add ceph-volume and ceph-volume-systemd man pages to CMakeLists file

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc: include ceph-disk and ceph-disk-volume man pages in index

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #16836 from heyingstar/bug-ms_dump_on_send-doubledata

messages/MOSDOp: a fixes of encode_payload

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

Merge pull request #17079 from joscollin/wip-use-after-free

msg: drop the unnecessary polling_stop()

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #16344 from rzarzynski/wip-rgwqa-tempest

rgw, qa: integrate Tempest to verify RadosGW's compliance with Swift API

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: enable 'rgw' application on created pools

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa: add optional 'application' to pool creation helpers

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #16818 from theanalyst/rgw-lc-add-filter-2

rgw_lc: add support for optional filter argument and make ID optional

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #17116 from Synesis-LLC/fix-20934

rgw: Prevent overflow of cached stats values

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #17152 from ceph/wip-rm21030

create the ceph-volume and ceph-volume-systemd man pages

Reviewed-by: Andrew Schoen <aschoen@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

rpm: include ceph-volume and ceph-volume-systemd man pages

Signed-off-by: Alfredo Deza <adeza@redhat.com>

debian: include ceph-volume and ceph-volume-systemd man pages

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc: eliminate useless cat statement

Useless Use of Cat Award!

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #17039 from dzafman/wip-18206

osd: Fixes for osd_scrub_during_recovery handling

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crush: fix type mismatch

Pool IDs are of type int64_t instead of uint64_t.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

crush: fix bucket_adjust_item_weight() won't update weight-set correctly

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

crush: CrushWrapper::add_bucket - do not allow caller pass in null 'idout'

*** Caught signal (Segmentation fault) **
 in thread 7f495c0f6300 thread_name:crushtool
 ceph version 12.1.2-768-gab69125 (ab6912523e779174f92f0b0fc10372bd0b645415) mimic (dev)
 1: (()+0x1a3d1) [0x7f495c1343d1]
 2: (()+0xf370) [0x7f4951deb370]
 3: (CrushWrapper::add_bucket(int, int, int, int, int, int*, int*, int*)+0x84) [0x7f49538ba084]
 4: (CrushCompiler::parse_bucket(__gnu_cxx::__normal_iterator<boost::spirit::tree_node<boost::spirit::node_val_data<char const*, boost::spirit::nil_t> >*, std::vector<boost::spirit::tree_node<boost::spirit::node_val_data<char const*, boost::spirit::nil_t> >, std::allocator<boost::spirit::tree_node<boost::spirit::node_val_data<char const*, boost::spirit::nil_t> > > > > const&)+0xef0) [0x7f49538dc170]
 5: (CrushCompiler::parse_crush(__gnu_cxx::__normal_iterator<boost::spirit::tree_node<boost::spirit::node_val_data<char const*, boost::spirit::nil_t> >*, std::vector<boost::spirit::tree_node<boost::spirit::node_val_data<char const*, boost::spirit::nil_t> >, std::allocator<boost::spirit::tree_node<boost::spirit::node_val_data<char const*, boost::spirit::nil_t> > > > > const&)+0x130) [0x7f49538dcba0]
 6: (CrushCompiler::compile(std::istream&, char const*)+0xb93) [0x7f49538deaa3]
 7: (main()+0x2615) [0x7f495c126015]
 8: (__libc_start_main()+0xf5) [0x7f49507ccb35]
 9: (()+0xf4b0) [0x7f495c1294b0]
2017-08-14 13:31:25.498050 7f495c0f6300 -1 *** Caught signal (Segmentation fault) **
 in thread 7f495c0f6300 thread_name:crushtool

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

doc index the ceph-volume and ceph-volume-systemd man pages

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/man create the ceph-volume man page

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/man create the ceph-volume-systemd man page

Signed-off-by: Alfredo Deza <adeza@redhat.com>

crush: update crush_choose_arg_map size on resizing

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

crush: fix bucket_remove_item() won't update weight-set simultaneously

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

crush: fix bucket_add_item() won't update weight-set simultaneously

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

crush: fix bucket index to weight-set

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

doc/cephfs: s/rados_max_object_size/osd_max_object_size

Signed-off-by: Zhu Shangzhong <zhu.shangzhong@zte.com.cn>

Merge pull request #17127 from tchaikov/wip-doc-cleanups

doc/conf.py: do not set html_use_smartypants explicitly

Reviewed-by: Dan Mick <dmick@redhat.com>
Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #17131 from scienceluo/wip-luo-check-fix-branch

osd: Drop the unused code in OSD::_collect_metadata

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

osd: Drop the unused code in OSD::_collect_metadata

Signed-off-by: Luo Kexue <luo.kexue@zte.com.cn>

doc/conf.py: do not set html_use_smartypants explicitly

because it's deprecated, and is on by default in the specified sphinx
version (v1.6.3):

this change silences following warning:

build-doc/virtualenv/local/lib/python2.7/site-packages/sphinx/environment/__init__.py:683:
RemovedInSphinx17Warning: html_use_smartypants option is deprecated.
Smart quotes are on by default; if you want to disable or customize
them, use the smart_quotes option in docutils.conf.
  RemovedInSphinx17Warning)

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: remove manpage-howto.txt

since manpages are built as part of "make" process, we can dispense with
the manual steps.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17115 from amitkumar50/cov-osd-1

osd: Initialization of pointer cls

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

osd: Fixes for osd_scrub_during_recovery handling

Fixes: http://tracker.ceph.com/issues/18206
Signed-off-by: David Zafman <dzafman@redhat.com>

tests: osd-scrub-snaps.sh minor cleanup

Signed-off-by: David Zafman <dzafman@redhat.com>

mds: check cap string only if !allow_all

This corrects a regression introduced by #16891 which fixes
http://tracker.ceph.com/issues/20990. Not using cephx would
cause all clients to fail auth with:

    2017-08-17 12:21:05.191958 7f5b788d4700  0 -- 127.0.0.1:0/65887226 >> 127.0.0.1:6805/3339248996 conn(0x1004be8a0 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER

Fixes: http://tracker.ceph.com/issues/21027
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #17120 into master

* refs/remotes/upstream/pull/17120/head:
rpm: ceph-fuse requires fuse package

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #17097 from ceph/wip-bz1478599

ceph-volume: Use a delimited CLI output parser instead of JSON

Reviewed-by: Andrew Schoen <aschoen@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #17068 from ceph/ceph-volume-docs

ceph-volume docs

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

PGPool::update: optimize with interval_set.swap

Use constant complexity swap instead of linear complexity
assignment.

Signed-off-by: Zac Medico <zmedico@gmail.com>

docs/ceph-volume add lvm/create to the index

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume add the lvm create reference

Signed-off-by: Alfredo Deza <adeza@redhat.com>

qa: add chown test clearing S_ISGID and S_ISUID

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

rpm: ceph-fuse requires fuse package

The ceph-fuse(8) man page describes using the "fusermount" tool to mount
CephFS. This utility is in the "fuse" package. Prior to this change, the
ceph-fuse RPM only depends on "fuse-libs", not "fuse".

The Debian packaging has always depended on the full "fuse" (previously
"fuse-utils") package, so we should make the RPM packaging do the same.

Fixes: http://tracker.ceph.com/issues/21057
Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

osd: Initialization of cls

Fixes the coverity issue:

** 1322812 Uninitialized pointer field
CID 1322812 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR)
2. uninit_member: Non-static class member cls is not initialized
in this constructor nor in any functions that it calls.

Signed-off-by: Amit Kumar amitkuma@redhat.com

debian: include ceph-volume-systemd CLI tool

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #16807 from fangyuxiangGL/revert-8765

rgw: revert PR #8765

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #17024 from cbodley/wip-20906

cls/log: cls_log_list always returns next marker

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #17048 from theanalyst/rgw-fix-get-obj-tag-attr

rgw: rgw_op: exit early if object has no attrs in GetObjectTagging

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #17050 from theanalyst/rgw-iam-get-bucket-acl

rgw: evaluate the correct bucket action for GetACL bucket operation

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

doc ignore ceph-volume when building man pages

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc: add LVM tag term to the glossary

Signed-off-by: Alfredo Deza <adeza@redhat.com>