Merge pull request #14327 from cbodley/wip-12863

rgw: switch from "timegm()" to "internal_timegm()" for better portability

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14190 from idryomov/wip-respect-epoch-barrier

osdc/Objecter: respect epoch barrier in _op_submit()

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #14220 from jcsp/wip-19407

mon: update mgrmap when active goes offline

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12437 from idealguo/remove-duplicate-flush

rgw: remove duplicate flush formatter

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #13934 from chuang-he/fix_copy_obj_data

rgw: fix configurable write obj window size

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #11963 from zhangsw/fix-rgw-lifecycle-bug

rgw: LCWorker's worktime is not the same as config rgw_lifecycle_work_time.

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Switch from "timegm()" to "internal_timegm()"

Changed timegm() to internal_timegm() for better portability

Fixes: http://tracker.ceph.com/issues/12863
Signed-off-by: Rishabh Kumar <kris.kr296@gmail.com>

Merge pull request #14042 from pritha-srivastava/wip-rgw-fix-accounting

rgw: Added code to correctly account for bytes sent/ received during a 'PUT' operation.

Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14125 from zhangsw/fix-rgw-cache-normalname-bug

rgw: add pool namespace to cache's key so that system obj can have unique key

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14072 from trociny/wip-notify

librbd: Notifier::notify API improvement

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #14160 from Liuchang0812/wip-19363-fix-lc-two-response

rgw/lifecycle: do not send lifecycle rules when GetLifeCycle failed

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14310 from tchaikov/wip-yield-for-ppc

common/simple_spin: use __ppc_yield() on all powerpc archs

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Tested-by: Andrew Solomon <asolomon@us.ibm.com>

Merge pull request #14289 from tchaikov/wip-conditionalize-crc32c

cmake,crc32c: conditionalize crc32c on different archs

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Tested-by: Andrew Solomon <asolomon@us.ibm.com>

Merge pull request #14275 from ceph/evelu-cephdisk-race

ceph-disk: Adding retry loop in get_partition_dev()

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #14185 from DHB-liuhong/objects

mon/PGMap: make si units more readable in PGMap summary

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14313 from wjin/clean_mgr

mgr: rm nonused main function

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-disk: Adding retry loop in get_partition_dev()

There is very rare cases where get_partition_dev() is called before the actual partition is available in /sys/block/<device>.

It appear that waiting a very short is usually enough to get the partition beein populated.

Analysis:
update_partition() is supposed to be enough to avoid any racing between events sent by parted/sgdisk/partprobe and
the actual creation on the /sys/block/<device>/* entrypoint.
On our CI that race occurs pretty often but trying to reproduce it locally never been possible.

This patch is almost a workaround rather than a fix to the real problem.
It offer retrying after a very short to be make a chance the device to appear.
This approach have been succesful on the CI.

Note his patch is not changing the timing when the device is perfectly created on time and just differ by a 1/5th up to 2 seconds when the bug occurs.

A typical output from the build running on a CI with that code.
command_check_call: Running command: /usr/bin/udevadm settle --timeout=600
get_dm_uuid: get_dm_uuid /dev/sda uuid path is /sys/dev/block/8:0/dm/uuid
get_partition_dev: Try 1/10 : partition 2 for /dev/sda does not in /sys/block/sda
get_partition_dev: Found partition 2 for /dev/sda after 1 tries
        get_dm_uuid: get_dm_uuid /dev/sda uuid path is /sys/dev/block/8:0/dm/uuid
get_dm_uuid: get_dm_uuid /dev/sda2 uuid path is /sys/dev/block/8:2/dm/uuid

fixes: #19428

Signed-off-by: Erwan Velu <erwan@redhat.com>

mgr: rm nonused main function

Signed-off-by: Wei Jin <wjin.cn@gmail.com>

Merge pull request #14174 from shinobu-x/wp-sk-typo-osdmonitor-20170328

mon/OSDMonitor: Clean up: delete extra S signature for plural

Reviewed-by: Kefu Chai <kchai@redhat.com>

crc32c,compressor: only compile functions on supported arch

so we can still compile if certain header file (<sys/auxv.h>) used
for detecting cpu capabilities on ppc is missing on x86 host machine.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/simple_spin: use __ppc_yield() on all powerpc archs

__ppc_yield() is declared in sys/platform/ppc.h by glibc, for better
portability we just use the inlined assembly here. the shared resource
hints are supported by PowerPC ISA 2.06 but on older PowerPC cores, they
are no-ops. so it's fine to do this way.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12243 from rzarzynski/wip-rgw-bulkupload

rgw: add support for the BulkUpload of Swift API
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #14120 from wangzhengyong/uninitialized

rgw: fix uninitialized fields

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #14217 from Liuchang0812/wip-18584-pg-info-dump

osd: print pg_info_t::purged_snaps as array, not string

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14036 from batrick/mds-debug-reply

mds: bump client_reply debug to match client_req

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #14296 from theanalyst/fix/script/credits

script: drop the -x arg for credits script

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #13401 from liewegas/wip-kill-subop

osd: kill all remaining MOSDSubOp users

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #14221 from jcsp/wip-19253-b

qa: whitelist another fullness log message

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #14279 from jimmyway/wip-fix-pg-comment

osd: fix comments about pg refs and lock

Reviewed-by: Pan Liu <liupan1111@gmail.com>

Merge pull request #14277 from atheism/fix-test-objectstore-chain_xattr

test: objectstore: chain_xattr: fix wrong memset usage to fill buf

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13800 from xiexingguo/wip-clean-pgcreate-throttle

common/config_opts.h: kill mon_pg_create_interval

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13621 from Jing-Scott/add-remov-x-delete-feature

rgw: add the remove-x-delete feature to cancel swift object expiration

Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #13862 from jcsp/wip-16523

qa, mds: add checks for fragmentation, and enable it by default

mon/OSDMonitor: Clean up: delete extra S signature for plural

Signed-off-by: Shinobu Kinjo <shinobu@redhat.com>

Merge pull request #14156 from smithfarm/release-notes-add-rbd-mirror

tools: ceph-release-notes: prefixes and pep8 compliance

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

script: drop the -x arg for credits script

while useful for debugging creates a whole lot of output which we don't
need normally

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #12084 from hansbogert/patch-4

rgw: Add --zonegroup-new-name  in usage
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #14269 from liewegas/wip-no-spinlock

mon/OSDMonitor: spinlock -> std::mutex

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14274 from liewegas/wip-osd-beacon-fix

osd: fix some osd beacon bugs

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14288 from Adirl/small_leaks

msg/async/rdma: Fix small memory leaks detected by valgrind

Reviewed-by: Haomai Wang <haomai@xsky.com>

cmake: point to asm_compiler to yasm only if HAVE_INTEL

as currently only isa uses it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: extract ppc64le detection into SIMDExt.cmake

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: add the check for Content-Length in BulkUpload.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: implement the full response generation in BulkUpload of Swift API.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: optimize metadata caching in BulkUpload of Swift API.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: implement the object creation in BulkUpload of Swift API.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: implement the container creation in BulkUpload of Swift API.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

msg/async/rdma: Fix small memory leaks detected by valgrind

Issue: 995322
Change-Id: Iccffbd0defd09aa51c4b51b78733921444ea1c5f
Signed-off-by: Amir Vadai <amir@vadai.me>

rgw: implement the AlignedStreamGetter.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: implement the basic security check for BulkUpload of Swift API.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #14170 from liewegas/wip-fewer-warnings

common/ceph_context: fewer warnings about experimental features

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14271 from joscollin/wip-krbd-warning

rbd: warning, ‘devno’ may be used uninitialized in this function

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13909 from kestrels/wip-19217

crc32c: Add crc32c function optimized for ppc architecture

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14124 from Fabian-Gruenbichler/ipv6_fix

common: fix segfault in public IPv6 addr picking

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14173 from badone/wip-do_request-abort

common: Fix heap buffer overflow in do_request

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14193 from liupan1111/wip-fix-delete-full

librados: set the flag CEPH_OSD_FLAG_FULL_TRY of Op in the right place.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14248 from badone/wip-simple_spin_lock-delay

common: Make spinlock delay more conventional

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13690 from jcsp/wip-17503

mgr: pass through cluster log to plugins

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw: implement the TAR extraction loop of Swift's BulkUpload.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: implement the TAR format interpreter.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: implement RGWBulkUploadOp::DecoratedStreamGetter.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: initial class structure for BulkUpload of Swift API.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

osd: fix comments about pg refs and lock

Signed-off-by: tang.jin <tang.jin@istuary.com>

Merge pull request #14276 from atheism/rgw-rgwop-remove-unused-iter

rgw: rgw_op: remove unused variable iter
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

test: objectstore: chain_xattr: fix wrong memset usage to fill buf

    Current usage:
        memset(buf, sizeof(buf), 0x1F);
    should be:
        memset(buf, 0x1F, sizeof(buf));
    to fill the buffer with '0x1F'.

Signed-off-by: Weibing Zhang <zhangweibing@unitedstack.com>

rgw: rgw_op: remove unused variable iter

    The variable iter is not used in  RGWInitMultipart::execute().

Signed-off-by: Weibing Zhang <zhangweibing@unitedstack.com>

Merge pull request #13811 from rzarzynski/wip-bs-bitmap-mem-cleanup

os/bluestore: memory and dereference clean-up in the BitAllocator

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14121 from majianpeng/bluefs-get-allocated

os/bluestore/BlueFS: optimize get_allocated

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14162 from majianpeng/optimize-bluestore-force-flush

os/bluestore: clean up flush logic

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14225 from ivancich/wip_fix_throttle_config_change

os/bluestore: make live changes for BlueStore throttle config work like initial config

Reviewed-by: Sage Weil <sage@redhat.com>

osd: only send beacon on connect if active

Otherwise the mon will just ignore it and we'll wait another
cycle.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: fix beacon condition

- && should have been ||
- change this to instead not send when monmap epoch==0,
  and only set last_sent_beacon if we actually send.  That
  way we'll send on the next tick after we get a real monmap.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedBackend: remove MOSDSubOp cruft from repop_commit

Signed-off-by: Sage Weil <sage@redhat.com>

osd: drop unused MOSDSubOp includes

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedBackend: rename sub_op_modify* do_repop*

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedBackend: no empty MOSDSubOps anymore

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedBackend: use more __func__

Signed-off-by: Sage Weil <sage@redhat.com>

osd: drop unnecessary declarations in headers

Signed-off-by: Sage Weil <sage@redhat.com>

osd: use MOSDScrubReserve instead of MOSDSubOp for scrub reservations

This is the last MOSDSubOp user.

Note that while the next step is to move to AsyncReserver internally,
this isn't quite yet possible since AsyncReserve "blocks" indefinitely
so we wouldn't generate a REJECT.  Changing how we schdule scrubs
internally will take a bit more work.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: use new MOSDRepScrubMap message for ScrubMap instead of MOSDSubOp

This is one of the last remaining users of MOSDSubOp!

Signed-off-by: Sage Weil <sage@redhat.com>

osd: use new MOSDPGBackfillRemove instead of MOSDSubOp

One of the last remaining MOSDSubOp users!

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedBackend: remove legacy push/pull op handling

These are never sent, and thus never recieved.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #14229 from ceph/wip-systemd

qa: Add reboot case for systemd test

Merge pull request #14254 from idryomov/wip-vstart-runner-ps

qa/vstart_runner: amend ps invocation

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #14226 from ceph/preset-mon

systemd: remove ceph-create-keys from presets

Merge pull request #14266 from tchaikov/wip-require-mgr-for-pg-dump

tests: add MGR=1 so 'pg dump' won't be blocked

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #14259 from johnwilkins/wip-doc-multisite

doc: wip-doc-multisite ports downstream multisite document upstream.

rbd: warning, ‘devno’ may be used uninitialized in this function

The following warning appears during make. Fixed in both unmap_image() functions in krbd.cc

--
krbd.cc: In function ‘int krbd_unmap_by_spec(krbd_ctx*, const char*, const char*, const char*, const char*)’:
krbd.cc:608:65: warning: ‘devno’ may be used uninitialized in this function [-Wmaybe-uninitialized]
   return do_unmap(ctx->udev, devno, build_unmap_buf(id, options));
                                                                 ^
krbd.cc:591:9: note: ‘devno’ was declared here
   dev_t devno;
--

Signed-off-by: Jos Collin <jcollin@redhat.com>

mon/OSDMonitor: spinlock -> std::mutex

I think spinlock is dangerous here: we're doing semi-unbounded
work (decode).  Also seemingly innocuous code like dout macros
take mutexes.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMap: use _pg_to_up_acting_osds() to get_pg_acting_primary()

avoid unnecessary swap() call of vector<int> which will be dropped on
the floor after getting its first element, which can be retrieved using
another parameter using _pg_to_up_acting_osds() directly.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr: use OSDMap::get_pg_acting_primary() to get the primary

OSDMap::pg_to_acting_osds() assumes that the "acting" parameter is not
null.

Signed-off-by: Kefu Chai <kchai@redhat.com>

tests: add MGR=1 so 'pg dump' won't be blocked

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/config_opts: kill mon_pg_create_interval

This is not needed anymore.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #14260 from dmick/wip-workunit-clone-with-branch

workunit: request branch when cloning

Reviewed-by: Kefu Chai <kchai@redhat.com>

common: Make spinlock delay more conventional

The accepted method of implementing a spinlock delay is the "pause"
instruction.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

tasks/workunit.py: specify the branch name when cloning a branch

c1309fb failed to specify a branch when cloning using --depth=1, which
by default clones the HEAD. and we can not "git checkout" a specific
sha1 if it is not HEAD, after cloning using '--depth=1', so in this
change, we dispatch "tag", "branch", "HEAD" using three Refspec classes.

Signed-off-by: Kefu Chai <kchai@redhat.com>
Signed-off-by: Dan Mick <dan.mick@redhat.com>

rgw: fix uninitialized fields

Signed-off-by: wangzhengyong@cmss.chinamobile.com

Merge pull request #14219 from ShiqiCooperation/master

os/bluestore: clean up Invalid return value judgment

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14228 from ifed01/wip-bluestore-fix-store-test

test/store_test: fix bluestore test cases disablement

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #13617 from liewegas/wip-mgr-commands

mon,mgr: tag some commands for ceph-mgr

Reviewed-by: Kefu Chai <kchai@redhat.com>