ceph_test_objectstore: error via derr too

This helps establish timing relative to the log.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: increase aio queue depth

We've seen some reports of the aio EAGAIN backoff in logs; use a larger
aio ring buffer to reduce the risk of this.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: reduce some debug level

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: put fsck structures in mempools

This lets us track them individually in debug mode.

Signed-off-by: Sage Weil <sage@redhat.com>

mempool: do not skip first 2 mempools

This broke when we sorted the pool definitions in the header.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: flush osr even on failure in _deferred_replay

This is one less thing for fsck to worry about.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: trim cache during fsck

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: be noisy about bitmap init (it's slow!)

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #14316 from liewegas/wip-bluestore-fsck-unused

os/bluestore: fsck: verify blob.unused field

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: use typedef for blob unused (uint16_t)

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fsck: verify blob.unused field

Two checks:

- verify that no logical extents reference the portion of the
  blob marked unused, and
- verify that the csum (if present) is zero for any unused
  region.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #14157 from liewegas/wip-19379

os/bluestore: fix deferred writes vs collection split race

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #14258 from liewegas/wip-bluestore-tool

ceph-bluestore-tool: rename from bluefs-tool; improve usage

Merge pull request #14294 from tchaikov/wip-check-last-osd-report-if-luminous

mon/OSDMonitor: check last_osd_report only when the whole cluster is lu…

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14303 from liewegas/wip-more-mgr-commands

mgr: move 'osd perf' and 'osd blocked-by' to mgr

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14010 from zhangsw/fix-rgw-fetchremote-bug

rgw: version id doesn't work in fetch_remote_obj

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #13310 from ceph/wip-ceph-disk-fix

ceph-disk: Add fix subcommand

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #14314 from rzarzynski/wip-rgw-bulkupload-freebsd

rgw: switch to std::array in RGWBulkUploadOp due to C++11 and FreeBSD.
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #14252 from cbodley/wip-19409

radosgw-admin: new 'global quota' commands update period config
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #14109 from cbodley/wip-rgw-stat-manifest

rgw: stat requests skip compression, manifest handling, etc
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #14281 from tchaikov/wip-19429

qa/tasks/workunit.py: use "overrides" as the default settings of workunit

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #14297 from Adirl/ibport

msg/async/rdma: Make port number an attribute of the Connection not o…

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #13766 from dachary/wip-18962-ceph-disk-zap

ceph-disk: Write 10M to all partitions before zapping

Reviewed-by: Sébastien Han <seb@redhat.com>

Merge pull request #13998 from wangzhengyong/aio_complete

rgw: fix memory leak in delete_obj_aio

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #12883 from mslovy/wip-rbd-import

rbd: import real thin-provision image

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #14327 from cbodley/wip-12863

rgw: switch from "timegm()" to "internal_timegm()" for better portability

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14190 from idryomov/wip-respect-epoch-barrier

osdc/Objecter: respect epoch barrier in _op_submit()

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #14220 from jcsp/wip-19407

mon: update mgrmap when active goes offline

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12437 from idealguo/remove-duplicate-flush

rgw: remove duplicate flush formatter

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #13934 from chuang-he/fix_copy_obj_data

rgw: fix configurable write obj window size

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #11963 from zhangsw/fix-rgw-lifecycle-bug

rgw: LCWorker's worktime is not the same as config rgw_lifecycle_work_time.

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Switch from "timegm()" to "internal_timegm()"

Changed timegm() to internal_timegm() for better portability

Fixes: http://tracker.ceph.com/issues/12863
Signed-off-by: Rishabh Kumar <kris.kr296@gmail.com>

Merge pull request #14042 from pritha-srivastava/wip-rgw-fix-accounting

rgw: Added code to correctly account for bytes sent/ received during a 'PUT' operation.

Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14125 from zhangsw/fix-rgw-cache-normalname-bug

rgw: add pool namespace to cache's key so that system obj can have unique key

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14072 from trociny/wip-notify

librbd: Notifier::notify API improvement

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #14160 from Liuchang0812/wip-19363-fix-lc-two-response

rgw/lifecycle: do not send lifecycle rules when GetLifeCycle failed

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: switch to std::array in RGWBulkUploadOp due to C++11 and FreeBSD.

Before this patch RGWBulkUploadOp::terminal_errors was declared as
std::initializer_list<int>. Unfortunately, a constexpr constructor
for it is available since C++14, not C++11. This was causing build
failures on FreeBSD.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

os/bluestore: do not extend blobs with unused bitmap

If we resize the blob we need to adjust the resolution of the
unused bitmap, and that is only possible for some bit patterns.
For now just ignore blobs with unused blocks.

Add an assert in add_tail() so that we don't forget that
add_tail is (probably) where we'd would do that adjustment.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: some whitespace

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: mark_used for wctx->writes

The refactored blob reuse code needs to mark blobs used when
processing wctx->writes.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: prefix fsck errors with 'fsck error:'

Greppable!

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async/rdma: Make port number an attribute of the Connection not of the Device

Since multiple connections on different ports could exist, shouldn't use
device->active_port, instead use conn->ibport.
Or in other words, now Device object doesn't have an active_port,
instead every port specific action (create_qp, get_lid, get_gid etc.)
need to specify the port number. The information about the port number
is known to the connection (RDMAConnectedSocket*) who is the caller of
those actions.

Issue: 995322
Change-Id: I482cb87c04ba99845dc44f6dd0547835fe814ebf
Signed-off-by: Amir Vadai <amir@vadai.me>

Merge pull request #14310 from tchaikov/wip-yield-for-ppc

common/simple_spin: use __ppc_yield() on all powerpc archs

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Tested-by: Andrew Solomon <asolomon@us.ibm.com>

Merge pull request #14289 from tchaikov/wip-conditionalize-crc32c

cmake,crc32c: conditionalize crc32c on different archs

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Tested-by: Andrew Solomon <asolomon@us.ibm.com>

Merge pull request #14275 from ceph/evelu-cephdisk-race

ceph-disk: Adding retry loop in get_partition_dev()

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #14185 from DHB-liuhong/objects

mon/PGMap: make si units more readable in PGMap summary

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14313 from wjin/clean_mgr

mgr: rm nonused main function

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-disk: Adding retry loop in get_partition_dev()

There is very rare cases where get_partition_dev() is called before the actual partition is available in /sys/block/<device>.

It appear that waiting a very short is usually enough to get the partition beein populated.

Analysis:
update_partition() is supposed to be enough to avoid any racing between events sent by parted/sgdisk/partprobe and
the actual creation on the /sys/block/<device>/* entrypoint.
On our CI that race occurs pretty often but trying to reproduce it locally never been possible.

This patch is almost a workaround rather than a fix to the real problem.
It offer retrying after a very short to be make a chance the device to appear.
This approach have been succesful on the CI.

Note his patch is not changing the timing when the device is perfectly created on time and just differ by a 1/5th up to 2 seconds when the bug occurs.

A typical output from the build running on a CI with that code.
command_check_call: Running command: /usr/bin/udevadm settle --timeout=600
get_dm_uuid: get_dm_uuid /dev/sda uuid path is /sys/dev/block/8:0/dm/uuid
get_partition_dev: Try 1/10 : partition 2 for /dev/sda does not in /sys/block/sda
get_partition_dev: Found partition 2 for /dev/sda after 1 tries
        get_dm_uuid: get_dm_uuid /dev/sda uuid path is /sys/dev/block/8:0/dm/uuid
get_dm_uuid: get_dm_uuid /dev/sda2 uuid path is /sys/dev/block/8:2/dm/uuid

fixes: #19428

Signed-off-by: Erwan Velu <erwan@redhat.com>

mgr: rm nonused main function

Signed-off-by: Wei Jin <wjin.cn@gmail.com>

Merge pull request #14174 from shinobu-x/wp-sk-typo-osdmonitor-20170328

mon/OSDMonitor: Clean up: delete extra S signature for plural

Reviewed-by: Kefu Chai <kchai@redhat.com>

crc32c,compressor: only compile functions on supported arch

so we can still compile if certain header file (<sys/auxv.h>) used
for detecting cpu capabilities on ppc is missing on x86 host machine.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/simple_spin: use __ppc_yield() on all powerpc archs

__ppc_yield() is declared in sys/platform/ppc.h by glibc, for better
portability we just use the inlined assembly here. the shared resource
hints are supported by PowerPC ISA 2.06 but on older PowerPC cores, they
are no-ops. so it's fine to do this way.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test: add test for rbd import,import-diff,copy with sparse-size option

Signed-off-by: Ning Yao <yaoning@unitedstack.com>

librbd: add sparse copy support

Fixes: fix issue #15648
http://tracker.ceph.com/issues/15648

Signed-off-by: yaoning <yaoning@unitedstack.com>

rbd: import/import-diff real thin-provision image

currently, import/import-diff check whether object is zero based on the whole object
therefore, holes inside object will be filled with zero

Signed-off-by: yaoning <yaoning@unitedstack.com>

rbd: add sparse size support in rbd

add --sparse_size in rbd command options
add calc_sparse_extent in rbd utils

Signed-off-by: yaoning <yaoning@unitedstack.com>

Merge pull request #12243 from rzarzynski/wip-rgw-bulkupload

rgw: add support for the BulkUpload of Swift API
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

doc/rgw: update section on global quotas

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: new 'global quota' commands update period config

Fixes: http://tracker.ceph.com/issues/19409
Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: load period config separately when there's no current period

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: synchronize period config with period updates

read period config from local storage on RGWPeriod::update(), and write
the new config to storage on RGWPeriod::reflect()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: allow RGWPeriodConfig to be stored separately

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: constify some rgw_pool args

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove unused RGWPeriod::update(RGWZoneGroupMap)

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14120 from wangzhengyong/uninitialized

rgw: fix uninitialized fields

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

mgr: move 'osd perf' and 'osd blocked-by' to mgr

Signed-off-by: Sage Weil <sage@redhat.com>

mon: move 'osd perf' and 'osd blocked-by' to PGMap

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #14217 from Liuchang0812/wip-18584-pg-info-dump

osd: print pg_info_t::purged_snaps as array, not string

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14036 from batrick/mds-debug-reply

mds: bump client_reply debug to match client_req

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #14296 from theanalyst/fix/script/credits

script: drop the -x arg for credits script

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #13401 from liewegas/wip-kill-subop

osd: kill all remaining MOSDSubOp users

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #14221 from jcsp/wip-19253-b

qa: whitelist another fullness log message

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #14279 from jimmyway/wip-fix-pg-comment

osd: fix comments about pg refs and lock

Reviewed-by: Pan Liu <liupan1111@gmail.com>

Merge pull request #14277 from atheism/fix-test-objectstore-chain_xattr

test: objectstore: chain_xattr: fix wrong memset usage to fill buf

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: flush osr on collection split

We need to ensure that any preceding txcs have finished their deferred
writes before splitting, or else later txcs on the child sequencer will
not order/wait for deferred writes correctly before deallocated extents.

Fixes: http://tracker.ceph.com/issues/19379
Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: make deferred_aggressive atomic

This is written and read outside of any mutex.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: use target object hash (not pgid.ps) for temp objects

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #13800 from xiexingguo/wip-clean-pgcreate-throttle

common/config_opts.h: kill mon_pg_create_interval

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13621 from Jing-Scott/add-remov-x-delete-feature

rgw: add the remove-x-delete feature to cancel swift object expiration

Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #13862 from jcsp/wip-16523

qa, mds: add checks for fragmentation, and enable it by default

mon/OSDMonitor: Clean up: delete extra S signature for plural

Signed-off-by: Shinobu Kinjo <shinobu@redhat.com>

Merge pull request #14156 from smithfarm/release-notes-add-rbd-mirror

tools: ceph-release-notes: prefixes and pep8 compliance

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

script: drop the -x arg for credits script

while useful for debugging creates a whole lot of output which we don't
need normally

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #12084 from hansbogert/patch-4

rgw: Add --zonegroup-new-name  in usage
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #14269 from liewegas/wip-no-spinlock

mon/OSDMonitor: spinlock -> std::mutex

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14274 from liewegas/wip-osd-beacon-fix

osd: fix some osd beacon bugs

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mon/OSDMonitor: only check last_osd_report if the whole cluster is luminous

if a peon is not luminous, it won't forward the beacon message. if any
OSD is not luminous, it won't report to mon with beacon messages. so, we
can trust the OSDMonitor::last_osd_report only if the whole cluster is
luminous.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14288 from Adirl/small_leaks

msg/async/rdma: Fix small memory leaks detected by valgrind

Reviewed-by: Haomai Wang <haomai@xsky.com>

cmake: point to asm_compiler to yasm only if HAVE_INTEL

as currently only isa uses it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: extract ppc64le detection into SIMDExt.cmake

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: add the check for Content-Length in BulkUpload.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: implement the full response generation in BulkUpload of Swift API.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: optimize metadata caching in BulkUpload of Swift API.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: implement the object creation in BulkUpload of Swift API.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: implement the container creation in BulkUpload of Swift API.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

msg/async/rdma: Fix small memory leaks detected by valgrind

Issue: 995322
Change-Id: Iccffbd0defd09aa51c4b51b78733921444ea1c5f
Signed-off-by: Amir Vadai <amir@vadai.me>

rgw: implement the AlignedStreamGetter.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: implement the basic security check for BulkUpload of Swift API.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #14170 from liewegas/wip-fewer-warnings

common/ceph_context: fewer warnings about experimental features

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/tasks/workunit.py: use "overrides" as the default settings of workunit

otherwise the settings in "workunit" tasks are always overridden by the
settings in template config. so we'd better follow the way of how
"install" task updates itself with the "overrides" settings: it uses the
"overrides" as the *defaults*.

Fixes: http://tracker.ceph.com/issues/19429
Signed-off-by: Kefu Chai <kchai@redhat.com>