AsyncMessenger: Kepp file_lock hold when accessing its event field

When process_event get a file event, other thread may delete this event later
but before we do event

Fix #13001
Signed-off-by: Haomai Wang <haomai@xsky.io>

Merge pull request #5896 from liewegas/wip-fix-spec

ceph.spec: include /etc/sysconfig/ceph

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

ceph.spec: include /etc/sysconfig/ceph

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5895 from jcsp/wip-doc-default-pools

doc: remove references to default data/metadata pools

doc: remove references to default data/metadata pools

These haven't existed since 0.84 -- the cephfs documentation
was updated at the time, but there were also references in the
rados documentation.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #5803 from ceph/wip-debian-systemd

debian systemd support; /etc/{sysconfig,default}/ceph; easy jemalloc enablement

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

ceph-common: explicitly trigger /run/ceph creation

This doesn't happen magically on debian.

Signed-off-by: Sage Weil <sage@redhat.com>

systemd: tmpfiles.d in /run, not /var/run

Signed-off-by: Sage Weil <sage@redhat.com>

debian: fix location of tmpfiles.d

/usr/lib/tmpfiles.d, not /lib/tmpfiles.d

Signed-off-by: Sage Weil <sage@redhat.com>

debian/rules: install systemd files and /etc/default/ceph

Fix the systemd units to use /etc/default instead of /etc/sysconfig.  There
is probably a better way...

Signed-off-by: Sage Weil <sage@redhat.com>

Makefile.am: include etc in tarball

Signed-off-by: Sage Weil <sage@redhat.com>

ceph.spec: install /etc/sysconfig/ceph

Make it easy to enable jemalloc, too.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5869 from ceph/wip-rpm-dirs

ceph.spec: make /var/{lib,log,run} owned by ceph

Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #5540 from ceph/wip-selinux-radosgw

selinux: Update policy for radosgw

Reviewed-by: Milan Broz <mbroz@redhat.com>

make /var/run/ceph 770 ceph:ceph

This allows members of the ceph group to make librados clients (like the
ceph cli and qemu) create sockets in the default /var/run/ceph/* location.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph.spec: make /var/{lib,log,run} owned by ceph

Move creation of /var/lib/ceph and /var/run/ceph to ceph-common, too.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5871 from liewegas/wip-10328

common: add debug option to deliberately leak some memory

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge remote-tracking branch 'gh/wip-tcmalloc'

.gitignore: radosgw-object-expirer

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5881 from cernceph/wip-logrotate

logrotate: ignore exit status 1 from killall

Reviewed-by: Sage Weil <sage@redhat.com>

ceph.spec.in: ceph-radosgw should require ceph-selinux

Signed-off-by: Boris Ranto <branto@redhat.com>

selinux: Update policy for radosgw

The current SELinux policy does not cover radosgw daemon. This patch
introduces the SELinux support for radosgw daemon (civetweb only).

Signed-off-by: Boris Ranto <branto@redhat.com>

logrotate: ignore postrotate error for radosgw

Fix the annoying logrotate error for radosgw.

Fixes: #13033
Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>

logrotate: ignore exit status 1 from killall

If any of ceph-osd, ceph-mon, ceph-mds were not running then
killall -q will exit status 1, leading to anacron sending a mail

  error: error running shared postrotate script for '/var/log/ceph/*.log '

Fix by overriding the exit status with || true.

Fixes: #13033
Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>

Merge pull request #5800 from ceph/wip-user

ceph-disk: do async startup on upstart; fix dmcrypt create/activate bug

Reviewed-by: Loic Dachary <ldachary@redhat.com>

rocksdb: recognized --without-tcmalloc

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5031 from trociny/wip-10098

ceph-objectstore-tool: add mark-complete operation

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #5866 from jmunhoz/wip-requestid-on-error

rgw: include RequestId as part of the Error response

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #5859 from jecluis/wip-test-mon-ping

test: mon/mon-ping.sh: make sure 'ceph mon ping' works as expected

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5873 from dillaman/wip-rbd-missing-return

rbd: missing return statement within do_metadata_remove

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rbd: missing return statement within do_metadata_remove

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #5780 from dzafman/wip-12387

osd: When generating past intervals due to an import end at pg epoch

Reviewed-by: Sage Weil <sage@redhat.com>

common: add debug option to deliberately leak some memory

This will be used to verify leak detection tests are working.

Signed-off-by: Sage Weil <sage@redhat.com>

test/Makefile-client: ship LibradosTestStub.h in tarball

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5868 from jmunhoz/wip-x-amz-request-id-doc

doc: rgw: update x-amz-request-id status

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

Merge pull request #5863 from ceph/wip-13015

rgw: preserve all attrs if intra-zone copy

doc: rgw: update x-amz-request-id status

Signed-off-by: Javier M. Mellid <jmunhoz@igalia.com>

rgw: include RequestId as part of the Error response

The RGW error responses must contain a RequestId to be compliant with
the Amazon S3 errors. This RequestId is the ID of the request associated
with the error.

Fixes: #13020
Signed-off-by: Javier M. Mellid <jmunhoz@igalia.com>

test: mon/mon-ping.sh: make sure 'ceph mon ping' works as expected

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #5343 from hjwsm1989/wip-12442

  mon/MonClient: fix error in 'ceph ping mon.id'

Reviewed-by: Joao Eduardo Luis <joao@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-objectstore-tool: add mark-complete operation

It is supposed to be used as a last resort to fix a cluster that has
PGs in 'incomplete' state, using the following procedure:

1) stop the osd that is primary for the incomplete PG;
2) run:
  ceph-objectstore-tool --data-path ... --journal-path ... --pgid $PGID --op mark-complete
3) start the osd.

Fixes: #10098
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #5816 from dillaman/wip-librbd-gmock

Initial support for gmock-based unit tests in librbd

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #5557 from guangyy/wip-12666

rgw: expose the number of unhealthy workers through admin socket

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5786 from ceph/wip-12911

rbd export-diff should export data chunks in-order

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #5858 from jecluis/wip-test-mon-scrub

test: mon/mon-scrub.sh: have test running on make check

Reviewed-by: Loic Dachary <ldachary@redhat.com>

rgw: preserve all attrs if intra-zone copy

Fixes: #13015
Intra zone copy requires that all objects' attributes are preserved.
This was broken at commit: e41d97c8e38bb60d7e09e9801c0179efe7af1734

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #5442 from ceph/wip-12551

ReplicatedPG,Objecter: copy_get should include truncate_seq and size

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5490 from dwj192/master

rgw:add --reset-regions for regionmap update

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

test/Makefile.am: run mon/mon-scrub.sh as part of checks

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

test: mon/mon-scrub.sh: port clashed with other tests

This will allow the test to be run during make check.

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #5828 from lebauce/fix-swift-prefix

rgw: add delimiter to prefix only when path is specified

Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

ReplicatedPG,Objecter: copy_get should include truncate_seq and size

Otherwise, we break CephFS over cache tiers.

Fixes: #12551
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #5837 from tchaikov/wip-12968

mon: disable gmt_hitset if not supported

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5842 from dzafman/wip-cot-failure

test: Fix failure test to find message anywhere in stderr

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5845 from wonzhq/async-rx

AsyncMessenger: add instance name in debug log when processing msg

Reviewed-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #5779 from dzafman/wip-misc-fix-guangyy

osd: check the length of the map before accessing the first element

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5843 from dillaman/wip-12885

librbd: diff_iterate needs to handle holes in parent images

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #5592 from xinxinsh/wip-fix-rbd-metadata

fix bugs of rbd metadata operation

Reviewed-by: Haomai Wang <haomaiwang@gmail.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

test: Fix failure test to find message anywhere in stderr

Consolidate test_failure() and test_failure_tty()

Signed-off-by: David Zafman <dzafman@redhat.com>

rados: Fix usage for "notify" command

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #5834 from ceph/wip-leveldb-hang

mon: MonitorDBStore: make get_next_key() work properly

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4955 from kylinstorage/librados-2read-segmentation-fault-v1

bug fix: librados segmentation fault, when two read ops share one Aio…

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4925 from XinzeChi/wip-hit-set-size

osd: fix hit_set_map size for tier pool

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5723 from H3C/wip-msg-bugfix1

msg: we should set the socket options before connect or listen in order to have it take effect.

Reviewed-by: Haomai Wang <haomaiwang@gmail.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5829 from dachary/wip-12731-jerasure-warnings

erasure-code: fix gf-complete warnings

Reviewed-by: Kefu Chai <kchai@redhat.com>

AsyncMessenger: add instance name in debug log when processing msg

To better debug.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

rgw:add --reset-regions for regionmap update

Fixes: #12964
Signed-off-by: Weijun Duan <duanweijun@h3c.com>

Merge pull request #5781 from dzafman/wip-eccleanup

osd, test: Minor clean-up from fast-read and error handling ec changes

Reviewed-by: Kefu Chai <kchai@redhat.com>

librbd: diff_iterate needs to handle holes in parent images

If a clone's parent image snapshot includes a discarded extent,
this was previously causing an assert failure.  Instead, ignore
any discard holes in the parent image.

Fixes: #12885
Backport: hammer
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: new test case for librbd diff_iterate over discard extents

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #5838 from ceph/wip-12861

qa/workunits/rados/test_alloc_hint.sh: sudo to ls files

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5396 from XinzeChi/wip-scrub-promote

bug fix: osd: requeue_scrub when kick_object_context_blocked

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #5772 from yuyuyu101/wip-12912

AsyncConnection: Close connection when unregistered connection met WAIT

Merge pull request #5733 from cxwshawn/rgw-fix

rgw: init_rados failed leads to repeated delete

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #5809 from rzarzynski/wip-rgw_putobj_debug_improvement

rgw: improve debugs in RGWPutObj and RGWPutObj_ObjStore_SWIFT.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #5567 from guce/rgw_check_key_parameter_conflict

rgw: add key parameter conflict check for radosgw-admin command line.

Merge pull request #5819 from rzarzynski/wip-12950

rgw: don't append empty ETag HTTP header.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge remote-tracking branch 'gh/infernalis'

qa/workunits/rados/test_alloc_hint.sh: sudo to ls files

The osd data dir is owned by ceph and not readable by other
non-root users.

Fixes: #12861
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5257 from t-miyamae/wip-mshec-fixlogic4

erasure code: shec performance optimization by decoding cache

Reviewed-by: Loic Dachary <ldachary@redhat.com>

rgw: init_rados failed leads to repeated delete

Fixes: #12978
Signed-off-by: Xiaowei Chen <chen.xiaowei@h3c.com>

Merge pull request #5463 from kylinstorage/wip-evict_ops

Allow evict operations to be throttled

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5745 from ceph/wip-12904

osd: sparse reads returning invalid extent map

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5580 from xinxinsh/wip-merge-two-setattr

merge two continuous OP_SETATTR ops into one OP_SETATTRS

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mon: disable gmt_hitset if not supported

the gmt_hitset is enabled by default in the ctor of pg_pool_t, this
is intentional. because we want to remove this setting and make
gmt_hitset=true as a default in future. but this forces us to
disable it explicitly when preparing a new pool if any OSD does
not support gmt hitset.

Fixes: #12968
Signed-off-by: Kefu Chai <kchai@redhat.com>

test: mon: mon-scrub.sh: test 'mon scrub'

In its current state is used to catch regressions on 'ceph mon scrub'
hanging due to an infinite loop.

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #5785 from liewegas/wip-12861

qa/workunits/rados/test_alloc_hint.sh: sudo to list files

mon: MonitorDBStore: make get_next_key() work properly

We introduced a significant bug with 2cc7aee, when we fixed issue #11786.
Although that patch would fix the problem described in #11786, we
managed to not increment the iterator upon returning the current key.
This would have the iterator iterating over the same key, forever and
ever.

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #5784 from dachary/wip-docker-helper

fix docker-test.sh for CentOS 7

Reviewed-by: Kefu Chai <kchai@redhat.com>

tests: drop docker-tests.sh root and /dev support

Now that ceph-disk.sh activation tests switched to the ceph-disk suite
as found in https://github.com/ceph/ceph-qa-suite/tree/master/suites/ceph-disk,
it does not need for root and /dev support from docker-tests.sh. There
currently is no other use case and since it's generally not a good idea
to run anything as root in a container anyway.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: CentOS 7 needs systemd-container

It is now needed by the Ceph dependencies and won't work unless the fake
version of it is replaced by the actual one.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

rgw: add delimiter to prefix only when path is specified

http://tracker.ceph.com/issues/12960
Fixes: #12960
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>

Merge pull request #5728 from trociny/wip-12706-infernalis

ceph-disk: use /sys/dev/block/maj:min/partition to see if partition

Reviewed-by: Loic Dachary <ldachary@redhat.com>

refine tests for metadata ops

Signed-off-by: xinxin shu <xinxin.shu@intel.com>

improve error handle of rbd metadata operation & format output

Signed-off-by: xinxin shu <xinxin.shu@intel.com>

fix metadata loading error if we open an image

Signed-off-by: xinxin shu <xinxin.shu@intel.com>

msg: we should set the socket options before connect or listen in order to have it take effect.

Signed-off-by: Ruifeng Yang <149233652@qq.com>

erasure-code: fix gf-complete warnings

Update to the latest gf-complete v2 branch which has the compilation
warning fixed. In addition the default compilation flag upstream has
been set to display such warnings.

git log --oneline --graph d384952c68a64d93ac7af6341d5519ea5d2958b9..9caeefbf2860e56a75502f4d3342deed5b5ba265
* 9caeefb add -Wsign-compare and address the warnings
* 5979f98 gf_w64.c: fix integer overflow
* 93a9845 gf_w64.c: fix integer overflow
* 513c87b gf_w64.c: fix integer overflow
* 4d6fa89 gf_w64.c: fix integer overflow

http://tracker.ceph.com/issues/12731 Fixes: #12731

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge remote-tracking branch 'gh/infernalis'

Merge pull request #4895 from XinzeChi/wip-outdata-set

bug fix: osd: avoid multi set osd_op.outdata in tier pool

Reviewed-by: Sage Weil <sage@redhat.com>