rgw: make radosgw-admin user rm idempotent

Fixes: #5378
Signed-off-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #5901 from yuyuyu101/wip-13001

AsyncMessenger: Kepp file_lock hold when accessing its event field

Merge pull request #5626 from majianpeng/objecter

Objecter: fix bus about objecter.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5840 from liewegas/wip-12991

qa/workunits/cephtool/test.sh: make mds epoch check more tolerant

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge remote-tracking branch 'gh/infernalis'

Merge pull request #5686 from liewegas/wip-hammer-feature

require osds are hammer v0.94.4 or later before upgrading past hammer

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5905 from ceph/wip-rpm-notcmalloc

ceph.spec: respect CEPH_EXTRA_CONFIGURE_ARGS

Merge pull request #5904 from ceph/wip-submodules

fix a few submodule issues

Reviewed-by: Loic Dachary <ldachary@redhat.com>

.gitignore: ignore src/ceph.tmpe

It makes the gitbuilders unhappy.  Not sure where it comes from...

Signed-off-by: Sage Weil <sage@redhat.com>

gmock: ignore *.pyc

Signed-off-by: Sage Weil <sage@redhat.com>

rocksdb: ignore m4

Signed-off-by: Sage Weil <sage@redhat.com>

ceph.spec: respect CEPH_EXTRA_CONFIGURE_ARGS

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/cephtool/test.sh: make mds epoch check more tolerant

This can race with an actual mdsmap epoch update for some other
reason.  We just need to make sure the epoch *increased*, not that
it is exactly old + 1.

Fixes: #12991
Signed-off-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/wip-10617-again'

Merge pull request #5334 from majianpeng/read-len-zero-for-ec

make read ec-object(offset=0, len=0) can work like read no-ec object.

Reviewed-by: Kefu Chai <kchai@redhat.com>

sd/PG: tolerate missing pgmeta object

Bug 10617 left stray PG dirs around in firefly.  Hammer correctly
ignores these, assuming they are leftover cruft.  We broke this
when we dropped compat support in cd4e676e6d45c8166290ef834d73c2a0bda98fa2

See also #13060.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: allow peek_map_epoch to return an error

Allow PG::peek_map_epoch to return an error indicating the PG
should be skipped.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5776 from wonzhq/tmap-update

osd: force promote for ops which ec base pool can't handle

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

osd/ReplicatedPG: remove stray debug line

This snuck in

Signed-off-by: Sage Weil <sage@redhat.com>

AsyncMessenger: Kepp file_lock hold when accessing its event field

When process_event get a file event, other thread may delete this event later
but before we do event

Fix #13001
Signed-off-by: Haomai Wang <haomai@xsky.io>

Merge pull request #5896 from liewegas/wip-fix-spec

ceph.spec: include /etc/sysconfig/ceph

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

ceph.spec: include /etc/sysconfig/ceph

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5895 from jcsp/wip-doc-default-pools

doc: remove references to default data/metadata pools

doc: remove references to default data/metadata pools

These haven't existed since 0.84 -- the cephfs documentation
was updated at the time, but there were also references in the
rados documentation.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #5803 from ceph/wip-debian-systemd

debian systemd support; /etc/{sysconfig,default}/ceph; easy jemalloc enablement

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

ceph-common: explicitly trigger /run/ceph creation

This doesn't happen magically on debian.

Signed-off-by: Sage Weil <sage@redhat.com>

systemd: tmpfiles.d in /run, not /var/run

Signed-off-by: Sage Weil <sage@redhat.com>

debian: fix location of tmpfiles.d

/usr/lib/tmpfiles.d, not /lib/tmpfiles.d

Signed-off-by: Sage Weil <sage@redhat.com>

debian/rules: install systemd files and /etc/default/ceph

Fix the systemd units to use /etc/default instead of /etc/sysconfig.  There
is probably a better way...

Signed-off-by: Sage Weil <sage@redhat.com>

Makefile.am: include etc in tarball

Signed-off-by: Sage Weil <sage@redhat.com>

ceph.spec: install /etc/sysconfig/ceph

Make it easy to enable jemalloc, too.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5869 from ceph/wip-rpm-dirs

ceph.spec: make /var/{lib,log,run} owned by ceph

Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #5540 from ceph/wip-selinux-radosgw

selinux: Update policy for radosgw

Reviewed-by: Milan Broz <mbroz@redhat.com>

make /var/run/ceph 770 ceph:ceph

This allows members of the ceph group to make librados clients (like the
ceph cli and qemu) create sockets in the default /var/run/ceph/* location.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph.spec: make /var/{lib,log,run} owned by ceph

Move creation of /var/lib/ceph and /var/run/ceph to ceph-common, too.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5871 from liewegas/wip-10328

common: add debug option to deliberately leak some memory

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge remote-tracking branch 'gh/wip-tcmalloc'

.gitignore: radosgw-object-expirer

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5881 from cernceph/wip-logrotate

logrotate: ignore exit status 1 from killall

Reviewed-by: Sage Weil <sage@redhat.com>

ceph.spec.in: ceph-radosgw should require ceph-selinux

Signed-off-by: Boris Ranto <branto@redhat.com>

selinux: Update policy for radosgw

The current SELinux policy does not cover radosgw daemon. This patch
introduces the SELinux support for radosgw daemon (civetweb only).

Signed-off-by: Boris Ranto <branto@redhat.com>

logrotate: ignore postrotate error for radosgw

Fix the annoying logrotate error for radosgw.

Fixes: #13033
Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>

logrotate: ignore exit status 1 from killall

If any of ceph-osd, ceph-mon, ceph-mds were not running then
killall -q will exit status 1, leading to anacron sending a mail

  error: error running shared postrotate script for '/var/log/ceph/*.log '

Fix by overriding the exit status with || true.

Fixes: #13033
Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>

Merge pull request #5800 from ceph/wip-user

ceph-disk: do async startup on upstart; fix dmcrypt create/activate bug

Reviewed-by: Loic Dachary <ldachary@redhat.com>

rocksdb: recognized --without-tcmalloc

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5031 from trociny/wip-10098

ceph-objectstore-tool: add mark-complete operation

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #5866 from jmunhoz/wip-requestid-on-error

rgw: include RequestId as part of the Error response

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #5859 from jecluis/wip-test-mon-ping

test: mon/mon-ping.sh: make sure 'ceph mon ping' works as expected

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5873 from dillaman/wip-rbd-missing-return

rbd: missing return statement within do_metadata_remove

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rbd: missing return statement within do_metadata_remove

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #5780 from dzafman/wip-12387

osd: When generating past intervals due to an import end at pg epoch

Reviewed-by: Sage Weil <sage@redhat.com>

common: add debug option to deliberately leak some memory

This will be used to verify leak detection tests are working.

Signed-off-by: Sage Weil <sage@redhat.com>

test/Makefile-client: ship LibradosTestStub.h in tarball

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5868 from jmunhoz/wip-x-amz-request-id-doc

doc: rgw: update x-amz-request-id status

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

Merge pull request #5863 from ceph/wip-13015

rgw: preserve all attrs if intra-zone copy

doc: rgw: update x-amz-request-id status

Signed-off-by: Javier M. Mellid <jmunhoz@igalia.com>

rgw: include RequestId as part of the Error response

The RGW error responses must contain a RequestId to be compliant with
the Amazon S3 errors. This RequestId is the ID of the request associated
with the error.

Fixes: #13020
Signed-off-by: Javier M. Mellid <jmunhoz@igalia.com>

test: mon/mon-ping.sh: make sure 'ceph mon ping' works as expected

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #5343 from hjwsm1989/wip-12442

  mon/MonClient: fix error in 'ceph ping mon.id'

Reviewed-by: Joao Eduardo Luis <joao@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-objectstore-tool: add mark-complete operation

It is supposed to be used as a last resort to fix a cluster that has
PGs in 'incomplete' state, using the following procedure:

1) stop the osd that is primary for the incomplete PG;
2) run:
  ceph-objectstore-tool --data-path ... --journal-path ... --pgid $PGID --op mark-complete
3) start the osd.

Fixes: #10098
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #5816 from dillaman/wip-librbd-gmock

Initial support for gmock-based unit tests in librbd

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #5557 from guangyy/wip-12666

rgw: expose the number of unhealthy workers through admin socket

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5786 from ceph/wip-12911

rbd export-diff should export data chunks in-order

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #5858 from jecluis/wip-test-mon-scrub

test: mon/mon-scrub.sh: have test running on make check

Reviewed-by: Loic Dachary <ldachary@redhat.com>

rgw: preserve all attrs if intra-zone copy

Fixes: #13015
Intra zone copy requires that all objects' attributes are preserved.
This was broken at commit: e41d97c8e38bb60d7e09e9801c0179efe7af1734

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #5442 from ceph/wip-12551

ReplicatedPG,Objecter: copy_get should include truncate_seq and size

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5490 from dwj192/master

rgw:add --reset-regions for regionmap update

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

test/Makefile.am: run mon/mon-scrub.sh as part of checks

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

test: mon/mon-scrub.sh: port clashed with other tests

This will allow the test to be run during make check.

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #5828 from lebauce/fix-swift-prefix

rgw: add delimiter to prefix only when path is specified

Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

ReplicatedPG,Objecter: copy_get should include truncate_seq and size

Otherwise, we break CephFS over cache tiers.

Fixes: #12551
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #5837 from tchaikov/wip-12968

mon: disable gmt_hitset if not supported

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5842 from dzafman/wip-cot-failure

test: Fix failure test to find message anywhere in stderr

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

test/librados: add test case for read object (off=0,len=0).

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

osd: make read(off=0,len=0) ec-object work.

When len=0, it mean read the whole object. It can work when reading
no-ec object off=0 & len=0. But for ec-object, it can't work.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #5845 from wonzhq/async-rx

AsyncMessenger: add instance name in debug log when processing msg

Reviewed-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #5779 from dzafman/wip-misc-fix-guangyy

osd: check the length of the map before accessing the first element

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5843 from dillaman/wip-12885

librbd: diff_iterate needs to handle holes in parent images

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #5592 from xinxinsh/wip-fix-rbd-metadata

fix bugs of rbd metadata operation

Reviewed-by: Haomai Wang <haomaiwang@gmail.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

test: Fix failure test to find message anywhere in stderr

Consolidate test_failure() and test_failure_tty()

Signed-off-by: David Zafman <dzafman@redhat.com>

rados: Fix usage for "notify" command

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #5834 from ceph/wip-leveldb-hang

mon: MonitorDBStore: make get_next_key() work properly

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4955 from kylinstorage/librados-2read-segmentation-fault-v1

bug fix: librados segmentation fault, when two read ops share one Aio…

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4925 from XinzeChi/wip-hit-set-size

osd: fix hit_set_map size for tier pool

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5723 from H3C/wip-msg-bugfix1

msg: we should set the socket options before connect or listen in order to have it take effect.

Reviewed-by: Haomai Wang <haomaiwang@gmail.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5829 from dachary/wip-12731-jerasure-warnings

erasure-code: fix gf-complete warnings

Reviewed-by: Kefu Chai <kchai@redhat.com>

AsyncMessenger: add instance name in debug log when processing msg

To better debug.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

rgw:add --reset-regions for regionmap update

Fixes: #12964
Signed-off-by: Weijun Duan <duanweijun@h3c.com>

Merge pull request #5781 from dzafman/wip-eccleanup

osd, test: Minor clean-up from fast-read and error handling ec changes

Reviewed-by: Kefu Chai <kchai@redhat.com>

librbd: diff_iterate needs to handle holes in parent images

If a clone's parent image snapshot includes a discarded extent,
this was previously causing an assert failure.  Instead, ignore
any discard holes in the parent image.

Fixes: #12885
Backport: hammer
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: new test case for librbd diff_iterate over discard extents

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #5838 from ceph/wip-12861

qa/workunits/rados/test_alloc_hint.sh: sudo to ls files

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5396 from XinzeChi/wip-scrub-promote

bug fix: osd: requeue_scrub when kick_object_context_blocked

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #5772 from yuyuyu101/wip-12912

AsyncConnection: Close connection when unregistered connection met WAIT

Merge pull request #5733 from cxwshawn/rgw-fix

rgw: init_rados failed leads to repeated delete

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #5809 from rzarzynski/wip-rgw_putobj_debug_improvement

rgw: improve debugs in RGWPutObj and RGWPutObj_ObjStore_SWIFT.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #5567 from guce/rgw_check_key_parameter_conflict

rgw: add key parameter conflict check for radosgw-admin command line.

Merge pull request #5819 from rzarzynski/wip-12950

rgw: don't append empty ETag HTTP header.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge remote-tracking branch 'gh/infernalis'

qa/workunits/rados/test_alloc_hint.sh: sudo to ls files

The osd data dir is owned by ceph and not readable by other
non-root users.

Fixes: #12861
Signed-off-by: Kefu Chai <kchai@redhat.com>