Merge remote-tracking branch 'gh/infernalis'

Merge pull request #5916 from xiaoxichen/fix1

os/OSD.cc cast osd_max_write_size to int64_t

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5940 from flyd1005/master

doc: Fixes a wrong directory name and a broken hyperlink

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

Merge pull request #5912 from badone/bug_7822

examples: hello_world.cc: fix the displayed content

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5933 from dachary/wip-max-stale

common: osd_pg_epoch_persisted_max_stale < map_cache_size

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: fix a broken hyperlink

Signed-off-by: Li Peng <lip@dtdream.com>

Merge pull request #5934 from cernceph/wip-logrotate

logrotate: logs are now owned by ceph:ceph

Reviewed-by: Sage Weil <sage@redhat.com>

doc: Fixes a wrong directory name.

Signed-off-by: Li Peng <lip@dtdream.com>

Merge pull request #5937 from ukernel/wip-13067

Wip 13067

Reviewed-by: John Spray <john.spray@redhat.com>

Examples: hello_world.cc, content displayed after read is not null terminated.

Copy contents of bufferlist to a string before sending to stdout since a
bufferlist is not null terminated.

Fixes: #7822
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

mds: reset MDSRank heartbeat timeout even when MDS is laggy

Fixes: #13067
Signed-off-by: Yan, Zheng <zyan@redhat.com>

common: osd_pg_epoch_persisted_max_stale < map_cache_size

1ff51a299d20dd73a95ba55fdec57498b8c71c13 reduced map_cache_size to 200,
osd_pg_epoch_persisted_max_stale must be lowered because it does not
make sense for it to be the equal or greater than map_cache_size.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

logrotate: logs are now owned by ceph:ceph

Add the su directive to avoid this error during logrotate:

error: skipping "/var/log/ceph/ceph-osd.0.log" because parent
directory has insecure permissions (It's world writable or writable
by group which is not "root") Set "su" directive in config file to
tell logrotate which user/group should be used for rotation.

Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>

os/OSD.cc cast osd_max_write_size to int64_t

Otherwise overflow may occur and negative values generated.

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

Merge pull request #5923 from ceph/wip-limits

set limits in systemd unit files

systemd: eliminate ceph-rgw tmpfiles.d file

This is for storing the rgw socket files for fastcgi, which we do not
want to enable by default.

Signed-off-by: Sage Weil <sage@redhat.com>

systemd: no need to preprocess ceph-osd@service

This used to be necessary but now is not.

Signed-off-by: Sage Weil <sage@redhat.com>

systemd: set nofile limit in unit files

Make it big so hopefully nobody has to change it.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: drop default map cache size from 500 -> 200

OSDs don't need this from each other anymore since they send
a pg_interval_map_t with pg notify.  This cache is only used
during startup when loading things up (and 200 is still > the
max advance of 150), and when clients appear with super old maps
(pretty unusual unless they are idle for long periods).

It should mean a pretty significant cut in memory...

Signed-off-by: Sage Weil <sage@redhat.com>

kill /etc/security/limits.d/ceph

This only affects login shells; not useful.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5901 from yuyuyu101/wip-13001

AsyncMessenger: Kepp file_lock hold when accessing its event field

Merge pull request #5626 from majianpeng/objecter

Objecter: fix bus about objecter.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5840 from liewegas/wip-12991

qa/workunits/cephtool/test.sh: make mds epoch check more tolerant

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge remote-tracking branch 'gh/infernalis'

Merge pull request #5686 from liewegas/wip-hammer-feature

require osds are hammer v0.94.4 or later before upgrading past hammer

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5905 from ceph/wip-rpm-notcmalloc

ceph.spec: respect CEPH_EXTRA_CONFIGURE_ARGS

Merge pull request #5904 from ceph/wip-submodules

fix a few submodule issues

Reviewed-by: Loic Dachary <ldachary@redhat.com>

.gitignore: ignore src/ceph.tmpe

It makes the gitbuilders unhappy.  Not sure where it comes from...

Signed-off-by: Sage Weil <sage@redhat.com>

gmock: ignore *.pyc

Signed-off-by: Sage Weil <sage@redhat.com>

rocksdb: ignore m4

Signed-off-by: Sage Weil <sage@redhat.com>

ceph.spec: respect CEPH_EXTRA_CONFIGURE_ARGS

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/cephtool/test.sh: make mds epoch check more tolerant

This can race with an actual mdsmap epoch update for some other
reason.  We just need to make sure the epoch *increased*, not that
it is exactly old + 1.

Fixes: #12991
Signed-off-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/wip-10617-again'

Merge pull request #5334 from majianpeng/read-len-zero-for-ec

make read ec-object(offset=0, len=0) can work like read no-ec object.

Reviewed-by: Kefu Chai <kchai@redhat.com>

sd/PG: tolerate missing pgmeta object

Bug 10617 left stray PG dirs around in firefly.  Hammer correctly
ignores these, assuming they are leftover cruft.  We broke this
when we dropped compat support in cd4e676e6d45c8166290ef834d73c2a0bda98fa2

See also #13060.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: allow peek_map_epoch to return an error

Allow PG::peek_map_epoch to return an error indicating the PG
should be skipped.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5776 from wonzhq/tmap-update

osd: force promote for ops which ec base pool can't handle

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

osd/ReplicatedPG: remove stray debug line

This snuck in

Signed-off-by: Sage Weil <sage@redhat.com>

AsyncMessenger: Kepp file_lock hold when accessing its event field

When process_event get a file event, other thread may delete this event later
but before we do event

Fix #13001
Signed-off-by: Haomai Wang <haomai@xsky.io>

Merge pull request #5896 from liewegas/wip-fix-spec

ceph.spec: include /etc/sysconfig/ceph

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

ceph.spec: include /etc/sysconfig/ceph

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5895 from jcsp/wip-doc-default-pools

doc: remove references to default data/metadata pools

doc: remove references to default data/metadata pools

These haven't existed since 0.84 -- the cephfs documentation
was updated at the time, but there were also references in the
rados documentation.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #5803 from ceph/wip-debian-systemd

debian systemd support; /etc/{sysconfig,default}/ceph; easy jemalloc enablement

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

ceph-common: explicitly trigger /run/ceph creation

This doesn't happen magically on debian.

Signed-off-by: Sage Weil <sage@redhat.com>

systemd: tmpfiles.d in /run, not /var/run

Signed-off-by: Sage Weil <sage@redhat.com>

debian: fix location of tmpfiles.d

/usr/lib/tmpfiles.d, not /lib/tmpfiles.d

Signed-off-by: Sage Weil <sage@redhat.com>

debian/rules: install systemd files and /etc/default/ceph

Fix the systemd units to use /etc/default instead of /etc/sysconfig.  There
is probably a better way...

Signed-off-by: Sage Weil <sage@redhat.com>

Makefile.am: include etc in tarball

Signed-off-by: Sage Weil <sage@redhat.com>

ceph.spec: install /etc/sysconfig/ceph

Make it easy to enable jemalloc, too.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5869 from ceph/wip-rpm-dirs

ceph.spec: make /var/{lib,log,run} owned by ceph

Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #5540 from ceph/wip-selinux-radosgw

selinux: Update policy for radosgw

Reviewed-by: Milan Broz <mbroz@redhat.com>

make /var/run/ceph 770 ceph:ceph

This allows members of the ceph group to make librados clients (like the
ceph cli and qemu) create sockets in the default /var/run/ceph/* location.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph.spec: make /var/{lib,log,run} owned by ceph

Move creation of /var/lib/ceph and /var/run/ceph to ceph-common, too.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5871 from liewegas/wip-10328

common: add debug option to deliberately leak some memory

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge remote-tracking branch 'gh/wip-tcmalloc'

.gitignore: radosgw-object-expirer

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5881 from cernceph/wip-logrotate

logrotate: ignore exit status 1 from killall

Reviewed-by: Sage Weil <sage@redhat.com>

ceph.spec.in: ceph-radosgw should require ceph-selinux

Signed-off-by: Boris Ranto <branto@redhat.com>

selinux: Update policy for radosgw

The current SELinux policy does not cover radosgw daemon. This patch
introduces the SELinux support for radosgw daemon (civetweb only).

Signed-off-by: Boris Ranto <branto@redhat.com>

logrotate: ignore postrotate error for radosgw

Fix the annoying logrotate error for radosgw.

Fixes: #13033
Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>

logrotate: ignore exit status 1 from killall

If any of ceph-osd, ceph-mon, ceph-mds were not running then
killall -q will exit status 1, leading to anacron sending a mail

  error: error running shared postrotate script for '/var/log/ceph/*.log '

Fix by overriding the exit status with || true.

Fixes: #13033
Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>

Merge pull request #5800 from ceph/wip-user

ceph-disk: do async startup on upstart; fix dmcrypt create/activate bug

Reviewed-by: Loic Dachary <ldachary@redhat.com>

rocksdb: recognized --without-tcmalloc

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5031 from trociny/wip-10098

ceph-objectstore-tool: add mark-complete operation

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #5866 from jmunhoz/wip-requestid-on-error

rgw: include RequestId as part of the Error response

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #5859 from jecluis/wip-test-mon-ping

test: mon/mon-ping.sh: make sure 'ceph mon ping' works as expected

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5873 from dillaman/wip-rbd-missing-return

rbd: missing return statement within do_metadata_remove

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rbd: missing return statement within do_metadata_remove

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #5780 from dzafman/wip-12387

osd: When generating past intervals due to an import end at pg epoch

Reviewed-by: Sage Weil <sage@redhat.com>

common: add debug option to deliberately leak some memory

This will be used to verify leak detection tests are working.

Signed-off-by: Sage Weil <sage@redhat.com>

test/Makefile-client: ship LibradosTestStub.h in tarball

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5868 from jmunhoz/wip-x-amz-request-id-doc

doc: rgw: update x-amz-request-id status

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

Merge pull request #5863 from ceph/wip-13015

rgw: preserve all attrs if intra-zone copy

doc: rgw: update x-amz-request-id status

Signed-off-by: Javier M. Mellid <jmunhoz@igalia.com>

rgw: include RequestId as part of the Error response

The RGW error responses must contain a RequestId to be compliant with
the Amazon S3 errors. This RequestId is the ID of the request associated
with the error.

Fixes: #13020
Signed-off-by: Javier M. Mellid <jmunhoz@igalia.com>

test: mon/mon-ping.sh: make sure 'ceph mon ping' works as expected

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #5343 from hjwsm1989/wip-12442

  mon/MonClient: fix error in 'ceph ping mon.id'

Reviewed-by: Joao Eduardo Luis <joao@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-objectstore-tool: add mark-complete operation

It is supposed to be used as a last resort to fix a cluster that has
PGs in 'incomplete' state, using the following procedure:

1) stop the osd that is primary for the incomplete PG;
2) run:
  ceph-objectstore-tool --data-path ... --journal-path ... --pgid $PGID --op mark-complete
3) start the osd.

Fixes: #10098
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #5816 from dillaman/wip-librbd-gmock

Initial support for gmock-based unit tests in librbd

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #5557 from guangyy/wip-12666

rgw: expose the number of unhealthy workers through admin socket

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5786 from ceph/wip-12911

rbd export-diff should export data chunks in-order

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #5858 from jecluis/wip-test-mon-scrub

test: mon/mon-scrub.sh: have test running on make check

Reviewed-by: Loic Dachary <ldachary@redhat.com>

rgw: preserve all attrs if intra-zone copy

Fixes: #13015
Intra zone copy requires that all objects' attributes are preserved.
This was broken at commit: e41d97c8e38bb60d7e09e9801c0179efe7af1734

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #5442 from ceph/wip-12551

ReplicatedPG,Objecter: copy_get should include truncate_seq and size

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5490 from dwj192/master

rgw:add --reset-regions for regionmap update

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

test/Makefile.am: run mon/mon-scrub.sh as part of checks

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

test: mon/mon-scrub.sh: port clashed with other tests

This will allow the test to be run during make check.

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #5828 from lebauce/fix-swift-prefix

rgw: add delimiter to prefix only when path is specified

Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

ReplicatedPG,Objecter: copy_get should include truncate_seq and size

Otherwise, we break CephFS over cache tiers.

Fixes: #12551
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #5837 from tchaikov/wip-12968

mon: disable gmt_hitset if not supported

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5842 from dzafman/wip-cot-failure

test: Fix failure test to find message anywhere in stderr

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

test/librados: add test case for read object (off=0,len=0).

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

osd: make read(off=0,len=0) ec-object work.

When len=0, it mean read the whole object. It can work when reading
no-ec object off=0 & len=0. But for ec-object, it can't work.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #5845 from wonzhq/async-rx

AsyncMessenger: add instance name in debug log when processing msg

Reviewed-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #5779 from dzafman/wip-misc-fix-guangyy

osd: check the length of the map before accessing the first element

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5843 from dillaman/wip-12885

librbd: diff_iterate needs to handle holes in parent images

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #5592 from xinxinsh/wip-fix-rbd-metadata

fix bugs of rbd metadata operation

Reviewed-by: Haomai Wang <haomaiwang@gmail.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

test: Fix failure test to find message anywhere in stderr

Consolidate test_failure() and test_failure_tty()

Signed-off-by: David Zafman <dzafman@redhat.com>

rados: Fix usage for "notify" command

Signed-off-by: David Zafman <dzafman@redhat.com>