journal: prevent race injecting new records into overflowed object

The recorded added after the overflow will not have been detached
from the object recorder.

Fixes: #15202
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

os/filestore: fix warning

os/filestore/FileStore.cc: In member function ‘int FileStore::_zero(const coll_t&, const ghobject_t&, uint64_t, size_t)’:
os/filestore/FileStore.cc:3294:24: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
       if (offset + len > st.st_size) {
                        ^

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #7932 from theanalyst/mailmap/10.0.4

mailmap for 10.0.4

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #8121 from liewegas/wip-15136

mon: remove 'mds setmap'

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #8137 from ceph/wip-15146-ceph-dep

rpm: remove sub-package dependencies on "ceph"

Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Jun Huang <hjwsm1989@gmail.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #7838 from rldleblanc/pg_prime-enable

mon: enable 'mon osd prime pg temp' by default

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7939 from yangdongsheng/remove_v1

mon: standardize Ceph removal commands

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7721 from xiexingguo/xxg-wip-fixdc

os/filestore: fix result handling logic of destroy_collection

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7754 from xiexingguo/xxg-wip-14845

osd: fix inaccurate counter and skip over queueing an empty transaction

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7970 from Yan-waller/yj-wip-async

msg/async: _try_send trim already sent for outcoming_bl more efficient

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #8127 from ceph/wip-15122

ceph_test_msgr: reduce test size to fix memory size

Reviewed-by: Sage Weil <sage@redhat.com>

Merge branch 'wip-13990' of git://github.com/tchaikov/ceph

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8160 from yuyuyu101/jewel-spdk

os/bluestore/NVMEDevice: make IO thread using dpdk launch

Merge pull request #8163 from liewegas/wip-15151

osd/ReplicatedPG: clear watches on change after applying repops

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #7359 from songbaisen/song9

mon: consider pool size when creating pool

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7674 from majianpeng/bluestore-sync-transaction

os/bluestore: make bluestore_sync_transaction = true can work.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7860 from majianpeng/bluestores-clone-overlay

os/bluestore: clone overlay data

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7930 from ceph/wip-debian-librgw-tests

debian: package librgw_file* tests

Merge remote-tracking branch 'gh/wip-strict-debian-deps'

Merge pull request #7957 from majianpeng/bluestore-read

os/filestore: require offset == length == 0 for full object read; add test

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #8039 from gregsfortytwo/wip-14962-greg

osd/ReplicatedPG: be more careful about calling publish_stats_to_osd()

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8096 from liewegas/wip-15073

osd: ensure new osdmaps commit before publishing them to pgs

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8167 from theanalyst/mstart/rgw_ports

mstart: start rgw on different ports as well

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

osd: assert that we don't try to use an OSDMap > superblock.newest_map

handle_osd_map now adds OSDMaps to the cache as it receives them
but before they are stable.  Ensure that we don't try to use them.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: commit osdmaps before exposing them to PGs

handle_osd_map and the PGs use different sequencers when writing
their updates.  We therefore need to make sure new osdmaps are
committed to disk before we expose them to PGs, lest they update
their info to reference a new osdmap that hasn't actually
committed yet.

This doesn't happen with FileStore because transactions are
ordered when they are queued, but it does affect BlueStore.

Fix by splitting handle_osd_map into two phases, one that just
persists stuff, and the second half that publishes the new maps to
the rest of the OSD.

Fixes: #15073
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #8170 from yehudasa/wip-rgw-sync-fixes-3

Wip rgw sync fixes 3

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mon: remove 'mds setmap' command

This command has no discernable purpose, and is difficult and
dangerous to support anyway.

Fixes: #15136
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #8179 from zhjwpku/zhjwpku

doc/architecture.rst: remove redundant word "across"

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/architecture.rst: remove redundant word "across"

Signed-off-by: Zhao Junwang <zhjwpku@gmail.com>

rgw: take a reference to the cr in async cr workers

and drop the cr when worker is done. The cr can be cleaned before
worker is done, so we should have it stick around as long as the
worker is alive.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: finalize reqs through a temporary set

can't call _finish_request() on iter->second when iterating, it clobbers the
map.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: update data_log only when completing the op

It doesn't make sense to update it earlier, the zone that follows will not sync before
it sees the completion.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #8169 from ceph/wip-15041

doc: initial draft of RBD mirroring admin documentation

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc: initial draft of RBD mirroring admin documentation

Fixes: #15041
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8168 from linuxbox2/rgw-ldap-fixes

Rgw ldap fixes

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-ldap: conditional build

Permit building without LDAP support--support is enabled by default.
Tested with CMake and autotools standalone builds.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #8165 from tchaikov/wip-fix-rgw-encoding

rgw/rgw_common.h: fix the RGWBucketInfo decoding

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge remote-tracking branch 'gh/jewel'

osd: populate the trim_thru epoch using MOSDMap.oldest_map

instead of filling MOSDMap with the local oldest_map, we share
the maximum MOSDMap.oldest_map received so far with peers. That
way one OSD's failure to trim won't prevent it from sharing with
others that they are allowed to trim.

Fixes: #13990
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8158 from dillaman/wip-rbd-mirror-fixes

rbd-mirror: fix image replay test failures

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mstart: start rgw on different ports as well

similar to the way we set CEPH_PORT, set CEPH_RGW_PORT as well so that
we've rgw instances starting on different ports

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #8133 from ceph/wip-15145-cephfs-tools

packaging: move cephfs repair tools to ceph-common

Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

rgw/rgw_common.h: fix the RGWBucketInfo decoding

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8154 from tchaikov/wip-fix-rgw-decoding

rgw/rgw_common.h: fix the RGWBucketInfo decoding

Reviewed-by: Loic Dachary <ldachary@redhat.com>

osd/ReplicatedPG: clear watches on change after applying repops

apply_and_flush_repops does any in-flight request side-effects,
including setting up new watches.  Clear out watches after that
so we don't leave any behind.

Fixes: #15151
Signed-off-by: Sage Weil <sage@redhat.com>

bluestore/NVMEDevice: make IO thread using dpdk launch

Previously we will call rte_eal_init in caller thread as well as dpdk
resources will init in that thread. It will cause all threads spawned
later will bind to the specified core.

And io thread will be spawned via dpdk api, otherwise, dpdk memory
has huge performance degraded problem.

Signed-off-by: Haomai Wang <haomai@xsky.com>

qa/workunits/rbd: rbd_mirror was extracting the incorrect image id

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits/rbd: use unique logs for each rbd-mirror daemon

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: refetch active object before defaulting to new tag

If a live replay is in progress, it's possible that object offset 0
was pulled and a new tag is discovered before the current object is
(re-)pulled to determine that the old tag still has entries remaining.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7894 from SUSE/wip-14828

 ceph.spec.in: do not install Ceph RA on systemd platforms

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

rgw/rgw_common.h: fix the RGWBucketInfo decoding

introduced by 416234b

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph.spec.in: Make ceph-common require libcephfs1

We recently moved fsck/repair tools to ceph-common package. We should
also make the version/release requirements tight.

Signed-off-by: Boris Ranto <branto@redhat.com>

Merge pull request #8107 from liewegas/wip-crush

crush: fix cli tests for new crush tunables

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8073 from ceph/wip-fsmap-upgrade-jewel

mds: fix FSMap upgrade with daemons in the map

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #8142 from tchaikov/wip-rgw-ftbfs

cls/rgw: fix FTBFS

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: fix FTBFS

BIVerObjEntry::mtime() is a method not a member variable

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8002 from rzarzynski/wip-rgw-putobjmeta

rgw: add support for metadata upload during PUT on Swift container.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #8138 from dillaman/wip-15141

rbd: add support for mirror image promotion/demotion/resync

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #8139 from yehudasa/wip-rgw-implicit-tenant

rgw: Allow an implicit tenant in case of Keystone

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

librbd: implement mirror image resync request API

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: track if a mirror image resync has been requested

A secondary image will store its resync request to the primary
image's journal. When the local rbd mirror process detects the
flag being set, it can initiate a resync from the remote.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: add placeholder for marking clients are disconnected

When a client is disconnected from the journal, entries can be
pruned regardless of the affected client's commit position.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: initial implementation of mirror image promote/demote

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: helper methods for manipulating journal tags from API

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: dump mirroring primary status on info command

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: expose image mirroring primary/secondary status

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test: update rbd cli tests to incorporate new mirror image commands

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: add mirror image promote/demote/resync commands

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: stub API methods for mirror image promotion/demotion/resync

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rgw: Allow an implicit tenant in case of Keystone

This, unfortunately, introduces possible double lookups, but
they should be cached. Also, the logic appears somewhat convoluted,
although the intent is quite simple: if you're an OpenStack user
with a Keystone authentication, we allow an implicit tenant of
the same name as the user.

Signed-off-by: Pete Zaitcev <zaitcev@redhat.com>
Conflicts:
src/rgw/rgw_swift.cc

Merge pull request #8095 from yehudasa/wip-rgw-sync-fixes

rgw sync fixes

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: data shard sync doesn't exit on error

We don't handle the error at the top level, so no
need to exit. Just continue with the backoff mechanism.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #8108 from yehudasa/wip-rgw-highres-3

Wip rgw highres 3

rgw ldap: s/memberattr/dnattr/;

The parameter really is dnattr (attribute used to form DNs), not
memberattr (attribute used to form group member strings).

Switch to the correct name for this attribute, to avoid confusion--
especially if we ever apply group membership checks in future.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw: propagate low time precision for user operations

When user set a conditional timestamp to compare (e.g. if-modified-since),
it uses a low precision time resolution. Need to adjust checks for that.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: use higher precision time when encoding / decoding external apis

When dealing with external entities (e.g., remote rgw), use higher precision time
format for object mtime.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

utime: add gmtime_nsec()

higher precision than gmtime()

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: extend date/time parsing

So that we can parse correctly time encoded using utime_t::gmtime()

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #7631 from liewegas/wip-agent-flush-low

osd: consider high/low mode when putting agent to sleep

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #8040 from ukernel/jewel-15050

mds: fix stray purging in 'stripe_count > 1' case

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #7713 from ukernel/wip-filelock-deadlock

Wip filelock deadlock

Reviewed-by: John Spray <john.spray@redhat.com>

packaging: remove sub-package dependencies on "ceph"

One side effect of the recent package split (in #10587) is that "ceph"
is now an empty meta-package.

We should not have any packages that depend on "ceph" any more. Change
the various sub-packages to either drop the dependency altogether, or
just Require: ceph-base instead.

Fixes: #15146
Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

osd: handle_osd_map whitespace

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #8136 from athanatos/wip-split-lfn

osd: filestore: fix race condition with split vs collection_move_rename and long object names

Reviewed-by: Sage Weil <sage@redhat.com>

packaging: move cephfs repair tools to ceph-common

The CephFS repair tools are generally useful to run on any node, not
just the MDS nodes themselves.

Move the utilities from the ceph-mds package to the ceph-common package.

Fixes: #15145
Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #8132 from dillaman/wip-librbd-memcheck

librbd: correct issues discovered via valgrind memcheck

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

journal: reschedule watch if no entries available during live replay

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: correct valgrind memcheck errors

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8120 from SUSE/wip-15138

RPM: fix openldap and openssl build dependencies for SUSE

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Jun Huang <hjwsm1989@gmail.com>

Merge pull request #8114 from liewegas/wip-tmap-migrate

qa/workunits/rados/test.sh: test tmap_migrate

Merge pull request #8119 from dillaman/wip-qa-rbd-copy

rbd: clone operation should default to image format 2

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #8090 from trociny/fixup-rbd-mirror-test

test: fixup and improvements for rbd-mirror test

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #8128 from jcsp/wip-15135

tools/cephfs: fix tmap_upgrade

Merge pull request #8129 from dillaman/wip-qa-rbd-permissions

test: fix issues discovered via the rbd permissions test case

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #8124 from dillaman/wip-qa-rbd-lock-fence

test: handle exception thrown from close during rbd lock test

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/workunits/rbd: disable deep-flatten during permissions testing

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tools/cephfs: fix tmap_upgrade

Move it up to handle the command before we try
and initialize DataScan::driver, because drivers
require a target filesystem, and that doesn't
apply to this command.

Fixes: #15135
Signed-off-by: John Spray <john.spray@redhat.com>

rbd: snap list should open image as read-only

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test_msgr: reduce test size to fix memory size

Fix #15122
Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge remote-tracking branch 'gh/jewel'