rgw: more data changes log implementation

Remove a bunch of hard coded stuff. Make renew thread updates
expiration. Only renew if there was more than one request through the
current window.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

RefCounteCond: keep return val, wait() returns it

It is necessary in some cases to notify waiters of the
actual return value for the action they were waiting on.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: changed data log renew thread

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: limit num of buckets in data changes log

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: use shared_ptr instead of RefCountedObject

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

lru_map: infrastructure for a bounded map

Useful for cache-like maps, where we want to control
the max number of entries in the map.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: data changes log, don't always send new requests

We may piggy back on older entries that hasn't expired yet.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: data changes log, naive implementation

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: decouple bucket data pool from bucket index pool

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: fix get_obj() with zero sized obj

Now that even zero sized objs have manifest a
test had to be modified.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: don't set shadow obj attr in object metadata

This is incorrect, was only right for pre-manifest objects.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw-admin: object stat also decodes policy

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: update cli test for radosgw-admin

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Makefile.am: add missing header file

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw-admin: add object stat command

for retrieving object metadata.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw-admin: bucket list also specifies object namespace

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: copy_obj uses req_id as tag

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: radosgw-admin bucket list --bucket lists bucket objects

Also lists tag for each object.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: call rgw_store_user_info() with objv_tracker

another rebase casualty

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw-admin: fix some more merge issues

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw-admin: fix user_id initialization

broken due to rebase

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: share object tag and index tag

object tag is now being written to the index, so that both
object and index hold the same tag. This is needed so that we
could know whether object and index refer to the same instance.

Also cleanup old legacy code that did atomic ops through cloning
objects.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: bucket index log fixes

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

radosgw-admin: bilog list gets marker and max-entries params

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

radosgw-admin, cls_rgw: list bucket index log

a new radosgw-admin command to list bucket index log.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_rgw: bucket index versioning

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: fix broken radosgw-admin user * commands

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_rgw, rgw: bucket index logs modifications

Add a log to the bucket index.

This commit also ties the "epoch" version that is kept
per index entry to the relevant ceph pool id. This fixes
the dependency on the specific pool where the object was
created, and makes it possible to copy rgw data pools.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: metadata rm

Still needs to fix the way we remove user entries; need to take
different path, similar to put_entry())

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: cls_log_trim boundary change

end_time is not inclusive

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: show metadata log through radosgw-admin

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: use new cls_log listing interface

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_log: adjust listing api

Listing api now also gets end time and a marker.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: log user operations

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: track object versions between reads and writes

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: initialize meta_mgr earlier

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

test_cls_log: remove warning

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_log: extend unitest

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_log: more fixes

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_log: unitest

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_log: fixes, other adjustments

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

objclass: provide new api for unique subop versioning

We need to be able to generate a unique identifier for
each subop. This can be useful e.g., if we want to keep multiple
omap data entries indexed by the same key. The unique id
is being generated by the current object id and the current
osd subop count. For write operations it's unique.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

osd: keep track of current osd subop num

This can be used later to generate unique subop versions
in compound write operations.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_log: trim works in chunks, returns -ENODATA when done

Also created a higher level interface that iterates until
done.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_log: a class to handle info sorted by timestamp

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: metadata list user, only show uids

don't show unrelated object names

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: user metadata updates also key version

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: add top level metadata handler

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: put metadata, other cleanups

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: metadata manager, api to list keys

Also, implement key listing for user metadata.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: metadata set/get infrastructure

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: rgw_get_system_obj() can return obj_version

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_version: add cls_version_read(ObjectReadOpeation&)

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

librados: add two more ObjectOperation::exec()

 - one that also gets out bufferlist and ret value pointer
 - one that gets a callback context

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: start tying metadata objs to version objclass

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_version: unitest

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_version: various fixes

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_version: create a new objclass

New objclass to track and modify objects versions.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: fix json decoding of rgw_bucket

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: regionmap update

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: get/set region map

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: zone list, setup changes

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

common/ceph_parser.cc: cleanup

remove extra logging to stdout

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: set region info, default region

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: can list regions, show default region info

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: some region/zone related cleanups/fixes

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: admin command to show region info

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: derr -> lderr

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: region creation

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: region management encoding/decoding changes

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: define region/zone data structures

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

v0.61

os/: default to dio for non-block journals

Workaround: #4910
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>

ceph-disk: use separate lock files for prepare, activate

Use a separate lock file for prepare and activate to avoid deadlock.  This
didn't seem to trigger on all machines, but in many cases, the prepare
process would take the file lock and later trigger a udev event and the
activate would then block on the same lock, either when we explicitly call
'udevadm settle --timeout=10' or when partprobe does it on our behalf
(without a timeout!).   Avoid this by using separate locks for prepare
and activate.  We only care if multiple activates race; it is
okay for a prepare to be in progress and for an activate to be kicked
off.

Signed-off-by: Sage Weil <sage@inktank.com>

ceph-test.install: add ceph-monstore-tool and ceph-osdomap-tool

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph.spec.in: remove twice listed ceph-coverage

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph.spec: add some files to ceph

Add installed, but not packaged files to ceph-test (ceph-monstore-tool,
ceph-osdomap-tool) rpm file section.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mon: fix init sequence when not daemonizing

We made the common_init_finish and chdir conditional on daemonize in commit
2e0dd5ae6c8751e33d456b2b06c1204b63db959a, breaking init (asok at least)
when -f is specified (as with upstart).

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

mon: avoid null deref in Monitor::_mon_status()

mikedawson reports:

*** Caught signal (Segmentation fault) **
 in thread 7f40ce270700

 ceph version 0.60-801-g7ec0151 (7ec01513970b5a977bdbdf60052b6f6e257d267e)
 1: /usr/bin/ceph-mon() [0x59d550]
 2: (()+0xfbd0) [0x7f40d3e38bd0]
 3: (operator<<(std::ostream&, entity_name_t const&)+0x16) [0x4d7c46]
 4: (operator<<(std::ostream&, entity_inst_t const&)+0x1b) [0x4d837b]
 5: (Monitor::_mon_status(std::ostream&)+0x2ce) [0x4d284e]
 6: (Monitor::do_admin_command(std::string, std::string, std::ostream&)+0x4f) [0x4d652f]
 7: (AdminHook::call(std::string, std::string, ceph::buffer::list&)+0x68) [0x4efa38]
 8: (AdminSocket::do_accept()+0x451) [0x64ab81]
 9: (AdminSocket::entry()+0x398) [0x64c528]
 10: (()+0x7f8e) [0x7f40d3e30f8e]
 11: (clone()+0x6d) [0x7f40d237ae1d]

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

ceph.spec: require xfsprogs

This is needed when creating new OSDs (via ceph-disk).  At least for most
people.  Eventually we'll want to include btrfs here.

Signed-off-by: Sage Weil <sage@inktank.com>

init-ceph: update osd crush map position on start

This is what the upstart ceph-osd.conf does; we need to do the same so that
new OSDs (e.g., that ceph-deploy creates) get added to the crush map.

Signed-off-by: Sage Weil <sage@inktank.com>

mon: fork early to avoid leveldb static env state

leveldb has static state that prevents it from recreating its worker thread
after our fork(), even when we close and reopen the database (tsk tsk!).
Avoid this by forking early, before we touch leveldb.

Hide the details in a Preforker class.  This is modeled after what
ceph-fuse already does; we should convert it later.

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

Merge remote-tracking branch 'gh/wip-mon-rank' into next

Reviewed-by: Greg Farnum <greg@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>

tools/: add paranoid option to ceph-osdomap-tool

Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

osd: default 'osd leveldb paranoid = false'

Signed-off-by: Sage Weil <sage@inktank.com>

librados,client: bump mount timeout to 5 min

30 seconds is pretty short.

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

OSD: also walk maps individually for start_split in consume_map()

We need to go map-by-map to get the parents right in consume_map()
just as we must in load_pgs().

Fixes: 4884
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

rgw: increase startup timeout to 5 min

30s is too short.

Signed-off-by: Sage Weil <sage@inktank.com>

Merge branch 'wip-paranoid' into next

Merge remote-tracking branch 'gh/wip-doc-cuttlefish' into next

Merge remote-tracking branch 'upstream/wip_4884' into next

Fixes: #4884
Reviewed-by: Greg Farnum <greg@inktank.com>

Makefile,gitignore: ceph-monstore-tool, not ceph_monstore_tool

Signed-off-by: Samuel Just <sam.just@inktank.com>

Makefile: put ceph_monstore_tool in bin_DEBUGPROGRAMS

Signed-off-by: Samuel Just <sam.just@inktank.com>

tools: ceph-osdomap-tool.cc

Add tool for dumping info from osd omap.

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSD: load_pgs() should fill in start_split honestly

In load_pgs(), we previously called assigned children starting
at the loaded pg created between its stored epoch and the current
osdmap to have that pg as their parent.  This is not correct, some
of the children may have been split in subsequent epochs from children
split in earlier epochs.  Instead, do each map individually.

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSD: cancel_pending_splits needs to cancel all descendants

expand_pg_num() and load_pgs() may result in a pg with children
in pending_splits which also have children in pending_splits (etc).

Signed-off-by: Samuel Just <sam.just@inktank.com>

osd: add --osd-leveldb-paranoid flag

Signed-off-by: Sage Weil <sage@inktank.com>

mon: add --mon-leveldb-paranoid flag

This is sort of equivalent to an fsck.

Signed-off-by: Sage Weil <sage@inktank.com>

dumper: fix Objecter locking

Locking expectations changed at some point, and the Dumper wasn't
updated to comply:
1) We need to take the lock for Objecter, as it
doesn't do so on its own any more.
2) We need to drop the lock in several places so that Objecter
can take delivery of messages

Signed-off-by: Greg Farnum <greg@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>

Revert "PaxosService: use get and put for version_t"

This reverts commit e725c3e210b244e090d70c77d937c94f4f63a2be.

These inadvertantely got rid of the prefix portion of the key, which
lead to overwriting the wrong keys.

Fixes: #4872
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>