Client.cc: rename client_logger -> client_counters

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Rename ProfLogger to Perfcounters: part 2

Part 2: class names, variable names

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Rename ProfLogger to PerfCounters

Part 1: file names

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

re-enable proflogger test

Re-enable the proflogger test. This time, it doesn't test the basic
socket functionality (that is what test/admin_socket.cc is for).

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

test/admin_socket: test message 0

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Move ProfLogger socket code into admin_socket

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

ProfLogger: add request codes

The client now starts by sending a 4-byte request code.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

ProfLogger: use double quotes, add commas

Technically, JSON always uses double quotes, even though Javascript
often uses single quotes for strings. Also add a comma that had been
omitted.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

ProfLogger: enable by default for daemons

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Put each ProfLogger's data into its own object

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

ProfLogger: don't start threads until it's safe

Introduce a new configuration variable, internal_safe_to_start_threads.
This will be set by common_init_finish once it is safe to start threads.
That will trigger callbacks for any configuration observers listening
for this event.

This is used by ProfLogger to hold off on starting the UNIX domain
socket thread until it is safe to do so.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

vstart.sh: use out/proflog.$name

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

proflogger: Fix compiler warnings

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

proflogger: test multiple profloggers

Test multiple profloggers, and removing profloggers.

Add a clear function which removes all profloggers.

Make the order in which profloggers are output to JSON determininstic.

Const cleanup on get() methods.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

test/profloger: test fl_avg variables

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

test/proflogger: test JSON formatting a bit

Test JSON formatting a bit. Use single quotes in our JSON because that
seems to make things easier.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

proflogger: Unlink our UNIX domain sockets on exit

When we're exiting normally, we ought to unlink our UNIX domain sockets.

Also fix a bug in ProfLogThread, and enable some more tests.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

test/proflogger.cc: read length of message first

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

ProfLogger: write out length of message first

Write out the length of the message first, so that it's easier to write
clients. Also, serialize ProfLogger instances to memory rather than
directly to the socket, to avoid blocking while holding one of the
ProfLogger locks.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

test/proflogger: Add TeardownSetup and SimpleTest

common/ProfLogger.cc: check the sock_path length.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Add test/proflogger.cc, fix ProfLogger::init()

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Rework ProfLogger

* Replace existing proflogger config options with "profiling_logger_uri".
This option controls profiling logger sinks.

* ProfLogger: replace file-writing code with code that sends the
information over a UNIX domain socket.

* handle_conf_change is now fully and correctly implemented. We never
read from the md_config_t structure except in this function, so there
are no races. We re-create the thread when the settings change (no need
for SIGHUP, etc.)

* Replace the single big lock with a lock per Proflogger.

* No need for favg any more; just use fset everywhere for floating-point
variables.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

ProfLoggerBuilder: new API

There are now three different types of variables that can be created:
u64, float, float average.

The distinction between "inc" and "set" variables is gone, since there
will be no clearing of variables in the new Proflogger.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Introduce ProfLoggerBuilder

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

ProfLogger: don't return val from inc/set/etc.

Nobody was using those return values.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Remove lingering references to g_conf

These were harmless, but silly.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rgw: fix range dump

mds: fix protocol versions

I screwed this up in a928f767b974cd36f4be6ea618c5cec366d28362.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

recalc_op_target must give an error if pool gone

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Makefile: remove dup testrados_list_parallel_SOURCES

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge remote branch 'origin/stable'

Merge remote branch 'origin/wip-1287'

rgw: fix compilation for certain architectures

botol_tool: add --xattr argument for PUT

boto_tool can now set extended attributes when performing a PUT
operation.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

boto_tool: add --get-obj-xattr

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rgw_op: mime-encode ugly xattrs in PUT

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rgw_common: remove unused variable

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

mime encoding: encode control chars

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Add control character detection

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

bloom filter: update to latest version

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

filepath: remove asserts

These were included as a safety check and haven't been triggered, so
there's no point slowing anything down with them.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

honor CINIT_FLAG_NO_DEFAULT_CONFIG_FILE

Don't use CEPH_CONF_FILE_DEFAULT when CINIT_FLAG_NO_DEFAULT_CONFIG_FILE
is set.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

crushtool: don't parse any generic args

Notably, crushtool uses -c, which elsewhere means parse this config file.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'rgw-sync-cache'

Conflicts:
src/rgw/rgw_op.cc

rgw: configurable way to enable/disable cache

ReplicatedPG: project changes to clone_overlap

Previously, changes to clone_overlap were incorrect since make_writeable
is called after do_osd_ops.  Now, ctx->modified_ranges will hold the set
of offsets modified during the transaction to be applied to
clone_overlap during make_writeable.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

Merge branch 'wip-codingstyle'

CodingStyle: whitespace

Signed-off-by: Sage Weil <sage@newdream.net>

CodingStyle: final decisions?

- _d suffix for naked struct/class types (not _t!)

- m_ prefix for class members

- prefer braces for single line ifs.

Signed-off-by: Sage Weil <sage@newdream.net>

ReplicatedPG: sub_op_push fix

We need to iterate over the intersection of data_subset and data_needed,
not the original data_needed.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

Merge remote branch 'origin/wip-1257'

Don't build build tests unless requested

Build tests (that check if there are unresolved symbols in libraries)
can slow down the build a lot. We should only enable them when
developers need them.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rados parallel tests: prepend "test"

Just to avoid any confusion, all these test apps should begin with the
word "test".

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rados_list_parallel: add test of list-while-modify

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rgw: unwatch cache sync object before exiting

testrados: fix

Change testrados back to a simple single-threaded test. The
multi-threaded tests need to be properly designed to run multi-threaded.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

systest_runnable: print line when joining runnables

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Create rados_delete_pools_parallel test

* add rados_delete_pools_parallel test

* change RETURN_IF_NOT_VAL -> RETURN1_IF_NOT_VAL.
We want to return a non-zero error code when the value is something we
don't expect, even if that unexpected value is 0.

* st_rados_list_objects: add option to ignore list errors (for the
deletion test)

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

ReplicatedPG: data_subset may not contain data_needed

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

encoding.h: need to pass by reference

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

ReplicatedPG: fix snaps decode in recover_primary

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

test/osd: TestSnaps now writes large, random objects with holes

In order to test clone recovery pathways, TestSnaps now writes to random
subsets of an object between snapshots.  Objects may now also be very
large.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

mds: fix handle_simple_lock on LOCK replica

We were ignoring caps.  To avoid this kind of mistake, use eval_gather(),
like handle_file_lock() does.

Signed-off-by: Sage Weil <sage@newdream.net>

mds: drop useless from field from MInodeFileCaps

While we're updating the protocol, clean this up too.

Signed-off-by: Sage Weil <sage@newdream.net>

mds: print cur+nested auth_pins on inode

And clean up a bit.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: move to MIX state if writer wanted and no wanted loner

We can just look at the target loner here, which also takes any caps wanted
by other replicas on other MDSs into account.  Otherwise we need to
to duplicate the CInode::calc_ideal_loner() logic.

This assumes the loner field is accurate.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: print mds_caps_wanted on inode

along with the client caps info

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: improve print() for MInodeFileCaps

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: migrate loner_cap state

It is tedious to infer what the old loner_cap was pre-migration. Just send
it over the wire and set it explicitly. Usually when we eval() we would
have come to the same conclusion, but when we didn't, we got into
inconsistent/impossible states where the issue caps don't match the loner
state (Asx issued but no loner_cap set).  That meant the next issue was
a revocation with no lock state change, which led to yet more problems
down the line.

This is an internal MDS protocol change.

Signed-off-by: Sage Weil <sage@newdream.net>

mds: clear [want_]loner_cap on cap export

These fields are meaningless on the replica.

Signed-off-by: Sage Weil <sage@newdream.net>

Move list objects logic to st_rados_list_objects

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rados_list_parallel: fix rados_ioctx_destroy uses

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rados_list_parallel: add test4

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rados_list_parallel: test adding objects + listing

Test one runnable adding objects while another runnable lists them.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Add rerun.sh

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rados_list_parallel: make num_objects selectable

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

add CrossProcessSem::reinit, fix rados_list_parallel

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Now we can put runnable logs into separate files

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Add rados_open_pools_parallel test

Also add rados_open_pools_parallel to .gitignore

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rgw: clean up temp object if failed to PUT

Conflicts:

src/rgw/rgw_op.cc

rgw: clean up temp object if failed to PUT

rgw: remove include of curl/types.h

test/rgw_escape: add JSON + utf-8 test

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

test/rgw_escape.cc: add utf-8 test

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

CodingStyle: Typos, whitespace.

rgw: fix escaping clobbering utf8

crushtool: Add call to global_init() to avoid dout segfault

Signed-off-by: Jim Schutt <jaschut@sandia.gov>
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Revert "mds: handle importer failure on exporter"

This reverts commit bf768cb146b23d345efcc51badfe668bf37d4046.

This won't actually happen with f44cffdb8ccc8cf258b418146dbbfc20a9b685f9

obsync: Depend on python-pyxattr instead of python-xattr

Signed-off-by: Wido den Hollander <wido@widodh.nl>
Signed-off-by: Sage Weil <sage@newdream.net>

mds: verify deferred messages aren't stale

We may defer processing of some messages because we are laggy (in hearing
from the monitor).  When we eventually get to those messages, make sure
they haven't since become stale (i.e., the source mds isn't now down).

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: handle importer failure on exporter

If the export is reversed but we still receive the ack, behave.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rados: fix warning

warning: rados.cc:319: large integer implicitly truncated to unsigned type

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

client: print ino along with &fh for ll_ file ops

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rados: fix warning

warning: rados.cc:319: large integer implicitly truncated to unsigned type

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rgw: bucket deletion updates caches

mime.c: avoid compiler warning

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

test/mime: test invalid decodes

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

mime.h: better comments

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Add mime encoding

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rgw: fix lru