rgw: ONLY improve code formatting in rgw_object_expirer_core.{cc,h}.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: fix skipping some objects to delete by RadosGW's object expirer.

Wei Qiaomiao has found that expired objects, which should be removed by
the object expirer of RadosGW, might be left unprocessed till next
restart of a RadosGW's instance. This happens when process_single_shard
method of RGWObjectExpirer class exhaust a time slot for a single round
and finishes without informing caller about the situation.

Fixes: http://tracker.ceph.com/issues/16705
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: fix trimming object expirer's hints.

Fixes: http://tracker.ceph.com/issues/16684
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #10306 from tchaikov/wip-no-mktemp-p

qa/workunits/cephtool/test.sh: s/TMPDIR/TEMP_DIR/

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #9972 from ceph/objclass-perm

osd: object class loading and execution permissions

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #9980 from gaowanlong/split_out_handle_pg_scrub

osd: small cleanups

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10126 from dx9/wip-fcntl-warns

test/libcephfs: fix gcc sys/fcntl.h warnings

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10166 from wjwithagen/wip-wjw-cmake-test_rados_tool.sh

test_rados_tool.sh: Make script work under ctest

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9782 from Ved-vampir/zlib_cleanup

compressor: zlib compressor plugin  cleanup

Reviewed-by: Kefu Chai <kchai@redhat.com>

test: run_seed_to_range.sh: check existance of a directory using [ -d "$dir" ]

sadly, sh evalutes `[ -d ]` to true. as it takes "-d" as a non-empty
string as true.

this fixes following failure
```
2016-07-12T23:22:02.839 INFO:teuthology.orchestra.run.mira084.stderr:cp:
missing destination file operand after ‘.’
2016-07-12T23:22:02.839 INFO:teuthology.orchestra.run.mira084.stderr:Try
'cp --help' for more information.
```
see
http://pulpito.ceph.com/kchai-2016-07-12_23:09:35-rados-wip-kefu-testing2---basic-mira/311334/

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits/cephtool/test.sh: s/TMPDIR/TEMP_DIR/

this fixes the test failure of
```
2016-07-12T23:29:40.935
INFO:tasks.workunit.client.0.mira101.stderr:/home/ubuntu/cephtest/workunit.client.0/cephtool/test.sh:
line 153: /CEPH_WATCH_9445: Permission denied
```
see
http://pulpito.ceph.com/kchai-2016-07-12_23:09:35-rados-wip-kefu-testing2---basic-mira/311333/

it's a regression introduced by e5c262b

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits/cephtool/test.sh: use mktemp $TEMP_DIR/XXX instead

mktemp -p is not supported on FreeBSD

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10131 from badone/wip-peering-doc-fixes

doc: peering.rst, fix typo

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10292 from badone/wip-perf-counters-doc-fixes

doc: perf_counters.rst fix trivial typo

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9491 from Yan-waller/yj-wip-fixbuildpushop

osd/ReplicatedBackend: add sanity check during build_push_op()

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #9510 from renhwztetecs/renhw-wip-common

common: unknown hash type of judgment modification

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9729 from stiopaa1/common_workqueue_addMove

common/WorkQueue: add std move

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #9770 from tanghaodong25/submit-transaction-perfcounter

kv/RocksDB: add perfcounter for submit_transaction_sync operation

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10274 from yehudasa/wip-16665

rgw: collect skips a specific coroutine stack

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10187 from dreamhost/wip-16618

rgw: fix multi-delete query param parsing.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #10289 from jdurgin/wip-googletest-centos

fix the rpm build for centos

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #10271 from linuxbox2/rgw-rename

rgw_file: fix rename cases and unify unlink
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #8618 from linuxbox2/rgw-file-setattr

rgw: file setattr
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

rgw_file: implement rgw_setattr

Introduce a new RGWSetattrs RGWOp descendant, to create or replace
sets of attrs on buckets or objects.

This version of the change uses the standard RGWRADOS::set_attrs op
(we want attribute changes to (e.g.) sync with other changes).

Previous versions of this changed incorrectly masked the values
of st->st_ino in RGWFileHandle::stat(), now fixed.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #10284 from linuxbox2/rgwfile_write_guard

rgw_file: refuse partial, out-of-order writes
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10258 from ukernel/wip-16655

ceph-fuse: link to libtcmalloc or jemalloc

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10168 from batrick/client-inode-des

client: move Inode specific cleanup to destructor

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10169 from batrick/client-inode-cons

client: note order of member init in cons

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10265 from tchaikov/wip-ceph-fs-warnings

include/ceph_fs.h: guard `#define CEPH_SETATTR_*` with #ifndef

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10142 from batrick/i16042

mds: move Finisher to unlocked shutdown

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10176 from renhwztetecs/renhw-wip-mon-mds

mon/mds: add err info when load_metadata is abnormal

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10192 from batrick/mon-minor

mon: use clearer code structure

Reviewed-by: songbaisen <song.baisen@zte.com.cn>

Merge pull request #10200 from renhwztetecs/renhw-wip-mds-journal

osdc/mds: fixup pos parameter in the journaler

Reviewed-by: John Spray <john.spray@redhat.com>

doc: peering.rst, fix typo

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

doc: perf_counters.rst fix trivial typo

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #9933 from chhabaramesh/master

kv: In memory keyvalue db implementation

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>

install-deps: Get the latest virtualenv in a temporary folder
to work around a bug in old virtualenv

Signed-off-by: Oleh Prypin <oleh@pryp.in>

googletest: avoid installing any pieces of it

This makes rpmbuild happy.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #10286 from ceph/wip_leftover_with_mds

Removes remaining reference to WITH_MDS

Reviewed-by: Ali Maredia <amaredia@redhat.com>

rgw: collect skips a specific coroutine stack

Fixes: http://tracker.ceph.com/issues/16665
Instead of drain_all_but() that specifies number of stacks to leave behind,
added drain_all_but_stack() that has a specific stack specified. This is needed
so that we don't call wakeup() through lease_cr->go_down() on a cr stack that
was already collected.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Not all references to WITH_MDS were removed (see PR #10186). This
removes the final one (dependency of vstart on ceph-mds).

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #10120 from oritwas/bz16412

rgw: remove bucket index objects when deleting the bucket

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #10161 from ceph/wip-dang-cmake

CMake - stop pip checking for updates

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Ali Maredia <amaredia@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10170 from ceph/wip-pg-log-errors-11

include write error codes in the pg log

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge branch 'googletest'

Update Ceph to use Google Test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

objclass: implement 'osd class default list'

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

rgw_file: refuse partial, out-of-order writes

A single file object may be opened only once per gateway
instance, and writes to that object must be complete, and in-order.
Enforce this.

If an invalid write is seen, deletes the current write transaction.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #10282 from ceph/wip-cmake-cxx-flags

cmake: fix the FTBFS introduced by dc8b3ba

Reviewed-by: Casey Bodley <cbodley@redhat.com>

cmake: fix the FTBFS introduced by dc8b3ba

the flags are not list

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10279 from ceph/wip-cmake

cmake changes

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #9784 from ceph/wip-ms-async-fixes

msg/async: connect authorizer fix + recv_buf size

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #10276 from tchaikov/wip-doc-drop-i386

doc: remove i386 from minimal hardware recommendations

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Vikhyat Umrao <vumrao@redhat.com>

Merge pull request #10254 from dillaman/wip-16536

rbd-mirror: include local pool id in resync throttle unique key

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #10245 from jcsp/wip-mds-cache-warn

mds: add health warning for oversized cache

Reviewed-by: Yan, Zheng <zyan@redhat.com>

cmake: use CMAKE_CXX_STANDARD for enabling C++11

hope we can drop the crufts when moving to cmake 3.1+

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10191 from batrick/mds-minor

mds: use reference to avoid copy

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10268 from batrick/i16664

doc: fix standby replay config

Reviewed-by: John Spray <john.spray@redhat.com>

cmake: do not check for availability of static_cast<> anymore

it's implied by C++11 compliance.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: use CHECK_STRUCT_HAS_MEMBER for checking member variables in struct

we were using CHECK_CXX_SOURCE_COMPILES() for this purpose, which was a
little bit clumsy.

Signed-off-by: Kefu Chai <kchai@redhat.com>

kv: In memory keyvalue db implementation

Signed-off-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

ceph-fuse: link to libtcmalloc or jemalloc

Fixes: http://tracker.ceph.com/issues/16655
Signed-off-by: Yan, Zheng <zyan@redhat.com>

doc: remove i386 from minimal hardware recommendations

as we don't build or test on i386 machines now. also, we don't build
i386 releases anymore. and regular users are on amd64 or arm.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10102 from liewegas/wip-mon-nosnap

mon/MonitorDBStore: do not use snapshot iterator; close on close

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10275 from ceph/wip-cmake

cmake: disable -fvar-tracking-assignments for ceph_dencoder.cc

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

cmake: disable -fvar-tracking-assignments for ceph_dencoder.cc

there are too many variables involved in main() of ceph_dencoder.cc, but
-fvar-tracking-assignments is enabled by default, when the compiler
finds that the number of variables exceeds the limit
"max-vartrack-size", it gives up, and tries again disabling this option.

this change avoids the recompilation of ceph_dencoder.cc and also silences
the warning of
```
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/encoding/ceph_dencoder.cc:
In function ‘int main(int, const char**)’:
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/encoding/ceph_dencoder.cc:289:5:
note: variable tracking size limit exceeded with
-fvar-tracking-assignments, retrying without
 int main(int argc, const char **argv)
     ^
```

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10113 from stiopaa1/msg_asyncmess_moveC_handle_reap

msg/AsyncMessenger: move C_handle_reap class to cc

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #10183 from stiopaa1/msg_asyncmess_addConstToSetAddrUnknowns

msg/Messenger: add const and override to function

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10141 from stiopaa1/msg_asyncmess_removeUnneededCast

msg/AsyncMessenger.cc: remove unneeded cast

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Kefu Chai <kefu@redhat.com>

include/ceph_fs.h: guard `#define CEPH_SETATTR_*` with #ifndef

this silences the warnings like:

```
from
/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-11.0.0-513-gfe98b0b/src/java/native/libcephfs_jni.cc:33:
warning:
/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-11.0.0-513-gfe98b0b/src/include/ceph_fs.h:373:0:
"CEPH_SETATTR_MODE" redefined [enabled by default]
^
In file included from
/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-11.0.0-513-gfe98b0b/src/java/native/libcephfs_jni.cc:32:0:
/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-11.0.0-513-gfe98b0b/src/include/cephfs/libcephfs.h:106:0:
note: this is the location of the previous definition
^
```

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10266 from tchaikov/wip-ceph-detect-init-py3

ceph-detect-init: fix the py3 test

Reviewed-by: Loic Dachary <ldachary@redhat.com>

googletest: use ceph branch

Added the same functionality as the previous gtest/gmock branches -
.gitignore and not installing shared libs that we only want to
statically link to test programs.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Replace gmock/gtest with googletest

Google Mock has been absorbed into the Google Test project, and
is now available under the googletest repository.

Signed-off-by: Anirudha Bose <ani07nov@gmail.com>

rgw_file: fix rename cases and unify unlink

Rather ordinary rename cases failed in the unlink step.  Fix this,
unifying the unlink path while at it.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

doc: fix standby replay config

I tried using these settings in tests without success. The correct config names
are prefixed with "mds".

Fixes: http://tracker.ceph.com/issues/16664
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-detect-init: fix the py3 test

the mock_open() does not support "for i in open()", and readline() and
readlines() support was added in py3.3 and py3.4 so for better backwards
compatibility, we should change the code being tested to use the plain
read() call.
and also use open(path) instead of open(path, 'rb') for simplicity,
otherwise we need to use bytestring for comparison and pass the same
parameters to mock.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test: ceph-detect-init: fix the indent

this fixes the pep8 failure:
```
pep8 runtests: commands[0] | flake8 ceph_detect_init tests
tests/test_all.py:121:17: E128 continuation line under-indented for
visual indent
tests/test_all.py:125:17: E128 continuation line under-indented for
visual indent
tests/test_all.py:129:17: E128 continuation line under-indented for
visual indent
tests/test_all.py:133:17: E128 continuation line under-indented for
visual indent
```

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10090 from yuyuyu101/wip-remove-async-lock

msg/async: remove file event lock

Reviewed-by: Kefu Chai <kefu@redhat.com>

ceph_test_msgr: clean previous sent queue infos

1. A -> B
2. goto standby
3. B mark down
4. A reconnect to B
5. got reset session and dispatch remote reset
6. because remote reset is executed in DispatchQueue, it will be delayed
7. A -> B successfully and begin to send message
8. assert because we found the first message is missing but it's reasonble

if policy.resetcheck is true

Signed-off-by: Haomai Wang <haomai@xsky.com>

ceph_test_msgr: use ceph log infrastructure to output

because we want to get the right log sequence which mixes ceph logginer and
cerr. Otherwise, cerr output make the logs a little disordered.

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async/AsyncConnection: make sure no STATE_WAIT -> STATE_STANDBY

Otherwise if message in queue, we will continue to reconnect right now,
it won't meet our expectation that we want our connect request delay

Signed-off-by: Haomai Wang <haomai@xsky.com>

test_msgr: add server_msgr cleanup

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async/AsyncConnection: add REPLACING write status to indicate replacing

When replacing, we don't expect any AsyncConnection dispatch new event which
will cause thing chaos

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async: hold lock avoid state changing

Fixes: http://tracker.ceph.com/issues/16554
Signed-off-by: Haomai Wang <haomai@xsky.com>

test_async_driver: accept new api changes

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async: dont delete time event when exchange eventcenter

When replacing and someone called mark_down, it will delete_time_event which
isn't allowed. Because we're exchaning EventCenter now!

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async/AsyncConnection: simplify shutdown socket flow

Let cleanup resources things all in shutdown_socket

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async: add assert to ensure create/delete file event within thread

Signed-off-by: Haomai Wang <haomai@xsky.com>

Event: remove file_lock

Now all EventCenter will exists within one thread, it will let all file events
api changes without locks.

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async/AsyncMessenger: make sure all connection cleanup handler finished

Fixes: http://tracker.ceph.com/issues/16552
Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async: remove close socket line

because if we are in STATE_CLOSED, fd must be -1

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async: make sure worker started before let msgr ready

When we create event thread, it need a little time to enter event loop(like
calling set_owner), if caller is going to call create_file_event before event
thread enter event loop, it will trigger assert.

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async: make EventCenter notify file event creating when set_owner

EventCenter::init is called by other thread instead of event thread, so we
need to move create_file_event to set_owner which is called by event thread.

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async/AsyncConnection: swap eventcenter when replacing

Previously we only exchange fd when replacing, now we will introduce dpdk
plugin in the near future. It needs all fd used locally which not like
kernel socket shared by all cores.

So we need to add EventCenter swapping to let each socket is associated to
EventCenter.

Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge pull request #10153 from tchaikov/wip-no-copy-plugins

test: test_{compression_plugin,async_compressor}: do not copy plugins

Reviewed-by: Haomai Wang <haomai@xsky.com>

msg/async/Event: fix delete_time_event leak

Signed-off-by: Haomai Wang <haomai@xsky.com>

AsyncMessenger: make create/delete_file_event within event thread

We are make each AsyncConnection/AsyncMessenger only modify its file event
in event thread. So make sure create/delete_file_event aren't directly called.

Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge pull request #10174 from theanalyst/test/rgw/multisite-cleanup

rgw: test_multi.py avoid creating mds

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10211 from oritwas/wip-rgw-fix-zone-modify

rgw: remove unused realm from radosgw-admin zone modify

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10256 from ceph/wip-kill-ceph_test_rados_api_tmap_migrate

test: remove ceph_test_rados_api_tmap_migrate

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10229 from badone/wip-libgmock_main.la-build-failures

makefile: change librgw_file_* as check_PROGRAMS

Reviewed-by: Kefu Chai <kchai@redhat.com>