Merge pull request #3527 from wonzhq/op-stat

osd: don't log op stats for proxy read in the cache tier

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3503 from ceph/wip-pg-reqids

osd: index PGLog with optional extra_reqids in each log entry

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Wang, Zhiqiang <zhiqiang.wang@intel.com>

ceph_test_rados: fix typo

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3452 from majianpeng/iohint

Iohint in librbd cache.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librbd: Don't do readahead for random read.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3516 from ceph/wip-mdsmon-args

mon/MDSMonitor: fix gid/rank/state parsing

Reviewed-by: Huang Jun <hjwsm1989@gmail.com>
Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #3551 from ceph/wip-10698

rgw: fail s3 POST auth if keystone not configured

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #3476 from ktdreyer/wip-rpm-radosgw-logrotate

ceph.spec.in: move rgw logrotate to rgw subpackage

Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3437 from diurchenko/patch-2

rgw: fix doc, wrong HTTP header name

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #3546 from zhouyuan/204_post_container

rgw: Response 204 when post on containers

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fail s3 POST auth if keystone not configured

Fixes: #10698
This fixes issue introduced in 8b3dfc9472022ea45ad24e02e0aa21dfdad798f8,
where if user does not exist, we try keystone authentication. However,
if keystone is not configured we justt fall through without failing.
This would have failed later on due to bucket permissions, unless bucket
had a public write permissions.

Reported-by: Valery Tschopp <valery.tschopp@switch.ch>
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

doc: Fixed hyperlink.

Signed-off-by: John Wilkins <jowilkin@redhat.com>

rgw: Response 204 when post on containers

Currently POST request on container returns 202 if success. But in Swift
the code is 204. Let's use 204 to keep align with the Swift API.

Fixes #10667
Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

Merge pull request #3526 from ceph/wip-rgw-versioning-4

Wip rgw versioning 4

Reviewed-by: Sage Weil <sage@redhat.com>

rgw: more merge related fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix merge artifact

s/XMLArgs/RGWHTTPArgs

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge remote-tracking branch 'origin/master' into wip-rgw-versioning-4

Conflicts:
src/rgw/rgw_admin.cc

Merge pull request #3125 from ceph/wip-10194-master

Wip 10194 master

Reviewed-by: Sage Weil <sage@redhat.com>

Merge branch 'prio_hb_pkts' of git://github.com/wenjianhn/ceph

Conflicts:
src/msg/Messenger.h

Merge pull request #3543 from ceph/wip-hints

librados: rename NOREUSE to NOCACHE

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3295 from majianpeng/fix2

osd: add per-pool fadvise-dontneed flag

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3392 from ceph/wip-crush-straw2

crush: new straw2 bucket

Reviewed-by: Joao Eduardo Luis <joao@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3495 from XinzeChi/wip-size-empty

cleanup: replace some size() or length() with !empty()

Reviewed-by: Loic Dachary <ldachary@redhat.com>

librados: rename NOREUSE to NOCACHE

As far as I can tell, the posix_fadvise() distinction between WONTNEED and
NOREUSE is subtle: one says I won't access the data, and the other says
I will access it one more time and then not access it.  That is, the
distinction is about time.  This thread seems to confirm this
interpretation:

 https://lkml.org/lkml/2011/6/27/44

Since we are attaching hints to the IO operations themselves, this
distinction doesn't make much sense for us.  (Backends should be careful
about which hint they use; or rather, they should use WONTNEED *after*
doing the IO since NOREUSE is presenting a no-op in Linux.)

However, we want to make a totally different distinction:

 WONTNEED - nobody will access this -> drop it from the cache
 NOCACHE  - *i* won't access this again -> don't let me affect your caching
            decisions or the working set you're maintaining for other
            clients.

The NOCACHE name is made-up and distinct from NOREUSE only so that it is
different from POSIX and doesn't introduce confusion for people familiar
with the POSIX meaning.  Perhaps a more accurate name would be IWONTNEED
but that is only one character apart and too error-prone IMO.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PGLog: only do slow extra_reqid search of object appears at least once

Signed-off-by: Sage Weil <sage@redhat.com>

osd: preserved extra_reqids on promote, flush

This fixes idempotency for operations across caching changes (promote,
flush and cache mode changes).

Fixes: #8935
Signed-off-by: Sage Weil <sage@redhat.com>

osd: preserve extra_reqids in PGLog

The log scraping is unfortunately O(n) in the size of the log.  We can
also follow objects and then the prior_version pointers, but we don't have
a version->entry index at the moment.

Only index by reqid if reqid_is_index.  Always index extra_reqids.
Notably, the PROMOTE op does not return true for reqid_is_indexed, but this
is where the promotion extra_reqids will appear.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: allow extra reqids to be stashed in each pg_log_entry_t

Signed-off-by: Sage Weil <sage@redhat.com>

cls_rgw: fix rgw_bucket_olh_log_entry::epoch initialization

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #3540 from athanatos/wip-sam-cache-object-context-across-op

Wip sam cache object context across op

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3536 from athanatos/wip-10616

Wip 10616

Reviewed-by: Sage Weil <sage@redhat.com>

rgw: reuse fcgx connection structure

Fixes: #10194
Reuse the fcgx connection structure. It forces FCGX_Accept_r() freeing
up or reusing the connection if possible. This is needed for certain
fastcgi module (mod-proxy-fcgi) that sets the FCGI_KEEP_CONN, which
prevents the connection from being closed when calling FCGX_Finish_r().

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

QueueRing: a reduced contention queue

A queue that provides multi-readers, multi-writers concurrency with a
reduced lock contention.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

test/encoding/types.h: add new structs for testing

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls_rgw: add missing struct test instance generator

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls_rgw: fix encoding for cls_rgw_obj

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: don't call send_response() explicitly

This will make us send the response twice and clobber the http
state.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: set explicit "null" instance when needed

When clearing a bucket, if instance is not found, set it to "null",
otherwise we'll just create a delete marker.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix a crash when getting ECANCELED in delete_obj()

bufferlist::c_str() returns NULL if bufferlist is empty. We were
clearing the state, later on assigning s->obj_tag to a string.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

workunits/rgw: adjust radosgw-admin output parsing

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

workunits/rgw: set radosgw-admin location through env variable

add new RGW_ADMIN to be able to set call.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: don't clear olh log if we lost in race

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: guard olh modification

Add a check that validates that object still exists. This shouldn't really be
an issue, as we later compare the tag.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: version id should not contain underscore

The problem is that we use underscore for the raw object name encoding.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #3524 from ceph/wip-8251-2

Wip 8251 2

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3535 from ceph/wip-librbd-tests

tests: ensure RBD integration tests exercise all features

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3530 from xinxinsh/wip-cleanup

cleanup : remove unused perf counter

Merge pull request #3528 from RobertJansen1/patch-1

corrected some confusing numbers

Reviewed-by: Loic Dachary <ldachary@redhat.com>

edited failure conclusion

Signed-off-by: Robert Jansen <r.jansen@fairbanks.nl>

corrected some confusing numbers

Signed-off-by: Robert Jansen <r.jansen@fairbanks.nl>

osd: don't log op stats for proxy read in the cache tier

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

cleanup : remove unused perf counter

Signed-off-by: xinxin shu <xinxin.shu@intel.com>

rgw: also convert sharded replicalog entries

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: get rid of replicalog index_by_instance param

We convert old entries anyway, just complicates everything. Only use
that was kept is the one needed for the conversion function.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fixing rebase casualties

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: convert old replicalog entries if needed

If reading a bucket replicalog entry and one doesn't exist, fall back to
the old key, and convert it to the new one. When updating entries, if
entry does not exist do the same.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: add replicalog update command

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #3514 from rzarzynski/wip-10550

mds: Wip 10550

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #3499 from ceph/wip-perf-filter

common: filtering in `perf dump`

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3508 from zhouyuan/submodule_https_git

Update git submodule to use the same https protocol

Reviewed-by: Loic Dachary <ldachary@redhat.com>

tests: ensure RBD integration tests exercise all features

The RBD_FEATURES environment variables was not being exported to
the Python and C++ integration tests.  This resulted in the same
test cases being run multiple times instead of testing different
RBD features.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mon/MDSMonitor: fix gid/rank/state parsing

Signed-off-by: John Spray <john.spray@redhat.com>

msg/Pipe: set dscp as CS6 for heartbeat socket

To prioritize the heartbeat packets in the the diffserv-aware device.
Thanks dtaht.

mds: don't join on thread which has not been runned.

Fixes: #10550
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Update git submodule to use the same https protocol

Make more convenient for hosts with git protocol blocked

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

Merge pull request #3505 from ceph/wip-fuse-regress-note

Update PendingReleaseNotes for ceph-fuse change in Giant

librbd: make librbd cache send read op with fadvise_flags.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

librbd: Don't do readahead for random read.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

rgw: extend replica log api (purge-all)

Adding a new purge-all option for the replica log api, which completely
removes a replica log object.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

utime: extend utime parsing

Since we sometimes encode it as <sec>.<usec>, also try to parse it using
this format.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: fix replica log indexing

Fixes: #8251
Previously we were indexing the replica log only by bucket name, even
though we were provided with the bucket instance id. Fix that, and also
add the option to be able to revert to the old behavior. For
radosgw-admin we can do it if --replicalog-index-by-instance=false is
provided. For the replica log REST api we now have the index-by-instance
http param (defaults to true).

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Merge pull request #3480 from liewegas/wip-10617

osd: continue interrupted pg deletion on pool removal (bug 10617)

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3454 from hjwsm1989/optracker-op-wq

  use shardid as a key of the shardinfo when "dump_op_pq_state".

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3430 from majianpeng/misc

Misc

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3311 from xinxinsh/fix1

disable tcmalloc by default when enable jemalloc

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3394 from ceph/wip-10555

mon: Do not allow empty pool names when creating

Reviewed-by: Sage Weil <sage@redhat.com>

Update PendingReleaseNotes for ceph-fuse change in Giant

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #3444 from ceph/wip-mon-pgtemp

osd: OSDMap: remove pg temps for inexistent pools

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3465 from liewegas/wip-mon-features

mon/OSDMonitor: validate crush features for osd (not just mon)

Reviewed-by: Joao Eduardo Luis <joao@redhat.com>

Merge pull request #3475 from tchaikov/osdmap-remove-unused-variables

osd/OSDMap: remove unused variables

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3477 from liewegas/wip-noreuse

librados: add NOREUSE fadvise hint

Reviewed-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #3471 from yuandong1222/rgw-format-mtime

rgw: format mtime of radosgw-admin bucket stats

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #3497 from trociny/test-10439

tests: bring back useful test 'ceph tell osd.foo'

Reviewed-by: Loic Dachary <ldachary@redhat.com>

common: filtering in `perf dump`

So that we can get out a particular subsystem
or particular counter without dumping
everything.  Should make tests that watch perf
counters much less spammy!

`logger` and `counter` params are used with
an exact comparison here but the interface
should be amenable to extending to e.g. globbing
if we wanted to in the future.

Also tidy up some iterator while()s into for()s.

Signed-off-by: John Spray <john.spray@redhat.com>

tests: bring back useful test 'ceph tell osd.foo'

The test was removed in 1189138 (mon: make ceph tell mon.* version
work) as it began to fail due to #10439. After it fixed in c4548f6
(pybind: ceph_argparse: validate incorrectly formed targets), the test
can be restored.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

cleanup: replace some length() with !empty()

Signed-off-by: Xinze Chi <xmdxcxz@gmail.com>

cleanup: replace some size() with !empty()

Signed-off-by: Xinze Chi <xmdxcxz@gmail.com>

Merge pull request #2563 from ceph/wip-7467

rgw: support multiple host names

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3492 from ceph/wip-gmock

tests: add Google C++ Mocking Framework

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3493 from ceph/wip-10637

librbd: trim header update not using AIO

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3388 from dachary/wip-make-check

tests: centos7 needs nc to run make check

Merge pull request #3345 from dachary/wip-install-epel

tests: better EPEL installation method

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

tests: better EPEL installation method

It does not depend on a specific version of the EPEL release package.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3455 from bveuille/wip-documentation

doc: Change Availability text in all of the man pages

Reviewed-by: Loic Dachary <ldachary@redhat.com>

tests: replace existing gtest 1.5.0 with gmock/gtest 1.7.0

Google Testing Framework is included by default within the Google
C++ Mocking Framework.  Update makefiles to use new gmock/gtest
libraries and remove old gtest source code.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: trim header update not using AIO

The original trim header update code was using blocking IO to
update the header.  After migrating to an asynchronous trim
which performs all work in librados callbacks, it exposed a
potential deadlock in the librados_test_stub when attempting
to do blocking IO within a librados callback.  This commit
changes the header update to use AIO.

Fixes: #10637
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

gmock: use Google C++ Mocking Framework for unit tests

The 'src/gmock' directory is extracted from the 1.7.0
release located at https://code.google.com/p/googlemock/

gmock is licensed under the New BSD license, see
src/gmock/LICENSE

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #3486 from ceph/wip-librbd-trim-error

librbd: trim would not complete if exclusive lock is lost

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3487 from ceph/wip-install-deps

install-deps: fix LC_ALL setting

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

install-deps: fix LC_ALL setting

On my box LC_ALL=C# includes the '#' in the value without
a space between C and '#' and things go completely bonkers.

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>