client: update the 'approaching max_size' code

The old 'approaching max_size' code expects MDS set max_size to
'2 x reported_size'. This is no longer true. The new code reports
file size when half of previous max_size increment has been used.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: limit client writable range increment

For very large file, setting the writable range to '2 * file_size'
causes file recovery to run a long time. To recover a 1T file, Filer
needs to probe 2T~1T range.

Fixes: http://tracker.ceph.com/issues/19955
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #15087 from liewegas/wip-thrash-ec

qa/suites/rados: fix ec thrashing

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15072 from liewegas/wip-crush-swap-bucket

mon,crush: add 'osd crush swap-bucket' command

Reviewed-by: Loic Dachary <loic@dachary.org>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15069 from tchaikov/wip-inttypes-cleanup

int_types.h: remove hacks to workaround old systems

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #15057 from liewegas/wip-osd-choose-cleanup

osd/PG: drop pre-firefly compat_mode for choose_*_acting

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #15042 from badone/wip-redundant-headers-3

common: Remove redundant includes

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14989 from liewegas/wip-19877

osd: fix past_intervals base case by adding epoch_pool_created to pg_history_t

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15113 from trociny/wip-unittest_librbd-cleanup

test/librbd: unit tests cleanup

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #15093 from dillaman/wip-19929

librbd: reacquire lock should update lock owner client id

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #10690 from ceph/wip-rgw-polymorphic-errors

rgw: polymorphic error codes

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15096 from jcsp/wip-journalrepair-test

qa: simplify TestJournalRepair

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #15110 from tchaikov/wip-build-using-cmake

doc: use do_cmake.sh instead of `cmake ..`

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15023 from tchaikov/wip-cmdparse-constness

cmdparse: more constness

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15112 from jcsp/wip-19940

ceph-create-keys: update client.admin if it already exists

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15097 from linuxbox2/wip-rgwfile-write3

rgw_file: v3: fix write-timer action

test/librbd: make unit tests pass when cache is disabled

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

test/librbd: TestMockImageRemoveRequest gmock warnings

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

ceph-create-keys: update client.admin if it already exists

This means that anyone who de-privileges their client.admin
user will find that it gets resets with full privileges again.
This is necessary because we cannot distinguish between an old
key and a deliberately de-privileged key, and it's okay because
this is the admin key, it's meant to have access to everything.

Fixes: http://tracker.ceph.com/issues/19940
Signed-off-by: John Spray <john.spray@redhat.com>

doc: use do_cmake.sh instead of `cmake ..`

user could be confused at seeing errors like:

src/CMakeLists.txt:644 (add_subdirectory): The source directory /src/lua
does not contain a CMakeLists.txt file.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15094 from fullerdj/wip-djf-19893

qa/cephfs: Fix for test_data_scan

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #14938 from michaelsevilla/f19362

mds: add perf counters for file system operations

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15018 from ukernel/wip-19890

pybind: fix open flags calculation

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #14769 from ukernel/wip-19755

osdc/Filer: truncate large file party by party

Reviewed-by: John Spray <john.spray@redhat.com>

rgw: Searching for error codes and dumping output based on protocol flags.

The previous commit uses dynamic allocation to search for s3/ swift specific
error codes and dump the results. In order to avoid dynamic allocation,
we are making use of protocol flags to achieve the above.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

polymorphic error codes.

make set_req_state_err a method of req_state

get rid of params.perr; it's never used

rgw - polymorphic error object.

polymorphic error object: this way sts, which has additional error fields,
and its own xml format for how errors should be produced, can overload
the base rgw_err class.

STL fixup: use real mappings for error mappings, not a linear search.
insubstantiate error mappings exactly once in src/rgw/rgw_common.cc, instead
of multiple copies from every source file that includes src/rgw/rgw_http_errors.h.

remove dump_errno(s, err) - it was never used, and the code looked bogus.

move rgw_http_error_to_errno from header to rgw_rest_client.cc,
the only place it's used.
move rgw_http_status_code and http_codes to rgw_rest.cc,
the only place it's used.

base class (rgw_err) implements s3 errors; swift & sts errors use
the base class and overlay their own additional error codes to
the ones recognized by the base class.

For a while, I was trying to just have a "generic" abort_early() that
used dump_more.  Now, I have an op aware form that works the old way.
I would like the http routines to known less about "op" structures,
which is why I'm making this distinction.

Splitting up generic and s3 flavors of some things:
rgw_rest_init() - split into s3 and generic forms.
end_header: s3 vs generic forms: generic form uses dump_more.
abort_early: s3 vs generic forms: generic form uses dump_more, error_content.

And fixing some mistakes from the merge:
Fixup: should not have deleted dump_errno() local storage forms.
Fixup:RGWRESTFlusher  - take a dump_more to make it generic.
Fixup: need RGWRESTOp::send_response()

Move swift_err up; use it in bulkdelete_respond
I Think I don't need s->prot_flags when using swift_err.

Signed-off-by: Marcus Watts <mwatts@redhat.com>

rgw_file: v3: fix write-timer action

For now, unify with v4 write-on-close path, by calling
RGWFileHandle::close() on write-timer expire, since it will
call write_finish() as a side-effect.

Fixes: http://tracker.ceph.com/issues/19932
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #15052 from batrick/i19734

mds: use debug_mds for most subsys

Reviewed-by: John Spray <john.spray@redhat.com>

qa: simplify TestJournalRepair

This was sending lots of metadata ops to MDSs to persuade
them to migrate some subtrees, but that was flaky.  Use
the shiny new rank pinning functionality instead.

Signed-off-by: John Spray <john.spray@redhat.com>

qa/cephfs: Fix for test_data_scan

Don't assume that test_data_scan will be run on exactly 2 MDS nodes.

Fixes: http://tracker.ceph.com/issues/19893
Signed-off-by: Douglas Fuller <dfuller@redhat.com>

Merge pull request #15091 from ceph/luminous

merge v12.0.2 release tag

mds: add perf counters for file system operations

Shows the types of operations that clients are doing by adding counters to the
spot where the MDS responds to clients. This avoids counting operations that
were restarted while waiting for locks or for metadata to load.  Does not
support per-client activity. This is the first step towards comprehensive
performance tracking in the file system.

Signed-off-by: Michael Sevilla <mikesevilla3@gmail.com>

Merge pull request #15079 from ztczll/master

mailmap: ztczll affiliation

Reviewed-by: Kefu Chai <kchai@redhat.com>

librbd: reacquire lock should update lock owner client id

Fixes: http://tracker.ceph.com/issues/19929
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/suites/rados/threas-erasure-code-*: fix luminous facet

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/thrash-erasure-code-overwrites: fix

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15081 from tchaikov/wip-fio-readme

fio: update README.md so only the fio ceph engine is built

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mon/OSDMonitor: require force flag is swap-bucket alg types differ

Signed-off-by: Sage Weil <sage@redhat.com>

crush: add crush_alg_name

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: add 'osd crush swap-bucket' command

Swap contents of two crush buckets.  Do it safely by default so that
the origin bucket has to be an orphan (unless forced).

Signed-off-by: Sage Weil <sage@redhat.com>

crush: implement swap_bucket

Swap contents between buckets.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15048 from trociny/wip-19907

rbd-mirror: resolve admin socket path names collision

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #15026 from ukernel/wip-19891

qa/suites/fs: reserve more space for mds in full tests

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15039 from ukernel/wip-19903

client: fix UserPerm::gid_in_group()

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15035 from batrick/quiet-mds-grow-shrink

qa: silence spurious insufficient standby health warnings

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #14991 from ukernel/wip-19828

mds: fix use-after-free in Locker::file_update_finish()

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #14978 from cernceph/dvanders_quota

client: client_quota no longer optional

Reviewed-by: John Spray <john.spray@redhat.com>

fio: update README.md so only the fio ceph engine is built

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15044 from lixiaoy1/master

test/fio_ceph_objectstore: fix fio plugin build failure by engine_data

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mailmap:ztczll affiliation

Signed-off-by: zhanglei <243290414@qq.com>

Merge pull request #15076 from atheism/atheism-mailmap

mailmap: Weibing Zhang mailmap affiliation

Merge pull request #15007 from ztczll/master

test/objectstore/: Check put_ref return value

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #15021 from runsisi/wip-fix-writefull

osd/PrimayLogPG: update modified range to include the whole object size for write_full op

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15046 from ukernel/wip-osdmap-encode

osd/OSDMap: bump encoding version for require_min_compat_client

Reviewed-by: Sage Weil <sage@redhat.com>

mailmap: Weibing Zhang mailmap affiliation

Signed-off-by: Weibing Zhang <atheism.zhang@gmail.com>

int_types.h: remove hacks to workaround old systems

Signed-off-by: Kefu Chai <kchai@redhat.com>

common: Remove redundant includes

Fixes: http://tracker.ceph.com/issues/19883 (Partially)
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

rbd-mirror: resolve admin socket path names collision

If global admin_socket parameter is set, for PoolReplayer admin
sockets generate and use unique paths in $run_dir. Provide the socket
paths in output of PoolReplayer's "rbd mirror status" admin socket
command.

A possible use case:

1) run "rbd mirror status $pool_replayer" command using global admin
   socket (known from configuration) and obtain socket paths for the
   PoolReplayer local and remote context sockets.
2) Use these paths to run commands provides by the contexts.

Fixes: http://tracker.ceph.com/issues/19907
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #13111 from cbodley/wip-rgw-mdlog-trim

rgw multisite: automated mdlog trimming

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #15004 from liewegas/wip-19880

os/bluestore: fix deferred write race

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #15029 from xiexingguo/wip-bluestore-20170510

os/bluestore: pre-calculate number of ghost buffers to evict

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15012 from tchaikov/wip-byteswap

byteorder: use gcc intrinsics for byteswap

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15019 from badone/wip-redundant-headers-2

common: Remove redundant includes

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15040 from tchaikov/wip-19902

osd/PrimaryLogPG: do not call on_shutdown() if (pg.deleting)

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15009 from tchaikov/wip-kill-warnings

libradosstriper: do not assign garbage to returned value

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #14940 from ceph/wip-radosgw-admin-package-move

debian/rpm: move radosgw-admin to ceph-common

Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

test/common: Check block_device_model return value

Signed-off-by: zhanglei <243290414@qq.com>

pybind: fix cephfs.OSError initialization

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "cephfs.pyx", line 672, in cephfs.LibCephFS.open (/home/zhyan/Ceph/ceph-2/build/src/pybind/cephfs/pyrex/cephfs.c:10160)
  File "cephfs.pyx", line 155, in cephfs.OSError.__init__ (/home/zhyan/Ceph/ceph-2/build/src/pybind/cephfs/pyrex/cephfs.c:1889)
TypeError: __init__() takes exactly 3 positional arguments (2 given)

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #13610 from liewegas/wip-snapset

osd: eliminate snapdir objects and move clone snaps vector into SnapSet

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

osd/PG: drop pre-firefly compat_mode for choose_*_acting

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15056 from liewegas/wip-fix-osdmap-comment

osd/OSDMap: fix feature commit comment

osd/OSDMap: fix feature commit comment

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #10260 from weiqiaomiao/wqm-wip-clean-opstate

rgw: clean unuse code in cls_statelog_check_state

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14411 from mikulely/rgw-cleanup-rados

rgw: rgw_rados drop deprecated global var

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14906 from mikulely/lc-cleanup-2

rgw: cleanup lc continuation

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #15010 from shashalu/fix-bucket-create

rgw: when create_bucket use the same num_shards with info.num_shards

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15017 from dillaman/wip-19863

common/bit_vector: utilize deep-copy during data decode

Reviewed-by: Sage Weil <sage@redhat.com>

mds: use debug_mds for most subsys

DOUT_COND was obsoleted by f41887e38d70970f4d0181fb73e835ece68cee6c.

Balancer and Mantle use a dout hack to continue working since we will probably
specifically want very verbose balancer output in the future.

Fixes: http://tracker.ceph.com/issues/19734
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

test/fio_ceph_objectstore: fix fio plugin build failure by engine_data

The commit in fio https://github.com/axboe/fio/commit/710bf9c592b849935e1bcc56ffc805fa1d5f9365
changed engine_data to a union which leads build of fio_ceph_objectstore failed.

Signed-off-by: Lisa Li xiaoyan.li@intel.com

osd/PrimayLogPG: update modified range to include the whole object size for write_full op

Signed-off-by: runsisi <runsisi@zte.com.cn>

osd/PrimaryLogPG: drop parameter count_bytes for write_update_size_and_usage

Signed-off-by: runsisi <runsisi@zte.com.cn>

osd/OSDMap: bump encoding version for require_min_compat_client

Otherwise, OSDMap::decode() throws expcetion when decoding old
version osdmap

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #14743 from ukernel/wip-19635

mds: issue new caps when sending reply to client

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #14704 from batrick/i19589

mds: fix mantle script to not fail for last rank

Reviewed-by: Michael Sevilla <mikesevilla3@gmail.com>

mds: fix use-after-free in Locker::file_update_finish()

The capability may have already been freed when executing
Locker::file_update_finish()

Fixes: http://tracker.ceph.com/issues/19828
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #14921 from batrick/ptrwaiter-balance

mds: relocate PTRWAITER put near get

osd/PrimaryLogPG: do not call on_shutdown() if (pg.deleting)

when a callback is called, it could be facing a PG already shut down by
OSD. but if that callback wants to shut that PG down. it should check
the PG's status first.

Fixes: http://tracker.ceph.com/issues/19902
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14598 from batrick/mds-balancer-pin

mds: support export pinning on directories

client: fix UserPerm::gid_in_group()

Fixes: http://tracker.ceph.com/issues/19903
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

qa/suites/fs: fix write size calculation in full tests

'max_avail' has already taken full_ratio into account

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

common: Remove redundant includes

Fixes: http://tracker.ceph.com/issues/19883 (Partially)
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

mds: delay export until dir is stable

This catches an assertion failure due to exports on unjournaled directories:

    2017-05-10 12:45:18.193640 7eff65a04700  7 mds.0.migrator export_dir [dir 10000000003 /a/ [2,head] auth pv=3 v=1 cv=0/0 ap=1+2+2 state=1073742082|complete f()->f(v0 m2017-05-10 12:45:14.977151 1=0+1) n()->n(v0 rc2017-05-10 12:45:14.977151 1=0+1) hs=0+1,ss=0+0 | child=1 authpin=1 0x55ef655de000] to 1
    2017-05-10 12:45:18.193654 7eff65a04700 15 mds.0.18 hit export target 10 @ 2017-05-10 12:45:18.193653
    2017-05-10 12:45:18.193661 7eff65a04700 10 mds.0.cache.dir(10000000003) auth_pin by 0x55ef650cc130 on [dir 10000000003 /a/ [2,head] auth pv=3 v=1 cv=0/0 ap=2+2+2 state=1073742082|complete f()->f(v0 m2017-05-10 12:45:14.977151 1=0+1) n()->n(v0 rc2017-05-10 12:45:14.977151 1=0+1) hs=0+1,ss=0+0 | child=1 authpin=1 0x55ef655de000] count now 2 + 2
    2017-05-10 12:45:18.193701 7eff65a04700  7 mds.0.cache request_start_internal request(mds.0:4) op 5377
    2017-05-10 12:45:18.193708 7eff65a04700  7 mds.0.migrator dispatch_export_dir request(mds.0:4)
    2017-05-10 12:45:18.198281 7eff66205700 20 mgrc operator() 234 counters, of which 0 new
    2017-05-10 12:45:18.198294 7eff66205700 20 mgrc send_report encoded 2006 bytes
    2017-05-10 12:45:18.198298 7eff66205700  1 -- 127.0.0.1:6825/693985647 --> 127.0.0.1:6826/1025 -- mgrreport(+0-0 packed 2006) v2 -- 0x55ef655abe00 con 0
    2017-05-10 12:45:18.198364 7eff6a9ab700 10 _calc_signature seq 14 front_crc_ = 2388339344 middle_crc = 0 data_crc = 0 sig = 5026349537430007662
    2017-05-10 12:45:18.198376 7eff6a9ab700 20 Putting signature in client message(seq # 14): sig = 5026349537430007662
    2017-05-10 12:45:18.198411 7eff65a04700 -1 /home/pdonnell/ceph/src/mds/MDCache.cc: In function 'void MDCache::make_trace(std::vector<CDentry*>&, CInode*)' thread 7eff65a04700 time 2017-05-10 12:45:18.194136
    /home/pdonnell/ceph/src/mds/MDCache.cc: 8225: FAILED assert(parent)

     ceph version 12.0.1-2198-ge757c02 (e757c025fa3270b12fb2fca17cf159fa1bd72747)
     1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x55ef5b780f02]
     2: (MDCache::make_trace(std::vector<CDentry*, std::allocator<CDentry*> >&, CInode*)+0x1cb) [0x55ef5b4fa12b]
     3: (Migrator::get_export_lock_set(CDir*, std::set<SimpleLock*, std::less<SimpleLock*>, std::allocator<SimpleLock*> >&)+0x55) [0x55ef5b5e8215]
     4: (Migrator::dispatch_export_dir(boost::intrusive_ptr<MDRequestImpl>&, int)+0xa74) [0x55ef5b5f72d4]
     5: (Migrator::export_dir(CDir*, int)+0x9ca) [0x55ef5b5ea75a]
     6: (MDBalancer::handle_export_pins()+0x7b4) [0x55ef5b61ab24]
     7: (MDBalancer::tick()+0x1e8) [0x55ef5b61b748]
     8: (MDSRankDispatcher::tick()+0x5f1) [0x55ef5b44bdb1]
     9: (Context::complete(int)+0x9) [0x55ef5b43bcc9]
     10: (SafeTimer::timer_thread()+0x452) [0x55ef5b77dd52]
     11: (SafeTimerThread::entry()+0xd) [0x55ef5b77f15d]
     12: (()+0x76ba) [0x7eff6d4bb6ba]
     13: (clone()+0x6d) [0x7eff6c52782d]

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

byteorder: refactor ceph_le{16,32,64} using template

to improve the readablity and easier for debugging.

Signed-off-by: Kefu Chai <kchai@redhat.com>

byteorder: use gcc intrinsics for byteswap

* use gcc intrinsics for byteswap
* use template to wrap them.
* add the modeline for emacs/vim
* update the caller of the mswab/swab accordingly

Signed-off-by: Kefu Chai <kchai@redhat.com>

radosstriper: do not use GCC extension to initialize struct members

it's obsolete since GCC 2.5 and is not portable. see
https://gcc.gnu.org/onlinedocs/gcc/Designated-Inits.html

Signed-off-by: Kefu Chai <kchai@redhat.com>

byteorder: remove the cruft to detect endianness

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/sctp_crc32: include acconfig.h for the detecting endianness

so byteorder.h can be a pure c++ header.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: use cmake module to detect endianness

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmdparse: more constness

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa: add health warning test for insufficient standbys

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>