Merge pull request #5466 from ceph/wip-12578

mds: open base inode's snaprealm after decoding snapblob

Reviewed-by: John Spray <john.spray@redhat.com>

mds: open base inode's snaprealm after decoding snapblob

base inode has no parent, so can we mark its snaprealm open
immediately. This change makes sure replica mdsdir's snaplream
is opened.

Fixes: #12578
Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #5140 from tchaikov/wip-pg-ls-update-pending-rel-notes

doc: add changes to "pg ls*" commands to pending release notes

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: add changes to "pg ls*" commands to pending release notes

"pg ls*" commands now accept recovering instead of recovery as
one of its states.

Fixes: #11569
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5337 from branch-predictor/bp-reduce-mallocs

Thread.cc: remove malloc/free pair

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5000 from ceph/wip-da-SCA-20150601

Various SCA and Coverity fixes

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5312 from xinxinsh/wip-sparse-copy

use SEEK_HOLE/SEEK_DATA for sparse copy

Reviewed-by: Haomai Wang <haomaiwang@gmail.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5420 from ceph/wip-12429

osd/OSDMap: handle incrementals that modify+del pool

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5431 from kylinstorage/test_cache_pool_script_fix

fix: qa/workunits/rados/test_cache_pool.sh

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5430 from ceph/wip-12537

rgw: url encode exposed bucket

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5400 from tchaikov/wip-12451

tools/rados: update the help message

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5440 from vumrao/wip-vumrao-12458-doc

doc: change --keyfile description in man page of rbd help

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc: change "--keyfile" description in man page of rbd help

"--keyring" has higher priority than "--keyfile".
"--keyfile" option is overridden by "--keyring"
if latter is also specified.

Signed-off-by: Vikhyat Umrao <vumrao@redhat.com>

Merge pull request #5455 from ceph/wip-cephfs-cmake

CMake fixes for CephFS

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5456 from kylinstorage/wip-do_autogen.sh

Allow do_autogen.sh pass configure parameters

Reviewed-by: Sage Weil <sage@redhat.com>

CMake: add crushtool

Signed-off-by: John Spray <john.spray@redhat.com>

CMake: fix librados build

Signed-off-by: John Spray <john.spray@redhat.com>

CMake: fix rbd build

Don't link with 'global' from rbd (to match
autotools update fa78739).  Fixes "g_ceph_context
defined twice" error.

Signed-off-by: John Spray <john.spray@redhat.com>

CMake: libblkid not only needed with rbd

It's linked in libcommon.

Signed-off-by: John Spray <john.spray@redhat.com>

CMake: smalliobenchrbd should dep on rbd

Signed-off-by: John Spray <john.spray@redhat.com>

CMake: set WITH_CEPHFS by default

Option was referenced in CMakeLists but
not actually set.

Signed-off-by: John Spray <john.spray@redhat.com>

CMake: fix cephfs cls build

Signed-off-by: John Spray <john.spray@redhat.com>

CMake: fix cephfs test targets

These had bad lib names.

Signed-off-by: John Spray <john.spray@redhat.com>

CMake: add cephfs tools

cephfs-table-tool, cephfs-data-scan, cephfs-journal-tool

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #5422 from ceph/wip-fix-filestore-getattr

tests: fix the ObjectStore/StoreTest.SimpleCloneTest in "ceph_test_objectstore"

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5375 from ceph/wip-config-root-ino

Make UID of new root inos configurable

add help to do_autogen.sh

Signed-off-by: Yunchuan Wen <yunchuanwen@ubuntukylin.com>
Reviewed-by: Li Wang <liwang@ubuntukylin.com>

Allow do_autogen.sh to pass configure parameters

Currently do_autogen.sh does not allow to pass parameters
of configure to disable/enable components during building, which is not
flexible, this patch add the ability to it.

Signed-off-by: Yunchuan Wen <yunchuanwen@ubuntukylin.com>
Reviewed-by: Li Wang <liwang@ubuntukylin.com>

Merge pull request #5445 from ceph/wip-12432

Wip 12432

rgw: set http status in civetweb

Need to set the http status in civetweb so that we report it correctly.
Fixes: #12432
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

civetweb: update submodule to support setting of http status

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #5258 from zhouyuan/rgw_skip_first_chunk

rgw: skip prefetch first chunk if range get falls to shadow objects

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

doc: krbd supports clones since 3.10

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #5438 from ceph/fix-doc-build

doc: fix the build of placement-groups.rst

Reviewed-by: Kefu Chai <kchai@redhat.com>

os/FileStore: getattr() should return 0 if success

* fix the test of ObjectStore/StoreTest.SimpleCloneTest, the expected
  return value of getattr() should be 0. otherwise it otherwise it fails
  on objectstore backends of "memstore" and "keyvaluestore".
* change FileStore::getattr() to return 0 on success
* tighten related tests in ceph_test_objectstore, to test the return value
  of getattr() using ASSERT_EQ(r, 0) instead of ASSERT_GE(r, 0)

Signed-off-by: Kefu Chai <kchai@redhat.com>

KeyValueStore: Fix incorrect rm_keys behaviors

GenericObjectMap will copyup parents keys when rm keys, but if
caller(KeyValueStore) already buffer "set keys", it shouldn't
copyup reset keys.

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

doc: fix the build of placement-groups.rst

add a reference target of "mapping pgs to osds"

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5432 from ceph/wip-fix-rbd-build

make: fix the build of "rbd"

Reviewed-by: Loic Dachary <ldachary@redhat.com>

fix: qa/workunits/rados/test_cache_pool.sh

Signed-off-by: Mingxin Liu <mingxinliu@ubuntukylin.com>

Merge pull request #5434 from tobegit3hub/remove_execute_mode

Remove execute mode of source files

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Remove execute mode of source files

Merge pull request #5398 from mingyuez/master

rbd:If the image has any protected snaps,'rbd snap purge image' will return failure without removing any snaps.

Reviewed-by: Josh During <jdurgin@redhat.com>
Reviewed-by: Haomai Wang <haomaiwang@gmail.com>

make: fix the build of "rbd"

`CEPH_GLOBAL` was removed in fa78739, but the linked libraries does not
offer all needed symbols for "rbd". so add them back.

Signed-off-by: Kefu Chai <kchai@redhat.com>

rbd:'rbd purge image' will return failure without removing any snaps if the image has a protected snap

Signed-off-by: mingyuez <zhao.mingyue@h3c.com>

Merge pull request #5395 from tchaikov/wip-12512

common/syncfs: fall back to sync(2) if syncfs(2) not available

Reviewed-by: Haomai Wang <haomaiwang@gmail.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5402 from ceph/wip-claim-sparse-read-buffer

osd/ReplicatedPG: claim the read buffer instead of copying it

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #5428 from ceph/wip-kill-rest-bench

kill rest-bench

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

rgw: skip prefetch first chunk if range get falls to shadow objects

Currently the head object will be prefetched in each GET:
 a) This is unnecessary if the Range GET falls to shadow objects.
 b) The GET request would be quite slow if we have a big head object
This patch adds some check on the Range. If it's not in the head
object(>=rgw_max_chunk_size) then skip the prefetch.

Fixes: #12539
Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

Merge remote-tracking branch 'origin/next'

Merge pull request #5425 from ceph/wip-12479-next

librbd: test crash during client upgrade suite

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rgw: url encode exposed bucket

Fixes: #12537
Don't send the bucket name back without url encoding it.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #5423 from oddomatik/patch-1

doc: No CRUSH involvement in deciding PG target

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5426 from ceph/wip-12383-next

rbd: remove dependency on non-ABI controlled CephContext

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #5399 from tchaikov/wip-12458

rbd: add "--keyring" option to help message

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #5414 from cxwshawn/master

doc: add bucket object version description.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

remove libs3 submodule

Signed-off-by: Sage Weil <sage@redhat.com>

remove rest-bench

This is a weak tool; users should look to cosbench or others instead.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5419 from ceph/wip-kfsx-invalidate

test_librbd_fsx: invalidate before discard in krbd mode

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

No CRUSH involvement in deciding PG target

Quoting from elsewhere "[we calculate] a placement group using the object name, a hash code, the number of PGs in the pool and the pool name"

osd/OSDMap: handle incrementals that modify+del pool

Because new_pools was processed after old_pools, if something
was modified and then deleted in the same incremental
map, then the resulting state would be a pool in the map
that had no entry in pool_name.

Fixes: #12429
Signed-off-by: John Spray <john.spray@redhat.com>

lockdep: allow lockdep to be dynamically enabled/disabled

librbd test cases attempt to enable lockdep coverage via the librados
API.  Use a configuration observer to register/unregister lockdep
support.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: librbd API test cannot use private md_config_t struct

Remove all depencencies on md_config_t and instead use librados API
methods to get/set configuration values.

Fixes: #12479
Backport: hammer
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librados_test_stub: implement conf get/set API methods

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: add "--keyring" option to help message

Refs: #12458
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5401 from ceph/wip-12450

tools/rados: change the default max_ops to 16

Reviewed-by: David Zafman <dzafman@redhat.com>

doc: add bucket object version description.
bucket object version has been supported, but do not have description in the docs,
so add this part.

Signed-off-by: shawn chen <cxwshawn@gmail.com>

test_librbd_fsx: invalidate before discard in krbd mode

Commit bd050240ceef ("test_librbd_fsx: flush before discard in krbd
mode") added an fsync() before BLKDISCARD.  Don't know what I was
thinking at the time, but I missed the invalidate part, for which we
need to use the BLKFLSBUF ioctl.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Log::reopen_log_file: take m_flush_mutex

Otherwise, _flush() might continue to write to m_fd after it's closed.
This might cause log data to go to a data object if the filestore then
reuses the fd during that time.

Fixes: #12465
Backport: firefly, hammer
Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit 8778ab3a1ced7fab07662248af0c773df759653d)

Merge pull request #5348 from athanatos/wip-12465

log: take mutex when reopening log fd

Reviewed-by: Sage Weil <sage@redhat.com>

tools/rados: change the first op id to 0

the first op id was 16 by default, which is okay, but a non-zero
magic number could lead to questions. max_op was mixed up with
max_ops, and changed to 16 in 51e402e3 by mistake.

Signed-off-by: Kefu Chai <kchai@redhat.com>

tools/rados: change the default max_ops to 16

otherwise the load-gen will not emit any ops by default.

Fixes: #12450
Signed-off-by: Kefu Chai <kchai@redhat.com>

tools/rados: update the help message

Fixes: #12451
Signed-off-by: Kefu Chai <kchai@redhat.com>

common/syncfs: fall back to sync(2) if syncfs(2) not available

Fixes: #12512
Signed-off-by: Kefu Chai <kchai@redhat.com>

rbd: remove dependency on non-ABI controlled CephContext

The rbd CLI tool no longer attempts to initialize a CephContext
and pass said context to librados since it's possible that the
structure will not be ABI compatible between rbd and librados.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #5139 from jmunhoz/wip-7817-ms

rgw: add minimum support for copy multipart part

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #5313 from guce/rgw_access_invalid_argument

rgw: check subuser illegal access parameter.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

crypto: use NSS_InitContext/NSS_ShutdownContex to avoid memory leak

Switched to context-aware NSS init/shutdown functions to avoid conflicts
with parent application.  Use a reference counter to properly shutdown the
NSS crypto library when the last CephContext is destroyed.  This avoids
memory leaks with the NSS library from users of librados.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #5364 from xdonghai/master

doc:modified librados-intro.rst an error of document representation

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #5191 from theanalyst/v9.0.2/mailmap

doc: mailmap for v9.0.2 release

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5130 from tchaikov/wip-12167

ceph-detect-init: do not require "argparse" on py2.6

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5338 from hjwsm1989/wip-12459

common/WorkQueue: fix the FTBFS with GCC 4.4.7

* add nested-name-specifier ThreadPool before WorkQueueVal

Reviewed-by: Kefu Chai <kchai@redhat.com>

tools/cephfs: respect mds_root_ino_[gid|uid]

...while injecting inodes.  So that if you're a cephfs
admin but not uid=0 on the client, you can get things
injected as your choice of uid/gid.

Definitely useful for dev/test, potentially useful
elsewhere too.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: configurable uid/gid on new root inos

This is to deal with the slightly perverse situation
where a user might have all the permissions they need
to create a filesystem and mount it with fuse, but then
be unable to do anything inside the mount because
we default to creating a root-owned "/" inode.

This is immediatley useful for dev/test environments without
root, but also potentially in production environments
where the filesystem configuration/mounting is done by
someone less privileged than root.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #5350 from ceph/wip-pgls-filter-enc

osdc: fix pgls_filter op encoding

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5355 from ceph/wip-anotate-s-files

Anotate all the .s files

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5116 from yuyuyu101/wip-compressor

Wip compressor

Reviewed-by: Sage Weil <sage@redhat.com>

modified librados-intro.rst

Signed-off-by: x11507 <xu.donghai@h3c.com>

Merge pull request #5254 from vuhuong/wip-xio

@vuhuong sorry for the latency, could you please rebase & repush , to poke the built bot again?

once the build completes, will get it merged.

thanks,

osd/ReplicatedPG: claim the read buffer instead of copying it

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5349 from yuyuyu101/wip-async-fix-21

AsyncConnection: Exit process loop if entering fault

Merge remote-tracking branch 'origin/next'

Merge pull request #5352 from ceph/wip-rbd-help-specs

rbd: cli help and arg parsing fixes

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #5336 from ceph/wip-migrate-event-seq

mds: fix val used in inode->last_journaled

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Anotate all the .s files

Recent update to erasure code library in 59aa6700 caused a regression
where .s files are no longer properly anotated and yasm sets the exec
stack for them. This patch brings back the anotations as was done before
by Dan Mick, see Bug #10114.

Signed-off-by: Boris Ranto <branto@redhat.com>

rbd: rename --object-extents option to --whole-object

--object-extents is a bit confusing - extent is generally something of
a varying length and here the meaning is "diff whole objects".  Rename
it to --whole-object (the name of diff_iterate() parameter).

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

rbd: du command should take spec as an argument

Change du to take <image-spec> | <snap-spec> as an argument instead of
going through --image option.  The new synopsis is

    (du | disk-usage) [<image-spec> | <snap-spec>]

This is to make it look more like the rest of the commands: the only
other command that takes pool as an argument is ls and it can't really
serve as a prototype for du, because the latter has to work on images
and snapshots as well.

Examples:

    # stats for pool rbd
    $ rbd du
    $ rbd -p rbd du

    # stats for pool foo
    $ rbd -p foo du

    # stats for snapshot mysnap of image baz in pool rbd
    $ rbd du baz@mysnap

    # stats for image bar in pool foo
    $ rbd du foo/bar

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

rbd: assorted cli arg parsing fixes

- import-diff needs image-spec, yet OPT_IMPORT_DIFF was in the list of
  commands that don't need imgname

- check for lock id after imgname, so that if imgname isn't specified
  we fail with imgname rather than lock id error message

- error out if locker isn't specified

- move feature-name validity check after imgname checks so that if
  imgname is malformed we fail with imgname rather than feature error
  message

- error out if image-meta key or value isn't specified

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

rbd: error out if dest_snapname is specified

No command uses it as of now, but only clone command fails; cp, mv and
import simply ignore it.  Check if it's set and exit with a generic
error message.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

rbd: import doesn't require image-spec arg, ditto for export and path

Mark those as such in help and clarify what image-spec defaults to.

Related, all command args in our man page are enclosed into brackets.
I suppose the reason is that they are optional in the sense that you
can have commands like

    $ rbd clone --pool a --image b --snap -c --dest-pool d --dest e

with no args.  Given that we are trying to push people towards

    $ rbd clone a/b@c d/e

undo that so that real optional arguments can be marked optional.
While at it, add synopsis for each command and use backticks for
denoting commands more consistently.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

doc: use spec syntax in rbd docs

Use spec syntax instead of --pool, --image and --snap to hopefully push
people towards the former.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>