rgw: Use attrs from source bucket on copy

On copy objects, when bucket source is the same as the destination, use attrs
from source bucket.

Fixes: #11639
Signed-off-by: Javier M. Mellid <jmunhoz@igalia.com>

Merge pull request #4226 from ceph/wip-11276

#11276: make ceph-fuse well behaved with pool quotas

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4622 from xiaoxichen/wip-rocksdb-conf

Wip rocksdb conf

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4606 from ceph/wip-rgw-init

init-radosgw: check in /var/lib/ceph/radosgw ; unify w/ sysv version

Reviewed-by: Travis Rhoden <trhoden@redhat.com>

init-radosgw.sysv: remove

Signed-off-by: Sage Weil <sage@redhat.com>

init-radosgw: unify init-radosgw[.sysv]

Signed-off-by: Sage Weil <sage@redhat.com>

init-radosgw: look in /var/lib/ceph/radosgw

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #4680 from ceph/wip-fix-group-in-doxygen

doc: fix group in doxygen

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: customize css for the group title

Fixes: #9442
Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: use @name to define a group, not @group

we are able to output a specified group using the directive
of `doxygengroup` in breathe. this directive prints out the
description of the group. but it's not realistic to enumerate
all groups defined in source code in the rst files. but the
doxygen command @name also helps to group functions together.
the downside of this approach is that we can not add more
items to a group later on. but it should be fine with us,
since in our case, all the grouped items are living in a single
header file.

Fixes: #9442
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4671 from dachary/wip-contributing

doc: recommendations related to performances / cleanup

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4674 from flaf/master

Doc: add a note about a CRUSH corner case with `ceph osd out`

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4673 from dachary/wip-bencher

tools: remove obsolete aio_bench argument placeholder

Reviewed-by: Sage Weil <sage@redhat.com>

Doc: add a note about a CRUSH corner case with `ceph osd out`

Signed-off-by: François Lafont <francois.lafont@ac-versailles.fr>

tools: remove obsolete aio_bench argument placeholder

It was a noop and has been removed by
1ac727982213bf676beefe9340d6822193dbb700

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4672 from dachary/wip-bencher

common: fix ObjBencher::aio_bench signature

Reviewed-by: Sage Weil <sage@redhat.com>

common: fix ObjBencher::aio_bench signature

It was broken by 1ac727982213bf676beefe9340d6822193dbb700

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4592 from ceph/wip-7746

rbd: add disk usage tool

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4308 from dachary/wip-bencher

common: clarify some obj_bencher code

Reviewed-by: Dmitry Yatsushkevich <dmitry.yatsushkevich@gmail.com>

Merge pull request #4650 from stiopaa1/bufferIterator

buffer.h: Removed unnecessary copy constructor, assignment operator a…

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: recommendations related to performances / cleanup

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4668 from ceph/wip-rados-binary-omapsetval

rados cli: setomapval can now read from stdin

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4473 from ceph/wip-10809-11135-10290

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

test: Add config changes to all tests to avoid order dependency

ReplayCorrupt was crashing because journal_ignore_corruption wasn't set true
Improve ReplayCorrupt by checking setting of bool corrupt and use ASSERT_FALSE()

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Add admin socket feature set_recovery_delay

Set the value of osd_recovery_delay_start to specified seconds

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: For import/export --debug dump the log

Move log dumping code to common function dump_log()

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Use ceph-objectstore-tool --no-overwrite for import-rados

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Test ceph-objectstore-tool --dry-run in select operations

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Fix ceph-objectstore-tool test missing fd.close()

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Fix ceph-objectstore-tool test error message

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: If object re-appears after removal, just skip it

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Add --no-overwrite flag for import-rados

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Remove list-lost because now we have --dry-run flag

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Add --dry-run option

Fixes: #10290
Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Add dump-info command to show object info

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Use empty string for <object> to specify pgmeta object

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Add a couple of strategically placed prints

Signed-off-by: John Spray <john.spray@redhat.com>

ceph-objectstore-tool: Clean up error handling

Use negative errors throughout and positive error to specify exit status
cpp_strerror() handles negative errors
Clean up main() code to use local "ret"
Let end of main() decide how to treat the final value of "ret"

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Create section around log/missing/divergent_priors of --op log

Remove extra new lines in the output

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Add divergent_priors handling

Add to --op log output
Add to metadata_section for export
Add filtering of divergent_priors in case of splits

Fixes: #11237
Signed-off-by: David Zafman <dzafman@redhat.com>

test: ceph-objectstore-tool: Remove duplicate debug messages, keep cmd/log/call together

Signed-off-by: David Zafman <dzafman@redhat.com>

test: ceph-objectstore-tool import after split testing

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Use CEPH_DIR where appropriate

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Limit how long ceph-objectstore-tool test will wait for health

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Add optional arg to vstart() to provide additional args to vstart

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Test ceph-objectstore-tool --op dump-journal output

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Pep8 fixes for ceph-objectstore-tool test

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Add --force option which is used for import only

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Fix pgid scan to skip snapdirs

Long standing bug where it wasn't skipping snapdirs if !debug
For debug output use stderr like all other cases

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Add dump-journal op

Have dump-journal op directly dump a filestore journal without
mounting or try to dump after mounting.

Fixes: #11135
Signed-off-by: David Zafman <dzafman@redhat.com>

osd: FileJournal: Add _fdump() that takes Formatter instead of ostream

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Fix ceph-objectstore-tool test, overwrite OTHERFILE so second check is meaningful

Signed-off-by: David Zafman <dzafman@redhat.com>

test: ceph_test_filejournal: Conform to test infrastructure requirements

Only call RUN_ALL_TESTS() once

Signed-off-by: David Zafman <dzafman@redhat.com>

test: ceph_test_filejournal need to force aio because testing with a file

Signed-off-by: David Zafman <dzafman@redhat.com>

test: ceph_test_filejournal fix missing argument to FileJournal constructor

Signed-off-by: David Zafman <dzafman@redhat.com>

test: ceph_test_filejournal  Add check of journalq in WriteTrim test

Commit all journal entries and check for empty journalq

Signed-off-by: David Zafman <dzafman@redhat.com>

filestore: Change int open_journal() to void new_journal()

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Fix log output in FileJournal::read_entry() coming from do_read_entry()

Signed-off-by: David Zafman <dzafman@redhat.com>

os: Add some logging used only by test code

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Add simple_dump() to FileJournal for unit testing

Allow a dump when entries aren't ObjectStore::Transaction's

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: FileJournal clean-up

Move aio / directio adjustments to constructor
Indicate functions which only operate before journal is active
Make functions const when possible
Move push_back() of journalq to read_entry()
Change dump() to use do_read_entry() to minimize side effects

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Dump header in FileJournal::dump()

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: FileJournal::read_entry() can't use a zero seq to check for corruption

Seen when using dump() on a journal which has all entries committed

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Fix flushing in FileJournal::dump()

Use ostream arg since caller is specifying
Move flush after last section close

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: On any exit release CephContext so logging can flush

Add myexit() which will put() the g_ceph_context if present before exit
Must umount() once the objectstore has been mounted

Fixes: #10809
Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Check for keyvaluestore experimental feature

Don't list keyvaluestore as a valid type if feature not enabled

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Eliminate obscure "Invalid params" error

Will fail later with "No such file or directory" for an empty data-path
because we've already checked for vm.count("data-path")

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Check pgid validity earlier like we did before

This code was not doing the umount() even though it was moved after mount()

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-dencoder: Add ability to import from stdin by specifying "-"

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #4375 from tchaikov/wip-ceph-unrecognized-exception

ceph.in: do not throw on unknown errno

Reviewed-by: Loic Dachary <ldachary@redhat.com>

buffer.h: added some consts

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

buffer.h: Removed unnecessary copy constructor and assignment operator

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #4151 from dmitryya/issue_7401

obj_bencher: status_printer waits before the first calculation

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4552 from stiopaa1/bufferLinkage

common/buffer.cc: change linkage of few variables to internal

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4663 from dachary/wip-11598-ceph-detect-init

ceph-detect-init: plain setup.py install on make install

Reviewed-by: Kernel Neophyte <neophyte.hacker001@gmail.com>

Merge pull request #4666 from SUSE/wip-11610-next

ceph.spec.in: include SUSE in _with_systemd

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

test/cli-integration/rbd: verify formatting for 'rbd disk-usage'

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

ceph.in: do not throw on unknown errno

some of the errnos are not listed in errno.errorcode. if we happen
to run into them, print 'Unknown' instead.

Fixes: #11354
Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits/rados/test_rados_tool: added test for omapsetval via stdin

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: add disk usage tool

The new disk usage tool uses the new fast diff object map feature
(when enabled) to quickly calculate the provisioned vs actual usage
of images and associated snapshots within a pool.

Fixes: #7746
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: disable deep-flatten feature for the krbd fsx test

krbd doesn't support fast-diff or deep-flatten yet.  Had to do this
with two commits because fast-diff is in next and deep-flatten isn't.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

rados cli: setomapval can now read from stdin

This will allow a user to store binary values within omap.  Since
RBD uses binary omap values, this will assist with certain recovery
operations.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge remote-tracking branch 'origin/next'

Merge pull request #4665 from ceph/wip-11551-next

tests: disable new RBD features for the krbd fsx test

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #4644 from ceph/wip-11405

librbd: new QA client upgrade tests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

tests: disable fast-diff feature for the krbd fsx test

The kernel does not yet support fast-diff.

Fixes: #11551
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #4627 from dachary/wip-install-deps-clean

install-deps.sh: only pip install after make clean

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph.spec.in: include SUSE in _with_systemd

The master specfile newly defines a _with_systemd variable that should be true
for the set of distros that are using systemd. Since this set of distros
includes SUSE/openSUSE (at least for the more recent versions where ceph is
supported), this commit sets _with_systemdto true on SUSE/openSUSE.

http://tracker.ceph.com/issues/11610 Fixes: #11610

Signed-off-by: Nathan Cutler <ncutler@suse.cz>

Merge pull request #4536 from ceph/wip-mds-snaps

Wip mds snaps

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

ceph-detect-init: plain setup.py install on make install

If no DESTDIR environment variable is set, just

  python setup.py install

instead of setting --root, --prefix or --install-layout. These options
are tested to do the right thing in the context of RPM and deb packaging
but they are presumably not what someone using make install manually
would want.

http://tracker.ceph.com/issues/11598 Fixes: #11598

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4646 from theanalyst/doc/release/v0.94.2

doc: more entries for release notes in v0.94.2

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: more entries for release notes in v0.94.2

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

install-deps.sh: only pip install after make clean

Do not re-install all python dependencies if there already exists a
wheelhouse directory. It makes it so running install-deps.sh twice will
only access the network the first time. The directories where the python
dependencies are installed are removed by make clean. Refreshing the
python dependencies cache can be done via make clean ; install-deps.sh

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4521 from tchaikov/wip-fix-malformed_input-macros

common: fix the macros for malformed_input::what()

Reviewed-by: Loic Dachary <ldachary@redhat.com>

common: fix the macros for malformed_input::what()

the thrown exception of malformed_input should carry the function name in
which it was thrown. but what we have now is something like:
"buffer::malformed_input: __PRETTY_FUNCTION__ unknown encoding version >
100"
because __PRETTY_FUNCTION__ is not a macro any more. see
- https://gcc.gnu.org/onlinedocs/gcc-3.1/gcc/Function-Names.html
- https://gcc.gnu.org/onlinedocs/gcc/Function-Names.html
and it is not a string literal, so neither can we can concat it with
the literal err message.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/KeyValueDBMemory : update init()

make it build.

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

os/RocksDBStore: interpret some configurations.

compact_on_mount,
compaction_threads,
flusher_threads,
disableWAL.

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

common/str_map: trim start/tailing writesapce in key and value.

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

/s/omap_rocksdb_option/filestore_rocksdb_option

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

os/RocksDBStore: compact_on_mount not support now

will fix it.

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>