Merge pull request #3893 from ceph/wip-10390

#10390: Refactor stray handling code and add throttling

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #3371 from nilamdyuti/wip-doc-ceph-object-gateway

Updates radosgw man page with configuration for mod_proxy_fcgi

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

Merge pull request #4137 from majianpeng/assert-version

osd/ReplicatedPG: For CEPH_OSD_OP_ASSERT_VER, it should use

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4141 from tchaikov/sync-rst-with-manpages

doc: update .rst file with their .8 counterparts

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4117 from ceph/wip-libcephfs-flock

Fixes for libcephfs flock tests

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

doc: Adds updated radosgw-admin man page under man/

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

doc: Updates radosgw-admin man page.

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

doc: Adds updated radosgw man page under man/

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

doc: Updates radosgw man page.

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

doc: Adds updated radosgw man page under man/

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

doc: Updates radosgw man page with configuration for mod_proxy_fcgi .

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

mds: be fairer in enqueuing purges

Don't let purges that require fewer ops skip ahead of those
that require more ops.

Signed-off-by: John Spray <john.spray@redhat.com>

doc: update .rst file with their .8 counterparts

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/ReplicatedPG: For CEPH_OSD_OP_ASSERT_VER, it should use
op.assert_ver rather than op.watch.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #4135 from ceph/wip-11198

Fix ceph_test_async_driver failed

Fix ceph_test_async_driver failed

This test will create 10000 sockets which will failed because of limited system fd. Actually we only need to create several hundreds sockets and it's enough to get the test's goal.

Fix bug #11198

Merge pull request #3873 from tchaikov/cli-human-readable-errno

rados: translate errno to str in CLI

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3841 from ceph/wip-10217

qa/workunits/fs/misc: fix filelock_interrupt.py

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #3829 from liewegas/wip-recovery-sleep

osd: inject simple sleep in recovery

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3685 from XinzeChi/wip-mark-started

osd: clear extra mark_started

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #3282 from tchaikov/pq-test

common/PrioritizedQueue: add basic tests for it

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3177 from davidzhang84/wip-inject-eio-ec

Add EIO injection on EC pool for mdata and data error only for now

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3889 from XinzeChi/wip-tier-stat

osd: add l_osd_op_cache_hit perf counter for cache pool

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4010 from majianpeng/osdmap-nspace

mon:make 'ceph osd map' accept namespace.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4130 from dachary/wip-backports

doc: what does it mean for a release to be supported

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

doc: what does it mean for a release to be supported

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4122 from trhoden/rgw-quickstart

doc: Add RGW quick start info

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4098 from ceph/wip-11128

auth: reinitialize NSS modules after fork()

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

auth: use crypto_init_mutex to protect NSS_Shutdown()

Signed-off-by: Yan, Zheng <zyan@redhat.com>

doc: Add RGW quick start info

Add details of how to use ceph-deploy to deploy the CivetWeb-based
RGW present in Hammer.

Signed-off-by: Travis Rhoden <trhoden@redhat.com>

Merge pull request #4120 from trhoden/quickstart-mon-create

doc: remove reference to old ceph-deploy steps

Reviewed-by: Sage Weil <sage@redhat.com>

doc: remove reference to old ceph-deploy steps

ceph-deploy 1.1.3 is ancient, and isn't easily found or installed.

No need to reference it and call out the steps necessary to use if
using it.

Signed-off-by: Travis Rhoden <trhoden@redhat.com>

Merge pull request #4103 from ceph/wip-error-str

mds: fix error_str formatting in scrub output

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4116 from ceph/wip-10368

mds: call wait_for_safe twice in command_flush_journal

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

mds: call wait_for_safe twice in command_flush_journal

...so that we wait until everyone who was waiting for the first
wait_for_safe has run their completion before we proceed, thus
avoiding the possibility that those completions will execute
during our trimming and cause problems.

Fixes: #10368
Signed-off-by: John Spray <john.spray@redhat.com>

auth: reinitialize NSS modules after fork()

Fixes: #11128
Signed-off-by: Yan, Zheng <zyan@redhat.com>

auth: properly return error of cephx_calc_client_server_challenge

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: improved doxygen comments in StrayManager

Signed-off-by: John Spray <john.spray@redhat.com>

mds: update StrayManager op limit on mds_max_purge_ops_per_pg

Previously was only updating on mds_max_purge_ops

Signed-off-by: John Spray <john.spray@redhat.com>

mds: fix parent check in MDCache::eval_remote

This code path wasn't touched until the change
in "mds/Server: fix stray reintegration".  Now we
see that it will crash in fsstress in some circumstances
when the client is doing a link operation, where
we are checking the projected linkage on the DN
before going into this function, but not the projected
parenthood of the inode.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: make purge op limit dynamic based on PGs/MDSs

Signed-off-by: John Spray <john.spray@redhat.com>

mds: throttle purge stray operations

Pull out the stray-handling code into a separate StrayManager
class.

There is an additional improvement to stray reintegration here,
by passing the DN that triggered the reintegration through,
we should make a better choice about *which* hardlink to
reintegrate the inode into, rather than just picking the first.

Fixes: #10390
Signed-off-by: John Spray <john.spray@redhat.com>

common/config_opts: add purge throttle options

These are to be respected by the MDS in the number
of concurrent RADOS delete operations and
number of files to be purged at once.

Signed-off-by: John Spray <john.spray@redhat.com>

osdc: make op count in Filer::purge configurable

Previously this was simply hardcoded to do up to
10 RADOS delete ops at a time.  Make this a config
option so that Filer consumers (like the MDS) can
know how many concurrent operations will be done
on their behalf.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: eval stray on remove dentry replica

This happens because when handling an MCacheExpire,
we expire inodes first and then dentries, so when
the inodes are put() their dentries still are replicated
so eval_stray fails.  Do an explicit call to maybe_eval_stray
when handling the expire for the dentry in order to catch
this.

One hits this path when migrating a stray away from
a stopping MDS rank.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: expire other ranks' mydirs during 'stopping'

Avoids the surviving MDSs thinking their base dir is still
replicated after a rank has been stopped.

Signed-off-by: John Spray <john.spray@redhat.com>

mds/Server: fix stray reintegration

This was checking tracei's primary parent DN for remoteness
as a condition to calling eval_remote.  However, the *primary*
parent DN in this case is the stray, which is not remote.

The correct check is tracedn (the remote dentry that got us here)
rather than tracei->get_parent_dn() (the primary dentry for the
inode that we have looked up).

This leads to eval_remote being invoked, and reintegration
of stray hard-linked inodes working.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #4114 from apeters1971/wip-isa-lrucache

erasure-code: make ErasureCodeIsaTableCache drop entries according to LRU

Reviewed-by: Loic Dachary <ldachary@redhat.com>

erasure-code: make ErasureCodeIsaTableCache drop entries according to LRU

There was a bug in the arguments used by the splice function to move elements from the LRU list to the head of the list.
To simplify the implementation the direction of the LRU list has been changed e.g. newest entries are pushed in front.

Signed-off-by: Andreas-Joachim Peters <andreas.joachim.peters@cern.ch>

Merge pull request #4084 from ceph/wip-switch-to-breathe

Wip switch to breathe

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: s/Asphyxiate/Breathe/ in documenting.rst

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: enable doxygen for enum

* since we switched to breathe, which supports rendering enum
  types

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: Switch doxygen integration back to breathe

* asphyxiate needs more toolings, see
  https://github.com/ceph/asphyxiate/issues/1
* this commit basically reverts c96064
* use `autodoxygen` directive to doxygen referenced source files
* do not call `doxygen` explicitly in `build-doc`, `autodoxygen`
  will take care of it.

Fixes: #6115
Fixes: #6115
Signed-off-by: Kefu Chai <kchai@redhat.com>

test/libcephfs/flock.cc: don't release lock before EWOULDBLOCK check

make sure the main thread/process not release the file lock before
workers execute the operation that supposes to return EWOULDBLOCK.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

test/libcephfs/flock.cc: fix synchronization points

In the threesome test cases, the main thread/process call PING_WORKER()
twice at each synchronization point. It's possible that one worker gets
both notifications and takes two steps forward, another worker gets no
notification.

The fix is use seperate semaphores for odd/even number synchronization
points.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #3983 from yuyuyu101/wip-async-fix-3

Wip async fix 3

Reviewed-by: Sage Weil <sage@redhat.com>

Thread.cc: Make set_affinity private and correct behavior

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

doc/release-notes: more hammer release notes

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #4085 from ceph/wip-11160

rgw: don't use rgw_socket_path if frontend is configured

Reviewed-by: Sage Weil <sage@redhat.com>

doc/release-notes: make hammer note about filestore format upgrade

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3885 from Ved-vampir/master

common: add perf counters description

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #3840 from ceph/wip-libcephfs-fsetxattr

libcephfs: add ceph_f{get,set,list,remove)_xattr

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #3750 from ceph/wip-dump-cache

mds: Add 'dump cache' asok

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

mds: implement 'dump cache' asok command

Retains old text format output when user specifies
a filename, else returns JSON.  JSON output will
cope less well with large caches, but will be more
useful for automated tests.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: use MDSCacheObject::dump in CDir::dump

Signed-off-by: John Spray <john.spray@redhat.com>

mds: add CDentry::dump

Signed-off-by: John Spray <john.spray@redhat.com>

mds: extended dump() for CInode

Signed-off-by: John Spray <john.spray@redhat.com>

mds: add dump() to MDSCacheObject

Subclasses extend this with their own attributes.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: add SimpleLock::dump

Signed-off-by: John Spray <john.spray@redhat.com>

mds: add a dump() method to MutationImpl

This is just an externally callable wrapper
to _dump that doesn't require caller
to pass in a time.

Signed-off-by: John Spray <john.spray@redhat.com>

include/frag: add a dump() method to fragtree_t

Signed-off-by: John Spray <john.spray@redhat.com>

mds: move auth_pins attrs into MDSCacheObject

These were common between CInode, CDir, CDentry.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge branch 'master' of github.com:ceph/ceph

Merge remote-tracking branch 'gh/hammer'

Conflicts:
PendingReleaseNotes
src/Makefile.am
src/gmock
src/test/Makefile.am

Merge pull request #4035 from ceph/wip-11113

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3988 from ceph/wip-refine-build-configuration-hammer

Refine build configuration for hammer

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4104 from rzarzynski/wip-11148

rgw: rectify 202 Accepted in response for PUT on already existing bucket

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #4105 from athanatos/wip-11110

Wip 11110

Reviewed-by: Sage Weil <sage@redhat.com>

init-radosgw*: don't require rgw_socket_path to be defined

Fixes: #11159
Backport: hammer, firefly

Scripts required rgw_socket_path to exist in order to start radosgw.
This is not needed.

Reported-by: Dan Mick <dmick@redhat.com>
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 6823bcdcd0ce72cd223e809291f46d82da76115c)

Merge pull request #4087 from ceph/wip-11159

init-radosgw*: don't require rgw_socket_path to be defined

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4064 from dachary/wip-10488-jerasure-idempotent

mon: informative message when erasure-code-profile set fails

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4066 from dachary/wip-11144-erasure-code-profile-rm

osd: erasure-code-profile incremental rm before set

Reviewed-by: Sage Weil <sage@redhat.com>

PG: ensure that info.last_epoch_started only increases

See doc/dev/osd_internals/last_epoch_started.rst

Fixes: #11110
Signed-off-by: Samuel Just <sjust@redhat.com>

doc: add last_epoch_started.rst

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #4054 from diurchenko/wip-10662

rgw: Swift API. Allows setting attributes with COPY object operation.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rectify 202 Accepted in response for PUT on existing bucket.

Fixes: #11148
Backport: hammer
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

mds: fix error_str formatting in scrub output

operator<< on an ostringstream actually gets
you an address.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #4025 from ceph/wip-fix-configure-noversion

configure.ac: add --disable-gitversion

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

configure.ac: add --disable-gitversion

During development each git commit cause currently a complete
rebuild of the source tree. By passing --disable-gitversion to
configure we set a generic version string for ceph to prevent
getting after each commit a new version set.

remove src/check_version and replace with a extended version
of src/make_version which allows detecting changes in .git_version
and ceph_ver.h to avoid not needed updates.

This change also forces to call make_version each time to make
sure that changes from configure or the NO_VERSION env variable
are processed.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

init-radosgw*: don't require rgw_socket_path to be defined

Fixes: #11159
Backport: hammer, firefly

Scripts required rgw_socket_path to exist in order to start radosgw.
This is not needed.

Reported-by: Dan Mick <dmick@redhat.com>
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: don't use rgw_socket_path if frontend is configured

Fixes: #11160
Backport: hammer, firefly

Previously if we wanted to use the tcp fcgi socket, we needed to clear
rgw_socket_path.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #3951 from ceph/wip-cephfs-snaps

Wip cephfs snaps

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4072 from javierguerragiraldez/master

rados.py: fix Object.write() method

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rados.py: fix Object.write() method

the file position should advance by the number of bytes written,
but the ioctx.write() function returns 0 on success

Signed-off-by: Javier Guerra <javier@guerrag.com>

Merge pull request #3898 from jdurgin/wip-rbd-merge-diff

qa: make rbd-fuse exit cleanly with lttng

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4071 from athanatos/wip-10718

Wip 10718

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4016 from ceph/wip-11123

osd: fix whiteout handling for delete+create compound ops

Reviewed-by: Samuel Just <sjust@redhat.com>

PG: make sure to update history.last_epoch_started with first write

Otherwise, we might remember the write, but not that the activation
interval was history.last_epoch_started.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #4067 from ceph/wip-rbd-warning

tests: remove unused variable

Reviewed-by: Haomai Wang <haomaiwang@gmail.com>

tests: remove unused variable

Signed-off-by: Josh Durgin <jdurgin@redhat.com>