mds: set ScrubHeader for dirfrag when doing non-recursive scrub

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: improve debug message for CInode::validate_disk_state

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: improve debug message for CDir::check_rstat()

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: remove unused 'Context *' parameter from MDCache::repair_foo_stats()

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: report dirfrag stats error found by scrub

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: use MDSInternalContextBase instead of Context in scrub stack code

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: queue non-recursive scrub to top of scrub stack

non-recursive scrub blocks admin socket, make it have earliest service

Signed-off-by: Yan, Zheng <zyan@redhat.com>

Continuation: delete self after calling _done()

make sure Continuation delete itself even when virtual function
_done() is overwrited.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: improve passed_validation check.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: fix typo

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: and 'force' options to scrub_path asok command

'force' means check whole subtree (don't skip scrubbing entries
not modified since last scrub)

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: remove obsoleted FIXME from scrubstack code

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: make scrubstack queue CInode instead of CDentry

There are two problems that enqueue CDentry in scrub stack. First,
the scrub stack code can't handle base inodes; second, the linkage
of a CDentry can change in the middle of scrubbing, this can confuse
the scrub stack code.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: don't ignore scrubbing non-regular non-directory inodes

The overhead of scrubbing them is low, so don't treat them specially.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: and 'recursive/repair' options to scrub_path asok command

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: remove old scrub_dentry code

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: use scrubstack code to scrub single dentry

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: handle non-recursive scrubbing

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: store ScrubHeader refernce in CInode's scrub_info_t

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: scrub fragstat.mtime and rstat.rctime

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: set bad dirfrag's version to 1

there are lots of places assume 'version == 0' means dirfrag is
uninitialized

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: don't getxattr and setxattr at the same time

If an osd request contains write operation, the reply will not
carry output data. So we can't getxattr and setxattr in single
OSD request.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: check all items even when CInode::validate_disk_state() finds errors

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: repair fragstat/rstat errors in inode

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: repair fragstat/rstat errors in dirfrag

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: properly free CInode::ValidationContinuation

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: don't crash when scrub finds bad fragstat/rstat

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: check both frag_stat and nest_stat when validating inode

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: check dirfrag rstat when dirfrag is fetched

This avoids the problem that dirfrag becomes imcomplete before
ValidationContinuation::_dirfrags() get called;

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: skip scrubbing dentries not modified since last scrub

Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #7946 from SUSE/wip-doc-dev-teuth

doc/dev: integrate testing into the narrative

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc/dev: integrate testing into the narrative

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #7862 from theanalyst/rgw/doc-insecure-keystone

doc: rgw explain keystone's verify ssl switch

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #7887 from cbodley/wip-rgw-test-multi-period

test/rgw: add multisite test for meta sync across periods

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #7827 from cbodley/wip-rgw-period-meta-logs

rgw: fixes for per-period metadata logs

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #7941 from SUSE/wip-teuos-intro

doc: detailed description of bugfixing workflow

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: detailed description of bugfixing workflow

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #7933 from ceph/wip-doc-dist-ceph-detect-init

doc: add ceph-detect-init(8) source to dist tarball

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #7937 from saru95/patch--1

ceph_daemon.py: Resolved ImportError to work with python3

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7936 from ceph/wip-fix-debian-build

debian/changelog: Remove stray 'v' in version

ceph_daemon.py: Resolved ImportError to work with python3

Signed-off-by: Sarthak Munshi <sarthakmunshi@gmail.com>

rgw: fix for RGWCoroutinesManager::run() return value

the 'int ret' variable of the inner scope was shadowing an 'int ret'
variable in the outer scope, so we weren't propagating any of the error
codes

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: use current period for InitSyncStatus

the InitSyncStatus coroutine records the position to start incremental
sync after finishing a full sync.  this should be the master's marker
from the current period, rather than its oldest log period

this also adds a check to run_sync() that restarts a full sync if it
sees that our sync period is behind the master's oldest log period

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: meta log rest handlers avoid get_log()

RGWMetadataManager::get_log() will allocate a log and keep it in memory.
this could lead to a potential denial of service by making requests with
lots of different period ids

RGWMetadataLog if effectively stateless (the only state is a set of
modified_shards, which are not touched by any of the rest api calls), so
we can use a temporary instead of calling get_log()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: pull first log period from master if not found locally

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: implement find_oldest_log_period for RGWMetadataManager

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: period history behaves with no current_period

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: initialize async_rados before meta_mgr

RGWMetadataManager::init() will depend on async_rados

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: make RGWMetadataLog::get_shard_oid public and const

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add RGWStatObjCR for async raw_obj_stat

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: only write meta sync shard markers if can_adjust_marker

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: make RGWMetaSyncCR wakeups thread-safe

now that the shards will be coming and going after startup, they need to
be reference counted and protected by a mutex

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw-admin: add read_current_period_id() helper function

Signed-off-by: Casey Bodley <cbodley@redhat.com>

debian/changelog: Remove stray 'v' in version

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #7918 from ceph/wip-rgw-payer-yehuda

rgw: link payer info to usage logging

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

doc: add ceph-detect-init(8) source to dist tarball

Prior to this commit, the tarball from "make dist" did not include the
ceph-detect-init(8) man page rST source.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #7929 from yehudasa/wip-submodules-fix

submodules: revert an accidental change

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #7910 from ceph/wip-14955

buffer: increment history alloc as well in raw_combined

Reviewed-by: Sage Weil <sage@redhat.com>

submodules: revert an accidental change

revert changes that went in in commit f1ac0de5bfd974258c257470feae5ff03054873e.
This affects ceph-erasure-code-corpus, civetweb

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #7916 from ceph/wip-14975

rgw: adjust error code when bucket does not exist in copy operation
rgw: don't override error when initializing zonegroup

Fixes: #14975
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

rgw: avoid showing payer when payer == owner

If the payer matches the owner it shows the owner only.

Signed-off-by: Javier M. Mellid <jmunhoz@igalia.com>

Merge pull request #7786 from ceph/wip-rgw-indexless

rgw: indexless buckets (Yehuda Sadeh)

 - can define a policy, for which buckets are indexless
 - users can then create buckets under the specified placement target
 - indexless buckets will not be synced across zones
 - does not work with (s3) versioned buckets

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: indexless buckets

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: configurable index type

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: don't override error when initializing zonegroup

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: adjust error code when bucket does not exist in copy operation

Fixes: #14975
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #7764 from rzarzynski/wip-rgw-etag-dlo

rgw: multiple improvements regarding etag calculation for SLO/DLO of Swift API.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

buffer: increment history alloc as well in raw_combined

Introduced in 73dcd26fc8798e09d93520c583d63edb28ffc631

Fixes: #14955
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge remote-tracking branch 'gh/jewel'

v10.0.4

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #7897 from branch-predictor/bp-bench-errors-fatal

common/obj_bencher.cc: make verify error fatal

Reviewed-by: Loic Dachary <ldachary@redhat.com>

common/obj_bencher.cc: make verify error fatal

When run without "--no-verify", all verification errors are noted,
but they are not forwarded/reported anywhere else but to cerr, which
will cause automated testing to ignore them. Make seq_read_bench and
rand_read_bench return -EIO on any verification error which will,
in turn, return it back to caller.

Fixes: #14971
Signed-off-by: Piotr Dałek <piotr.dalek@ts.fujitsu.com>

Merge pull request #7757 from ukernel/jewel-14684

mon/MDSMonitor.cc: properly note beacon when health metrics changes

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #7882 from branch-predictor/bp-unconfuse-health-1

mon: unconfuse object count skew message

Reviewed-by: Sage Weil <sage@redhat.com>

PGMonitor: unconfuse object count skew message

"Pool <pool> has too few pgs" is okay assuming it does not take other
pools into account. And since it does, it is confusing in the following
scenario:

1. Create two pools, one with small pg count and one with large
   pg count
2. Put a whole lot of objects in smaller pool, resulting in "too few
   pgs" warning on that pool, which is expected behavior.
3. Put a whole lot of objects in larger pool, warning goes away.
   Suddenly smaller pool has plenty of PGs?

Current message suggests adding more nodes (or PGs) to pool, when
actually it's warning about significantly more objects in that
particular pool than in the other pools.

Signed-off-by: Piotr Dałek <piotr.dalek@ts.fujitsu.com>

Merge pull request #7898 from oritwas/wip-cmake-add-fs_types

cmake: add common/fs_types.cc to libcommon

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #7881 from aclamk/ceph-df-used-fix

mon: fix calculation of %USED

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7892 from majianpeng/bluestore-leak-unwritten-extent

osd: bluestore: fix check for write falling within the same extent

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7886 from stiopaa1/blueStoreOverrides

osd: bluestore: add 'override' to virtual functions

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7895 from majianpeng/bluestore-trim-overlay

os/bluestore/BlueStore: Don't leak trim overlay data before write.

os/bluestore/BlueStore: Don't leak trim overlay data before write.

Suppose: bluestore_overlay_max_length=bluestore_min_alloc_size;
bluestore_overlay_max = 2;
For the following ops:
write(off=0, len=4096) --->write into overlay
write(off=4096, len=4096)-->write into overlay
write(off=0, len=bluestore_min_alloc_size)-->because overlay_map.size()
>=2, it allocate a extent.

It should trim overlay data(0,4096) &(4096, 4096),and then write(0,
bluestore_min_alloc_size).
But the original code don't trim overlay data.
This make the later read data is orignal data rather that new data.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

cmake: add common/fs_types.cc to libcommon

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

os/bluestore/BlueStore: Fix bug when calc offset & end whether locate in the a extent.

Suppose: bluestore_overlay_max_length == bluestore_min_alloc_size
The orignal code which calc content of written whether locate in a
extent:
(offset / min_alloc_size) == (offset + length) /min_alloc_size
This will make the case which offset=0 & length =min_alloc_size locate
in the different extent.
In fact, this content is in the same extent.

Change end = offset + length - 1 make work.

Fixes: #14954
Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

common/obj_bencher.cc: use more readable constant instead of magic number

When clean_up_slow() fails, it returns "-5" which is equal to -EIO.
Change it in source, so it's not confusing for someone who does not
remember all error codes (functionality remains the same).

Signed-off-by: Piotr Dałek <piotr.dalek@ts.fujitsu.com>

Merge pull request #7098 from liewegas/wip-file-layout

mds, client: add namespace to file_layout_t (previously ceph_file_layout)

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Bluestore.h:Added 'override' to virtual functions

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

test/rgw: add multisite test for meta sync across periods

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #7849 from ceph/wip-cmake

Wip cmake

cmake: simplified heap_profiler_objs dependencies

The heap_profiler object library only needs to be
added to mon,mds, and osd instead of the numerous
targets it is part of

Signed-off-by: Ali Maredia <amaredia@redhat.com>

client/Inode: fix has_dir_layout

Compare to default file_layout_t() -- not zeroed memory.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #7867 from SUSE/wip-14940

RPM: move %post(un) ldconfig calls to ceph-base

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #7869 from rzarzynski/wip-rgw-aws4-puthash

rgw: calculate payload hash in RGWPutObj_ObjStore only when necessary.

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge remote-tracking branch 'gh/jewel'

Merge pull request #7612 from liewegas/wip-buffer-combined

buffer: raw_combined allocations buffer and ref count together

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #7690 from xiexingguo/xxg-wip-14804

common/TrackedOp: fix inaccurate counting for slow requests

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7704 from xiexingguo/xxg-wip-14814

osd: filestore: fix wrong scope of result code for error cases during mkfs

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7812 from xiexingguo/xxg-wip-14872

osd: fix wrong return type of find_osd_on_ip()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7815 from branch-predictor/bp-conditional-crc

msg/async: don't calculate msg header crc when not needed

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #7711 from xiexingguo/xxg-wip-14817

osd: filestore: fix result code overwritten for clone

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7790 from tchaikov/wip-log-segv

log: segv in a portable way

Reviewed-by: Nathan Cutler <ncutler@suse.com>