osd/OSDMap: bump encoding version for require_min_compat_client

Otherwise, OSDMap::decode() throws expcetion when decoding old
version osdmap

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #14743 from ukernel/wip-19635

mds: issue new caps when sending reply to client

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #14704 from batrick/i19589

mds: fix mantle script to not fail for last rank

Reviewed-by: Michael Sevilla <mikesevilla3@gmail.com>

Merge pull request #14921 from batrick/ptrwaiter-balance

mds: relocate PTRWAITER put near get

Merge pull request #14598 from batrick/mds-balancer-pin

mds: support export pinning on directories

mds: delay export until dir is stable

This catches an assertion failure due to exports on unjournaled directories:

    2017-05-10 12:45:18.193640 7eff65a04700  7 mds.0.migrator export_dir [dir 10000000003 /a/ [2,head] auth pv=3 v=1 cv=0/0 ap=1+2+2 state=1073742082|complete f()->f(v0 m2017-05-10 12:45:14.977151 1=0+1) n()->n(v0 rc2017-05-10 12:45:14.977151 1=0+1) hs=0+1,ss=0+0 | child=1 authpin=1 0x55ef655de000] to 1
    2017-05-10 12:45:18.193654 7eff65a04700 15 mds.0.18 hit export target 10 @ 2017-05-10 12:45:18.193653
    2017-05-10 12:45:18.193661 7eff65a04700 10 mds.0.cache.dir(10000000003) auth_pin by 0x55ef650cc130 on [dir 10000000003 /a/ [2,head] auth pv=3 v=1 cv=0/0 ap=2+2+2 state=1073742082|complete f()->f(v0 m2017-05-10 12:45:14.977151 1=0+1) n()->n(v0 rc2017-05-10 12:45:14.977151 1=0+1) hs=0+1,ss=0+0 | child=1 authpin=1 0x55ef655de000] count now 2 + 2
    2017-05-10 12:45:18.193701 7eff65a04700  7 mds.0.cache request_start_internal request(mds.0:4) op 5377
    2017-05-10 12:45:18.193708 7eff65a04700  7 mds.0.migrator dispatch_export_dir request(mds.0:4)
    2017-05-10 12:45:18.198281 7eff66205700 20 mgrc operator() 234 counters, of which 0 new
    2017-05-10 12:45:18.198294 7eff66205700 20 mgrc send_report encoded 2006 bytes
    2017-05-10 12:45:18.198298 7eff66205700  1 -- 127.0.0.1:6825/693985647 --> 127.0.0.1:6826/1025 -- mgrreport(+0-0 packed 2006) v2 -- 0x55ef655abe00 con 0
    2017-05-10 12:45:18.198364 7eff6a9ab700 10 _calc_signature seq 14 front_crc_ = 2388339344 middle_crc = 0 data_crc = 0 sig = 5026349537430007662
    2017-05-10 12:45:18.198376 7eff6a9ab700 20 Putting signature in client message(seq # 14): sig = 5026349537430007662
    2017-05-10 12:45:18.198411 7eff65a04700 -1 /home/pdonnell/ceph/src/mds/MDCache.cc: In function 'void MDCache::make_trace(std::vector<CDentry*>&, CInode*)' thread 7eff65a04700 time 2017-05-10 12:45:18.194136
    /home/pdonnell/ceph/src/mds/MDCache.cc: 8225: FAILED assert(parent)

     ceph version 12.0.1-2198-ge757c02 (e757c025fa3270b12fb2fca17cf159fa1bd72747)
     1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x55ef5b780f02]
     2: (MDCache::make_trace(std::vector<CDentry*, std::allocator<CDentry*> >&, CInode*)+0x1cb) [0x55ef5b4fa12b]
     3: (Migrator::get_export_lock_set(CDir*, std::set<SimpleLock*, std::less<SimpleLock*>, std::allocator<SimpleLock*> >&)+0x55) [0x55ef5b5e8215]
     4: (Migrator::dispatch_export_dir(boost::intrusive_ptr<MDRequestImpl>&, int)+0xa74) [0x55ef5b5f72d4]
     5: (Migrator::export_dir(CDir*, int)+0x9ca) [0x55ef5b5ea75a]
     6: (MDBalancer::handle_export_pins()+0x7b4) [0x55ef5b61ab24]
     7: (MDBalancer::tick()+0x1e8) [0x55ef5b61b748]
     8: (MDSRankDispatcher::tick()+0x5f1) [0x55ef5b44bdb1]
     9: (Context::complete(int)+0x9) [0x55ef5b43bcc9]
     10: (SafeTimer::timer_thread()+0x452) [0x55ef5b77dd52]
     11: (SafeTimerThread::entry()+0xd) [0x55ef5b77f15d]
     12: (()+0x76ba) [0x7eff6d4bb6ba]
     13: (clone()+0x6d) [0x7eff6c52782d]

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #15030 from chardan/jfw-wip-halflife_atomic_t-fix-compare_exchange_weak

tools: change compare_exchange_weak to compare_exchange_strong

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #14975 from runsisi/wip-io-logging

librbd: cleanup logging code under librbd/io

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #14905 from johnzzpcrystal/aio

test/fsx:  Remove the dead code associated with aio backend

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14911 from ovh/bp-small-enc-eradication

os/bluestore: use denc for varint encoding

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14957 from ovh/bp-crc-cache-miss-counter

buffer, osd: add missing crc cache miss perf counter

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14996 from xiexingguo/wip-fix-dump_basic

mon/PGMap.cc: fix "osd_epochs" section of dump_basic

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15003 from badone/wip-redundant-headers

common: Remove redundant includes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15013 from TsaiJin/wip-pg-lock-cleanup

osd/PG.cc: unify the call of checking whether lock is held

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14937 from SpandanKumarSahu/bug#17453

mgr: remove non-existent MDS daemons from FSMap

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

librbd: cleanup logging code under librbd/io

Signed-off-by: runsisi <runsisi@zte.com.cn>

tools: change compare_exchange_weak to compare_exchange_strong

On non x86 platforms without a dedicated compare-exchange instruction, std::compare_exchange_weak()
could fail.

Signed-off-by: Jesse Williamson <jwilliamson@suse.de>

Merge pull request #15016 from dachary/wip-19888-ceph-disk

ceph-disk: separate ceph-osd --check-needs-* logs

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #14854 from ceph/wip-reduce-smoke

qa/suites: Reduce fs combination tests for smoke, use bluestore

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14959 from liewegas/wip-crush-compat

mon,osd: add require_min_compat_client setting to enforce and clarify client compatibility

Reviewed-by: Loic Dachary <ldachary@redhat.com>

test/cli: fix cli tests

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: 'osd set-require-min-compat-client ...'

Signed-off-by: Sage Weil <sage@redhat.com>

qa/releases/luminous: set require_min_compat_client after upgrade

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites: set initial require_min_compat_client

For cases where we are selecting crush tunables beyond the default
min of hammer.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMap: add CRUSH_CHOOSEARGS feature bit

Signed-off-by: Sage Weil <sage@redhat.com>

crush/CrushWrapper: has_[incompat_]chooseargs()

We'll use this for feature bit compat checks with clients and osds.

Signed-off-by: Sage Weil <sage@redhat.com>

include/ceph_features: rename OSDMAP_REMAP -> OSDMAP_PG_UPMAP

Missed this with the remap -> upmap rename a few weeks back.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: prevent injection of crush map violating require_min_compat_client

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: drop allow_pg_upmap option; use require_min_compat_client instead

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: switch primary-{affinity,temp} checks to require_min_compat_client

If require_min_compat_client is set (we've finished luminous upgrade), use
that to enforce primary_temp and primary_affinity settings.  Otherwise,
use the legacy config option.

Drop the assert in update_from_paxos.

After luminous we can remove the config options entirely.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: set OSDMap require_min_compat_client on upgrade

Fill in this field once require_luminous is set.

Signed-off-by: Sage Weil <sage@redhat.com>

crush: move default tunables up to hammer

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: default require_min_compat_client to hammer for new clusters

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDmap: add require_min_compat_client field

This is a persistent record of the compatibility for old clients that
we will enforce.  Once set, the mon will eventually prevent newer features
of the OSDMap or CRUSH maps from being used if they violate this
requirement.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMap: get_min_compat_client()

Calculate the oldest client we allow to connect.  Provide both a release
name (e.g., firefly) and a version string (e.g., "0.80").

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15015 from smithfarm/wip-sphinx

build/ops: rpm: fix python-Sphinx package name for SUSE

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #14952 from tchaikov/wip-19813

mgr: do the shutdown in the right order

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14478 from theanalyst/doc/dev/ccache

doc: dev add a note about ccache

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14886 from badone/wip-async-osd-scrub-sleep

osd: Implement asynchronous scrub sleep

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #14985 from johnzzpcrystal/store_errno

msg/async/net_handler: errno should be stored before calling next function

Reviewed-by: Haomai Wang <haomai@xsky.com>

ceph-disk: separate ceph-osd --check-needs-* logs

It is using the OSD id zero but have nothing to do with OSD zero and
this is confusing to the user. The log themselves do not need to be kept
around and are stored in the run directory so that they can be disposed
of after reboot.

Fixes: http://tracker.ceph.com/issues/19888
Signed-off-by: Loic Dachary <loic@dachary.org>

build/ops: rpm: fix python-Sphinx package name for SUSE

This commit moves "BuildRequires: python-sphinx" down to the RH/CentOS/Fedora
distro conditional and adds a "BuildRequires: python-Sphinx" to the SUSE
conditional.

Signed-off-by: Jan Matejek <jmatejek@suse.com>
Signed-off-by: Nathan Cutler <ncutler@suse.com>

osd/PG.cc: unify the call of checking whether lock is held

Signed-off-by: Jin Cai <caijin.caij@alibaba-inc.com>

doc: dev add a note about ccache

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

mds: fix dir auth calculation in CDir::merge

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: properly cleanup export states

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

buffer, osd: add missing crc cache miss perf counter

This helps tracking if/how effective is the CRC cache in bufferlists.

Signed-off-by: Piotr Dałek <piotr.dalek@corp.ovh.com>

mds: update export targets even when not active

Problem caught test_migration_on_shutdown. Migration could not proceed because
export_targets were not being updated.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #14984 from dmick/wip-osd-dup

test/osd/osd-dup.sh: lower wb fd throttle limits

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14907 from ukernel/wip-19820

ceph-fuse: use user space permission check by default

common: Remove redundant includes

Fixes: http://tracker.ceph.com/issues/19883 (Partially)
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #14976 from trociny/wip-19871

rbd-nbd: relax size check for newer kernel versions

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #14656 from chardan/jfw-wip-halflife_atomic_t-rodan

rbd, librbd: migrate atomic_t to std::atomic

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #14925 from fangyuxiangGL/fix_unneccesary_write

rgw: don't do unneccesary write if buffer with zero length

Merge pull request #14990 from wjin/fix

osd: clean nonused work queue

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14962 from liewegas/wip-denc-dump

include/denc: support ENCODE_DUMP

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #14995 from SUSE/wip-mgr-rest-exit

pybind/mgr/rest: completely terminate cherrypy in shutdown

Reviewed-by: Kefu Chai <kchai@redhat.com>

include/denc: clean up ENCODE_DUMP a bit

- PATH_MAX for string
- ENCODE_DUMP -> ENCODE_DUMP_PATH
- some comments!

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #14999 from ivancich/wip-mod-git-subtree-README

doc: Re-word the warnings about using git subtrees.

doc: Re-word the warnings about using git subtrees.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

rbd_mirror: migrate std::atomic<unsigned> to std::atomic<bool>

Signed-off-by: Jesse Williamson <jwilliamson@suse.de>

librbd: migrate atomic_t to std::atomic<>

Signed-off-by: Jesse Williamson <jwilliamson@suse.de>

tools (rbd*): migrate atomic_t to std::atomic<>

Signed-off-by: Jesse Williamson <jwilliamson@suse.de>

mon/PGMap.cc: fix "osd_epochs" section of dump_basic

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

pybind/mgr/rest: completely terminate cherrypy in shutdown

cherrypy.engine.stop() doesn't actually completely kill cherrypy,
which means cherrypy.engine.block() never returns, so this module's
thread never completes cleanly.

Signed-off-by: Tim Serong <tserong@suse.com>

Merge pull request #14956 from liewegas/wip-bdev-ioc-lock

os/bluestore: fix use after free race with aio_wait

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Jianpeng Ma <jianpeng@intel.com>

osd: clean nonused work queue

it was restructured by 3cc48278bf0ee5c9535d04b60a661f988c50063b

Signed-off-by: Wei Jin <wjin.cn@gmail.com>

Merge pull request #14919 from tchaikov/wip-19825

mon: check is_shutdown() in timer callbacks

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14968 from TsaiJin/wip-lock-pg-when-scrub-available

osd: sched_scrub() lock pg only if all scrubbing conditions are fulfilled

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Pan Liu <liupan1111@gmail.com>

mds: check mdsdir against dest

Introduced by aebc1caee86b70f51d7d2d16f25f0ee9e4119f43. Found by Zheng.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mgr: do the shutdown in the right order

Fixes: http://tracker.ceph.com/issues/19813
Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr: do not use pointers for MgrStandby members

it's simpler and easier to manage the life cycle of them this way.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14970 from dachary/wip-crush-documentation

crush: update documentation for negative choose step

Reviewed-by: Loic Dachary <ldachary@redhat.com>

mds: handle aux subtree when splitting/merging dirfrag

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

msg/async/net_handler: errno should be stored before next calling

Signed-off-by: Zhou Zhengping <johnzzpcrystal@gmail.com>

mds: call maybe_export_pin on all fetched dirfrags

If the dirfrag is pinned to the current mds, it needs to be made into an aux
subtree.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: handle export pin on unjournaled directory

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: improve time handling for test_exports test

Also catches corner-case found by Zheng where an unjournaled directory will
cause export pinning to fail because it cannot be made a subtree until its
parent is stable.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #14979 from liewegas/wip-19874

mon: fix prime_pg_temp overrun
common/Finisher: fix uninitialized variable warning

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14958 from ovh/bp-fix-finisher-warning

common/Finisher: fix uninitialized variable warning
common/Finisher: fix uninitialized variable warning

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14793 from dmick/wip-prio-and-daemonperf

ceph: perfcounter priorities and daemonperf updates to use them

Merge pull request #14510 from liewegas/wip-bluefs-sync-write

os/bluestore/BlueFS: add bluefs_sync_write option

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #14305 from liewegas/wip-zipkin-partial

osd: zipkin tracing

messages/MCommand: fix type on decode

Wow, this has been broken since v0.38, but apparently
the message never made it into the object corpus so
we never noticed!

In reality the bug is harmless: decode_message() will
set_header which clobbers whatever version the default
ctor fills in, so this only affects ceph-dencoder's
test.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-object-corpus: kraken objects

Signed-off-by: Sage Weil <sage@redhat.com>

test/osd/osd-dup.sh: lower wb fd throttle limits

osd-dup.sh was losing OSDs to EMFILE, mostly on arm64 (not clear why).

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #14981 from cbodley/wip-rgw-bi-list-plain

cls/rgw: list_plain_entries() stops before bi_log entries
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

test/encoding/generate-corpus-objects: fix typo

Signed-off-by: Sage Weil <sage@redhat.com>

cls/rgw: list_plain_entries() stops before bi_log entries

list_plain_entries() was using encode_obj_versioned_data_key() to set
its end_key, which gives a prefix of BI_BUCKET_OBJ_INSTANCE_INDEX[=2]

that range between start_key and end_key would not only span the
BI_BUCKET_OBJS_INDEX[=0] prefixes, but BI_BUCKET_LOG_INDEX[=1] prefixes
as well. this can result in list_plain_entries() trying and failing to
decode a rgw_bi_log_entry as a rgw_bucket_dir_entry

Fixes: http://tracker.ceph.com/issues/19876
Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/encoding/generate-corpus-objects.sh: simplify object corpus population

Script that generates corpus objects.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/rgw/run-s3tests.sh: run s3-tests

This works out of the box with a vstart environment and

 RGW=1 ../src/vstart.sh -n -l
 PATH=bin:$PATH ../qa/workunits/rgw/run-s3tests.sh

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #14051 from SpandanKumarSahu/bug#17195

rbd: stop indefinite thread waiting in krbd udev handling

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

os/bluestore: remove useless IOContext::num_reading

If we are a syncrhonous read, we don't need this: we don't aio_wait for
sync reads.  If we are an aio_read, we are in the aio_running count anyway,
and there is also no purpose for this counter.

I'm a bit unsure about the NVME use of this counter; I switched it to use
num_running (pretty sure we aren't mixing reads and writes on a single
IOContext) *but* it might make more sense to switch to a private counter.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlockDevice: fix racy aio_{wake,wait}() on IOContext

Thread 1 (_do_read)                 Thread 2 (_aio_thread)
queues aio
ioc->aio_wait()
  locks ioc->lock
  num_waiting++
                                    finishes aios
                                    ioc->aio_wake
                                      if (num_waiting)
                                        lock ioc->lock (blocks)
  num_running == 0, continue
  ioc->unlock
                                        ioc->lock taken
do_read destroys ioc
                                        use after free, may block forever

The last bit of the race may vary; the key thing is that thread 2 is
taking a lock that thread 1 can destroy; thread 2 doesn't have it pinned
in memory.

Fix this by simplifying the aio_wake, aio_wait.  Since it is mutually
exclusive with a callback completion, we can avoid calling it at all when
a callback in present, and focus on keeping it simple.

Avoid use-after-free by making sure the last decrement happens under
the lock in the aio_wake/wait case.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/KernelDevice: remove weird aio_wake from sync read()

I'm not entirely sure why we were ever signaling the cond here; the read
is synchronous!

Signed-off-by: Sage Weil <sage@redhat.com>

CMakeLists.txt: WITH_BLKIN off for now

We are unconditionally linking lttng-ust, which make selinux complain. We
should either

 - fix selinux rules and unconditionally link
 - dlopen at runtime based on an option (like we do for the current
   tracepoints)
 - ???

Signed-off-by: Sage Weil <sage@redhat.com>

blkin: fix test

Signed-off-by: Sage Weil <sage@redhat.com>

blkin: fix cmake compile options

Signed-off-by: Sage Weil <sage@redhat.com>

blkin: remove installed files

Signed-off-by: Sage Weil <sage@redhat.com>

blkin: fix -fPIC option

Signed-off-by: Sage Weil <sage@redhat.com>