tools: monmaptool: support monmap features

Adds three new options:

  --feature-list     will list monmap features, as well as available
                     features.
  --feature-set      sets a feature on the monmap
  --feature-unset    unsets a feature on the monmap

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

test/mon: few more cases for mon_feature_t

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

mon: Monitor: mon debug cli via admin socket

Purpose: allow a dev to test certain portions of the monitors on-the-fly
that would otherwise be nearly impossible without requiring shutting
them off.

Description:

  Currently we are adding support to debug monitors on-the-fly via the
  ceph tool cli, via the admin socket.

  An initial attempt was made to make this interface available via
  over-the-network ceph cli, but that would require a quorum to exist,
  or significant changes to be made to the handling of commands to allow
  this interface to be an exception to the rule.

  Therefore, this interface will live on the admin sockets for the time
  being; potentially forever, because the jury is still out on whether
  there's added value making this available via any other medium that's
  not the admin socket.

  The work means to test the newly added monmap features. This new
  interface will allow one to set, unset and list supported features.

  This will make it easier to implement functional tests for the
  features themselves, for quorum features that are expected to be set
  on-quorum, as well as to test upgrade paths from version A to version
  B.

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

mon: MonMap: move functions from header to source file

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

mon: MonMap: add a new test instance

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

mon: MonMap: mon addr and all now held by mon_info_t

Instead of keeping several maps mapping back and forth from string to
entity_addr_t for each monitor, have the monitor's address and name in
purpose-specific class, mon_info_t, and everything references it
instead.

Although the benefits may not be obvious right now, it will allow us to
add other ip addresses for the monitors a bit more easily, instead of
having to kludge around the existing structures.

We also keep compatibility with older versions by maintaining the
traditional 'mon_addr' map. This is a logical change inside the monitor,
so we can easily accomodate older monitors without having to rely on
quorum features; just needs a bit of care and foo.

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

krbd.cc: don't rely on MonMap internal members

Use the public interface instead. That's stable and not as prone to
change.

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

mon: don't allow downgrading once we go the kraken way

Otherwise monitors - downgraded monitors - will not honour the monmap
features required to form quorum.

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

mon: Elector: ignore mon without required mon features

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

mon: Monitor: dump features on 'mon_status'

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

mon: Monitor: get required features from monmap

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

mon: MonmapMonitor: set monmap features according to quorum

Once an election finishes, make sure the monmap reflects the quorum's
mon-specific features.

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

mon: rename quorum_features to quorum_con_features

Because now we have quorum_mon_features too.

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

mon: Elector: support mon-specific features

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

messages: MMonElection: support mon-specific features

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

test/mon: add test_mon_types

Currently testing solely mon_feature_t.

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

mon/mon_types.h: add 'mon_feature_t' for mon-specific features

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

common/bit_str: Helper functions to output bitmask's flags

These functions will take a given function 'f' as parameter, and will
call 'f' for each bit that is set on a provided bitmask. The
responsibility of printing the appropriate flag name falls on 'f'.

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

mon: Monitor: show probe's required_features instead

For some reason, we were logging to derr with our internally established
'required_features', instead of the value from the MMonProbe message
stating what are the actual required_features.

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

mon: Monitor: dedup code on {win,lose}_election()

refactor out the loop that calls 'election_finished()' on all paxos
services.

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

mon: Monitor: drop useless 'todo' comment

I can't recall if it caused trouble or not, or what 'it' was. But
there's no good reason to keep this comment around.

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #11397 from dzafman/wip-scrub-misc

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11677 from ceph/wip-rocksdb-no-march-native

rocksdb: do not build with --march=native

Reviewed-by: Bassam Tabbara <bassam.tabbara@quantum.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #11678 from tchaikov/wip-silence-rbd-warning

librbd: silence the unused variable warning

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: silence the unused variable warning

Signed-off-by: Kefu Chai <kchai@redhat.com>

rocksdb: do not build with --march=native

we can not assume that building host has the same arch (or the same cpu
capability) with the target machine.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11663 from stiopaa1/mds_mdslog_addConstToMemFunc

mds/MDLog: add const to member functions

Merge pull request #11668 from Enproduktion/master

doc: Fixed mapping error in legacy mds command

Merge pull request #11446 from ukernel/wip-cephfs-scan-links

tools/cephfs: add scan_links command which fixes linkages errors

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11609 from stiopaa1/mds_snaprealm_unneededuseofcount

mds/SnapRealm: remove unneeded use of count

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11625 from stiopaa1/mds_damageTable_removeUnneededCount

mds/DamageTable.cc: remove unneeded use of count

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #9304 from dzafman/wip-13937

osd: handle EC recovery read errors

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11645 from dillaman/wip-librbd-discard-after-write

test: new librbd discard after write test case

Merge pull request #11670 from stiopaa1/mds_mutation_addConstToMemFun

mds/Mutation.h: add const to member functions

Merge pull request #11608 from atheism/mds-duplicated-check

client: Client.cc: remove duplicated op type checking against CEPH_MD…

client: Client.cc: correct the op type checking for CEPH_MDS_OP_CREATE

Signed-off-by: Weibing Zhang <zhangweibing@unitedstack.com>

global: Always log loading erasure code for tests

Fix for broken test-erasure-code.sh and test-erasure-eio.sh

Signed-off-by: David Zafman <dzafman@redhat.com>

test.sh: Make check for flags more robust

Low space broke test, saw "flags nearfull,pauserd,pausewr...."

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Remove extra objectstore_tool call which causes a recovery

Caused by: 70e000a9a42c50eda48f5d3b9e432ffc3a70f75b

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Handle object removals in a non-racey way

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Fix hang on unfound object after mark_unfound_lost is done

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Handle recovery read errors

Fixes: http://tracker.ceph.com/issues/13937
Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Fix log messages

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: CLEANUP: Remove unused pending_read member

Remove unused struct

Signed-off-by: David Zafman <dzafman@redhat.com>

Revert "test: Disable tests due to recovery race"

This reverts commit 5bc55338f5a1645bc651811fae2f89ad855ff86e.

Conflicts:
src/test/Makefile.am (no longer exists)
src/test/erasure-code/Makefile.am (no longer exists)

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #11582 from batrick/ceph-dumpable

core: set dumpable flag after setuid

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

mds/Mutation.h: add const to member functions

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

doc: Fixed mapping error in legacy mds command

Signed-off-by: Malte Fiala <maltefiala@produktion.io>

Merge pull request #11666 from tchaikov/wip-cmake

cmake: remove include/Makefile.am

cmake: remove include/Makefile.am

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11638 from tchaikov/wip-clang

include/enc: make clang happy

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11658 from hjwsm1989/doc

doc: change the osd_max_backfills default to 1

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mds/MDLog: add const to member functions

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #10887 from michaelsevilla/mantle

fs: Mantle: A Programmable Metadata Load Balancer

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11641 from vshankar/rbd-data-pool-assert

librbd: possible assert failure creating image when using data pool

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

common/cmdparse,msg/async: remove unnecessary std::move()

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11570 from tchaikov/wip-revise-SubmittingPatches

doc: reformat SubmittingPatches with more rst syntax

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #11630 from cbodley/wip-rgw-curl-upgrade

rgw: remove suggestion to upgrade libcurl
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #11385 from bassamtabbara/wip-rocksdb-external-project

cmake: use external project for rocksdb

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: remove ending period in section titles

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: change the osd_max_backfills default to 1
Fixes: http://tracker.ceph.com/issues/17701
Signed-off-by: huangjun <hjwsm1989@gmail.com>

Merge pull request #11653 from ktdreyer/doc-epel-instructions

doc: standardize EPEL instructions

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: use external project for rocksdb

This commit makes rocksdb an external cmake project. This ensures
that rocksdb is no longer built from src/rocksdb, and that make clean
works correctly.

Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>

pickup new rocksdb with support for linux cmake

This picksup a new version of rocksdb with support for
linux cmake.

Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>

Merge pull request #11619 from dachary/wip-17662-ceph-disk

ceph-disk: allow using a regular file as a journal

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11652 from ktdreyer/doc-mgr-syntax

doc: fix mgr literal block rST syntax

Reviewed-by: Kefu Chai <kchai@redhat.com>

librbd: assert failure when using data pool

This fixes a silly assert that's hit during image creation
(cli/api) when the data pool specified is same as the pool
specified by -p/--pool option (or the default).

Signed-off-by: Venky Shankar <vshankar@redhat.com>

doc: standardize EPEL instructions

Prior to this change, the documentation pages contained different ways
to enable EPEL. Pick a simple, secure (https) way and standardize on
that.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

doc: fix mgr literal block rST syntax

Prior to this change, there were double-colon ("::") characters in the
HTML docs to indicate literal blocks.

The proceeding paragraphs were not pre-formatted because there was no
newline between the double-colon and the to-be-pre-formatted paragraph.

Move the "::" characters inline to simplify the syntax and make Sphinx
properly interpret these texts as literal blocks.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #11594 from cbodley/wip-rgw-init-bucket-sync

rgw: fix for passing temporary in InitBucketSyncStatus

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #11505 from cbodley/wip-17571

rgw: clean up RGWShardedOmapCRManager on early return

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #11504 from cbodley/wip-17568

rgw: RGWSimpleRadosReadCR tolerates empty reads

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

mantle: re-design and fixes
- fix legacy encoding in mds map and add balancer to dumps
- fix blocking rados read and remove temporary files
- fix beacon message spamming

Signed-off-by: Michael Sevilla <mikesevilla3@gmail.com>

mantle: store balancer in RADOS, balancer version in MDSMap
- add docs and sample balancer (greedy-spill)

Signed-off-by: Michael Sevilla <mikesevilla3@gmail.com>

mantle: write balancer policies in Lua

Introduces Mantle, a programmable metadata load balancer. Policies for making
migration decisions are written in Lua but the Migrator and Balancer modules
still do fragmentation and migration. If the Lua balancer fails, control falls
back to the original balancer implementation.

Signed-off-by: Michael Sevilla <mikesevilla3@gmail.com>

Merge pull request #11352 from dachary/wip-17526-rados-bench

doc: document hostname constraints for rados bench

reviewed-by: John Wilkins <jowilkin@redhat.com>

Merge pull request #11602 from dillaman/wip-17665

doc: clarify QEMU rbd cache settings

reviewed-by: John Wilkins <jowilkin@redhat.com>

Merge pull request #9825 from liewegas/wip-addrvec

msg: add entity_addr_t types; add new entity_addrvec_t type

Reviewed-by: Kefu Chai <kchai@redhat.com>

test: new librbd discard after write test case

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11517 from ukernel/wip-17562

Fix #17562 (backtrace check fails when scrubbing directory created by fsstress)

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11593 from ukernel/wip-17275

osdc/ObjectCacher: wake up dirty stat waiters after removing buffers

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11604 from stiopaa1/mds_inotable_addOverride

mds/InoTable.h: add override to virtual functs

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11613 from stiopaa1/mds_cdir_removeUnneededUseofCount

mds/CDir.cc: remove unneeded use of count

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11611 from batrick/i17670

mds: check if down mds is known

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11191 from renhwztetecs/renhw-wip-add-libcephfs

libcephfs: add ceph_fsetattr&&ceph_lchmod&&ceph_lutime

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11560 from ukernel/wip-mds-context-wrapping

MDS: reduce usage of context wrapper

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11345 from jcsp/wip-mgr-assert

messages: fix out of range assertion

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/encoding: fix the build with clang

put the overloaded operator<<() into namespace std, so clang's name
resolution is able to find it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add definitions for glibc

"-D_GNU_SOURCE -D_FILE_OFFSET_BITS=64" is not for gcc, it's for glibc
actually. so enable it on LINUX.

Signed-off-by: Kefu Chai <kchai@redhat.com>

include/denc: cast (possible) ints to bool in enable_if

This makes clang happy.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11632 from VictorDenisov/group_pool_fix

rbd: fix parsing of group and image specific pools

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd: fix parsing of group and image specific pools

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

Merge pull request #11036 from tomy-ac/wip-tc-rados_append

Added append functionality to rados tool.

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #11221 from stiopaa1/osd_osdmap_removeUseOfCount

osd/OSDMap.cc: remove unneeded use of count

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11388 from ukernel/wip-17177

os/ObjectStore: properly clear object map when replaying OP_REMOVE

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

rgw: remove suggestion to upgrade libcurl

Reported-by: Ken Dreyer <kdreyer@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>

mds/DamageTable.cc: remove unneeded use of count

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #11143 from kylinstorage/wip-fix-write-hang

osd: fix rados write op hang

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #11532 from javeme/run_in_thread-return

pybind: avoid "exception 'int' object is not iterable"

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11610 from liewegas/wip-mon-forward-features

messages/MForward: reencode forwarded message if target has differing features

Reviewed-by: Kefu Chai <kchai@redhat.com>