osd/PGMonitor: always update pgmap with latest osdmap

without this change, we always update the pgmap with the latest osdmap in
PGMonitor::check_osd_map(), but assign pending_inc.osdmap_epoch with the
epoch passed to PGMonitor::check_osd_map(), even if the osdmap could be
be much newer than "epoch". for example, we could end up with an
MOSDPGCreate with an ancient epoch, but carrying some creating pgs whose
pools do not exist in osdmap of that ancient epoch yet. this happens
when RetryCheckOSDMap is involved.

Fixes: http://tracker.ceph.com/issues/19398
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14726 from batrick/i19583

mds: change_attr++ and set ctime for set_vxattr

Reviewed-by: Yan, Zheng <zyan@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #14652 from jcsp/wip-18816

mds: remove "mds log" config option

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #14636 from jcsp/wip-19691

osdc: remove journaler_allow_split_entries option

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #14317 from ukernel/wip-19306

client: make seeky readdir more efficiency

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #12935 from stiopaa1/17855_evictClient

mds/Server.cc: Don't evict a slow client if...

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #14388 from shashalu/fix-create-bucket

rgw: using the same bucket num_shards as master zg when create bucket in secondary zg

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14479 from swamireddy/v12.0.1_credits

mailmap: V12.0.1 credits

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #14611 from liewegas/wip-mon-warn-cleanup

mon/PGMonitor: clean up min/max span warning

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #14741 from trociny/wip-19405

test/rbd_mirror: race in TestMockLeaderWatcher.AcquireError

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12792 from david-z/wip-mds-use-tcmalloc

mds: remove boost::pool usage and use tcmalloc directly

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #14575 from jcsp/wip-cephfs-suites

qa: Tidy up fs/ suite

Reviewed-by: Yan, Zheng <zyan@redhat.com>

qa: switch off btrfs for multimds and kcephfs

Signed-off-by: John Spray <john.spray@redhat.com>

qa: update links for modified fs subsuite names

Signed-off-by: John Spray <john.spray@redhat.com>

qa/tasks/cephfs: clean up mount point setup

Previously were sometimes trying to maintain a mounted
client across a filesystem destroy/create.

Signed-off-by: John Spray <john.spray@redhat.com>

qa/suites: don't use btrfs for cephfs testing

This change happened a while back, but it got rolled back
when the generic objectstore/ dir had its filestore
entry split out into xfs and btrfs in 208675af.

Signed-off-by: John Spray <john.spray@redhat.com>

qa: remove unused fs/ dir

The objectstore/ dir is now selecting the backing fs.

Signed-off-by: John Spray <john.spray@redhat.com>

qa/suites/fs: more meaningful names for sub suites

The "recovery" sub suite was originally tests for
client/mds recovery in certain failure cases, it has
since grown to include lots of unit testing of
various features using CephFSTestCase.

The "basic" suite is now specifically just running workloads
now that I've moved out the smaller functional tests.

Signed-off-by: John Spray <john.spray@redhat.com>

qa/suites/fs: put simple functional tests together

Most of what's in basic/ is "workload" type testing
(i.e. a simple cluster cluster configuration and then
running a script inside the client), which gets
permuted in various ways.  Move the simpler
functional tests out with the others like themselves.

Signed-off-by: John Spray <john.spray@redhat.com>

qa: don't permute object store for functional tests

These are unit tests for specific CephFS functionality,
it is gratuitous to repeat them with different underlying
RADOS object stores.

We retain coverage of XFS vs. bluestore in the workload tests.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #14018 from jcsp/wip-17939

client: getattr before returning quota/layout xattrs

Reviewed-by: Yan, Zheng <zyan@redhat.com>

mailmap: shiqi name normalization

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>

mds: remove boost::pool usage and use tcmalloc directly

Signed-off-by: Zhi Zhang <zhangz.david@outlook.com>

Merge pull request #14729 from joscollin/wip-cleanup-osdc-comparision

osdc: silence warning from `-Wsign-compare`

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14711 from jcsp/wip-19738

mon: don't call propose_pending in prepare_update()

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14740 from petermaloney/master

doc: fix link that pointed to a nonexistent file

Reviewed-by: Kefu Chai <kchai@redhat.com>

osdc: warning: comparison between signed and unsigned integer expressions

The following warning appears during make. Fixed the review comments too.
[ 28%] Building CXX object src/mds/CMakeFiles/mds.dir/__/osdc/Journaler.cc.o
./src/osdc/Journaler.cc: In member function ‘void Journaler::_do_flush(unsigned int)’:
./src/osdc/Journaler.cc:643:16: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
     if (newlen < len) {
         ~~~~~~~^~~~~

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #14705 from joscollin/wip-googletest-warning

test: warning: comparison between signed and unsigned integer expressions

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

rgw: using the same bucket num_shards as master zg when create bucket in secondary zg

create bucket in secondary zonegroup will forward to master. The master may have different num_shards option.
So when create bucket in local, should use master's num_shards instead of local num_shards option.

Fixes: http://tracker.ceph.com/issues/19745
Signed-off-by: Shasha Lu <lu.shasha@eisoo.com>

Merge pull request #13898 from songbaisen/m5

osd: check fsid is normal before osd mkfs

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10893 from yonghengdexin735/wip-zzz-1

rados: check for negative return value of rados_create_with_context() as its comment put

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14442 from liewegas/wip-midkraken-upgrade

mon/OSDMonitor: tolerate upgrade from post-kraken dev cluster

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/rbd_mirror: race in TestMockLeaderWatcher.AcquireError

Due to handle_get_locker is executed in async context, shut_down could
start before handle_get_locker.

Fixes: http://tracker.ceph.com/issues/19405
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

doc: fix link that pointed to a nonexistent file

commit d788bae60dac374dcf5c7a30fca04426cbee660a renamed config.rst to config-fcgi.rst

Reported-by: fusl in #ceph on irc.oftc.net
Signed-off-by: Peter Maloney <peter.maloney@brockmann-consult.de>

Merge pull request #14712 from dillaman/wip-rbd-mirror-watch

rbd-mirror: remove tracking of image names from pool watcher

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #14722 from cbodley/wip-19746

radosgw-admin: warn that 'realm rename' does not update other clusters
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #14338 from aclamk/fix-fio-ceph-objectstore-lifetime

fio_ceph_objectstore: fixes improper write request data lifetime

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

mds/Server.cc: Don't evict a slow client if...

... it's the only client

Fixes: http://tracker.ceph.com/issues/17855
Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #14335 from hrchu/radosgw-admin-api-subuser

doc: rgw: Clean up create subuser parameters

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14707 from wjwithagen/wip-wjw-freebsd-kqueue-logging

msg: Increase loglevels on some messages

Reviewed-by: Haomai Wang <haomai@xsky.com>

mds: change_attr++ and set ctime for set_vxattr

Fixes: http://tracker.ceph.com/issues/19583
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #9853 from dx9/wip-alpine-dev-support

alpine: add alpine linux dev support

Merge pull request #14189 from tchaikov/wip-mgr-py-stderr

mgr: redirect python stdout,stderr to ceph log

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #14559 from liewegas/wip-pg-map

mon: move 'pg map' to OSDMonitor

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14583 from joscollin/wip-cleanup-osdmap-sections

osd: renamed the new vector name in OSDMap::build_simple_crush_map_from_conf

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14721 from dmick/wip-crypto-arm

CMakeLists.txt: don't do crypto/isa-l if not Intel

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14501 from mdw-at-linuxbox/wip-rgw-rvk

rgw: swift: disable revocation thread if sleep == 0

radosgw-admin: warn that 'realm rename' does not update other clusters

Fixes: http://tracker.ceph.com/issues/19746
Signed-off-by: Casey Bodley <cbodley@redhat.com>

CMakeLists.txt: don't do crypto/isa-l if not Intel

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #14716 from tchaikov/wip-mgr-pystring-leak

mgr: release allocated PyString

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #14720 from liewegas/wip-19591

mgr: fix session leak

Reviewed-by: Kefu Chai <kchai@redhat.com>

vstart.sh: add --valgrind_args

To do leak checking, you can do

 vstart.sh ... \
--valgrind_mgr memcheck \
--valgrind_args '--leak-check=full --show-reachable=yes'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/MgrSession: allow ref tracing, and fix ref leak

The first RefCountedObj ctor arg is a cct, not a ref count.

Fixes: http://tracker.ceph.com/issues/19591
Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: add --valgrind_mgr

Signed-off-by: Sage Weil <sage@redhat.com>

mailmap: Yuri Weinstein name normalization

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>

mailmap: Nathan Cutler name normalization

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>

mailmap: Ian Colle affiliation

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>

mailmap: Kyr Shatskyy affiliation

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>

mailmap: haodong tang affiliation

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>

mailmap: craigchi affiliation

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>

mailmap: Enming Zhang affiliation

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>

mailmap: Saumay Agrawal affiliation

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>

mailmap: Tahia Khan affiliation

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>

mailmap: Taeksang Kim affiliation

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>

mailmap: hechuang affiliation

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>

mailmap: Tang Wenjun affiliation

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>

mailmap: Jos Collin affiliation

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>

mailmap: Guillaume Abrioux affiliation

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>

mailmap: Nikita Gerasimov affiliation

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>

mailmap: DanielBar-On affiliation

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>

mailmap: wei xiao affiliation

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>

mailmap: Ziye Yang affiliation

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>

mailmap: Michael Eischer affiliation

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>

Merge pull request #12272 from kylinstorage/wip-writesame

osd: use append(bufferlist &) to avoid unnecessary copy

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11684 from runsisi/wip-fix-asok-mode

common/admin_socket: add config for admin socket permission bits

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13019 from wonzhq/wip-long-io

osd: support dumping long ops

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14358 from majianpeng/osd-remove-modsubop

osd/ReplicatedBackend: remove MOSDSubOp cruft from repop_applied

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14268 from hrchu/swift/java

doc: rgw: Rewrite Java swift examples

Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10384 from wjwithagen/wip-wjw-reduce-noise

common/ceph_context.cc: Use CEPH_DEV to reduce logfile noise

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14693 from mogeb/init-ceph-libs

init-ceph: add ceph libraries path to environment

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12898 from XinzeChi/wip-interval-return

common/interval_set: return int64_t for size()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14232 from jcsp/wip-19412

mgr: fix python module teardown & add tests

Reviewed-by: Kefu Chai <kchai@redhat.com>

mon/OSDMonitor: cleanup: use range-based loop

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph_manager: use 'pg map' for get_pg_{primary,replica}

Pulling this out of the 'pg dump' heap is inefficient.
Also, pg dump data comes from the mgr and may be stale.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #14584 from tchaikov/wip-19631

qa/suites: Revert "qa/suites: add mon-reweight-min-pgs-per-osd = 4"

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14504 from tchaikov/wip-another-mgr-command

mon,mgr: trim osdmap without the help of pgmap

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14055 from hrchu/radosgw-admin-api-user-caps

doc: correct and improve add user capability section

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #13956 from hrchu/rgw-admin-doc

Indicate how to add multiple admin capbabilies

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14372 from hrchu/rgwadmin-admin-doc-usage

doc: rgw: correct get usage parameter default value

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr: release allocated PyString

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14547 from smithfarm/wip-create-pool

rgw: cleanup: fix variable name in RGWRados::create_pool() declaration

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14697 from ceph/wip-rgw-upgrade-s3tests-encryption

qa/rgw: don't scan radosgw logs for encryption keys on jewel upgrade test

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14424 from dachary/wip-jerasure-sync

erasure-code: sync jerasure/gf-complete submodules

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #14558 from liewegas/wip-18293

osd: fix occasional MOSDMap leak

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14633 from SUSE/wip-yasm-wrapper-strip-E-flag

yasm-wrapper: strip -E (stops ccache trashing source files)

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14640 from ovh/bp-fiemap-burden

osd, os: reduce fiemap burden

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14644 from liewegas/wip-rados-df-json

rados: out json 'df' values as numbers, not strings

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #14645 from liewegas/wip-mgr-mon

mon/MgrMonitor: only propose if we updated

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14648 from gregsfortytwo/wip-17916-master

osd: pglog: with config, don't assert in the presence of stale diverg…

Reviewed-by: Sage Weil <sage@redhat.com>

mon: don't call propose_pending in dispatch

This was happening indirectly because the command handling
code was calling tick() as a shortcut to "promote something"
and tick calls propose_pending because.

Fixes: http://tracker.ceph.com/issues/19738
Signed-off-by: John Spray <john.spray@redhat.com>