Merge pull request #19480 from smithfarm/wip-22407-luminous

luminous: cephfs: client: implement delegation support in userland cephfs

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #19775 from ukernel/luminous-19578

luminous: mds: track dirty dentries in separate list

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #19827 from pdvian/wip-22587-luminous

luminous: mds: reduce debugging level for balancer messages

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Amit Kumar <amitkuma@redhat.com>

Merge pull request #20082 from smithfarm/wip-21525-luminous

luminous: cephfs: client: dual client segfault with racing ceph_shutdown

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20085 from pdvian/wip-22763-luminous

luminous: cephfs-journal-tool: add "set pool_id" option

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20086 from pdvian/wip-22765-luminous

luminous: client: avoid recursive lock in ll_get_vino

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #18019 from ukernel/luminous-21091

luminous: mds: fix StrayManager::truncate()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #19485 from dillaman/wip-21788-luminous

luminous: librbd: refresh image after applying new/removing old metadata

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #18834 from shinobu-x/wip-22073-luminous

luminous: [api] compare-and-write methods not properly advertised

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19058 from shinobu-x/wip-22190-luminous

luminous: class rbd.Image discard----OSError: [errno 2147483648] error discarding region

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19066 from pdvian/wip-22172-luminous

luminous: [rbd-nbd] Fedora does not register resize events

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Amit Kumar <amitkuma@redhat.com>

Merge pull request #19107 from pdvian/wip-22208-luminous

luminous: rbd: disk usage on empty pool no longer returns an error message

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19305 from pdvian/wip-21700-luminous

luminous: rbd-mirror: Allow a different data-pool to be used on the secondary cluster

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19447 from smithfarm/wip-21646-luminous

luminous: librbd: Image-meta should be dynamically refreshed

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19479 from smithfarm/wip-22376-luminous

luminous: rbd: Python RBD metadata_get does not work

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Amit Kumar <amitkuma@redhat.com>

Merge pull request #19484 from dillaman/wip-21644-luminous

luminous: rbd-mirror: sync image metadata when transfering remote image

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #19503 from shinobu-x/wip-22395-luminous

luminous: librbd: cannot clone all image-metas if we have more than 64 key/value pairs

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19504 from shinobu-x/wip-22393-luminous

luminous: librbd: cannot copy all image-metas if we have more than 64 key/value pairs

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19554 from shinobu-x/wip-22454-luminous

 luminous: cluster resource agent ocf:ceph:rbd - wrong permissions

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19625 from shinobu-x/wip-22497-luminous

 luminous: [rbd-mirror] new pools might not be detected

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19800 from smithfarm/wip-21641-luminous

luminous: rbd ls -l crashes with SIGABRT

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #19802 from smithfarm/wip-21690-luminous

luminous: tests: rbd_mirror_helpers.sh request_resync_image function saves image id to wrong variable

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19853 from pdvian/wip-22593-luminous

luminous: rbd: librbd: filter out potential race with image rename

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18722 from shinobu-x/wip-luminous-21946

luminous: mds: set PRIO_USEFUL on num_sessions counter

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #18730 from shinobu-x/wip-luminous-22030

luminous: List of filesystems does not get refreshed after a filesystem deletion

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #18754 from shinobu-x/wip-21636-luminous

luminous: ceph-monstore-tool --readable mode doesn't understand FSMap, MgrMap

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20053 from dillaman/wip-22740-luminous

luminous: qa/suites/upgrade: disable broken kraken upgrade cls_rbd test

Reviewed-By: Yuri Weinstein <yuriw@redhat.com>

Merge pull request #18564 from kmroz/wip-21688-luminous

luminous: Possible deadlock in 'list_children' when refresh is required

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18698 from shinobu-x/wip-luminous-21808

luminous: rbd: fix crash during map

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19038 from pdvian/wip-22183-luminous

luminous: rgw: set sync_from_all as true when no value is seen

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19446 from smithfarm/wip-22404-luminous

luminous: doc: crush_ruleset is invalid command in luminous

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #18747 from shinobu-x/wip-21794-luminous

luminous: backoff causes out of order op

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #20087 from dzafman/wip-misc-luminous

qa/standalone: Minor fixes for ceph_objectstore_tool.py test

Reviewed-by: Kefu Chai <kchai@redhat.com>

test: Fix ceph-objectstore-tool usage check

Caused by: c7b7a1f04f78fa62890c567d0ca53874c8d75eb7

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 0364ae104afc15effadfe968b5f60221704d933c)

test: ceph_objectstore_tool.py bad --filestore option to vstart

Luminous doesn't have --filestore option

Caused by: fd7d53bacb10d5b2094f1f579c96899d8a0a4b4f

Signed-off-by: David Zafman <dzafman@redhat.com>

client: avoid recursive lock in ll_get_vino

Fixes: http://tracker.ceph.com/issues/22629
Signed-off-by: dongdong tao <tdd21151186@gmail.com>
(cherry picked from commit fdfbe40c3f16c14dc9c3352f3eea66d34138ee9d)

cephfs-journal-tool: add usage help info for header set <field>

Signed-off-by: dongdong tao <tdd21151186@gmail.com>
(cherry picked from commit 2b06bff950a78b15aca083b20eaa812b4214e8d4)

mds: add error log info

Signed-off-by: dongdong tao <tdd21151186@gmail.com>
(cherry picked from commit a7c2ecd6391d22c5312a9e568ff742cff252749e)

cephfs-journal-tool: add "set pool_id" option

Fixes: http://tracker.ceph.com/issues/22631
Signed-off-by: dongdong tao <tdd21151186@gmail.com>
(cherry picked from commit b3a5e313117028b23133f3e0d72e0b6b7159a178)

Merge pull request #18742 from shinobu-x/wip-21924-luminous

luminous: ceph_test_objectstore fails ObjectStore/StoreTest.Synthetic/1 (filestore) buffer content mismatch

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #18744 from shinobu-x/wip-21922-luminous

luminous: Objecter::C_ObjectOperation_sparse_read throws/catches exceptions on -ENOENT

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #18745 from shinobu-x/wip-21921-luminous

luminous: Objecter::_send_op unnecessarily constructs costly hobject_t

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

test: make the LibCephFS.ShutdownRacer test even more thrashy

Have each thread do the startup and shutdown in a loop for a specified
number of times.

Tracker: http://tracker.ceph.com/issues/21512
Signed-off-by: Jeff Layton <jlayton@redhat.com>
(cherry picked from commit f877e365a42f8f34af6fc3382593ef09101a50d0)

lockdep: free_ids and lock_ref hashes must be truly global

It's possible for the teardown of g_lockdep_ceph_ctx to occur, followed
by a new context being registered as the lockdep context. When that
occurs, we can end up reusing lock id's that were previously handed out
to consumers. We need for those IDs to be persistent across lockdep
enablement and disablement.

Make both the free_ids table, and the lock_refs map persistent across
lockdep_unregister_ceph_context and lockdep_register_ceph_context cycles.
Entries in those tables will only be deleted by the destruction of the
associated mutex.

When lockdep_unregister is called, do the refcounting like we normally
would, but only clear out the state when the lockid is registered
in the lock_names hash.

Finally, we do still need to handle the case where g_lockdep has gone
false even when there are outstanding references after the decrement.
Only log the message if that's not the case.

With this, we can deal with the case of multiple clients enabling and
disabling lockdep in an unsynchronized way.

Tracker: http://tracker.ceph.com/issues/21512
Signed-off-by: Jeff Layton <jlayton@redhat.com>
(cherry picked from commit 82fe4e94bcb706434f9215bc8405ff60770cf14f)

common: add a clear_g_str_vec() function to clear g_str_vec

Prefix str_vec and str_vec_lock with "g_" to make it clear that they are
truly global values. Add a new clear_g_str_vec function to allow it to
be explicitly cleaned out by callers that need that functionality
(mostly testcase for now).

Tracker: http://tracker.ceph.com/issues/21512
Signed-off-by: Jeff Layton <jlayton@redhat.com>
(cherry picked from commit f72ace0a5666db49bb390ff4ec6808e6e55a779f)

Merge pull request #18746 from shinobu-x/wip-21916-luminous

luminous: msg/async/AsyncConnection.cc: 1835: FAILED assert(state == STATE_CLOSED)

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

common: make it safe to call env_to_vec multiple times

After it has been called once and we have outstanding CephContexts with
pointers into str_vec, we can't call get_str_vec on it again.

Add a static local mutex to protect access to str_vec.

Tracker: http://tracker.ceph.com/issues/21512
Signed-off-by: Jeff Layton <jlayton@redhat.com>
(cherry picked from commit 90e898de5f4b8d22f1a6d0e2aedf9e8c50cf72d5)

lockdep: fix Mutex tests to disable lockdep properly

...and make g_lockdep a bool.

Signed-off-by: Jeff Layton <jlayton@redhat.com>
(cherry picked from commit 0cd0bd778a6149d3e2fe657060e223be3675aed7)

client: test shutdown race

Spawn threads that bring up a bunch of ceph_mounts with individual
CephContext objects, and then tear them down in parallel.

Tracker: http://tracker.ceph.com/issues/20988
Signed-off-by: Jeff Layton <jlayton@redhat.com>
(cherry picked from commit 8252f3168a1bc90add7f7515c549e9eecffa54b7)

client: fix signed/unsigned comparison compiler warning

The build says:

src/client/Client.cc: In member function ‘void Client::trim_caps(MetaSession*, int)’:
src/client/Client.cc:4121:22: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
   if (s->caps.size() > max)
       ~~~~~~~~~~~~~~~^~~~~

Signed-off-by: Jeff Layton <jlayton@redhat.com>
(cherry picked from commit e057b6770b5545f327990cbaf17e7a391e3e8a50)

lockdep: fix races with concurrent lockdep teardown

If the cct is unregistered while other threads are flogging mutexes,
then we can hit all sorts of bugs. Ensure that we handle that
situation sanely, by checking that g_lockdep is still set after
we take the lockdep_mutex.

Also, remove an assertion from lockdep_unregister, and just turn it into
an immediate return. It's possible to have a call to
lockdep_unregister_ceph_context, and then a call to
lockdep_register_ceph_context while a mutex is being held by another
task.

In that case, it's possible the lock does not exist in the map
when we go to unregister it. That's not a bug though, just a natural
consequence of that series of actions.

Tracker: http://tracker.ceph.com/issues/20988
Signed-off-by: Jeff Layton <jlayton@redhat.com>
(cherry picked from commit 75f41a95782a7ee83a243d91963e8d591402f8a6)

lockdep: don't take lockdep_mutex twice for new lock registrations

We can do it under the same mutex, which should be more efficient.

Signed-off-by: Jeff Layton <jlayton@redhat.com>
(cherry picked from commit 01863bb6fb62ea89aa3e21e43bf4dc4f3da9cfcb)

Merge pull request #18749 from shinobu-x/wip-21785-luminous

luminous: OSDMap cache assert on shutdown

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #18753 from shinobu-x/wip-21697-luminous

luminous: OSDService::recovery_need_sleep read+updated without locking

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #18715 from shinobu-x/wip-luminous-21964

luminous: ceph_test_cls_log failures related to cls_cxx_subop_version()

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18732 from shinobu-x/wip-luminous-21875

luminous: ceph-mgr spuriously reloading OSD metadata on map changes

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #18735 from shinobu-x/wip-22029-luminous

luminous: restarting active ceph-mgr cause glitches in bps and iops metrics

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

qa/suites/upgrade: disable broken kraken upgrade cls_rbd test

Fixes: https://tracker.ceph.com/issues/22740
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19257 from ifed01/wip-ifed-rocksdb-fix

luminous: rocksdb: fixes early metadata spill over to slow device in

Merge pull request #19071 from smithfarm/wip-22181-luminous

luminous: rgw segfaults after running radosgw-admin data sync init

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #19077 from smithfarm/wip-22171-luminous

luminous: rgw: log keystone errors at a higher level

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19810 from adamemerson/wip-luminous-21901

luminous: rgw: bucket policy evaluation logical error

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19847 from adamemerson/wip-22541-luminous

luminous: rgw: put bucket policy panics RGW process

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19785 from linuxbox2/luminous-22410

luminous: rgw: fix chained cache invalidation to prevent cache size growth

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #19788 from adamemerson/wip-luminous-cache

luminous: rgw: Random 500 errors in Swift PutObject (needs cache fixes)

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #19053 from shinobu-x/wip-22187-luminous

luminous: rgw: add cors header rule check in cors option request

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19819 from adamemerson/wip-backport-22601

luminous: rgw: S3 API Policy Conditions IpAddress and NotIpAddress do not work

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #19050 from shinobu-x/wip-22184-luminous

luminous: Dynamic bucket indexing, resharding and tenants seems to be broken

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19085 from shinobu-x/wip-22215-luminous

luminous: rgw: bucket index object not deleted after radosgw-admin bucket rm --purge-objects --bypass-gc

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18870 from shinobu-x/wip-21949-luminous

luminous: rgw: null instance mtime incorrect when enable versioning

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18764 from smithfarm/wip-22017-luminous

luminous: rgw: Segmentation fault when starting radosgw after reverting .rgw.root

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18765 from smithfarm/wip-22024-luminous

luminous: RGWCrashError: RGW will crash if a putting lc config request does not include an ID tag in the request xml

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18766 from smithfarm/wip-22021-luminous

luminous: rgw: modify s3 type subuser access permission fail

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18867 from shinobu-x/wip-22027-luminous

luminous: multisite: destination zone does not compress synced objects

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18868 from shinobu-x/wip-22026-luminous

luminous: Policy parser may or may not dereference uninitialized boost::optional sometimes

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #19538 from shinobu-x/wip-22434-luminous

luminous: rgw: user stats increased after bucket reshard

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19489 from dplyakin/luminous-get-website-error-code

luminous: rgw: fix GET website response error code

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19090 from pdvian/wip-22179-luminous

luminous: rgw: Fix swift object expiry not deleting objects

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Amit Kumar <amitkuma@redhat.com>

Merge pull request #19434 from cbodley/wip-luminous-pr-16807

luminous: rgw: revert PR #16807

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #19502 from shinobu-x/wip-22397-luminous

luminous: rgw: radosgw-admin reshard command argument error.

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #19506 from shinobu-x/wip-22388-luminous

luminous: rgw: 501 is returned When init multipart is using V4 signature and chunk encoding

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com

Merge pull request #19929 from zmc/wip-luminous-prom-pg_deep

luminous: mgr/prometheus: add missing 'deep' state to PG_STATES in ceph-mgr pro…

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #19943 from dillaman/wip-22676-luminous

luminous: cls/rbd: remove incompatible group features from partial implementation

Reviewed-by: Mykola Golub <mgolub@suse.com>

qa/suites: filter out cls_rbd tests against unsupported methods

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18865 from shinobu-x/wip-21631-luminous

luminous: doc: remove region from "INSTALL CEPH OBJECT GATEWAY"

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

cls/rbd: remove incompatible group features from partial implementation

Fixes: https://tracker.ceph.com/issues/22676
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mgr/prometheus: add missing 'deep' state to PG_STATES in ceph-mgr prometheus plugin

without this the endpoint throws 500s when any PG is doing a deep scrub.

Signed-off-by: Peter Woodman <peter@shortbus.org>
Fixes: http://tracker.ceph.com/issues/22116
(cherry picked from commit 8c93c0eeaff406af0878916ef5b9ce958896d586)

pybind/mgr/prometheus: don't crash when encountering an unknown PG state

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 67a1b66b06bf74274c2cf1d4b5c900f2c56a990d)

Merge pull request #19925 from jcsp/wip-dashboard-trim-luminous2

luminous: mgr: remove unused static files from dashboard module

Reviewed-by: Sebastien Han <shan@redhat.com>

mgr/dashboard: remove unneeded bits of AdminLTE

AdminLTE is delivered in a handy all-in format,
but we don't need all this stuff at runtime.

Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 38df5150ee026c6dc9ec763f757713c7e4d99cdb)

mgr/dashboard: remove non-minified bootstrap

Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 5ce51ebbf047e373eac624e49e2d2bcfb46f09ff)

mgr/dashboard: remove un-minified datatables source

Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 888f0f934e59fe28fdd97bca044702ce445bfe2e)

mgr/dashboard: don't include remote js/css

This will help anyone running away from the internet.

Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 199d7ff26959961f200b86062f6290c9c17cba9a)

mgr/dashboard: re-arrange static files

Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 85a87da7aa46266abd8d4c42218bcbccce386316)

Merge pull request #19880 from jcsp/wip-doc-dashboard-standby-luminous

doc: update mgr/dashboard doc about standbys

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: update mgr/dashboard doc about standbys

Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit fa14812cbb584b61cb2ce687ae4c14225eecd9fe)

Merge pull request #19844 from jcsp/wip-22615

luminous: mgr/dashboard: Fix PG status coloring

Reviewed-by: Wido den Hollander <wido@42on.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #19858 from joscollin/wip-luminous-doc-misc-fixes

luminous: doc: misc fixes for CephFS best practices

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

doc: misc fixes

misc fixes for best-practices.

Signed-off-by: Jos Collin <jcollin@redhat.com>
(cherry picked from commit ecacd1078db5020e6d65436f9d56cc9ed0016666)

doc: fix heading

Fixed 'Which kernel version?' heading not having brown color background.

Signed-off-by: Jos Collin <jcollin@redhat.com>
(cherry picked from commit a068fd842f431a54f9db2d437a71cdc11907a8e1)