Patrick Donnelly [Fri, 22 Mar 2024 15:56:08 +0000 (11:56 -0400)]
Merge PR #56271 into main
* refs/pull/56271/head:
qa/cephfs: stop ignoring MON_DOWN globally
qa: extend mon timeout coming up after mondb creation
qa: update dashboard schema for mon_status
mon: do not log MON_DOWN if monitor uptime is less than threshold
Reviewed-by: Leonid Usov <leonid.usov@ibm.com> Reviewed-by: Venky Shankar <vshankar@redhat.com>
Venky Shankar [Fri, 22 Mar 2024 07:24:29 +0000 (12:54 +0530)]
Merge PR #54581 into main
* refs/pull/54581/head:
doc/dev: update quiesce developer document
qa: wrap quiesce verification to dump debugging on error
qa: update quiesce tests for control via locallock
qa: set archive path in vstart_runner
qa: refactor CephFSMount.kill_background to optionally kill all background jobs
qa: use kwarg for rank parameter
qa: simplify calls to (rank|mds)_(tell|asok)
Revert "pybind/mgr/volumes: block quiesce for critical .meta file"
mds: remove is_root indication on quiesce_inode op
mds: prevent new lock cache cons when invalidating an existing one
mds: use XLOCK_WAIT For local lock xlockers
mds: prevent new wrlocks on LocalLock if there exists any xlock waiter
mds: block import discover when parent directory inode is quiesced
mds: avoid issuing exclusive caps to clients lacking w caps
mds: print lock cache during invalidation
mds: use inodeno_t to track quiesce requests
mds: dispatch quiesce_inode ops after dir traversal
mds: remove quiescelock handling for SimpleLock type
mds: quiescelock as local lock + cap masking
qa: run quiesce unit tests in fs:functional
qa: add quiesce protocol unit tests
qa: detect partial migrations during large config of dist epin
qa: use stdin-killer to timeout run_shell_payload
qa: simplify run_shell argument processing
doc: add dev docs for quiesce protocol
pybind/mgr/volumes: block quiesce for critical .meta file
mds: add vxattr to block quiesce on an inode
mds: convert encoded ephemeral dist pin to flags
mds: add counter to throttle quiesce
mds: add quiesce set feature flag
mds: skip non-head inodes for quiesce
mds: add quiesce op
mds: print all SimpleLock flags in debug output
mds: pretty print mutation when dumping lock
mds: add new inode quiescelock
mds: use 128 bits for waiters on MDSCacheObject
mds: provide mechanism to authpin while freezing
mds: add command to get specific op
mds: finish request before completing internal req
mds: complete internal op if killed
mds: avoid killing dead requests
mds: add command to kill request
mds: add path argument to `ops` and `dump tree` to stream result to local file
mds: print internal_request filepaths if present
mds: add more information to debug message
mds: remove redundant parenthesis
mds: implement Mutation::dump method
mds: make LockType fields const
mds: annotate mdr with try_rdlock_snap_layout failure
mds: refactor if into switch
mds: call Locker method using this
mds: simplify assert
mds: dump locks passed to Locker::acquire_locks
mds: add LockOp::print method for debugging
mds: use new insert template via print
mds: add request result to mutation for analysis by tests
mds: add comment on locking order rules
mds: allow specifying rdlock position
mds: remove dead method
common: provide a template for object dumps
common: support long running ops without slow warnings
common: simplify loop
common: add JSONFormatterFile class
common: use more efficient vector for stack
include: use larger int for large gathers
Reviewed-by: Venky Shankar <vshankar@redhat.com> Reviewed-by: Leonid Usov <leonid.usov@ibm.com>
Afreen [Fri, 1 Mar 2024 07:26:25 +0000 (12:56 +0530)]
mgr/dashboard:Mark placement targets as non-required in bucket form
Fixes https://tracker.ceph.com/issues/64708
- adds info text explaining about placement targets
- adds fieldset to security, policy and tags for a11y
- removes placement target as required field
- when no placement target provided, request omits placement_target
field
- hides placement atrgets ection for edit form since its not possible to
edit after creation
- updates rgw-bucket.service.spec.ts unit test
- minor cleanups related to unncesseary css present, and changes in help
text for object locking
- updated e2e tests for buckets to incorporate changes
This flag is no longer necessary as the volumes plugin issues quiesce calls
against the data (i.e. root) directory of the subvolume rather than the
subvolume directory (with its associated .meta file).
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
mds: prevent new lock cache cons when invalidating an existing one
The previous scheme invalidated a lock cache and then immediately removed it
from its Capability list. The lock cache would eventually be deleted but a new
one could be constructed shortly after. The main reason for this is that simply
invalidating the lock cache does not drive a state change in the local locks
preventing new writers. This is mostly important for acquiring the quiescelock.
This commit also corrects a bug where a MDLockCache would be created for a
given opcode type (like create) when the capability does not have the issued
cap (CEPH_CAP_DIR_CREATE). The bug would not cause any negative side-effects
but would hold locks unnecessarily when only MDS ops (and not the client
executing ops asynchronously) are acquiring the locks.
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
mds: prevent new wrlocks on LocalLock if there exists any xlock waiter
Otherwise, an xlock waiter can become starved as a LocalLock supports multiple
writers.
Strictly speaking, a new lock state would be appropriate for this but we cheat
frequently with the LocalLock -- there is only one state. All transition checks
are already manually performed by the Locker.
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
mds: block import discover when parent directory inode is quiesced
This is to prevent two racing ranks quiescing some root from exporting a tree
under a completely quiesced directory (inode). The state of that imported tree
may take time to quiesce and cause the root to be QUIESCED before all inodes
under it are actually quiesced.
If a dirfrag to be imported is discovered before the parent is quiesced, then
the quiesce traversal will issue a quiesce_inode op normally for parent which
will attempt to authpin the parent. That will block if the export is still
in-progress (causing quiesce to wait for the export to finish or abort).
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
If a CInode is removed from cache before the quiesce_inode request can process
it (and pin it in cache), a new CInode may be created with the same address.
That pointer still exists in MutationImpl::quiesce_ops and would prevent
issuing a quiesce_inode op for the new inode.
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
Patrick Donnelly [Sat, 17 Feb 2024 15:26:14 +0000 (10:26 -0500)]
qa: detect partial migrations during large config of dist epin
This method would wrongly "succeed" when looking for setup of distributed
ephemerally pinned directory fragments. If the migrator splits a subtree during
the course of migration (to reduce the migration size) then the operation may
not actually be complete.
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
Patrick Donnelly [Wed, 24 Jan 2024 02:25:35 +0000 (21:25 -0500)]
qa: use stdin-killer to timeout run_shell_payload
- simplify argument processing / forwarding
- use stdin-killer to kill all sub-processes of the shell
- do not needlessly use run_shell to execute the command as it adds a timeout
to the stdout/stderr processing
- provide a stdin (PIPE) by default otherwise teuthology's code closes stdin
and triggers stdin-killer to timeout the shell.
- use a 15 minute timeout by default
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
Patrick Donnelly [Tue, 20 Feb 2024 22:08:32 +0000 (17:08 -0500)]
mds: use 128 bits for waiters on MDSCacheObject
Adding a new inode lock will overflow inode wait bits into the MDSCacheObject
wait bits. Make space for the quiescelock.
This includes a minor refactor to no longer attempt scoping the set of masks we
test in MDSCacheObject::waiting when calling MDSCacheObject::is_waiter_for.
This optimization wasn't worth the overhead and would be awkard to keep as
std::bitset cannot be used as a key for a std::multimap (easily). Instead, we
use the sequence number as a key which helps us to avoid allocating another map
whenever we call MDSCacheObject::take_waiting.
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
Patrick Donnelly [Tue, 13 Feb 2024 16:07:26 +0000 (11:07 -0500)]
mds: provide mechanism to authpin while freezing
When a subtree is freezing, it's no longer possible to acquire new authpins.
This is a problem when a compound request like quiescing a subtree is trying to
acquire authpins for each sub-op. This creates a situation where some quiesce
sub-ops complete with authpins (thereby preventing the tree from becoming
"frozen") and new sub-ops cannot acquire authpins (because the tree is
"freezing"). To circumvent this, allow some authpin requests to proceed if
FLAG_BYPASSFREEZING is set.
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
projects / ceph.git / log