client: fix error handling in check_pool_perm

client: fix error handling in check_pool_perm

Previously, on an error such as a pool not existing,
the caller doing the check would error out, but
anyone waiting on waiting_for_pool_perm would
block indefinitely (symptom was that reads on a
file with a bogus layout would block forever).

Fix by triggering the wait list on errors and
clear the CHECKING state so that the other callers
also perform the check and find the error.

Additionally, don't return the RADOS error code
up to filesystem users, because it can be
misleading.  For example, nonexistent pool is
ENOENT, but we shouldn't give ENOENT on IO
to a file which does exist, we should give EIO.

Signed-off-by: John Spray <john.spray@redhat.com>