tests: populate /dev/disk/by-partuuid for scsi_debug

The scsi_debug SCSI devices do not have a symlink in /dev/disk/by-partuuid
because they are filtered out by 60-persistent-storage.rules. That was
worked around by 60-ceph-partuuid-workaround-rules which has been
removed by 9f76b9ff31525eac01f04450d72559ec99927496.

Add create rules targetting this specific case, only for tests since the
problem does not show in real use cases.

Fixes: http://tracker.ceph.com/issues/17100
Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 7cbf1f0a5eced402e6c7242015550668e3b568f3)

Merge pull request #11389 from jcsp/wip-17466-jewel

jewel: mon: don't crash on invalid standby_for_fscid

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10997 from batrick/i17105-backport

jewel: multimds: allow_multimds not required when max_mds is set in ceph.conf at startup

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10958 from ukernel/jewel-16764

jewel: client: fix shutdown with open inodes

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10921 from jcsp/wip-client-lock-backport

jewel: client: add missing client_lock for get_root

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11400 from ceph/wip-rbd-cli-jewel

jewel: krbd-related CLI patches

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #11171 from ceph/wip-rasize-doc-jewel

jewel: doc: fix description for rsize and rasize

Reviewed-by: Loic Dachary <ldachary@redhat.com>

rbd: expose rbd unmap options

Reuse rbd map -o infrastructure to expose rbd unmap options in
a similar fashion.  Currently it's just one bool option, but we may
need more in the future.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 620f5e1455fdcb05cd3873c1e260141849829e35)

Conflicts:
doc/man/8/rbd.rst [ PR #9151 ("rbd: add methods to set and get
  snapshot limits") not in jewel ]

rbd: fix up terminology in help texts

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit ba8f9ee791d55c4cf25ad751213ea6ba770b7434)

rbd: recognize lock_on_read option

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit e857b7896527b676155d6e01c78567337dc33b1c)

doc: clarify rbd size units

It wasn't clear from the man page that --size defaults to M.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit d38dc29ed588b8933da3b66e46e1f0dd4cbb2cf4)

Conflicts:
doc/man/8/rbd.rst [ PR #9066 ("rbd: introduce a simple bench
  for read") not in jewel ]

Merge pull request #10812 from SUSE/wip-17094-jewel

jewel: build/ops: ceph-osd-prestart.sh fails confusingly when data directory does not exist

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #11335 from linuxbox2/jewel-nfs

jewel: rgw: nfs backports

Merge pull request #11332 from linuxbox2/jewel-ldap

jewel: rgw: combined LDAP backports

mds: remove max_mds config option

It is now required that all changes to max_mds use the run-time `ceph fs
set max_mds` command. The rationale for this change is that it is
confusing to have a configuration for max_mds which is only observed at
file system creation.

Fixes: http://tracker.ceph.com/issues/17105
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 52f4e545e0ce4782d5f2239d31080d392c12eba2)

Merge pull request #10761 from dachary/wip-16377-jewel

jewel: msgr/async: Messenger thread long time lock hold risk

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10278 from SUSE/wip-16667-jewel

jewel: incorrect value of CINIT_FLAG_DEFER_DROP_PRIVILEGES

Reviewed-by: Sage Weil <sage@redhat.com>

mds: make mds_role_t member order match constructor

Previously this was mildy confusing.

Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 63d711d4e9decd2940c55aa697e6a588593da23a)

mon: don't crash on invalid standby_for_fscid

Fixes: http://tracker.ceph.com/issues/17466
Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 92fdcf36bd7d57b97f5d887d0370a181860e388c)

messages: fix MMDSBeacon constructor

This was leaving garbage in some fields during decode
when decoding messages with an older version.

Fixes: http://tracker.ceph.com/issues/17466
Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit f1fb667dfd21cd241f4f45ce9ef3f8393d114c60)

Merge pull request #10794 from SUSE/wip-17082-jewel

jewel: disable LTTng-UST in openSUSE builds

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #9739 from vumrao/wip-vumrao-16337

jewel: osd: add peer_addr in heartbeat_check log message

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #9388 from vumrao/wip-vumrao-16069

jewel: mon: Display full flag in ceph status if full flag is set

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #10496 from Abhishekvrshny/wip-16583-jewel

jewel: mon crash: crush/CrushWrapper.h: 940: FAILED assert(successful_detach)

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11018 from ceph/wip-17223-jewel

jewel: krbd qa scripts and concurrent.sh test fix

Reviewed-by: Loic Dachary <ldachary@redhat.com>

rgw_file:  pre-assign times

Set unix timestamps based on RGW values for creation, modification
time for buckets, objects in stat requests.  Regard any saved
value of these in unix attributes as an overlay.

Fixes: http://tracker.ceph.com/issues/17367
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit 969750e96380859a350e25015bc8cbf9bdf88512)

Fixes: http://tracker.ceph.com/issues/17394

rgw_file: fix set_attrs operation

The effective part of this change is to always set a value
for RGW_ATTR_UNIX_KEY1 (because it is expected later).

Secondarily, do not pass the address of the to-set attributes
buffer::list as remove attrs--this is confusing.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit 08afb40b2812270a6af3cd1965e8c03c10392ace)

Fixes: http://tracker.ceph.com/issues/17393

rgw_file: implement rgw_setattr

Introduce a new RGWSetattrs RGWOp descendant, to create or replace
sets of attrs on buckets or objects.

This version of the change uses the standard RGWRADOS::set_attrs op
(we want attribute changes to (e.g.) sync with other changes).

Previous versions of this changed incorrectly masked the values
of st->st_ino in RGWFileHandle::stat(), now fixed.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit 4de1c3c260265f821ebee842d49cb35bf49d8e4e)

Fixes: http://tracker.ceph.com/issues/17311
Fixes: http://tracker.ceph.com/issues/17332

rgw: remove duplicated calls to getattr

Signed-off-by: Weibing Zhang <zhangweibing@unitedstack.com>
(cherry picked from commit 87b550ed2fba472d4ef2e1f771bb5f9fd248804f)

Fixes: http://tracker.ceph.com/issues/17327

rgw: rgw file fix bug of rgw_lookup can not exact match file name

bug reproduce steps:
in nfs-client (connect to nfs-ganesha server):
1. mv file file.rename
2. stat file // get file attrs, not return with -ENOENT

the reason is that:
RGWStatLeafRequest does not exact match the file name,
just take file name as a prefix filter

Signed-off-by: Min Chen <chenmin@xsky.com>
(cherry picked from commit 9d813bafc8e197507457c58ab4f365ccdb7f3589)

Fixes: http://tracker.ceph.com/issues/17326

rgw: fix the wrong return value

Here `parent->readdir` will return negative numbers When there is an error.
so we should just return it.

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>
(cherry picked from commit c979be7810793534715c24c6a9f7c8b0ed31de8d)

Fixes: http://tracker.ceph.com/issues/17325

rgw_file: restore local definition of RGWLibFS gc interval

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit 31936caa09b46e86700faad969adfa9d43176206)

Fixes: http://tracker.ceph.com/issues/17323

rgw_file:  explain semantics of RGWFileHandle::commit

Add a comment explaining why the method currently returns 0
unconditionally.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit b1da1354320cc5f69277b225a293a03cc2a5054f)

ceph_timer: prefer using directive to typedef

This change replaces the existing member-hook typedef as well as
the new set-type typedefs, so committed separately.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit f7d4d418658d2091652033045abdb4bfbeb637aa)

librgw: add API version defines for librgw and rgw_file

This change borrows the major, minor+extra format used by libcephfs.
The version numbering is starting at 1,1,0 on the theory that the
implicit version at Jewel is 1,0,0.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit 355ccc5ded7f0b459bca24bc8b504b41807c583d)

rgw file: allow RGWLibFS::write_completion_interval_s to be set in conf

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit 70dad0fba561edaf0bb4e746e29b92a4e9b014b2)

rgw_file:  log on write completion events at level 10

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit d1e512d8c0e20df5a96ccb5f775a7a2ca123efbe)

rgw_file: add RGW_OPEN_FLAG_STATELESS as a synonym for V3

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit da596eb3af69c2cdf191893eb44a8947662634cf)

rgw_file: implement stateless write completion timer

Implements a temporal mechanism to enforce write completion for setups
which lack open state tracking (e.g., NFS3).

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit 2c83ed4bd43fcb5d5497151a157b1dc08022fed9)

rgw_file: fix/update rgw_create flag arguments

Allow passing POSIX open flags as well as api call flags.  Needed
for NFS3 support.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit 2d0ea2b4649e76cbbb87f44c57754d7ea9c46dce)

rgw_file: fix/update rgw_open flag arguments

Allow passing POSIX open flags as well as api call flags.  Needed
for NFS3 support.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit ada29f7f8323be2301588941bcd5c60190b32a4b)

rgw file: remove busy-wait in RGWLibFS::gc()

This is a background thread.  However, CPU is wasted.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit ca33241286f52d849dbde8092507131b8b1108cc)

Fixes: http://tracker.ceph.com/issues/17321
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw_file: unlock() must precede out label

In lookup_handle(...).

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit 9f64f34e481278a683e962b17c7aa811005783a8)

rgw nfs: fix write continuations

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit b4d079f2b5a4d49b2ba7576b272a2a3f3b0d66c8)

rgw nfs: don't leak fh->mtx in lookup_handle()

This change fixes a serious latent locking problem, noticed after
updating the ganesha/rgw driver invalidation after renames.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit d74d46170d7104a6553674f111bbdbe3a116cf54)

rgw file: refuse to rename directories

The representation of paths as an aggregate involving any
number of objects argues against ever permitting such
operations.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit 297220fd2a96c0cf34701127b8cf724e24e6865f)

rgw_file: refuse ops on deleted targets

Detect these illegal cases, as they are indicate various incorrect
behaviors/bugs.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit 34224ecd8360e03603a1a5dd1c1c9f0a31f1c1d5)

rgw_file: implement rgw_setattr

Introduce a new RGWSetattrs RGWOp descendant, to create or replace
sets of attrs on buckets or objects.

This version of the change uses the standard RGWRADOS::set_attrs op
(we want attribute changes to (e.g.) sync with other changes).

Previous versions of this changed incorrectly masked the values
of st->st_ino in RGWFileHandle::stat(), now fixed.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit 4de1c3c260265f821ebee842d49cb35bf49d8e4e)

rgw_file: refuse partial, out-of-order writes

A single file object may be opened only once per gateway
instance, and writes to that object must be complete, and in-order.
Enforce this.

If an invalid write is seen, deletes the current write transaction.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit 12aded803e24539266ce9698c678088e2158a82a)

rgw_file: fix rename cases and unify unlink

Rather ordinary rename cases failed in the unlink step.  Fix this,
unifying the unlink path while at it.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit 05af1fbb870a905ea8903594dfc607d346c9dd31)

add ldap auth custom search filter feature patch - http://tracker.ceph.com/issues/17185

Signed-off-by: Harald Klein <hari@vt100.at>
(cherry picked from commit c935885ae9d5fd413495448a0b0e5fce899c9b73)

Fixes: http://tracker.ceph.com/issues/17185

rgw ldap:  protect rgw::from_base64 from non-base64 input

Also adds unit tests for:
1. empty output from from_base64 (turns out to be harmless)
2. random and specific non-base64 and sort strings
(modified from upstream to avoid alteration of src/test/test_rgw_token.cc)

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit 0a4c91ec7652d02673a9b156cd16144d778a3844)

Fixes: http://tracker.ceph.com/issues/17324

rgw ldap: enforce simple_bind w/LDAPv3

Found by Harald Klein <hklein@redhat.com>.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit 08d54291435e4d1cb5e02cda3951bc6e8510b0e2)

rgw: add reinit/rebind logic (ldap)

Gracefully handle stale LDAP connections by rebinding--verified
with MS AD.  Rebind is attempted once per request until rebound--
not worse than misconfiguring the environment.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit 9b8fccf68789ba5c5986766c544b23aeada0e87e)

Merge pull request #10656 from dachary/wip-16564-jewel

jewel: cors auto memleak

Merge pull request #11252 from ceph/wip-offline-split-jewel

ceph-objectstore-tool: add a way to split filestore directories offline

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #10866 from dachary/wip-17122-jewel

jewel: COPY broke multipart files uploaded under dumpling

Merge pull request #10890 from dachary/wip-17140-jewel

jewel: rgw: period commit loses zonegroup changes: region_map converted repeatedly

Merge pull request #11021 from cbodley/wip-16794

jewel: rgw: back off bucket sync on failures, don't store marker

Merge pull request #10657 from dachary/wip-16792-jewel

jewel: metadata sync can skip markers for failed/incomplete entries

ceph-objectstore-tool: add a way to split filestore directories offline

Use the usual split code, and split each dir that meets the
usual split criteria.

This can be run with lower than usual split settings, to avoid costly
online splits. To make sure the directories are not merged again, use
a load merge threshold (e.g. filestore merge threshold = 1), and
adjust the split multiplier accordingly.

Fixes: http://tracker.ceph.com/issues/17220
Signed-off-by: Josh Durgin <jdurgin@redhat.com>
(cherry picked from commit e7b0428e0e8d8f5459311dc698d94a3ac4f04684)

Conflicts:
src/os/filestore/FileStore.cc
src/tools/ceph_objectstore_tool.cc
(trivial resolution)

Merge pull request #10888 from dachary/wip-17062-jewel

jewel: tests: fix TestClsRbd.mirror_image failure in upgrade:jewel-x-master-distro-basic-vps

doc: fix description for rsize and rasize

Signed-off-by: Andreas Gerstmayr <andreas.gerstmayr@gmail.com>
(cherry picked from commit a004254a02cafdee848a382f893f89c3c5a9845a)

10.2.3

rgw: fix collection of object sync errors

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit aacc31f72cef6f151459e4a0543850edeeaf1938)

rgw: fix marker tracker completion handling

Was not tracking high markers correctly. Could only work if there was a single
hole in the completion range. Just keep a map of all the complete entries.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit f5801c726efdc2f3067a071e6bb5ac83fd0cd147)

rgw: collect() stops if error encoutered

and returns true if needed to be called again

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 5528932e4c327a0fb0ce34a676eb59707a575325)

rgw: back off bucket sync on failures, don't store marker

Fixes: http://tracker.ceph.com/issues/16742
If we fail on any single entry in bucket, skip updating the marker tracker
so that next time we'll go over that entry, and back off. This will trigger
a report to the data sync error repo and eventually a retry on the failing
object.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 1f3fec807043fd313ef7c66ff48f18b82c8bfa66)

krbd: don't segfault if images are unmapped concurrently

"rbd map c" can die from a NULL dereference on any of this_pool,
this_image or this_snap in wait_for_udev_add():

    <image a is mapped>
    rbd map c
                                    rbd map b
                                    rbd unmap a
                                    rbd unmap b

However unlikely, this segfault is triggered by the rbd/concurrent.sh
workunit on a regular basis.

Similarly, "rbd showmapped" can die if an image to be listed is
unmapped.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 2bfecb1c0a0b2a314a5d137e6ca2cccd9ddc9b54)

qa: rbd/simple_big.sh: drop extraneous sudo

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 6643f4d9723e5d9c5195680ac9abb9915ef42a56)

qa: rbd/concurrent.sh: suppress rbd map output

rbd_map_image() echoes the id of the image and is used in command
substitution:

    id=$(rbd_map_image "${image}")

Output from rbd map isn't consumed and clobbers the return.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit a7d4189587e343a581660f1a118f6667c26e5bd4)

qa: rbd: don't modprobe, chown sysfs files or udevadm settle

This is a followup to 38a572011dbb ("qa: rbd/map-snapshot-io.sh: don't
chown sysfs files") for the rest of the workunits:

- /sys/bus/rbd/add_single_major is used if present, so chown'ing
  /sys/bus/rbd/add doesn't help.  Use sudo on rbd map instead.

- Don't modprobe or udevadm settle - rbd CLI tool takes care of that.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 6bddef8ecff97f732bf707d9ba6d2386bc147f5a)

qa: rbd/map-snapshot-io.sh: don't chown sysfs files

/sys/bus/rbd/add_single_major is used if present, so chown'ing
/sys/bus/rbd/add doesn't help.  Use sudo on rbd map instead.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 38a572011dbbf8298cb7ba8ebf43909967eec689)

client: fix shutdown with open inodes

This piece of code was dereferencing an invalid
iterator (invalidated by call to erase())

Fixes: http://tracker.ceph.com/issues/16764
Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit d642b4faec3266f609e4871ccdccdcd73707dc23)

client: add missing client_lock for get_root

A segmentation fault was observed in a recent test failure
(test_client_cache_size) [1]. An analysis of the core dump revealed that the
client->root inode was NULL. The client log [2] showed that the root inode was
deleted prior to the segfault:

    2016-06-27 14:24:13.358500 7fe75cff9700 20 client.4161 _ll_put 0x7fe76000ba40 100000003e9 1 -> 0
    2016-06-27 14:24:13.358502 7fe75cff9700 10 client.4161 put_inode on 100000003e9.head(faked_ino=0 ref=1 ll_ref=0 cap_refs={} open={} mode=41777 size=0/0 mtime=2016-06-27 14:24:10.700562 caps=pAsLsXsFs(0=pAsLsXsFs) 0x7fe76000ba40)
    2016-06-27 14:24:13.358513 7fe75cff9700 15 inode.put on 0x7fe76000ba40 100000003e9.head now 0
    2016-06-27 14:24:13.358515 7fe75cff9700 10 client.4161 remove_cap mds.0 on 100000003e9.head(faked_ino=0 ref=0 ll_ref=0 cap_refs={} open={} mode=41777 size=0/0 mtime=2016-06-27 14:24:10.700562 caps=pAsLsXsFs(0=pAsLsXsFs) 0x7fe76000ba40)
    2016-06-27 14:24:13.358538 7fe75cff9700 15 client.4161 remove_cap last one, closing snaprealm 0x7fe76000c1b0
    2016-06-27 14:24:13.358544 7fe75cff9700 20 client.4161 put_snap_realm 1 0x7fe76000c1b0 1002 -> 1001
    2016-06-27 14:24:13.358546 7fe75cff9700 10 client.4161 put_inode deleting 100000003e9.head(faked_ino=0 ref=0 ll_ref=0 cap_refs={} open={} mode=41777 size=0/0 mtime=2016-06-27 14:24:10.700562 caps=- 0x7fe76000ba40)

After looking through the reference count inc/dec, I noticed this mismatched inc/dec:

    2016-06-27 14:24:13.352622 7fe75dffb700  3 client.4161 ll_forget 100000003e9 1
    2016-06-27 14:24:13.352601 7fe7767fc700  1 -- 172.21.15.58:0/3762258427 >> 172.21.15.57:6804/11121 conn(0x7fe789bf3b60 sd=20 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=7 cs=1 l=0). == rx == mds.0 seq 1473 0x7fe764037a80 client_reply(???:470 = 0 (0) Success safe) v1
    2016-06-27 14:24:13.352665 7fe75dffb700 20 client.4161 _ll_put 0x7fe76000ba40 100000003e9 1 -> 1
    2016-06-27 14:24:13.352687 7fe75d7fa700  3 client.4161 ll_lookup 0x7fe76000ba40 testdir
    2016-06-27 14:24:13.352699 7fe75d7fa700 20 client.4161 _lookup have dn testdir mds.-1 ttl 0.000000 seq 0
    2016-06-27 14:24:13.352705 7fe75d7fa700 15 inode.get on 0x7fe760011da0 100000003ea.head now 537
    2016-06-27 14:24:13.352693 7fe76effd700  1 -- 172.21.15.58:0/3762258427 <== mds.0 172.21.15.57:6804/11121 1473 ==== client_reply(???:470 = 0 (0) Success safe) v1 ==== 27+0+0 (3458149698 0 0) 0x7fe764037a80 con 0x7fe789bf3b60
    2016-06-27 14:24:13.352708 7fe75d7fa700 10 client.4161 _lookup 100000003e9.head(faked_ino=0 ref=3 ll_ref=1 cap_refs={} open={} mode=41777 size=0/0 mtime=2016-06-27 14:24:10.700562 caps=pAsLsXsFs(0=pAsLsXsFs) 0x7fe76000ba40) testdir = 100000003ea.head(faked_ino=0 ref=537 ll_ref=999 cap_refs={} open={} mode=40755 size=0/0 mtime=2016-06-27 14:24:13.325297 caps=pAsLsXsFsx(0=pAsLsXsFsx) COMPLETE parents=0x7fe750001f50 0x7fe760011da0)

This sequence of ll_forget/ll_lookup is repeated in this test several hundred times. This is the prior sequence (in one thread):

    2016-06-27 14:24:13.324896 7fe75dffb700  3 client.4161 ll_forget 100000003e9 1
    2016-06-27 14:24:13.324904 7fe75dffb700 20 client.4161 _ll_put 0x7fe76000ba40 100000003e9 1 -> 1
    2016-06-27 14:24:13.324915 7fe75dffb700  3 client.4161 ll_lookup 0x7fe76000ba40 testdir
    2016-06-27 14:24:13.324921 7fe75dffb700 20 client.4161 _lookup have dn testdir mds.-1 ttl 0.000000 seq 0
    2016-06-27 14:24:13.324931 7fe75dffb700 15 inode.get on 0x7fe760011da0 100000003ea.head now 621
    2016-06-27 14:24:13.324904 7fe7767fc700  1 -- 172.21.15.58:0/3762258427 >> 172.21.15.57:6804/11121 conn(0x7fe789bf3b60 sd=20 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=7 cs=1 l=0). == rx == mds.0 seq 1387 0x7fe764026720 client_reply(???:385 = 0 (0) Success safe) v1
    2016-06-27 14:24:13.324934 7fe75dffb700 10 client.4161 _lookup 100000003e9.head(faked_ino=0 ref=3 ll_ref=2 cap_refs={} open={} mode=41777 size=0/0 mtime=2016-06-27 14:24:10.700562 caps=pAsLsXsFs(0=pAsLsXsFs) 0x7fe76000ba40) testdir = 100000003ea.head(faked_ino=0 ref=621 ll_ref=998 cap_refs={} open={} mode=40755 size=0/0 mtime=2016-06-27 14:24:13.321021 caps=pAsLsXsFsx(0=pAsLsXsFsx) COMPLETE parents=0x7fe750001f50 0x7fe760011da0)

The key line is `_lookup 100000003e9.head(...)`. ll_ref=2 is the expected count
after this sequence but, as we see at 14:24:13.352708, ll_ref=1.

This racing behavior is not serialized by client_lock because
Client::get_root() does not lock the client_lock. Additionally, this race is
not easy to identify in the logs because Client::get_root() does not use
Client::_ll_get to increment the reference, which hides the increments in the
logs. Instead it directly increments the reference using Inode:ll_get().

This commit adds the client_lock for Client::get_root() and
Client::get_root_ino() (which may not strictly be necessary but can't hurt).

[1] http://pulpito.ceph.com/pdonnell-2016-06-27_06:54:23-fs-wip-pdonnell-i16164---basic-smithi/280361/
[2] http://qa-proxy.ceph.com/teuthology/pdonnell-2016-06-27_06:54:23-fs-wip-pdonnell-i16164---basic-smithi/280361/remote/smithi058/log/ceph-client.0.23448.log.gz

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 208e3b6fd58237b309aae15b18dcd82672b133af)

Merge pull request #10804 from dachary/wip-17089-jewel

jewel: OSD failed to subscribe skipped osdmaps after ceph osd pause

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10760 from dachary/wip-16865-jewel

jewel: saw valgrind issues in ReplicatedPG::new_repop

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10662 from dachary/wip-16901-jewel

jewel: segfault in RGWOp_MDLog_Notify

Reviewed-by: Yehuda Sadeh <ysadehwe@redhat.com>

Merge pull request #10763 from dachary/wip-17034-jewel

jewel: rgw: object expirer's hints might be trimmed without processing in some circumstances
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10658 from dachary/wip-16862-jewel

jewel: default zone and zonegroup cannot be added to a realm
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10659 from dachary/wip-16863-jewel

jewel: use zone endpoints instead of zonegroup endpoints
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10660 from dachary/wip-16864-jewel

jewel: multisite segfault on ~RGWRealmWatcher if realm was deleted
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

rgw: delete region map after upgrade to zonegroup map

convert_regionmap() reads the region map and uses it to initialize the
zonegroup map.  but it doesn't remove the region_map afterwards, so
radosgw (and some radosgw-admin commands) will keep doing this on
startup, overwriting any changes made to the period/zonegroup map

Fixes: http://tracker.ceph.com/issues/17051
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 215cd6f6ff7b8cb74df6dc3b94a5928f56ab9540)

qa/workunits: support filtering cls_rbd unit test cases

Fixes: http://tracker.ceph.com/issues/16529
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 0891ede57cd254aa75c66301fde4a2eb6957a3d6)

Merge pull request #10847 from gregsfortytwo/jewel-fix-mds

mds: fix double-unlock on shutdown

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #10663 from dachary/wip-16934-jewel

jewel: Add zone rename to radosgw_admin

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10664 from dachary/wip-16958-jewel

jewel: Bug when using port 443s in rgw.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10764 from dachary/wip-17032-jewel

jewel: multisite: RGWPeriodPuller tries to pull from itself

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10765 from dachary/wip-17006-jewel

jewel: Increase log level for messages occuring while running rgw admin command

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10845 from cbodley/wip-16702

jewel: rgw: add tenant support to multisite sync

Reviewed-by: Loic Dachary <ldachary@redhat.com>

rgw: adjust manifest head object

adjust the manifest head object:
 - when reading manifest, set the head object to the object
   we read the manifest from (and its size). Some manifests are
   broken and point at a different object
 - when copying multipart object, set the manifest head object to
   point at the new head object

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 419264586fc46337322f0d60b0ffcdcce3bb5c5a)

rgw: adjust objs when copying obj with explicit_objs set

If the head object contains data, need to set it in the list of
objs (in addition to being pointed at by head_obj).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 1e012112b99004745952accc6fb11ada5e8e0045)

rgw: patch manifest to handle explicit objs copy issue

Fixes: http://tracker.ceph.com/issues/16435
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit b51476466e5bb03bbaa6e428bb6bb189a259b9fd)

Merge pull request #9405 from SUSE/wip-16083-jewel

jewel: mds: wrongly treat symlink inode as normal file/dir when symlink inode is stale on kcephfs

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10654 from dachary/wip-9577-jewel

jewel: mon: "mon metadata" fails when only one monitor exists

Reviewed-by: John Spray <john.spray@redhat.com>

mds: fix double-unlock on shutdown

We did a bad backport or something and accidentally ended up with two Unlock()
calls on mds_lock. Don't.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

rgw: collect skips a specific coroutine stack

Fixes: http://tracker.ceph.com/issues/16665
Instead of drain_all_but() that specifies number of stacks to leave behind,
added drain_all_but_stack() that has a specific stack specified. This is needed
so that we don't call wakeup() through lease_cr->go_down() on a cr stack that
was already collected.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 5a2e8f0526db92a290c711f82627fc5042c290ea)

Merge pull request #10791 from dachary/wip-17004-jewel

jewel: rbd-mirror: FAILED assert(m_state == STATE_STOPPING)

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10827 from dillaman/wip-16950-jewel

jewel: librbd: delay acquiring lock if image watch has failed

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rgw: fix compilation

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
(cherry picked from commit 874de58024e12d893ae050cd421ef67c9521c753)