erasure-code: shec plugin feature

erasure-code: shec plugin feature

There is one new plugin (shec). When upgrading a cluster, there
must be a protection against the following scenario:

   * the mon are upgraded but not the osd
   * a new pool is created using plugin shec
   * the osd fail to load the shec plugin because they have not been
     upgraded

A feature bit is added : PLUGINS_V3. The monitor will only agree to
create an erasure code profile for the shec plugin if all OSDs
supports PLUGINS_V3. Once such an erasure code profile is stored in the
OSDMap, an OSD can only boot if it supports the PLUGINS_V3 feature,
which means it is able to load the shec plugin.

The monitors will only activate the PLUGINS_V3 feature if all monitors
in the quorum support it. It protects against the following scenario:

   * the leader is upgraded the peons are not upgraded
   * the leader creates a pool with plugin=shec because all OSD have
     the PLUGINS_V3 feature
   * the leader goes down and a non upgraded peon becomes the leader
   * an old OSD tries to join the cluster
   * the new leader will let the OSD boot because it does not contain
     the logic that would excluded it
   * the old OSD will fail when required to load the plugin shec

This is going to be needed each time new plugins are added, which is
impractical. A more generic plugin upgrade support should be added
instead, as described in http://tracker.ceph.com/issues/7291.

See also 9687150ceac9cc7e506bc227f430d4207a6d7489 for the PLUGINS_V2
implementation.

http://tracker.ceph.com/issues/10887 Fixes: #10887

Signed-off-by: Loic Dachary <ldachary@redhat.com>