osd: use get_pgid() instead of info.pgid

Signed-off-by: Myoungwon Oh <omwmw@sk.com>

osd: use global id instead of entity_inst_t

Signed-off-by: Myoungwon Oh <omwmw@sk.com>

osd: move pgid into OpQueueItem

this pave the road to generalize OpWQ

We're going to want to be able to queue things that are not ordered by
the PG lock.  To that end, this patch genearlizes OSD::ShardedOpWQ to
use a type which can specify an ordering token and locking structure
other than a PG.

There is a lot of collateral damage which I didn't feel was worth
separating out into other commits.  The code in ShardedOpWQ itself got
some superficial cleanup.  Also, the item being queued has been switched
to not use a boost::variant.  It was a cute way before to make the type
easily copyable, but adding more visitors for the locking support would
have been annoying.  Instead, the variant is a unique_ptr to an
interface.  This makes the queue item type no longer copyable, which is
just as well since we don't really want to be copying queue items anyway
(duplicates would be most likely a bug)

Signed-off-by: Samuel Just <sjust@redhat.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>
Signed-off-by: Myoungwon Oh <omwmw@sk.com>

src/test: fix move semantics

Signed-off-by: Myoungwon Oh <omwmw@sk.com>

common,osd: enforce move semantics in WorkQueue

enable the WorkQueue to support move-only template types.

Signed-off-by: Samuel Just <sjust@redhat.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>
Signed-off-by: Myoungwon Oh <omwmw@sk.com>

enforce move semantics of Request type

Signed-off-by: Samuel Just <sjust@redhat.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

OSD: s/PGQueueable/OpQueueItem/g

PGQueueable was always a pretty bad name, and going forward, we won't
necessarily be scheduling things ordered only by the PG lock, so let's
change it.

Signed-off-by: Samuel Just <sjust@redhat.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>
Signed-off-by: Myoungwon Oh <omwmw@sk.com>

PG.h: remove some unnecessary virtual declarations

PG users do not need these public methods.

Signed-off-by: Samuel Just <sjust@redhat.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>
Signed-off-by: Myoungwon Oh <omwmw@sk.com>

Merge pull request #18190 from JustL/master

msg/async/rdma: unnecessary reinitiliazation of an iterator

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #18058 from Yan-waller/wip-walle-0927pgstatexpandstate

osd/PG:  extend pg state bits to fix pg ls commands error

Reviewed-by: Sage Weil <sage@redhat.com>

msg/async/rdma: unnecessary reinitiliazation of an iterator

current_buffer iterator does not need to be reinitalized as no changes
occur between the initialization and iteration steps.

Signed-off-by: Justinas Lingys jlingys@connect.ust.hk

Merge pull request #18178 from joscollin/wip-radosgw-admin-reshard-fix

rgw: update usage() with status
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #18180 from oritwas/wip-rgw-21617

rgw: add reshard commands

Reviewed-by: Jos Collin <jcollin@redhat.com>

rgw: Fixed typos and status in help.t

Fixed typos and status in help.t

Signed-off-by: Jos Collin <jcollin@redhat.com>

rgw: add reshard commands to the man pages

Fixes: http://tracker.ceph.com/issues/21617
Signed-off-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #18162 from tchaikov/wip-ceph-disk-retry-on-dev-not-ready

ceph-disk: retry on OSError

Reviewed-by: Loic Dachary <ldachary@redhat.com>

rgw: update usage() with status

Fixed the missing 'status' in usage().

Signed-off-by: Jos Collin <jcollin@redhat.com>

osd/PG: extend pg state to 64 bits

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

os/bluestore: drop unused class FreelistManager

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

Merge pull request #18157 from batrick/ptl-tool-branch-checkout

ptl-tool: checkout branch after creation

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #17708 from liewegas/wip-pg

osd: initial minimal efforts to clean up PG interface

Merge pull request #18125 from cernceph/dvanders_conf

mon: mark mon_allow_pool_delete as observed

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18144 from hjwsm1989/wip-21425-part2

qa/suites/rest/basic/tasks/rest_test: whiltelist OSD_DOWN

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #18159 from liewegas/wip-21712-b

os: fix 0-length zero semantics, add tests

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: 0-length zero should not change object size

Fixes: http://tracker.ceph.com/issues/21712
Signed-off-by: Sage Weil <sage@redhat.com>

os/filestore: make 0-length zero avoid touching object length

Fixes: http://tracker.ceph.com/issues/21712
Signed-off-by: Sage Weil <sage@redhat.com>

os/ObjectStore: 0-length zero does not change length of object

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18141 from adamemerson/wip-log-time

log: Use the coarse real time clock in log timestamps

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17976 from adamemerson/wip-warning-sign

mgr,osd,kv: Fix various warnings for Clang and GCC7

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-disk: retry on OSError

we are likely to
1) create partition, for instance, sdc1
2) partprobe sdc
3) udevadm settle
4) check the device by its path: /dev/sdc1

but there is chance that the uevent sent from kernel fails to reach udev
before we call "udevadm", hence "/dev/sdc1" does not exist even after
"udevadm settle" returns. so we retry in case of OSError here.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-disk: factor out the retry logic into a decorator

Signed-off-by: Kefu Chai <kchai@redhat.com>

log: Mark timestamps for coarseness and print appropriately

Have each timestamp know whether it's coarse or fine. Print six figures
for fine stamps and three for coarse.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

log: Add log_coarse_timestamps option

And set up an observer in CephContext to coarsen/refine time stamps as
requested.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

config: Add templated _get_val for use in observers

When notified by a config observer, the config lock is already held,
create a _get_val template to manipulate newstyle options when
notified.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

log: Use log_clock, defaulting to coarse time

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

log: Add optional msg parameter to create_entry and test with it

The tests should, really, get their time from the same source as the
log, and also probably use the same interface to creating entries that
other people do. So add an optional message argument and have the
tests use that rather than using the new operator directly.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

log: Create log_clock to replace use of utime_t

We create a new time type to support using coarse timestamps (for
performance) by default while still allowing the user to switch to
fine timestamps.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

osd/PGLog: drop old compat coll/oid args to read_log_and_missing

The oldest version we care about is 10, and these were for <8.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18122 from liewegas/wip-monthrash-peering

qa/suites/rados/monthrash: tolerate PG_AVAILABILITY during mon thrashing

Merge pull request #18047 from gregsfortytwo/wip-20416-bitwise-assert

osd: make the PG's SORTBITWISE assert a more generous shutdown

Reviewed-by: Sage Weil <sage@redhat.com>

ceph-objectstore-tool: bring past_intervals up to date on import

We can't blinding fast-forward the pg epoch to the latest map without
also updating the pg history and PastIntervals.  Do that at import time.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-objectstore-tool: hide DIAGNOSTIC stuff behind debug/verbose mode

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-objectstore-tool: drop unused biginfo_oid

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-objectstore-tool: refuse to import PG older that OSD's oldest_epoch

We don't have a way to construct a valid PastIntervals history.

Signed-off-by: Sage Weil <sage@redhat.com>

ptl-tool: checkout branch after creation

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-objectstore-tool: remove rm-past-intervals op

The OSD doesn't rebuild this on demand anymore.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-objectstore-tool: do not clear same_interval_since and PastIntervals

We dropped the OSD recalculation code for this.  Import it.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-objectstore-tool: print pg_info_t to stdoutt

Helpful for debugging

Signed-off-by: Sage Weil <sage@redhat.com>

osd: print summary for forced backfill/recovery to debug

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: drop OSD friend, finally!

Signed-off-by: Sage Weil <sage@redhat.com>

osd: drop redundant is_active() check before scrub_sched()

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: make update_snap_mapper_bits public

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: make on_removal() public

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: make get_primary() public

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: add get_pool()

Signed-off-by: Sage Weil <sage@redhat.com>

osd: use pg_id.pool() for pg pool id

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: make sched_scrub() public

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: get_must_scrub()

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PGBackend: remove scrub_supported()

Everybody supports scrub.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: change 'debug dump_missing'

- json
- move into PG class

No users in the qa suite.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: move bits of delete_dir (pg removal) into PG

We don't want to expose things like SnapMapper, OSDriver.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: define public interface for heartbeat peers

Signed-off-by: Sage Weil <sage@redhat.com>

osd: use accessors for pg resurrection

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: simplify start_recovery_ops return value meaning

Instead of complex check in caller, return simple bool indicating whether
find_unfound() should be called with a rctx.

There is one remaining condition that can probably be simplified:

  if (!recovering.empty() ||
      work_in_progress || recovery_ops_active > 0 || deferred_backfill)
    return !work_in_progress && have_unfound();

but we will leave it for now.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: simplify unfound recovery case; rename method

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: drop unneeded write_if_dirty

Nothing in start_recovery_ops or stuck_on_unfound dirties state.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: move a few assertions, flag change into start_recovery_ops()

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: move some more split machinery into PG

and move other split pieces into the public interface.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: move write_if_dirty inside handle_{loaded,create}

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: get_need_up_thru(), get_same_interval_since()

Signed-off-by: Sage Weil <sage@redhat.com>

osd: bit less verbose while loading pgs

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: move more recovery logic into PG

I suspect we eventually want to move the create_context and
dispatch_context into OSDService (if it isn't there already) and move
even more of this logic into PG.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: move some osd logic into handle_pg_trim

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: hide peering_queue from OSD

Signed-off-by: Sage Weil <sage@redhat.com>

osd: remove build_past_intervals_parallel

This was added in the dumpling era to handle the case where we have lots
of PGs with stale or missing past_intervals.  These days, we update
PastIntervals as we inject new makes (via advance_map), when we
learn about new PGs via notify with get the PastIntervals at the same time,
and during PGCreate handling we have a separate path in
build_initial_pg_history.  This code is obsolete.

It also reaches deep into the PG class where it shouldn't!

Signed-off-by: Sage Weil <sage@redhat.com>

osd: use PG::is_deleting()

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: move load_pgs cruft into read_state()

This only happens during initial load; move it there.

Also, since update_store_with_options

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: make read_state, peek_map_epoch public

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: simplify/streamline upgrade process

There's no reason for this to be exposed to the OSD.  Just do it directly
in read_state().

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: drop bufferlist passed from peek_map_epoch to read_state/read_info

This is no longer used or needed.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: make init() public

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: on_shutdown -> shutdown in public interface

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: add get_num_ref()

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: add get_pg_stats() to public interface

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: add dump_pgstate_history() to public interface

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: make get_watchers public, hide locking

Signed-off-by: Sage Weil <sage@redhat.com>

osd: drop legacy infos object cleanup

This won't be there post-luminous

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: make osr public

Signed-off-by: Sage Weil <sage@redhat.com>

osd: use is_deleting()

Signed-off-by: Sage Weil <sage@redhat.com>

osd: use pg->pg_id const member, not get_pgid()

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: reorder some members

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: protect get_last_peering_reset

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: protect get_role()

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: cleanup and move ref counting public bits

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: protect pg state accessors

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: add is_forced_recovery_or_backfill()

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ECBackend: use Listener instead of PG interface to check undersized

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: collection more of public at top of class

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: expose static methods for ceph-objectstore-tool

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: move C_PG_FinishRecovery into header

Less code than delcaring it in 2 places (by making it a friend).

Signed-off-by: Sage Weil <sage@redhat.com>