osd: remove chunk_map.clear() becuase new chunk_map
     will be overwritten entirely

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

osd: do not reply EAGAIN if get_lock_type() is false

This will be retried later

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

osd: correct the if case to handle exceptional case

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

osd: move setting callback and manifest_ops to caller

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

osd: fix memory leak in C_SetManifestRefCountDone

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

osd: remove unused bufferlist

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

osd,test: remove cdc_window_size entirely

To avoid miscalculation caused by boundary mismatch,
we prefer to read whole object.

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

test: adjust DedupFlushRead and add a scenario

add a scenario--object size > window size > chunk size

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

osd: combine two ifs

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

osd: move create_or_get_ref to refcount_manifest()

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

osd: make start_dedup() and do_cdc() simple with removing unnecessary copy

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

osd: fix comment, declaration

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

osd: remove unnecesssary if{}, and make simple using
calc_refs_to_inc_on_set()

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

osd: return EINVAL instead of 0 if fingerprint is not set

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

osd: add comment to explain why EC pool is disallowed

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

test: correct error and add checking refcount in ManifestFlushDupCount

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

osd: fix reference counting during tier_flush

When dealing with reference on manifest object including snapshot,
consider increment/decrement refcount between old and new chunk_map.
See the comment in this commit.

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

osd: return 0 early if fingerprint algorithm is not set

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

test: add a test to confirm that proper error is return if tier_flush fails

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

osd: fix error handlding if flush fails

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

osd: fix initialization for C_SetDedupChunks

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

test: add a duplicated count test when tier_flush() is called

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

osd: send decrement message to only target chunks (not all chunks)

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

osd: do not increment refcount if a previous snapshot have a same chunk

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

osd: return 0 immediately if the object is evicted

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

osd: return an error earlier in start_flush() if non-blocking is enabled

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

osd: return zero if there is no entry to dedup

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

osd: check completion state and rework finish_set_dedup

make finish_set_dedup() to handle all error or success cases

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

osd: remove unnecessary argument in finish_set_dedup

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

osd: return error imediately if last_peering_reset != pg->get_last_peering_reset

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

osd: fix uninitialized value

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

mon: add error check and a comment to explain why interr.clear() is needed

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

osd, test: do not allow ec pool to be set as a base tier in dedup

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

test: add tier_flush tests

1. Flush and read data
2. Flush snapshotted object

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

osd: add tier_flush for dedup

As described in TODO, tier_flush triggers running CDC, then
increases reference count after CDC

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

osd, mon : add dedup options

add dedup_tier, dedup_chunk_algorithm, dedup_cdc_window_size
and dedup_cdc_chunk_size.

dedup_tier: a pool where chunks are stored.
dedup_chunk_algorithm: chunking algorithm.
dedup_cdc_window_size: the maximum data size to run CDC.
 Dedup process reads amount of dedup_cdc_window_size,
 then calculate chunks by using CDC.
dedup_cdc_chunk_size: chunk size for CDC.

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

Merge pull request #36979 from tspmelo/wip-npm-update-20-8

mgr/dashboard: Update npm packages

Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #37449 from ceph/47676-zone-empty-realm-id

mgr/dashboard: get rgw daemon zonegroup name from mgr

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #37375 from ceph/47615-fix-endpoint-responses

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #37064 from bk201/wip-47372

mgr/dashboard: fix error when typing existing paths in the Ganesha form

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #37596 from rhcs-dashboard/telemetry_link_cleanup

mgr/dashboard: Improve HTML formatting of the telemetry report preview

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

mgr/dashboard: Update npm packages

Angular 10.1 includes extraction of TS translations,
so @locl/cli is no longer needed.

Updating Angular automatically removed the usage of base tsconfig.

Downgraded css-loader to be able to use css modules.

angular-tree-component has been deprecated and
replaced by @circlon/angular-tree-component

Fixes: https://tracker.ceph.com/issues/47289
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge PR #32778 into master

* refs/pull/32778/head:
msg: Use sockets instead of pipes for wakeup events on win32
common: Convert remote errors on win32
common,msg: Initialize Windows WSA and TLS
common,os,kv: Define aligned_free
common: avoid CLOCK_*_COARSE warnings on win32
global: Windows support

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #36554 into master

* refs/pull/36554/head:
mgr/volumes: Make number of cloner threads configurable

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Shyamsundar R <srangana@redhat.com>

Merge PR #36457 into master

* refs/pull/36457/head:
qa/tasks/cephfs: fix fs authorize cmd args
mds: add root_squash mode in MDS auth caps
vstart_runner: add write_file and sudo_write_file

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #36556 into master

* refs/pull/36556/head:
doc: cephfs mirroring design document

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #37483 from dzafman/wip-46405

osd/osd-rep-recov-eio.sh: TEST_rados_repair_warning:  return 1

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #37500 from ricardoasmarques/fix-bootstrap-container-init

cephadm: Fix error setting 'mgr/cephadm/container_init' config

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #37582 from ivancich/wip-fix-bucket-list-namespace

rgw: fix setting of namespace in ordered and unordered bucket listing

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #37572 from ivancich/wip-orphan-list-namespace-locator

rgw: allow rgw-orphan-list to note when rados objects are in namespace

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #37511 from ivancich/wip-paginate-rgw-bucket-list

rgw: radosgw-admin should paginate internally when listing bucket

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

mgr/dashboard: Improve HTML formatting of the telemetry report preview

Convert the string "Community Data License Agreement - Sharing - Version 1.0" to a clickable link

Fixes: https://tracker.ceph.com/issues/47790
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #37584 from rhcs-dashboard/report-preview-helper

mgr/dashboard: Add short descriptions to the telemetry report preview

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge PR #37591 into master

* refs/pull/37591/head:
ceph-volume: don't exit before empty report can be printed

Reviewed-by: Joshua Schmid <jschmid@suse.de>

doc: cephfs mirroring design document

Signed-off-by: Venky Shankar <vshankar@redhat.com>

ceph-volume: don't exit before empty report can be printed

get_plan() called exit in case of an empty plan. This prevented a report
being printed under these circumstances. Avoid exit in this case. Also
adds tests to ensure an empty report is printed.

Fixes: https://tracker.ceph.com/issues/47760
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #37561 from tchaikov/wip-cmake-node-mirror

cmake: support Node.js and npm registry mirror

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #37570 from ceph/wip-install-ceph-deploy

doc: add a link to ceph-deploy document

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #37446 into master

* refs/pull/37446/head:
mds: remove dead mdsmap broadcast code

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge PR #37497 into master

* refs/pull/37497/head:
client,mds: Drop execute permission bits

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #37571 into master

* refs/pull/37571/head:
qa/tasks/cephfs/nfs: Check ganesha daemon status after restarting mgr

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #37579 into master

* refs/pull/37579/head:
SubmittingPatches: use "doc" prefix in title of doc-only commits

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/dashboard: Add short descriptions to the telemetry report preview

A small info icon with a popover on the telemetry report preview form step2 for
Report ID and Report Preview.

Report ID: "A randomized UUID to identify a particular cluster over the course of several telemetry reports."
Report preview: "The actual telemetry data that will be submitted."

Fixes: https://tracker.ceph.com/issues/47610
Signed-off-by: Nizamudeen A <nia@redhat.com>

rgw: fix setting of namespace in ordered and unordered bucket listing

The namespace is not always set correctly during bucket listing. This
can, for example, cause the listing of incomplete multipart uploads,
which are in the _multipart_ namespace, to not paginate correctly, and
cause entries to be re-listed.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge PR #37577 into master

* refs/pull/37577/head:
PendingReleaseNotes: add note about batch refactor

Reviewed-by: Nathan Cutler <ncutler@suse.com>

rgw: allow rgw-orphan-list to note when rados objects are in namespace

Currently namespaces and locators are ignored when `rados ls` is run
by rgw-orphan-list to record RADOS's known objects.

However there have been cases where RADOS objects have a locator, and
when one is included in the listing, the script does not handle it
correctly. Now when objects have locators, we will prevent their
output from entering the .intermediate file.

Additionally we do not expect RGW data objects to be in RADOS
namespaces, so when a namespaced object is detected, we'll error out
with a message.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

PendingReleaseNotes: add note about batch refactor

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

SubmittingPatches: use "doc" prefix in title of doc-only commits

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37498 from jan--f/cadm-new-batch-report

pybind/mgr/orchestrator: fix preview for new c-v batch output

Merge pull request #37556 from dillaman/wip-rbd-status

rbd: cleanup code and output formatting in status action

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37521 from jschmid1/remove_orch_osd_spec_mention

docs: <orch osd spec> is no longer present

Merge pull request #37574 from tchaikov/wip-crimson-doc

doc/crimson: cleanups

Reviewed-by: Myoungwon Oh <myoungwon.oh@samsung.com>

doc/crimson/poseidonstore: reformat

to render it as expected

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: move dev/crimson.rst into dev/crimson/

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36343 from myoungwon/wip-doc-poseidon-store

doc: introduce PoseidonStore

Reviewed-by: Samuel Just <sjust@redhat.com>

mgr/dashboard: get rgw daemon zonegroup name from mgr

Fixes: https://tracker.ceph.com/issues/47676
Signed-off-by: Alfonso Martinez <almartin@redhat.com>

Merge pull request #37567 from liewegas/wip-stable-mod

include: ceph_stable_mod in public domain

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/tasks/cephfs/nfs: Check ganesha daemon status after restarting mgr

Fixes: https://tracker.ceph.com/issues/47591
Signed-off-by: Varsha Rao <varao@redhat.com>

doc: add a link to ceph-deploy document

Merge pull request #37565 from tchaikov/wip-crimson-cleanup

crimson: do not capture unused variables

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

crimson/os: mark class with "final" dtor "final"

to silence warning like

scheduler.cc:145:30: warning: class with destructor marked 'final' cannot be inherited from [-Wfinal-dtor-non-final-class]
  ~ClassedOpQueueScheduler() final {};
                             ^
/home/jenkins-build/build/workspace/ceph-perf-crimson/ceph-pr/src/crimson/osd/scheduler/scheduler.cc:52:7: note: mark 'ClassedOpQueueScheduler<T>' as 'final' to silence this warning
class ClassedOpQueueScheduler : public Scheduler {
      ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: mark dtor of interface class "virtual"

we use a smart_ptr in Seastore to manage the life cycle of
`TransactionManager` which is in turn a subclass
`ExtentCallbackInterface`, so should declare the dtor of latter as
virtual.

Signed-off-by: Kefu Chai <kchai@redhat.com>

rbd: cleanup code and output formatting in status action

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

include: ceph_stable_mod in public domain

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #37395 from tchaikov/wip-doc-crimson

doc/dev/crimson: add instruction on running install-deps.sh for build…

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: do not capture unused variables

also silence warnings like:

Signe-off-by: Kefu Chai <kchai@redhat.cop_btree_node.h:103:36: warning: lambda capture 'this' is not used [-Wunused-lambda-capture]
                                  [this, ec, len, &extents] (auto i) {
                                   ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

seastar: pick up document fixes

pick up changes to address the split script failure with HTML generated
with the latest pandoc.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/dev/crimson: add links to seastar documents

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37523 from Exotelis/wip-42404

mgr/dashboard: Show warning when replicated size is 1

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

doc/dev/crimson: add instruction on running install-deps.sh for building crimson

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: introduce Poseidon store

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>
Signed-off-by: Jiwoong Park <jw12.park@samsung.com>
Signed-off-by: Kyuho Son <kyuho.son@samsung.com>
Signed-off-by: Sungkyu Park <sk921.park@samsung.com>

mgr/dashboard: api doc: fix endpoint responses doc generation.

Fixes: https://tracker.ceph.com/issues/47615
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

cmake: read $NPM_REGISTRY for npm registry mirror

in case the offical npm registry is slow or unreachable.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: extract add_npm_options()

for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: read $NODE_MIRROR for Node.js mirror

in case the offical Node.js https://nodejs.org/dist/ is slow or
unreachable.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37134 from myoungwon/wip-tier-evict

osd: add tier_evict

Reviewed-by: Samuel Just <sjust@redhat.com>

mgr/dashboard: fix error when typing existing paths in the Ganesha form

- The `CephFS.ls_dir()` implementation in the backend had changed, the code in the
  UI endpoint `/ui-api/nfs-ganesha/lsdir` needs to adapt.
- Add fs_name as resource_id in `/ui-api/nfs-ganesha/lsdir/<fs_name>` to distinguish FS.
- Add more checks and unit tests.

Fixes: https://tracker.ceph.com/issues/47372
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

Merge pull request #37189 from orozery/striper-assemble-multiple-extents

librbd: support assembling results of multiple object extents

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37509 from lixiaoy1/fix_rwl

rbd/tools: fix rwl related names

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd/tools: fix rwl related names

These attributes have been renamed.

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>