osd: fix ObjectContextRef reference leak

After start_dedup() is called on the object,
several CREATE_OR_GET_REFs are sent.
In the meantime, the ObjectContextRef can be evicted
from the cache while the deduplication is in progress.

To avoid this situation, this commit adds ObjectContextRef
to ManifestOp as FlushOp works.

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

Merge pull request #43870 from benhanokh/restore_alloc_file

NCB::refresh allocation-file after FSCK remove

BlueStore: Fix a bug when FSCK is invoked in mount()/umount()/mkfs() with DEEP option
Fixes: https://tracker.ceph.com/issues/53185
NCB mishandles fsck DEEP in mount()/umount()/mkfs() case causing it to remove the allocation-file without destaging a new copy (which will cost us a full rebuild on startup)
There are also few confiliting calls to open_db()/close_db() passing inconsistent read-only flag

We fix both issues by storing open-db type (read-only/read-write) and using it for close-db (which won't pass read-only flag anymore)
We also move allocation-file destage to close-db so it will be refreshed after being removed by fsck and such

Signed-off-by: Gabriel Benhanokh <gbenhano@redhat.com>

Merge PR #44155 into master

* refs/pull/44155/head:
mgr: limit changes to pg_num

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #44108 into master

* refs/pull/44108/head:
mgr: fix locking for MetadataUpdate::finish

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #44204 from songtongshuai/sts_ceph

test/librbd: add get_group test

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #44189 from zdover23/wip-doc-2021-12-02-documenting-ceph

doc/start: update documenting-ceph.rst (1 of x)

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge PR #44017 into master

* refs/pull/44017/head:
mgr/cephadm: Do not propogate access logs from cherrypy

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #44139 from linuxbox2/wip-rgw-lcselect

rgwlc: permit lifecycle processing for a single bucket

Merge PR #44132 into master

* refs/pull/44132/head:
mgr/prometheus: define module options for standby

Reviewed-by: Laura Flores <lflores@redhat.com>

test/librbd: add get_group test

Signed-off-by: songtongshuai_yewu <songtongshuai_yewu@cmss.chinamobile.com>

doc/start: update documenting-ceph.rst (1 of x)

This PR updates the content on documenting-ceph,
which is, as of December 2021, in need of an
update.

This is the first of what I estimate to be three
to five PRs against this .rst file.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #44103 from majianpeng/librbd-pwl-flush-by-blockgurad

librbd/cache/pwl: Using BlockGuard control overlap ops order when flu…

Reviewed-by: Mykola Golub mgolub@suse.com
Reviewed-by: Sunny Kumar <sunkumar@redhat.com>
Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #44144 from majianpeng/librbd-fix-discard-granularity

librbd: fix discard granularity for pwl cache

Reviewed-by: mgolub@suse.com
Reviewed-by: Sunny Kumar <sunkumar@redhat.com>
Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #43848 from dvanders/doc_bench

doc: add disk benchmarking and cache tuning recommendations

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #43691 from curtbruns/use_optimal_for_min_alloc

os/bluestore: Set min_alloc_size to optimal io size

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #43578 from ronen-fr/wip-rf-log-options

common: hide internal logger configuration strings from clients

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #43493 from myoungwon/wip-52872

test: increase retry duration when calculating manifest ref. count

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #43944 from aclamk/wip-aclamk-fix-crush-location-hook-50659

crush: Fix segfault in update_from_hook

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #44194 from ceph/wip-yuriw-crontab-master

qa/tests: switch all gibba machines to smithi

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #44178 from Huber-ming/admin_lc_fix

radosgw-admin: fix error message of OPT::LC_RESHARD_FIX

qa/tests: switch all gibba machines to smithi

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #43336 from ifed01/wip-fix-bluefs-volumes-ops

qa/osd-bluefs-volume-ops: fix bluefs volumes ops test case

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

mgr/cephadm: Do not propogate access logs from cherrypy

The only messages we're really interested in are errors that
would come from the error logs. The acces logs only provide
messages like

- [18/Nov/2021:23:55:32] "POST /data HTTP/1.1" 200 - "" "Python-urllib/3.8"

which we don't want spammed to the log, especially since they are
logged at INFO level

Signed-off-by: Adam King <adking@redhat.com>

Merge PR #44035 into master

* refs/pull/44035/head:
mgr/cephadm: less log noise when config checks fail

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #44089 from benhanokh/ncb_fsck_fix

os/bluestore: bug-fix for NCB-FSCK

doc: add disk benchmarking and cache recommendations

Advise operators on how to benchmark devices for BlueStore, and how to
tune the volatile write cache for optimal OSD performance.

Fixes: https://tracker.ceph.com/issues/53161
Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>

Merge pull request #44063 from vshankar/tr-52487

qa: wait for purge queue operations to finish

Reviewed-by: Venky Shankar <vshankar@redhat.com>

radosgw-admin: fix error message of OPT::LC_RESHARD_FIX

Signed-off-by: Huber-ming <zhangsm01@inspur.com>

Merge PR #44125 into master

* refs/pull/44125/head:
qa/suites/rados/thrash-old-clients: use better-support cephadm distro/podman

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

mgr: fix locking for MetadataUpdate::finish

We need to hold the DaemonState lock here since we are both reading and
writing its content.

Fixes: https://tracker.ceph.com/issues/53393
Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: less log noise when config checks fail

We are already raising health alerts--there is no need to spam the log
every few seconds when these checks are evaluated.

Signed-off-by: Sage Weil <sage@newdream.net>

rgwlc: optimize single-bucket lifecycle processing

Looks up the shard index of the corresponding bucket, and only
buckets in the corresponding shard are considered for processing.
This has a side effect of matching buckets by id, and also adds
support for --tenant.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #44161 from neha-ojha/wip-fix-upgrades

qa/suites/upgrade/octopus-x: bunch of fixes and cleanup

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>
Reviewed-by: Sridhar Seshasayee <sseshasa@redhat.com>

Merge pull request #43149 from sebastian-philipp/cephadm-force-last-admin

mgr/cephadm: Add client.admin keyring when upgrading from older version

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sage Weil <sage@newdream.net>

Merge pull request #44109 from sebastian-philipp/doc-crush-types

doc/cephadm: host location: add link to types

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #44134 from liewegas/cephadm-device-enhanced-scan

mgr/cephadm: avoid repeated calls to get_module_option

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #44071 from tchaikov/wip-atomic-mips64

cmake: test for 16-byte atomic support on mips also

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #43540 from fengchunsong/dpdk-test

test/msgr: remove DPDK Non-runtime configure items

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #43981 from lxbsz/wip-53216

qa/cephfs: correct the parameters' order

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #44087 from guits/guits-fix-cv-rootfs

ceph-volume: remove --root param from nsenter cmd

Reviewed-by: Sébastien Han <seb@redhat.com>

Merge pull request #44143 from devlikai/master

doc/mgr/diskprediction: fix a typo.

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #44116 from lxbsz/caps_doc1

doc: fix the style of the cephfs capability doc

Reviewed-by: Venky Shankar <vshankar@redhat.com>

librbd/cache/pwl: it should in apply_metadata set discard_granularity for pwl cache.

Function apply_meta can overwrite discard_granularity_bytes
based on option.

Fixes:https://tracker.ceph.com/issues/53434

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #44129 from spdfnet/patch-1

doc: fix typo in cephadm host management

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

qa: correct the parameters' order

The parameters' order is incorrect and missing the client_config.

Introduced-by: 242585656c6bc282f5adbd073e83bafa86b5c0d2
Fixes: https://tracker.ceph.com/issues/53216
Signed-off-by: Xiubo Li <xiubli@redhat.com>

qa: move the optional client_config parameter to the end

Fixes: https://tracker.ceph.com/issues/53216
Signed-off-by: Xiubo Li <xiubli@redhat.com>

qa: rename and save the client_config for kernel mount

Fixes: https://tracker.ceph.com/issues/53216
Signed-off-by: Xiubo Li <xiubli@redhat.com>

qa: wait for purge queue operations to finish

TestFragmentation.test_deep_split relies on `num_strays`
to reach zero expecting that the purge threads would
have deleted the directory entries. However, checking
`num_strays` cannot be relied on since PurqeQueue merely
journals the purge item (see PurgeQueue::push) followed
by the StrayManager marking the stray as removed thereby
accounting `num_strays`.

So, add an additional condition to check if the purge
threads have finished processing items.

Fixes: http://tracker.ceph.com/issues/52487
Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #44038 from lxbsz/wip-53082

client: fix crash when iterating and deleting sessions

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #43878 from jtlayton/wip-53214

qa: account for split of the kclient "metrics" debugfs file

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #43850 from batrick/i53194

mds: defer messages to bootstrapping ranks

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #43297 from yongseokoh/test-dir-max-entries

qa: add mds_dir_max_entries workunit test case

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #41334 from vshankar/wip-kcephfs-new-mount-syntax

mount: introduce new device mount syntax

Reviewed-by: Venky Shankar <vshankar@redhat.com>

qa/*/octopus-x/stress-split-erasure-code-no-cephadm: set quincy flags

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/suites/upgrade/octopus-x/stress-split-no-cephadm: remove msgr2

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa: test upgrades with hybrid allocator

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa: rename octopus install correctly

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa: remove leftovers from nautilus

pglog_hardlimit and msgr2

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/suites/upgrade/octopus-x/stress-split-no-cephadm: set quincy flags

not pacific

Signed-off-by: Neha Ojha <nojha@redhat.com>

mgr: limit changes to pg_num

We need to avoid making drastic changes to pg_num that outpace pgp_num or
else we will may hit the per-osd pg limits.

Fixes: https://tracker.ceph.com/issues/53442
Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #42710 from yehudasa/wip-rgw-mgr-module

mgr/rgw: new rgw manager module

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge PR #44140 into master

* refs/pull/44140/head:
python-common/ceph/deployment/drive_group: fix 'orch ls --format yaml'

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

python-common/rgw: fix style issues

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

doc/rgw: fix docs build

Workaround rgw modules conflict, as there are two separate modules named
rgw: src/pybind/rgw, src/pybind/mgr/rgw

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #44072 from ronen-fr/wip-rf-latescrub-count

qa/standalone: osd-scrub-repair.sh: fix expected "not scrubbed since"…

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sridhar Seshasayee <sseshasa@redhat.com>

Merge pull request #44118 from sebastian-philipp/cephadm-inventory-changed-while-iterated

mgr/cephadm: Inventory: Fix `dictionary changed size during iteration `

Reviewed-by: Michael Fritch <mfritch@suse.com>

qa/suites/rados/thrash-old-clients: use better-support cephadm distro/podman

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #44115 from rhcs-dashboard/fix-tooltip-fast-diff

mgr/dashboard: avoid tooltip if disk_usage=null and fast-diff enabled

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #44083 from wangbo-yw/wangbo-yw

mgr/dashboard: add some test for controllers/pool.py

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #43855 from zhangmengqianyw/zmq-unittest

mgr/dashboard:add unittest in test_osd.py

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #44082 from pcuzner/fix-prometheus-timings

mgr/prometheus: Fix the per method stats exported

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/mgr/diskprediction: fix a typo.

doc: remove extra comma.

This commit remove extra comma of "To disable prediction,:".

Fixes: https://tracker.ceph.com/issues/53433
Signed-off-by: devlikai <likai_lc@inspur.com>

Merge pull request #43529 from curtbruns/rgw-lua-storageclass

rgw/lua: allow read/write of StorageClass field

Merge pull request #42504 from arjune123/rgw-bug-fixes

rgw/notification: assigning the value of zonegroup to awsRegion

doc, man: document new device syntax for kclient mount

Fxies: http://tracker.ceph.com/issues/47277
Signed-off-by: Venky Shankar <vshankar@redhat.com>

qa: use v1 mount-syntax for upgrade tests

Signed-off-by: Venky Shankar <vshankar@redhat.com>

test: mount kclient using new-style (v2) syntax

But, do not throw away the old style mount syntax since we would
want to continue testing it since users (scripts) might still be
using it.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mount: use mds_mode=prefer-crc with new-style mount syntax if possible

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mount: optionally do not fall-back to old-style mount syntax

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mount: introduce new mount syntax

Old mount device (source) have the following problems:

- mounts to the same cluster but with different fsnames
  and/or creds have identical device string which can
  confuse xfstests.

- device shown in /proc/mounts is different that what
  is used to mount.

New proposed syntax is as follows:

  # mount -t ceph cephuser@<cluster-fsid>.mycephfs2=/path

The cluster-id is optional but is always passed to the kernel
via mount() syscall (by fetching it from cluster configuration
file if available).

The mount helper tries the mounting the file system with
the new device syntax, falling back to using the old
syntax if required.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mount: record cluster fsid when reading ceph config file

This will be required when switching to new mount device
syntax when cluster fsid will be passed to kernel as mount
option.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mount: accept monitor host (`mon_host`) mount option

With new mount device syntax monitor host(s) can be passed
during mount as option parameter. This option is not mandatory
as it can be fetched from the cluster configuration file if
available.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

python-common/ceph/deployment/drive_group: fix 'orch ls --format yaml'

In f76c02a658ed1de54a380381932bb58b6ab91d41 we may leave out the placement
key when it is empty.  If we take that path, we need to tolerate specs
that are missing that key.

Fixes: f76c02a658ed1de54a380381932bb58b6ab91d41
Signed-off-by: Sage Weil <sage@newdream.net>

rgwlc: permit lifecycle processing for a single bucket

Permit a --bucket option to be passed to radosgw-admin lc process,
and propagate the bucket name to lifecycle processing, and process
only the named bucket if one is provided.

Fixes: https://tracker.ceph.com/issues/53430
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #43760 from yaarith/release-checklists

doc/dev/release-checklist: add manual review of telemetry reports

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mgr/cephadm: avoid repeated calls to get_module_option

We already stash these as MgrModule members.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge PR #44018 into master

* refs/pull/44018/head:
mon: fix quorum_age() regression

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #44030 into master

* refs/pull/44030/head:
mgr/cephadm: add some debug output for serve loop
ceph-volume: adjust arguments for 'ceph-volume raw activate'
ceph-volume: add raw support for db/wal for list and activate

Reviewed-by: Sébastien Han <seb@redhat.com>

Merge PR #44107 into master

* refs/pull/44107/head:
qa/tasks/cephadm_cases/test_cli: fix test_daemon_restart

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

mgr/prometheus: define module options for standby

We need to define the options for the standby class as well in order for
it to successfully make use of the default value for
standby_error_status_code (and stadnby_behavior).

Fixes: https://tracker.ceph.com/issues/53287
Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: add some debug output for serve loop

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-volume: adjust arguments for 'ceph-volume raw activate'

Take a list of devices, so that we can selectively activate a raw osd
with db/wal.

Remove the argument type kludge introduced in 2c228a9a409176c0f1679f176443fd3ead219c7a
since it is no longer needed.

Note that we're making this change because (1) it allows db/wal and (2)
because there are no known users of 'raw activate'.  The only known user
is via 'ceph-volume activate' and we've fixed that caller in this commit.

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-volume: add raw support for db/wal for list and activate

Currently 'prepare' doesn't support db/wal, but we want it in list and
activate because 'ceph-volume activate ...' tries raw before lvm.

Note that I'm not sure we really want to accept --block.db and --block.wal
here at all.

Fixes: 3d7ceec684b0ac5b83fae4c397b134236fac485e
Signed-off-by: Sage Weil <sage@newdream.net>

qa/tasks/cephadm_cases/test_cli: fix test_daemon_restart

We cannot schedule a daemon start if there is another daemon action
with a higher priority (including stop) scheduled.  However,
that state isn't cleared until *after* the osd goes down, the
systemctl command returns, and mgr/cephadm gets around to updating
the inventory scheduled_daemon_action state.

Semi-fix: (1) wait for the orch status to change, and then (2)
wait a few more seconds after that.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #44100 from adk3798/infer-config-fix

cephadm: only infer conf from mon if fsid matches

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #44101 from adk3798/agent-down-multiplier

mgr/cephadm: agent: allow agent down multiplier to be configured

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

doc: fix typo in cephadm host management

Merge pull request #42378 from sebastian-philipp/no-grafana-admin

mgr/cephadm: Add GrafanaSpec.initial_admin_password

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

mgr/cephadm: serve.py: put _write_client_files into it's own method

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>