From 68990bad356d99869185257a12a6cf49b6644565 Mon Sep 17 00:00:00 2001 From: Kefu Chai Date: Tue, 27 Jan 2026 15:08:28 +0800 Subject: [PATCH] cmake: migrate Python module installation from setup.py to pip Replace 'setup.py install' with 'pip install --use-pep517' to fix Cython compilation failures and eliminate deprecation warnings. Problem Statement: The build process for Cython modules involves preprocessing .pyx files (e.g., generating rbd_processed.pyx from rbd.pyx) and then cythonizing with specific compiler_directives. The previous approach using separate 'setup.py build' and 'setup.py install' commands caused this failure: ``` Error compiling Cython file: ------------------------------------------------------------ ... """ name = cstr(name, 'name') cdef: rados_ioctx_t _ioctx = convert_ioctx(ioctx) char *_name = name librbd_progress_fn_t _prog_cb = &no_op_progress_callback ^ ------------------------------------------------------------ rbd_processed.pyx:781:44: Cannot assign type 'int (*)(uint64_t, uint64_t, void *) except? -1' to 'librbd_progress_fn_t'. Exception values are incompatible. Suggest adding 'noexcept' to type 'int (uint64_t, uint64_t, void *) except? -1'. ``` This occurs because: 1. 'setup.py build build_ext' successfully preprocesses and cythonizes with compiler_directives from setup.py's cythonize() call 2. 'setup.py install' internally triggers a rebuild that: - Regenerates the preprocessed .pyx files - Re-runs cythonize() through Cython.Distutils.build_ext - Does NOT apply the compiler_directives from setup.py - Fails on the regenerated files missing required directives New Options Explained: --use-pep517: Addresses deprecation warning: ``` DEPRECATION: Building 'rados' using the legacy setup.py bdist_wheel mechanism, which will be removed in a future version. pip 25.3 will enforce this behaviour change. ``` Uses the modern PEP 517 build backend which: - Performs a single build pass with all compiler_directives applied - Prevents the implicit rebuild that caused CompileError - Future-proofs against pip 25.3+ which will require this --no-build-isolation: Ensures that environment variables set by CMake are respected: - CC, LDSHARED (compiler toolchain) - CPPFLAGS, LDFLAGS (compilation flags) - CYTHON_BUILD_DIR, CEPH_LIBDIR (build paths) Without this flag, pip would create an isolated build environment that ignores these critical build settings. --no-deps: Prevents pip from attempting to install Python dependencies listed in setup.py's install_requires. All dependencies are managed by CMake and the distribution's package manager, not pip. --ignore-installed: Addresses installation error when DESTDIR is set: ``` ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: '/usr/lib/python3/dist-packages/rados-2.0.0.egg-info' OSError: [Errno 18] Invalid cross-device link: '/usr/lib/python3/dist-packages/rados-2.0.0.egg-info' -> '/tmp/pip-uninstall-...' ``` This error occurs because pip detects an existing system installation and tries to uninstall it before installing to DESTDIR. With --ignore-installed, pip skips the uninstall step and directly installs to the DESTDIR staging directory, which is the correct behavior for packaging. Removed Options: `--install-layout=deb`: This Debian-specific patch to 'setup.py install' is no longer needed. Modern pip automatically detects the distribution and uses the correct layout (dist-packages on Debian, site-packages on RPM distros). `--single-version-externally-managed`: This option was specific to 'setup.py install' to prevent egg installation. With pip, this is handled automatically. `--record /dev/null`: No longer needed as pip manages installation records internally. `egg_info --egg-base`: Not needed with pip as metadata is generated automatically during the build process. Working Directory Change: Changed from `CMAKE_CURRENT_SOURCE_DIR` to `CMAKE_CURRENT_BINARY_DIR` to keep pip's temporary files and logs in the build directory rather than polluting the source tree. Additional Dependencies: Since the build process uses pip and creates a wheel distribution, we need to add `pip` and `wheel` Python modules as build dependencies. Future Improvements We considered implementing a custom `build_templates` command or using setuptools' `sub_commands` mechanism to avoid regenerating `*_processed.pyx` files on every build (tracking dependencies via file modification times or hash-based checks). However, to keep `setup.py` simple and maintainable, we've deferred this optimization for future work. The current solution using `pip install --use-pep517` ensures correct builds without additional complexity. This solution works correctly for both Debian and RPM packaging workflows, both of which use DESTDIR-based staged installations. Fixes: 719b749846 Signed-off-by: Kefu Chai --- ceph.spec.in | 2 ++ cmake/modules/Distutils.cmake | 21 ++++++++++----------- debian/control | 2 ++ 3 files changed, 14 insertions(+), 11 deletions(-) diff --git a/ceph.spec.in b/ceph.spec.in index 9b154a72a9a..f0b6b921e84 100644 --- a/ceph.spec.in +++ b/ceph.spec.in @@ -291,6 +291,8 @@ BuildRequires: python%{python3_pkgversion} BuildRequires: python%{python3_pkgversion}-devel BuildRequires: python%{python3_pkgversion}-setuptools BuildRequires: python%{python3_pkgversion}-Cython +BuildRequires: python%{python3_pkgversion}-pip +BuildRequires: python%{python3_pkgversion}-wheel BuildRequires: snappy-devel BuildRequires: sqlite-devel BuildRequires: sudo diff --git a/cmake/modules/Distutils.cmake b/cmake/modules/Distutils.cmake index 76b5fb23d91..fc8501807a4 100644 --- a/cmake/modules/Distutils.cmake +++ b/cmake/modules/Distutils.cmake @@ -137,25 +137,24 @@ function(distutils_install_cython_module name) set(ENV{CYTHON_BUILD_DIR} \"${CMAKE_CURRENT_BINARY_DIR}\") set(ENV{CEPH_LIBDIR} \"${CMAKE_LIBRARY_OUTPUT_DIRECTORY}\") - set(options --prefix=${CMAKE_INSTALL_PREFIX}) + set(options + --prefix=${CMAKE_INSTALL_PREFIX} + --use-pep517 + --no-build-isolation + --no-deps + --ignore-installed) if(DEFINED ENV{DESTDIR}) - if(EXISTS /etc/debian_version) - list(APPEND options --install-layout=deb) - endif() list(APPEND options --root=\$ENV{DESTDIR}) else() list(APPEND options --root=/) endif() execute_process( COMMAND - ${Python3_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/setup.py - build ${maybe_verbose} --build-base ${CYTHON_MODULE_DIR} - --build-platlib ${CYTHON_MODULE_DIR}/lib.3 - build_ext --cython-c-in-temp --build-temp ${CMAKE_CURRENT_BINARY_DIR} --cython-include-dirs ${PROJECT_SOURCE_DIR}/src/pybind/rados - install \${options} --single-version-externally-managed --record /dev/null - egg_info --egg-base ${CMAKE_CURRENT_BINARY_DIR} + ${Python3_EXECUTABLE} -m pip install + \${options} ${maybe_verbose} - WORKING_DIRECTORY \"${CMAKE_CURRENT_SOURCE_DIR}\" + ${CMAKE_CURRENT_SOURCE_DIR} + WORKING_DIRECTORY \"${CMAKE_CURRENT_BINARY_DIR}\" RESULT_VARIABLE install_res) if(NOT \"\${install_res}\" STREQUAL 0) message(FATAL_ERROR \"Failed to build and install ${name} python module\") diff --git a/debian/control b/debian/control index 18eb8429867..3d4c84984dd 100644 --- a/debian/control +++ b/debian/control @@ -110,7 +110,9 @@ Build-Depends: automake, python3-onelogin-saml2 , python3-jinja2, python3-markupsafe, + python3-pip, python3-setuptools, + python3-wheel, python3-sphinx, python3-venv, python3-yaml, -- 2.47.3