Technical Tutorials

Python has become the lingua franca of data science, yet the ecosystem's reliance on pure Python often masks a critical bottleneck: performance. For heavy numerical computations, data scientists frequently turn to libraries built on top of C, C++, or Fortran, such as NumPy, SciPy, and Pandas. However, distributing these libraries presents a unique challenge for maintainers. Unlike standard Python packages, libraries with native extensions cannot simply be installed from source on every user's machine without requiring a complex build environment, compiler toolchains, and compatible dependencies. The result is often "source-only" distributions that are slow to install and prone to build failures across different operating systems.

This is where modern Python packaging tools step in. The traditional setuptools configuration with setup.py scripts is increasingly being replaced by build backends that adhere to PEP 517 and PEP 621. Among these, scikit-build-core has emerged as the gold standard for projects requiring C/C++ extensions. In this guide, we will explore how to leverage scikit-build-core to create robust, architecture-specific binary wheels, ensuring your data science library delivers peak performance to every user, regardless of their environment.

The Evolution of Python Packaging

To understand why scikit-build is the solution, we must first acknowledge the pain points of the past. Previously, developers relied on setup.py files that mixed build logic with installation logic. This procedural approach often led to "build-once, deploy-anywhere" failures when the build environment didn't match the target environment. Furthermore, creating wheels (binary distributions) for multiple platforms (Linux, macOS, Windows) and architectures (x86_64, ARM64) required intricate, manual configuration.

Python packaging has shifted towards declarative configurations. PEP 621 allows project metadata to be defined in pyproject.toml, decoupling build logic from the project configuration. Scikit-build-core is a modern CMake-based build backend designed specifically for this declarative workflow. It handles the complexities of invoking CMake, managing compilers, and bundling binaries into wheels automatically.

Project Structure and Configuration

Setting up a project with scikit-build-core begins with a well-structured directory layout. You need a source folder for your Python code, a src directory for your C/C++ code, and a pyproject.toml file that tells the build system what to do.

project-root/
├── src/
│   ├── my_package/
│   │   ├── __init__.py
│   │   └── _core.c
├── pyproject.toml
└── build_requirements.txt

The heart of the configuration lies in pyproject.toml. We define the build backend as scikit_build_core and specify the CMake requirements. This replaces the old setup.py entirely.

[build-system]
requires = ["scikit-build-core", "cmake>=3.15", "setuptools"]
build-backend = "scikit_build_core.build"

[project]
name = "fast-data-lib"
version = "0.1.0"
description = "High-performance data processing using C extensions"
requires-python = ">=3.9"
dependencies = [
    "numpy>=1.24.0",
]

[tool.scikit-build]
cmake.version = "3.20.0"
build.type = "Release"
wheel.install-dir = "fast_data_lib"
# This tells scikit-build to build for all compatible platforms
manylinux.manylinux_x86_64 = true

Writing the C-Extension with CMake

Once the Python configuration is set, the CMake setup takes over. We need a CMakeLists.txt file in the root of our project that describes how to compile the C source and link it as a Python module. Scikit-build-core automatically passes variables like PYTHON_INCLUDE_DIRS and PYTHON_LIBRARIES to CMake, simplifying the linkage process.

cmake_minimum_required(VERSION 3.20)
project(fast_data_lib C)

# Use scikit-build-core's Python configuration
include_directories(${PYTHON_INCLUDE_DIRS})

# Define the extension module
python_add_library(_core MODULE src/my_package/_core.c)

# Link against standard Python libraries
target_link_libraries(_core ${PYTHON_LIBRARIES})

# Ensure the library is installed to the correct location
set_target_properties(_core PROPERTIES
    LIBRARY_OUTPUT_DIRECTORY "${PROJECT_SOURCE_DIR}/src/my_package"
)

Here, the python_add_library command is a high-level macro provided by CMake that handles the creation of a shared library (`.so` on Linux, `.dll` on Windows, `.dylib` on macOS) compatible with the Python interpreter. The extension name (_core) must match the import name in your Python code.

Building and Distributing Binary Wheels

The power of scikit-build-core lies in its ability to build wheels for various platforms. You can build a local wheel using:

pip wheel . --no-build-isolation

However, for true cross-platform distribution, we recommend using cibuildwheel, a tool that runs inside a CI/CD pipeline (like GitHub Actions). It automatically builds wheels for Linux, macOS, and Windows across all supported Python versions. When you upload these wheels to PyPI, pip will intelligently select the correct binary wheel for the user's machine, avoiding the need to compile anything locally.

# Example .github/workflows/build.yml snippet
jobs:
  build_wheels:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        os: [ubuntu-latest, macos-latest, windows-latest]
    steps:
      - uses: actions/checkout@v4
      - name: Build wheels
        uses: pypa/cibuildwheel@v2.19
      - uses: pypa/gh-action-pypi-publish@release/v1

Conclusion

Delivering high-performance data science libraries requires more than just optimized algorithms; it requires a robust packaging strategy. By migrating from legacy setup.py scripts to the modern, declarative approach of scikit-build-core, developers can streamline the distribution of C-extensions and binary wheels. This ensures that your users get the speed benefits of native code without the friction of complex local compilation environments. As the Python ecosystem continues to evolve, adopting these tools is not just a best practice—it is a necessity for maintaining high-quality, accessible scientific software.