tencent-goosefs-rust-sdk

goosefs-sdk is a standalone Rust crate that talks directly to GooseFS Master and Worker services over gRPC (via tonic/prost), without any JNI or FFI bridge to the Java GooseFS client. GooseFS is Tencent Cloud’s distributed caching filesystem built on top of COS object storage, and this crate is the Layer 3 building block in a Lance → OpenDAL → GooseFS stack aimed at accelerating big-data and AI/ML workloads.

The crate exposes both a high-level FileSystem API (GoosefsFileWriter/GoosefsFileReader, backed by a shared FileSystemContext connection pool) and lower-level MasterClient/WorkerClient primitives for metadata CRUD and block streaming. Recent releases add a disk-backed local page cache with pluggable LRU/LFU/S3-FIFO eviction, a short-circuit read path that mmaps blocks directly when client and worker are co-located, HA master failover with cancel-safe leader discovery, and a Python binding with lazy, batched metadata APIs.

What You Get

• A high-level FileSystem API (GoosefsFileWriter, GoosefsFileReader, GoosefsFileInStream) built on tokio + tonic for end-to-end async file read/write pipelines
• Lower-level MasterClient, WorkerClient, WorkerMgrClient, and VersionClient primitives for direct metadata CRUD, worker discovery, and block streaming
• A disk-backed local page cache (LocalCacheManager) with striped page locks, LRU/LFU eviction, multi-directory allocation, and TTL-based background eviction
• A short-circuit read path that uses zero-copy mmap with madvise prefetch when the client is co-located with a GooseFS worker, falling back transparently to gRPC
• HA master discovery with cancel-safe leader election and connection-pooled channels using P2C (Power of Two Choices) scheduling for high-concurrency metadata RPCs
• A Python binding (via PyO3) exposing async and sync clients with batched, order-preserving fan-out APIs for metadata and lifecycle operations

Common Use Cases

• Building the storage backend layer for Lance vector datasets and other AI/ML data formats that need low-latency access to GooseFS-cached COS data
• Implementing an OpenDAL service backend for GooseFS so Rust data pipelines can address it through a unified object-storage interface
• Serving analytical workloads that benefit from local page-cache hits and short-circuit mmap reads when compute is co-located with GooseFS workers
• Replacing JNI/FFI bridges to the Java GooseFS client with a pure-Rust, async gRPC implementation to avoid JVM overhead in Rust services
• Driving batch metadata operations (stat, list, create, delete, rename) from Python via the PyO3 binding without per-call GIL overhead

Under The Hood

Architecture The crate is organized as a layered gRPC client: context::FileSystemContext owns the shared connection pool and is the only entry point that performs TCP/SASL handshakes, fs::base_filesystem/fs::filesystem implement the high-level FileSystem trait, and io:: hosts the streaming GoosefsFileWriter/GoosefsFileReader/GoosefsFileInStream types built on top of them. Below that, client:: holds per-service gRPC clients (MasterClient for metadata, WorkerClient for block I/O, WorkerMgrClient for worker discovery, VersionClient for handshakes, plus a HA-aware master_inquire module for leader election). A separate block:: module (with a short_circuit submodule) maps file ranges to block read plans and routes them via consistent hashing in WorkerRouter, falling back from short-circuit mmap reads to the standard GrpcBlockReader/GrpcBlockWriter streaming path on any error. cache:: implements the opt-in local page cache as its own subsystem (cache/store) with pluggable evictors and an io_uring-backed store on Linux. This separation means the short-circuit and caching layers can be disabled or bypassed independently without touching the core gRPC client, and callers who only need metadata operations never pull in the block-streaming or caching code paths.

Tech Stack The crate targets Rust 1.88+ and builds on tonic 0.14 (client-only, channel+codegen features, no server-side router stack) with tonic-prost/prost 0.14 for generated protobuf stubs, tokio 1.23 trimmed to rt/macros/sync/time/io-util/fs for the library (examples/tests opt into the full feature set), thiserror 2 for the crate’s Error enum, arc-swap for lock-free hot-path state (worker maps, auth state), and an optional io-uring 0.7 backend (Linux-only, cfg-gated) for the page-cache store. A metrics-pushgateway feature pulls in reqwest with rustls-tls only when Prometheus pushgateway export is needed, keeping the default dependency graph gRPC-only. Protobuf code is pre-generated and checked in under src/generated/, so downstream consumers do not need protoc unless they regenerate from proto/ via the opt-in regen-proto feature. A bindings/python workspace member provides the PyO3-based Python package alongside the Rust crate.

Code Quality The repository ships an extensive test suite: unit and hermetic integration tests run via cargo nextest run --workspace --lib --tests, plus cargo test --workspace --doc for doctests, across a Linux/Windows CI matrix (ci.yml). Dedicated end-to-end suites cover short-circuit consistency (sc_consistency.rs, sc_inv_s3.rs, short_circuit_e2e.rs), page-cache correctness (page_cache_e2e.rs, page_cache_consistency.rs), auth/retry behavior, connection reuse, and write-degradation paths, with a separate ci_integration.yml workflow for cluster-backed e2e tests against a Docker GooseFS fixture. The Python binding has its own ruff-linted, pytest-covered suite. Error handling is centralized in a thiserror-derived Error enum with explicit From conversions rather than ad-hoc unwrap/panic!, and Drop implementations for writer/reader handles perform best-effort cleanup (aborting background tasks, invoking remove_blocks) instead of silently leaking resources.

API Design The crate favors a small, high-level surface for common cases (GoosefsFileWriter::write_file_with_context / GoosefsFileReader::read_file_with_context) that reuses a single shared FileSystemContext connection pool, avoiding the common footgun of re-establishing TCP/SASL per call. Lower-level MasterClient/WorkerClient types are still exposed for callers that need direct metadata or block control. The Python binding mirrors this with sync and async client classes and adds batch entry points (batch_get_status, batch_exists, batch_create_file, etc.) that fan out over futures::stream::buffered while preserving Rust’s single-op BaseFileSystem as the low-level building block for advanced fan-out. Configuration is layered (builder methods, GOOSEFS_* env vars, and properties files) rather than requiring one canonical style, which adds some surface area but keeps the SDK usable in both programmatic and ops-driven deployment contexts.