A Rust-Python thing I am working on. Apache 2 licence
Key takeaways
- Overview Nairobi OS is a high-performance, distributed data science infrastructure designed for extreme resource efficiency.
- Zero-Copy Ingestion: Hardware-accelerated data loading using io_uring and 1GB Huge Pages.
- git clone https://github.com/KevinKenya/nairobi-connector-open-source cd nairobi-connector-open-source Setup Virtual Environment:
Overview Nairobi OS is a high-performance, distributed data science infrastructure designed for extreme resource efficiency. It enables processing of massive datasets in constrained environments (Edge, Io T, Serverless) by leveraging a specialized Rust-based refinery daemon. By utilizing kernel-level features such as io_uring, memfd, and Huge Pages, Nairobi OS achieves sub-millisecond IPC overhead and zero-copy data pipelines.
Zero-Copy Ingestion: Hardware-accelerated data loading using io_uring and 1GB Huge Pages. Hardware-Accelerated Visualization: Interactive Jupyter plotting via the Lagos Vision engine (wgpu and egui). Fused Analytics Pipeline: Ingest, crunch, and correlate data in a single D-Bus round trip. Kernel-Bypass Performance: Vectorized analytics leveraging Polars and Rayon for maximum hardware saturation. Sovereign Interface: A fluent Python API that hides the complexity of low-level IPC and memory management. Architecture Nairobi OS is built on a triad of specialized components connected via D-Bus and shared memory:
[ Data Source ] -> (io_uring/Huge Pages) -> [ Axum Refinery ] | (D-Bus / memfd / iceoryx2) | [ Nairobi Hub ] / \ [ Nairobi Python ] [ Lagos Vision ] | | [ Jupyter Notebook ] <-> [ Visual Output ] Installation Prerequisites Operating System: Linux or WSL2 (Kernel 5.10+ required for io_uring and memfd). Rust: 1.70+ Python: 3.10+ System Libraries: sudo apt-get update && sudo apt-get install -y \ build-essential \ pkg-config \ libdbus-1-dev \ python3-dev \ dbus-x11 \ libosmesa6-dev \ mesa-utils Build from Source Clone the Repository: