rust python interops

2026-03-18
, ,

Daily Drivers

I have a working systemd service that runs the web UI of Minio with services.minio in NixOS declaratively, with run-agenix.d.mount and run-keys.mount as dependencies 014 since I encrypt all my access, security, and root credential keys for the Minio server and client and load them decrypted at runtime. It works because I set the service env for service user/group minio:minio with bash export of vars and values for the server, as those keys are NixOS options for services.minio (server side).

For the client there is no such NixOS option. I am writing a similar systemd service for the client to create a bucket declaratively after the server runs, to test the server config. But unlike the server, the access and secret keys being decrypted and loaded at runtime are not found for the client, despite being present in /run/agenix like the others. The Minio server/API callback URL variable also needs to be defined correctly for the client service env.

The solution: commands that require elevated privileges (sudo) go in ExecStartPre or ExecStartPost sections of the systemd service file, while the main service command that runs as a specific user goes in ExecStart. Configuration and environment setup is handled in a separate script called from ExecStartPre, ensuring all necessary environment variables are set before the main service starts.

Context

Rust uses programming concepts which are either implemented differently in Python or not implemented at all. Unlike Python, Rust enforces that no variable is mutable until you specify so, and implements borrow checks, macros, and lifetimes from the ground up.

This gives memory guarantees and far less boilerplate, among other things, which the build would not have had in pure Python.

PyO3 generates API 1 for Python types in Rust, and handles most SQL ops, so you almost always write a Python client to do what you were using Python for otherwise 014 especially for data ops, ML, and NLP.

Truth

I was always biased against Python, especially learning it with frameworks like Django, but its seamless utility made me look at it again. I began working around its limitations as soon as I learned about the Curry-Howard correspondence 2 014 and here we are.

Why compile for FFIs

Doing so allows multiple programs to use the same library code at runtime, reducing memory usage and improving efficiency. The necessary compiler-related tools for this include codegen 3, bindgen 4, Foreign Function Interface, wrappers, or language bindings.

If the language whose functions you export 014 and the one you want to import this dynamic shared library object into 014 allows it by design, natively or within doable data marshalling 5 and runtime overhead, you can cross the language boundary cleanly.

Qualifying with illustrations in Rust and PyO3

  1. Meta goal

    Illustrating these steps with the data ops template in Rust.

    DIY 014 visit the attached SourceHut links of snippets for the full workspace code.

  2. Proc Macro 014 I

    1. Rust library

      To generate:

      • a companion builder struct with the same fields wrapped in Option<T> to track if values are set
      • setter methods for each field on the builder struct, taking the value and storing it as Some(value), returning &mut self for chaining
      • a build method to check and construct
      • support for optional fields in the original struct, allowing those to be omitted without error

    2. A main function

      Use the builder pattern to get a struct Command with fields executable, args, env, and current_dir, and derive the builder macro to generate CommandBuilder.

      The source for the main function is below 014 checked and built:

      -- see code/chaining/app/src/main.rs on SourceHut

  3. Using PyO3 to generate API for Python types in Rust

    The lib below defines simple Rust structs, classes, methods, and functions to compile the dynamic shared library object 014 a native Python extension which you can import as a module in a Python client script to create instances of Rust classes exposed as Python classes and call those methods.6

    -- see code/ownership/src/lib.rs on SourceHut

  4. What next

    ARM64 SBC builds for data processing, machine learning, or RTOS applications 7, requiring high efficiency 8.

Footnotes

  1. API invocation is consistent and can be error-free↩︎

  2. https://en.wikipedia.org/wiki/Curry%E2%80%93Howard_correspondence#bodyContent↩︎

  3. create bindings that allow programs written in one programming language to call functions or use services from another language. It involves automatically generating the necessary code to interface with external libraries, making it easier to integrate native code into applications.↩︎

  4. automatically generate Rust bindings for C and C++ libraries, allowing Rust code to call functions and use types defined in C/C++ headers. This simplifies the process of integrating existing C/C++ libraries into Rust projects without needing to rewrite them.↩︎

  5. Marshaling the inputs and outputs of a function, from the Swift perspective, means to serialize the input values into strings, and receive the output value as a string which is then decoded into a Swift value. The Haskell perspective is dual.

    Marshaling/serializing is a robust solution to foreign language interoperability. While there is a small overhead of encoding and decoding at each function call, it almost automatically extends to all sorts of data transported across the language boundary, without being vulnerable to compiler implementation details and memory representation incompatibilities.↩︎

  6. Note that while Rust depends on the ownership model, borrowing, and freeing to manage memory, Haskell OTOH relies on lazy eval 014 more efficient than OCaml or Scheme’s GC. See the Hackage req-conduit.↩︎

  7. https://docs.rs/deltalake/latest/deltalake/↩︎

  8. https://www.youtube.com/channel/UCGKEMK3s-ZPbjVOIuAV8clQ↩︎

Webmentions

Leave a comment

Comments are verified via IndieAuth. You will be redirected to authenticate before your comment is published.