useR! 2024

The {mirai} package provides the latest in parallel computing technology, and the {crew} package extends {mirai} to high-performance computing environments. {mirai} and {crew} offer distributed computing for demanding workflows such as machine learning, simulation, and Bayesian data analysis. Each package is designed according to the ‘moju-kapu’ (modular encapsulation) concept. Interfaces are simple and easy to use standalone, while also inviting third-party extensions and integrations. The result is a multi-tiered, flexible framework with convenient entry points at every level of complexity. Together, {mirai} and {crew} form an elegant and performant foundation for an emerging landscape of asynchronous and parallel programming tools in R. They already provide new backends for {parallel}, {promises}, {plumber}, {targets}, and Shiny, as well as new high-level interfaces such as {crew.cluster} for traditional clusters and {crew.aws.batch} for the cloud. This talk explores the layered engineering approach of {mirai} and {crew} through the lens of ‘moju-kapu’, in which each package is designed to be at the same time self-sufficient, but also a modular part of an encapsulating system.