useR! 2024

The dplyr package is a powerful tool for data manipulation in R. It provides a consistent grammar for manipulating data frames and is widely used by data scientists and analysts. However, dplyr requires that the entire dataset fit into memory, and can be slow for large datasets. The duckdb package is a new in-memory database that is designed to be blazing fast and efficient for analytical workloads. A relational frontend, modeled after Codd's relational algebra, is provided alongside an SQL interface. The new duckplyr package uses this relational frontend: unlike dbplyr, which translates dplyr commands into SQL, duckplyr translates dplyr commands into relational algebra. The package has been designed to be a fully compatible drop-in replacement for dplyr from day one. Operations are run in duckdb when possible, and fall back to dplyr when not. The project's goal is to speed up more and more dplyr verbs, R functions, and data types, towards becoming the primary implementation of the dplyr grammar of data manipulation. In this talk, I will present duckdb and duckplyr, and discuss the design of duckplyr and the supporting tools.