roadmap.qmd

---
title: Project Roadmap
description: "Lots of backward-compatible improvements and extensions ahead"
preview: images/site_preview.png
format: html
---

Below are some^[This list is manually curated and may not be comprehensive.  Please see the issue trackers and milestones of the different package repositories for a more up-to-date view, e.g. <https://github.com/futureverse/future/milestones>.] of the bigger milestones on the roadmap.  The issues linked that are examples of feature requests from end-users are prefixed 'FR', which will be resolved by the other non-FR issues.


## Core Future API

<style>
.list-unstyled ul {
  list-style: none;
  padding-left: 2ex;
}

.list-unstyled li {
  display: flex;       /* Aligns icon and text in a row */
  align-items: flex-start; /* Keeps icon at the top of multi-line text */
  gap: 0.5rem;         /* Space between icon and text */
  margin-bottom: 0.5rem;
}

</style>

::: {.list-unstyled}
* {{< bi square-fill >}}<span> Restarting of failed or terminated futures, e.g. `reset(f)` and `v <- value(f)`.  Also automatic restarting of futures when parallel worker dies [~~[future#FR188]~~, ~~[future#205]~~, ~~[future.callr#FR11]~~, ~~[parallelly#32]~~] (solved in **future** 1.40.0; future backends will be updated accordingly)</span>
* {{< bi square-fill >}}<span>Cancellation of futures with interruption, i.e. `cancel(f)`. All futures can be canceled, but not all can be interrupted. If a future backend does not support interrupting futures, then the interrupt requested is silently ignored. [~~[future#93]~~, ~~[future#FR213]~~, ~~[future.batchtools#FR27]~~, ~~[parallelly#33]~~] (solved in **future** 1.49.0; all future backends support cancellation and interruption as of 2025-08-25)</span>
* {{< bi square >}}<span>Troubleshooting: Improve debugging of futures that produce errors, e.g. capture call frames in parallel workers and let the user inspect them via tools such as `utils::browser()` and `utils::recover()` [[future#253]]</span>
* {{< bi square >}}<span>Support for controlling R options and environment variables via `future()` [[future#480]]</span>
* {{< bi square >}}<span>Flattening nested parallelism: Process nested futures via a common future queue instead of via nested future plans [[future#FR361]]</span>
:::



## Resource Management and Scheduling

::: {.list-unstyled}
* {{< bi square-fill >}}<span>Profiling: Support for built-in _time_ profiling to identify bottlenecks and improve overall performance [~~[future#142]~~, ~~[future#FR437]~~]</span>
* {{< bi square-half >}}<span>Profiling: Support for built-in _memory_ profiling, e.g. **autometrics** [[future#59]]</span>
* {{< bi square-half >}}<span>Random number generation: Support for custom, alternative RNGs to the built-in L'Ecuyer-CMRG algorithm</span>
* {{< bi square-half >}}<span>Marshaling: Make it possible to parallelize also some of the [object types that are currently "non-exportable"](https://future.futureverse.org/articles/future-4-non-exportable-objects.html), e.g. objects by packages such as **DBI**, **keras**, **ncdf4**, **rstan**, **sparklyr**, **xgboost**, and **xml2** (Project: **[marshal]** and R Consortium ISC **[Marshaling and Serialization in R]** Working Group)</span>
* {{< bi square-half >}}<span>Shared memory: Support for low-memory sharing of globals among worker processes running on the same machine by leveraging the **mori** package)</span>
* {{< bi square >}}<span>Content-addressable shared memory, e.g. efficient local and remote caching</span>
* {{< bi square-half >}}<span>Resources: Make it possible to declare "resources" that a future requires in order to be resolved, e.g. memory, runtime, CPUs, GPUs, networking & internet, files, sandboxing, etc. [~~[doFuture#FR63]~~, [future#FR181], [future#FR301], [future#FR346], [future#FR430], [future#FR450], ~~[parallelly#18]~~, [future.batchtools#FR75]]</span>
* {{< bi square >}}<span>Scheduling: Add generic support for queuing such that futures can be queued on the local computer, on a remote scheduler, or in a peer-to-peer scheduler among collaborators [[future#FR256], [future.apply#FR63], [future.batchtools#FR23]]</span>
* {{< bi square >}}<span>Multiple backends: Send different futures to different backends, based on which parallel backend can provide the requested resources</span>
* {{< bi square >}}<span>Observability: Support for OpenTelemetry (when the **otel** stack has matured)</span>
:::


## Map-Reduce Future API

::: {.list-unstyled}
* {{< bi square-fill >}}<span>Early stopping, e.g. have `future_lapply(X, FUN)` terminate active futures as soon as one of the `FUN(X[[i]])` calls produces an error [~~[future#FR213]~~, ~~[future.apply#FR75]~~]</span>
* {{< bi square-fill >}}<span>Robustness of foreach: Implement bug fixes and outstanding **foreach** feature requests in the **doFuture** adapter until resolved upstream.  For example, add `withDoRNG()` and `%dofuture%` [~~[doFuture#61]~~] (solved in **doFuture** 0.13.0 and **doFuture** 1.0.0)</span>
* {{< bi square-fill >}}<span>Generic support for parallelizing common map-reduce functions, e.g. `y <- lapply(X, slow_fcn) |> futurize()` and `y <- X |> purrr::map(slow_fcn) |> futurize()` (solved in **futurize** 0.1.0)</span>
* {{< bi square-fill >}}<span>Generic support for progress updates for common map-reduce functions, e.g. `y <- lapply(X, slow_fcn) |> progressify()` and `y <- X |> future_map(slow_fcn) |> progressify() |> futurize()` [~~[progressr#FR85]~~, ~~[progressr#113]~~] (solved in **progressify** 0.1.0)</span>
* {{< bi square-half >}}<span>Automatic map-reduce: Support for merging of many futures into chunks of futures to be processed more efficiently with less total overhead. This could make `fs <- lapply(X, function(x) future(FUN(x))` and `vs <- value(fs)` automatically as efficient as `lapply(X, FUN) |> futurize()` removing the need for custom implementations</span>
* {{< bi square >}}<span>Efficient chunking of large objects in map-reduce calls via in-line prologue declaration for code to be processed in current session prior to parallelization, e.g. subsetting of input data before distributing them to parallel workers</span>
* {{< bi x-square >}}<span>~~Develop a **[future.mapreduce]** package to provide an essential, infrastructure Map-Reduce Future API, e.g. load balancing, chunking, parallel random number generation (RNG), and early stopping.  Packages such as **future.apply**, **furrr**, and **doFuture** are currently implementing their own version of these features.  Consolidating these essentials into a shared utility package will guarantee a consistent behavior for all together with faster deployment of bug fixes and improvements [[future.apply#20], [future.apply#FR32], [future.apply#FR44], [future.apply#59], [future.apply#FR60]]~~ (discontinued, superseded by **[futurize]**)</span>
:::


## Parallel backends

Below is a list of future backends that are worked on or in the plans:

::: {.list-unstyled}
* {{< bi square-half >}}<span>**[future.p2p]** - a backend to resolve futures via a peer-to-peer network of trusted collaborators [available as of 2025-08-10]</span>
* {{< bi square-half >}}<span>**future.sandbox** - a backend to resolve untrusted futures via locked-down, sandboxed Linux containers (Docker, Singularity, ...) without access to the hosts file system or network. This can already be partially achieved by low-level container setups via **parallelly**</span>
* {{< bi square >}}<span>**[future.clustermq]** - a backend to resolve futures on a HPC cluster via the **clustermq** package [[future#FR204], [future#FR267]]</span>
* {{< bi square >}}<span>**future.aws.lambda** - a backend to resolve futures via the Amazon AWS Lambda service. A group of several people is actively working on this since the end of 2020 [[future#FR423]]</span>
* {{< bi square >}}<span>**future.aws.batch** - a backend to resolve futures via the Amazon AWS Batch service. We have a working group of several people actively working on this [[future#FR423]]</span>
* {{< bi square >}}<span>**future.google.cloud.functions** - a backend to resolve futures via the Google Cloud Functions service. Work on this will start when a stable **future.aws.lambda** prototype has been established</span>
* {{< bi square >}}<span>**future.sparklyr** - a backend to resolve futures in Spark via the **sparklyr** package [[future#FR286], [sparklyr#FR1935]]</span>
* {{< bi x-square >}}<span>~~**[future.redis]** - a backend to resolve futures via a Redis queue using the **redux** package [[future#FR151]]~~ (discontinued, superseded by **[future.mirai]**)</span>
* {{< bi x-square >}}<span>~~**[future.rrq]** - a backend to resolve futures via a Redis queue using the **rrq** package [[future#FR151]]~~ (discontinued, superseded by **[future.mirai]**)</span>
:::



## Miscellaneous

::: {.list-unstyled}
* {{< bi square-half >}}<span>Internationalization (i18n): Make all error and warning messages translatable via R's `gettext()` framework.  Invite community to provide message translations.</span>
* {{< bi square-half >}}<span>Migrate to permissive license Apache License (>= 2), where possible</span>
:::


[doFuture#61]: https://github.com/HenrikBengtsson/doFuture/issues/61
[doFuture#FR63]: https://github.com/HenrikBengtsson/doFuture/issues/63

[future#59]:  https://github.com/futureverse/future/issues/59
[future#93]:  https://github.com/futureverse/future/issues/93
[future#134]: https://github.com/futureverse/future/issues/134
[future#142]: https://github.com/futureverse/future/issues/142
[future#FR151]: https://github.com/futureverse/future/issues/151
[future#FR181]: https://github.com/futureverse/future/issues/181
[future#FR188]: https://github.com/futureverse/future/issues/188
[future#FR204]: https://github.com/futureverse/future/issues/204
[future#FR267]: https://github.com/futureverse/future/issues/267
[future#205]: https://github.com/futureverse/future/issues/205
[future#FR213]: https://github.com/futureverse/future/issues/213
[future#253]: https://github.com/futureverse/future/issues/253
[future#FR256]: https://github.com/futureverse/future/issues/256
[future#FR286]: https://github.com/futureverse/future/issues/286
[future#FR301]: https://github.com/futureverse/future/issues/301
[future#FR346]: https://github.com/futureverse/future/issues/346
[future#FR361]: https://github.com/futureverse/future/issues/361
[future#FR423]: https://github.com/futureverse/future/issues/423
[future#FR430]: https://github.com/futureverse/future/issues/430
[future#FR437]: https://github.com/futureverse/future/issues/437
[future#FR450]: https://github.com/futureverse/future/issues/450
[future#480]: https://github.com/futureverse/future/issues/480

[future.apply#20]: https://github.com/HenrikBengtsson/future.apply/issues/20
[future.apply#FR32]: https://github.com/HenrikBengtsson/future.apply/issues/32
[future.apply#FR44]: https://github.com/HenrikBengtsson/future.apply/issues/44
[future.apply#59]: https://github.com/HenrikBengtsson/future.apply/issues/59
[future.apply#FR60]: https://github.com/HenrikBengtsson/future.apply/issues/60
[future.apply#FR63]: https://github.com/HenrikBengtsson/future.apply/issues/63
[future.apply#FR75]: https://github.com/HenrikBengtsson/future.apply/issues/75

[future.batchtools#FR23]: https://github.com/HenrikBengtsson/future.batchtools/issues/23
[future.batchtools#FR27]: https://github.com/HenrikBengtsson/future.batchtools/issues/27
[future.batchtools#FR75]: https://github.com/futureverse/future.batchtools/issues/75

[future.callr#FR11]: https://github.com/HenrikBengtsson/future.callr/issues/11

[parallelly#18]: https://github.com/HenrikBengtsson/parallelly/issues/18
[parallelly#32]: https://github.com/HenrikBengtsson/parallelly/issues/32
[parallelly#33]: https://github.com/HenrikBengtsson/parallelly/issues/33

[progressr#FR85]: https://github.com/HenrikBengtsson/progressr/issues/85
[progressr#113]: https://github.com/HenrikBengtsson/progressr/issues/113

[sparklyr#FR1935]: https://github.com/sparklyr/sparklyr/issues/1935


[futurize]: https://futurize.futureverse.org
[future.p2p]: https://future.p2p.futureverse.org
[future.mapreduce]: https://future.mapreduce.futureverse.org
[future.mirai]: https://future.mirai.futureverse.org
[marshal]: https://marshal.futureverse.org
[future.clustermq]: https://github.com/HenrikBengtsson/future.clustermq
[future.rrq]: https://github.com/HenrikBengtsson/future.rrq
[future.redis]: https://github.com/bwlewis/future.redis

[Marshaling and Serialization in R]: https://github.com/RConsortium/marshalling-wg