-
Notifications
You must be signed in to change notification settings - Fork 12
Expand file tree
/
Copy pathmanuscript.qmd
More file actions
1824 lines (1651 loc) · 83.1 KB
/
Copy pathmanuscript.qmd
File metadata and controls
1824 lines (1651 loc) · 83.1 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
---
title: Orchestrating Microbiome Analysis with Bioconductor
format:
nature-pdf:
# journal.cite-style is included in the tex file but ignored by pandoc if
# cite-method is not `natbib`.
journal:
cite-style: sn-nature
# `citeproc` is the pandoc default. Set `cite-method: natbib` if required
# to use the bst styles from the upstream template.
cite-method: natbib
keep-tex: true
equal-margins: false
# These settings below enable word calculation
filters:
- at: pre-quarto
path: _extensions/andrewheiss/wordcount/citeproc.lua
- at: pre-quarto
path: _extensions/andrewheiss/wordcount/wordcount.lua
affiliations:
- id: 1
name: University of Turku
department: Department of Computing
address: Turku, Finland
- id: 2
name: University of Turku
department: Department of Life Technologies
address: Turku, Finland
- id: 3
name: University of Oxford
department: Department of Biology
address: Oxford, United Kingdom
- id: 4
name: University of Oxford
department: Queen’s College
address: Oxford, United Kingdom
- id: 5
name: University College Cork
department: Department of Anatomy and Neuroscience
address: Cork, Ireland
- id: 6
name: University College Cork
department: APC Microbiome Ireland
address: Cork, Ireland
- id: 7
name: LMU München
department: Department of Statistics
address: Munich, Germany
- id: 8
name: Cornell University
department: Department of Population Health Sciences, Weill Cornell Medicine
address: New York, USA
- id: 9
name: Cornell University
department: Department of Biomedical Informatics & Data Science
address: New York, USA
- id: 10
name: Cornell University
department: Department of Biostatistics
address: New York, USA
- id: 11
name: Yale University
department: Department of Biomedical Informatics & Data Science
address: New Haven, USA
- id: 12
name: Yale University
department: Department of Biostatistics
address: New Haven, USA
- id: 13
name: University of Helsinki
department: Department of Microbiology
address: Helsinki, Finland
- id: 14
name: Helmholtz Munich
address: Munich, Germany
- id: 15
name: Flatiron Institute
address: New York, USA
- id: 16
name: Institute for Molecular Medicine Finland, FIMM-HiLIFE
address: Helsinki, Finland
- id: 17
name: Danone Research and Innovation
address: Utrecht, Netherlands
- id: 18
name: CUNY Graduate School of Public Health and Health Policy
department: Department of Epidemiology and Biostatistics
address: New York, USA
- id: 19
name: No affiliation
- id: 20
name: Amsterdam University Medical Centres
department: Department of Psychiatry
address: Amsterdam, Netherlands
- id: 21
name: Amsterdam University Medical Centres
department: Amsterdam Public Health
address: Amsterdam, Netherlands
- id: 22
name: Cornell University
department: Division of Gastroenterology and Hepatology, Weill Cornell Medicine
address: New York, USA
- id: 23
name: Cornell University
department: Department of Statistics and Data Science
address: New York, USA
author:
- name: Tuomas Borman
orcid: 0000-0002-8563-8884
email: tuomas.v.borman@utu.fi
affiliations:
- ref: 1
attributes:
corresponding: true
- name: Giulio Benedetti
orcid: 0000-0002-8732-7692
affiliations:
- ref: 1
attributes:
equal-contributor: true
- name: Geraldson Muluh
orcid: 0000-0002-9721-5192
affiliations:
- ref: 1
attributes:
equal-contributor: true
- name: Aura Raulo
orcid: 0000-0003-4860-7840
affiliations:
- ref: 3
- ref: 4
- name: Benjamin Valderrama
orcid: 0000-0002-4740-8965
affiliations:
- ref: 5
- ref: 6
- name: Artur Sannikov
orcid: 0000-0001-7765-123X
affiliations:
- ref: 2
- name: Stefanie Peschel
orcid: 0000-0002-7936-7627
affiliations:
- ref: 7
- name: Yihan Liu
orcid: 0009-0008-0462-7501
affiliations:
- ref: 8
- ref: 9
- ref: 10
- ref: 11
- ref: 12
- name: Rasmus Hindström
orcid: 0009-0004-5731-178X
affiliations:
- ref: 1
- name: OMA consortium
affiliations:
- ref: 19
- name: Katariina Pärnänen
orcid: 0000-0003-2220-1738
affiliations:
- ref: 13
- name: Christian L. Müller
orcid: 0000-0002-3821-7083
affiliations:
- ref: 7
- ref: 14
- ref: 15
- name: Aki S. Havulinna
orcid: 0000-0002-4787-8959
affiliations:
- ref: 1
- ref: 16
- name: Sudarshan Shetty
orcid: 0000-0001-7280-9915
affiliations:
- ref: 17
- name: Marcel Ramos
orcid: 0000-0002-3242-0582
affiliations:
- ref: 18
- name: Domenick J. Braccia
orcid: 0000-0002-4606-6254
affiliations:
- ref: 19
- name: Héctor Corrada Bravo
orcid: 0000-0002-1255-4444
affiliations:
- ref: 19
- name: Felix M. Ernst
orcid: 0000-0001-5064-0928
affiliations:
- ref: 19
- name: Levi Waldron
orcid: 0000-0003-2725-0694
affiliations:
- ref: 18
- name: Thomaz F. S. Bastiaanssen
orcid: 0000-0001-6891-734X
affiliations:
- ref: 20
- ref: 21
attributes:
equal-contributor: true
- name: Himel Mallick
orcid: 0000-0003-4956-2429
affiliations:
- ref: 8
- ref: 22
- ref: 23
attributes:
equal-contributor: true
- name: Leo Lahti
orcid: 0000-0001-5537-637X
email: leo.lahti@utu.fi
affiliations:
- ref: 1
attributes:
corresponding: true
abstract: |
The expansion of microbiome research has led to the accumulation of
interlinked datasets encompassing versatile taxonomic and functional
assays. The analysis of increasingly large and heterogeneous multi-modal
microbiome data would benefit from unified approaches supporting the design
of modular data science workflows through interoperable methods. Building upon
the optimized statistical programming framework for multi-assay data
integration recently developed by the Bioconductor project,
we introduce a community-developed open source framework
for microbiome data science. In contrast to the previous alternatives,
the methodology is specifically designed to support joint analysis
of hierarchical, interlinked, and heterogeneous multi-table datasets that
are increasingly common in modern microbiome research. This
framework encompasses open data, methods, tutorials, and an active online
community, thereby supporting standardized and reproducible data
wrangling, joint analysis, and reporting. We have detailed the functionality
and usage in the OMA book
([https://bioconductor.org/books/release/OMA](https://bioconductor.org/books/release/OMA)),
which offers guidance for prospective users and contributors.
keywords: [microbiome, Bioconductor, data science, data integration]
bibliography: bibliography.bib
---
# Introduction {#sec-intro}
Modern data science applications critically rely on open-source methods created
by the research community [@barker2022; @pearson2025]. Open
software ecosystems have become central innovation hubs, driving
technological and cultural transformation [@hocquet2024]. The
Bioconductor project has become a significant distribution
channel for open data science methods in the life sciences.
Its vast developer community maintains over 2,300 quality-controlled R
packages for various fields of life science informatics [@gentleman2004].
Additionally, Bioconductor has evolved into a global community that supports
data science collaboration and training [@drnevich2025], playing an increasingly
important role in advancing computational skills that are essential for modern
microbiology education [@timmis2024].
Microbiome research focuses on how microbial communities vary, function and
interact with their environment and a potential host [@moreno-indias2021]. The
technologies used in this field generate vast amounts of high-dimensional and
hierarchically structured data, commonly in the form of microbial DNA sequences
detected in a set of samples. Following the rapid expansion of microbiome
research and technological advances, metagenomic sequencing is increasingly
complemented by other types of high-throughput measurements. This has
led to the accumulation of multi-modal datasets—collections
that combine different types of molecular data. These interlinked datasets
encompass diverse taxonomic and functional assays, ranging from microbial
abundances and phylogenetic relationships to functional profiles.
Microbiome data scientists
are thus increasingly facing the practical challenges of integrating
heterogeneous data across multiple data modes into reproducible workflows.
The unique statistical properties and complexity of microbiome data
necessitate specialized approaches in their analysis
[@moreno-indias2021; @jeganathan2021; @willis2019], yet the diversity of
proposed solutions and inconsistent outcomes
[@nearing2022; @pelto2025; @gamboa-tuz2025] can make it
difficult to identify the optimal methods and build interoperable workflows
[@wen2023; @shetty2019]. Accordingly, a need for standardization has been
highlighted [@moreno-indias2021].
Open microbiome data science _frameworks_ have emerged to provide sets of
interoperable methodologies in specific computational environments
[@eren2021; @schloss2009; @bolyen2019; @mcmurdie2012], and a broader ecosystem
of digital resources, engaged user communities and collaboration culture that
extend the capacities far beyond the underlying technical framework [@katz2019].
However, frameworks focused solely on microbiome data science fall short on
addressing the need to integrate data across multiple modalities, such as
metagenomics, metabolomics, and host genomics.
The Bioconductor project integrates the methods of microbiome data science
into a wider statistical programming framework, where
interoperable methods and
workflows for single-cell omics [@amezquita2020], spatial transcriptomics [@crowell2025],
metabolomics [@gatto2025], proteomics [@gatto2025; @anwar2025], and other
fields are concurrently developed.
Here, we introduce a stable and optimized data science ecosystem supporting the
integration and analysis of multi-modal datasets in microbiome research.
It provides harmonized data import and representation, and fast,
robust methods for transforming abundance tables into biological insights. These
methods have been extensively tested and distributed through R/Bioconductor
packages
([https://bioconductor.org/packages](https://bioconductor.org/packages)),
supporting the community-driven development of the framework and integration
within the broader Bioconductor ecosystem. To promote
evidence-based best practices, we introduce the online book _Orchestrating
Microbiome Analysis with Bioconductor_ (OMA), which represents a comprehensive
compilation of best practices and current trends in microbiome data science using the
R/Bioconductor ecosystem.
# Data analytical frameworks {#sec-data_frameworks}
Several data analytical frameworks are available for microbiome data science,
each distinguished by its underlying paradigms, user communities, and scope.
For example, tools such as QIIME2 [@bolyen2019], Anvi'o
[@eren2021], and Mothur [@schloss2009] emphasize standardized, modular
approaches tailored to microbiome analysis. These frameworks provide
validated and user-friendly solutions, enabling development of efficient
end-to-end workflows for microbiome research. However, they are typically
designed around specific data types, which limits flexibility
when integrating heterogeneous data modalities within a unified analytical
framework. In contrast, Bioconductor embraces a more exploratory, statistically
oriented paradigm spanning multiple areas of bioinformatics beyond
microbiome research.
This data analytical paradigm of Bioconductor is similar to scikit-bio's
[@aton2025]. While scikit-bio as a Python-based ecosystem benefits from rapidly
evolving machine learning and artificial intelligence tools, Bioconductor’s
strengths lie in its vast statistical ecosystem dedicated to omics research and
strong interoperability with other computational environments.
Another key distinction is community structure: scikit-bio is more
centralized, whereas Bioconductor packages are developed and maintained by an
open community, resulting in a broad toolkit and collaboration enabling easy
reuse of common solutions.
Although this approach can introduce some challenges, such as dependency management,
package compatibility, and differences in package stability, these are actively
addressed through several coordinated mechanisms. First, submitted packages undergo
peer review, ensuring relevance and adherence to standards such as documentation
and working examples. Secondly, Bioconductor follows synchronized, semiannual
release cycle, during which packages are frozen and mutually compatible. Thirdly,
all packages are continuously tested within Bioconductor's automated build system,
which ensures that breaking changes are rapidly detected before release. Each package has an active
maintainer who is notified of failures and resolves them in a timely manner.
Finally, community guidelines and continuous integration practices, together
with shared data structures such as (Tree)SummarizedExperiment, further improve
interoperability between packages.
# Data infrastructure {#sec-infra}
Data scientists routinely deal with interlinked assays, hierarchical data
(*e.g.*, ontologies, trees), relational data (*e.g.*, networks), and supporting
side information. This side information can include
sample attributes such as treatment group, collection site, or time point. The
ability to integrate heterogeneous data elements is essential in biological
research, where the individual components of a system can seldom be studied in
isolation. However, this growing complexity introduces additional overhead, as
researchers must track samples and features across multiple tables and manage an
increasing number of data elements and non-trivial links between them. Moreover,
input data formats vary across different methods, leading to technical
complications and time lost on data wrangling. Bioconductor is addressing these
challenges through standardized data structures that are supported by an
ecosystem of quality-controlled, interoperable methods. This allows the analyst
to invest more time in core analytical tasks.
Bioconductor data structures support statistical analysis of complex
data combinations [@huber2015] and are widely adopted by various
research communities. They have been implemented through specialized
object-oriented classes that integrate multiple data elements into
a single structure known as a *data container*
[@chambers2014]. The SummarizedExperiment (SE) class is
the primary data container underlying the Bioconductor ecosystem
[@huber2015; @morgan2025]. Its position among the top 1\% most-downloaded
packages in Bioconductor reflects its widespread recognition and
increasing adoption among developers (@fig-bioc_packages). SE provides a
standardized solution to store tabular data by linking numeric abundance
matrices with side information on features or rows (*e.g.*, taxonomy),
and samples or columns (*e.g.*, origin, collection time, host health).
The interoperability of SE with other Bioconductor data structures
enables the transfer of methods across different fields of omics research,
thereby promoting the development of modular and scalable data science methods
and ultimately yielding improved overall quality control and user support [@drnevich2025].
In particular, the SE framework has been extended to
similar data integration tasks, notably in single-cell
and spatial omics through the SingleCellExperiment (SCE) [@amezquita2020] and
SpatialExperiment [@crowell2025] classes.
```{r}
#| label: bioc_packages
#| eval: false
#| echo: false
#| message: false
if (!require("BiocManager")) {
install("BiocManager")
library("BiocManager")
}
pkgs <- c("BiocPkgTools", "lubridate", "tidyverse")
temp <- sapply(pkgs, function(pkg) {
if (!require(pkg, character.only = TRUE)) {
install(pkg)
library(pkg, character.only = TRUE)
}
})
# Get packages and info based on their BiocViews
pkgs <- biocPkgList()
# Get how many times these packages are loaded
df <- biocDownloadStats()
# Take only packages that depend on SE
pkgs[["SE"]] <- sapply(pkgs[["Depends"]], function(x) any(grepl("summarizedexperiment", x, ignore.case = TRUE)))
pkgs <- pkgs[pkgs[["SE"]], ]
# Extract the package topic based on biocView
fields <- sapply(pkgs[["biocViews"]], function(x) {
field <- "Other"
if (any(grepl("genomic|genetic|gene|genom|DNA", x, ignore.case = TRUE))) field <- "Genomics"
if (any(grepl("proteomics|protein", x, ignore.case = TRUE))) field <- "Proteomics"
if (any(grepl("metabolomics|metabolome|metabolite|Lipidom|massspectro", x, ignore.case = TRUE))) field <- "Metabolomics"
if (any(grepl("transcripto|RNA-seq|RNA", x, ignore.case = TRUE))) field <- "Transcriptomics"
if (any(grepl("immuno", x, ignore.case = TRUE))) field <- "Immunology"
if (any(grepl("cytom", x, ignore.case = TRUE))) field <- "Cytometrics"
if (any(grepl("microarray|chip", x, ignore.case = TRUE))) field <- "Microarray"
if (any(grepl("single-cell|singlecell", x, ignore.case = TRUE))) field <- "Single-cell"
if (any(grepl("metagenom|microbiome|16S|microbiota|amplicon|shotgun|microb|metatranscript|metametabolo|metaproteo", x, ignore.case = TRUE))) field <- "Microbiome"
return(field)
})
pkgs[["Field"]] <- fields
# Subset downloads table and add field to it
df <- df[df[["Package"]] %in% pkgs[["Package"]], ]
df[["Field"]] <- pkgs[match(df[["Package"]], pkgs[["Package"]]), "Field"][[1]]
# Get information on when the package became available in Bioconductor
dates <- df[["Date"]] |> unique()
dates <- dates[dates < floor_date(Sys.time(), "month")]
available <- lapply(dates, function(date) {
temp <- df[df[["Date"]] == date, ]
temp <- temp[match(unique(df[["Package"]]), temp[["Package"]]), ]
temp <- temp[["Nb_of_distinct_IPs"]] > 0 | temp[["Nb_of_downloads"]] > 0
temp[is.na(temp)] <- FALSE
as.numeric(temp)
})
available <- do.call(cbind, available) |> as.data.frame()
colnames(available) <- dates
rownames(available) <- unique(df[["Package"]])
# Create a table showing the number of packages by field through time
ind <- pkgs[match(unique(df[["Package"]]), pkgs[["Package"]]), "Field"][[1]]
pkgs_date <- rowsum(available, group = ind)
pkgs_date <- pkgs_date |>
rownames_to_column("Field") |>
pivot_longer(
cols = -Field,
names_to = "Date",
values_to = "N"
)
pkgs_date[["Date"]] <- as.Date(pkgs_date[["Date"]])
# Save results
pkgs_date |>
write.csv(file = file.path("data", "bioc_packages_stats.csv"), row.names = FALSE)
```
```{r}
#| label: fig-bioc_packages
#| eval: true
#| echo: false
#| message: false
#| warning: false
#| fig-height: 3
#| fig-width: 6
#| fig-cap: "**Adoption of the SummarizedExperiment (SE) data container among
#| Bioconductor developers.** The growth in the number of Bioconductor
#| packages supporting SE over time is shown, categorized by their respective
#| fields. Packages are counted by the date they were added to Bioconductor.
#| Some packages added the SE support after their original release."
if (!require("BiocManager")) {
install("BiocManager")
library("BiocManager")
}
pkgs <- c("tidyverse", "see")
temp <- sapply(pkgs, function(pkg) {
if (!require(pkg, character.only = TRUE)) {
install(pkg)
library(pkg, character.only = TRUE)
}
})
# Import data
pkgs_date <- read.csv(file.path("data", "bioc_packages_stats.csv"))
pkgs_date[["Date"]] <- pkgs_date[["Date"]] |> as.Date()
# Create the plot
p <- ggplot(pkgs_date, aes(x = Date, y = N, fill = Field)) +
geom_area() +
theme_classic() +
scale_fill_okabeito() +
labs(x = "Year", y = "Number of packages") +
scale_y_continuous(expand = c(0, 0), limits = c(0, NA)) +
scale_x_date(expand = c(0, 0), limits = c(as.Date("2015-01-01"), max(pkgs_date[["Date"]])))
p
```
**Hierarchical data structures.** Microbiome data scientists frequently need to
integrate information on feature phylogenies and sample hierarchies
to accurately reflect fine-scale variability in microbiome data.
While maintaining interoperability with
the broader Bioconductor ecosystem, the TreeSummarizedExperiment
(TreeSE) class [@huang2021] extends the SE and SCE data structures to
hierarchical datasets by incorporating both feature and sample organizations
as tree structures (@fig-datacontainers a). Moreover, TreeSE supports the
storage of other common elements of microbiome data, for example, reference
sequences, and incorporates results from common analytical procedures such as
dimensionality reduction, statistical transformations, and agglomeration by
taxonomic, functional or other information. The interoperability with other
Bioconductor classes facilitates scalable analysis and the design of modular
yet flexible workflows, as TreeSE inherits full compatibility with methods
designed for SE and SCE. In particular, the direct interoperability with the
widely adopted SE data science ecosystem distinguishes our TreeSE
framework from phyloseq, which is another popular Bioconductor data container
developed initially with a focus on 16S amplicon data analysis [@mcmurdie2012].
TreeSE provides a more general framework than phyloseq, as the former not only
accommodates the typical components of a phyloseq object
(taxa abundances, sample metadata, phylogenetic tree, and reference sequences),
but also extends to multi-modal datasets encompassing
parallel taxonomic levels, functional predictions, and resistome, metabolome or
other profiles and alternative feature representations. It also offers
substantial gains in computing speed and memory efficiency over phyloseq
(@fig-benchmarking). While speedyseq is faster in certain operations, TreeSE
tends to scale more efficiently in terms of execution time (t) and allocated
memory (m) as sample size (n) increases. This benefits from the built-in support
for sparse and delayed matrices in SE-based data containers, which can improve
performance [@huber2015]. Despite these advantages, further optimizing time and
memory consumption represents a key area for further development in the (Tree)SE
framework. To facilitate migration from phyloseq, the OMA book provides
conversion functions between the two data container formats together with a
mapping table of commonly used functions. These resources enable users to
incrementally migrate existing phyloseq workflows to the (Tree)SE framework while
preserving familiar coding patterns and minimizing changes to established
analysis pipelines. Despite the advantages of the (Tree)SE framework,
further optimization of time and memory consumption remains an area for future
development.
```{r}
#| label: benchmark_analysis
#| eval: false
#| echo: false
#| message: false
# Adapted from https://github.com/microbiome/benchmarking/tree/main/article
# Import libraries
if (!require("BiocManager")) {
install("BiocManager")
library("BiocManager")
}
pkgs <- c(
"bench", "DelayedArray", "mia", "microbiome", "microbiomeDataSets", "philr",
"phyloseq", "picante", "tidyverse", "speedyseq"
)
temp <- sapply(pkgs, function(pkg) {
if (!require(pkg, character.only = TRUE)) {
install(pkg)
library(pkg, character.only = TRUE)
}
})
# Set seed for reproducibility
set.seed(123)
# Set benchmarking hyperparameters
n_iter <- 10
memory_threshold <- 10000
beta_threshold <- c(1000, 10000)
N <- c(10, 100, 1000, 10000)
# Define class names
classes <- c(tse = "TreeSE", pseq = "phyloseq", spseq = "speedyseq")
# Define method names
methods <- c(
alpha = "Faith diversity",
beta = "UniFrac dissimilarity",
melt = "Melting",
trans = "PhILR transformation",
agg = "Family agglomeration"
)
# Import dataset
GTSD <- GrieneisenTSData()
# Agglomerate by Genus to reduce size
GTSD <- mia::agglomerateByRank(GTSD, rank = "Genus")
# Run benchmark for each sample size
benchmark_out <- bench::press(
N = N,
{
# Select a random subset of samples
tse <- GTSD[, sample(ncol(GTSD), N)]
# Convert TreeSE to phyloseq
pseq <- mia::convertToPhyloseq(tse)
# List expressions to benchmark
expressions <- list(
# Estimate faith from phyloseq
alpha_pseq = quote(picante::pd(samp = t(otu_table(pseq)), tree = phy_tree(pseq))[, 1]),
# Estimate faith from TreeSE
alpha_tse = quote(mia::getAlpha(tse, index = "faith_diversity")),
# philr transform phyloseq
trans_pseq = quote(philr::philr(t(otu_table(pseq)), tree = phy_tree(pseq), pseudocount = 1)),
# philr transform TreeSE
trans_tse = quote(mia::transformAssay(tse, method = "philr", MARGIN = 1L, pseudocount = 1)),
# Melt phyloseq
melt_pseq = quote(phyloseq::psmelt(pseq)),
# Melt speedyseq
melt_spseq = quote(speedyseq::psmelt(pseq)),
# Melt TreeSE
melt_tse = quote(mia::meltSE(tse, add.row = TRUE, add.col = TRUE)),
# Agglomerate phyloseq
agg_pseq = quote(phyloseq::tax_glom(pseq, taxrank = "Family")),
# Agglomerate speedyseq
agg_spseq = quote(speedyseq::tax_glom(pseq, taxrank = "Family")),
# Agglomerate TreeSE
agg_tse = quote(mia::agglomerateByRank(tse, rank = "Family"))
)
if (N <= beta_threshold[[1]]) {
# Estimate unifrac from phyloseq
expressions[["beta_pseq"]] <- quote(
phyloseq::UniFrac(pseq)
)
} else {
# Use dummy expression because all results must have the same length
expressions[["beta_pseq"]] <- quote(1 + 1)
}
if (N <= beta_threshold[[2]]) {
# Estimate unifrac from TreeSE
expressions[["beta_tse"]] <- quote(
mia::getDissimilarity(tse, method = "unifrac")
)
} else {
# Use dummy expression because all results must have the same length
expressions[["beta_tse"]] <- quote(1 + 1)
}
# Run benchmark
bench::mark(
iterations = n_iter,
memory = if (N <= memory_threshold) TRUE else FALSE,
check = FALSE,
exprs = expressions
)
}
)
# Retrieve benchmarking results for each experiment and iteration
benchmark_df <- benchmark_out |>
unnest(c(time, gc)) |>
dplyr::select(expression, N, time, gc, mem_alloc) |>
separate_wider_delim(
cols = expression, delim = "_",
names = c("method", "object")
) |>
filter(method != "beta" |
(object == "pseq" & N <= beta_threshold[[1]]) |
(object == "tse" & N <= beta_threshold[[2]]))
# Summarise benchmarking results with mean time and standard deviation
benchmark_df <- benchmark_df |>
group_by(method, object, N) |>
summarise(
Time = as.numeric(mean(time)), Memory = as.numeric(mean(mem_alloc)),
TimeSD = sd(time), TimeSE = TimeSD / sqrt(n_iter),
NoGC = sum(gc == "none"), .groups = "drop"
) |>
mutate(
method = factor(method, levels = names(methods)),
object = factor(object, levels = names(classes))
)
# Write to file
benchmark_df |>
mutate(Time = as.numeric(Time), Memory = as.numeric(Memory)) |>
write.csv(file = file.path("data", "benchmark_results.csv"), row.names = FALSE)
```
```{r}
#| label: fig-benchmarking
#| echo: false
#| message: false
#| warning: false
#| fig-cap: "**Execution time and memory consumption for microbiome data
#| containers.** The Bioconductor's TreeSE, phyloseq and speedyseq are
#| commonly used data containers in microbiome analysis. We benchmarked them
#| for five common operations applied to random subsets of samples from a
#| large study on wild baboons [@grieneisen2021] (accessible through the
#| microbiomeDataSets package in Bioconductor). Execution time (s) was
#| measured as the mean value over 10 iterations, whereas allocated memory
#| (MB) was estimated based on a single iteration. The benchmarking was
#| conducted in R using the bench library with 8 CPU cores and 32 GB RAM.
#| All operations, except for UniFrac dissimilarity with phyloseq, could
#| also be performed on a regular laptop (*e.g.*, 4 CPU cores and 16 GB RAM).
#| The source code for the benchmarking experiments is available through the
#| OMA book."
#| fig-width: 12
#| fig-height: 6
# Adapted from https://github.com/microbiome/benchmarking/tree/main/article
pkgs <- c("patchwork", "tidyverse")
temp <- sapply(pkgs, function(pkg) {
if (!require(pkg, character.only = TRUE)) {
install(pkg)
library(pkg, character.only = TRUE)
}
})
classes <- c(tse = "TreeSE", pseq = "phyloseq", spseq = "speedyseq")
methods <- c(
alpha = "Faith diversity",
beta = "UniFrac dissimilarity",
melt = "Melting",
trans = "PhILR transformation",
agg = "Family agglomeration"
)
# Import results
benchmark_df <- read.csv(file.path("data", "benchmark_results.csv")) %>%
mutate(method = factor(method, levels = names(methods)),
object = factor(object, levels = names(classes)))
# Select only points from 10, 100, 1000, 10000
benchmark_df <- benchmark_df[benchmark_df[["N"]] %in% c(10, 100, 1000, 10000), ]
# Specify plot layouts
scientific_10 <- function(y) {
sapply(y, function(z) {
if (is.na(z)) {
return(NA)
} else if (z == 1) {
return("1")
} else if (z == 10) {
return("10")
} else {
return(parse(text = paste0("10^", log10(z))))
}
})
}
breaks <- benchmark_df[["N"]][log10(benchmark_df[["N"]]) %% 1 == 0] |> unique()
# Visualise benchmarking results: time
p1 <- ggplot(benchmark_df, aes(x = N, y = Time, colour = object)) +
geom_errorbar(aes(ymin = Time - TimeSE, ymax = Time + TimeSE), width = 0) +
geom_line() +
geom_point() +
scale_x_log10(breaks = breaks, limits = benchmark_df[["N"]] |> range(), labels = scientific_10) +
scale_y_log10(labels = scientific_10) +
scale_colour_manual(
labels = classes,
values = c("black", "darkgrey", "lightgrey")
) +
facet_grid(. ~ method, labeller = labeller(method = methods)) +
labs(x = "# Samples", y = "Execution time (s)", colour = "Object") +
theme_bw() +
theme(
axis.title.x = element_blank(),
axis.title.y = element_text(size = 15),
axis.text.x = element_blank(),
axis.text.y = element_text(size = 12),
axis.ticks.x = element_blank(),
strip.text = element_text(size = 12),
strip.background = element_blank()
)
# Visualise benchmarking results: memory
p2 <- ggplot(benchmark_df, aes(x = N, y = Memory / 1e6, colour = object)) +
geom_line() +
geom_point() +
scale_x_log10(breaks = breaks, limits = benchmark_df[["N"]] |> range(), labels = scientific_10) +
scale_y_log10(labels = scientific_10) +
scale_colour_manual(
labels = classes,
values = c("black", "darkgrey", "lightgrey")
) +
facet_grid(. ~ method,
labeller = labeller(method = methods)
) +
labs(x = "Sample size (n)", y = "Allocated memory (MB)", colour = "Object") +
theme_bw() +
theme(
axis.title = element_text(size = 15),
axis.text = element_text(size = 12),
strip.text = element_blank()
) +
guides(colour = "none")
# Combine results
p <- (p1 / p2) +
plot_layout(guides = "collect") &
theme(
legend.position = "bottom",
legend.text = element_text(size = 12),
legend.title = element_text(size = 15),
legend.key.size = unit(1.2, "cm")
)
p
```
**Multi-assay data structures.** Contemporary microbiome research frequently
involves data across multiple measurement modalities for enhanced functional
and mechanistic insights. This brings the need to find an appropriate
representation for each data type and map the links between them. Whereas the
TreeSE data container is broadly applicable to different modalities such as
taxonomic, resistome, transcriptome, proteome, and metabolome profiling data,
dedicated data containers have been developed for certain modalities by the
Bioconductor community to address their specific characteristics; examples
include single cell sequencing [@amezquita2020] and host genomics
[@lawrence2013]. Linking data across modalities presents an additional technical
challenge, as multi-omics datasets are often sparse and only a subset of the
samples may be shared across multiple modalities [@sherwani2025]. In other
cases, a single sample in one modality may link to several samples in
another modality [@husso2023] (*e.g.*, due to technical or spatial replication).
These scenarios require the ability to integrate multiple
data modalities into a single object and create flexible sample mappings across them.
The MultiAssayExperiment (MAE) data container [@ramos2017] addresses these
challenges by introducing a formal mapping layer (sampleMap) that links
biological samples to their corresponding measurements in each modality
(@fig-datacontainers b). This design supports one-to-one, one-to-many, and
partially overlapping sample relationships across experiments. Importantly, this
mapping is also accessible for downstream operations. For instance when
subsetting a MAE object
(*e.g.*, selecting samples with both microbiome and metabolome data), the
framework automatically resolves the intersection of available samples and
ensures consistent alignment across all modalities. Such structured handling of
sample relationships is particularly important for integrative methods, as it
reduces the risk of sample mismatches, a common source of irreproducibility in
ad hoc multi-table analyses. For instance, diagonal integration methods
[@xu2022; @argelaguet2021computational]—where shared features across
data types may not be explicitly defined—can benefit from
such a systematic framework that enforces anchor alignment. This reflects lessons
learned from single-cell data integration, where shared anchors and hierarchical
mapping have proven essential for interpretability. To summarize, flexible mapping of samples
across several data objects is key for efficient data integration, method sharing,
and overall interoperability.
@tbl-containers summarizes the integrative data containers for microbiome
analysis, whereas @tbl-elements introduces the available data elements in each
container.
| Data container | Target use |
|-----------------------------------------------|-------------------------------------------------------------|
| SummarizedExperiment (SE)[@huber2015] | generic container linking multiple data tables (assays) |
| TreeSummarizedExperiment (TreeSE)[@huang2021] | extension incorporating hierarchical data structures |
| MultiAssayExperiment (MAE)[@ramos2017] | binding omic-specific containers in multi-omics experiments |
: Integrative data containers for microbiome analysis.
{#tbl-containers tbl-colwidths="[50,50]"}
| Slot | Content | SE | TreeSE | MAE$^a$ |
|---------------|----------------------------------------------|:--:|:------:|:-------:|
| **Tables** | | | | |
| assays | tables of features (rows) and samples (cols) | X | X | X |
| altExps$^b$ | experiments with variable number of features | | X | X |
| reducedDims | results of dimensionality reduction | | X | X |
| **Rows** | | | | |
| rowData | side information on features | X | X | X |
| rowPairs | linkage between pairs of associated features | | X | X |
| rowTrees | hierarchical organization of features | | X | X |
| referenceSeq | reference sequences of features | | X | X |
| **Columns** | | | | |
| colData | side information on samples | X | X | X |
| colTrees | hierarchical organization of samples | | X | X |
| sampleMap$^c$ | flexible sample mapping across experiments | | | X |
| **Other** | | | | |
| metadata | additional information on the experiment | X | X | X |
: Key data elements in the integrative data containers.
{#tbl-elements tbl-colwidths="[17,63,3,9,8]"}
$^a$ While MAE does not contain conventional slots (with the exception
of colData), its experiments can include any number of data containers that
provide those slots. $^b$ altExp samples must match assay samples in a 1-to-1
mapping. $^c$ The sampleMap supports diverse mappings between samples from
different experiments (1-to-1, 1-to-many, many-to-1 and many-to-many).
![**Bioconductor data containers for microbiome data.** **(a)
TreeSummarizedExperiment (TreeSE)** supports hierarchical, multi-table
microbiome analysis. A data object can accommodate: i) “assays” that describe
the abundances of features in each sample (*e.g.* taxonomic units, resistance
genes, or predicted functions), ii) “colData” with tabular metadata on each
sample, iii) “rowData” with tabular metadata on each feature (*e.g.* taxonomic
mappings), and iv) tree information describing feature hierarchies, such as
phylogenetic relations between the taxonomic units (rowTree) or host species
(colTree). TreeSE can thus link multiple numeric abundance tables and their
derived versions (*e.g.* statistical transformations, different taxonomic
levels, dimensionality reduction) with tabular and hierarchical side
information on the features (rows) and samples (columns). Additional slots for
reference sequences and experiment metadata are available. (Figure adapted from
[@huang2021]) **(b) MultiAssayExperiment (MAE)** facilitates the integration of
interlinked datasets that may represent different omics modalities
linked by potentially non-trivial sample mappings. (Figure adapted from
[@ramos2017])](figures/data_containers.png){#fig-datacontainers}
**Data resources.** Several open microbiome data resources are supported,
enabling direct data import into the aforementioned data containers for further
downstream analysis within the Bioconductor methods ecosystem
(@tbl-dataresources). Examples of the available data resources include
curatedMetagenomicData [@pasolli2017], HoloFood [@rogers2024],
microbiomeDataSets [@microbiomedatasets], and the EBI/MGnify database
[@gurbich2023]. In addition, Bioconductor packages often
include built-in demonstration datasets. Collectively, these resources
encompass taxonomic and functional profiles from thousands of
published microbiome studies and tens of thousands samples that can
be accessed with the available tools for research and educational purposes.
| Data resource | Package | # studies or datasets | # samples |
|---------------------------------------------|--------------------------------------------------------------------------|-----------------------|-----------|
| Curated metagenomic data [@pasolli2017] | curatedMetagenomicData [@pasolli2017] | 93 | 22,588 |
| HoloFood [@rogers2024] | HoloFoodR [@holofoodr] | - | 9,990 |
| MGnify [@gurbich2023] | MGnifyR [@mgnifyr] | 5,129 | 629,821 |
| microbiomeDataSets [@microbiomedatasets] | microbiomeDataSets [@microbiomedatasets] | 6 | 19,100 |
| Microbiome Benchmark Data [@gamboa-tuz2025] | MicrobiomeBenchmarkData [@gamboa-tuz2025] | 6 | 1,125 |
| Human Microbiome Project 2 [@hmp2data] | HMP2data [@hmp2data] | 3 | 11,493 |
| A compilation of open microbiome datasets | [https://github.com/microbiome/data](https://github.com/microbiome/data) | 8 | 18,777 |
: Supporting open microbiome data resources (accessed 29/09/2025).
{#tbl-dataresources tbl-colwidths="[30,40,15,15]"}
# Data preparation {#sec-process}
While several tools for microbiome data analysis are available [@jeganathan2021; @willis2019;
@marcoszambrano2023], their implementations typically differ in terms of
expected input and output formats. Common data standards promote
modular data science workflows, speeding up methods development,
benchmarking, and replication. Building upon these principles, we describe a set
of interoperable software libraries that support microbiome data integration and
analysis based on the two integrative data containers, TreeSE and MAE
(Figure \ref{fig:workflow}).
This framework specifically supports the downstream analysis of
high-throughput sequencing data from microbiome experiments after summarizing
the original measurements into abundance
tables (*e.g.*, taxonomic or functional profiles). Taxonomic
abundance tables are one of the most commonly encountered data types.
Common measurement platforms include amplicon-based marker gene
studies (*e.g.*, 16S rRNA) and metagenomic sequencing, but alternative profiling
techniques are available including phylogenetic microarrays
[@rajilicstojanovic2009] and high-density optical mapping
[@abakumov2024]. The interoperability of the data containers
support modular workflow design for integrating different
omics data types into the broader Bioconductor ecosystem. This