Whoops, I wrote up a reprex too, not knowing that @Jeff-Thompson12 had already done the same. I'll just share it anyway.

Instead of just toggling one metric with an NA assessment/score to 0, I toggle them all below. As you can see below, the pkg {dplyr} has 7 assessments that are NA given the following pkg_ref call and the score drops from 0.45 to 0.13 when zero'd! I would expect the default score (with NAs) should be 0.13; @emilliman5, do you agree? That is, the existence of an NA shouldn't hurt the riskmetric score by default. It should automatically exclude that metric from the score calculation? That way, the onus would be on the user to tell riskmetric which metrics are important to them by explicitly passing a 1 to the weights vector arg. Is that a adequate summary of our discussion from last Friday?

library(dplyr)
library(riskmetric)
packageVersion("riskmetric")

> [1] ‘0.2.2’

assessed <- "dplyr" %>%
  pkg_ref(source = "pkg_cran_remote", repos = c("https://cran.rstudio.com")) %>%
  as_tibble() %>%
  pkg_assess()

initial_scoring <- assessed %>% pkg_score()
initial_scoring$pkg_score %>% round(2)

> [1] 0.45
  
# riskmetric doesn't appear to be picking up certain metrics
# so we'll set their weights to zero here.
metric_scores <- initial_scoring %>%
  select(-c(package, version, pkg_ref, pkg_score)) %>%
  t

> metric_scores
                           [,1]
bugs_status          0.53333333
covr_coverage                NA
size_codebase                NA
export_help                  NA
r_cmd_check                  NA
dependencies         0.01098694
reverse_dependencies 1.00000000
license                      NA
has_maintainer       1.00000000
remote_checks        0.88461538
exported_namespace           NA
has_website          1.00000000
downloads_1yr        0.99339687
has_news             1.00000000
has_vignettes        1.00000000
has_examples                 NA
has_source_control   1.00000000
has_bug_reports_url  1.00000000
news_current         1.00000000

metric_weights <- ifelse(is.na(metric_scores[,1]), 0, 1)

# re-score the pkg, explicitly setting NAs with a new weight = 0
assessed %>%
  pkg_score(weights = metric_weights) %>%
  pull("pkg_score") %>%
  round(2)

> [1] 0.13

Uh oh!

pkg_score returns different value if metic is NA and weights are set explicitly #293

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

pkg_score returns different value if metic is `NA` and weights are set explicitly #293