Version map cannot be validated - not mapping to new bioconductor website

[aehrenberg]

Submitting the basic installation command for Bioconductor:
if (!require("BiocManager", quietly = TRUE)) install.packages("BiocManager") BiocManager::install(version = "3.18")

in R (version 4.3.0) returns the following error:
Error: Bioconductor version cannot be validated; no internet connection? See #troubleshooting section in vignette

None of the suggested solutions I've found online (adjusting the proxy/firewall settings, ensuring internet connection, etc) have helped.

The documentation suggests that the version map can be found at https://bioconductor.org/config.yaml which, as of today (11/13/2023), cannot be reached. There is a version map I can reach through the new bioconductor website at this link, though, https://new.bioconductor.org/config.yaml.

It seems like the version map cannot be validated because the bioconductor.org/config.yaml is a broken link, whereas the new.bioconductor.org/config.yaml would work. Can this be fixed?

[vjcitn]

I am not sure what to say about your internet issue. I find the following succeeds, while browsing to the config.yaml does not.

z2 = httr::GET("https://bioconductor.org/config.yaml")
cat(rawToChar(content(z2)))

[lshep]

@vjcitn If you browse in a web broswer it likely downloaded the config to your computer if you did not get an error. I also cannot produce this. If you run the code above that vince ran what do you get @aehrenberg ?

[aehrenberg]

When I ran the code suggested by @vjcitn :

z2 = httr::GET("https://bioconductor.org/config.yaml")
cat(rawToChar(content(z2)))

I got the following:

Error in curl::curl_fetch_memory(url, handle = handle) :  schannel: failed to receive handshake, SSL/TLS connection failed

I thought to try the url to the new bioconductor website as follows

z1 = httr::GET("https://new.bioconductor.org/config.yaml")  
cat(rawToChar(content(z1)))

and got the following:

---
## Lines that must be updated when we release a new software version are 
## indicated in comments below

output_dir: output
staging_deploy_root: biocadmin@staging.bioconductor.org:/loc/www/bioconductor-test.fhcrc.org
production_deploy_root: webadmin@master.bioconductor.org:/extra/www/bioc

## CHANGE THIS WHEN WE RELEASE A VERSION:
release_version: "3.18"
r_version_associated_with_release: "4.3.0"
r_version_associated_with_devel: "4.3.0"

## CHANGE THIS WHEN WE RELEASE A VERSION:
devel_version: "3.19"

## CHANGE THIS WHEN WE ADD A VERSION:
## This is the list of all versions for which we want to generate "new" 
## bioc-views, package index, and package detail pages.
## if old versions (no older than 2.5) need to be rebuilt for any reason,
## add them here, but in most cases this should just contain the
## release and devel versions (though after a release you should
## let the no-longer-release version build one last time so package
## landing pages won't say "release version"):
versions:
- "3.18"
- "3.19"

## CHANGE THIS (i.e., uncomment) as various parts of the new devel version 
## become available. set to "[]" if none are available.
devel_repos:
- "bioc"
- "data/experiment"
- "workflows"
#- "data/annotation"

## CHANGE this when the build machines change:
## also, don't include machines that are not building yet (comment them out)
active_release_builders:
  linux: "nebbiolo2"
  windows: "palomino4"
  mac_monterey: "lconway"
active_devel_builders:
  linux: "nebbiolo1"
  windows: "palomino3"
  mac_monterey: "merida1"

## CHANGE this when the single package builder builds
## on a newer version of bioc
single_package_builder:
  bioc_version: "3.18"
  r_version: "4.3"

rss_hub_url: "https://pubsubhubbub.appspot.com/"
## CHANGE these as AMI IDS change
ami_ids:
 bioc2_8: ami-3a2ef452
 bioc2_9: ami-2623f94e
 bioc2_10: ami-5621fb3e
 bioc2_11: ami-f827fd90
 bioc2_12: ami-7224fe1a
 bioc2_13: ami-4a25ff22
 bioc2_14: ami-9c25fff4
 bioc3_0: ami-be7917d6
 bioc3_1: ami-a3d126c8
 bioc3_2: ami-85d88de0
 bioc3_3: ami-abd0b3bc
 bioc3_4: ami-8946709f
 bioc3_5: ami-279a315d
 bioc3_6: ami-ac5df1d3
 bioc3_7: ami-01bcd08e357360496
 bioc3_8: ami-0565362d8bfb9cbed
 bioc3_9: ami-0f5d1990d8c571cdf
 bioc3_10: ami-0c5ab50ca03a54468
 bioc3_11: ami-071b80cf0d8ca085c
 bioc3_12: ami-04c69d122c1cf7e81
 bioc3_13: ami-0e7efd11a6eab85a6
string_pattern_type: legacy
data_sources:
- items_root: /
  layouts_root: /
  type: filesystem_unified
  identifier_type: legacy
  encoding: utf-8
  config:
    encoding: utf-8
- items_root: /
  type: biostar_list
  identifier_type: legacy
  encoding: utf-8
  config:
    gmane_rss_url: 'http://rss.gmane.org/messages/excerpts/gmane.science.biology.informatics.conductor'
    cache_file: tmp/gmane_list_cache_file.yaml
    encoding: utf-8
    ttl: 10
- items_root: /help/bioc-views/package-pages
  layouts_root: /
  type: bioc_views
  identifier_type: legacy
  encoding: utf-8
  config:
    json_dir: "assets/packages/json"
    encoding: utf-8
- items_root: /help/publications/papers/pubmed
  type: pubmed_papers
  identifier_type: legacy
  encoding: utf-8
  config:
    cache_file: tmp/pubmed_cache_file.yaml
    retmax: 20
    ttl: 24
    baseurl: http://eutils.ncbi.nlm.nih.gov/entrez/eutils/
#   queried database, e.g. "pubmed" or "pmc"; can be a vector
    db: [pubmed, pmc]
    term: bioconductor
#   Sorting of query results. This one is database-specific; possible values include
#   pubmed: pub+date, first+author, last+author, journal, title, relevance
#   pmc: print+pub+date, electronic+pub+date, journal, pmc+live+date
#   For default sorting leave this field out
    sort: [ "", electronic+pub+date]
# CHANGE THIS when it becomes apparent which future versions of BioC will
# work with which versions of R:
r_ver_for_bioc_ver:
  "1.6": "2.1"
  "1.7": "2.2"
  "1.8": "2.3"
  "1.9": "2.4"
  "2.0": "2.5"
  "2.1": "2.6"
  "2.2": "2.7"
  "2.3": "2.8"
  "2.4": "2.9"
  "2.5": "2.10"
  "2.6": "2.11"
  "2.7": "2.12"
  "2.8": "2.13"
  "2.9": "2.14"
  "2.10": "2.15"
  "2.11": "2.15" # R switching to yearly releases, BioC remaining the same
  "2.12": "3.0"
  "2.13": "3.0"
  "2.14": "3.1"
  "3.0": "3.1"
  "3.1": "3.2"
  "3.2": "3.2"
  "3.3": "3.3"
  "3.4": "3.3"
  "3.5": "3.4"
  "3.6": "3.4"
  "3.7": "3.5"
  "3.8": "3.5"
  "3.9": "3.6"
  "3.10": "3.6"
  "3.11": "4.0"
  "3.12": "4.0"
  "3.13": "4.1"
  "3.14": "4.1"
  "3.15": "4.2"
  "3.16": "4.2"
  "3.17": "4.3"
  "3.18": "4.3"
  "3.19": "4.4"
# UPDATE THIS when we release a version
release_dates: # old info from http://en.wikipedia.org/wiki/Bioconductor#Milestones
  "1.0": "1/5/2001"
  "1.1": "11/19/2002"
  "1.2": "5/29/2003"
  "1.3": "10/30/2003"
  "1.4": "5/17/2004"
  "1.5": "10/25/2004"
  "1.6": "5/18/2005"
  "1.7": "10/14/2005"
  "1.8": "4/27/2006"
  "1.9": "10/4/2006"
  "2.0": "4/26/2007"
  "2.1": "10/8/2007"
  "2.2": "5/1/2008"
  "2.3": "10/22/2008"
  "2.4": "4/21/2009"
  "2.5": "10/28/2009"
  "2.6": "4/23/2010"
  "2.7": "10/18/2010"
  "2.8": "4/14/2011"
  "2.9": "11/1/2011"
  "2.10": "4/2/2012"
  "2.11": "10/3/2012"
  "2.12": "4/4/2013"
  "2.13": "10/15/2013"
  "2.14": "4/14/2014"
  "3.0": "10/14/2014"
  "3.1": "4/17/2015"
  "3.2": "10/14/2015"
  "3.3": "5/4/2016"
  "3.4": "10/18/2016"
  "3.5": "4/25/2017"
  "3.6": "10/31/2017"
  "3.7": "05/01/2018"
  "3.8": "10/31/2018"
  "3.9": "05/03/2019"
  "3.10": "10/30/2019"
  "3.11": "04/28/2020"
  "3.12": "10/28/2020"
  "3.13": "05/20/2021"
  "3.14": "10/27/2021"
  "3.15": "04/27/2022"
  "3.16": "11/02/2022"
  "3.17": "04/26/2023"
  "3.18": "10/25/2023"
mirrors:
 - 0-Bioconductor:
   - institution: Bioconductor, automatic redirection to servers worldwide
     institution_url: https://bioconductor.org
     contact: Bioconductor Maintainer
     contact_email: maintainer@bioconductor.org
     city: World-wide
     mirror_url: http://bioconductor.org/
     https_mirror_url: https://bioconductor.org/
     rsync: secure_mirror_from_master
     country_code: us
 - Germany:
   - institution: Department of Statistics, TU Dortmund
     institution_url: https://www.statistik.tu-dortmund.de/
     contact: Uwe Ligges
     contact_email: ligges@statistik.tu-dortmund.de
     mirror_url: http://bioconductor.statistik.tu-dortmund.de/
     https_mirror_url: https://bioconductor.statistik.tu-dortmund.de/
     rsync: 
     city: Dortmund
     country_code: de
   - institution: GWDG
     institution_url: https://ftp.gwdg.de/
     contact: Tim Ehlers and Steffen Klemer
     contact_email: ftpmaster@gwdg.de
     mirror_url: http://ftp.gwdg.de/pub/misc/bioconductor/
     https_mirror_url: https://ftp.gwdg.de/pub/misc/bioconductor/
     rsync: secure_mirror_from_master
     city: Gottingen
     country_code: de
 - Japan:
   - institution: The Institute of Statistical Mathematics
     institution_url: https://www.ism.ac.jp/
     contact: Keisuke Honda
     contact_email: khonda@ism.ac.jp
     mirror_url: http://bioc.ism.ac.jp/
     https_mirror_url: https://bioc.ism.ac.jp/
     rsync: secure_mirror_from_master
     country_code: jp
     city: Tachikawa
   - institution: RIKEN Advanced Center for Computing and Communication
     institution_url: https://accc.riken.jp/en/
     contact: Itoshi NIKAIDO, Ph.D.
     contact_email: dritoshi@gmail.com
     mirror_url: http://bioconductor.riken.jp/
     https_mirror_url: https://bioconductor.riken.jp/
     rsync: secure_mirror_from_master
     country_code: jp
     city: Wako
 - China:
   - institution: TUNA Mirror Site, Tsinghua University 
     institution_url: mirrors.tuna.tsinghua.edu.cn
     contact: Miao Wang 
     contact_email: mirroradmin@tuna.tsinghua.edu.cn 
     mirror_url: http://mirrors.tuna.tsinghua.edu.cn/bioconductor/
     https_mirror_url: https://mirrors.tuna.tsinghua.edu.cn/bioconductor/ 
     rsync: secure_mirror_from_master
     country_code: cn 
     city: Peking 
   - institution: eScience Center, Nanjing University
     institution_url: https://sci.nju.edu.cn/
     contact: Ge Yao
     contact_email: yaoge@nju.edu.cn
     mirror_url: http://mirrors.nju.edu.cn/bioconductor/
     https_mirror_url: https://mirrors.nju.edu.cn/bioconductor/
     rsync: secure_mirror_from_master
     city: Nanjing
     country_code: cn
   - institution: University of Science and Technology of China (USTC)
     institution_url: https://ustc.edu.cn
     contact: Linux User Group (LUG) @ USTC
     contact_email: lug@ustc.edu.cn
     mirror_url: http://mirrors.ustc.edu.cn/
     https_mirror_url: https://mirrors.ustc.edu.cn/
     rsync: secure_mirror_from_master
     city: Hefei Anhui
     country_code: cn
 - Norway:
   - institution: University of Bergen
     institution_url: https://www.uib.no/
     contact: Trond Davidsen
     contact_email: Trond.Davidsen@it.uib.no
     mirror_url: http://bioconductor.uib.no/
     https_mirror_url: https://bioconductor.uib.no/
     rsync: secure_mirror_from_master
     city: Bergen
     country_code: no
 - Italy:
   - institution: University of Pisa
     institution_url: https://www.unipi.it/
     contact: Fabio Pratelli
     contact_email: fabio.pratelli@unipi.it 
     mirror_url: http://bioconductor.unipi.it
     https_mirror_url: https://bioconductor.unipi.it
     rsync: secure_mirror_from_master
     city: Pisa
     country_code: it
 - Korea:
   - institution: CRAN Asia
     institution_url: https://cran.asia
     contact: Seongho Bae
     contact_email: admin@cran.asia
     mirror_url: http://cran.asia
     https_mirror_url: https://cran.asia
     rsync: secure_mirror_from_master
     city: Seoul
     country_code: kr
 - Australia:
   - institution: AARNet Pty Ltd
     institution_url: https://aarnet.edu.au
     contact: AARNet Operations Center
     contact_email: mirror@aarnet.edu.au
     mirror_url: http://mirror.aarnet.edu.au/pub/bioconductor
     https_mirror_url: https://mirror.aarnet.edu.au/pub/bioconductor
     rsync: secure_mirror_from_master
     city: Sydney
     country_code: au
 - Denmark:
   - institution: dotsrc
     institution_url: https://dotsrc.org/
     contact: Anders Trier Olesen
     contact_email: staff@dotsrc.org
     mirror_url: http://mirrors.dotsrc.org/bioconductor/
     https_mirror_url: https://mirrors.dotsrc.org/bioconductor/
     rsync: secure_mirror_from_master
     city: Aalborg
     country_code: dk
 - Sweden:
   - institution: Academic Computer Club Umea
     institution_url: https://www.accum.se/
     contact: FTP admins
     contact_email: ftp-adm@accum.se
     mirror_url: http://mirror.accum.se/mirror/bioconductor.org/
     https_mirror_url: https://mirror.accum.se/mirror/bioconductor.org/
     rsync: secure_mirror_from_master
     city: Umea
     country_code: se

Again, it seems like there is an issue with referencing the old bioconductor website, no?

[mtmorgan]

For what it's worth, BiocManager is using download.file() (to avoid a dependency on httr). So one would like to look at ``` download.file("https://bioconductor.org/config.yaml", fl <- tempfile()) ``` and then validate the content of fl. The vignette has a section 'Offline config.yaml' that would allow you to continue working while this issue gets resolved https://bioconductor.github.io/BiocManager/articles/BiocManager.html#offline-config-yaml.

[aehrenberg]

This is helpful info. I got the same result using the download.file function as before (didn't work with the regular url but did seem to work for the new url. I was able to get the config.yaml offline, but am getting hit with a ton of errors when I try to do basic things like install Deseq...

BiocManager::install("DESeq2")
'getOption("repos")' replaces Bioconductor standard repositories, see 'help("repositories", package = "BiocManager")'
for details.
Replacement repositories:
    CRAN: https://cran.rstudio.com/
Warning: unable to access index for repository https://bioconductor.org/packages/3.18/bioc/src/contrib:
  cannot open URL 'https://bioconductor.org/packages/3.18/bioc/src/contrib/PACKAGES'
Warning: unable to access index for repository https://bioconductor.org/packages/3.18/data/annotation/src/contrib:
  cannot open URL 'https://bioconductor.org/packages/3.18/data/annotation/src/contrib/PACKAGES'
Warning: unable to access index for repository https://bioconductor.org/packages/3.18/data/experiment/src/contrib:
  cannot open URL 'https://bioconductor.org/packages/3.18/data/experiment/src/contrib/PACKAGES'
Warning: unable to access index for repository https://bioconductor.org/packages/3.18/workflows/src/contrib:
  cannot open URL 'https://bioconductor.org/packages/3.18/workflows/src/contrib/PACKAGES'
Warning: unable to access index for repository https://bioconductor.org/packages/3.18/books/src/contrib:
  cannot open URL 'https://bioconductor.org/packages/3.18/books/src/contrib/PACKAGES'
Bioconductor version 3.18 (BiocManager 1.30.22), R 4.3.2 (2023-10-31 ucrt)
Installing package(s) 'BiocVersion', 'DESeq2'
Warning: unable to access index for repository https://bioconductor.org/packages/3.18/bioc/src/contrib:
  cannot open URL 'https://bioconductor.org/packages/3.18/bioc/src/contrib/PACKAGES'
Warning: unable to access index for repository https://bioconductor.org/packages/3.18/data/annotation/src/contrib:
  cannot open URL 'https://bioconductor.org/packages/3.18/data/annotation/src/contrib/PACKAGES'
Warning: unable to access index for repository https://bioconductor.org/packages/3.18/data/experiment/src/contrib:
  cannot open URL 'https://bioconductor.org/packages/3.18/data/experiment/src/contrib/PACKAGES'
Warning: unable to access index for repository https://bioconductor.org/packages/3.18/workflows/src/contrib:
  cannot open URL 'https://bioconductor.org/packages/3.18/workflows/src/contrib/PACKAGES'
Warning: unable to access index for repository https://bioconductor.org/packages/3.18/books/src/contrib:
  cannot open URL 'https://bioconductor.org/packages/3.18/books/src/contrib/PACKAGES'
Warning: unable to access index for repository https://bioconductor.org/packages/3.18/bioc/bin/windows/contrib/4.3:
  cannot open URL 'https://bioconductor.org/packages/3.18/bioc/bin/windows/contrib/4.3/PACKAGES'
Warning: unable to access index for repository https://bioconductor.org/packages/3.18/data/annotation/bin/windows/contrib/4.3:
  cannot open URL 'https://bioconductor.org/packages/3.18/data/annotation/bin/windows/contrib/4.3/PACKAGES'
Warning: unable to access index for repository https://bioconductor.org/packages/3.18/data/experiment/bin/windows/contrib/4.3:
  cannot open URL 'https://bioconductor.org/packages/3.18/data/experiment/bin/windows/contrib/4.3/PACKAGES'
Warning: unable to access index for repository https://bioconductor.org/packages/3.18/workflows/bin/windows/contrib/4.3:
  cannot open URL 'https://bioconductor.org/packages/3.18/workflows/bin/windows/contrib/4.3/PACKAGES'
Warning: unable to access index for repository https://bioconductor.org/packages/3.18/books/bin/windows/contrib/4.3:
  cannot open URL 'https://bioconductor.org/packages/3.18/books/bin/windows/contrib/4.3/PACKAGES'
Warning: unable to access index for repository https://bioconductor.org/packages/3.18/bioc/src/contrib:
  cannot open URL 'https://bioconductor.org/packages/3.18/bioc/src/contrib/PACKAGES'
Warning: unable to access index for repository https://bioconductor.org/packages/3.18/data/annotation/src/contrib:
  cannot open URL 'https://bioconductor.org/packages/3.18/data/annotation/src/contrib/PACKAGES'
Warning: unable to access index for repository https://bioconductor.org/packages/3.18/data/experiment/src/contrib:
  cannot open URL 'https://bioconductor.org/packages/3.18/data/experiment/src/contrib/PACKAGES'
Warning: unable to access index for repository https://bioconductor.org/packages/3.18/workflows/src/contrib:
  cannot open URL 'https://bioconductor.org/packages/3.18/workflows/src/contrib/PACKAGES'
Warning: unable to access index for repository https://bioconductor.org/packages/3.18/books/src/contrib:
  cannot open URL 'https://bioconductor.org/packages/3.18/books/src/contrib/PACKAGES'
Warning message:
packages ‘BiocVersion’, ‘DESeq2’ are not available for Bioconductor version '3.18'

Versions of these packages for your version of R might be available elsewhere,
see the ideas at
https://cran.r-project.org/doc/manuals/r-patched/R-admin.html#Installing-packages 

[mtmorgan]

What specifically is the output with

download.file("https://bioconductor.org/config.yaml", fl <- tempfile())

? Solving this problem will, I bet, solve all your problems. It is not likely an issue with bioconductor.org per se.

What is sessionInfo()? Have you tried alternative download methods, via options(download.file.method=) as documented on ?download.file

[aehrenberg]

When I put that in, I get the following error and the tempfile path, fl, maps to a file that doesn't exist:

download.file("https://bioconductor.org/config.yaml", fl <- tempfile())

trying URL 'https://bioconductor.org/config.yaml'
Error in download.file("https://bioconductor.org/config.yaml", fl <- tempfile()) : 
  cannot open URL 'https://bioconductor.org/config.yaml'
In addition: Warning message:
In download.file("https://bioconductor.org/config.yaml", fl <- tempfile()) :
  URL 'https://bioconductor.org/config.yaml': status was 'SSL connect error'

Session info is as follows:

R version 4.3.2 (2023-10-31 ucrt)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 11 x64 (build 22621)

Matrix products: default


locale:
[1] LC_COLLATE=English_United States.utf8  LC_CTYPE=English_United States.utf8    LC_MONETARY=English_United States.utf8
[4] LC_NUMERIC=C                           LC_TIME=English_United States.utf8    

time zone: America/Los_Angeles
tzcode source: internal

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] BiocManager_1.30.22

loaded via a namespace (and not attached):
[1] compiler_4.3.2 tools_4.3.2   

I tried running the following:

if (!require("BiocManager", quietly = TRUE))
    install.packages("BiocManager")
BiocManager::install(version = "3.18")

on all download.file.method options ("internal", "libcurl", "wget", "curl" and "wininet") and got the same error each time:

Error: Bioconductor version cannot be validated; no internet connection?  See #troubleshooting section in vignette

[mtmorgan]

Did you say that you were behind a firewall? If so, what specifically have you done to set up a proxy? If this is at an institution or company and you cannot browse to https://bioconductor.org, I would suggest contacting your local tech support with that request 'help me connect to...'

If this were a general problem, then likely there would have been much more gnashing of teeth on the support site / mailing list / slack.

[lokapal]

1. Add to your .Renviron the following string:

CURL_SSL_BACKEND=openssl

2. Add to your .Rprofile the following string:

options(download.file.method="libcurl", url.method="libcurl")

Restart R. Enjoy!