Classification export fails on entrez error

[davmlaw]

https://shariant.org.au/classification/export_redirect?share_level=public&build=GRCh38&type=json&exclude_orgs=sa_pathology

Died with:

Traceback (most recent call last):
  File "/opt/variantgrid/annotation/models/models_citations.py", line 678, in _load_from_entrez
    records = list(Medline.parse(handle))
  File "/opt/variantgrid/env/lib/python3.11/site-packages/Bio/Medline/__init__.py", line 168, in parse
    for line in handle:
  File "/usr/lib/python3.11/http/client.py", line 678, in read1
    return self._read1_chunked(n)
  File "/usr/lib/python3.11/http/client.py", line 719, in _read1_chunked
    chunk_left = self._get_chunk_left()
  File "/usr/lib/python3.11/http/client.py", line 580, in _get_chunk_left
    raise IncompleteRead(b'')
IncompleteRead: IncompleteRead(0 bytes read)Traceback (most recent call last):
  File "/usr/lib/python3.11/http/client.py", line 578, in _get_chunk_left
    chunk_left = self._read_next_chunk_size()
  File "/usr/lib/python3.11/http/client.py", line 545, in _read_next_chunk_size
    return int(line, 16)
ValueError: invalid literal for int() with base 16: b''

I have tried to recreate this, but can't. I suspect because this is cached?

[davmlaw]

Was able to get a somewhat similar issue to happen on Shariant test (had to switch to system w/o cache)

This was done purely in GUI via
https://test.shariant.org.au/classification/api/classifications/export?share_level=public&build=GRCh38&type=json&exclude_orgs=sa_pathology

https://app.rollbar.com/a/jimmy.andrews/fix/item/VariantGrid/6272/occurrence/466143923390#detail

HTTPError HTTP Error 400: Bad Request
_load_from_entrez(/opt/variantgrid/annotation/models/models_citations.py:679)
handle = Entrez.efetch(db=entrez_db, id=request_ids, rettype='medline', retmode='text')

Do we only retrieve this stuff with the JSON export?

Maybe we should have a task to do this offline (scheduled or as things come in) - would make exports faster and more robust

[TheMadBug]

I was confused by this because JSON download didn't used to get the citations, but checking the code, we changed it so it did for the purposes of the cut down export.

Specifically instead of providing literature with the full free text of literature, we now present it with the full list of citations scanned from the record.

(The MVL download also used to do this but that's not relevant anymore).

We load the citations if somebody looks at a classification form that includes them, or on download.

Import process takes long enough and is intensive enough as it is so I wouldn't want to do it then, but a scheduled process to prefetched any outstanding PubMed links would be good.

[EmmaTudini]

Discussed 13/08/2026 - Retrieve pubmed links nightly or upon import, instead of lazily upon first export of data.
Options: 1. Nightly job. 2. Could do as async upon import (not blocking import). James preference is nightly job. Could also do both? Dave in favour of this.

[davmlaw]

🤖 Written by Claude

This is a separate problem from #1780 (that one is Monarch condition-text automatching during upload; this is Entrez/PubMed citation loading during export), so it is still open.

However PR #1781 is a good reference template for the nightly/async approach discussed above: it uses a pending_automatch flag on the model plus a periodic celery beat sweep (classification/tasks/condition_text_automatch_task.py) that claims each pending row with a single-statement UPDATE ... WHERE pending and does the external API call outside any transaction. The same shape would work here — flag citations needing an Entrez fetch on import, then have a scheduled task drain them in batches so export never has to call PubMed lazily.