Migrate to Python 3 and add 35 new conference scrapers (214k papers) - #3
Open
justi wants to merge 31 commits into
Open
Migrate to Python 3 and add 35 new conference scrapers (214k papers)#3justi wants to merge 31 commits into
justi wants to merge 31 commits into
Conversation
- cPickle -> pickle, cStringIO -> io, file() -> open() - print statements -> print() functions - dict.has_key() -> 'key' in dict / 'key' not in dict - urllib -> urllib.request / urllib.parse - BeautifulSoup v3 -> bs4 (beautifulsoup4) - os.startfile (Windows-only) -> webbrowser.open (cross-platform) - PDFMiner old API (process_pdf) -> pdfminer.six high-level API (extract_text) - google_search.py: replace dead Google AJAX API with basic scraping fallback - nips_download_parse.py: update to proceedings.neurips.cc, years 2006-2023 - Binary mode for pickle files (wb/rb) - Add requirements.txt with beautifulsoup4 and pdfminer.six Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The proceedings site uses <ul class="paper-list"> with <li> entries containing <a title="paper title"> and <span class="paper-authors">. Also adds User-Agent header and extracts PDF links. Successfully scrapes 17,366 papers from 2006-2023. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Years without proceedings are skipped gracefully via 404 handling. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…onary In Python 3 dict.keys() returns a view, not a list, so deleting keys while iterating raised RuntimeError. Replaced with iterating over stopwords and using dict.pop(). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- nips_add_pdftext.py: update PDF URL construction for new neurips.cc format (hash->file, Abstract->Paper), remove dead NIPS local fallback - repool_util.py: fix missing comma between 'those' and 'may' in stopwords list (strings were silently concatenated to 'thosemay') Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Scrapes ICML proceedings from 2013-2025 (14,281 papers). Extracts title, authors, and direct PDF links from GitHub. Update .gitignore to cover all pubs_* pickle files. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Scrapes CVPR proceedings from 2013-2025 (18,452 papers). Handles both old (.py suffix, per-day pages) and new URL formats. Falls back to per-day fetching when day=all is not supported. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Scrapes ACL proceedings from 2000-2025 (21,490 papers). Extracts title, authors, and PDF links. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Scrapes ICCV proceedings from 2013-2025 (9,145 papers). ICCV occurs every 2 years (odd years only). Reuses same parsing logic as CVPR scraper. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Scrapes EMNLP proceedings from 2000-2025 (20,646 papers). Same parsing logic as ACL scraper. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Scrapes NAACL proceedings from 2000-2025 (8,925 papers). NAACL doesn't run every year; missing years are skipped gracefully. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Scrapes WACV proceedings from 2020-2026 (4,435 papers). Reuses same parsing logic as CVPR/ICCV scrapers. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Scrapes AAAI proceedings from 2019-2025 (15,097 papers). Collects issue URLs from paginated archive, then parses papers from each issue page. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Scrapes AISTATS proceedings from 2010-2025 (4,613 papers). Same parsing logic as ICML scraper. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Scrapes COLT (Conference on Learning Theory) proceedings from 2011-2025 (1,598 papers). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- COLING (aclanthology.org): 2000-2025, 8,740 papers - EACL (aclanthology.org): 2003-2024, 3,138 papers - UAI (proceedings.mlr.press): 2019-2025, 1,368 papers - CoRL (proceedings.mlr.press): 2017-2025, 1,490 papers - ALT (proceedings.mlr.press): 2017-2025, 381 papers Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- IJCAI (ijcai.org): 2013-2025, 9,940 papers - ECCV (ecva.net): 2018-2024, 6,166 papers Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- CoNLL (aclanthology.org): 2000-2025, 1,547 papers - SemEval (aclanthology.org): 2007-2025, 3,217 papers - AACL (aclanthology.org): 2020-2025, 938 papers - ACML (proceedings.mlr.press): 2010-2024, 834 papers - RSS (roboticsproceedings.org): 2005-2025, 1,469 papers - INTERSPEECH (isca-archive.org): 2016-2024, 8,785 papers Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- JMLR (jmlr.org): 2000-2025, 4,135 papers (journal) - ICLR (openreview.net API): 2018-2025, 11,015 papers Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ICCAI, OSDI, NSDI PMLR (proceedings.mlr.press): - MIDL: 2019-2024, 504 papers - L4DC: 2020-2025, 669 papers - MLHC: 2016-2025, 323 papers - AutoML: 2016-2025, 97 papers - CLeaR: 2022-2025, 190 papers - PGM: 2016-2024, 222 papers ACL Anthology: - IJCNLP: 2005-2025, 5,337 papers New sources: - MICCAI (papers.miccai.org): 2024-2025, 1,883 papers - OSDI (usenix.org): 2012-2025, 472 papers - NSDI (usenix.org): 2012-2025, 821 papers Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Migrates the project to Python 3 and significantly expands stage-1 coverage by adding many new conference/journal scrapers that normalize proceedings into pubs_* pickle files for downstream querying and analysis.
Changes:
- Ported core utilities and analysis/demo scripts to Python 3 (pickle I/O, printing,
has_key, PDF parsing, browser opening). - Added a large set of new
*_download_parse.pyscrapers across multiple sources (CVF, ACL Anthology, PMLR, OpenReview, USENIX, etc.). - Added baseline packaging hygiene (
requirements.txt,.gitignore) and updated README to reflect the expanded dataset and usage.
Reviewed changes
Copilot reviewed 45 out of 47 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
.gitignore |
Ignores caches, generated pubs_* pickles, and temp PDF artifacts. |
README |
Updates installation, usage, and the list of supported venues/scrapers. |
requirements.txt |
Adds runtime deps for bs4 and pdfminer.six. |
repool_util.py |
Python 3 pickle I/O (binary) and cross-platform PDF opening via webbrowser. |
repool_analysis.py |
Python 3 updates for similarity scoring logic/output. |
pdf_read.py |
Migrates PDF extraction to pdfminer.six high-level API. |
google_search.py |
Replaces retired Google AJAX API with basic HTML scraping approach. |
demo1.py |
Python 3 compatibility updates for MNIST-related example. |
demo2.py |
Python 3 compatibility updates for “deep in title” example. |
demo3.py |
Python 3 compatibility updates for similarity-search example. |
nips_download_parse.py |
Replaces old NIPS books site scraping with proceedings.neurips.cc scraper. |
nips_add_pdftext.py |
Updates stage-2 PDF text enrichment to work with new NeurIPS URL patterns. |
aaai_download_parse.py |
Adds AAAI proceedings scraper from ojs.aaai.org archive. |
aacl_download_parse.py |
Adds AACL proceedings scraper from ACL Anthology. |
acl_download_parse.py |
Adds ACL proceedings scraper from ACL Anthology. |
acml_download_parse.py |
Adds ACML proceedings scraper from PMLR. |
aistats_download_parse.py |
Adds AISTATS proceedings scraper from PMLR. |
alt_download_parse.py |
Adds ALT proceedings scraper from PMLR. |
automl_download_parse.py |
Adds AutoML proceedings scraper from PMLR. |
clear_download_parse.py |
Adds CLeaR proceedings scraper from PMLR. |
coling_download_parse.py |
Adds COLING proceedings scraper from ACL Anthology. |
colt_download_parse.py |
Adds COLT proceedings scraper from PMLR. |
conll_download_parse.py |
Adds CoNLL proceedings scraper from ACL Anthology. |
corl_download_parse.py |
Adds CoRL proceedings scraper from PMLR. |
cvpr_download_parse.py |
Adds CVPR proceedings scraper from CVF Open Access. |
eacl_download_parse.py |
Adds EACL proceedings scraper from ACL Anthology. |
eccv_download_parse.py |
Adds ECCV proceedings scraper from ecva.net. |
emnlp_download_parse.py |
Adds EMNLP proceedings scraper from ACL Anthology. |
iccv_download_parse.py |
Adds ICCV proceedings scraper from CVF Open Access. |
iclr_download_parse.py |
Adds ICLR scraper using OpenReview APIs (v1/v2) across multiple year formats. |
icml_download_parse.py |
Adds ICML proceedings scraper from PMLR. |
ijcai_download_parse.py |
Adds IJCAI proceedings scraper handling both old/new site formats. |
ijcnlp_download_parse.py |
Adds IJCNLP proceedings scraper from ACL Anthology. |
interspeech_download_parse.py |
Adds INTERSPEECH proceedings scraper from ISCA archive. |
jmlr_download_parse.py |
Adds JMLR scraper with year extraction from metadata and PDF normalization. |
l4dc_download_parse.py |
Adds L4DC proceedings scraper from PMLR. |
miccai_download_parse.py |
Adds MICCAI scraper from papers.miccai.org (limited years). |
midl_download_parse.py |
Adds MIDL proceedings scraper from PMLR. |
mlhc_download_parse.py |
Adds MLHC proceedings scraper from PMLR. |
naacl_download_parse.py |
Adds NAACL proceedings scraper from ACL Anthology. |
nsdi_download_parse.py |
Adds NSDI proceedings scraper from usenix.org technical-sessions pages. |
osdi_download_parse.py |
Adds OSDI proceedings scraper from usenix.org technical-sessions pages. |
pgm_download_parse.py |
Adds PGM proceedings scraper from PMLR. |
rss_download_parse.py |
Adds RSS proceedings scraper from roboticsproceedings.org. |
semeval_download_parse.py |
Adds SemEval proceedings scraper from ACL Anthology. |
uai_download_parse.py |
Adds UAI proceedings scraper from PMLR using volume-to-year mapping. |
wacv_download_parse.py |
Adds WACV proceedings scraper from CVF Open Access. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
Comment on lines
+47
to
+49
| new_pub['title'] = title_tag.text.strip() | ||
| new_pub['pdf'] = 'https://proceedings.neurips.cc' + title_tag['href'] | ||
|
|
Comment on lines
+23
to
+25
| # convert abstract page URL to direct PDF URL | ||
| pdf_url = p['pdf'].replace('-Abstract-Conference.html', '-Paper-Conference.pdf').replace('/hash/', '/file/') | ||
|
|
Comment on lines
+5
to
16
| from pdfminer.high_level import extract_text | ||
| from io import BytesIO | ||
| import urllib.request | ||
|
|
||
| import urllib | ||
|
|
||
| def convertPDF(pdf_path, codec='ascii'): | ||
| def convertPDF(pdf_path, codec='utf-8'): | ||
| """ | ||
| Takes path to a PDF and returns the text inside it as string | ||
| pdf_path: string indicating path to a .pdf file. Can also be a URL starting | ||
|
|
||
| pdf_path: string indicating path to a .pdf file. Can also be a URL starting | ||
| with 'http' | ||
| codec: can be 'ascii', 'utf-8', ... | ||
| returns string of the pdf, as it comes out raw from PDFMiner |
Comment on lines
19
to
22
| if pdf_path[:4] == 'http': | ||
| print 'first downloading %s ...' % (pdf_path,) | ||
| urllib.urlretrieve(pdf_path, 'temp.pdf') | ||
| print('first downloading %s ...' % (pdf_path,)) | ||
| urllib.request.urlretrieve(pdf_path, 'temp.pdf') | ||
| pdf_path = 'temp.pdf' |
Comment on lines
34
to
36
| # a random thing I just thought of 5 seconds ago | ||
| overlap = sum([1 for x in words if x in p['pdf_text'].keys()]) | ||
| scores[i] = 2.0 * overlap / (wnum_train + wnum_test) |
| import simplejson | ||
| import urllib.request | ||
| import urllib.parse | ||
| import json |
Comment on lines
+16
to
+35
| def fetch(url): | ||
| req = urllib.request.Request(url, headers={'User-Agent': 'Mozilla/5.0'}) | ||
| with urllib.request.urlopen(req) as f: | ||
| return f.read() | ||
|
|
||
|
|
||
| def get_pages_for_year(year): | ||
| if year <= 2020: | ||
| base = "%s/WACV%d.py" % (BASE_URL, year) | ||
| else: | ||
| base = "%s/WACV%d" % (BASE_URL, year) | ||
|
|
||
| try: | ||
| html = fetch(base + "?day=all") | ||
| soup = BeautifulSoup(html, 'html.parser') | ||
| if soup.find('dt', {'class': 'ptitle'}): | ||
| return [html] | ||
| except: | ||
| pass | ||
|
|
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- pdf_read.py: remove unused codec param and BytesIO import, use tempfile.NamedTemporaryFile instead of fixed temp.pdf, add timeout - google_search.py: remove unused json import - repool_analysis.py: use set intersection for overlap calculation - nips_add_pdftext.py: only rewrite URL if it contains Abstract HTML - wacv/cvpr/iccv scrapers: add timeout=30, replace bare except Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- test_repool_util.py: save/load pickle roundtrip, stringToWordDictionary (stopwords, dict mutation regression, those/may comma fix) - test_repool_analysis.py: similarity scoring with set intersection - test_pdf_read.py: removed codec param, tempfile usage, no fixed temp.pdf - test_scrapers.py: fetch() timeout, no bare except, URL rewrite logic Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Copilot review: pdf field should point to actual PDF, not abstract page. Now pdf contains direct PDF URL (.../file/...-Paper-Conference.pdf) and url contains the abstract page (.../hash/...-Abstract-Conference.html). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
13 tests hitting NeurIPS, ICML, CVPR, ACL, AAAI, OpenReview, and PDF extraction. Verify HTML structure, paper counts, PDF links. OpenReview test skips gracefully on 403. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Handle three abstract URL patterns: - -Abstract-Conference.html (2021+) - -Abstract-Datasets_and_Benchmarks.html (2021+ datasets track) - -Abstract.html (2006-2020) Now 97.9% of URLs point to actual PDFs. Verified with full scrape + demo1/2/3 pipeline on fresh clone. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- test_google_search.py: PDF extraction, notfound fallback, error handling - test_scrapers.py: all 3 NeurIPS URL formats (Conference, Datasets, old -Abstract.html), no-double-rewrite, unchanged for other sources Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Migrates the project to Python 3 and significantly expands Stage 1 coverage by adding many new conference scrapers, plus accompanying tests and docs updates to support the larger dataset and updated dependencies.
Changes:
- Port core utilities and pipeline scripts to Python 3 (pickle I/O, URL handling, PDF extraction via
pdfminer.six, cross-platform PDF opening). - Add many new
*_download_parse.pyscrapers to generatepubs_*pickles across additional conferences/sources. - Add a pytest test suite (unit + integration) and update docs/config (
requirements.txt,.gitignore, README).
Reviewed changes
Copilot reviewed 51 out of 53 changed files in this pull request and generated 12 comments.
Show a summary per file
| File | Description |
|---|---|
| .gitignore | Ignore caches and generated pubs_* outputs. |
| README | Update installation, usage, and list of available scrapers/conferences. |
| requirements.txt | Add runtime dependencies for HTML parsing and PDF extraction. |
| repool_util.py | Python 3 pickle I/O + cross-platform PDF opening + stopword/dict-iteration fix. |
| repool_analysis.py | Python 3 updates + set-based overlap for similarity scoring. |
| pdf_read.py | Migrate to pdfminer.six high-level API and URL download path. |
| google_search.py | Replace retired Google AJAX API with HTML scraping approach. |
| nips_download_parse.py | Update NeurIPS scraping to proceedings.neurips.cc with dynamic year range + PDF URL rewrite. |
| nips_add_pdftext.py | Update PDF-text enrichment to use new PDF URL rewriting and Python 3 conventions. |
| demo1.py | Python 3 print/update and doc cleanup. |
| demo2.py | Python 3 print/update and doc cleanup. |
| demo3.py | Python 3 print/update and doc cleanup. |
| tests/test_scrapers.py | Add tests enforcing fetch timeout/bare-except constraints for selected scrapers. |
| tests/test_repool_util.py | Add unit tests for pickle roundtrip and word-dictionary behavior/regressions. |
| tests/test_repool_analysis.py | Add unit tests for publicationSimilarityNaive. |
| tests/test_pdf_read.py | Add unit tests for convertPDF behavior (timeouts/tempfiles/no temp.pdf). |
| tests/test_google_search.py | Add unit tests for new getPDFURL behavior via mocking. |
| tests/test_integration.py | Add live-site integration tests for multiple sources (NeurIPS/PMLR/CVF/ACL/etc.). |
| aaai_download_parse.py | New AAAI scraper (OJS) with archive traversal. |
| aacl_download_parse.py | New AACL scraper (ACL Anthology). |
| acl_download_parse.py | New ACL scraper (ACL Anthology). |
| acml_download_parse.py | New ACML scraper (PMLR). |
| aistats_download_parse.py | New AISTATS scraper (PMLR). |
| alt_download_parse.py | New ALT scraper (PMLR). |
| automl_download_parse.py | New AutoML scraper (PMLR). |
| clear_download_parse.py | New CLeaR scraper (PMLR). |
| coling_download_parse.py | New COLING scraper (ACL Anthology). |
| colt_download_parse.py | New COLT scraper (PMLR). |
| conll_download_parse.py | New CoNLL scraper (ACL Anthology). |
| corl_download_parse.py | New CoRL scraper (PMLR). |
| cvpr_download_parse.py | New CVPR scraper (CVF Open Access). |
| eacl_download_parse.py | New EACL scraper (ACL Anthology). |
| eccv_download_parse.py | New ECCV scraper (ECVA). |
| emnlp_download_parse.py | New EMNLP scraper (ACL Anthology). |
| iccv_download_parse.py | New ICCV scraper (CVF Open Access). |
| iclr_download_parse.py | New ICLR scraper (OpenReview API v1/v2 support). |
| icml_download_parse.py | New ICML scraper (PMLR). |
| ijcai_download_parse.py | New IJCAI scraper with multi-format parsing support. |
| ijcnlp_download_parse.py | New IJCNLP scraper (ACL Anthology). |
| interspeech_download_parse.py | New INTERSPEECH scraper (ISCA archive). |
| jmlr_download_parse.py | New JMLR scraper (jmlr.org) with year extraction. |
| l4dc_download_parse.py | New L4DC scraper (PMLR). |
| miccai_download_parse.py | New MICCAI scraper (papers.miccai.org). |
| midl_download_parse.py | New MIDL scraper (PMLR). |
| mlhc_download_parse.py | New MLHC scraper (PMLR). |
| naacl_download_parse.py | New NAACL scraper (ACL Anthology). |
| nsdi_download_parse.py | New NSDI scraper (USENIX). |
| osdi_download_parse.py | New OSDI scraper (USENIX). |
| pgm_download_parse.py | New PGM scraper (PMLR). |
| rss_download_parse.py | New RSS scraper (roboticsproceedings.org). |
| semeval_download_parse.py | New SemEval scraper (ACL Anthology). |
| uai_download_parse.py | New UAI scraper (PMLR). |
| wacv_download_parse.py | New WACV scraper (CVF Open Access). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
Comment on lines
18
to
+25
| if pdf_path[:4] == 'http': | ||
| print 'first downloading %s ...' % (pdf_path,) | ||
| urllib.urlretrieve(pdf_path, 'temp.pdf') | ||
| pdf_path = 'temp.pdf' | ||
|
|
||
| rsrcmgr = PDFResourceManager() | ||
| retstr = StringIO() | ||
| laparams = LAParams() | ||
| device = TextConverter(rsrcmgr, retstr, codec=codec, laparams=laparams) | ||
|
|
||
| fp = file(pdf_path, 'rb') | ||
| process_pdf(rsrcmgr, device, fp) | ||
| fp.close() | ||
| device.close() | ||
|
|
||
| str = retstr.getvalue() | ||
| retstr.close() | ||
|
|
||
| return str | ||
| print('first downloading %s ...' % (pdf_path,)) | ||
| with tempfile.NamedTemporaryFile(suffix='.pdf', delete=False) as tmp: | ||
| with urllib.request.urlopen(pdf_path, timeout=30) as resp: | ||
| tmp.write(resp.read()) | ||
| pdf_path = tmp.name | ||
|
|
||
| return extract_text(pdf_path) |
Comment on lines
+1
to
+23
| """ | ||
| Integration tests that hit real conference proceedings sites. | ||
| These verify that scrapers still work against current HTML structures. | ||
|
|
||
| Run with: pytest tests/test_integration.py -v | ||
| Slower than unit tests (~30s) as they make real HTTP requests. | ||
| """ | ||
|
|
||
| import os | ||
| import sys | ||
| import urllib.request | ||
| import pytest | ||
| from bs4 import BeautifulSoup | ||
|
|
||
| sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) | ||
|
|
||
| HEADERS = {'User-Agent': 'Mozilla/5.0'} | ||
|
|
||
|
|
||
| def fetch(url, timeout=15): | ||
| req = urllib.request.Request(url, headers=HEADERS) | ||
| with urllib.request.urlopen(req, timeout=timeout) as f: | ||
| return f.read() |
Comment on lines
+24
to
35
| words = set(test_pub['pdf_text'].keys()) | ||
|
|
||
| for i, p in enumerate(train_pubs): | ||
| if(i % 100 == 0): print("%d/%d..." % (i, len(train_pubs))) | ||
|
|
||
| if 'pdf_text' not in p: continue | ||
|
|
||
| # find score of the match | ||
| wnum_train = len(p['pdf_text']) | ||
|
|
||
| #a random thing I just thought of 5 seconds ago | ||
| overlap = sum([1 for x in words if x in p['pdf_text'].keys()]) | ||
|
|
||
| overlap = len(words & set(p['pdf_text'].keys())) | ||
| scores[i] = 2.0 * overlap / (wnum_train + wnum_test) |
Comment on lines
+15
to
+19
|
|
||
|
|
||
| def fetch(url): | ||
| req = urllib.request.Request(url, headers={'User-Agent': 'Mozilla/5.0'}) | ||
| with urllib.request.urlopen(req) as f: |
| url = "%s/issue/archive/%d" % (BASE_URL, page) | ||
| try: | ||
| html = fetch(url) | ||
| except: |
Comment on lines
35
to
+42
| try: | ||
| floc = p['pdf'].index('NIPS') | ||
| fname = p['pdf'][floc:] | ||
| txt = convertPDF('downloads/'+fname) | ||
| print('downloading pdf for [%s] and parsing...' % (p.get('title', 'an un-titled paper'))) | ||
| txt = convertPDF(pdf_url) | ||
| processed = True | ||
| print 'found %s in file!' % (p['title'],) | ||
| print('processed!') | ||
| except: | ||
| pass | ||
|
|
||
| if not processed: | ||
| # download the PDF and convert to text | ||
| try: | ||
| print 'downloading pdf for [%s] and parsing...' % (p.get('title', 'an un-titled paper')) | ||
| txt = convertPDF(p['pdf']) | ||
| processed = True | ||
| print 'processed from url!' | ||
| except: | ||
| print 'error: unable to open download the pdf from %s' % (p['pdf'],) | ||
| print 'skipping...' | ||
|
|
||
| print('error: unable to download the pdf from %s' % (pdf_url,)) | ||
| print('skipping...') |
Comment on lines
+46
to
+49
| try: | ||
| p['pdf_text'] = stringToWordDictionary(txt) | ||
| except: | ||
| print 'was unable to convert text to bag of words. Skipped.' | ||
|
|
||
|
|
||
| print '%d/%d = %.2f%% done.' % (i+1, len(pubs), 100*(i+1.0)/len(pubs)) | ||
|
|
||
| savePubs('pubs_nips', pubs_all) No newline at end of file | ||
| print('was unable to convert text to bag of words. Skipped.') |
Comment on lines
+25
to
+31
| req = urllib.request.Request(url, headers={'User-Agent': 'Mozilla/5.0'}) | ||
| try: | ||
| with urllib.request.urlopen(req) as response: | ||
| html = response.read().decode('utf-8') | ||
| except Exception as e: | ||
| print('Error searching Google: %s' % (e,)) | ||
| return 'notfound' |
Comment on lines
+30
to
+33
| req = urllib.request.Request(url, headers={'User-Agent': 'Mozilla/5.0'}) | ||
| try: | ||
| with urllib.request.urlopen(req) as f: | ||
| s = f.read() |
Comment on lines
+30
to
+34
| def fetch_json(url): | ||
| """Fetch a URL and return parsed JSON.""" | ||
| req = urllib.request.Request(url, headers=HEADERS) | ||
| with urllib.request.urlopen(req) as f: | ||
| return json.loads(f.read()) |
- pdf_read.py: clean up temp file in finally block, add User-Agent - test_integration.py: skip unless RUN_INTEGRATION_TESTS=1 - repool_analysis.py: use dict lookup O(1) instead of set() - aaai, nips, google_search, uai, iclr scrapers: add timeout - aaai, nips_add_pdftext: replace bare except with Exception as e - cvpr: document fetch() returns bytes Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Full Python 3 migration of the original codebase plus 35 new scrapers covering all major open-access AI/ML/NLP/CV conferences. Total: 214,222 papers from 36 conferences across 12 sources.
Python 3 Migration
cPickle->pickle,cStringIO->io,file()->open()printstatements ->print()functionsdict.has_key()->'key' in dicturllib->urllib.request/urllib.parseBeautifulSoupv3 ->beautifulsoup4(bs4)os.startfile(Windows-only) ->webbrowser.open(cross-platform)process_pdf) ->pdfminer.six(extract_text)google_search.py: replaced dead Google AJAX APInips_download_parse.py: updated toproceedings.neurips.cc, dynamic year rangedictionary changed size during iterationbug instringToWordDictionary'those'+'may'were concatenated)wb/rb)requirements.txtwithbeautifulsoup4andpdfminer.six.gitignoreNew Scrapers (35 conferences)
All scrapers follow the same pattern as the original
nips_download_parse.py: fetch proceedings HTML, parse with BeautifulSoup, save as pickle.Test plan
loadPubs/savePubswork with new binary pickle formatstringToWordDictionaryno longer crashes (dict iteration fix)🤖 Generated with Claude Code