scholar2sql streamlines literature reviews by automatically extracting structured data from scientific papers and storing it in a SQL database. Overall, this package offers two key advantages: (1) automated literature review processing (parsing, extraction, and structured organization of key information) and (2) proposed literature screening through customizable input parameters targeting your personal literature research outcomes.
This tool was used for the creation of the sql table of the drug screening literature review on ion channel found here on channelpedia. The configuration file is located in the examples folder of this repository.
Scholar2Sql mimics human expert literature review, extracting key features from articles, but scales the process to analyze a vastly larger number of publications than possible manually.
collapse
The process of getting the Open-Access article: 1) first get the full-text from pubmed central 2) download the abstract 3) download the pdf and 4) parse it using GROBID.
collapse
Scholar2sql extracts features from text with sources by: 1) using a RAG architecture. (the best sections are found using a bm25 algorithm.) 2) Engineering the prompt by combining the best sections to a template (determined by the output features) and the examples (helping the model behavior). Finally ChatGPT fills the template using the informations from the sections with the help of the examples.
collapse
Once you have defined your research question along with the input parameters (aw many as you like). Scholar2sql will then loop over them and for each find the most relevant articles, loop again and for each article: 1) try to download its content (concept 1) 2) extract the features (concept 2) and 3) fill the sql table.
Automated literature screening with large language models (LLMs) offers significant advantages for navigating the intricate world of scientific literature:
Researchers are finding new ways to use large language models (LLMs) to help process and analyze information. Our project aims to make the insights from these LLMs more widely available by organizing them in an easy-to-understand format. This could help people from various backgrounds access and use this information. Rather than having to repeatedly ask LLMs questions, which can be time-consuming and costly, our approach offers a way to access pre-organized information. We hope this might make research more efficient and accessible for more people.
The sheer volume of scientific publications can be overwhelming, making manual reviews impractical. Our approach aims to create a tool that not only processes large datasets swiftly but also facilitates the comparison of scientific results. This helps in distinguishing truth from falsehood, filtering out irrelevant information, and pinpointing inconsistencies and errors in the literature, thereby highlighting critical areas for further investigation.
git clone https://github.com/journeadrien/scholar2sql.git
pip install .
- Create a YAML configuration file, many fields are required (see Configuration section for details). This is a long process; take your time! There are examples provided in the 'examples' folder.
- To enable pdf parsing, start a grobid service (I recommend using docker here)
- Try loading the config file and resolve the validation errors (if they occur)
s2s_load_settings path_to_config.yaml # s2s_load_settings examples/ic_drug.yaml- reset sql table
s2s_reset_sql_table path_to_config.yaml # s2s_reset_sql_table examples/ic_drug.yaml- run
s2s_run path_to_config.yaml # s2s_run examples/ic_drug.yamlThe package requires a YAML configuration file which specifies various settings. Below are explanations of each section in the configuration file:
| Field | Description | Required |
|---|---|---|
| host | Database host (use "localhost" for local) | Yes |
| username | Database username | Yes |
| password | Database password | Yes |
| database | Name of the database | Yes |
| table | Name of the table | Yes |
| Field | Description | Default |
|---|---|---|
| to_save | Metadata to add to sql (["pubmed_id", "format", "sections"]) | ["pubmed_id", "format", "sections"] |
| Field | Description | Options | Default |
|---|---|---|---|
| level | Logging level for the main package | debug, info, warn, error | info |
| external_packages | Logging level for external packages | debug, info, warn, error | error |
| Field | Description | Required | Default |
|---|---|---|---|
| top_sections_per_article | Number of top sections to select per article using BM25 | No | 5 |
| Email for PubMed and Unpaywall API | Yes |
| Field | Description | Required | Default |
|---|---|---|---|
| top_articles_per_search | Number of top articles to select per PubMed search | No | 10 |
| api_key | PubMed API key | No | "" |
| additional_search_keywords | Additional keywords to include in PubMed search | No | "" |
| tmp_pmc_folder | Folder to store PubMed Central XML files | No | tmp/pmc |
| tmp_abstract_folder | Folder to store abstracts from PubMed | No | tmp/abstract |
| Field | Description | Required | Default |
|---|---|---|---|
| url | URL of the GROBID service | Yes | |
| tmp_pdf_folder | Folder to store downloaded PDF files | No | tmp/pdf |
| tmp_tei_folder | Folder to store TEI files (GROBID format) | No | tmp/tei |
| Field | Description | Required | Default |
|---|---|---|---|
| token | OpenAI API key | Yes | |
| model | Name of the OpenAI model to use | No | gpt-o-mini |
| temperature | OpenAI API key | No | 0. |
| verbose | Whether to display full prompts | No | False |
| Field | Description | Required | Default |
|---|---|---|---|
| overwrite_existing | Whether to overwrite existing records in the database | No | False |
This section defines the research goal, questions, input parameters, and output features for the literature review.
| Field | Description | Required | Default |
|---|---|---|---|
| research_goal | Overall goal of the literature search review | Yes | |
| information_to_exclude | Information to be excluded from the analysis | No | "" |
| research_question | Specific question to be answered (use {} for variables) | Yes |
Define the input parameters of interest (e.g., drugs, proteins, compounds). You can have multiple inputs.
| Field | Description | Required | Default |
|---|---|---|---|
| name | Name of the input parameter | Yes | |
| description | Description of the input parameter | No | "" |
| max_length | Maximum length of the input value | Yes | |
| value | List of possible values for the parameter | Yes |
For each value:
| Field | Description | Required | Default |
|---|---|---|---|
| name | Main name of the value | Yes | |
| pubmed_alias | Aliases to be used in PubMed search (JSON list format) | No | "" |
| llm_aliases | Aliases to be used for LLM extraction (JSON list format) | No | "" |
Define the features you want to extract from the literature.
| Field | Description | Required | Default |
|---|---|---|---|
| name | Name of the output feature | Yes | |
| description | Description of the output feature | No | "" |
| data_type | Data type of the feature (e.g., str, int, dict) | Yes | |
| required | Whether the field is required in the LLM output | No | True |
| multiple_values | Whether the feature expects a list of values | No | False |
| max_length | Maximum length of the feature value (for strings) | Yes | |
| allowed_values | List of allowed values (if applicable) | No | None |
For allowed values:
| Field | Description | Required | Default |
|---|---|---|---|
| name | Name of the allowed value | Yes | |
| alias | Alias to be used by the LLM | No | "" |
| description | Description of the allowed value | No | "" |
Provide examples to guide the LLM's behavior. Each example should include:
- input_parameters
- sections (mock article content)
- output_features (expected output)
Contributions are welcome! Please feel free to submit a Pull Request.
The development of this software was supported by funding to the Blue Brain Project, a research center of the École polytechnique fédérale de Lausanne (EPFL), from the Swiss government’s ETH Board of the Swiss Federal Institutes of Technology.
Copyright (c) 2024 Blue Brain Project/EPFL