Extracting and utilizing multimodal datasets of images and text with large language models скачать в хорошем качестве

Extracting and utilizing multimodal datasets of images and text with large language models

Abstract: With the recent exponential growth in publication rates, it has become impossible for a scientist to keep up with all publications related to a specific topic. Although there are notable efforts to automate text parsing from literature, there are many instances where important information is communicated through images or tables in papers.1 In this talk, I will present the latest developments in two software tools developed at the Center of Nanoscale Materials (CNM): i) EXSCLAIM! for data mining from scientific literature2, and ii) Plot2Spectra for image segmentation related to spectral images, with the aim of creating metadata.3 EXSCLAIM! has been enhanced with Large Language Models (LLMs), i.e., ChatGPT and appropriate prompt engineering to extract image-text pairs from scientific journals, which can be foundational for creating multimodal models and advancing semantic searches. In this presentation, I will demonstrate various applications of the extracted multimodal datasets in building knowledge graphs, conducting semantic searches, and performing topic modelling. Additionally, I will illustrate how to utilize the image segmentation workflow in Plot2Spectra to extract additional metadata and create datasets suitable for machine learning (ML) and high-throughput experimentation. Presenter: Katerina Aikaterini Vriza , Argonne National Laboratory Repo: github.com/katerinavr/data_mining_workshop