add data extraction tutorial

tutorial/data_extraction_tutorial_2.ipynb

@@ -0,0 +1,210 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "source": [
+    "# 🔍 Using LLMs for Scientific Information Extraction\n",
+    "\n",
+    "In this notebook, we focus on making calls to a Large Language Model (LLM) using [LiteLLM](https://github.com/BerriAI/litellm), a lightweight abstraction layer for various LLM providers (e.g. OpenAI, Anthropic, Azure).\n",
+    "\n",
+    "We define a simple function that takes a **system prompt** and a **user prompt** as parameters and returns the model's response.\n",
+    "\n",
+    "This allows us to flexibly test different prompt formulations and system instructions – a key step in the design of robust extraction workflows.\n"
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "b87aec46155e987c"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "from litellm import completion\n",
+    "\n",
+    "def call_llm(system_prompt, user_prompt, model=\"gpt-4\"):\n",
+    "    \"\"\"\n",
+    "    Sends a prompt to an LLM using LiteLLM and returns the response.\n",
+    "\n",
+    "    Parameters:\n",
+    "        system_prompt (str): The system message (sets model behavior).\n",
+    "        user_prompt (str): The actual user query or input.\n",
+    "        model (str): The model name to use (default: \"gpt-4\").\n",
+    "\n",
+    "    Returns:\n",
+    "        str: The text response from the model.\n",
+    "    \"\"\"\n",
+    "    response = completion(\n",
+    "        model=model,\n",
+    "        messages=[\n",
+    "            {\"role\": \"system\", \"content\": ___}, # TODO: add the system prompt\n",
+    "            {\"role\": \"user\", \"content\": ___} # TODO: add the user prompt\n",
+    "        ]\n",
+    "    )\n",
+    "    return response['choices'][0]['message']['content']"
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "2450aeed97426e78"
+  },
+  {
+   "cell_type": "markdown",
+   "source": [
+    "## 📄 Running the LLM on a Text Document\n",
+    "\n",
+    "Now that we have our cleaned and chunked text files, we will try a first simple LLM call.\n",
+    "\n",
+    "We load one text file and prompt the model to extract all polymer names mentioned in it.\n",
+    "\n",
+    "This serves as a first test of the LLM’s capabilities and helps us evaluate the basic prompt structure."
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "95684e8036b569e5"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "\n",
+    "# Load the .txt file\n",
+    "file_path = \"example_paper.txt\"\n",
+    "with open(file_path, \"r\", encoding=\"utf-8\") as f:\n",
+    "    document_text = f.read()\n",
+    "    \n",
+    "print(document_text[:500])  # Preview first 500 characters"
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "328d0c75c3469a28"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "# Define a basic system and user prompt\n",
+    "system_prompt = \"You are an expert in polymer science. Extract relevant scientific information.\"\n",
+    "user_prompt = f\"Extract all polymer names from the following text:\\n\\n{document_text}\"\n",
+    "\n",
+    "# Call the model\n",
+    "response = call_llm(___, ___) # TODO: add the prompts to call the call_llm function\n",
+    "\n",
+    "print(response)"
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "1a3159992ee1dcea"
+  },
+  {
+   "cell_type": "markdown",
+   "source": [
+    "Since the user and system prompts can affect the performance of the data extraction significantly, you can now try different combinations of system and user prompts to improve results. "
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "6ae855444903ef44"
+  },
+  {
+   "cell_type": "markdown",
+   "source": [
+    "## ✳️ One-Shot Prompting\n",
+    "\n",
+    "Instead of only giving the model an instruction, we can include **a single example** in the prompt to help guide its output.\n",
+    "\n",
+    "This technique is called **one-shot prompting** and can improve the quality and consistency of the model's responses.\n",
+    "\n",
+    "Below, we provide the model with a sample input-output pair and then ask it to perform the same extraction on a new text.\n"
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "d9509b18dfba643d"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "# User prompt that includes one-shot example inline\n",
+    "user_prompt = f\"\"\"\n",
+    "Here is an example of what I want:\n",
+    "\n",
+    "Input:  # TODO: add a short example text including min one polymer name. \n",
+    "\n",
+    "\n",
+    "Output: # TODO: add the desired output to your example text \n",
+    "\n",
+    "\n",
+    "---\n",
+    "\n",
+    "Now do the same for this input:\n",
+    "\n",
+    "Input:\n",
+    "{document_text}\n",
+    "\n",
+    "Output:\n",
+    "\"\"\"\n",
+    "\n",
+    "system_prompt = \"You are a chemistry assistant. Extract all polymer names from the given text.\"\n",
+    "\n",
+    "# Run the LLM\n",
+    "response = call_llm(system_prompt, user_prompt)\n",
+    "\n",
+    "print(response)"
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "91e538e0f36f1410"
+  },
+  {
+   "cell_type": "markdown",
+   "source": [
+    "## 🔭 Beyond Basic LLM Calls\n",
+    "\n",
+    "This notebook demonstrated the basic workflow of using a Large Language Model to extract structured information from scientific text.\n",
+    "\n",
+    "However, LLMs can be integrated into **more advanced workflows** that go far beyond single-prompt extraction:\n",
+    "\n",
+    "- 🤖 **Agent systems** that combine multiple reasoning steps, tools, and memory to guide extraction over many documents.\n",
+    "- 🧪 **Vision–Language Models (VLMs)** that can process both text and images (e.g. extract data from figures, tables, or chemical diagrams).\n",
+    "\n",
+    "You can explore more examples, notebooks, and real-world use cases in the **[Matextract project](https://matextract.pub)**.\n"
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "78780d48c6f65ef"
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 2
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython2",
+   "version": "2.7.6"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}