LLM Science Battle

• OpenAI’s GPT-4o: The latest flagship, renowned for its multimodal prowess and sophisticated reasoning.
• Google’s Gemini 2 Flash: A nimble powerhouse balancing speed and intelligence, built for real-world applications.
• Meta’s Llama 3.2 (3B): A smaller but capable iteration of Meta’s hugely popular open-weight model family, pushing the boundaries of open-source AI.

​

The Gauntlet: Turning Scientific Jargon into Actionable Knowledge Graphs

• Spotting the Actors (Entity Extraction - NER): This is about identifying these key terms and correctly labelling them – Is this a Method, a Task, or a Dataset?
• Mapping the Connections (Relation Extraction - RE): This is the trickier part. How do these actors relate? Is “BERT” Used-For “Named Entity Recognition”? Is “Method A” being Compared to “Method B”? Is “Method C” Evaluated-On “ImageNet”?

​

The Proving Ground: Why SciER?

• Gold Standard: It’s meticulously annotated by humans, providing a reliable “ground truth” to measure against.
• Real-World Complexity: It uses actual scientific abstracts, forcing the models to grapple with authentic complexity and jargon.
• Community Trust: It’s an established benchmark used by researchers to evaluate NER and RE systems.

​

Our Playbook: Prompting, Extracting, and Scoring with Morphik

1. Data Wrangling Made Easy: We took the SciER records, cleverly grouped sentences by document (abstract), and ingested them smoothly into Morphik. Crucially, we stored the human-annotated “ground truth” entities and relations alongside the text as metadata – essential for evaluation later. (Illustrative code snippet below shows the logic)

# Example: Preparing SciER data for Morphik ingestion
def prepare_text_for_ingestion(records: List[Dict]) -> List[Dict]:
    documents = []
    doc_groups = defaultdict(list)
    for record in records: doc_groups[record["doc_id"]].append(record) # Group by document

    for doc_id, recs in doc_groups.items():
        text = "\n".join([r["sentence"] for r in recs]) # Combine sentences
        gt_entities = [ner for r in recs for ner in r["ner"]] # Aggregate ground truth
        gt_relations = [rel for r in recs for rel in r["rel"]]
        documents.append({
            "text": text,
            "metadata": { "doc_id": doc_id, "ground_truth_entities": gt_entities, "ground_truth_relations": gt_relations }
        })
    return documents
# The evaluation code is available in the main Morphik repository: https://github.com/morphiklabs/morphik/tree/main/evaluations

2. Consistent Prompting is Key: We engineered a detailed prompt, guiding the LLMs on exactly what entities (Dataset, Method, Task) and relationships (Used-For, Compare, Evaluate-On, etc.) to extract, and crucially, how to format the output as JSON. Morphik allowed us to define this prompt once, complete with few-shot examples (EntityExtractionExample), and apply it identically to GPT-4o, Gemini 2 Flash, and Llama 3.2 (GraphPromptOverrides). This consistency is vital for a fair comparison. (Illustrative prompt setup below)

# Example: Setting up the prompt structure in Morphik
def create_graph_extraction_override(entity_types: List[str]) -> EntityExtractionPromptOverride:
    # ... (Define examples like EntityExtractionExample(label="BERT", type="Method"))
    prompt_template = """
    Your task is to read scientific text and extract Entities (Datasets, Methods, Tasks) and their Relationships (Used-For, Compare, etc.).
    **Entity Examples (brief list):** {examples}
    **Relationship Types:** Used-For, Feature-Of, Hyponym-Of, Part-Of, Compare, Evaluate-For, Conjunction, Evaluate-On, Synonym-Of.
    **Instructions:** Extract entities, then relationships between ONLY those entities. Stick to the text. Output a single JSON: {{"entities": [...], "relationships": [...]}}.
    **Text to analyze:** {content}
    """
    # ... (Return EntityExtractionPromptOverride)
# The evaluation code is available in the main Morphik repository: https://github.com/morphiklabs/morphik/tree/main/evaluations

3. Orchestrated Extraction: For every abstract and each LLM, Morphik handled the behind-the-scenes dance: sending the text chunk (we used a 600-token chunk size with 300-token overlap – parameters ripe for future tuning!), applying the consistent prompt, calling the correct model API, retrieving the response, and parsing the LLM’s JSON output into a structured graph format using db.create_graph. This streamlined the generation of comparable knowledge graph snippets from each model.

# Example: Morphik orchestrating graph creation for a specific model
graph = db.create_graph(
    name=f"scier_{model_name}_{run_id}",
    documents=doc_ids, # List of ingested document IDs
    prompt_overrides=graph_overrides # The consistent prompt defined earlier
)
# The evaluation code is available in the main Morphik repository: https://github.com/morphiklabs/morphik/tree/main/evaluations

4. Smarter Evaluation (Semantic Similarity): Exact text matching is too brittle for this task. Does “Convolutional Neural Network” not match “CNN”? To get a more realistic score, we implemented semantic similarity using OpenAI’s fast text-embedding-3-small model. We compared the meaning of extracted entities and relations against the ground truth, considering a match if the cosine similarity score was 0.80 or higher. This acknowledges variations in wording while focusing on conceptual accuracy. Morphik’s structured output made it straightforward to feed data into our evaluation script.

# Example: Using semantic similarity for evaluation
similarity_calculator = OpenAIEmbedding(threshold=0.80)
entity_metrics = evaluate_entity_extraction(
    all_gt_entities, extracted_entities, similarity_calculator, ...
)
relation_metrics = evaluate_relation_extraction(
    all_gt_relations, extracted_relationships, entity_match_map, similarity_calculator, ...
)
# The evaluation code is available in the main Morphik repository: https://github.com/morphiklabs/morphik/tree/main/evaluations

5. Visualizing the Verdict: We calculated standard Precision, Recall, and F1-Score for both entity and relation extraction. Morphik’s structured data allowed us to easily generate comparison charts.

​

The Results: Ding, Ding, Ding!

GPT-4o F1 Scores: Strong Entity Extraction (0.870 F1), but Relation Extraction proves much tougher (0.278 F1), highlighting the challenge of connecting entities.

Gemini 2 Flash F1 Scores: Impressive Entity Extraction (0.841 F1) and surprisingly leads the pack in Relation Extraction (0.362 F1).

Llama 3.2 (3B) F1 Scores: Solid Entity Extraction (0.807 F1), but struggles more with Relation Extraction (0.211 F1), typical for this harder task.

GPT-4o Counts: Extracted more entities (265 vs 213 GT), aligning with high recall but hinting at some over-extraction. Extracted fewer relations (90 vs 148 GT).

Gemini 2 Flash Counts: Also extracted more entities (265 vs 213 GT). Notably extracted the most relations (105 vs 148 GT), aligning with its higher RE F1 score.

Llama 3.2 (3B) Counts: Extracted entities closer to the ground truth count (216 vs 213 GT). Extracted the fewest relations (61 vs 148 GT).

​

Unpacking the Results: Insights and Caveats

• Entities Easier Than Relations (Big Surprise!): All models found identifying entities (NER) significantly easier than figuring out their relationships (RE). F1 scores dropped drastically (around 50-70%) for RE across the board. This isn’t shocking – understanding the nuanced connections between terms is inherently harder than just spotting the terms themselves.
• GPT-4o: The Recall Champion (NER): GPT-4o excelled at finding most of the entities, boasting the highest recall (95.8%) and the top NER F1 score (0.870). However, its higher extracted entity count suggests it might sometimes be slightly over-eager.
• Gemini 2 Flash: The Relation Whisperer? The surprise standout was Gemini 2 Flash in Relation Extraction. It achieved the best RE F1 score (0.362) and the highest RE precision (0.398), suggesting it was more accurate (though still far from perfect) in identifying valid relationships compared to the others in this specific setup.
• Llama 3.2 (3B): Solid Open-Source Contender: Llama 3.2 (3B) delivered respectable NER performance, showing a good balance between precision and recall (and extracting a number of entities very close to the ground truth). Its RE performance lagged, but it remains a strong open-weight option.
• The Claude Conundrum: We initially wanted to include Anthropic’s Claude in this comparison. However, we encountered difficulties getting Claude to consistently adhere to the strict JSON output format required for our automated evaluation pipeline, especially for the complex relationship structures. This wasn’t necessarily a reflection on Claude’s reasoning ability, but it highlighted a practical challenge: for automated KG building workflows like those Morphik enables, reliable, structured output from the LLM is non-negotiable.
• Room for Improvement (Prompts, Chunking): These results reflect our specific experimental setup. We used one detailed prompt and a specific text chunking strategy (600 tokens, 300 overlap). More sophisticated prompt engineering (e.g., chain-of-thought, different examples) or experimenting with chunk sizes could potentially boost performance for all models. This is an iterative game!

​

Conclusion: LLMs Need Orchestration to Truly Shine in Science

• NER is Getting Good: State-of-the-art LLMs are becoming adept at identifying key entities in scientific text.
• RE is the Next Frontier: Accurately extracting the complex web of relationships remains a major challenge, though models like Gemini 2 Flash show encouraging progress.
• Model Choice Matters: Different LLMs exhibit different strengths. GPT-4o excels at broad identification, while Gemini 2 Flash showed an edge here in relational understanding. Llama 3.2 (3B) offers a strong open alternative.

• Ingest and Prepare diverse data sources.
• Manage and Apply complex, consistent prompts across different models.
• Orchestrate API calls and handle model-specific quirks.
• Structure the often unpredictable LLM outputs into usable formats (like graphs!).
• Integrate smoothly with sophisticated evaluation techniques (like semantic similarity).