{"resourceId":"jena-intel-llms-blind-to-atom-coordinates-nature-comp-sci-20261012","versions":[{"version":"external-8d34d21af58110dde2c4531707c8c24a774989a5700dc17bdc696229b17c3452","resource":{"id":"jena-intel-llms-blind-to-atom-coordinates-nature-comp-sci-20261012","title":"Nature Computational Science (Jena/Intel Labs): LLMs fine-tuned for materials property prediction use atom types but not atom positions, and scaling to 70B parameters or 2 million structures does not fix it","organization":"Friedrich Schiller University Jena (Jablonka lab) with Intel Labs; published in Nature Computational Science","sector":"AI for materials science and scientific machine learning (academic research)","geography":"Germany (Jena) and United States (Intel Labs); open benchmark","publishedAt":"Nature Computational Science, open access, published 12 October 2026; accompanying Research Briefing (s43588-026-01044-3) published 13 October 2026","publicationDate":"2026-10-12","eventDate":null,"sourceName":"Nature Computational Science","sourceLabel":"Peer-reviewed academic study","sourceUrl":"https://www.nature.com/articles/s43588-026-01045-2","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["knowledge-work","infrastructure","operating-model"],"finding":"Using synthetic datasets with tunable contributions from coordinates and categories, plus the MatText benchmarking framework, the authors report that LLMs consistently fail to learn from coordinate information in coordinate–category data while learning category patterns well. The paper says this persists regardless of model size (up to 70 billion parameters), dataset scale (up to 2 million structures) or text representation. An analysis of the MatBench leaderboard across six property tasks shows graph neural networks consistently outperforming language-model approaches (the authors' 'GNN-LM wall'). The authors recommend using LLMs for composition-dominated tasks and geometric architectures for structure-dominated ones. The companion Research Briefing (13 Oct) summarises the result as LLMs learning which atoms are present, not where they are.","sledRelevance":"University materials, chemistry and AI-for-science groups, and the research computing teams that fund GPU time for them, are being asked to back general-purpose LLMs for scientific prediction. This peer-reviewed result gives a specific reason to match model architecture to the problem and to test LLM approaches against geometric baselines before committing compute.","evidence":"Opened the open-access Nature Computational Science article via HTTPS (curl; full HTML text extracted and read, including abstract, GNN-LM wall analysis, Discussion, Limitations, data/code availability and affiliations). Also opened the 13 Oct Research Briefing page (s43588-026-01044-3); its body is subscription-only, so only its visible summary was used.","architectureImplications":"Interpretation: pipelines for structure-dependent prediction should keep geometric models (for example GNNs) as the primary or baseline model and use LLMs where composition or text dominates.","governanceImplications":"Research computing allocation reviews could ask applicants to report a geometric baseline before granting large LLM fine-tuning budgets for property prediction.","securityPrivacyImplications":"No personal data; public benchmark datasets and MIT-licensed code.","caveats":"Three properties (shear modulus, bulk modulus, perovskite formation energy) from a limited set of benchmarks; only BERT-style and Llama-2 architectures evaluated; scaling study used parameter-efficient fine-tuning up to 70B, so full fine-tuning or pretraining on coordinate-rich corpora is untested; multimodal models with explicit coordinate embeddings were not evaluated; leaderboard comparison is descriptive. One author is from Intel Labs (now Lila Sciences).","streamIds":["research"],"roles":{"sales":"Sales framing: this is peer-reviewed evidence that bigger general-purpose LLMs are not the answer for structure-dependent scientific prediction. For research computing and AI-for-science leaders, the credible offer is an architecture-selection and benchmarking review before committing large fine-tuning budgets. The evidence covers three materials properties and two model families, so it cannot support blanket claims that LLMs fail at all scientific prediction.","engineering":"Engineering framing: the fit is any project predicting properties from atomic or other coordinate data. The study shows text-encoded coordinates are largely ignored by fine-tuned LLMs, while GNNs perform better on MatBench. Prerequisites are a geometric baseline and a controlled test that varies coordinate signal. Constraints include limited properties and untested multimodal designs; a proof of value should rerun MatText on the user's task.","delivery":"Delivery framing: delivery means adding an evaluation gate to AI-for-science projects. Work includes setting baselines, running coordinate-sensitivity tests, and documenting model choice. Ownership sits with the PI and a research software or ML engineer, with research computing reviewing allocation. Proposed acceptance measures include a reported geometric baseline per project; risks include overgeneralising beyond materials tasks and dismissing multimodal approaches the study did not test."},"roleDetails":{"sales":{"customerProblem":"Research groups and computing centres are spending GPU budget on LLM approaches to scientific prediction without evidence they suit the task.","stakeholders":["Research computing director","Materials and chemistry PIs","AI-for-science institute lead","Research software engineering lead"],"discoveryQuestions":["Which prediction tasks depend on 3D structure or coordinates?","Do projects report geometric baselines before LLM fine-tuning?","How is GPU time allocated to AI-for-science projects?"],"valueHypothesis":"An architecture-selection review may avoid spending fine-tuning budgets on models that cannot use structural information; savings are not quantified in the study.","potentialEngagement":"Three-week benchmarking review of one or two AI-for-science projects using the open MatText code.","unsupportedClaims":["LLMs fail on all scientific prediction tasks","Multimodal or coordinate-aware models share the limitation","Quantified compute savings"]},"engineering":{"fit":"Property prediction from crystal structures, molecular geometries or other coordinate–category data.","architectureIntegration":"Use geometric neural networks as baseline or primary models for structure-dominated tasks and LLMs for composition- or text-dominated tasks, evaluated in the same benchmark harness (MatText, per the paper).","prerequisites":["Labelled structure–property dataset","Geometric baseline model","GPU time for fine-tuning comparisons"],"constraints":["Validated on three materials properties","Only BERT and Llama-2 families tested","Parameter-efficient fine-tuning only","Multimodal coordinate-embedding models untested"],"security":"Public datasets and MIT-licensed code; confidentiality concerns arise only with proprietary structures.","proofOfValue":["Rerun the coordinate-sensitivity test on an in-house task","Compare an LLM against a GNN baseline on held-out data","Test one coordinate-aware or multimodal model the study did not cover"]},"delivery":{"work":"Add a model-selection and baseline evaluation step to AI-for-science project plans and allocation requests.","dependencies":["Benchmark datasets","Research software engineering support","GPU allocation"],"ownership":"Principal investigator with a research software or ML engineer; research computing reviews allocations.","skillsAdoption":"Train researchers on inductive bias, baselines and coordinate-sensitivity testing.","governanceCheckpoints":["Baseline reported before large fine-tuning allocation","Model-choice rationale documented","Results reviewed against held-out data"],"acceptanceCriteria":["Proposed: every structure-dependent project reports a geometric baseline","Proposed: coordinate-sensitivity test documented","Proposed: model-choice rationale recorded in the allocation request"],"risks":["Overgeneralising beyond materials tasks","Ignoring untested multimodal designs","Added evaluation time delaying projects"]}},"retrievedAt":"2026-10-13T21:58:21-05:00","enrichedAt":"2026-10-13T22:03:45-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Researchers need enough ML literacy to understand inductive bias and choose architectures, not only prompt or fine-tune LLMs.","procurementImplications":"Supports budgeting for specialised model development and GPU time for GNNs, rather than assuming larger LLMs will close accuracy gaps.","operatingModelImplications":"Argues for an architecture-selection step in AI-for-science project design.","sourceVerification":{"openedUrl":"https://www.nature.com/articles/s43588-026-01045-2","referenceExcerpt":"we find that LLMs consistently fail to capture coordinate information while excelling at category patterns","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}