Post-Doctoral Researcher

ABOUT THE ROLE

As a Research Associate, you will take a leading role in the project Trustworthy AI for Peer Review: Developing and Validating Composite Research Quality Indicators. Your work will focus on constructing high-quality REF-aligned corpora of research outputs, using and extending the full-text cache and metadata pipelines developed through our software: Macroscope. You will collect, clean, link, and document text and bibliographic data so that it can support rigorous testing of AI-assisted research assessment.

You will be responsible for developing reproducible data workflows that link REF2021 submissions, Units of Assessment, outcome profiles, institutional metadata, bibliographic records, abstracts, and, where licensing permits, full-text content. This will involve working with sources such as OpenAlex, Crossref, Unpaywall, Scopus and REF data, designing robust extraction and matching methods, maintaining provenance and quality-control records, and producing well-structured datasets for downstream analysis.

The role will also contribute to the project’s LLM-based evaluation work, helping to design, run, and analyse experiments that test how large language models perform in REF-style assessment tasks. This will include supporting prompt and model comparison, repeated-run evaluation, assessment of stability and disagreement, and analysis of potential biases or confounding effects. The post provides an opportunity to work at the frontier of trustworthy AI, research evaluation, scientometrics, and large-scale text infrastructure, with scope to publish, present, and help shape policy-relevant demonstrators.

The successful candidate will also have the opportunity to support in the development of Semantic Space, an Academic Intelligence company that supports strategic decision making for research intensive organisations. 

The role is fixed term for 12 months. 

ABOUT THE PROJECT

Trustworthy AI for Peer Review: Developing and Validating Composite Research Quality Indicators is a UKRI/ESRC Metascience project investigating how artificial intelligence can support high-stakes research assessment without displacing expert judgement. Research assessment through peer review, journal editorial processes and the Research Excellence Framework (REF) is essential to the research system, but it is increasingly costly, time-consuming and difficult to scale. At the same time, large language models are creating new possibilities for analysing and evaluating research outputs.

The project addresses a central challenge: AI-derived research indicators are often presented as single scores, with limited visibility of uncertainty, disagreement, instability or bias. The project will develop and validate new approaches that make the reliability of AI-augmented assessment explicit and usable by decision-makers. It will combine benchmark datasets, LLM evaluation, scientometric indicators, uncertainty-aware modelling, Trust Cards, and user-centred demonstrators for REF panels, journal editors, publishers and research managers.

Key research areas include:

• Building FAIR, multi-resolution benchmark datasets linking REF submissions, journal peer-review data, books, bibliographic metadata, abstracts and, where appropriate, full-text corpora.
• Developing and testing LLM-based approaches to REF-style and peer-review assessment, including repeated-run evaluation, prompt comparison, model comparison, and calibration against known assessment outcomes.
• Creating indicators that communicate uncertainty, disagreement, stability, bias sensitivity and limits of valid use, rather than reducing complex judgements to opaque single scores.
• Developing transparent governance and communication tools, including Trust Cards, open-source workflows, documentation and demonstrators for responsible use of AI in research evaluation.

The project is delivered by an interdisciplinary consortium led by Loughborough University with Northumbria University, Durham University, Heriot-Watt University, the University of Southampton, the University of Aberdeen and the University of Wolverhampton, bringing together expertise in machine learning, scientometrics, accountable AI, computational reasoning, human-centred design, metascience and research evaluation practice.

ABOUT THE TEAM

The project team brings together researchers working across trustworthy AI, machine learning, scientometrics, human-centred design, computational reasoning, research policy and responsible innovation. The wider consortium includes specialists in uncertainty-aware AI, peer-review analysis, accountable AI, argumentation, stakeholder engagement and research evaluation.

At Northumbria University, the role will be based within Professor Martyn Dade-Robertson’s research environment, building on existing work on Macroscope: a software and data pipeline for mapping, analysing and interpreting large corpora of academic research. The Northumbria contribution focuses on the empirical data backbone of the project, including corpus construction, full-text and metadata processing, REF-aligned benchmarking, and interpretable demonstrators that help experts understand complex research landscapes (see www.semanticspace.ai).

ABOUT YOU

We are looking for a highly motivated postdoctoral Research Associate with strong skills in data handling, computational text analysis and reproducible research. You will have experience of programming for data-intensive research, ideally using Python and associated tools for APIs, web data extraction, text processing, databases, data cleaning, version control and workflow documentation. Experience with NLP, large language models, bibliometrics, scientometrics, research information systems, OpenAlex/Crossref/Unpaywall, REF data, or full-text extraction pipelines would be especially valuable. You do not need prior experience of REF assessment, but you should be interested in how data and AI can be used responsibly in high-stakes research evaluation.

Further information about the requirements of the role is available in the person specification.

If you would like an informal discussion about the role, please contact Prof Martyn Dade-Robertson ([email protected]).

Salary: £41,064 - £46,049