Skip to main content

Elicit's limitations

What Elicit does and does not do well

Here are a few limitations in Elicit to be aware of:

Limitations specific to Elicit

  • Elicit is built on language models, which are capable but imperfect. We've customized the models so their summaries and extractions reflect what a paper actually says, rather than what the model expects to be true in general (often called hallucination). We err on the side of Elicit saying nothing rather than saying something wrong, but it can still miss the nuance of a paper or misunderstand what a source refers to.

  • We write about Elicit's performance and limitations on our blog, for instance this post about Research Reports and this one about Systematic Reviews.

  • Other people have also shared thoughts on Elicit's limitations, including this thread.

Limitations that apply to research or search tools in general

  • Elicit is only as good as the sources underlying it. While we think researchers are a very careful and rigorous group on average, there is research with questionable methodology and even fraud. Elicit does not yet know how to evaluate whether one paper is more trustworthy than another, except by giving you some imperfect heuristics like citation count, journal, critiques from other researchers who cited the paper, and certain methodological details (sample size, study type, etc.). We're actively researching how best to help with quality evaluation but, today, Elicit summarizes the findings of a bad study just like it summarizes the findings of a good study.

  • In the same way that good research involves looking for evidence for and against various arguments, we recommend searching for papers presenting multiple sides of a position to avoid confirmation bias.

  • Elicit works better for some questions and domains than others. We eventually want to help with all domains and types of research but, to date, we've focused on empirical research (e.g. randomized controlled trials in social sciences or biomedicine) so that we can apply lessons from the systematic review discipline.

This is not a comprehensive list of possible limitations.

Did this answer your question?