openFDA is designed primarily for real-time queries. Using combinations of the
parameters you can page through a result set that has up to
This limit is in place to protect openFDA infrastructure and is sufficient in most cases;
however, sometimes it is desirable to navigate through a result set that exceeds
26,000 search matches.
If you are unable to narrow your search criteria to decrease the number of hits, consider the following strategies
to obtain data from large result sets:
Use the “Search After” feature that permits scrolling through a result set of unlimited size, up to the size of the dataset itself. The following are the basic steps you need to follow:
Execute your initial query that produces a large number of matches. Make sure not to include the
skip parameter, because
search_after do not work together (technical explanation is here).
The initial query will return your first page of data. For example:
Link HTTP header contained in the response. Detailed information about the purpose and structure
Link header can be found here.
In short, this header will contain a
rel="Next" URL representing the query you need to use to obtain the next page of data. Missing
header indicates you are already on the last page. An example of extracting the header value using
bash-3.2$ curl -sIg 'https://api.fda.gov/drug/event.json?search=patient.drug.openfda.product_type.exact:%22HUMAN%20PRESCRIPTION%20DRUG%22&limit=100&sort=receivedate:asc' | grep "Link: " Link: <https://api.fda.gov/drug/event.json?search=patient.drug.openfda.product_type.exact%3A%22HUMAN%20PRESCRIPTION%20DRUG%22&limit=100&sort=receivedate%3Aasc&skip=0&search_after=0%3D1068076800000%3B1%3Dsafetyreport%25234022687>; rel="next"
Use the extracted URL to obtain next page of data. Note it includes the
search_after query parameter.
Repeat the cycle until the
Link header is no longer present in the response, which indicates you are on the last page.