## Paginate search results

URL: https://docs.atlan.com/product/capabilities/build-apps/sdks/python/search/references/paging

> Page through search results programmatically using the Atlan Python SDK (pyatlan) with automatic pagination and page size control.

# Paginate search results

## Automatically (via SDK)

Our SDKs are designed to simplify paging, so you don't need to worry about the underlying details. You can simply iterate through a search response and the SDK will automatically fetch the next pages when it needs to (lazily).

The SDKs will even add a default sort by GUID to make sure stable results across pages, even when you don't provide any sorting criteria yourself.

### Java

```java showLineNumbers title="Automatic paging"
client.assets.select() // (1)
 .pageSize(50) // (2)
 .stream() // (3)
 .limit(100) // (4)
 .filter(a -> !(a instanceof ILineageProcess)) // (5)
 .forEach(a -> log.info("Do something with each result: {}", a)); // (6)
```

1. You can start building a query across all assets using the `select()` method on the `assets` member of any client. You can chain as many mandatory (`where()`) conditions, mandatory exclusion (`whereNot()`) conditions, and set of conditions some of which must match (`whereSome()`) as you want.
2. The number of results to include (per page).
3. You can stream the results direct from the response. This will also lazily load and loop through each page of results.

 :::tip[Can be chained without creating a request in-between]
You can actually chain the `stream()` method directly onto the end of your query and request construction, without creating a `request` or `response` object in-between.
 :::
4. With streaming, you can apply your own limits to the maximum number of results you want to process.

 :::tip[Independent of page size]
Note that this is independent of page size. You could page through results 50 at a time, but only process a maximum of 100 total results this way. Since the results are lazily-loaded when streaming, only the first two pages of results would be retrieved in such a scenario.
 :::
5. You can also apply your own logical filters to the results.

 :::tip[Push-down as much as you can to the query]
You should of course push-down as many of the filters as you can to the query itself, but if you have a particular complex check to apply that can't be encoded in the query this can be a useful secondary filter over the results.
 :::
6. The `forEach()` on the resulting stream will then apply whatever actions you want with the results that come through.

### Python

```python showLineNumbers title="Build the query"
from pyatlan.client.atlan import AtlanClient
from pyatlan.model.assets import Asset, Process
from pyatlan.model.fluent_search import CompoundQuery, FluentSearch

builder = (
 FluentSearch() # (1)
 .where(CompoundQuery.active_assets()) # (2)
).to_request() # (3)
results = client.asset.search(index) # (4)
for asset in results: # (5)
 if not isinstance(asset, Process): # (6)
 non_process = asset
```

1. You can start building a query using a `FluentSearch` object. You can have as many mandatory (`where()`) conditions, mandatory exclusion (`where_not()`) conditions, and set of conditions some of which must match (`where_some()`) as you want.
2. This helper provides a query that ensures results are active (not archived) assets.
3. You can now build all of this search configuration into a request.
4. You can then run the search against this request.
5. This will iterate through all the results without the need to be concerned with pages.

 :::note[Iterating over results produces a [Generator](https://wiki.python.org/moin/Generators)]
This means that results are retrieved from the backend a page at time. This also means that you can only iterate over the results once.
 :::
6. Remember that each result is a generic `Asset`. You should of course push-down as many of the filters as you can to the query itself, but if you have a particular complex check to apply that can't be encoded in the query this can be a useful secondary filter over the results.

### Kotlin

```kotlin showLineNumbers title="Automatic paging"
client.assets.select() // (1)
 .pageSize(50) // (2)
 .stream() // (3)
 .limit(100) // (4)
 .filter { it !is ILineageProcess } // (5)
 .forEach { log.info { "Do something with each result: $it" } } // (6)
```

1. You can start building a query across all assets using the `select()` method on the `assets` member of any client. You can chain as many mandatory (`where()`) conditions, mandatory exclusion (`whereNot()`) conditions, and set of conditions some of which must match (`whereSome()`) as you want.
2. The number of results to include (per page).
3. You can stream the results direct from the response. This will also lazily load and loop through each page of results.

 :::tip[Can be chained without creating a request in-between]
You can actually chain the `stream()` method directly onto the end of your query and request construction, without creating a `request` or `response` object in-between.
 :::
4. With streaming, you can apply your own limits to the maximum number of results you want to process.

 :::tip[Independent of page size]
Note that this is independent of page size. You could page through results 50 at a time, but only process a maximum of 100 total results this way. Since the results are lazily-loaded when streaming, only the first two pages of results would be retrieved in such a scenario.
 :::
5. You can also apply your own logical filters to the results.

 :::tip[Push-down as much as you can to the query]
You should of course push-down as many of the filters as you can to the query itself, but if you have a particular complex check to apply that can't be encoded in the query this can be a useful secondary filter over the results.
 :::
6. The `forEach()` on the resulting stream will then apply whatever actions you want with the results that come through.

### Raw REST API

:::tip[Use an SDK]
The SDKs manage making multiple requests and parsing results to make subsequent requests in the most efficient way possible. You will need to make many different API requests if you want to do the same directly via the raw REST APIs.
:::

## Manually (via Elastic)

For curious minds, though, you can page through search results using a combination of the following properties[^1]:

| Property | Description | Example |
|---|---|---|
| `from` | Indicates the starting point for the results. | `0` |
| `size` | Indicates how many results to include per response (page). As a general rule of thumb we would recommend a size from `20`-`100`, making `50` a common starting point. | `50` |
| `track_total_hits` | Includes an accurate number of total results, if set to `true`. With its default value on the raw REST APIs (`false`) the maximum number of results you will see in the `approximateCount` field in the response is 10000. (Again, the SDKs set this to `true` by default to avoid this confusion.) | `true` |

:::warning[Constraints with this approach]
To have the most consistent results you can when paging, you must always use some sorting criteria and include at least one sorting criteria as a tie-breaker. (You must also keep that criteria the same for every page.)

Furthermore, as you get to larger `from` sizes (more than ~10,000) Elastic will begin to use significantly more resources to process your paging. To reduce this impact, if you need to page through many results you should implement your own timestamp-based offset mechanism so that the `from` size is kept consistently low.

(Again, the SDKs do both of these for you automatically.)
:::
For example:

### Java

```java showLineNumbers title="Annotated sort options, as you would define them in the Java SDK"
SortOptions byUpdate = Asset.UPDATE_TIME.order(SortOrder.Desc); // (1)
SortOptions byGuid = Asset.GUID.order(SortOrder.Asc); // (2)
```

1. Include any of your own sorting, like this example putting the most recently-updated assets first in the results.
2. Also consider a tie-breaker sorting mechanism. In this case, we use an asset's GUID to further sort any results that have the same last modified timestamp, since GUID is guaranteed to be unique for every asset.

```java title="Build the request"
IndexSearchRequest index = IndexSearchRequest.builder(
 IndexSearchDSL.builder(someQuery) // (1)
 .from(100) // (2)
 .size(50) // (3)
 .trackTotalHits(true) // (4)
 .sortOption(byUpdate) // (5)
 .sortOption(byGuid)
 .build())
 .build();
```

1. You still need a query, to get some results 😉.
2. Starting point for the page of results being requested. In this example, you would be asking for the third page. (`0` would be from `0-50` for the first page, `50` would be from `50-100` for the second page, and this gives us `100-150` for the third page.)
3. The number of results per page (in this example, `50` results per page).
4. Enable `trackTotalHits` so that your response includes an accurate total number of results. (Actually the Java SDK enables this by default, so this step is redundant unless you want to turn it *off*.)
5. And we need to include the sorting criteria we defined just above.

```java title="Iterate through multiple pages of results"
IndexSearchResponse response = index.search(client); // (1)
long totalResults = response.getApproximateCount(); // (2)
for (Asset result : response)
response.forEach(a -> log.info("Found asset: {}", a.getGuid())); // (4)
response.stream() // (5)
 .filter(a -> !(a instanceof ILineageProcess)) // (6)
 .limit(100) // (7)
 .forEach(a -> log.info("Found asset: {}", a.getGuid())) // (8)
```

1. Keep the response object from the initial search, as it has a helper method for paging.
2. Since we set `trackTotalHits` to `true` (the default for the Java SDK even if we don't set it), the `.getApproximateCount()` will give us the total number of results. This can be over 10,000.
3. Iterate through all the results, across all pages (each page is lazily-loaded, so you can break out at any time without actually retrieving all pages of results).
4. Alternatively, you can iterate through all the results using `forEach()` on the response. (This uses the same underlying iterable-based implementation.)
5. Alternatively, you can stream the results. Streaming will also lazily-load only the pages of results necessary to meet the chained criteria for processing the stream.
6. When streaming, you can further filter the results to apply any complex filtering logic you couldn't push-down as part of the query itself.
7. When streaming, you can also limit the total number of results you want to process—independently of the page size.
8. Don't forget to actually _do_ something with the results in the stream 😉

### Python

```python showLineNumbers title="Annotated sort options, as you would define them in the Python SDK"
from pyatlan.client.atlan import AtlanClient
from pyatlan.model.enums import SortOrder
from pyatlan.model.assets import Referenceable
from pyatlan.model.search import IndexSearchRequest, DSL

by_update = Referenceable.UPDATE_TIME.order(SortOrder.DESCENDING) # (1)
by_guid = Referenceable.GUID.order(SortOrder.ASCENDING) # (2)
```

1. Include any of your own sorting, like this example putting the most recently-updated assets first in the results.
2. Also consider a tie-breaker sorting mechanism. In this case, we use an asset's GUID to further sort any results that have the same last modified timestamp, since GUID is guaranteed to be unique for every asset.

```python title="Build the request"
index = IndexSearchRequest(
 dsl=DSL(
 query=someQuery, # (1)
 from_=100, # (2)
 size=50, # (3)
 track_total_hits=True, # (4)
 sort=[ # (5)
 by_update,
 by_guid
 ],
 )
)
```

1. You still need a query, to get some results 😉.
2. Starting point for the page of results being requested. In this example, you would be asking for the third page. (`0` would be from `0-50` for the first page, `50` would be from `50-100` for the second page, and this gives us `100-150` for the third page.)
3. The number of results per page (in this example, `50` results per page).
4. Enable `track_total_hits` so that your response includes an accurate total number of results. (Actually the Python SDK enables this by default, so this step is redundant unless you want to turn it *off*.)
5. And we need to include the sorting criteria we defined just above.

```python title="Iterate through multiple pages of results"
client = AtlanClient()
response = client.asset.search(index) # (1)
total_results = response.count # (2)
for result in response: # (3)
 # Do something with each result of the search...
```

1. Keep the response object from the initial search, as it has a helper method for paging.
2. Since we set `track_total_hits` to `True` (the default for the Python SDK even if we don't set it), the `.count` property will give us the total number of results. This can be over 10,000.
3. Iterate through all the results, across all pages (each page is lazily-loaded, so you can break out at any time without actually retrieving all pages of results). Don't forget to actually _do_ something with the results in the stream 😉

### Kotlin

```kotlin showLineNumbers title="Annotated sort options, as you would define them in the Java SDK"
val byUpdate = Asset.UPDATE_TIME.order(SortOrder.Desc) // (1)
val byGuid = Asset.GUID.order(SortOrder.Asc) // (2)
```

1. Include any of your own sorting, like this example putting the most recently-updated assets first in the results.
2. Also consider a tie-breaker sorting mechanism. In this case, we use an asset's GUID to further sort any results that have the same last modified timestamp, since GUID is guaranteed to be unique for every asset.

```kotlin title="Build the request"
val index = IndexSearchRequest.builder(
 IndexSearchDSL.builder(someQuery) // (1)
 .from(100) // (2)
 .size(50) // (3)
 .trackTotalHits(true) // (4)
 .sortOption(byUpdate) // (5)
 .sortOption(byGuid)
 .build())
 .build()
```

1. You still need a query, to get some results 😉.
2. Starting point for the page of results being requested. In this example, you would be asking for the third page. (`0` would be from `0-50` for the first page, `50` would be from `50-100` for the second page, and this gives us `100-150` for the third page.)
3. The number of results per page (in this example, `50` results per page).
4. Enable `trackTotalHits` so that your response includes an accurate total number of results. (Actually the Java SDK enables this by default, so this step is redundant unless you want to turn it *off*.)
5. And we need to include the sorting criteria we defined just above.

```kotlin title="Iterate through multiple pages of results"
val response = index.search(client) // (1)
val totalResults = response.approximateCount // (2)
for (result in response)
response.forEach { log.info { "Found asset: ${it.guid}" } } // (4)
response.stream() // (5)
 .filter { it !is ILineageProcess } // (6)
 .limit(100) // (7)
 .forEach { log.info { "Found asset: ${it.guid}" } } // (8)
```

1. Keep the response object from the initial search, as it has a helper method for paging.
2. Since we set `trackTotalHits` to `true` (the default for the Java SDK even if we don't set it), the `.getApproximateCount()` will give us the total number of results. This can be over 10,000.
3. Iterate through all the results, across all pages (each page is lazily-loaded, so you can break out at any time without actually retrieving all pages of results).
4. Alternatively, you can iterate through all the results using `forEach()` on the response. (This uses the same underlying iterable-based implementation.)
5. Alternatively, you can stream the results. Streaming will also lazily-load only the pages of results necessary to meet the chained criteria for processing the stream.
6. When streaming, you can further filter the results to apply any complex filtering logic you couldn't push-down as part of the query itself.
7. When streaming, you can also limit the total number of results you want to process—independently of the page size.
8. Don't forget to actually _do_ something with the results in the stream 😉

### Raw REST API

```json showLineNumbers title="POST /api/meta/search/indexsearch"
{
 "dsl": {
 "from": 100, // (1)
 "size": 50, // (2)
 "track_total_hits": true, // (3)
 "query": {...}, // (4)
 "sort": [ // (5)}, // (6)} // (7)
 ]
 }
}
```

1. Starting point for the page of results being requested. In this example, you would be asking for the third page. (`0` would be from `0-50` for the first page, `50` would be from `50-100` for the second page, and this gives us `100-150` for the third page.)
2. The number of results per page (in this example, `50` results per page).
3. Enable `track_total_hits` so that your response includes an accurate total number of results.
4. You still need a query, to get some results 😉.
5. When paging, we should always sort the results (for consistency across the pages).
6. Include any of your own sorting, like this example putting the most recently-updated assets first in the results.
7. Also consider a tie-breaker sorting mechanism. In this case, we use the GUID of an asset to further sort any results that have the same last modified timestamp.

```json showLineNumbers title="Annotated response, in plain JSON"
{
 "queryType": "INDEX",
 "searchParameters": {
 "showSearchScore": false,
 "suppressLogs": false,
 "allowDeletedRelations": false,
 "query": "{\"from\":100,\"size\":50,\"track_total_hits\":true,\"query\":{...},\"sort\":[{\"__modificationTimestamp\":\"asc\"},{\"__guid\":\"asc\"}]}" // (1)
 },
 "entities": [ // (2),
 {...},
 ...
 ],
 "approximateCount": 24631 // (3)
}
```

1. Note that every response to a search includes the query that was run. You can deconstruct this programmatically to always determine the `from` you will need for the next page of results. (Basically: `from = from + size`.) And in fact, since you can programmatically extract both the query and sorting criteria from this you have everything you need to get the next page—the `query`, the `from`, the `size` and the `sort`.
2. The results themselves are the objects in the `entities` array. The size of this array will be *at most* `size` elements. Of course, your final page of results may not have a complete page of results, so it's possible that this array will be *less* than `size` (in particular, when you are on the final page).
3. Since the request sets `track_total_hits` to `true`, the `approximateCount` in the response will have an accurate number of total results. Note that this can go beyond 10,000.

[^1]: If you're familiar with Elasticsearch there are an alternative paging options using `search_after` and point-in-time (PIT) state preservation. (There also used to be scrolling, but this is no longer recommended by Elasticsearch.) We don't currently expose the `search_after` or PIT approaches through Atlan's search. However, you should still be able to page beyond the first 10,000 results using the approach outlined above.

---