The current FRR implementation lets you fetch particular config objects in one iteration.
This becomes problematic when querying a bulk state data (e.g. millions of routes) as it holds the CLI/Object
until it is fully displayed. There is no way to limit requests to "n" first objects, etc...
In such a case, you are reading the entire object and skipping the uninteresting data, which is very inefficient.
There should be an easy-to-use and robust mechanism to split (paginate) the data into manageable pieces.
This problem has already received some attention from the frr community, but the proper solution was not
1. Previously, the northbound architect created a pull with the solution, but after multiply rounds of review, he stopped active contributions to the project. So that PR was not merged in the end. https://github.com/FRRouting/frr/pull/6371 2. Currently, the community with engineers from the `VMware` team is working on the new `Centralised Management Daemon (MGMTD)` One of the development goals is - '13. Support for batching and pagination for display of large sets of operation data.` https://github.com/FRRouting/frr/wiki/FRR-Centralized-Management-Requirements The pull request tracking the development: https://github.com/FRRouting/frr/pull/10000 Is in development for about 2 years, ~20k lines of code, doesn't look like it will be merged soon
The  is being actively developed, but a lot of work must be done before it can be used in practice.
It is good enough for some basic tests, but the DB connection is available only for staticd and part of zebra.
More details will be described in the follow-up comments.
Regarding the :
After trying to develop some hacky solutions, I realized that most of my ideas are already implemented in the dropped PR.
Since there have not been many changes to the northbound architecture, it is possible to merge it back by hand and customize it for our needs.
The basic demo can be found at:
The PR introduces a new cli flags to perform data fetching with pagination
Fetching #n first elements show yang operational-data <xpath> max-elements <n> <demon> Fetching all elements by #n elements per iteration show yang operational-data /frr-interface:lib max-elements 3 repeat zebra
To demonstrate how it can be extended, I have introduced a new flag next that does an iteration
starting from the previous one.
Demo example :
Demo visualization (GIF):
Details on the development and the timeline will be in the following comments.