I am developing a custom API application in .NET/MVC that performs the JSON request for the various API's. So far all is working and I've pulled in many records, however, one of the largest data pulls is for the PageViews API.
My question is regarding how to get the delta data or new data since the last request. The complete processed pulled in over a million records and this took a couple days to process. I have enhanced the code since then but the full call still takes awhile.
I'd like to get just the new data. What I have done so far is to create a new table and save the total pages and index it that way so I can begin the next request from the last page called. However, i've seen that new data is not at the beginning or end of a request.
Has anyone run into this issue and found a solution? If not, is there something similar that was done to what I am attempting?
Nevada State College