Getting More Than 100 Records With Airtable

I'm building an App that gets a database on Airtable with the Requests library and transform it into a list. My issue is that I can only get 100 records with the URL solicitation.

Solution 1:

As I saw in other posts, many people were dealing with the same issue. I tried to find solucions, but I coudn't fix it with the URL.

Although, thank God, it is easier to get all data from Airtable in Python with the Airtable API library. (http://airtable-python-wrapper.readthedocs.io/en/master/)

There is a function called get_all(), wich acept the maxRecords argument. Just call x.get_all(), with no argument inside, and the API will return every record in the table.

Solution 2:

TLDR: You could try using a recursive function that executes when an offset exists in the http response (json).

Acknowledgment: This solution took 7 hours of research, troubleshooting, and advice from the great, and powerful Doug. The project uses Alamofire to perform http requests, and SwiftyJson to access the JSON.

Cause: In the Airtable documentation, they state their rate limit is 100 items per request. If a request has more than 100 items, the request will include an offset.

They give an instruction to include the offset in your next request to get the next 100 results. But, they don't explain or demonstrate how to do it.

Answer : It's been tested to retrieve 2,565 items from 25 http requests. Written in Swift, the logic is simple:

Write a recursive function with an argument for an optional offset. Call the function without the offset. Check the http response (json) for the offset. If the offset exists, store the http request (json) in an array outside the function. Then, call that same function from inside itself - this time with the offset.

Extended code here.

funcrequestAirtableRecords(forTabletable: String, withTableViewtableView: String, withOffsetoffset: String?, completion: @escaping ([JSON]) -> ()) {
    let parameters: [String: Any] = offset !=nil? ["view": tableView, "offset": offset!] : ["view": tableView]
    do {
        let url: URLRequest=tryself.requestRecordsURL(table: table, method: HttpRequest.get, parameters: parameters)!Alamofire.request(url).responseJSON { (response) inswitch response.result {
            case .success(_):
                let json =JSON(response.result.value!)
                self.jsonArray.append(json)
                let nextOffset = json["offset"]
                if nextOffset.exists() {
                    self.requestAirtableRecords(forTable: table, withTableView: tableView, withOffset: nextOffset.stringValue, completion: { _in
                        completion(self.jsonArray)
                    })
                } else {
                    completion(self.jsonArray)
                }
            case .failure(let error):
                print(error)
            }
        }
    } catch {
        print("Error: Unable to request records from Airtable.")
    }
}

Solution 3:

This is the code I used in Python to solve this issue - in line with the answer by Christian, I used a recursive call. However, I'm posting this code b/c the OP requested an answer in Python.

defget_airtable_data(email_dict=None, offset=None):
        """
        queries airtable and creates a dict of {name: email} from records that are 
        listed as subscribed
        """if email_dict isNone:
        email_dict = {}
    r = requests.get(request_url, headers={"Authorization": bearer_token}, params={"pageSize": 100, "offset": offset})
    r_json = json.loads(r.text)
    for item in r_json["records"]:
        if item["fields"]["subscribed"] == 1:
            email_dict[item["fields"]["Name"]] = item["fields"]["Primary Email"]

    if r_json["offset"] isNone:
        print("all data loaded")
    else:
        try:
            get_airtable_data(email_dict, r_json["offset"])
        except KeyError:
            print("all data loaded")
    return email_dict