Troubleshooting

ABS Data APIs have an API Gateway with the following limitations:

• Maximum response time is 30 seconds. Exceeding this results in a 504 error and the message "Endpoint request timed out".
• Maximum response size is 10MB. Exceeding this can result in a 504 or 500 error.

Solutions

Make sure you are calling the API with compression enabled, ensure your headers include: Accept-Encoding: gzip, deflate, br. Compression reduces API response size significantly. However, vary large data requests will still exceed the API gateway limit even with compression.

If you are still receiving an error then you will need to break your data request into several smaller requests. The simplest way to do this is using the startPeriod and endPeriod query parameters to subset data on time range. Or use dataKey to subset data on dimension members (e.g. request all data for each region separately). For information on this see: GET Data, Path Parameters, dataKey.

If you need to get all data, the following API call will return all data for the specified dataflow as a CSV file with codes and labels: https://api.data.abs.gov.au/data/ABS,DataflowId,Version/all?dimensionAtObservation=AllDimensions. Make sure you use following header in your call: Accept: application/vnd.sdmx.data+csv;urn=true;file=true;labels=both

The maximum allowed length of the ‘dataKey’ section of the URL is 5,000 characters. Requests that exceed this limit will return a 400 error.

Solution

To work around the maximum allowed URL length, we recommend using wildcards to request all members of a given dimension rather than specifying the code for each dimension member individually in the API call. Simply remove all codes for a given dimension to request all members for that dimension. For more information on this see: GET Data, Path Parameters, dataKey.

A request that used to return data or structures now returns a 404 error and a message that requested structures could not be found.

Solution

The most likely cause for this error is an incorrect version number. All structures in the ABS Data API have a version number e.g. "1.0.0". If a structure has been updated to a new version and the old version removed, any requests to the old version will result in a 404 error and a message that the requested structures could not be found. Old versions of data structures are removed to prevent access to out of date data.

Data structures and Dataflows may be updated to new versions occasionally. We recommend not including version numbers in your data or data structure API requests to ensure you always return the latest version.

Removing the version number from your requests will default to requesting the latest available version. E.g: https://api.data.abs.gov.au/datastructure/ABS/JV/ or https://api.data.abs.gov.au/data/ABS,JV,/all