Data sources and compilation methods

4.1 In Australia, data from the Department of Home Affairs (and its predecessors) has been the primary source of international merchandise trade statistics for over 100 years. The ABS uses information from the Department of Home Affairs which has been provided to them by importers, exporters, shipping companies, airlines, freight forwarders, or agents on behalf of these businesses, so that the goods can be cleared for entry to, or export from, Australia.

4.2 This chapter describes the exports and imports information which is collected by the Department of Home Affairs and transferred to the ABS. The way the ABS receives, transforms, quality assures and aggregates data for dissemination is explained. The small number of non-Department of Home Affairs data sources are also described.

4.3 The Department of Home Affairs is the federal authority responsible for the collection and management of export and import documentation. The documentation is used by the Department of Home Affairs to assess and collect duty and other revenue payable on imported goods, and to monitor and control the physical movement of goods into and out of Australia.

4.4 For exports, declaration information must be lodged by exporters (or their agents) with the Department of Home Affairs prior to, and up to six months before, the departure of the goods. In mid-2013, 99.7% of all export declarations were submitted electronically. Electronic cargo manifests are mandatory for each international voyage and flight.

4.5 For imports, a declaration must be lodged with, and cleared by the Department of Home Affairs before access to the goods can be obtained. Import declarations may be lodged well in advance of the expected arrival of the goods. Over 99.5% of import declarations are reported electronically to the Department of Home Affairs.

4.6 The system used by the Department of Home Affairs to collect and process export and import information is known as the Integrated Cargo System (ICS). When a declaration is entered into the ICS a range of checks are applied to ensure that only valid codes for the commodity, country, port etc are used and that all mandatory information has been entered. Warning messages are sent to alert importers and exporters of possible errors. The Department of Home Affairs checks are aimed at ensuring compliance with customs laws in an environment of self-regulation and risk assessment. In practice, this means that while most declarations are processed without intervention, compliance is assured through the Department of Home Affairs legislative power to monitor and intercept cargo and to audit company records and practices. Where evidence of non-compliance with customs legislation is identified, compliance can be enforced by imposing sanctions or fines or by prosecution.

4.7 Every day (including weekends and public holidays) the ICS produces a file that includes export declarations and manifests processed on the previous day and transmits the file to the ABS overnight. Table 4.1 lists the export data items received from the Department of Home Affairs in the daily files. While some data items have been grouped in the table below, for example, quantity has been grouped with the unit of quantity, each item included in the grouping is listed in the description. In addition to the data listed below, the ABS also receives management information, to verify the full receipt of Department of Home Affairs data files. Appendix 5 lists these data items and explains whether they are available in ABS aggregate output.

4.8 After midnight each day the ICS produces a file of import and clearance records which were finalised by the Department of Home Affairs in the previous 24 hours. The file is transmitted to the ABS overnight. Table 4.2 below lists the import data received daily from the Department of Home Affairs. While some data items have been grouped in the table, for example, duty calculation fields, each item included in the grouping is listed in the description. Appendix 5 (in the Data downloads section) lists these data items and explains whether they are available in ABS aggregate output.

4.9 The relationship between the ABS and the Department of Home Affairs is defined in a signed agreement between the two agencies. Meetings between the deputies of both agencies are held annually and regular discussions occur at the work level. Together these arrangements ensure that both agencies understand the requirements of the other agency and ensures that the relationship remains strong.

4.10 The ABS receives a range of information about the customs procedures applied to transactions. This includes the line nature type, treatment code, duty fields and GST fields. The information supplied by Department of Home Affairs can be used to ensure the correct treatment of transactions in ABS statistics.

4.11 In 2001 when the concepts, sources and methods (CSM) for international merchandise trade statistics was first published, ABS statistics were processed on a mainframe computer. Between 2003 and 2006 the international merchandise trade system was re-designed, tested and transferred to a new platform. Over a similar period the Department of Home Affairs were re-engineering their systems for collecting exports and imports information. While these system changes were significant in terms of the way data are collected, delivered and processed there was only minimal impact on statistical output (see Appendix 2 in the Data downloads section). If users of the statistics are interested in the previous systems and methods they should read the 2001 version of the CSM.

4.12 The IMT system uses standard ABS statistical infrastructure and tools with adaptations to cover the type of data (export and import declarations, not survey questions and answers) and the volume of data received on a daily basis.

4.13 The IMT system has a separate test environment for testing changes or new functionality prior to implementation in the production environment. The test environment is also used as a training environment.

4.14 The timing of the processing cycle differs between exports and imports due to exports being compiled on a departure date basis and additional time allowed for the finalisation of export manifests. Consequently, for exports the commencement of the new processing month normally occurs around the middle of the reference month. For imports the commencement of the new processing month happens earlier, generally on the sixth working day.

4.15 Threshold edit values are set to manage processing workloads. These are used to ensure that all records with values above the thresholds and records with fatal or warning error messages are checked by ABS editors.

4.16 Before processing commences it is necessary to ensure that metadata changes taking effect from the current reference month have been implemented correctly. For example, if a new 10 digit commodity code is active in the ICS from 1 January 2014 this new code must be loaded to the IDW metadata before processing of January 2014 data commences.

4.17 The IDW is a key database in the processing of international merchandise trade data. It is the repository for all export/import data and metadata, including:

• all the source data (including all versions of each transaction) from the Department of Home Affairs
• all amendments
• information added by ABS editors about the amendments
• all the metadata used by the IMT system (e.g. commodity, country and port codes and descriptions) and some metadata which can be used to link international merchandise trade data to other ABS economic surveys e.g. ABN
• additional transactions which are added when customs data are not available, see the Other Data Sources section (paragraphs 4.55-4.56).

4.18 The data load processes initially re-format and map some customs data items to ABS data items. The most important re-mapping is for any fields which use the UN/Locode (i.e. the country, port and state fields). This is because when the Department of Home Affairs introduced the UN/Locode in their system (with the introduction of the ICS) the ABS decided to minimise changes to the IMT system and statistical output by continuing to use the previous country, port and State codes, (see Diagram 6.1 in the Country, Overseas Port, State and Australian Port chapter).

4.19 Processing stage codes are added to records in the data load to IDW. These codes are used throughout the IMT system to identify particular record types and their status at particular processing points.

4.20 The load process identifies out of scope transactions which are not edited but are still loaded to the IDW for analytical purposes. The following record types are out of scope of international merchandise trade statistics.

• Consular records: These are transactions covering goods sent to Australian diplomats overseas (do not subtract from Australia's stock of material resources) or brought to Australia for foreign diplomats stationed here (do not add to Australia's stock of material resources). These transactions are internal flows and are out of scope for international merchandise trade statistics (see paragraph 2.47 in the Scope, Coverage and Treatments chapter).
• Customised software records: Some goods transactions are considered services. Where possible these are identified in the load e.g. computer software with a calculated unit value (reported customs value divided by reported quantity) greater than $10,000.00 is an indication that the software is customised and not an off the shelf product (see paragraph 2.49 in the Scope, Coverage and Treatments chapter).
• Value below the threshold: The customs thresholds which require the lodgement of a customs declaration are described in the Scope, Coverage and Treatments chapter (paragraph 2.8). Import consignment lines which are below the threshold ($1,000) are often received because they are part of a consignment of goods which includes lines above the threshold or because the goods for import are tobacco and alcohol which require a full declaration regardless of value. These low value lines are treated as out of scope. Export consignments with values below the consignment threshold ($2,000) are also received as an export permit may apply to the commodity. These are also treated as out of scope. This treatment ensures a consistent approach as the records identified as out of scope only represent a small proportion of the value and volume of goods below these thresholds. Value estimates for goods below the thresholds are included in Australia's balance of payments statistics.
• Outside the revision period: Amendments for records which are outside the ABS revision period (see paragraphs 10.15-10.16 in the Data Dissemination chapter).
• Bank notes in circulation: These are out of scope when they are valued at their face value and represent a financial transaction (see paragraph 2.46 in the Scope, Coverage and Treatments chapter). As these records are not easily identified in the load process it is more likely that they are changed to out of scope by an ABS editor.

4.21 Replacement or amendment records which alter an out of scope record to in-scope record are edited and included in international merchandise trade statistics.

4.22 Once the export and import data are loaded to the IDW, all in-scope records are passed through their respective edit process (exports or imports edit). The edit process checks the details of each transaction against a set of edit conditions which are categorised as fatal, warning (notifiable) or informative. Transactions which fail fatal or warning edits then have messages sent to the edit/amendment system for investigation by an ABS editor. The investigation may require contact with the exporter, importer or agent to resolve. Informative edits usually indicate an automated system adjustment has been applied. Transactions which fail informative edits are not viewed by an ABS editor unless they also contain a fatal or warning edit.

4.23 The following list describes the most common edit checks applied to both export and import data:

• identification of all transactions with values equal to or above a large value threshold. These transactions are subject to detailed checks including confirmation with the exporter, importer or their agent if necessary
• internal consistency checks e.g. if the mode of transport is reported as sea, the Australian and overseas ports should not be airports
• unit value and for exports gross weight unit value edits are applied. These edits are generally informative and are only assigned to editors if they fail a warning or fatal edit
• automatic adjustment of certain import records, where the unit values fall well outside the expected range for the relevant HS classification. The quantity is assumed to be in error and is estimated

4.24 To ensure that ABS editors focus on the most significant transactions, and to minimise contact with exporters, significance editing is used in conjunction with many of these edit checks. Significance editing gives a score to all transactions to indicate how the reported value (exports FOB value, imports customs value) differs from a predicted value. The size of the score is an indication of the potential error. Transactions with scores beyond the significance score thresholds are flagged for review by an ABS editor. The score is allocated based on data calculated by the input editing benchmarks. The input editing benchmarks use the previous 12 months of data to determine if quantity or gross weight is the best predictor of value.

4.25 The re-edit process is essentially the same as the edit process but occurs following editor amendment whereas the edit process occurs when a new transaction (or revised version of a transaction) is received from the Department of Home Affairs.

4.26 Chart 4.3 below shows the imports data load where the IDN information from the Department of Home Affairs is re-formatted and loaded to the IDW. Earlier versions of an IDN which are already on the IDW are identified (to ensure that the new version is not loaded as a new transaction) and up-dated in the Load to the IDW process. In Chart 4.3 below, this is shown as data moving from the IDW back to the 'Load to IDW' stage and then through to the IDW again. The metadata which is used in the Reformat and Imports edit processes includes the edit checks and country/port metadata. After the edit check a new version of the transaction with any relevant alerts is created in the IDW (shown as a two way arrow between the IDW and Imports edit).

4.27 There are many similarities in the exports and imports data load processes. However, they are separate processes involving different data items and metadata and there are some aspects which are unique to the exports load. The unique export processes are described in the following sections.

4.28 Exporters must complete an export declaration (EDN) with the Department of Home Affairs in advance of the goods arriving at the wharf for export. The EDN is an intention to export and includes preliminary information about when the goods will leave Australia and on what carrier the goods will be transported. The actual departure details are listed on the ship or aircraft manifest. When the ABS receives both the EDN and its manifest, the transaction is said to be acquitted 'matched' and the goods exported.

4.29 The exports matching process matches EDNs and manifests and adds the departure details from the manifest to the EDN information. In most cases the EDN is received by the ABS before the manifest. The EDN will be loaded to the IDW and identified as an EDN awaiting acquittal. Once the corresponding manifest is received the EDN is acquitted and, if the transaction has not been reported in Australian dollars, the Reserve Bank of Australia exchange rate for the day of departure will be used to convert values to Australian dollars. If the manifest is received first it is also loaded to the IDW but identified as a manifest record. When the EDN is received, the matching process retrieves the manifest from the IDW and the EDN is acquitted. This process of retrieving EDNs/manifests from the IDW and matching them with their corresponding manifests/EDNs is depicted with a two way arrow between the exports matching process and the IDW.

4.30 A small number of records are 'self acquitting'. This means the EDN is marked as acquitted (and therefore exported) without being shown on a manifest. Self acquitting EDNs are those export declarations which do not get recorded on a manifest. These are postal records, ships and aircraft exported under their own power, ship and aircraft stores and spares and unaccompanied baggage, identified by the relevant export goods type.

4.31 Because EDNs are lodged and passed to the ABS in advance of export, the exchange rate can be for a future date (and is therefore unknown). When these EDNs are loaded to the IDW the FOB value is estimated using the latest available exchange rate. To update these estimated records the load process checks for new exchange rates and retrieves the estimated records from the IDW. These records are passed through the load process again to update the information. In Chart 4.4 below, this is shown as data moving from the IDW back to the Match and 'Load to IDW' stages (with input of exchange rates) and then through to the IDW again.

4.32 Towards the end of the processing month, a report is produced showing all unacquitted (no manifest received) large value EDNs with expected departure dates within the processing month. If the report shows an abnormally high value of unacquitted EDNs sometimes a decision to load more daily files to the IDW is made in the hope that manifests to acquit the EDNs are received. However, there comes a point where meeting the publication deadline means that no more data is loaded.

4.33 The edit/amendment system is used to view and, if necessary, amend transactions that have been flagged by the edit process as requiring examination by an ABS editor. The system is also used to view and, if appropriate to do so, amend unusual transactions identified as requiring ABS editor attention (e.g. by output edit processes). Amendments are applied as a result of contact with the exporter or importer, identification of a repeated mis-reporting error or other information provided in the transaction or similar transactions. Any amendment of a transaction requires the ABS editor to include comments (which are appended to the transaction) to explain the change(s) made. All amended transactions go through the re-edit process to ensure data integrity and transactions failing re-edit checks are returned to the edit/amendment system for re-examination. The system also provides the means to view any transaction using its unique identifier. The system is also used by ABS editors and managers to monitor work flow e.g. the number of transactions awaiting editor attention (because they have failed fatal or warning edits).

4.34 Aggregate editing is an important component of statistical compilation and finalisation of output for release. Aggregate editing involves creating monthly totals by particular levels of commodity (HS 6-digit or SITC 2-digit), country, state, mode of transport and unit of quantity and comparing these to previously released aggregates. Significance scoring (scoring the current month aggregates against a benchmark derived from the previous 12 months of data) is used to highlight the most significant aggregate movements and to set workloads. Where necessary further follow-up with exporters and importers is undertaken and transactions amended. Records of any investigations and findings are kept and reviewed by more senior staff during the finalisation of output.

4.35 Output editing involves examining the data for particular commodities or 2-digit chapters of the HS. Examination of data at these levels may reveal a significant change. When this occurs the transactions contributing to the change are identified and scrutinised.

4.36 The trade query tool is an integral component of aggregate and output editing. The query tool allows staff to retrieve data from the IDW using a defined set of criteria. Simple reports can be produced, data aggregated and sorted to help ABS editors to understand movements or to resolve mis-reporting errors in the data. The trade query tool is also used in editing and verifying individual transactions. Where mis-reporting errors are found the relevant transactions are amended using the edit/amendment system.

4.37 International merchandise trade statistics are based on a large number of individual transactions and are released at a very detailed aggregated level e.g. 8-digit exports and 10-digit imports by Australian port, country and overseas port. As the ABS has limited resources available for ensuring the quality of international merchandise trade statistics, resources are focussed on the data required for macroeconomic statistics.

4.38 Once editing is finalised and prior to the public release of international merchandise trade statistics, data are prepared for inclusion in the balance of payments.

4.39 The preparation of data for the balance of payments signals the first approval stage in the release of international merchandise trade statistics. The export and import data supplied for the balance of payments includes totals on a SITC and, for imports only, a BoPBEC basis, information about significant month to month movements and other information used to compile goods data on a balance of payments basis. More information regarding classifications such as SITC and BoPBEC and how data is prepared for the balance of payments is presented in the Classifications and Relationship with Other ABS Macroeconomic Statistics chapters respectively.

4.40 International merchandise exports statistics are first released on a balance of payments basis in International Trade in Goods (cat. no. 5368.0). Goods credits and debits on a balance of payments basis are closely related to international merchandise exports and imports but have been adjusted to take account of the principles of economic ownership and residence. More information can be found in the Relationship with Other ABS Macroeconomic Statistics chapter (paragraphs 12.4 to 12.7).

4.41 Australia's balance of payments is published quarterly in Balance of Payments and International Investment Position, Australia (cat. no. 5302.0). In this publication, the quarterly estimates of goods and services credits and debits are the sum of the months published in the corresponding release of International Trade in Goods (cat. no. 5368.0) without any changes. The quarterly exports and imports statistics which are included in the Expenditure on GDP estimates in the publication Australian National Accounts: National Income, Expenditure and Product (cat. no. 5206.0) are the same as those published in the quarterly balance of payments (so they include both goods and services and are on a balance of payments basis). Detailed exports and imports data on an international merchandise trade basis are supplied to the National Accounts Branch for use in compiling the annual supply-use tables, for input-output analysis and for other analytical purposes.

4.42 On a quarterly basis, data for the Prices Branch of the ABS are also prepared. The data include detailed Harmonized System commodity level data (8 digit export and 10 digit import) by country, value, quantity and gross weight. The data are used in the compilation of the International Trade Price Indexes, Australia (cat. no. 6457.0).

4.43 International merchandise trade data provided to the areas of the ABS which compile the balance of payments, national accounts and prices statistics include aggregates and some transactions level data so that the compilers of those statistics are able to fully understand and interpret the data for their

4.44 The ABS Information Warehouse (ABSIW) is the repository for data available for dissemination, including international merchandise trade statistics. Also stored on the ABSIW are all the metadata which make the statistics meaningful e.g. international merchandise trade commodity and country codes and descriptions and associated information. The ABSIW was created to enable all ABS data to be captured, defined and delivered in a consistent way.

4.45 At the end of the processing month after the data have been finalised and the output files created, international merchandise trade statistics are loaded to the ABSIW User database (an intermediate database where data are checked prior to release). The publication tables, time series spreadsheets and data cubes found on the ABS website are all extracted from the ABSIW User database. At 11.30am (Canberra time) on the day of release, the publication, all storage tables involving international merchandise trade data and metadata for the particular collection, time series spreadsheets and data cubes are copied from the ABSIW User database to the ABSIW Output database (output environment) and ABS Website.

4.46 Consultancy and subscription data are created from the ABSIW Output database from 11.30am on the day of release of the publication.

4.47 There are only a small number of regular non-Department of Home Affairs data sources used in the compilation of international merchandise trade statistics. These data sources are:

• RBA exchange rates - these are used to convert exports FOB values to Australian dollars using the exchange rate applicable on the date of departure from Australia (or in some cases the expected date of export), see paragraph 4.34 in the ABS Compilation Methods section above
• public information - newspapers and websites are used to verify unusual transactions
• other ABS surveys or compilation areas - occasionally more up to date or additional information about goods being exported or imported might be provided by another ABS survey, for example, the quarterly Survey of New Capital Expenditure.

4.48 Export declarations are not required for fuel provided to foreign airlines and ships in Australia. The value of these transactions is estimated and included in Australia's merchandise export statistics using price and historical quantity data while alternative data sources are explored. In addition, because exports are recorded on a date of departure basis, occasionally research shows that a significant export has occurred for which no details have yet been received from the Department of Home Affairs. These transactions are added to the IDW towards the end of the processing month and processed in the same way as customs records i.e. once added the transaction(s) are passed through the edit process, subject to output editing and included in the final results.