Story 5: Into the cloud

For the 2021 Census, the ABS took a large step forward in its journey to move more of its services and systems into the cloud. This started with the Census Digital Service and also included:

• the Operations Insights platform
• the Call Centre Agent tool for use by Services Australia and ABS staff
• the Census Knowledge Base
• the Paper Form Request Service
• the MyWork mobile app for field staff
• Claire, the chatbot
• online training courses
• the field staff payroll solution
• Dynatrace (system monitoring software).

From August to November 2017, the ABS successfully made its first real foray into using cloud services with the Australian Marriage Law Postal Survey. During this process we got to experience the scale and security available in the cloud. We built an online form in a cloud platform primarily for Australians on the electoral roll living overseas. While this only accounted for 34,447 responses, it was tested on millions of transactions in an hour, easily proving the benefits of cloud services for large volume IT solutions. They are far more scalable and when well designed, can be more cost effective than traditional local hosted environments.

Extensive security testing and validation by IT security experts, including the Australian Cyber Security Centre, demonstrated that cloud-based solutions are both viable and preferable, especially from a security perspective. They are now ABS’ preferred solution for all future ICT developments.

When we created the requirements for the 2021 Census Digital Service (CDS), we chose a cloud based solution despite there being very few Australian government cloud-based services at that time.

ABS contracted PwC Australia after an open tender in May 2019 to deliver the CDS in partnership with Amazon Web Services and hosted on the Amazon cloud platform in Australia. They were able to show how a system could be delivered that was highly secure, reliable and could perform at the scale needed. We had a target of 350 form submissions per second at peak or 1.26 million responses per hour on Census day.

The system was built primarily using Amazon’s “native technologies” that are specifically designed for their cloud services, including Lambda, Dynamo DB and S3 buckets. Amazon Web Services provided architectural design input, conducted ‘well architected reviews’ and a reliability review.

The system integrated with third party services that we used for the Census including Google’s ReCaptcha, Geoscape’s Address Look-Up system and Clevertar’s Census guided chatbot, Claire.

The CDS included:

• the online form
• the Census website
• a content management system
• self-service facilities such as ‘not-at-home’ reporting, request a new Census number and paper form requests.

"A key focus of the 2021 Census was to ensure the digital service had redundancy, performance and protection against cyber threats." -  Gwil Davies, Partner, Digital Innovation and Cloud Engineering, PwC Australia.

Our next large cloud development was the Operations Insights platform, also hosted on the Amazon cloud. ABS staff, Amazon Web Services, and their partners Shine and the ARQ Group, worked together to create this first cloud-based data lake for the ABS to store data on how the Census was progressing.

This system ingested operations data from multiple data sources and produced approximately 100 reports showing progress of Census operations and enabling us to make numerous critical, real-time decisions.

The volume of data, complexity of the data transformations, and volume of reports made it an ideal candidate to be cloud hosted. See Story 8 - Monitoring operations during the Census for more detail on the use of this service.

While cloud systems can be scalable and secure, they need to be designed and built with that in mind. They have complexities that need to be clearly understood. To maximise success, partnering with experts across a range of disciplines was essential. It is important to begin with the end in mind in terms of performance, scale and security.

Independent testing by third party vendors and other government agencies (the Australian Cyber Security Centre in particular) was critical for reassuring us we had the right solution.

Our use of other fully established cloud services such as Geoscape’s Address Lookup, Google’s Captcha and Amazon Web Services Connect, provided confidence that the system could handle the load and performance required.