During the first phase of the project, we put together a basic web interface to prove out the concept and allow some limited remote data collection, since it isn’t always possible to have the equipment being monitored in the lab or to have a laptop connected to something in the field.

As the project developed, however, it became apparent that collecting lots of data would be required to develop the product further. This meant we would be putting the relational database we had selected for ease of prototyping to the test.

We quickly realized that the demo approach would not be adequate for the amount of data we needed to collect, and that we would need to beef up our pipeline to avoid locking up the database and potentially missing data.

Inserts are some of the longer operations on a relational database. The solution we came up with was to use AWS SQS to create a queue that could temporarily receive all of the incoming data, then after a short interval, insert it into the database in a single batch.

Even this relatively small step took us from handling tens of devices to comfortably passing a load test with ten thousand devices reporting at the same time

‟customer comment”

— author