How do you Move a Vast Imagery Archive?

Satellite imaging company DigitalGlobe recently wrote an interesting blog post about their vast archive of imagery and how they are moving it to the cloud.

Some interesting statistics from the post:
One image from WorldView-3 can be 30 gigabytes.
The DigitalGlobe archive is around 100 petabytes.
Their archive increases by 10 petabytes per year.

Comparing this to what we learned about Planet’s archive earlier this year, Planet has an archive of 7 petabytes. Before Planet launched the latest batch of 88 satellites and purchased Terra Bella from Google they were adding 7 terabytes daily or 2.5 petabytes per year to their archive. We estimate that they are now adding about two and a half times that daily or over 6 petabytes per year.

So, although Planet has the world’s largest imaging satellite fleet (actually, it is the world’s largest commercially-operated fleet of any type of satellite), their volume of imagery being gathered in terms of data storage required appears to be lower than DigitalGlobe’s. This is probably because DigitalGlobe’s satellites are higher resolution and thus produce much more data per area covered. It is also probable that DigitalGlobe’s satellites gather more optical wavelength bands.

DigitalGlobe have, until now, kept most of their archive in-house in a vast tape-based storage system. However, they are now moving to the cloud (Amazon Web Services). Rather than attempt to upload 100 petabytes to the internet, Amazon sent them a whole shipping container, a solution known as ‘Snowmobile’, complete with built in data storage, which DigitalGlobe then copied their data to and sent it back to Amazon. Interestingly, according to Amazon, 100 petabytes is the upper limit of their ‘Snowmobile’ solution (per truck).

We don’t know the exact size of the Google Earth database, but we estimate it to be several petabytes at least. So if you ever needed an offline version, then you would probably need a large suitcase-sized container at least.

The DigitalGlobe blog post features one of their images showing ‘Snowmobile’ outside DigitalGlobe’s headquarters but that image has not made it into Google Earth. However, we did manage to spot it in Street View:

Ground based imagery looking at an archive of satellite imagery.