Here’s a good way to upload data from an ecommerce store’s API using hourly updates. Here’s what I did.

You can easily run this code on a linux based machine or Google Compute Engine using an hourly crontab.

For the two files below, I created a folder for them to both go into. The first script uses jq, a command line processor. Since Google BigQuery only accepts new-line delimited JSON files, you can use <a href="https://stedolan.github.io/jq/">jq</a> and set the -c flag to convert the results to that format.

You’ll need to update the xxxx information with your own data.

This JSON file below is the actual schema you’ll use in your BigQuery table.