Downloads & automation

Request format

GET https://www.bigdatacloud.com/download?package={compact|network|complete}&format={csv|mmdb}&key={YOUR_API_KEY}

Authentication

Pass your API key as the key query parameter, exactly as you do for the API: &key=YOUR_API_KEY. We hide your key in our request logs. Prefer not to put the key in URLs? Send it in the x-bdc-key HTTP header instead (x-bdc-key: YOUR_API_KEY) and use the two-step download below. If both are present, the header is used.

Examples

CSV:

curl -fL -o bigdatacloud-compact-csv.zip "https://www.bigdatacloud.com/download?package=compact&format=csv&key=YOUR_API_KEY"

-L follows the link and saves bigdatacloud-compact-csv.zip.

MMDB:

curl -fL -o bigdatacloud-complete-mmdb.zip "https://www.bigdatacloud.com/download?package=complete&format=mmdb&key=YOUR_API_KEY"

This saves bigdatacloud-complete-mmdb.zip. See What each request returns for the files in each ZIP.

Prefer not to put the key in URLs?

Send the key in the x-bdc-key header and download in two steps: read the link from the Location header, then fetch it without your key.

CSV:

URL=$(curl -s -D - -o /dev/null -H 'x-bdc-key: YOUR_API_KEY' \
  "https://www.bigdatacloud.com/download?package=compact&format=csv" \
  | awk 'tolower($1)=="location:" { sub(/\r$/, "", $2); print $2 }')
curl -fo bigdatacloud-compact-csv.zip "${URL:?no download link: check the key and package}"

The first command gets the download link; the second saves the ZIP without sending your key.

Don't add -L to this command: curl would send your key on with the download.

MMDB:

URL=$(curl -s -D - -o /dev/null -H 'x-bdc-key: YOUR_API_KEY' \
  "https://www.bigdatacloud.com/download?package=complete&format=mmdb" \
  | awk 'tolower($1)=="location:" { sub(/\r$/, "", $2); print $2 }')
curl -fo bigdatacloud-complete-mmdb.zip "${URL:?no download link: check the key and package}"

What each request returns

Scheduled download

A daily job that fetches the current MMDB release, unzips it to a new folder and switches over when the download succeeds.

#!/bin/sh
# fetch-bigdatacloud.sh — run daily from cron, e.g.  30 3 * * *  /opt/bdc/fetch-bigdatacloud.sh
set -eu
PACKAGE=compact
FORMAT=mmdb
DEST=/opt/bdc
NEW="$DEST/$PACKAGE-$FORMAT.$(date +%Y%m%d)"
mkdir -p "$DEST"
STAGE=$(mktemp -d "$DEST/$PACKAGE-$FORMAT.XXXXXX")
OLD=""
cleanup() {
  if [ -n "$STAGE" ]; then
    rm -rf "$STAGE"
  fi
  # A swap that stops before the new folder is in place puts the previous release back.
  # Once the new folder is in place, drop the aside copy. Either way, no .old folder remains.
  if [ -n "$OLD" ]; then
    if [ -d "$NEW" ]; then
      rm -rf "$OLD"
    else
      mv "$OLD" "$NEW"
    fi
  fi
}
trap cleanup EXIT

curl -fL -o "$STAGE/download.zip" \
  "https://www.bigdatacloud.com/download?package=$PACKAGE&format=$FORMAT&key=$BDC_API_KEY"
unzip -q "$STAGE/download.zip" -d "$STAGE"
rm "$STAGE/download.zip"

# Only after the unzip succeeded: move today's folder aside, put the new release in its place, then point "current" at it
if [ -d "$NEW" ]; then
  OLD="$NEW.old.$$"
  mv "$NEW" "$OLD"
fi
mv "$STAGE" "$NEW"
STAGE=""
ln -sfn "$NEW" "$DEST/$PACKAGE-$FORMAT.current"
if [ -n "$OLD" ]; then
  rm -rf "$OLD"
fi
OLD=""
trap - EXIT

# Remove releases more than 7 days old. The name is exactly eight digits, so an aside folder is not matched.
find "$DEST" -maxdepth 1 -name "$PACKAGE-$FORMAT.2[0-9][0-9][0-9][0-9][0-9][0-9][0-9]" ! -path "$NEW" -mtime +7 -exec rm -rf {} +

Prefer not to put the key in URLs? Replace the curl -fL line with the header two-step. It still writes into the fresh temporary directory, so a second run the same day does not prompt:

URL=$(curl -s -D - -o /dev/null -H "x-bdc-key: $BDC_API_KEY" \
  "https://www.bigdatacloud.com/download?package=$PACKAGE&format=$FORMAT" \
  | awk 'tolower($1)=="location:" { sub(/\r$/, "", $2); print $2 }')
curl -fo "$STAGE/download.zip" "${URL:?no download link: check the key and package}"

Readers that keep an MMDB file open need to reopen it to pick up the new release.