GDAL access to private GCS buckets

How to access e.g. COGS on private GCS buckets via GDAL

One of the more frustrating endeavours: Question is how to authenticate ourselves against GCS from gdal. Super useful if we e.g. want to add COGs to QGIS.

As with so many things with GDAL, stuff is hidden and requires knowledge that I have no idea one would find. It is that sourspot just slightly beyond trivial. Examples are usually very basic. The gdal documentation states the configuration options, but it is not even obvious how to pass them to gdal.

One might think something like could work

gdalinfo /vsigs/spatialthoughts-public-data/viirs_ntl_2021_global.tif --config GS_SECRET_ACCESS_KEY key --config GS_ACCESS_KEY_ID acces key

But I actually have no idea how to correctly pass more than one config parameter to gdal.

What we can do is set CPL_GS_CREDENTIALS_FILE

gdalinfo /vsigs/spatialthoughts-public-data/viirs_ntl_2021_global.tif --config CPL_GS_CREDENTIALS_FILE ~/.boto

Now of course nobody really tells you what the .boto config file should look like; well actuall; this post does.

[Credentials]
gs_access_key_id=<YOURKEY>
gs_secret_access_key=<YOURSECRET>

Of course gdal calls it gs_access_key_id and gs_secret_access_key, while google gives you an Access key and a Secret. I don’t know if this is just me, or if it isn’t obvious to anyone. … I had to guess which one is which: The gs_access_key_id is the Access key (the longer one) and the gs_secret_access_key is the Secret.

With a ~/.boto in place, QGIS can access COGs in private buckets.