Creating a jupyter conda kernel for geospatial stuff

Setting up a jupyter kernel to be used in JupyterHub for geospatial analysis

  1. Setting up the kernel
    1. Creating Kernel
    2. Removing kernel
    3. Installing Resource manager
    4. Installing mpl widgets
  2. Setup geospatial (stare) env
    1. Gdal
    2. Cartopy, and proj, rasterio
    3. Geopandas
    4. DB interactions
    5. Dask
    6. STARE
    7. CCL

Setting up the kernel

Creating Kernel

From https://gdcoder.com/

pip3 cache purge
conda clean -y --all
conda update conda
ENVNAME=stare
conda create -y -n $ENVNAME python=3.11
conda activate $ENVNAME
conda install -y -c conda-forge ipykernel
python -m ipykernel install --user --name $ENVNAME --display-name $ENVNAME 
jupyter kernelspec list

Removing kernel

conda env list
conda activate $ENVNAME
jupyter-kernelspec uninstall $ENVNAME
conda deactivate
conda env remove -y -n $ENVNAME

Installing Resource manager

pip install jupyter-resource-usage

Installing mpl widgets

Has to be installed on the env that jupyterhub is running on as well. e.g.:

sudo /opt/jupyterhub/bin/python3 -m pip install ipympl

Then in the kernel:

pip install ipympl
conda install -y -c conda-forge ipympl

Regarding “Error displaying widget: model not found”

  • Try to restart the jupyterhub process
  • Also clear ~/.local/share/jupyter*
  • downgrade
pip3 index versions ipywidgets
pip3 install ipywidgets==7.7.2

pip3 install jupyterlab-widgets==3.0.0
pip3 install widgetsnbextension

Setup geospatial (stare) env

The dependencies are odd. Going in seems tomatter

Gdal

might as well throw the outdated system gdal out

sudo apt remove libgdal-dev gdal-bin
conda install -c conda-forge gdal

I ran into issues with gdal, failing to veriyf popler-data. I cleaned all to fix it:

conda clean --all

Other attempts

conda clean --packages --tarballs
conda remove poppler
conda install poppler
conda install -f poppler

Cartopy, and proj, rasterio

Make sure that pyproj and cartopy have not been installed with (gdal) binaries. By default, they ship with a small binary version of gdal. We need to make sure they use a full gdal instead. Either the system’s from apt or from conda. gh issue

pip3 uninstall -y pyproj cartopy rasterio
conda install -c conda-forge proj
conda install -c conda-forge cartopy
conda install -c conda-forge rasterio
conda install -c conda-forge pyproj
conda install -c conda-forge scipy
conda install -c conda-forge rioxarray
conda install -c conda-forge contextily

or rather:

conda install -c conda-forge proj cartopy rasterio pyproj scipy rioxarray contextily geopandas

Installing from pip appears to work; but the kernel crashes when using cartopy

pip3 install cartopy --no-binary cartopy
pip3 install pyproj --no-binary pyproj
pip3 install rasterio --no-binary rasterio

To transform images, we also need scipy

pip3 install scipy

Geopandas

Careful; If you have installed cartopy from pypi, DON’T install geopandas from conda. It will create a conflict (I think with shapely) and cause a runtime error when importing cartopy

conda install -y -c conda-forge geopandas
pip3 install geopandas

DB interactions

conda install -c conda-forge psycopg2
conda install -c conda-forge sqlalchemy
conda install -c conda-forge geoalchemy2
sudo apt install libpq-dev
pip3 install psycopg2 
pip3 install geoalchemy2

Dask

I am not sure if this might conflict with starepandas. Also, there was an issue with 2022.8.0 and starepandas

conda install -c conda-forge dask

STARE

pip3 install pystare
pip3 install starepandas

CCL

pip3 install connected-components-3d