How to prepare your docker-compose file

Note

This section aims a breaking down and explaining the different components of the docker-compose.yml file.

The docker-compose.yml file allows the deployment of multiple containers at once.

docker-compose-template.yml
 1########################################################################
 2# For details about the docker-compose file, please refer to           #
 3# https://docs.smartscope.org/installation/docker-compose_details.html #
 4########################################################################
 5version: "3"
 6services:
 7  smartscope:
 8    image: localhost/smartscope:0.7
 9    container_name: smartscope
10    build:
11      context: .
12    restart: always
13    volumes:
14      ######## FILL THIS ##########
15      - :/mnt/data/
16      ######## ADD YOUR MICROSCOPES #########
17      # The synthax from the microscope volumes is as follows:
18      # - /path/to/where/serialem/will/write/files/:/mnt/your_scope_name_here/
19      # Example:
20      # - /mnt/gatan_Raid_X/smartscope/:/mnt/arctica/
21    environment:
22      SECRET_KEY: 'd41d8cd98f00b204e9800998eag' #Make sure this is unique! Generate a new one with: $RANDOM | md5sum | head -c 25
23    depends_on:
24      - db
25      - cache
26    networks:
27      - smartscopenet
28  db:
29    image: mariadb:10.5
30    container_name: smartscope-db
31    restart: always
32    volumes:
33      ######## FILL ONLY THE FIRST VALUE ##########
34      - :/var/lib/mysql
35      - ./config/docker/initialdb.sql:/docker-entrypoint-initdb.d/initialdb.sql ##Copies the initial db file to initialize the datase on first launch##
36    environment:
37      MYSQL_DATABASE: smartscope
38      MYSQL_USER: root
39      MYSQL_ROOT_PASSWORD: pass
40    networks:
41      - smartscopenet
42  cache:
43    image: redis:6.2-alpine
44    restart: always
45    command: redis-server --save 20 1 --loglevel warning
46    networks:
47      - smartscopenet
48  nginx:
49    image: nginx
50    ports:
51      - 48000:80
52    volumes:
53      ######## FILL ONLY THE FIRST VALUE ##########
54      - :/mnt/data/ #This is the same what is bound to /mnt/data in the smartscope service.
55      - ./static/:/opt/smartscope/static/ #Doesn't need to be changed
56      - ./config/docker/templates_noSSL/:/etc/nginx/templates/ #Change if using SSL, view docs for more info.
57    networks:
58      - smartscopenet
59    depends_on:
60      - smartscope
61
62networks:
63  smartscopenet: {}

The file is essentially a list of services that need to be run. Each service contains a list of settings, some of which need to be adapted for each system.

Services

SmartScope requires 4 different services/containers to run.

smartscope

This container is the most important as it runs the SmartScope workflow and Web Server.

db

This container runs a MariaDB SQL database server to which SmartScope connects to save and query data.

cache

This container runs a Redis cache mainly involved in websocket communication between the SmartScope workflow and the Web Interface.

nginx

This container runs the Nginx reverse-proxy to serve as a proxy for the Web Server and all the different static files that need to be served to the WebUI. It also handles the https/SSL encryption (needs to be specifically enabled).

At this point the docker-compose file could be simplified to this:

 1version: "3"
 2services:
 3  smartscope: 
 4    ...smartscope settings here
 5  db: 
 6    ...db settings here
 7  cache: 
 8    ...cache settings here
 9  nginx: 
10    ...nginx settings here

Services settings

In each service, there are multiple sections to fill, some of which need to be edited to each systems. This section will break down the options and

image

The name of the image that is going to be used. When running the podman-compose up, podman will look if the image is already present in the local computer and otherwise will download the the image from the selected repository.

Note

When prompted to download the images, select the docker.io option as the other repository may not work properly.

build

When the image is specified at localhost/imagename and is not found, it will build it from the Dockerfile at the specificed context location. Other arguments can be passed to the image building process.

Only the smartscope service has a build section:

Minimal build block
smartscope:
  build:
    context: .

The section has optional arguments which can be added as follows

Optional build arguments
smartscope:
  build:
    context: .
    args:
      - USER_ID=
      - GROUP_ID=

These arguments are used to control the uid and gid other user account within the conatiner. The advantage of these arguments are mostly for a file permission standpoint. By setting these values to a user and group on the host system, it streamlines the permissions and file access later on without having to create new permission set.

For example, if you would like to files to be owned by a user named smartscope_user with a uid of 1003 set the group to cryoem_staff with a gid 1005, you may use those values in the args section.

Example
build:
  context: .
  args:
    - USER_ID=1003
    - GROUP_ID=1005

To find uid and gid:

#Find the uid of a user
$ id -u smartscope_user
1003

#Find the gid of a group
$ getent group cryoem_staff
cryoem_staff:*:1005:list,of,users,in,group

container-name

If not specified, it will assign a name to the container usually under the form of directory_service_1. The most likely containers that, as a user, you may want to interact with the command line are the smartscope and db. This is why the container-name has been fixed to for these services:

container-name
version: "3"
services:
  smartscope:
    container_name: smartscope
  db:
    container_name: smartscope-db

To connect to the shell of these services:

Example
$ podman exec -it smartscope /bin/bash
smartscope_user@53310f8baf12:/opt/smartscope$

volumes

Warning

This is most important section to modify according to your system specifications. All services except cache have changes in the volume section.

Warning

All directories and files bound to the smartscope container must have read-write permission to user in the container (default uid:1000) or the uid/gid specified in the build section.

Every item in the volume section serves to bind (or mount) a path from the host system to the container. The synthax goes as follows: /path/on/host/system/:/path/on/container

For example, /data/smartscope/:/mnt/data/ will bound the content of /data/smartscope/ to the /mnt/data/ location inside the container.

Note

When modifying the the docker-compose.yml file, only the path before the colon “:” should be modified.

Here is the list of volumes for each services:

smartscope

Required mounts

  • Main data location:

    This is where all the images and metadata for smartscope will be saved. Binds to /mnt/data/` in the container.

  • Microscope data locations:

    For each microscope to connect to the smartscope instance, one volume must be created. This is where SerialEM will save the files so it must be a location that is available to both SerialEM and the SmartScope computer. You may choose the location name within the container in /mnt/.

    Example
    # Let's say we created a Smartscope folder in our serialEM computer
# which is mounted in /mnt/microscope_glacios/ on our linux system.
# We may want to mount it to /mnt/glacios/ in the smartscope container.
smartscope:
  volumes:
    - /mnt/microscope_glacios/Smartscope/:/mnt/glacios/

Optional mounts

  • Log files locations:

    It is recommended to bind a directory to /opt/logs/ to keep a record of the logs in case of errors and crashes.

  • Long term data location:

    If the plan is to offload the main data location to another storage location, you can bind that location to /mnt/longterm/ and set USE_LONGTERMSTORAGE: 'True' in the enviroment section. This will allow smartscope to still load and display the images in the WebUI after the data has been moved.

  • Test files:

    The test files are required to run smartscope in fake-scope mode. Mount the downloaded test files to /mnt/testfiles/.

  • SmartScope source code:

    Although the image contains the SmartScope source code already. Most small updates and hotfixes can be pulled without having to re-build the entire image. Mounting the downloaded repository to the container serves that purpose. Mount to /opt/smartscope/ and it will override the default version.

  • AI models and other template files:

    That is the directory downloaded in the installation procedure which contains the current AI models for the squares and holes detectors and some other template files. It should be bound to /opt/Template_files/

db

Required mounts

  • Database location:

    This is where all the database data will be saved. Needs to be mounted to /var/lib/mysql/

nginx

Required mounts

  • Main data location:

    This is the same as the Main data location in the smartscope volumes

  • Static files location:

    Doesn’t need change To be able to properly render the webpage javascript and css content. It mounts the SmartScope repository’s static directory as such /opt/smartscope/static/

  • Nginx config file:

    Needs change to use SSL encryption Default: ./config/docker/templates_noSSL/:/etc/nginx/templates/

    To enable SSL in your server:

    Replace by: ./config/docker/templates_SSL/:/etc/nginx/templates/ Add certificate and private key to the volumes:

    /path/to/certs/cert.pem:/opt/certs/smartscope.crt /path/to/certs/privkey.pem:/opt/certs/smartscope.key

Optional mounts

  • Long term data location:

    If specified previously, this is the same as the Long term data location in the smartscope volumes

environment

Note

This is another section where some values may be changed

List of environment variables to pass to the container. Click here for details about the environment variables.

depends_on

Other containers that need to run prior to starting this one

networks

A private network allocation for the communication between the process. Needs a network section in the root of the yaml file as well.

command

If a specific command needs to run when the container is started

deploy

Used in the smartscope` service to pass the nvidia GPU to to the container.

Note

The deploy section is optional as smartscope can also run on cpu only.

To enable add:

Example
smartscope:
  deploy:
    resources:
      reservations:
        devices:
          - driver: nvidia
            count: 1
            capabilities: [ gpu ]

ports

A list of ports that are forwarded from the container to the host system. Currently used in nginx to pass the webserver port 80 to 48000 on the host. The synthax goes as follows: HostPort:ContainerPort and, as for volumes, only the host side should be changed.

In the nginx container section, if you use ssl, use - 443:443 instead of the default - 48000:80