Project

General

Profile

Actions

Feature #1961

closed

Installation via Docker

Added by Bernhard Koschiček-Krombholz over 1 year ago. Updated about 1 year ago.

Status:
Closed
Priority:
Normal
Category:
Administration
Target version:
Start date:
2023-02-08
Estimated time:

Description

It is now possible to install OpenAtlas via Docker, see docker-compose.yml and /install/Dockerfile.
We tested it, but it is still experimental and feedback is as always welcome.

Actions #1

Updated by Alexander Watzinger over 1 year ago

  • Status changed from Closed to In Progress
My bad for deciding yesterday to already close it. I opened it again because:
  • Making it work without errors, maybe let someone else test it too
  • Document in INSTALL.md, we may have to rewrite the top part too or at least mention that we don't advise (for now) to use docker installation for production servers
Actions #2

Updated by Enric Rodellas over 1 year ago

I got to install and run. My environment:
  • Windows 10
  • WSL 2 activated
  • Docker Desktop (from Microsoft) installed
  • PyCharm for get some debug
My command:
  • cd to repository of openatlas
  • docker compose up -d

With some trouble, but not not attributable to developers. Caused by git, that convert LF to CRLF, and that is cause that scripts like SQLs and entrypoint.sh are not well interpreted inside linux docker.

I need to:
  • repeat all proces and trust
  • try export and import process
  • try upgrade process

I would like decide directory or place of installation. In my context (git in Windows) has intalled postgres database in 'data' directory inside repo of git

Take in acoount that I'm not an expert in dockers, and I understand the concept, but I'm not a master of dockers

Actions #3

Updated by Enric Rodellas over 1 year ago

Agghh !!

I wanted to be sure that I could reproduce setup.

I have stopped services, deleted db created before. But no success.
When I try to repeat:

C:\Users\rodellas\GITHUBenricapbes\OpenAtlas> docker compose up -d

I got:

/var/www/openatlas /

2023-02-12T20:11:33.637401000Z psql: error: could not connect to server: Connection refused
2023-02-12T20:11:33.637430600Z     Is the server running on host "postgres" (172.19.0.2) and accepting
2023-02-12T20:11:33.637436900Z     TCP/IP connections on port 5432?
Actions #4

Updated by Enric Rodellas over 1 year ago

If after previous execution of 'docker compose' I add

        - OVERWRITE_DATABASE=TRUE

in docker-compose.yaml in environment of services of openatlas and rexecute service openatlas.

After this web and db runs perfectly

Actions #5

Updated by Bernhard Koschiček-Krombholz over 1 year ago

  • Description updated (diff)

I was on holiday and couldn't response in time, sorry.
I'm very happy, that you could make it run so far. I will write an installation notice in this week.

For your info, the OVERWRITE_DATABASE variable is there for the first run to install the database on the postgis container. There is a script (install/check_database.sh), which looks if there is a database running and everything is set up. If a database is running, but there is no OpenAtlas database installed and OVERWRITE_DATABASE is TRUE, it will install the basic database.
Keep in mind, that everything is in an experimental state, and we didn't test it enough to use it as a production service. Also, as you noticed with the SQL Backup, not everything is thought through right now.

And big thanks for your effort to set it up, get it running and your important feedback!

Actions #6

Updated by Bernhard Koschiček-Krombholz over 1 year ago

Enric Rodellas wrote:

I would like decide directory or place of installation. In my context (git in Windows) has intalled postgres database in 'data' directory inside repo of git

I found no way to do this via variables, but you can play around with the postgis volumes in docer-compose.yml:

volumes:
   - ./data/db:/var/lib/postgresql/data

This ./data/db part is the path where the database is stored, just add an absolute path, where you want it to install.

Actions #7

Updated by Bernhard Koschiček-Krombholz over 1 year ago

  • Description updated (diff)
Actions #8

Updated by Bernhard Koschiček-Krombholz over 1 year ago

  • Description updated (diff)

Updated composer file and install.md.
At the ./install.md is a new Section at the bottom for Docker. There is also a way to install OpenAtlas with an existing dump (e.g. if you exported another database with "Export SQL").
But keep in mind, our Docker installation is NOT tested and proven for productive services.

Actions #9

Updated by Bernhard Koschiček-Krombholz over 1 year ago

  • Status changed from In Progress to Closed
Actions #10

Updated by Bernhard Koschiček-Krombholz over 1 year ago

  • Target version changed from 7.12.0 to 7.11.0
Actions #11

Updated by Enric Rodellas over 1 year ago

  • Target version changed from 7.11.0 to 7.12.0

Bernhard,
When I execute

docker compose up --detach

openatlas docker is stopped and I see log (Docker Desktop):

2023-02-16 21:42:07 /var/www/openatlas /
2023-02-16 21:42:08 Database does not exist
2023-02-16 21:42:08 Install necessary postgres extensions
2023-02-16 21:42:08 psql: error: could not connect to server: Connection refused
2023-02-16 21:42:08     Is the server running on host "postgres" (172.22.0.2) and accepting
2023-02-16 21:42:08     TCP/IP connections on port 5432?
2023-02-16 21:42:08 psql: error: could not connect to server: Connection refused
2023-02-16 21:42:08     Is the server running on host "postgres" (172.22.0.2) and accepting
2023-02-16 21:42:08     TCP/IP connections on port 5432?

My environment: Windows 10, WSL2, Docker Desktop, Pycharm, Powershell

If I rexecute, I got

2023-02-16 21:42:07 /var/www/openatlas /
2023-02-16 21:42:08 Database does not exist
2023-02-16 21:42:08 Install necessary postgres extensions
2023-02-16 21:42:08 psql: error: could not connect to server: Connection refused
2023-02-16 21:42:08     Is the server running on host "postgres" (172.22.0.2) and accepting
2023-02-16 21:42:08     TCP/IP connections on port 5432?
2023-02-16 21:42:08 psql: error: could not connect to server: Connection refused
2023-02-16 21:42:08     Is the server running on host "postgres" (172.22.0.2) and accepting
2023-02-16 21:42:08     TCP/IP connections on port 5432?
2023-02-16 21:47:42 ERROR:  relation "web.settings" does not exist
2023-02-16 21:47:42 LINE 1: SELECT 1 FROM openatlas.web.settings
2023-02-16 21:47:42                       ^
2023-02-16 21:47:42 check_database.sh: line 24: OVERWRITE_DATABASE: unbound variable
2023-02-16 21:49:52 ERROR:  relation "web.settings" does not exist
2023-02-16 21:49:52 LINE 1: SELECT 1 FROM openatlas.web.settings
2023-02-16 21:49:52                       ^
2023-02-16 21:49:52 check_database.sh: line 24: OVERWRITE_DATABASE: unbound variable
2023-02-16 21:47:42 /var/www/openatlas /
2023-02-16 21:47:42 Database already exists
2023-02-16 21:49:52 /var/www/openatlas /
2023-02-16 21:49:52 Database already exists

Actions #12

Updated by Bernhard Koschiček-Krombholz over 1 year ago

  • Target version changed from 7.12.0 to 7.11.0

Hi Enric,

Did you delete or rebuild your old image? In the newest develop version I deleted the variable OVERWRITE_DATABASE, that is why I think you are starting an old image.

Frist try to remove your old openatlas image. With

docker images

you see all your images and then delete the openatlas-openatlas image with
docker rmi <your-image-id>

If you have installed the necessary PyCharm Plugins (Docker), you can do it on the PyCharm GUI.
If this not work, you can do a rebuild.

docker compose up openatlas --detach --build

This should only rebuild openatlas.

Additionally, both containers are not lightweight, they need some time (like 5–10 seconds) to start. So it is normal, that the openatlas container need a restart because the postgis container need some time to get running.

Hope this helps.

Actions #13

Updated by Bernhard Koschiček-Krombholz over 1 year ago

Andi and I rewrote the script. Please pull the newest develop version and run

docker compose up --detach --build

Everything should be rebuilt, and your database should not be overwritten.

Actions #14

Updated by Enric Rodellas over 1 year ago

Bernard,

I don't get recovered my last dump following instructions uncommenting

      # Uncomment to import SQL dump
      - ./files/export/dump.sql:/docker-entrypoint-initdb.d/dump.sql

and assuring (berfore executing) that ./files/export/dump.sql is last dump

Actions #15

Updated by Enric Rodellas over 1 year ago

I think would be very useful to add some external port to acces to database. Specially for adminsitrative tasks.

I test:

    ports:
      - "25432:5432" 

inside postgres section

Actions #16

Updated by Bernhard Koschiček-Krombholz over 1 year ago

Enric Rodellas wrote:

I don't get recovered my last dump following instructions uncommenting

With the last commit, I forgot to write a comment, that the installation scripts above has to be commented out. And you have to delete the /data/db folder or change the path of the variable:

./data/db:/var/lib/postgresql/data

Because, the scripts won't run if a database exists.

So to import a dump you have to do following steps:

  1. Delete or rename database path (e.g. ./data/db_dump:/var/lib/postgresql/data)
  2. Comment out installation scripts
    #      - ./install/extensions.sql:/docker-entrypoint-initdb.d/dump.sql
    #      - ./install/1_structure.sql:/docker-entrypoint-initdb.d/1_structure.sql
    #      - ./install/2_data_model.sql:/docker-entrypoint-initdb.d/2_data_model.sql
    #      - ./install/3_data_web.sql:/docker-entrypoint-initdb.d/3_data_web.sql
    #      - ./install/4_data_type.sql:/docker-entrypoint-initdb.d/4_data_type.sql
    
  3. Uncomment path to dump script
    ./files/export/dump.sql:/docker-entrypoint-initdb.d/dump.sql
    
  4. run docker compose up --detach --build

If the dump is quite large, it will take some time, and it may be, that you have to restart the postgis container.

Enric Rodellas wrote:

I think would be very useful to add some external port to acces to database. Specially for adminsitrative tasks.

I test:
[...]

inside postgres section

Yes and no. For debugging reason, yes. But I don't know you this will interfere with other installation processes (e.g. kubernetes). I wrote it in the composer file and commented it.

In my point of view, the composer file is the file, where the user can and should add personal installation options. This is the place to customize your installation and therefore, I don't want to fill it up with things, that are not necessarily needed. It will soon get bloated.

Actions #17

Updated by Enric Rodellas about 1 year ago

Bernhard,
Thank you very much for your notes.
I have carefully followed your instructions, ( but I can't initialize the database with the Dump file. In the process a Postgres BD is initialized, but does not create any of the openatlas tables and finally launch an error.

I comment and uncomment:

    volumes:
      - ./data/db:/var/lib/postgresql/data
#     Installation of an empty database
      #- ./install/extensions.sql:/docker-entrypoint-initdb.d/dump.sql
      #- ./install/1_structure.sql:/docker-entrypoint-initdb.d/1_structure.sql
      #- ./install/2_data_model.sql:/docker-entrypoint-initdb.d/2_data_model.sql
      #- ./install/3_data_web.sql:/docker-entrypoint-initdb.d/3_data_web.sql
      #- ./install/4_data_type.sql:/docker-entrypoint-initdb.d/4_data_type.sql
#      Import dump (comment install section above out)
      - ./files/export/dump.sql:/docker-entrypoint-initdb.d/dump.sql

And this is the log in openatlas docker:

2023-02-22 18:56:04 Traceback (most recent call last):
2023-02-22 18:56:04   File "/var/www/openatlas/install/upgrade/database_upgrade.py", line 50, in <module>
2023-02-22 18:56:04     settings = Settings.get_settings(cursor)
2023-02-22 18:56:04   File "/var/www/openatlas/openatlas/database/settings.py", line 14, in get_settings
2023-02-22 18:56:04     cursor.execute("SELECT name, value FROM web.settings;")
2023-02-22 18:56:04   File "/usr/lib/python3/dist-packages/psycopg2/extras.py", line 146, in execute
2023-02-22 18:56:04     return super(DictCursor, self).execute(query, vars)
2023-02-22 18:56:04 psycopg2.errors.UndefinedTable: relation "web.settings" does not exist
2023-02-22 18:56:04 LINE 1: SELECT name, value FROM web.settings;
2023-02-22 18:56:04                                 ^

Actions #18

Updated by Bernhard Koschiček-Krombholz about 1 year ago

Your dump file is named dump.sql and is in the folder ./files/export/ (this is in the OpenAtlas directory)?
The only way I get this error message is, that my dump.sql file doesn't exist.

So to import an SQL dump, please do the following things:

  1. docker compose down
  2. Delete or rename (if you want to keep it) the database folder. In your case ./data/db/
  3. Move the dumped SQL file to ./files/export/.
  4. Either rename the .sql file to dump.sql or change this path to your filename - ./files/export/ dump.sql :/docker-entrypoint-initdb.d/dump.sql
  5. docker compose up -d

If everything went well, the services should start automatically. We implemented a health check (in the develop branch, not main), so the openatlas-openatlas-1 container will not start, if the openatlas-postgres-1 container is unhealthy (not ready)

If the error still occurs, I can only ask for you dump file, so I can check if everything is ok with it.

Actions

Also available in: Atom PDF