More changes
This commit is contained in:
Brett
parent
77767f4e05
commit
d1fe6d1e99
66
OLD.README.txt
Normal file
66
OLD.README.txt
Normal file
@ -0,0 +1,66 @@
|
|||||||
|
### Please note that this re-purposed CKAN Docker repo is a WORK IN PROGRESS ###
|
||||||
|
### It should not be used to install CKAN via Docker until this page is updated ###
|
||||||
|
|
||||||
|
-------------------------------------
|
||||||
|
(From Adria)
|
||||||
|
Be limited in scope, and act as a base that users can extend to their own needs
|
||||||
|
Be opinionated, and provide one way to do things
|
||||||
|
Be automatically tested
|
||||||
|
-------------------------------------
|
||||||
|
|
||||||
|
should we have an nginx container included? maybe just instructions on how to include one
|
||||||
|
|
||||||
|
ARG should be used for sensitive variables so as to keep the values of these variables
|
||||||
|
out of the build and hence will not show with an "inspect image" command
|
||||||
|
|
||||||
|
Do we pre-build a CKAN image and use that (and extend) as the base image OR
|
||||||
|
just build it from scratch and while saving to storage what we need to
|
||||||
|
|
||||||
|
To use local storage for the ckan.ini file and the CKAN src code, do the following:
|
||||||
|
|
||||||
|
# docker-compose up -d --build
|
||||||
|
# mkdir local
|
||||||
|
# docker cp ckan:/srv/app/ckan.ini ./local/ckan.ini
|
||||||
|
# docker cp ckan:/srv/app/src ./local/src
|
||||||
|
Stop/Remove ckan container
|
||||||
|
Start ckan container as follows
|
||||||
|
Use a bind mount for the config file (ckan.ini)
|
||||||
|
# docker run -p 0.0.0.0:5000:5000 --net ckan-docker_default --hostname ckan --name ckan \
|
||||||
|
--env-file=./environ --mount type=bind,source=$(pwd)/local/ckan.ini,target=/srv/app/ckan.ini \
|
||||||
|
--mount type=bind,source=$(pwd)/local/src,target=/srv/app/src \
|
||||||
|
-d ckan-docker_ckan
|
||||||
|
|
||||||
|
Maybe include a script to replace docker-compose if required
|
||||||
|
|
||||||
|
Dev Mode (OKFN)
|
||||||
|
|
||||||
|
The differences between Docker dev abd Docker base is as the following:
|
||||||
|
|
||||||
|
docker-compose.dev.yml
|
||||||
|
solr: explicitly puts in ports (8983:8983)
|
||||||
|
db: Doesn't pass in environment and arg values
|
||||||
|
ckan: has extra volume bind mount (./src:/srv/app/src_extensions)
|
||||||
|
|
||||||
|
Dockerfile.dev
|
||||||
|
Takes the base image and
|
||||||
|
Adds a new directory (SRC_EXTENSIONS_DIR=/srv/app/src_extensions)
|
||||||
|
installs libffi-dev
|
||||||
|
installs dev-requirements.txt
|
||||||
|
runs different start script (start_ckan_development.sh) which installs any extension located in SRC_EXTENSIONS_DIR
|
||||||
|
runs a typical Dev install
|
||||||
|
pip install -r pip-requirements.txt
|
||||||
|
pip install -r requirements.txt
|
||||||
|
pip install -r dev-requirements.txt
|
||||||
|
python3 setup.py develop
|
||||||
|
ckan config-tool test.ini
|
||||||
|
ckan config-tool $CKAN_INI -s DEFAULT "debug = true"
|
||||||
|
ckan config-tool $CKAN_INI "ckan.plugins = $CKAN__PLUGINS"
|
||||||
|
ckan config-tool $SRC_DIR/ckan/test-core.ini \
|
||||||
|
"sqlalchemy.url = $TEST_CKAN_SQLALCHEMY_URL" \
|
||||||
|
"ckan.datastore.write_url = $TEST_CKAN_DATASTORE_WRITE_URL" \
|
||||||
|
"ckan.datastore.read_url = $TEST_CKAN_DATASTORE_READ_URL" \
|
||||||
|
"solr_url = $TEST_CKAN_SOLR_URL" \
|
||||||
|
"ckan.redis.url = $TEST_CKAN_REDIS_URL"
|
||||||
|
|
||||||
|
All other steps are similar to the base Dockerfile
|
||||||
|
|
||||||
75
README.txt
75
README.txt
@ -1,66 +1,13 @@
|
|||||||
### Please note that this re-purposed CKAN Docker repo is a WORK IN PROGRESS ###
|
# July 7 2022
|
||||||
### It should not be used to install CKAN via Docker until this page is updated ###
|
# This repo will be actively worked on from now.
|
||||||
|
# This file will be my ToDo list of things to take care of
|
||||||
|
|
||||||
-------------------------------------
|
- have the ckan-docker repo for the "get CKAN up on Docker" repo
|
||||||
(From Adria)
|
- create a new repo (maybe 'ckan-docker-dev') for a Dev orientated approach
|
||||||
Be limited in scope, and act as a base that users can extend to their own needs
|
- get rid of the "patches/" stuff like in OKFN
|
||||||
Be opinionated, and provide one way to do things
|
- Get rid of 2.8 and 2.7. We should only support Docker on CKAN 2.9 (or latest stable which will soon be 2.10)
|
||||||
Be automatically tested
|
- Get rid of nginx, just provide instructions on how to add it
|
||||||
-------------------------------------
|
- Use pre-built CKAN base images, eventually move them over to an official CKAN DockerHub repo with the correct tags
|
||||||
|
- No extensions but provide instructions on how to add to a Dockerfile
|
||||||
should we have an nginx container included? maybe just instructions on how to include one
|
- no hard-coded container names, use environment variables
|
||||||
|
|
||||||
ARG should be used for sensitive variables so as to keep the values of these variables
|
|
||||||
out of the build and hence will not show with an "inspect image" command
|
|
||||||
|
|
||||||
Do we pre-build a CKAN image and use that (and extend) as the base image OR
|
|
||||||
just build it from scratch and while saving to storage what we need to
|
|
||||||
|
|
||||||
To use local storage for the ckan.ini file and the CKAN src code, do the following:
|
|
||||||
|
|
||||||
# docker-compose up -d --build
|
|
||||||
# mkdir local
|
|
||||||
# docker cp ckan:/srv/app/ckan.ini ./local/ckan.ini
|
|
||||||
# docker cp ckan:/srv/app/src ./local/src
|
|
||||||
Stop/Remove ckan container
|
|
||||||
Start ckan container as follows
|
|
||||||
Use a bind mount for the config file (ckan.ini)
|
|
||||||
# docker run -p 0.0.0.0:5000:5000 --net ckan-docker_default --hostname ckan --name ckan \
|
|
||||||
--env-file=./environ --mount type=bind,source=$(pwd)/local/ckan.ini,target=/srv/app/ckan.ini \
|
|
||||||
--mount type=bind,source=$(pwd)/local/src,target=/srv/app/src \
|
|
||||||
-d ckan-docker_ckan
|
|
||||||
|
|
||||||
Maybe include a script to replace docker-compose if required
|
|
||||||
|
|
||||||
Dev Mode (OKFN)
|
|
||||||
|
|
||||||
The differences between Docker dev abd Docker base is as the following:
|
|
||||||
|
|
||||||
docker-compose.dev.yml
|
|
||||||
solr: explicitly puts in ports (8983:8983)
|
|
||||||
db: Doesn't pass in environment and arg values
|
|
||||||
ckan: has extra volume bind mount (./src:/srv/app/src_extensions)
|
|
||||||
|
|
||||||
Dockerfile.dev
|
|
||||||
Takes the base image and
|
|
||||||
Adds a new directory (SRC_EXTENSIONS_DIR=/srv/app/src_extensions)
|
|
||||||
installs libffi-dev
|
|
||||||
installs dev-requirements.txt
|
|
||||||
runs different start script (start_ckan_development.sh) which installs any extension located in SRC_EXTENSIONS_DIR
|
|
||||||
runs a typical Dev install
|
|
||||||
pip install -r pip-requirements.txt
|
|
||||||
pip install -r requirements.txt
|
|
||||||
pip install -r dev-requirements.txt
|
|
||||||
python3 setup.py develop
|
|
||||||
ckan config-tool test.ini
|
|
||||||
ckan config-tool $CKAN_INI -s DEFAULT "debug = true"
|
|
||||||
ckan config-tool $CKAN_INI "ckan.plugins = $CKAN__PLUGINS"
|
|
||||||
ckan config-tool $SRC_DIR/ckan/test-core.ini \
|
|
||||||
"sqlalchemy.url = $TEST_CKAN_SQLALCHEMY_URL" \
|
|
||||||
"ckan.datastore.write_url = $TEST_CKAN_DATASTORE_WRITE_URL" \
|
|
||||||
"ckan.datastore.read_url = $TEST_CKAN_DATASTORE_READ_URL" \
|
|
||||||
"solr_url = $TEST_CKAN_SOLR_URL" \
|
|
||||||
"ckan.redis.url = $TEST_CKAN_REDIS_URL"
|
|
||||||
|
|
||||||
All other steps are similar to the base Dockerfile
|
|
||||||
|
|
||||||
|
|||||||
@ -1,113 +0,0 @@
|
|||||||
FROM alpine:3.7
|
|
||||||
|
|
||||||
# Internal environment variables
|
|
||||||
ENV APP_DIR=/srv/app
|
|
||||||
ENV SRC_DIR=/srv/app/src
|
|
||||||
ENV CKAN_INI=${APP_DIR}/ckan.ini
|
|
||||||
ENV PIP_SRC=${SRC_DIR}
|
|
||||||
ENV CKAN_STORAGE_PATH=/var/lib/ckan
|
|
||||||
ENV GIT_URL=https://github.com/ckan/ckan.git
|
|
||||||
# CKAN version to build
|
|
||||||
ENV GIT_BRANCH=ckan-2.9.3
|
|
||||||
# Customize these on the .env file if needed
|
|
||||||
ENV CKAN_SITE_URL=http://localhost:5000
|
|
||||||
ENV CKAN__PLUGINS image_view text_view recline_view datastore datapusher envvars
|
|
||||||
|
|
||||||
WORKDIR ${APP_DIR}
|
|
||||||
|
|
||||||
# Install necessary packages to run CKAN
|
|
||||||
RUN apk add --no-cache tzdata \
|
|
||||||
git \
|
|
||||||
gettext \
|
|
||||||
postgresql-client \
|
|
||||||
python3 \
|
|
||||||
apache2-utils \
|
|
||||||
libxml2 \
|
|
||||||
libxslt \
|
|
||||||
musl-dev \
|
|
||||||
uwsgi-http \
|
|
||||||
uwsgi-corerouter \
|
|
||||||
uwsgi-python3 \
|
|
||||||
py3-gevent \
|
|
||||||
uwsgi-gevent \
|
|
||||||
libmagic \
|
|
||||||
curl \
|
|
||||||
sudo && \
|
|
||||||
# Packages to build CKAN requirements and plugins
|
|
||||||
apk add --no-cache --virtual .build-deps \
|
|
||||||
postgresql-dev \
|
|
||||||
gcc \
|
|
||||||
make \
|
|
||||||
g++ \
|
|
||||||
autoconf \
|
|
||||||
automake \
|
|
||||||
libtool \
|
|
||||||
python3-dev \
|
|
||||||
py3-virtualenv \
|
|
||||||
libxml2-dev \
|
|
||||||
libxslt-dev \
|
|
||||||
linux-headers && \
|
|
||||||
# Create SRC_DIR
|
|
||||||
mkdir -p ${SRC_DIR}
|
|
||||||
|
|
||||||
# Install pip
|
|
||||||
RUN curl -o ${SRC_DIR}/get-pip.py https://bootstrap.pypa.io/get-pip.py && \
|
|
||||||
python3 ${SRC_DIR}/get-pip.py
|
|
||||||
|
|
||||||
# Set up Python3 virtual environment
|
|
||||||
RUN cd ${APP_DIR} && \
|
|
||||||
python3 -m venv ${APP_DIR} && \
|
|
||||||
source ${APP_DIR}/bin/activate
|
|
||||||
|
|
||||||
# Virtual environment binaries/scripts to be used first
|
|
||||||
ENV PATH=${APP_DIR}/bin:${PATH}
|
|
||||||
|
|
||||||
# Install CKAN, uwsgi plus extensions
|
|
||||||
RUN pip3 install -e git+${GIT_URL}@${GIT_BRANCH}#egg=ckan && \
|
|
||||||
pip3 install uwsgi && \
|
|
||||||
cd ${SRC_DIR}/ckan && \
|
|
||||||
cp who.ini ${APP_DIR} && \
|
|
||||||
pip install --no-binary :all: -r requirements.txt && \
|
|
||||||
# Install CKAN envvars to support loading config from environment variables
|
|
||||||
pip3 install -e git+https://github.com/okfn/ckanext-envvars.git#egg=ckanext-envvars && \
|
|
||||||
# Install CKAN extensions
|
|
||||||
pip3 install -e 'git+https://github.com/DataShades/ckanext-xloader@py3#egg=ckanext-xloader' && \
|
|
||||||
pip3 install -r $CKAN_VENV/src/ckanext-xloader/requirements.txt && \
|
|
||||||
pip3 install -U requests[security] && \
|
|
||||||
pip3 install -e 'git+https://github.com/DataShades/ckanext-harvest.git@py3#egg=ckanext-harvest' && \
|
|
||||||
pip3 install -r $CKAN_VENV/src/ckanext-harvest/pip-requirements.txt && \
|
|
||||||
pip3 install -e 'git+https://github.com/DataShades/ckanext-syndicate@py3#egg=ckanext-syndicate' && \
|
|
||||||
pip3 install -r $CKAN_VENV/src/ckanext-syndicate/requirements.txt && \
|
|
||||||
pip3 install -e 'git+https://github.com/ckan/ckanext-scheming.git@master#egg=ckanext-scheming' && \
|
|
||||||
pip3 install -r $CKAN_VENV/src/ckanext-scheming/requirements.txt
|
|
||||||
|
|
||||||
# Create and update CKAN config
|
|
||||||
RUN ckan generate config ${CKAN_INI}
|
|
||||||
|
|
||||||
# Install and configure supervisor
|
|
||||||
RUN pip3 install supervisor && \
|
|
||||||
mkdir /etc/supervisord.d
|
|
||||||
|
|
||||||
# Copy all setup files
|
|
||||||
COPY setup ${APP_DIR}
|
|
||||||
COPY setup/supervisor.worker.conf /etc/supervisord.d/worker.conf
|
|
||||||
COPY setup/supervisord.conf /etc/supervisord.conf
|
|
||||||
|
|
||||||
# Create a local user and group to run the app
|
|
||||||
RUN addgroup -g 92 -S ckan && \
|
|
||||||
adduser -u 92 -h /srv/app -H -D -S -G ckan ckan
|
|
||||||
|
|
||||||
# Create local storage folder
|
|
||||||
RUN mkdir -p $CKAN_STORAGE_PATH && \
|
|
||||||
chown -R ckan:ckan $CKAN_STORAGE_PATH
|
|
||||||
|
|
||||||
# Create entrypoint directory for children image scripts
|
|
||||||
ONBUILD RUN mkdir /docker-entrypoint.d
|
|
||||||
|
|
||||||
RUN chown ckan -R /srv/app
|
|
||||||
|
|
||||||
EXPOSE 5000
|
|
||||||
|
|
||||||
HEALTHCHECK --interval=10s --timeout=5s --retries=5 CMD curl --fail http://localhost:5000/api/3/action/status_show || exit 1
|
|
||||||
|
|
||||||
CMD ["/srv/app/start_ckan.sh"]
|
|
||||||
@ -1,15 +0,0 @@
|
|||||||
[uwsgi]
|
|
||||||
http-socket = :5000
|
|
||||||
uid = ckan
|
|
||||||
guid = ckan
|
|
||||||
plugins = python3
|
|
||||||
wsgi-file = /srv/app/wsgi.py
|
|
||||||
virtualenv = /srv/app
|
|
||||||
module = wsgi:application
|
|
||||||
master = true
|
|
||||||
processes = 5
|
|
||||||
pidfile = /tmp/%n.pid
|
|
||||||
harakiri = 50
|
|
||||||
max-requests = 5000
|
|
||||||
vacuum = true
|
|
||||||
callable = application
|
|
||||||
@ -1,194 +0,0 @@
|
|||||||
import os
|
|
||||||
import sys
|
|
||||||
import subprocess
|
|
||||||
import psycopg2
|
|
||||||
import urllib3
|
|
||||||
import time
|
|
||||||
import re
|
|
||||||
|
|
||||||
ckan_ini = os.environ.get('CKAN_INI', '/srv/app/ckan.ini')
|
|
||||||
|
|
||||||
RETRY = 5
|
|
||||||
|
|
||||||
def update_plugins():
|
|
||||||
|
|
||||||
plugins = os.environ.get('CKAN__PLUGINS', '')
|
|
||||||
print('[prerun] Setting the following plugins in {}:'.format(ckan_ini))
|
|
||||||
print(plugins)
|
|
||||||
cmd = ['ckan', 'config-tool', ckan_ini,
|
|
||||||
'ckan.plugins = {}'.format(plugins)]
|
|
||||||
subprocess.check_output(cmd, stderr=subprocess.STDOUT)
|
|
||||||
print('[prerun] Plugins set.')
|
|
||||||
|
|
||||||
|
|
||||||
def check_main_db_connection(retry=None):
|
|
||||||
|
|
||||||
conn_str = os.environ.get('CKAN_SQLALCHEMY_URL')
|
|
||||||
if not conn_str:
|
|
||||||
print('[prerun] CKAN_SQLALCHEMY_URL not defined, not checking db')
|
|
||||||
return check_db_connection(conn_str, retry)
|
|
||||||
|
|
||||||
|
|
||||||
def check_datastore_db_connection(retry=None):
|
|
||||||
|
|
||||||
conn_str = os.environ.get('CKAN_DATASTORE_WRITE_URL')
|
|
||||||
if not conn_str:
|
|
||||||
print('[prerun] CKAN_DATASTORE_WRITE_URL not defined, not checking db')
|
|
||||||
return check_db_connection(conn_str, retry)
|
|
||||||
|
|
||||||
|
|
||||||
def check_db_connection(conn_str, retry=None):
|
|
||||||
|
|
||||||
if retry is None:
|
|
||||||
retry = RETRY
|
|
||||||
elif retry == 0:
|
|
||||||
print('[prerun] Giving up after 5 tries...')
|
|
||||||
sys.exit(1)
|
|
||||||
|
|
||||||
try:
|
|
||||||
connection = psycopg2.connect(conn_str)
|
|
||||||
|
|
||||||
except psycopg2.Error as e:
|
|
||||||
print(str(e))
|
|
||||||
print('[prerun] Unable to connect to the database, waiting...')
|
|
||||||
time.sleep(10)
|
|
||||||
check_db_connection(conn_str, retry=retry - 1)
|
|
||||||
else:
|
|
||||||
connection.close()
|
|
||||||
|
|
||||||
|
|
||||||
def check_solr_connection(retry=None):
|
|
||||||
|
|
||||||
if retry is None:
|
|
||||||
retry = RETRY
|
|
||||||
elif retry == 0:
|
|
||||||
print('[prerun] Giving up after 5 tries...')
|
|
||||||
sys.exit(1)
|
|
||||||
|
|
||||||
url = os.environ.get('CKAN_SOLR_URL', '')
|
|
||||||
search_url = '{url}/select/?q=*&wt=json'.format(url=url)
|
|
||||||
http = urllib3.PoolManager()
|
|
||||||
try:
|
|
||||||
r = http.request('GET', search_url)
|
|
||||||
except urllib3.exceptions.ConnectionError as e:
|
|
||||||
print(str(e))
|
|
||||||
print('[prerun] Unable to connect to solr, waiting...')
|
|
||||||
time.sleep(10)
|
|
||||||
check_solr_connection(retry=retry - 1)
|
|
||||||
else:
|
|
||||||
print('[prerun] Connection Status from SOLR is ', (r.status))
|
|
||||||
|
|
||||||
def init_db():
|
|
||||||
|
|
||||||
db_command = ['ckan', '-c', ckan_ini,
|
|
||||||
'db', 'init']
|
|
||||||
print('[prerun] Initializing or upgrading db - start')
|
|
||||||
try:
|
|
||||||
subprocess.check_output(db_command, stderr=subprocess.STDOUT)
|
|
||||||
print('[prerun] Initializing or upgrading db - end')
|
|
||||||
except subprocess.CalledProcessError as e:
|
|
||||||
if 'OperationalError' in e.output:
|
|
||||||
print(e.output)
|
|
||||||
print('[prerun] Database not ready, waiting a bit before exit...')
|
|
||||||
time.sleep(5)
|
|
||||||
sys.exit(1)
|
|
||||||
else:
|
|
||||||
print(e.output)
|
|
||||||
raise e
|
|
||||||
|
|
||||||
|
|
||||||
def init_datastore_db():
|
|
||||||
|
|
||||||
conn_str = os.environ.get('CKAN_DATASTORE_WRITE_URL')
|
|
||||||
if not conn_str:
|
|
||||||
print('[prerun] Skipping datastore initialization')
|
|
||||||
return
|
|
||||||
|
|
||||||
datastore_perms_command = ['ckan', '-c', ckan_ini,
|
|
||||||
'datastore', 'set-permissions']
|
|
||||||
|
|
||||||
connection = psycopg2.connect(conn_str)
|
|
||||||
cursor = connection.cursor()
|
|
||||||
|
|
||||||
print('[prerun] Initializing datastore db - start')
|
|
||||||
try:
|
|
||||||
datastore_perms = subprocess.Popen(
|
|
||||||
datastore_perms_command,
|
|
||||||
stdout=subprocess.PIPE)
|
|
||||||
|
|
||||||
perms_sql = datastore_perms.stdout.read().decode('utf-8')
|
|
||||||
# Remove internal pg command as psycopg2 does not like it
|
|
||||||
perms_sql = re.sub('\\\\connect \"(.*)\"', '', perms_sql)
|
|
||||||
cursor.execute(perms_sql)
|
|
||||||
for notice in connection.notices:
|
|
||||||
print(notice)
|
|
||||||
|
|
||||||
connection.commit()
|
|
||||||
|
|
||||||
print('[prerun] Initializing datastore db - end')
|
|
||||||
print(datastore_perms.stdout.read().decode('utf-8'))
|
|
||||||
except psycopg2.Error as e:
|
|
||||||
print('[prerun] Could not initialize datastore')
|
|
||||||
print(str(e))
|
|
||||||
|
|
||||||
except subprocess.CalledProcessError as e:
|
|
||||||
if 'OperationalError' in e.output:
|
|
||||||
print(e.output)
|
|
||||||
print('[prerun] Database not ready, waiting a bit before exit...')
|
|
||||||
time.sleep(5)
|
|
||||||
sys.exit(1)
|
|
||||||
else:
|
|
||||||
print(e.output)
|
|
||||||
raise e
|
|
||||||
finally:
|
|
||||||
cursor.close()
|
|
||||||
connection.close()
|
|
||||||
|
|
||||||
|
|
||||||
def create_sysadmin():
|
|
||||||
|
|
||||||
name = os.environ.get('CKAN_SYSADMIN_NAME')
|
|
||||||
password = os.environ.get('CKAN_SYSADMIN_PASSWORD')
|
|
||||||
email = os.environ.get('CKAN_SYSADMIN_EMAIL')
|
|
||||||
|
|
||||||
if name and password and email:
|
|
||||||
|
|
||||||
# Check if user exists
|
|
||||||
command = ['ckan', '-c', ckan_ini, 'user', 'show', name,]
|
|
||||||
|
|
||||||
out = subprocess.check_output(command)
|
|
||||||
if 'User:None' not in re.sub(r'\s', '', out.decode()):
|
|
||||||
print('[prerun] Sysadmin user exists, skipping creation')
|
|
||||||
return
|
|
||||||
|
|
||||||
# Create user
|
|
||||||
command = ['ckan', '-c', ckan_ini, 'user', 'add',
|
|
||||||
name,
|
|
||||||
'password=' + password,
|
|
||||||
'email=' + email]
|
|
||||||
|
|
||||||
subprocess.call(command)
|
|
||||||
print('[prerun] Created user {0}'.format(name))
|
|
||||||
|
|
||||||
# Make it sysadmin
|
|
||||||
command = ['ckan', '-c', ckan_ini, 'sysadmin', 'add',
|
|
||||||
name]
|
|
||||||
|
|
||||||
subprocess.call(command)
|
|
||||||
print('[prerun] Made user {0} a sysadmin'.format(name))
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == '__main__':
|
|
||||||
|
|
||||||
maintenance = os.environ.get('MAINTENANCE_MODE', '').lower() == 'true'
|
|
||||||
|
|
||||||
if maintenance:
|
|
||||||
print('[prerun] Maintenance mode, skipping setup...')
|
|
||||||
else:
|
|
||||||
check_main_db_connection()
|
|
||||||
init_db()
|
|
||||||
update_plugins()
|
|
||||||
check_datastore_db_connection()
|
|
||||||
init_datastore_db()
|
|
||||||
check_solr_connection()
|
|
||||||
create_sysadmin()
|
|
||||||
@ -1,46 +0,0 @@
|
|||||||
#!/bin/bash
|
|
||||||
|
|
||||||
# Run the prerun script to init CKAN and create the default admin user
|
|
||||||
sudo -u ckan -EH python3 prerun.py
|
|
||||||
|
|
||||||
# Run any startup scripts provided by images extending this one
|
|
||||||
if [[ -d "/docker-entrypoint.d" ]]
|
|
||||||
then
|
|
||||||
for f in /docker-entrypoint.d/*; do
|
|
||||||
case "$f" in
|
|
||||||
*.sh) echo "$0: Running init file $f"; . "$f" ;;
|
|
||||||
*.py) echo "$0: Running init file $f"; python "$f"; echo ;;
|
|
||||||
*) echo "$0: Ignoring $f (not an sh or py file)" ;;
|
|
||||||
esac
|
|
||||||
echo
|
|
||||||
done
|
|
||||||
fi
|
|
||||||
|
|
||||||
|
|
||||||
# Check whether http basic auth password protection is enabled and enable basicauth routing on uwsgi respecfully
|
|
||||||
if [ $? -eq 0 ]
|
|
||||||
then
|
|
||||||
if [ "$PASSWORD_PROTECT" = true ]
|
|
||||||
then
|
|
||||||
if [ "$HTPASSWD_USER" ] || [ "$HTPASSWD_PASSWORD" ]
|
|
||||||
then
|
|
||||||
# Generate htpasswd file for basicauth
|
|
||||||
htpasswd -d -b -c /srv/app/.htpasswd $HTPASSWD_USER $HTPASSWD_PASSWORD
|
|
||||||
# Start supervisord
|
|
||||||
supervisord --configuration /etc/supervisord.conf &
|
|
||||||
# Start uwsgi with basicauth
|
|
||||||
sudo -u ckan -EH uwsgi -i ckan-uwsgi.ini
|
|
||||||
else
|
|
||||||
echo "Missing HTPASSWD_USER or HTPASSWD_PASSWORD environment variables. Exiting..."
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
else
|
|
||||||
# Start supervisord
|
|
||||||
supervisord --configuration /etc/supervisord.conf &
|
|
||||||
# Start uwsgi
|
|
||||||
sudo -u ckan -EH uwsgi -i ckan-uwsgi.ini
|
|
||||||
fi
|
|
||||||
else
|
|
||||||
echo "[prerun] failed...not starting CKAN."
|
|
||||||
fi
|
|
||||||
|
|
||||||
@ -1,12 +0,0 @@
|
|||||||
[program:ckan-worker]
|
|
||||||
command=ckan -c /srv/app/ckan.ini jobs worker
|
|
||||||
priority=501
|
|
||||||
autostart=true
|
|
||||||
autorestart=true
|
|
||||||
redirect_stderr=true
|
|
||||||
stdout_logfile=/dev/stdout
|
|
||||||
stdout_logfile_maxbytes=0
|
|
||||||
stderr_logfile=/dev/stdout
|
|
||||||
stderr_logfile_maxbytes=0
|
|
||||||
user=ckan
|
|
||||||
environment=HOME="/srv/app",USER="ckan"
|
|
||||||
@ -1,23 +0,0 @@
|
|||||||
[unix_http_server]
|
|
||||||
file = /tmp/supervisor.sock
|
|
||||||
chmod = 0777
|
|
||||||
chown = nobody:nogroup
|
|
||||||
|
|
||||||
[supervisord]
|
|
||||||
logfile = /tmp/supervisord.log
|
|
||||||
logfile_maxbytes = 50MB
|
|
||||||
logfile_backups=10
|
|
||||||
loglevel = info
|
|
||||||
pidfile = /tmp/supervisord.pid
|
|
||||||
nodaemon = true
|
|
||||||
umask = 022
|
|
||||||
identifier = supervisor
|
|
||||||
|
|
||||||
[supervisorctl]
|
|
||||||
serverurl = unix:///tmp/supervisor.sock
|
|
||||||
|
|
||||||
[rpcinterface:supervisor]
|
|
||||||
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
|
|
||||||
|
|
||||||
[include]
|
|
||||||
files = /etc/supervisord.d/*.conf
|
|
||||||
@ -1,9 +0,0 @@
|
|||||||
import os
|
|
||||||
from ckan.config.middleware import make_app
|
|
||||||
from ckan.cli import CKANConfigLoader
|
|
||||||
from logging.config import fileConfig as loggingFileConfig
|
|
||||||
config_filepath = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'ckan.ini')
|
|
||||||
abspath = os.path.join(os.path.dirname(os.path.abspath(__file__)))
|
|
||||||
loggingFileConfig(config_filepath)
|
|
||||||
config = CKANConfigLoader(config_filepath).get_config()
|
|
||||||
application = make_app(config)
|
|
||||||
252
local/ckan.ini
252
local/ckan.ini
@ -1,252 +0,0 @@
|
|||||||
#
|
|
||||||
# CKAN - Pylons configuration
|
|
||||||
#
|
|
||||||
# These are some of the configuration options available for your CKAN
|
|
||||||
# instance. Check the documentation in 'doc/configuration.rst' or at the
|
|
||||||
# following URL for a description of what they do and the full list of
|
|
||||||
# available options:
|
|
||||||
#
|
|
||||||
# http://docs.ckan.org/en/latest/maintaining/configuration.html
|
|
||||||
#
|
|
||||||
# The %(here)s variable will be replaced with the parent directory of this file
|
|
||||||
#
|
|
||||||
|
|
||||||
[DEFAULT]
|
|
||||||
|
|
||||||
# WARNING: *THIS SETTING MUST BE SET TO FALSE ON A PUBLIC ENVIRONMENT*
|
|
||||||
# With debug mode enabled, a visitor to your site could execute malicious commands.
|
|
||||||
debug = false
|
|
||||||
|
|
||||||
[app:main]
|
|
||||||
use = egg:ckan
|
|
||||||
|
|
||||||
## Development settings
|
|
||||||
ckan.devserver.host = localhost
|
|
||||||
ckan.devserver.port = 5000
|
|
||||||
|
|
||||||
|
|
||||||
## Session settings
|
|
||||||
cache_dir = /tmp/%(ckan.site_id)s/
|
|
||||||
beaker.session.key = ckan
|
|
||||||
|
|
||||||
# This is the secret token that the beaker library uses to hash the cookie sent
|
|
||||||
# to the client. `ckan generate config` generates a unique value for this each
|
|
||||||
# time it generates a config file.
|
|
||||||
beaker.session.secret = EHU6ef6rWzspiSQutl2F4G28n
|
|
||||||
|
|
||||||
# `ckan generate config` generates a unique value for this each time it generates
|
|
||||||
# a config file.
|
|
||||||
app_instance_uuid = 52ba6253-cf07-4f0b-ba3f-1c685b4a708c
|
|
||||||
|
|
||||||
# repoze.who config
|
|
||||||
who.config_file = %(here)s/who.ini
|
|
||||||
who.log_level = warning
|
|
||||||
who.log_file = %(cache_dir)s/who_log.ini
|
|
||||||
# Session timeout (user logged out after period of inactivity, in seconds).
|
|
||||||
# Inactive by default, so the session doesn't expire.
|
|
||||||
# who.timeout = 86400
|
|
||||||
|
|
||||||
## Database Settings
|
|
||||||
sqlalchemy.url = postgresql://ckan_default:pass@localhost/ckan_default
|
|
||||||
|
|
||||||
#ckan.datastore.write_url = postgresql://ckan_default:pass@localhost/datastore_default
|
|
||||||
#ckan.datastore.read_url = postgresql://datastore_default:pass@localhost/datastore_default
|
|
||||||
|
|
||||||
# PostgreSQL' full-text search parameters
|
|
||||||
ckan.datastore.default_fts_lang = english
|
|
||||||
ckan.datastore.default_fts_index_method = gist
|
|
||||||
|
|
||||||
|
|
||||||
## Site Settings
|
|
||||||
|
|
||||||
ckan.site_url =
|
|
||||||
#ckan.use_pylons_response_cleanup_middleware = true
|
|
||||||
|
|
||||||
## Authorization Settings
|
|
||||||
|
|
||||||
ckan.auth.anon_create_dataset = false
|
|
||||||
ckan.auth.create_unowned_dataset = false
|
|
||||||
ckan.auth.create_dataset_if_not_in_organization = false
|
|
||||||
ckan.auth.user_create_groups = false
|
|
||||||
ckan.auth.user_create_organizations = false
|
|
||||||
ckan.auth.user_delete_groups = true
|
|
||||||
ckan.auth.user_delete_organizations = true
|
|
||||||
ckan.auth.create_user_via_api = false
|
|
||||||
ckan.auth.create_user_via_web = true
|
|
||||||
ckan.auth.roles_that_cascade_to_sub_groups = admin
|
|
||||||
ckan.auth.public_user_details = true
|
|
||||||
ckan.auth.public_activity_stream_detail = true
|
|
||||||
ckan.auth.allow_dataset_collaborators = false
|
|
||||||
ckan.auth.create_default_api_keys = false
|
|
||||||
|
|
||||||
## API Token Settings
|
|
||||||
api_token.nbytes = 60
|
|
||||||
api_token.jwt.encode.secret = string:EHU6ef6rWzspiSQutl2F4G28n
|
|
||||||
api_token.jwt.decode.secret = string:EHU6ef6rWzspiSQutl2F4G28n
|
|
||||||
api_token.jwt.algorithm = HS256
|
|
||||||
|
|
||||||
## API Token: expire_api_token plugin
|
|
||||||
expire_api_token.default_lifetime = 3600
|
|
||||||
|
|
||||||
## Search Settings
|
|
||||||
|
|
||||||
ckan.site_id = default
|
|
||||||
#solr_url = http://127.0.0.1:8983/solr
|
|
||||||
|
|
||||||
|
|
||||||
## Redis Settings
|
|
||||||
|
|
||||||
# URL to your Redis instance, including the database to be used.
|
|
||||||
#ckan.redis.url = redis://localhost:6379/0
|
|
||||||
|
|
||||||
|
|
||||||
## CORS Settings
|
|
||||||
|
|
||||||
# If cors.origin_allow_all is true, all origins are allowed.
|
|
||||||
# If false, the cors.origin_whitelist is used.
|
|
||||||
# ckan.cors.origin_allow_all = true
|
|
||||||
# cors.origin_whitelist is a space separated list of allowed domains.
|
|
||||||
# ckan.cors.origin_whitelist = http://example1.com http://example2.com
|
|
||||||
|
|
||||||
|
|
||||||
## Plugins Settings
|
|
||||||
|
|
||||||
# Note: Add ``datastore`` to enable the CKAN DataStore
|
|
||||||
# Add ``datapusher`` to enable DataPusher
|
|
||||||
# Add ``resource_proxy`` to enable resorce proxying and get around the
|
|
||||||
# same origin policy
|
|
||||||
ckan.plugins = image_view text_view recline_view datastore datapusher envvars
|
|
||||||
|
|
||||||
# Define which views should be created by default
|
|
||||||
# (plugins must be loaded in ckan.plugins)
|
|
||||||
ckan.views.default_views = image_view text_view recline_view
|
|
||||||
|
|
||||||
# Customize which text formats the text_view plugin will show
|
|
||||||
#ckan.preview.json_formats = json
|
|
||||||
#ckan.preview.xml_formats = xml rdf rdf+xml owl+xml atom rss
|
|
||||||
#ckan.preview.text_formats = text plain text/plain
|
|
||||||
|
|
||||||
# Customize which image formats the image_view plugin will show
|
|
||||||
#ckan.preview.image_formats = png jpeg jpg gif
|
|
||||||
|
|
||||||
## Front-End Settings
|
|
||||||
|
|
||||||
ckan.site_title = CKAN
|
|
||||||
ckan.site_logo = /base/images/ckan-logo.png
|
|
||||||
ckan.site_description =
|
|
||||||
ckan.favicon = /base/images/ckan.ico
|
|
||||||
ckan.gravatar_default = identicon
|
|
||||||
ckan.preview.direct = png jpg gif
|
|
||||||
ckan.preview.loadable = html htm rdf+xml owl+xml xml n3 n-triples turtle plain atom csv tsv rss txt json
|
|
||||||
ckan.display_timezone = server
|
|
||||||
|
|
||||||
# package_hide_extras = for_search_index_only
|
|
||||||
#package_edit_return_url = http://another.frontend/dataset/<NAME>
|
|
||||||
#package_new_return_url = http://another.frontend/dataset/<NAME>
|
|
||||||
#ckan.recaptcha.publickey =
|
|
||||||
#ckan.recaptcha.privatekey =
|
|
||||||
#licenses_group_url = http://licenses.opendefinition.org/licenses/groups/ckan.json
|
|
||||||
# ckan.template_footer_end =
|
|
||||||
|
|
||||||
|
|
||||||
## Internationalisation Settings
|
|
||||||
ckan.locale_default = en
|
|
||||||
ckan.locale_order = en pt_BR ja it cs_CZ ca es fr el sv sr sr@latin no sk fi ru de pl nl bg ko_KR hu sa sl lv
|
|
||||||
ckan.locales_offered =
|
|
||||||
ckan.locales_filtered_out = en_GB
|
|
||||||
|
|
||||||
## Feeds Settings
|
|
||||||
|
|
||||||
ckan.feeds.authority_name =
|
|
||||||
ckan.feeds.date =
|
|
||||||
ckan.feeds.author_name =
|
|
||||||
ckan.feeds.author_link =
|
|
||||||
|
|
||||||
## Storage Settings
|
|
||||||
|
|
||||||
#ckan.storage_path = /var/lib/ckan
|
|
||||||
#ckan.max_resource_size = 10
|
|
||||||
#ckan.max_image_size = 2
|
|
||||||
|
|
||||||
## Webassets Settings
|
|
||||||
#ckan.webassets.use_x_sendfile = false
|
|
||||||
#ckan.webassets.path = /var/lib/ckan/webassets
|
|
||||||
|
|
||||||
|
|
||||||
## Datapusher settings
|
|
||||||
|
|
||||||
# Make sure you have set up the DataStore
|
|
||||||
|
|
||||||
#ckan.datapusher.formats = csv xls xlsx tsv application/csv application/vnd.ms-excel application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
|
|
||||||
#ckan.datapusher.url = http://127.0.0.1:8800/
|
|
||||||
#ckan.datapusher.assume_task_stale_after = 3600
|
|
||||||
|
|
||||||
# Resource Proxy settings
|
|
||||||
# Preview size limit, default: 1MB
|
|
||||||
#ckan.resource_proxy.max_file_size = 1048576
|
|
||||||
# Size of chunks to read/write.
|
|
||||||
#ckan.resource_proxy.chunk_size = 4096
|
|
||||||
|
|
||||||
## Activity Streams Settings
|
|
||||||
|
|
||||||
#ckan.activity_streams_enabled = true
|
|
||||||
#ckan.activity_list_limit = 31
|
|
||||||
#ckan.activity_streams_email_notifications = true
|
|
||||||
#ckan.email_notifications_since = 2 days
|
|
||||||
ckan.hide_activity_from_users = %(ckan.site_id)s
|
|
||||||
|
|
||||||
|
|
||||||
## Email settings
|
|
||||||
|
|
||||||
#email_to = errors@example.com
|
|
||||||
#error_email_from = ckan-errors@example.com
|
|
||||||
#smtp.server = localhost
|
|
||||||
#smtp.starttls = False
|
|
||||||
#smtp.user = username@example.com
|
|
||||||
#smtp.password = your_password
|
|
||||||
#smtp.mail_from =
|
|
||||||
#smtp.reply_to =
|
|
||||||
|
|
||||||
## Background Job Settings
|
|
||||||
ckan.jobs.timeout = 180
|
|
||||||
|
|
||||||
## Logging configuration
|
|
||||||
[loggers]
|
|
||||||
keys = root, ckan, ckanext, werkzeug
|
|
||||||
|
|
||||||
[handlers]
|
|
||||||
keys = console
|
|
||||||
|
|
||||||
[formatters]
|
|
||||||
keys = generic
|
|
||||||
|
|
||||||
[logger_root]
|
|
||||||
level = WARNING
|
|
||||||
handlers = console
|
|
||||||
|
|
||||||
[logger_werkzeug]
|
|
||||||
level = WARNING
|
|
||||||
handlers = console
|
|
||||||
qualname = werkzeug
|
|
||||||
propagate = 0
|
|
||||||
|
|
||||||
[logger_ckan]
|
|
||||||
level = INFO
|
|
||||||
handlers = console
|
|
||||||
qualname = ckan
|
|
||||||
propagate = 0
|
|
||||||
|
|
||||||
[logger_ckanext]
|
|
||||||
level = DEBUG
|
|
||||||
handlers = console
|
|
||||||
qualname = ckanext
|
|
||||||
propagate = 0
|
|
||||||
|
|
||||||
[handler_console]
|
|
||||||
class = StreamHandler
|
|
||||||
args = (sys.stderr,)
|
|
||||||
level = NOTSET
|
|
||||||
formatter = generic
|
|
||||||
|
|
||||||
[formatter_generic]
|
|
||||||
format = %(asctime)s %(levelname)-5.5s [%(name)s] %(message)s
|
|
||||||
Loading…
x
Reference in New Issue
Block a user