Gitlab CI/CD

For the development environment.

Install with docker

Read the official documentation for how to install docker.

I installed docker on the MacOS.

gitlab

documentation

sudo docker run --detach \
  --hostname 192.168.8.226 \
  --publish 443:443 --publish 80:80 --publish 8022:22 \
  --name gitlab \
  --restart always \
  --volume ~/gitlab/config:/etc/gitlab \
  --volume ~/gitlab/logs:/var/log/gitlab \
  --volume ~/gitlab/data:/var/opt/gitlab \
  gitlab/gitlab-ce:latest

gitlab-runner

Order of initialization:

  • install
  • register
  • custom config

documentation

There is no difficulty to install gitlab and gitlab-runner in docker.

sudo docker run -d --name gitlab-runner --restart always \
  -v ~/gitlab-runner/config:/etc/gitlab-runner \
  -v /var/run/docker.sock:/var/run/docker.sock \
  gitlab/gitlab-runner:latest

I was confused with the config of gitlab-runner, so I wrote these down.

register:

  • docker run --rm -t -i -v ~/gitlab-runner/config:/etc/gitlab-runner gitlab/gitlab-runner register, cmd for register a runner.
    • the register info will record in the ~/gitlab-runner/config/config.toml
    • url is the url of the gitlab
    • token is from your gitlab, the project runner and shared runner is different. You can registe it for shared runner and enable for the specified project.
    • executor choose docker, I use the image python:3.7 for my python project.
    • manual operation to edit the config.toml:
      • pull_policy = "if-not-present" default value is always for every time pull the image.
      • network_mode = "host" default value is bridge that runner cannot connect other machines in the same net.

for non-interactive register:

docker run --rm -v /srv/gitlab-runner/config:/etc/gitlab-runner gitlab/gitlab-runner register \
  --non-interactive \
  --executor "docker" \
  --docker-image alpine:latest \
  --url "https://gitlab.com/" \
  --registration-token "PROJECT_REGISTRATION_TOKEN" \
  --description "docker-runner" \
  --tag-list "docker,aws" \
  --run-untagged="true" \
  --locked="false" \
  --access-level="not_protected"

config.toml

privileged 如果遇到mkdir permission denied，设为true，可以解决

[[runners]]
  name = "python3.7"
  url = "http://192.168.8.226/"
  token = "xxxxxx"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.docker]
    tls_verify = false
    image = "python:3.7"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
    pull_policy = "if-not-present"
    network_mode = "host"
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
  [runners.custom]
    run_exec = ""

cache with local files. Npm example:

volumes = ["/opt/gitlab-runner/cache:/cache"]

ls /opt/gitlab-runner/cache/GROUP/PROJECT_NAME/builds/GROUP/PROJECT_NAME
cache.zip

cache.zip is the node_modules.zip file.

.gitlab-ci.yml

  • Use the ssh connect development machine and run the shell command to update the project.
  • cache after the CI. The runner will cache the pip and venv packages.
# This file is a template, and might need editing before it works on your project.
image: python:3.7

# This folder is cached between builds
# http://docs.gitlab.com/ce/ci/yaml/README.html#cache
cache:
  paths:
    - .cache/pip
    - venv/

before_script:
  ##
  ## Install ssh-agent if not already installed, it is required by Docker.
  ## (change apt-get to yum if you use an RPM-based image)
  ##
  - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'

  ##
  ## Run ssh-agent (inside the build environment)
  ##
  - eval $(ssh-agent -s)

  ##
  ## Add the SSH key stored in SSH_PRIVATE_KEY variable to the agent store
  ## We're using tr to fix line endings which makes ed25519 keys work
  ## without extra base64 encoding.
  ## https://gitlab.com/gitlab-examples/ssh-private-key/issues/1#note_48526556
  ##
  - echo "$SSH_PRIVATE_KEY" > deploy.key
  - chmod 0600 deploy.key
  - ssh-add deploy.key
  - mkdir -p ~/.ssh

  - chmod 700 ~/.ssh
  - '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'

  ##
  ## Optionally, if you will be using any Git commands, set the user name and
  ## and email.
  ##
  #- git config --global user.email "user@example.com"
  #- git config --global user.name "User name"
  - python -V  # Print out python version for debugging

test:
  script:
  - python -m venv ./venv
  - source venv/bin/activate
  - pip install -U -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple/
  - cd lq_end
#  - cp -f config/ci.py config/config.py
#  - python manage.py devops drop_and_recreate_ci_database
#  - flask db upgrade
#  - flask init-db
  - python -m pytest --cov=.
  - ssh root@192.168.8.109 "cd /root/test-project && ./script.sh"

#pep8:
#  script:
#  - python -m venv ./venv
#  - source venv/bin/activate
#  - pip install -U flake8
#  - cd projectpath
#  - flake8