Skip to content

Setup sc_pack in three steps

Preliminaries

For simplicity, we will assume that you have a directory in your home, /home/shimmercat/test/shimmercat-accelerator, where you want to install sc_pack. We will call this directory the install_dir.

You can use any other directory of course, and if you do so please remember to change the path for the commands below.

1. Installation

  • Move to the install_dir directory, for example via command line:
$ cd /home/shimmercat/test/shimmercat-accelerator
  • Download the latest sc_pack to the directory with the below download command that requires jq. You can find the download_url at this API page.
$ curl -O $(curl -s  https://accelerator.shimmercat.com/presentation-api/v1/download/sc_pack/?format=json | jq -r .download_url)
  • Install dependencies:
$ sudo apt-get install python3 python3-dev virtualenv git
  • Create a Python 3 virtualenv:
$ virtualenv venv -p python3
  • Activate the virtualenv:
$ source venv/bin/activate
  • Install sc_pack on your virtualenv:
$ pip install /home/shimmercat/test/shimmercat-accelerator/<sc_pack-version>.whl
  • To verify that sc_pack is properly installed, run:
$ sc_pack -h

2. Directory hierarchy and configuration

The programs need a directory as the top of the hierarchy, and some information for configuration. This information contains details about your setup, and secrets needed for the programs to send and receive data from our cloud service.

The configuration file

All configuration is structured in a YAML file, called sc_pack.conf.yaml, which has nesting levels. That's what we take at here to understand more details about what each field means in sc_pack.conf.yaml. We have created an example file below - remember to edit the fields according to your details, mainly make sure the install_dir is the one you have been using so far in the commands you have already run before.

---
# This is a YAML document with the configuration items
# we need to set up the server with sc_pack.
# For the format, see example below.
#
# All the directories defined are relative to install_dir
shimmercat:
    root_dir: www
    devlove_file_path: devlove.yaml
    scratch_dir_name: shimmercat-scratch-folder
    listen_port: "4043"
    http2https_port: "8084"
    bots_views_dir: views-dir
# Update this value with the value of your authentication token
api_access_token: <your_authentication_token>
user: shimmercat
user_group: shimmercat
# Update this path with the path you want the sc_pack files to be at
install_dir: /home/shimmercat/test/shimmercat-accelerator
logs_host_name: localhost
# Update this value with the value of your deployment site's `name16` you will create on the next step
deployment_site_long_name: <your_name16>
# Update this value with the value of your deployment site's `secret16` you will create on the next step
deployment_site_long_secret: <your_secret16>
amqp_server_url: amqp.staging.c.shimmercat.com
amqp_vhost: skyloft
enable_bots_blocking: True
humanity_validator_host: localhost
humanity_validator_port: 8080
google_recaptcha_site_key: 6LfimFIUAAAAAACMycLvA3wetRR9v6ejSbf7uu2v
google_recaptcha_site_secret: 6LfimFIUAAAAAEpCe1QNHwr2p46AMKbUoOBdLWbE
transit_encryption_key: test-accelerator.shimmercat.com.no_enc
deployment_group: test

If you don't have a user called shimmercat you can either create it, or use a different one. In any case, please be sure to update the user and user_group above accordingly.

  • Copy the content above to /home/shimmercat/test/shimmercat-accelerator/sc_pack.conf.yaml

You can also get a similar copy of the config file by using the sc_pack command:

$ sc_pack extract example.sc_pack.conf.yaml

Directory hierarchy

Once the configuration file is created, it's time to setup the directory hierarchy. If you have already a file called sc_pack.conf.yaml in the directory from where you are running that command, run:

$ sudo sc_pack create

Otherwise you can let sc_pack create the deployment environment by typing:

$ sc_pack create -f /home/shimmercat/test/shimmercat-accelerator/sc_pack.conf.yaml

Subject to the same rule, it's possible to omit the -f in many sc_pack sub-commands. E.g, if the current directory is /home/shimmercat/test/shimmercat-accelerator and the file /home/shimmercat/test/shimmercat-accelerator/sc_pack.conf.yaml exists and you run sc_pack create, the configuration file taken is /home/shimmercat/test/shimmercat-accelerator/sc_pack.conf.yaml.

  • Check that the directories are properly created:
$ ls
$ celery redis shimmercat   devlove.yaml sc_logs_agent shimmercat-scratch-folder views-dir  # And some others
  • Start the accelerator:
$ sc_pack supervisord

You will see some logs similar to:

2019-05-18 14:52:00,385 INFO success: redis entered RUNNING state,
process has stayed up for > than 10 seconds (startsecs)
2019-05-18 14:52:00,385 INFO success: shimmercat entered RUNNING state,
process has stayed up for > than 10 seconds (startsecs)
2019-05-18 14:52:00,385 INFO success: kv_store entered RUNNING state,
process has stayed up for > than 10 seconds (startsecs)
2019-05-18 14:52:00,385 INFO success: humanity_validator entered RUNNING state,
process has stayed up for > than 10 seconds (startsecs)

To start the accelerator at boot time, see here.

Inside every service directory there is a directory named data, where the logs for each service are stored. If there are errors, please check the logs for more details. For example, if you see the error celery (exit status 1; not expected), you can check more details in the log file /home/shimmercat/test/shimmercat-accelerator/celery/data/celery.log.

Logging

Note that it's expected that the celery and the sc_logs_agent don't start with the current configuration. The reason is that we are still missing the info for deployment_site_long_name and deployment_site_long_secret that are needed to let Celery authenticate with Celery broker.

3. Create deployment sites and domains

A DeploymentSite represents a site where sc_pack has been deployed, and it is the root of the tree to configure everything else. In the commands below, make sure to replace <your_authentication_token> with your authentication token.

Create a deployment site

To create a DeploymentSite use the below command:

$ curl -X POST https://accelerator.shimmercat.com/presentation-api/v1/deployment-site/ -H 'Authorization: Token <your_authentication_token>' -H 'Content-type: application/json' -d '{"description": "Deployment site on server test1"}'

You will get the response data:

{
  "id":<your_deployment_id>,
  "customer": {
      "id": <your_customer_id>,
      "name":"ShimmerCat Accelerator Test"
  },
  "description":"Deployment site on server test1",
  "name16":"<your_deployment_site_long_name>",
  "secret16":"<your_deployment_site_long_secret>"
}

Now edit the file sc_pack.conf.yaml and add the responses for <your_deployment_site_long_name> and <your_deployment_site_long_secret>. To update the configuration files on the file system with the name16, and secret16 execute:

$ sc_pack create

Whenever you change sc_pack.conf.yaml you should run sc_pack create - otherwise the changes won't be synchronized to all the configuration files used by the services.

Create a domain for the deployment site

To create a domain for the deployment site, use the command below:

$ curl -X POST https://accelerator.shimmercat.com/presentation-api/v1/domain/ -H 'Authorization: Token <your_authentication_token>' -H 'Content-type: application/json' -d '{"name": "example-accelerator.shimmercat.com", "deployment_site_id": <your_deployment_id>}'

Note that you need to change <your_deployment_id> to make sure that the domain you are creating will be linked to the deployment site with this id. You will get the response:

{"id":<your_domain_id>,"name":"example-accelerator.shimmercat.com","sub_domains":[]}

If you would like to add more deployment sites to your domain, see more information on Adding a new deployment for an existing domain. You are now ready to serve a website with ShimmerCat.

4. Serve the example website

In your install_dir (/home/shimmercat/test/shimmercat-accelerator) there is a server config file named devlove.yaml, which was automatically created in the previous sections. If you want to try our static website example, edit the devlove.yaml file according to:

---
shimmercat-devlove:
      domains:
          elec example-accelerator.shimmercat.com:
              root-dir: www

Then download our static website example and decompress it in the directory /home/shimmercat/test/shimmercat-accelerator/www. Be sure to check that you have the file /home/shimmercat/test/shimmercat-accelerator/www/index.html.

Now edit your /etc/hosts to have the line 127.0.0.1 example-accelerator.shimmercat.com.

Last thing is to run the command:

$ sc_pack supervisord

Then open a browser and visit: https://example-accelerator.shimmercat.com:4043/.