Skip to content

Latest commit

 

History

History
103 lines (64 loc) · 9.99 KB

architecture.md

File metadata and controls

103 lines (64 loc) · 9.99 KB
Raw
sidebar_position
2

Architecture

Telescope is both a product and a project. It is designed to meet a number of practical requirements (e.g., blog aggregation), but also to give students the opportunity to work on a real-world open source project using a wide range of technologies. We could build something simpler; but our design is also about creating a safe space for technical exploration. As a result, Telescope is architected like many modern distributed services.

Initial Design

Our initial design was captured in this rough diagram:

Original Telescope architecture

It shows RSS/Atom blog feeds (on the left) being downloaded (1) and then queued (2) in order to be downloaded and parsed into a database. Parsing each feed also included the idea of analysis, to look at word count, reading time, etc. This database of parsed feeds is then exposed to various clients via a REST API. We imagined being able to provide multiple clients, written for the web or mobile, but also connecting to Slack and other systems.

Current Design

The following gives an overview of the current (i.e. 2.4.0) design of Telescope. There are a number of major systems that make up the app:

  • legacy monolithic back-end (1.0)
  • back-end microservices (2.0)
  • front-end web app
  • SAML2-based authentication and authorization
  • autodeployment server
  • test, build, and other automation tooling

Legacy Monolithic Back-end (1.0)

Telescope's back-end began as a single, monolithic node.js app. During the 2.0 release, much of the back-end was split into separate microservices (see below). However, parts of the legacy back-end code were still in use (see src/backend) but they were eventually replaced by the parser service during 3.0 release.

Back-end Microservices (2.0)

The legacy back-end has been split into a series of microservices. Each microservice follows a similar architectural pattern and has a single focuses. The current microservices include:

  • SSO Auth Service (src/api/sso) - authentication and authorization with Seneca's SSO service
  • Feed-Discovery Service (src/api/feed-discovery) - blog feed discovery and validation
  • Image Service (src/api/image) - dynamic image generation for the front-end hero banner
  • Posts Service (src/api/posts) - API for accessing Post and Feed data in Redis (probably not well named at this point)
  • Search Service (src/api/search) - API for doing searches against Elasticsearch
  • Status Service (src/api/status) - API for accessing Telescope status information, as well as providing the Dashboards
  • Parser Service (src/api/parser) - feed and post parsing which was disabled in 2.0, but now enabled in 3.0 release.

All microservices are built on a common foundation, the Satellite module. Satellite provides a common set of features for building Express-based microservices, with proper logging, health checks, headers, authorization middleware, as well as connections to Redis and Elasticsearch. It saves us having to manage the same set of dependencies a dozen times, and repeat the same boilerplate code.

The microservices are deployed as Docker containers. Each defines its own Dockerfile, and the entire system is managed via docker-compose, see src/docker, where we have a base service definition in src/docker/docker-compose.yml as well as overrides for various environments: development.yml (local dev), production.yml (staging and production), and gitpod.yml (running Telescope in GitPod). Each of these service definitions uses an associated environment file, which are defined in src/config. For example, Telescope's production server uses the src/docker/production.yml file and src/config/env.production environment.

When deployed, the services are not directly accessible via the network (i.e. you can't go to localhost:5555 to access the server). Instead, you have to access them via an API gateway called Traefik, which also runs in its own container. Traefik lives at api.telescope.cdot.systems (production), dev.api.telescope.cdot.systems (staging), and localhost:80 (development). It provides routing to each of the service containers via labels defined in the src/docker/docker-compose.yml file. For example, the Image Service is available at /v1/image. In production you access it at https://api.telescope.cdot.systems/v1/image, in staging at https://dev.api.telescope.cdot.systems/v1/image, and locally at http://localhost/v1/image. The other services are similar. Traefik makes the different services look like one thing, but they are actually deployed and running separately.

The microservices can be controlled either using the docker-compose command directly, or with the following npm scripts:

  • pnpm run services:start starts all services. You can also list one or more services to start a subset, npm run services:start image parser
  • pnpm run services:stop stops all services, or those specified
  • pnpm run logs image shows logs for the image service (specify whichever service you want to view)

On Windows and macOS, Docker Desktop must be running before you can call these scripts, see the Docker documentation.

Front-end Web App

The front-end web app is written in TypeScript and uses next.js and Material UI for routing, component system, etc. The code lives in src/web/app.

The web app accesses many of the microservices at runtime, and we depend on a build step to export and include the necessary runtime variables. In the front-end app, they are available at runtime via src/web/app/src/config.ts. These are populated via the src/web/app/next.config.js during build time. See Environment Setup for details on how to populate these with different values for local, CI, staging, or production environments. See also Front End for more details about the front-end.

The front-end is run via its own container, and hosted statically by nginx. The container is defined in src/web/app/Dockerfile. Locally, it is available at http://localhost:8000, on staging at https://dev.telescope.cdot.systems/, and on production at https://telescope.cdot.systems/.

SAML2-based authentication and authorization

Telescope uses SAML2-based authentication via Seneca's Single Sign-On in combination with token-based authorization via the SSO Auth and Users service. A good discussion of the main ideas and implementation is available in this blog post and this Pull Request.

In development, we use a test PHP-based SAML Identity Provider to test against. See Login.

Autodeployment Server

Our release infrastructure is built primarily on GitHub Actions, see .github/workflows/release.yml. However, we also host our own build service at tools/autodeployment. When a Release is created, the autodeployment server receives a webhook event from GitHub. We use this to trigger a new build and deploy it when complete. The build script that gets run is at tools/autodeployment/deploy.sh, which does a green/blue style of deployment (i.e., keep one version running while you build a new one, swap them when ready).

The autodeployment server is run outside of Docker, since it manages our docker containers. It has an API available at https://telescope.cdot.systems/deploy/status (production) and https://dev.telescope.cdot.systems/deploy/status (staging), and build logs are available at /deploy/status/log.

Dashboards

The Status service provides a number of real-time dashboards. These are available at https://api.telescope.cdot.systems/v1/status (production) and https://dev.api.telescope.cdot.systems/v1/status (staging).

The Dashboards intentionally do not use React, next.js, or other technologies from the normal front-end. This was done in order to provide students who don't want to work on React a place to contribute. Instead, it uses plain HTML, SCSS/CSS, and JavaScript, and uses server-side rendering with Express. It also uses a variety of front-end DOM code written in JavaScript.

The Status service uses this Autodeployment API to show a live build log, for example: https://api.telescope.cdot.systems/v1/status/build on production, https://dev.api.telescope.cdot.systems/v1/status/build on staging.

Test, Build, and other Automation Tooling

We use the following tools:

  • pnpm to manage packages within our monorepo, see pnpm-workspace.yaml
  • jest (unit, snapshot) and Playwright (e2e) to write and run tests. We define configs (see ./jest*.js) in the root of the project, and use/override them in sub-folders. Our tests are in test/ and also in test/ folders deeper in the tree
  • eslint to lint our code, see .eslintrc.js and .eslintignore
  • prettier to format our code, with Husky managing a precommit hook to automatically apply it on every commit, see .prettierrc, .prettierignore, and .husky/
  • various VSCode settings and extensions, defined in .vscode/
  • GitHub Actions Workflows for testing, see .github/
  • Renovatebot to manage dependency updates, see Renovate
  • GitPod for cloud development, see Gitpod, .gitpod.yml and gitpod.Dockerfile

We also have a number of tools running in containers at runtime (most only in staging/production) (see docker/production.yml):

  • nginx, which we use to host static web sites and also as a reverse proxy for node APIs. Our nginx configs are built from a template that lives in ./config/nginx.conf.template
  • certbot, which manages our Let's Encrypt SSL Certificates (i.e., automatically renewing them)
  • portainer, to provide a user-friendly front-end to our main containers at run-time