Request Baskets

Request Baskets is a web service to collect arbitrary HTTP requests and inspect them via RESTful API or simple web UI.

It is strongly inspired by ideas and application design of the RequestHub project and reproduces functionality offered by RequestBin service.

Table of Contents

• Introduction
  • Features
  • Screenshots
• Install
  • Build from source
  • Run docker container
• Configuration
  • Parameters
• Usage
  • Bolt database
  • PostgreSQL database
  • MySQL database
• Docker
  • Build docker image
  • Run container as a service
  • Cleanup

Introduction

Request Baskets service is available on our demonstration server: https://rbaskets.in

However, we encourage you to set up your own server and keep control over the information passed through and collected by Request Baskets service.

Features

Distinguishing features of Request Baskets service:

• RESTful API to manage and configure baskets, see Request Baskets API documentation in interactive mode
• All baskets are protected by unique tokens from unauthorized access; end-points to collect requests do not require authorization though
• Individually configurable capacity for every basket
• Pagination support to retrieve collections: basket names, collected requests
• Configurable responses for every HTTP method
• Alternative storage types for configured baskets and collected requests:
  • In-memory - ultra fast, but limited to available RAM and collected data is lost after service restart
  • Bolt DB - fast persistent storage for collected data based on embedded bbolt database (maintained fork of Bolt), service can be restarted without data loss and storage is not limited by available RAM
  • SQL database - classical data storage, multiple instances of service can run simultaneously and collect data in shared data storage, which makes the solution more robust and scaleable (PostgreSQL and MySQL are only supported at the moment)
  • Can be extended by custom implementations of storage interface

Screenshots

Basket requests overview:

Configuration of basket responses:

Install

Build from source

Build latest:

$ go get github.com/darklynx/request-baskets

Run:

$ export PATH=$PATH:$GOPATH/bin
$ request-baskets

Run docker container

$ docker pull darklynx/request-baskets
$ docker run -p 55555:55555 darklynx/request-baskets

Configuration

Request Baskets service supports several command line configuration parameters. Use -h or --help to print command line help:

$ request-baskets --help
Usage of bin/request-baskets:
  -db string
      Baskets storage type: "mem" - in-memory, "bolt" - Bolt DB, "sql" - SQL database (default "mem")
  -file string
      Database location, only applicable for file or SQL databases (default "./baskets.db")
  -conn string
      Database connection string for SQL databases, if undefined "file" argument is considered
  -l string
      HTTP listen address (default "127.0.0.1")
  -p int
      HTTP service port (default 55555)
  -page int
      Default page size (default 20)
  -size int
      Initial basket size (capacity) (default 200)
  -maxsize int
      Maximum allowed basket size (max capacity) (default 2000)
  -token string
      Master token, random token is generated if not provided
  -basket value
      Name of a basket to auto-create during service startup (can be specified multiple times)
  -prefix string
      Service URL path prefix
  -mode string
      Service mode: "public" - any visitor can create a new basket, "restricted" - baskets creation requires master token (default "public")
  -theme string
      CSS theme for web UI, supported values: standard, adaptive, flatly (default "standard")

Parameters

List of command line parameters with corresponding ENVVAR for docker container:

• -p port (PORT) - HTTP service listener port, default value is 55555
• -l IP address (LISTEN) - HTTP listener IP address, default 127.0.0.1 (docker default: 0.0.0.0)
• -page size (PAGE) - default page size when retrieving collections
• -size size (SIZE) - default new basket capacity, applied if basket capacity is not provided during creation
• -maxsize size (MAXSIZE) - maximum allowed basket capacity, basket capacity greater than this number will be rejected by service
• -token token (TOKEN) - master token to gain control over all baskets, if not defined a random token will be generated when service is launched and printed to stdout
• -db type (DB) - defines baskets storage type: mem - in-memory storage (default), bolt - bbolt database (docker default), sql - SQL database
• -file location (FILE) - location of Bolt database file, only relevant if appropriate storage type is chosen
• -conn connection (CONN) - database connection string for SQL databases, if undefined -file argument is considered
• -basket value (BASKET) - name of a basket to auto-create during service startup, this parameter can be specified multiple times
• -prefix URL path prefix (PATHPREFIX) - allows to host API and web-UI of baskets service under a sub-path instead of domain ROOT
• -mode mode (MODE) - defines service operation mode: public - when any visitor can create a new basket, or restricted - baskets creation requires master token
• -theme theme (THEME) - CSS theme for web UI, supported values: standard, adaptive, flatly

Usage

Open http://localhost:55555 in your browser. The main page will display a list of baskets that may be accessed if the basket token is known. It is possible to create a new basket if the name is not in use.

If basket was successfully created the authorization token is displayed. It is important to remember the token because it authorizes the access to management features of created basket and allows to retrieve collected HTTP requests. The token is temporary stored in browser session to simplify UI integration and improve user experience. However, once browser tab is closed, the token will be lost.

To collect HTTP requests send them (GET, POST, PUT, DELETE, etc.) to http://localhost:55555/<basket_name>

To view collected requests and manage basket:

• Open basket web UI http://localhost:55555/web/<basket_name>
• Use RESTful API exposed at http://localhost:55555/api/baskets/<basket_name>/...

It is possible to forward all incoming HTTP requests to arbitrary URL by configuring basket via web UI or RESTful API.

Bolt database

By default Request Baskets service keeps configured baskets and collected HTTP requests in memory. This data is lost after service or server restart. However a service can be configured to store collected data on file system. In this case the service can be restarted without loosing created baskets and collected data.

To start service in persistent mode simply configure the appropriate storage type, such as bbolt database (maintained fork of Bolt):

$ request-baskets -db bolt -file /var/lib/request-baskets/baskets.db
2016/01/08 23:15:28 [info] generated master token: abcdefgh1234567...
2016/01/08 23:15:28 [info] using Bolt database to store baskets
2016/01/08 23:15:28 [info] Bolt database location: /var/lib/rbaskets/baskets.db
2016/01/08 23:15:28 [info] starting HTTP server on port: 55555
...

Any other kind of storages or databases (e.g. MySQL, MongoDb) to keep collected data can be introduced by implementing following interfaces: BasketsDatabase and Basket

PostgreSQL database

The first attempt to implement SQL database storage for Request Baskets service is now available for evaluation. Even though the logic to organize the data within SQL database is written in the generic SQL dialect, the code make use of parametrized SQL queries that unfortunately do not have standard to express parameter placeholders across different databases.

Current implementation is based on PostgreSQL syntax. So running Request Baskets service with PostgreSQL database as a storage is fully supported.

Use following example to start the Request Baskets service with PostgreSQL database:

$ request-baskets -db sql -conn "postgres://rbaskets:pwd@localhost/baskets?sslmode=disable"
2018/01/25 01:06:25 [info] generated master token: mSEAcYvpDlg...
2018/01/25 01:06:25 [info] using SQL database to store baskets
2018/01/25 01:06:25 [info] SQL database type: postgres
2018/01/25 01:06:25 [info] creating database schema
2018/01/25 01:06:25 [info] database is created, version: 1
2018/01/25 01:06:25 [info] HTTP server is listening on 127.0.0.1:55555
...

The documentation of Go driver for PostgreSQL provides detailed description of connection string and its parameters.

If no configured instance of PostgreSQL server is available to test the Request Baskets service with, there is a quick way to launch one using Docker with following command:

$ docker run --rm --name pg_baskets -e POSTGRES_USER=rbaskets -e POSTGRES_PASSWORD=pwd \
    -e POSTGRES_DB=baskets -d -p 5432:5432 postgres

# following command will stop and destroy the instance of PostgreSQL container
$ docker stop pg_baskets

MySQL database

Added driver and application support within the SQL basket database for MySQL (or MariaDB) database.

Use following example to start the Request Baskets service with MySQL database:

$ request-baskets -db sql -conn "mysql://rbaskets:pwd@/baskets"
2018/01/28 23:39:59 [info] generated master token: aPgyuLxw723q...
2018/01/28 23:39:59 [info] using SQL database to store baskets
2018/01/28 23:39:59 [info] SQL database type: mysql
2018/01/28 23:39:59 [info] creating database schema
2018/01/28 23:39:59 [info] database is created, version: 1
2018/01/28 23:39:59 [info] HTTP server is listening on 127.0.0.1:55555
...

The documentation of Go driver for MySQL provides detailed description of connection string and its parameters.

If no configured instance of MySQL server is available to test the Request Baskets service with, there is a quick way to launch one using Docker with following command:

$ docker run --rm --name mysql_baskets -e MYSQL_USER=rbaskets -e MYSQL_PASSWORD=pwd \
    -e MYSQL_DATABASE=baskets -e MYSQL_RANDOM_ROOT_PASSWORD=yes -d -p 3306:3306 mysql

# following command will stop and destroy the instance of MySQL container
$ docker stop mysql_baskets

Docker

Build docker image

$ docker build -t request-baskets .

This will create a docker image using multi-stage docker builds approach with 2 stages: compiling the service and packaging the result to a tiny alpine container. The resulting size of built image is ~12 Mb.

Note: since the first stage is using golang:latest container to build the service executable, there is no need to have Go lang SDK installed on the machine where process of building the container is taking place.

See docker folder for alternative docker builds with detailed explanation of the process for every variant.

Run container as a service

$ docker run --name rbaskets -d -p 55555:55555 request-baskets
$ docker logs rbaskets

Cleanup

Stop and delete docker container:

$ docker stop rbaskets
$ docker rm rbaskets

Delete docker image:

$ docker rmi request-baskets