add a webapi and webhooks for a simple http/json-based api

for applications to compose/send messages, receive delivery feedback, and
maintain suppression lists.

this is an alternative to applications using a library to compose messages,
submitting those messages using smtp, and monitoring a mailbox with imap for
DSNs, which can be processed into the equivalent of suppression lists. but you
need to know about all these standards/protocols and find libraries. by using
the webapi & webhooks, you just need a http & json library.

unfortunately, there is no standard for these kinds of api, so mox has made up
yet another one...

matching incoming DSNs about deliveries to original outgoing messages requires
keeping history of "retired" messages (delivered from the queue, either
successfully or failed). this can be enabled per account. history is also
useful for debugging deliveries. we now also keep history of each delivery
attempt, accessible while still in the queue, and kept when a message is
retired. the queue webadmin pages now also have pagination, to show potentially
large history.

a queue of webhook calls is now managed too. failures are retried similar to
message deliveries. webhooks can also be saved to the retired list after
completing. also configurable per account.

messages can be sent with a "unique smtp mail from" address. this can only be
used if the domain is configured with a localpart catchall separator such as
"+". when enabled, a queued message gets assigned a random "fromid", which is
added after the separator when sending. when DSNs are returned, they can be
related to previously sent messages based on this fromid. in the future, we can
implement matching on the "envid" used in the smtp dsn extension, or on the
"message-id" of the message. using a fromid can be triggered by authenticating
with a login email address that is configured as enabling fromid.

suppression lists are automatically managed per account. if a delivery attempt
results in certain smtp errors, the destination address is added to the
suppression list. future messages queued for that recipient will immediately
fail without a delivery attempt. suppression lists protect your mail server
reputation.

submitted messages can carry "extra" data through the queue and webhooks for
outgoing deliveries. through webapi as a json object, through smtp submission
as message headers of the form "x-mox-extra-<key>: value".

to make it easy to test webapi/webhooks locally, the "localserve" mode actually
puts messages in the queue. when it's time to deliver, it still won't do a full
delivery attempt, but just delivers to the sender account. unless the recipient
address has a special form, simulating a failure to deliver.

admins now have more control over the queue. "hold rules" can be added to mark
newly queued messages as "on hold", pausing delivery. rules can be about
certain sender or recipient domains/addresses, or apply to all messages pausing
the entire queue. also useful for (local) testing.

new config options have been introduced. they are editable through the admin
and/or account web interfaces.

the webapi http endpoints are enabled for newly generated configs with the
quickstart, and in localserve. existing configurations must explicitly enable
the webapi in mox.conf.

gopherwatch.org was created to dogfood this code. it initially used just the
compose/smtpclient/imapclient mox packages to send messages and process
delivery feedback. it will get a config option to use the mox webapi/webhooks
instead. the gopherwatch code to use webapi/webhook is smaller and simpler, and
developing that shaped development of the mox webapi/webhooks.

for issue #31 by cuu508

README.md

@@ -33,6 +33,8 @@ See Quickstart below to get started.
   support is limited).
 - Webserver with serving static files and forwarding requests (reverse
   proxy), so port 443 can also be used to serve websites.
+- Simple HTTP/JSON API for sending transaction email and receiving delivery
+  events and incoming messages (webapi and webhooks).
 - Prometheus metrics and structured logging for operational insight.
 - "mox localserve" subcommand for running mox locally for email-related
   testing/developing, including pedantic mode.
@@ -133,12 +135,13 @@ https://nlnet.nl/project/Mox/.
 
 ## Roadmap
 
+- Aliases, for delivering to multiple local accounts.
 - Webmail improvements
-- HTTP-based API for sending messages and receiving delivery feedback
 - Calendaring with CalDAV/iCal
 - More IMAP extensions (PREVIEW, WITHIN, IMPORTANT, COMPRESS=DEFLATE,
   CREATE-SPECIAL-USE, SAVEDATE, UNAUTHENTICATE, REPLACE, QUOTA, NOTIFY,
   MULTIAPPEND, OBJECTID, MULTISEARCH, THREAD, SORT)
+- SMTP DSN extension
 - ARC, with forwarded email from trusted source
 - Forwarding (to an external address)
 - Add special IMAP mailbox ("Queue?") that contains queued but
@@ -447,6 +450,23 @@ messages, for example by replacing your Message-Id header and thereby
 invalidating your DKIM-signatures, or rejecting messages with more than one
 DKIM-signature.
 
+## Can I use mox to send transactional email?
+
+Yes. While you can use SMTP submission to send messages you've composed
+yourself, and monitor a mailbox for DSNs, a more convenient option is to use
+the mox HTTP/JSON-based webapi and webhooks.
+
+The mox webapi can be used to send outgoing messages that mox composes. The web
+api can also be used to deal with messages stored in an account, like changing
+message flags, retrieving messages in parsed form or individual parts of
+multipart messages, or moving messages to another mailbox or deleting messages
+altogether.
+
+Mox webhooks can be used to receive updates about incoming and outgoing
+deliveries. Mox can automatically manage per account suppression lists.
+
+See https://www.xmox.nl/features/#hdr-webapi-and-webhooks for details.
+
 ## Can I use existing TLS certificates/keys?
 
 Yes. The quickstart command creates a config that uses ACME with Let's Encrypt,

apidiff/next.txt

@@ -13,6 +13,7 @@ Below are the incompatible changes between v0.0.10 and next, per package.
 # iprev
 
 # message
+- (*Composer).TextPart: changed from func(string) ([]byte, string, string) to func(string, string) ([]byte, string, string)
 - From: changed from func(*log/slog.Logger, bool, io.ReaderAt) (github.com/mjl-/mox/smtp.Address, *Envelope, net/textproto.MIMEHeader, error) to func(*log/slog.Logger, bool, io.ReaderAt, *Part) (github.com/mjl-/mox/smtp.Address, *Envelope, net/textproto.MIMEHeader, error)
 - NewComposer: changed from func(io.Writer, int64) *Composer to func(io.Writer, int64, bool) *Composer
 

apidiff/packages.txt

@@ -16,3 +16,5 @@ spf
 subjectpass
 tlsrpt
 updates
+webapi
+webhook

config/config.go

@@ -182,6 +182,8 @@ type Listener struct {
 	AdminHTTPS   WebService `sconf:"optional" sconf-doc:"Admin web interface listener like AdminHTTP, but for HTTPS. Requires a TLS config."`
 	WebmailHTTP  WebService `sconf:"optional" sconf-doc:"Webmail client, for reading email. Default path is /webmail/."`
 	WebmailHTTPS WebService `sconf:"optional" sconf-doc:"Webmail client, like WebmailHTTP, but for HTTPS. Requires a TLS config."`
+	WebAPIHTTP   WebService `sconf:"optional" sconf-doc:"Like WebAPIHTTP, but with plain HTTP, without TLS."`
+	WebAPIHTTPS  WebService `sconf:"optional" sconf-doc:"WebAPI, a simple HTTP/JSON-based API for email, with HTTPS (requires a TLS config). Default path is /webapi/."`
 	MetricsHTTP  struct {
 		Enabled bool
 		Port    int `sconf:"optional" sconf-doc:"Default 8010."`
@@ -210,7 +212,7 @@ type Listener struct {
 	} `sconf:"optional" sconf-doc:"All configured WebHandlers will serve on an enabled listener. Either ACME must be configured, or for each WebHandler domain a TLS certificate must be configured."`
 }
 
-// WebService is an internal web interface: webmail, account, admin.
+// WebService is an internal web interface: webmail, webaccount, webadmin, webapi.
 type WebService struct {
 	Enabled   bool
 	Port      int    `sconf:"optional" sconf-doc:"Default 80 for HTTP and 443 for HTTPS."`
@@ -356,6 +358,19 @@ type Route struct {
 
 // todo: move RejectsMailbox to store.Mailbox.SpecialUse, possibly with "X" prefix?
 
+// note: outgoing hook events are in ../queue/hooks.go, ../mox-/config.go, ../queue.go and ../webapi/gendoc.sh. keep in sync.
+
+type OutgoingWebhook struct {
+	URL           string   `sconf-doc:"URL to POST webhooks."`
+	Authorization string   `sconf:"optional" sconf-doc:"If not empty, value of Authorization header to add to HTTP requests."`
+	Events        []string `sconf:"optional" sconf-doc:"Events to send outgoing delivery notifications for. If absent, all events are sent. Valid values: delivered, suppressed, delayed, failed, relayed, expanded, canceled, unrecognized."`
+}
+
+type IncomingWebhook struct {
+	URL           string `sconf-doc:"URL to POST webhooks to for incoming deliveries over SMTP."`
+	Authorization string `sconf:"optional" sconf-doc:"If not empty, value of Authorization header to add to HTTP requests."`
+}
+
 type SubjectPass struct {
 	Period time.Duration `sconf-doc:"How long unique values are accepted after generating, e.g. 12h."` // todo: have a reasonable default for this?
 }
@@ -368,6 +383,12 @@ type AutomaticJunkFlags struct {
 }
 
 type Account struct {
+	OutgoingWebhook          *OutgoingWebhook `sconf:"optional" sconf-doc:"Webhooks for events about outgoing deliveries."`
+	IncomingWebhook          *IncomingWebhook `sconf:"optional" sconf-doc:"Webhooks for events about incoming deliveries over SMTP."`
+	FromIDLoginAddresses     []string         `sconf:"optional" sconf-doc:"Login addresses that cause outgoing email to be sent with SMTP MAIL FROM addresses with a unique id after the localpart catchall separator (which must be enabled when addresses are specified here). Any delivery status notifications (DSN, e.g. for bounces), can be related to the original message and recipient with unique id's. You can login to an account with any valid email address, including variants with the localpart catchall separator. You can use this mechanism to both send outgoing messages both with and without unique fromid for a given address."`
+	KeepRetiredMessagePeriod time.Duration    `sconf:"optional" sconf-doc:"Period to keep messages retired from the queue (delivered or failed) around. Keeping retired messages is useful for maintaining the suppression list for transactional email, for matching incoming DSNs to sent messages, and for debugging. The time at which to clean up (remove) is calculated at retire time. E.g. 168h (1 week)."`
+	KeepRetiredWebhookPeriod time.Duration    `sconf:"optional" sconf-doc:"Period to keep webhooks retired from the queue (delivered or failed) around. Useful for debugging. The time at which to clean up (remove) is calculated at retire time. E.g. 168h (1 week)."`
+
 	Domain                       string                 `sconf-doc:"Default domain for account. Deprecated behaviour: If a destination is not a full address but only a localpart, this domain is added to form a full address."`
 	Description                  string                 `sconf:"optional" sconf-doc:"Free form description, e.g. full name or alternative contact info."`
 	FullName                     string                 `sconf:"optional" sconf-doc:"Full name, to use in message From header when composing messages in webmail. Can be overridden per destination."`
@@ -383,10 +404,11 @@ type Account struct {
 	NoFirstTimeSenderDelay       bool                   `sconf:"optional" sconf-doc:"Do not apply a delay to SMTP connections before accepting an incoming message from a first-time sender. Can be useful for accounts that sends automated responses and want instant replies."`
 	Routes                       []Route                `sconf:"optional" sconf-doc:"Routes for delivering outgoing messages through the queue. Each delivery attempt evaluates these account routes, domain routes and finally global routes. The transport of the first matching route is used in the delivery attempt. If no routes match, which is the default with no configured routes, messages are delivered directly from the queue."`
 
-	DNSDomain      dns.Domain     `sconf:"-"` // Parsed form of Domain.
-	JunkMailbox    *regexp.Regexp `sconf:"-" json:"-"`
-	NeutralMailbox *regexp.Regexp `sconf:"-" json:"-"`
-	NotJunkMailbox *regexp.Regexp `sconf:"-" json:"-"`
+	DNSDomain                  dns.Domain     `sconf:"-"` // Parsed form of Domain.
+	JunkMailbox                *regexp.Regexp `sconf:"-" json:"-"`
+	NeutralMailbox             *regexp.Regexp `sconf:"-" json:"-"`
+	NotJunkMailbox             *regexp.Regexp `sconf:"-" json:"-"`
+	ParsedFromIDLoginAddresses []smtp.Address `sconf:"-" json:"-"`
 }
 
 type JunkFilter struct {

config/doc.go

@@ -386,6 +386,35 @@ See https://pkg.go.dev/github.com/mjl-/sconf for details.
 				# limiting and for the "secure" status of cookies. (optional)
 				Forwarded: false
 
+			# Like WebAPIHTTP, but with plain HTTP, without TLS. (optional)
+			WebAPIHTTP:
+				Enabled: false
+
+				# Default 80 for HTTP and 443 for HTTPS. (optional)
+				Port: 0
+
+				# Path to serve requests on. (optional)
+				Path:
+
+				# If set, X-Forwarded-* headers are used for the remote IP address for rate
+				# limiting and for the "secure" status of cookies. (optional)
+				Forwarded: false
+
+			# WebAPI, a simple HTTP/JSON-based API for email, with HTTPS (requires a TLS
+			# config). Default path is /webapi/. (optional)
+			WebAPIHTTPS:
+				Enabled: false
+
+				# Default 80 for HTTP and 443 for HTTPS. (optional)
+				Port: 0
+
+				# Path to serve requests on. (optional)
+				Path:
+
+				# If set, X-Forwarded-* headers are used for the remote IP address for rate
+				# limiting and for the "secure" status of cookies. (optional)
+				Forwarded: false
+
 			# Serve prometheus metrics, for monitoring. You should not enable this on a public
 			# IP. (optional)
 			MetricsHTTP:
@@ -855,6 +884,53 @@ See https://pkg.go.dev/github.com/mjl-/sconf for details.
 	Accounts:
 		x:
 
+			# Webhooks for events about outgoing deliveries. (optional)
+			OutgoingWebhook:
+
+				# URL to POST webhooks.
+				URL:
+
+				# If not empty, value of Authorization header to add to HTTP requests. (optional)
+				Authorization:
+
+				# Events to send outgoing delivery notifications for. If absent, all events are
+				# sent. Valid values: delivered, suppressed, delayed, failed, relayed, expanded,
+				# canceled, unrecognized. (optional)
+				Events:
+					-
+
+			# Webhooks for events about incoming deliveries over SMTP. (optional)
+			IncomingWebhook:
+
+				# URL to POST webhooks to for incoming deliveries over SMTP.
+				URL:
+
+				# If not empty, value of Authorization header to add to HTTP requests. (optional)
+				Authorization:
+
+			# Login addresses that cause outgoing email to be sent with SMTP MAIL FROM
+			# addresses with a unique id after the localpart catchall separator (which must be
+			# enabled when addresses are specified here). Any delivery status notifications
+			# (DSN, e.g. for bounces), can be related to the original message and recipient
+			# with unique id's. You can login to an account with any valid email address,
+			# including variants with the localpart catchall separator. You can use this
+			# mechanism to both send outgoing messages both with and without unique fromid for
+			# a given address. (optional)
+			FromIDLoginAddresses:
+				-
+
+			# Period to keep messages retired from the queue (delivered or failed) around.
+			# Keeping retired messages is useful for maintaining the suppression list for
+			# transactional email, for matching incoming DSNs to sent messages, and for
+			# debugging. The time at which to clean up (remove) is calculated at retire time.
+			# E.g. 168h (1 week). (optional)
+			KeepRetiredMessagePeriod: 0s
+
+			# Period to keep webhooks retired from the queue (delivered or failed) around.
+			# Useful for debugging. The time at which to clean up (remove) is calculated at
+			# retire time. E.g. 168h (1 week). (optional)
+			KeepRetiredWebhookPeriod: 0s
+
 			# Default domain for account. Deprecated behaviour: If a destination is not a full
 			# address but only a localpart, this domain is added to form a full address.
 			Domain:
@@ -1233,8 +1309,8 @@ See https://pkg.go.dev/github.com/mjl-/sconf for details.
 # Examples
 
 Mox includes configuration files to illustrate common setups. You can see these
-examples with "mox example", and print a specific example with "mox example
-<name>". Below are all examples included in mox.
+examples with "mox config example", and print a specific example with "mox
+config example <name>". Below are all examples included in mox.
 
 # Example webhandlers