Enhanced typings with optional environment variables

[jsamr]

This PR offers three major improvements:

• Infer optional environment variables in the cleaned environment record when explicitly setting default to undefined
• Fine-grained type inference from spec objects (enhancement from Ensure default values don't go through validation, fix type narrowing from defaults #176, also addresses the same fix as fix: validator spec functions return their generic type #181)
• Typescript type testing for validators and cleanEnv.

Remark Should probably considered breaking for TS users and shipped within a major release.

Changes

Supports optional presence of environment variables

const cleaned = cleanEnv(env, {
  STR: str(),
  STR_OPT: str({ default: undefined }),
  STR_CHOICES: str({ choices: ["foo", "bar"] }),
  STR_REQ: str({ default: "foo" }),
  STR_DEFAULT_CHOICES: str({ default: "foo", choices: ["foo", "bar"] }),
});

Output type (VScode)

Notice a few things:

• STR_OPT property is of type string | undefined
• STR_CHOICES is narrowed down to 'foo' | 'bar'
• STR_REQ is widened to type string while STR_DEFAULT_CHOICES is narrowed-down to 'foo' | 'bar'

Better JSON support

JSON output type can now be inferred from usage (default):

const cleaned = cleanEnv(env, {
  JSON_ANY: json(),
  JSON_REQ_ANY: json({ default: {} as any }),
  JSON_DEFAULT: json({ default: { foo: 'bar' } }),
  JSON_DEV_DEFAULT: json({ devDefault: { foo: 'bar' } }),
  JSON_DEFAULT_OPT: json<{ foo: 'bar' }>({ default: undefined }),
})

Output type (VScode)

Make Validator Wrappers

There are now three types of validators which can be created with distinct functions. These functions are identical runtime-wise, but provide fine-grained typings to cover different scenarios:

• makeBaseValidator<BaseT> when you want the output to be narrowed-down to a subtype of BaseT (this is used in str and all string-based internal validators).
• makeExactValidator<T> when you want the output to be widened to T (this is used in bool validator).
• makeMarkupValidator for input which can produce arbitrary output types (this is used in json validator).

makeValidator is still there, but is typed like makeBaseValidator.

Guarantee types stability with thorough types testing

Thanks to expect-type, we can test all scenarios and maintain high types quality.

envalid/tests/types.test.ts

Lines 1 to 299 in a0f5e31

	import {
	cleanEnv,
	str,
	bool,
	num,
	RequiredValidatorSpec,
	OptionalValidatorSpec,
	makeMarkupValidator,
	json,
	makeBaseValidator,
	} from '../src'
	import { expectTypeOf } from 'expect-type'
	describe('validators types', () => {
	test('boolean validator', () => {
	const validator = bool
	expectTypeOf(validator()).toEqualTypeOf<RequiredValidatorSpec<boolean>>()
	expectTypeOf(
	validator({
	default: false,
	}),
	).toEqualTypeOf<RequiredValidatorSpec<boolean>>()
	expectTypeOf(
	validator({
	choices: [true, false],
	default: true,
	}),
	).toEqualTypeOf<RequiredValidatorSpec<boolean>>()
	expectTypeOf(validator({ default: undefined })).toEqualTypeOf<OptionalValidatorSpec<boolean>>()
	expectTypeOf(validator({ devDefault: undefined })).toEqualTypeOf<
	RequiredValidatorSpec<boolean>
	>()
	expectTypeOf(validator({ devDefault: false })).toEqualTypeOf<RequiredValidatorSpec<boolean>>()
	})
	test('number-based validators', () => {
	const validator = makeBaseValidator<number>(() => 1)
	// Specifying default or devDefault value should cause validator spec type param to widen
	expectTypeOf(
	validator({
	default: 0,
	}),
	).toEqualTypeOf<RequiredValidatorSpec<number>>()
	expectTypeOf(
	validator({
	devDefault: 0,
	}),
	).toEqualTypeOf<RequiredValidatorSpec<number>>()
	// But this inference can be overridden by specifying a type parameter
	expectTypeOf(
	validator<0>({
	default: 0,
	}),
	).toEqualTypeOf<RequiredValidatorSpec<0>>()
	expectTypeOf(
	validator<0>({
	devDefault: 0,
	}),
	).toEqualTypeOf<RequiredValidatorSpec<0>>()
	// Choices
	expectTypeOf(
	validator({
	choices: [1, 2],
	}),
	).toEqualTypeOf<RequiredValidatorSpec<1 | 2>>()
	expectTypeOf(
	validator({
	choices: [1, 2],
	default: 1,
	}),
	).toEqualTypeOf<RequiredValidatorSpec<1 | 2>>()
	// @ts-expect-error - 3 is not assignable to 1 | 2
	validator({ choices: [1, 2], default: 3 })
	// @ts-expect-error - 3 is not assignable to 1 | 2
	validator({ choices: [1, 2], devDefault: 3 })
	// Basic
	expectTypeOf(validator()).toEqualTypeOf<RequiredValidatorSpec<number>>()
	expectTypeOf(validator<1>()).toEqualTypeOf<RequiredValidatorSpec<1>>()
	expectTypeOf(validator({ default: undefined })).toEqualTypeOf<OptionalValidatorSpec<number>>()
	expectTypeOf(validator({ devDefault: undefined })).toEqualTypeOf<
	RequiredValidatorSpec<number>
	>()
	expectTypeOf(validator<2>({ devDefault: 2 })).toEqualTypeOf<RequiredValidatorSpec<2>>()
	})
	test('string-based validators', () => {
	const validator = makeBaseValidator<string>(() => '')
	// Specifying default or devDefault value should cause validator spec type param to widen
	expectTypeOf(
	validator({
	default: 'foo',
	}),
	).toEqualTypeOf<RequiredValidatorSpec<string>>()
	expectTypeOf(
	validator({
	devDefault: 'foo',
	}),
	).toEqualTypeOf<RequiredValidatorSpec<string>>()
	// But this inference can be overridden by specifying a type parameter
	expectTypeOf(
	validator<'foo'>({
	default: 'foo',
	}),
	).toEqualTypeOf<RequiredValidatorSpec<'foo'>>()
	expectTypeOf(
	validator<'foo'>({
	devDefault: 'foo',
	}),
	).toEqualTypeOf<RequiredValidatorSpec<'foo'>>()
	expectTypeOf(
	validator({
	choices: ['foo', 'bar'],
	}),
	).toEqualTypeOf<RequiredValidatorSpec<'foo' | 'bar'>>()
	expectTypeOf(
	validator({
	choices: ['foo', 'bar'],
	default: 'foo',
	}),
	).toEqualTypeOf<RequiredValidatorSpec<'foo' | 'bar'>>()
	//@ts-expect-error - baz is not assignable to 'foo' | 'bar'
	validator({ choices: ['foo', 'bar'], default: 'baz' })
	// Basic
	expectTypeOf(validator()).toEqualTypeOf<RequiredValidatorSpec<string>>()
	expectTypeOf(validator<'foo'>()).toEqualTypeOf<RequiredValidatorSpec<'foo'>>()
	expectTypeOf(validator({ default: undefined })).toEqualTypeOf<OptionalValidatorSpec<string>>()
	expectTypeOf(validator({ devDefault: undefined })).toEqualTypeOf<
	RequiredValidatorSpec<string>
	>()
	expectTypeOf(validator({ default: undefined })).toEqualTypeOf<OptionalValidatorSpec<string>>()
	expectTypeOf(validator({ devDefault: undefined })).toEqualTypeOf<
	RequiredValidatorSpec<string>
	>()
	expectTypeOf(validator({ devDefault: 'foo' })).toEqualTypeOf<RequiredValidatorSpec<string>>()
	expectTypeOf(validator<'foo'>({ devDefault: 'foo' })).toEqualTypeOf<
	RequiredValidatorSpec<'foo'>
	>()
	expectTypeOf(validator({ default: 'foo', devDefault: 'foo' })).toEqualTypeOf<
	RequiredValidatorSpec<string>
	>()
	expectTypeOf(validator<'foo' | 'bar'>({ default: 'foo', devDefault: 'bar' })).toEqualTypeOf<
	RequiredValidatorSpec<'foo' | 'bar'>
	>()
	expectTypeOf(
	validator<'foo' | 'bar'>({ choices: ['foo', 'bar'], devDefault: 'bar' }),
	).toEqualTypeOf<RequiredValidatorSpec<'foo' | 'bar'>>()
	})
	test('structured data validator', () => {
	const validator = makeMarkupValidator(() => ({}))
	expectTypeOf(validator()).toEqualTypeOf<RequiredValidatorSpec<any>>()
	expectTypeOf(validator({ default: {} as any })).toEqualTypeOf<RequiredValidatorSpec<any>>()
	expectTypeOf(validator({ default: undefined })).toEqualTypeOf<OptionalValidatorSpec<any>>()
	expectTypeOf(validator({ default: undefined, choices: [{ foo: 'bar' }] })).toEqualTypeOf<
	OptionalValidatorSpec<{ foo: 'bar' }>
	>()
	expectTypeOf(validator({ devDefault: undefined })).toEqualTypeOf<RequiredValidatorSpec<any>>()
	expectTypeOf(validator({ devDefault: { foo: 'bar' } })).toEqualTypeOf<
	RequiredValidatorSpec<{ foo: string }>
	>()
	expectTypeOf(validator<{ foo: 'bar' }>()).toEqualTypeOf<RequiredValidatorSpec<{ foo: 'bar' }>>()
	expectTypeOf(
	validator({
	default: {
	hello: 'world',
	},
	}),
	).toEqualTypeOf<
	RequiredValidatorSpec<{
	hello: string
	}>
	>()
	expectTypeOf(
	validator<{ hello: 'world' }>({
	default: {
	hello: 'world',
	},
	}),
	).toEqualTypeOf<
	RequiredValidatorSpec<{
	hello: 'world'
	}>
	>()
	expectTypeOf(validator<{ hello: string }>()).toEqualTypeOf<
	RequiredValidatorSpec<{
	hello: string
	}>
	>()
	expectTypeOf(
	validator({
	choices: [
	{ hello: 'world', option: false },
	{ hello: 'world', option: true },
	],
	}),
	).toEqualTypeOf<
	RequiredValidatorSpec<{
	hello: string
	option: boolean
	}>
	>()
	expectTypeOf(
	validator({
	choices: [
	{ hello: 'world', option: false },
	{ hello: 'world', option: true },
	],
	default: { hello: 'world', option: false },
	}),
	).toEqualTypeOf<
	RequiredValidatorSpec<{
	hello: string
	option: boolean
	}>
	>()
	})
	})
	test('cleanEnv', () => {
	const env = {
	STR: 'FOO',
	STR_OPT: undefined,
	STR_CHOICES: 'foo',
	STR_REQ: 'BAR',
	STR_DEFAULT_CHOICES: 'bar',
	BOOL: 'false',
	BOOL_OPT: undefined,
	BOOL_DEFAULT: undefined,
	NUM: '34',
	NUM_DEFAULT_CHOICES: '3',
	JSON_ANY: JSON.stringify(true),
	JSON_REQ_ANY: JSON.stringify('Foo bar'),
	JSON_DEFAULT: JSON.stringify({ foo: 'bar' }),
	JSON_DEFAULT_OPT: undefined,
	}
	const specs = {
	STR: str(),
	STR_OPT: str({ default: undefined }),
	STR_CHOICES: str({ choices: ['foo', 'bar'] }),
	STR_REQ: str({ default: 'foo' }),
	STR_DEFAULT_CHOICES: str({ default: 'foo', choices: ['foo', 'bar'] }),
	BOOL: bool(),
	BOOL_OPT: bool({ default: undefined }),
	BOOL_DEFAULT: bool({
	default: false,
	}),
	NUM: num(),
	NUM_DEFAULT_CHOICES: num({ default: 1, choices: [1, 2, 3] }),
	JSON_ANY: json(),
	JSON_REQ_ANY: json({ default: {} as any }),
	JSON_DEFAULT: json({ default: { foo: 'bar' } }),
	JSON_DEV_DEFAULT: json({ devDefault: { foo: 'bar' } }),
	JSON_DEFAULT_OPT: json<{ foo: 'bar' }>({ default: undefined }),
	}
	interface TestedCleanedEnv {
	readonly STR: string
	readonly STR_OPT?: string
	readonly STR_CHOICES: 'foo' | 'bar'
	readonly STR_REQ: string
	readonly STR_DEFAULT_CHOICES: 'foo' | 'bar'
	readonly BOOL: boolean
	readonly BOOL_OPT?: boolean
	readonly BOOL_DEFAULT: boolean
	readonly NUM: number
	readonly NUM_DEFAULT_CHOICES: 1 | 2 | 3
	readonly JSON_ANY: any
	readonly JSON_REQ_ANY: any
	readonly JSON_DEFAULT: { foo: string }
	readonly JSON_DEV_DEFAULT: { foo: string }
	readonly JSON_DEFAULT_OPT?: { foo: string }
	}
	expectTypeOf(cleanEnv(env, specs)).toMatchTypeOf<TestedCleanedEnv>()
	// Should also work when specs inlined
	expectTypeOf(
	cleanEnv(env, {
	STR: str(),
	STR_OPT: str({ default: undefined }),
	STR_CHOICES: str({ choices: ['foo', 'bar'] }),
	STR_REQ: str({ default: 'foo' }),
	STR_DEFAULT_CHOICES: str({ default: 'foo', choices: ['foo', 'bar'] }),
	BOOL: bool(),
	BOOL_OPT: bool({ default: undefined }),
	BOOL_DEFAULT: bool({
	default: false,
	}),
	NUM: num(),
	NUM_DEFAULT_CHOICES: num({ default: 1, choices: [1, 2, 3] }),
	JSON_ANY: json(),
	JSON_REQ_ANY: json({ default: {} as any }),
	JSON_DEFAULT: json({ default: { foo: 'bar' } }),
	JSON_DEV_DEFAULT: json({ devDefault: { foo: 'bar' } }),
	JSON_DEFAULT_OPT: json<{ foo: 'bar' }>({ default: undefined }),
	}),
	).toMatchTypeOf<TestedCleanedEnv>()
	})

[jsamr]

@af should be good!

[af]

Thanks for this and sorry for the delay in looking at it! Hoping to review and test it out this weekend 👍

[af]

Added some initial feedback, mostly about the exposed API and docs.

First of all, the JSON inference is super nice! And I really appreciate that you tidied up a lot of the kludges I still had in place from migrating this thing to TS.

Some of the types are pretty complex so I'm still digesting them. Overall looking very good though!

[af]

I don't know if it makes sense to expose makeMarkupValidator in the public API. What other use cases can you think of for it? If we keep it private for now we can always expose it later. Given the domain of "parsing env var strings" I think json() would be the dominant case

[jsamr]

I was thinking query parameters, e.g. "option1=true&option2=false&token=XXX"; or POSIX like flags e.g. -b -c --long-flag=/bin/xxx, /etc/fstab, XML, YAML... JSON5; honestly I don't know much about backend developers practices , but I had imagined exposing this one would give plenty of flexibility for consumers of this library, freeing them from the hassle of typings these validators correctly.

PS1: didn't test but I think we should also forbid specs with choices in markup validators, correct?
PS2: I went back and forth between naming this validator markupValidator and structuredValidator ; English is not my native language and if you have better naming ideas I'm all for it

[af]

I think structuredValidator would be a better name for sure, when most people hear "markup" they probably think of a specific type of format like HTML/XML/etc.

Also re choices, a nice side effect of removing structuredValidator from the public API is we don't have to worry about that interaction :)

[jsamr]

Also re choices, a nice side effect of removing structuredValidator from the public API is we don't have to worry about that interaction :)

Well, with one important exception: json validator. In any case, I did implement this specific logic here:
328e056

Yet, I'm happy to not export makeStructuredValidator if you don't want to.

[af]

Could probably use examples of each case (makeValidator vs makeExactValidator) here as I don't think it'll be clear to most readers without diving into the source

[jsamr]

Let me know if that's enough: 9727612

[jsamr]

Couldn't find the comment about formatting again; but fixed it here: ce1007f

[jsamr]

First of all, the JSON inference is super nice! And I really appreciate that you tidied up a lot of the kludges I still had in place from migrating this thing to TS.

Some of the types are pretty complex so I'm still digesting them. Overall looking very good though!

Hey @af ! It's cool if this PR proves useful. Although it was hard at times, I had a lot of fun with it. I don't know if it will help, but perhaps a trick to read those validator types is following:

// Such validator only works for subtypes of BaseT.
export interface BaseValidator<BaseT> {
  // These overloads enable nuanced type inferences for optimal DX
  // This will prevent specifying "default" alone from narrowing down output type
  (spec: RequiredChoicelessSpecWithType<BaseT>): RequiredValidatorSpec<BaseT>
  <T extends BaseT>(spec?: RequiredSpec<T>): RequiredValidatorSpec<T>
  <T extends BaseT>(spec: OptionalSpec<T>): OptionalValidatorSpec<T>
}

Think of it as function overloads. For instance, this:

export const num = makeValidator<number>((input: string) => {
  const coerced = parseFloat(input)
  if (Number.isNaN(coerced)) throw new EnvError(`Invalid number input: "${input}"`)
  return coerced
})

is strictly equivalent to this:

export function num(spec: RequiredChoicelessSpecWithType<number>): ValidatorSpec<number>
export function num<T extends number>(spec?: RequiredSpec<T>): RequiredValidatorSpec<T>
export function num<T extends number>(spec: OptionalSpec<T>): OptionalValidatorSpec<T>
export function num(spec?: Spec<number>): ValidatorSpec<number> {
  return internalMakeValidator((input: string) => {
    const coerced = parseFloat(input)
    if (Number.isNaN(coerced)) throw new EnvError(`Invalid number input: "${input}"`)
    return coerced
  })
}

In the call site, TypeScript will simply try each of these signatures, starting at the topmost, and stop at the one matching the combination of type parameters and arguments. E.g.:

const MY_NUM_VAR = num<1|2|3>();

In this instance, TypeScript picks the second overload from the top because:

• There is a type parameter;
• There is exactly zero argument.

[af]

A few requested changes but looking good. Almost ready to merge!

[af]

Thanks for the revisions! Latest CI run failed but once that's resolved I'll merge 👍

[jsamr]

@af Should be good now!

[jsamr]

@af Please excuse the disturbance; I have a small request: would you be OK to release an alpha/beta with these changes? We use this package extensively in multiple projects, so I'll be able to beta-test this with reasonable coverage.

[af]

Thanks for the nudge, I'd meant to do that but it got lost in the shuffle. Just published 8.0.0-alpha.1, let me know how it goes!

[jsamr]

@af Happy to report that we've been using alpha.2 version in our 6 projects, and it's been all smooth!

[af]

Great to hear, thanks! I will take it for a spin on a few of my projects as well, then after some long-overdue docs updates I'll push out a beta unless anything comes up