Use pulumi convert to automate conversion of examples #1999

viveklak · 2022-06-01T22:09:42Z

Proposed changes

This change partially addresses #1981 - i.e. it adds support for yaml but not java which pulumi convert doesn't currently support. In the process it automates the conversion of canonical yaml examples to the various languages.

Related issues (optional)

#1981

github-actions · 2022-06-01T22:14:20Z

Does the PR have any schema changes?

Looking good! No breaking changes found.
No new resources/functions.

github-actions · 2022-06-01T22:37:07Z

Does the PR have any schema changes?

Looking good! No breaking changes found.
No new resources/functions.

github-actions · 2022-06-01T22:56:47Z

Does the PR have any schema changes?

Looking good! No breaking changes found.
No new resources/functions.

lblackstone · 2022-06-01T23:03:08Z

sdk/nodejs/apps/v1/statefulSet.ts

 *         selector: {
 *             matchLabels: {
 *                 app: "nginx",
 *             },
 *         },
- *         serviceName: nginxService.metadata.name,
- *         replicas: 3,
+ *         serviceName: service.metadata.apply(metadata => metadata?.name),


Side note: I'm guessing the .metadata.name property isn't marked as required in the schema, which results in the ugly apply rather than the lifted form.

Yeah interestingly this is one of the reasons to use pulumi convert over kube2pulumi here - since it allows us to embed resource references and produce more pulumi-esque code.

lblackstone

Loving this change overall! It would also be nice to connect it to the build process, but we could defer that later since the convert function is currently a bit slow.

lblackstone · 2022-06-02T18:04:44Z

provider/pkg/gen/examples/upstream/generate.go

+	md := strings.NewReplacer(".yaml", ".md", ".yml", ".md").Replace(base)
+
+	buf := bytes.Buffer{}
+	_, err = buf.WriteString("{{% examples %}}\n")


I think it would be easier to generate the markdown file using a template, but I won't block on this since the automation is already an improvement over the status quo.

Yeah I thought about that. Definitely worth revisiting this with a template. There are still some rough edges with pulumi convert - as in it tries to install dependencies etc. which may fail. Mostly think of this as a script for now which can be run to update the examples as they are added. This is not even integrated into CI.

lblackstone · 2022-06-02T18:06:02Z

provider/pkg/gen/examples/upstream/generate.go

+		contract.IgnoreError(os.RemoveAll(dir))
+	}()
+
+	fmt.Fprintf(os.Stderr, "New dir: %q\n", dir)


Should this be on stderr?

lblackstone · 2022-06-02T18:07:51Z

provider/pkg/gen/examples/upstream/ingress.md

+description: Create an Ingress with auto-naming
+name: yaml-example
+resources:
+    ingress:


It might be helpful to name the resources something different than the type to make the usage clearer.

lblackstone · 2022-06-08T20:41:22Z

Related: #2001

…into vl/yamlgen

github-actions · 2022-07-07T22:04:48Z

Does the PR have any schema changes?

Looking good! No breaking changes found.
No new resources/functions.

github-actions · 2022-07-07T22:55:47Z

Does the PR have any schema changes?

Looking good! No breaking changes found.
No new resources/functions.

github-actions · 2022-07-07T23:00:48Z

Does the PR have any schema changes?

Looking good! No breaking changes found.
No new resources/functions.

github-actions · 2022-07-07T23:02:08Z

Does the PR have any schema changes?

Looking good! No breaking changes found.
No new resources/functions.

github-actions · 2022-07-07T23:10:09Z

Does the PR have any schema changes?

Looking good! No breaking changes found.
No new resources/functions.

github-actions · 2022-07-07T23:11:12Z

Does the PR have any schema changes?

Looking good! No breaking changes found.
No new resources/functions.

github-actions · 2022-07-08T22:38:06Z

Does the PR have any schema changes?

Looking good! No breaking changes found.
No new resources/functions.

github-actions · 2022-07-11T18:47:57Z

Does the PR have any schema changes?

Looking good! No breaking changes found.
No new resources/functions.

lblackstone

Thanks for cleaning this up! This will be very helpful.

lblackstone · 2022-07-11T19:54:07Z

provider/pkg/gen/examples/upstream/yaml/service.yaml

-      targetPort: 9376
+name: yaml-example
+runtime: yaml
+description: Create a Service with autonaming


Suggested change

description: Create a Service with autonaming

description: Create a Service with auto-naming

lblackstone · 2022-07-11T19:54:17Z

provider/pkg/gen/examples/upstream/yaml/statefulset.yaml

-    app: nginx
+name: yaml-example
+runtime: yaml
+description: Create a StatefulSet with autonaming


Suggested change

description: Create a StatefulSet with autonaming

description: Create a StatefulSet with auto-naming

lblackstone · 2022-07-11T19:57:47Z

.github/workflows/run-acceptance-tests.yml

@@ -525,3 +525,32 @@ jobs:
    name: lint
    if: github.event_name == 'repository_dispatch' ||
      github.event.pull_request.head.repo.full_name == github.repository
+  ensure-examples-gen:


I don't think we want to add this to the acceptance tests at this time. The convert process is a bit slow, and the examples are currently static. Let's add a Makefile target instead, and then we can update the examples as needed.

github-actions · 2022-07-11T20:59:12Z

Does the PR have any schema changes?

Looking good! No breaking changes found.
No new resources/functions.

lblackstone · 2022-07-12T16:28:20Z

/run-acceptance-tests

pulumi-bot · 2022-07-12T16:29:55Z

Please view the PR build: https://github.com/pulumi/pulumi-kubernetes/actions/runs/2657977474

github-actions · 2022-07-12T17:32:14Z

Does the PR have any schema changes?

Looking good! No breaking changes found.
No new resources/functions.

github-actions · 2022-07-12T17:33:44Z

Does the PR have any schema changes?

Looking good! No breaking changes found.
No new resources/functions.

github-actions · 2022-07-12T17:34:45Z

Does the PR have any schema changes?

Looking good! No breaking changes found.
No new resources/functions.

github-actions · 2022-07-12T18:18:15Z

Does the PR have any schema changes?

Looking good! No breaking changes found.
No new resources/functions.

We have multiple ways of generating docs, and including docs overlays for the Docker:Image resource was done via this tool. We have since moved to keeping "extra" documentation as part of the provider repository, examples include pulumi/pulumi-databricks#38 and pulumi/pulumi-kubernetes#1999, as well as altering documentation in our terraform provider forks. We have the ability to keep example documentation in the schema for Docker:Image as of pulumi-docker v4, and therefore this functionality is obsolete, and removal is a key step in consolidating how we do supplemental provider documentation.

viveklak requested a review from lblackstone June 1, 2022 22:09

viveklak added the impact/no-changelog-required This issue doesn't require a CHANGELOG update label Jun 1, 2022

viveklak changed the title ~~Use pulumi convert to automate creation of examples~~ Use pulumi convert to automate conversion of examples Jun 1, 2022

Use pulumi convert to automate conversion of examples

71cc790

Adding schema and SDK changes

5275a90

viveklak force-pushed the vl/yamlgen branch from 5c6a376 to 5275a90 Compare June 1, 2022 22:51

lblackstone reviewed Jun 1, 2022

View reviewed changes

lblackstone approved these changes Jun 2, 2022

View reviewed changes

hliuson self-assigned this Jul 7, 2022

hliuson added 2 commits July 7, 2022 14:55

Merge branch 'master' of https://github.com/pulumi/pulumi-kubernetes …

9c9094f

…into vl/yamlgen

merged in main

031f96d

hliuson and others added 2 commits July 7, 2022 15:51

CI (#2061)

a94e6bf

generate examples on PR

03c803e

renamed workflow file

2d64e1e

hliuson added 5 commits July 7, 2022 16:05

fix formatting

f22ce7a

tweak

f6d2ed0

tweak

9aad66e

tweak

c88bbcf

tweak

9a19dd7

tweak

8eb70cc

hliuson added 4 commits July 7, 2022 16:13

tweak

44d8ed7

tweak

1bd42a7

refactored generation

e7a61fa

reworked generation

5cc1bc7

check examples generated during tests

d8e8b65

hliuson requested a review from lblackstone July 11, 2022 18:43

lblackstone requested changes Jul 11, 2022

View reviewed changes

makefile and description changes

ab60cf8

hliuson requested a review from lblackstone July 11, 2022 20:55

lblackstone approved these changes Jul 12, 2022

View reviewed changes

lblackstone force-pushed the vl/yamlgen branch from d0d2396 to ab60cf8 Compare July 12, 2022 17:29

regenerated schema and SDKs

b059e89

Pin Helm release to fix test

9a35d11

lblackstone merged commit f93f9b6 into master Jul 12, 2022

pulumi-bot deleted the vl/yamlgen branch July 12, 2022 19:12

guineveresaenger mentioned this pull request Dec 7, 2022

Add code to generate example code from YAML for registry examples pulumi/pulumi-docker#440

Closed

guineveresaenger mentioned this pull request Feb 3, 2023

Add Example documentation for Image pulumi/pulumi-docker#479

Merged

guineveresaenger mentioned this pull request Feb 4, 2023

Remove overlay functionality pulumi/registrygen#17

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use pulumi convert to automate conversion of examples #1999

Use pulumi convert to automate conversion of examples #1999

viveklak commented Jun 1, 2022

github-actions bot commented Jun 1, 2022

github-actions bot commented Jun 1, 2022

github-actions bot commented Jun 1, 2022

lblackstone Jun 1, 2022

viveklak Jun 2, 2022

lblackstone left a comment

lblackstone Jun 2, 2022

viveklak Jun 3, 2022

lblackstone Jun 2, 2022

lblackstone Jun 2, 2022

lblackstone commented Jun 8, 2022

github-actions bot commented Jul 7, 2022

github-actions bot commented Jul 7, 2022

github-actions bot commented Jul 7, 2022

github-actions bot commented Jul 7, 2022

github-actions bot commented Jul 7, 2022

github-actions bot commented Jul 7, 2022

github-actions bot commented Jul 8, 2022

github-actions bot commented Jul 11, 2022

lblackstone left a comment

lblackstone Jul 11, 2022

lblackstone Jul 11, 2022

lblackstone Jul 11, 2022

github-actions bot commented Jul 11, 2022

lblackstone commented Jul 12, 2022

pulumi-bot commented Jul 12, 2022

github-actions bot commented Jul 12, 2022

github-actions bot commented Jul 12, 2022

github-actions bot commented Jul 12, 2022

github-actions bot commented Jul 12, 2022

	description: Create a Service with autonaming
	description: Create a Service with auto-naming

	description: Create a StatefulSet with autonaming
	description: Create a StatefulSet with auto-naming

Use pulumi convert to automate conversion of examples #1999

Use pulumi convert to automate conversion of examples #1999

Conversation

viveklak commented Jun 1, 2022

Proposed changes

Related issues (optional)

github-actions bot commented Jun 1, 2022

Does the PR have any schema changes?

github-actions bot commented Jun 1, 2022

Does the PR have any schema changes?

github-actions bot commented Jun 1, 2022

Does the PR have any schema changes?

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lblackstone left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lblackstone commented Jun 8, 2022

github-actions bot commented Jul 7, 2022

Does the PR have any schema changes?

github-actions bot commented Jul 7, 2022

Does the PR have any schema changes?

github-actions bot commented Jul 7, 2022

Does the PR have any schema changes?

github-actions bot commented Jul 7, 2022

Does the PR have any schema changes?

github-actions bot commented Jul 7, 2022

Does the PR have any schema changes?

github-actions bot commented Jul 7, 2022

Does the PR have any schema changes?

github-actions bot commented Jul 8, 2022

Does the PR have any schema changes?

github-actions bot commented Jul 11, 2022

Does the PR have any schema changes?

lblackstone left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Jul 11, 2022

Does the PR have any schema changes?

lblackstone commented Jul 12, 2022

pulumi-bot commented Jul 12, 2022

github-actions bot commented Jul 12, 2022

Does the PR have any schema changes?

github-actions bot commented Jul 12, 2022

Does the PR have any schema changes?

github-actions bot commented Jul 12, 2022

Does the PR have any schema changes?

github-actions bot commented Jul 12, 2022

Does the PR have any schema changes?