-
Notifications
You must be signed in to change notification settings - Fork 9.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
darwin/arm64 build #27257
Comments
Hi @dansimau! Thanks for raising this. There are some things that must happen before we could produce official Terraform releases on arm64 macOS, or before such releases would be practically useful:
Given that Apple's strategy for gradual ecosystem migration is emulation, I feel optimistic that problems in the current version of the emulation layer will be gradually addressed as the Apple team becomes aware of the problems, so hopefully future updates to the arm64 version of macOS will address the problems that are currently causing the assertion failure you saw. We'll monitor Go releases to see if there are any minor release updates to Go 1.15 that might be issued to improve compatibility with Apple's emulator when running x86_64 binaries, in which case we may be able to include them in v0.14.x minor releases. |
Makes sense. After using it a bit more, I have concluded that Terraform is basically unusable under emulation on macOS arm64 (Big Sur 11.01). Here are a sample of some more errors I have been getting:
(Eventually I killed the provider processes).
(Is the second time I have had to reinit).
At the moment it's roughly a coin toss as to whether a command will work or not. |
@dansimau as it happens, I just got a an M1 Macbook, so I'm able to reproduce this. I did a state import of a security group, and it worked perfectly for me. However, I have my shell set to run under emulation by default, and when I use the default shell, I (on one run, haven't been able to reproduce since then) see the same behavior as you where it hangs waiting for the plugin to start. Can you try running a shell under rosetta with a command like |
@danieldreier I was already running an x86_64 bash (v5), because I was using |
Just FWIW, another symptom: I randomly opened Activity Monitor and found this provider process using 100% CPU. Not even sure where it came from because I stopped running Terraform locally (unless the vscode language server runs it?) Anyway, I will update to macOS 11.1 and see if there is any change (there is at least one report of Rosetta bug fixes in 11.1). Side note, I had a lot of issues with other (non-Terraform) Go binaries compiled for darwin/x86_64 (e.g. my VSCode environment ground to a halt). I compiled Go HEAD for arm64, trashed my $GOHOME and reinstalled/recompiled everything. Now that environment is mostly working and stable (which is to be expected) with the notable exception of anything using cgo. |
I had same problem when kubectl & terraform x86 went crazy and ate up a lot of CPU. But upgrading to macOS 11.1 fixed that. |
Upgraded to 11.1 and it didn't change anything. The first time I ran Terraform it hung with the same symptoms (provider at ~100% CPU). Here's a sample of the process: |
For v0.12 users, |
For anyone who just wants to get on with being able to do their job...
Make sure your brew bin is in your PATH (e.g. Make sure to restart your terminal if you've just changed your PATH or installed terraform to
Then copy from
Then back in your workspace:
It should pick up the arm64 build from your global plugins (https://www.terraform.io/docs/commands/cli-config.html#implied-local-mirror-directories) The same basic process should work for all providers The resultant |
The strategy of building the providers locally for unsupported architectures is certainly a possible workaround. Thanks for sharing the instructions, @billinghamj. One thing I'd add here is that following your instructions exactly will cause the If you are in an environment where you need to mix both supported and unsupported platforms then probably the best approach would be to review the git diff created by that final Relatedly, I'm not sure if the local build process described here serves as a reproducible build such that two different people running those steps on their own laptops would end up producing identical binaries. If not, you may run into similar trouble if you have multiple people in your environment that are using this new platform and creating their own builds. To mitigate that, you could produce one "official-within-my-company" build of the provider and share it with all users, thus causing everyone to agree on what the checksum ought to be. |
Yeah, in my case I'm just probably not going to commit the modified lock file, and we likely will just share the binaries between each other too. |
(pprof) top I have the same problem on mac M1 |
Today on % make tools && make build
cd awsproviderlint && GO111MODULE=on go install .
cd tools && GO111MODULE=on go install github.com/bflad/tfproviderdocs
cd tools && GO111MODULE=on go install github.com/client9/misspell/cmd/misspell
cd tools && GO111MODULE=on go install github.com/golangci/golangci-lint/cmd/golangci-lint
cd tools && GO111MODULE=on go install github.com/katbyte/terrafmt
../../../go/pkg/mod/github.com/hashicorp/[email protected]/tfinstall/download.go:12:2: missing go.sum entry for module providing package github.com/hashicorp/go-getter (imported by github.com/hashicorp/terraform-exec/tfinstall); to add:
go get github.com/hashicorp/terraform-exec/[email protected]
make: *** [tools] Error 1 But even with the local build described by @billinghamj in our environment there are at least 2 other providers requiring attention for the Apple Silicon M1 ("github" and "external" providers in our case):
Inconvenient but totally doable: just compiled Obviously, for this to be practical, a concerted effort to test and CI as many plugins as possible for this to be usable in day-to-day devops tasks at this point in time, unfortunately :/ Does @hashicorp have some kind of (public) roadmap for this Apple M1 porting effort somewhere other than the |
I think it's really crazy that Hashicorp are refusing to touch this until Go 1.16 is released. The beta/RC is drastically more stable than running TF in Rosetta, which is what thousands of engineers are now being forced to do. Brew has already been distributing TF built with 1.16 for M1 and it works perfectly. I really don't think there's any excuse or justification for the current stance :\ Every other tool and runtime we use has been working flawlessly for months at this point. Terraform is the sole outlier. |
Since the providers all communicate with Terraform over gRPC, I anticipate (educated speculation) that Terraform core needs to go first (since everything is driven through that). As long as the other providers aren't broken on arm64 and can run in Rosetta, it seems plausible that they could continue to work via Rosetta since CPU instructions wouldn't directly affect gRPC communication between the provider and Core. Separately, @brainstorm wrote:
I would love a centralized resource, but I don't think there's anything requiring Hashicorp to put it together. Any member of the community could put together a dashboard, and ask people to contribute the up-to-date information for the providers they care about. That's all that Hashicorp would be doing if they were to do it themselves, I suspect. I started filing tickets for the providers I care about. There's nothing preventing others from doing the same and linking them together. I don't have the bandwidth at the moment to lead such an effort, but I'd happily participate since I'm watching those providers pretty closely anyway. Inspiration: |
Go 1.16 is out now. Please could this be prioritised reasonably highly so providers can start accepting PRs etc to add arm support? |
|
right, so I followed the instructions above, and built 2 providers. that got me past the terraform init stage, but when I tried to terraform apply (using remote backend ) it gives me these errors.
|
@apparentlymart v0.15 just came out but seems to not be using Go 1.16 still? |
@billinghamj the go.mod is still using 1.14 as the minimum though? |
The entry in The |
Ah that makes sense. So I'll go ahead and build it now on my own. Do you expect you guys to add darwin_arm64 to the releases directory anytime soon? |
Due to the requirement that the macOS builds be notarized via HashiCorp's Apple developer credentials that our team doesn't have direct access to, we're relying on support from a centralized release engineering team to add this new platform to the release pipeline. Their schedule is not under our direct control, so I can't say with any certainty when we'll be able to get that included, but on the plus side it's a process step common to all HashiCorp products distributed in this way so I expect that the approach for one will be the approach for all, once it's designed, and there are similar requests pending from all of the other product teams too. If all of the providers you intend to use have |
It's available on brew already too: https://github.com/Homebrew/homebrew-core/blob/master/Formula/terraform.rb So don't really need to wait for Hashicorp to post arm64 builds if you don't want |
Imho we do need to wait (or compile) since this will require the notarized plugins which are not available yet. |
@andig The plugins work fine with the brew version already - they have been working great for a couple of months now, and @apparentlymart seems to have indicated that the plugins do not need to be notarized. |
@billinghamj as far as I understand terraform will need to download providers that match the architecture. For that purpose it is not sufficient to have a brew arm64 binary:
If the respective provider is available with matching platform things are fine again:
|
I have hacked together a quick way to check which versions of a provider support macOS on arm64: PROVIDER="hashicorp/google"
curl --silent https://registry.terraform.io/v1/providers/${PROVIDER}/versions | jq -r '.versions[] | select(.platforms[] | contains({os: "darwin", arch: "arm64"})) | .version' the above would for instance output:
I haven't found any other obvious way of detecting which platforms each provider version are available as, so this could be handy for figuring out when all the providers you use are available for arm64. |
@andig Yeah that's right. A lot of providers have already published arm64/darwin binaries, but indeed some have not yet. It looks like hashicorp/google does as of v3.63.0 though. But either way, notarization does not appear to be required for plugins - just updating to Go 1.16 |
For those encountering this issue with
Here's a workaround to run it on darwin/arm64:
Hope it helps. |
Work or me only when file copied to ~/.terraform.d/plugins/registry.terraform.io/hashicorp/template/2.2.0/darwin_arm64/ |
Reference: hashicorp/terraform#27257 Next release will automatically include `darwin/arm64` port.
I am working in a lot of different Terraform projects and many of them have pinned providers version to old version. So either they are pre-darwn/arm64 or don't have a pre-compiled binary. After following the workarounds manually every time I decided to write a small CLI to handle all the details: https://github.com/kreuzwerker/m1-terraform-provider-helper Hope this of help to anyone! |
Hi all, Terraform v1.0.2 is the first release to include an official build for macOS on arm64. We've not yet added this new platform to our supported platforms in the v1.0 compatibility promises. Although we have tested this build on our own systems and believe it to work correctly, we'd like to let this new version soak for a release or two before we commit to compatibility with it in the same sense as we have for the other supported platforms, just in case we learn from feedback that something isn't behaving in an ideal way and thus we may need to make adjustments to some details. With that said, we don't have any specific plans to change or remove that release in the future, and so this is still an official release that we recommend for use, since our testing suggests that it is functioning correctly. We intend to add it as a fully-supported platform in the near future. If you try it and run into strange behaviors that you didn't previously see with the macOS amd64 releases then please open a new GitHub issue about that and let us know. The supported platforms for individual providers are decided by the teams who release those providers and so we (the Terraform Core team) can't promise that all providers you use will have matching support for Thanks for your patience while we worked through getting this implemented. Properly supporting a new platform is not a trivial matter but fortunately it's a relatively rare event! Again, please open a new bug report issue if you encounter problems with the new build, so that we can investigate each distinct problem separately. |
I didn't see an existing issue, so I thought I'd open an issue to track building an arm64 (Apple Silicon) binary for macOS.
After migrating to a new Mac, I have seen at least one issue using Terraform almost straight away:
(CTRL-C did not kill it; when I killed the provider processes they became zombies; I had to SIGKILL the main terraform process.)
Current Terraform Version
References
The text was updated successfully, but these errors were encountered: