macos-latest runners do not always have the same OS version

[paulmueller]

Description

The readme for the macOS 12 runner says that the OS version is 12.7.

However, I am seeing runners with 12.6.8 and 12.6.9 when I set the os to "macos-latest".

Is this behavior intended?

Background: My problem is apparently that my tests fail with the macOS 12.6.9 image but succeed with the macOS 12.6.8 and 12.7 images.

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 20.04
• Ubuntu 22.04
• macOS 11
• macOS 12
• macOS 13
• Windows Server 2019
• Windows Server 2022

Image version and build link

macOS 12.7: https://github.com/DCOR-dev/ckanext-dcor_schemas/actions/runs/6619079303/job/17978854847
macOS 12.6.9: https://github.com/DCOR-dev/ckanext-dcor_schemas/actions/runs/6623038566/job/17989672981

Is it regression?

no

Expected behavior

If I specify macos-latest, I expect to always get the same runner image with currently macOS 12.7

Actual behavior

Sometimes macOS 12.6.8 or 12.6.9 images are used.

Repro steps

Just use the macos-latest runner.

[ilia-shipitsin]

Hello, @paulmueller . it is not regarding images, I'd say it is how VM is chosen for your build.
I'll ask colleagues

[jan-janssen]

The issue is in the images:

Current runner version: '2.311.0'
Operating System
  macOS
  12.7
  21G816
Runner Image
  Image: macos-12
  Version: 20230921.4
  Included Software: https://github.com/actions/runner-images/blob/macOS-12/20230921.4/images/macos/macos-12-Readme.md
  Image Release: https://github.com/actions/runner-images/releases/tag/macOS-12%2F20230921.4

vs.

Current runner version: '2.311.0'
Operating System
  macOS
  12.6.9
  21G726
Runner Image
  Image: macos-12
  Version: 20231017.6
  Included Software: https://github.com/actions/runner-images/blob/macOS-12/20231017.6/images/macos/macos-12-Readme.md
  Image Release: https://github.com/actions/runner-images/releases/tag/macOS-12%2F20231017.6

The version number of the 12.6.9 image 20231017.6 is higher than the version number of the 12.7 image 20230921.4 so the runner chooses the image with 12.6.9 rather than 12.7. Is there anyway to enforce the image version ?

The link for the image seems to be broken as well: https://github.com/actions/runner-images/releases/tag/macOS-12%2F20231017.6 vs. https://github.com/actions/runner-images/releases/tag/macOS-12%2F20230921.4 . So it seems to me the image should be removed.

[Release-Candidate]

Another problem of these images is that different Mac OS 12 images use either Python 2 or Python 3 when using shell: python in the workflow YAML, depending on the exact runner: #8842

[akhilamoyila9]

It appears that there is inconsistency in the macOS versions used by the GitHub Actions runners labeled as macos-12. In some cases, the runner reports macOS version 12.6.9, while in other instances, it reports macOS version 12.7.1.

macOS 12.6.9: https://github.com/magma/magma/actions/runs/6963469881/job/18952865170
macOS 12.7.1: https://github.com/magma/magma/actions/runs/6956320863/job/18926933323
This variability is unexpected as macos-12 is generally expected to provide the most recent macOS version.

[ilia-shipitsin]

sorry for delayed reply.

it is not only version split 12.7.1 / 12.6.9, it is rather underlying hardware difference.
macos 12 build are routed to 2 kind of virtual machines . It is not something we can address on image generation level, unfortunately.

some day all builds hopefully be routed to single hardware (I was told that it should happen in 2024).

also, some statement from my colleagues (regarding difference in virtualization on those hardware)

We invested in upgraded hardware to support macOS 13 and are retiring older hardware, both of these changes impact macOS 12. The older generation Mac hardware was more efficient at nested virtualization. We recommend using larger linux runners for workflows that require nested virtualization. More information can be found here.

[tapasmishra]

it is not only version split 12.7.1 / 12.6.9, it is rather underlying hardware difference.
macos 12 build are routed to 2 kind of virtual machines . It is not something we can address on image generation level, unfortunately.

@ilia-shipitsin As per my understanding, the runners always use the latest image version, and that is one of the reasons why the option to specify/use a particular image version is restricted in the workflows. If my understanding is correct, then no matter which hardware the job is getting routed to, it should use the latest OS image version. I am a little confused when you mention two different types of hardware using two different versions of the same OS. It sounds illogical. Can you help us identify the correct repository to raise an issue if this issue is not related to runner images?

[ilia-shipitsin]

your understanding absolutely makes sense.
I suggest to try https://support.github.com , but I heard from colleagues that for some period of time we'll be running macos 12 like that, unfortunately

[DeeDeeG]

The link for the image seems to be broken as well: https://github.com/actions/runner-images/releases/tag/macOS-12%2F20231017.6 vs. https://github.com/actions/runner-images/releases/tag/macOS-12%2F20230921.4 .

I can confirm this is still happening, the links being broken for (some?) macOS 12 runners.

It is somewhat troublesome for me when the links are broken/docs aren't available for the runner version, since I'm trying to debug some obscure failures in my app that are only happening in the CI context, and I can't review documentation of which software is on the runner, so debugging is much harder. For example, knowing the Python version is very helpful, but I will probably have to throw some which -a python3 into my workflow yaml to find this out, and so on for every tool I want to know which version is in the runner.

The usual documentation here is very strong, so not having access to it makes me appreciate what is usually there all the more, and miss it in its absence. Thanks for all your work on GitHub Actions / Azure DevOps. I hope this aspect of the missing/broken documentation (+ links) can be resolved. Thank you for considering.

[joepeding]

I'm running into this issue too. I have a workflow to build my library for and run tests on MacOS; but it requires installing MySQL 8.1. Something as simple as brew install [email protected] worked for a while, but recently broke.

Rerunning the same workflow, I see versions 12.6.9 and 12.7.1 and 12.7.2 and it only gets past the brew install [email protected] step on 12.6.9; but this took me 3 succesful and 6 failing runs to work out.

[hluaces]

Most of my jobs are using old images (20231029) which break all my workflows. This has been happening ever since #8730 was closed.

[mikhailkoliada]

The problem have been resolved now, sorry for inconvenience!

[jan-janssen]

At least for me the issue is not fixed. I see randomly crashing tests in pyiron/executorlib#243 . At the moment I cannot identify an obvious difference between the tests which work and those which do not work. My tests use the message passing interface (MPI) for internal communication. As a consequence, my tests rely on consistent network configuration, still at the moment it seems the network configuration of the hosts changes which at least I cannot verify based on the logs. This is just an assumption, but maybe others who also struggle with this issue can confirm that their tests also rely on a consistent network configuration and an inconsistent network configuration can lead to their tests failing randomly. Once more I want to emphasise that I have not experienced any similar issues on Linux or Windows tests environments.

[tmota900]

Still happening here too, having issues in some sort of volume binding when building dmg files:

An unhandled rejection has occurred inside Forge:
Error: Cannot find module '../build/Release/volume.node'
Require stack:
- /Users/runner/work/creator/creator/node_modules/.pnpm/[email protected]/node_modules/macos-alias/lib/create.js
- /Users/runner/work/creator/creator/node_modules/.pnpm/[email protected]/node_modules/macos-alias/index.js
- /Users/runner/work/creator/creator/node_modules/.pnpm/[email protected]/node_modules/ds-store/index.js
- /Users/runner/work/creator/creator/node_modules/.pnpm/[email protected]/node_modules/appdmg/lib/appdmg.js
- /Users/runner/work/creator/creator/node_modules/.pnpm/[email protected]/node_modules/appdmg/index.js
- /Users/runner/work/creator/creator/node_modules/.pnpm/[email protected]/node_modules/electron-installer-dmg/src/index.js
- /Users/runner/work/creator/creator/node_modules/.pnpm/@[email protected]/node_modules/@electron-forge/maker-dmg/dist/MakerDMG.js
- /Users/runner/work/creator/creator/node_modules/.pnpm/@[email protected]/node_modules/@electron-forge/core/dist/util/require-search.js
- /Users/runner/work/creator/creator/node_modules/.pnpm/@[email protected]/node_modules/@electron-forge/core/dist/util/plugin-interface.js
- /Users/runner/work/creator/creator/node_modules/.pnpm/@[email protected]/node_modules/@electron-forge/core/dist/util/forge-config.js
- /Users/runner/work/creator/creator/node_modules/.pnpm/@[email protected]/node_modules/@electron-forge/core/dist/util/index.js
- /Users/runner/work/creator/creator/node_modules/.pnpm/@[email protected]/node_modules/@electron-forge/core/dist/api/index.js
- /Users/runner/work/creator/creator/node_modules/.pnpm/@[email protected]/node_modules/@electron-forge/cli/dist/electron-forge-publish.js
at Module._resolveFilename (node:internal/modules/cjs/loader:946:15)
    at Module._load (node:internal/modules/cjs/loader:787:27)
    at Module.require (node:internal/modules/cjs/loader:1012:19)
    at require (node:internal/modules/cjs/helpers:102:18)
    at Object.<anonymous> (/Users/runner/work/creator/creator/node_modules/.pnpm/[email protected]/node_modules/macos-alias/lib/create.js:7:13)
    at Module._compile (node:internal/modules/cjs/loader:1112:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1[166](https://github.com/clutch-creator/creator/actions/runs/7338689671/job/19981604048#step:8:167):10)
    at Module.load (node:internal/modules/cjs/loader:988:32)
    at Module._load (node:internal/modules/cjs/loader:834:12)
    at Module.require (node:internal/modules/cjs/loader:1012:19)
    at require (node:internal/modules/cjs/helpers:102:18)
    at Object.<anonymous> (/Users/runner/work/creator/creator/node_modules/.pnpm/[email protected]/node_modules/macos-alias/index.js:1:18)
    at Module._compile (node:internal/modules/cjs/loader:1112:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1166:10)
    at Module.load (node:internal/modules/cjs/loader:988:32)
    at Module._load (node:internal/modules/cjs/loader:834:12)
 ELIFECYCLE  Command failed with exit code 1.

 
 This came from original issue #8796 which the workaround I found is no longer working.

[mikhailkoliada]

@tmota900 please report to npm directly, we are unlikely to help with the external rpm dependencies