macOS x64 CI: fix dependency install and OpenMP runtime copy (use Homebrew libomp, adjust Helix payload) by asp2286 · Pull Request #7510 · dotnet/machinelearning

asp2286 · 2025-09-18T07:17:08Z

PR: macOS x64 CI: fix dependency install and OpenMP runtime copy

Summary

This PR fixes MachineLearning-CI failures on macOS x64 where jobs stop at Install MacOS build dependencies with:

Bash exited with code '1'

The breakage comes from two areas:

Dependency install: The pipeline relied on a custom libomp.rb path that no longer works on hosted macOS images.
Helix payload: The script attempted to copy both libomp.dylib and libiomp5.dylib, but libiomp5.dylib is not available when installing libomp from Homebrew core.

Fixes #7509

Changes

`build/ci/job-template.yml`

Replace custom brew install …/build/libomp.rb with standard Homebrew:

brew update
brew install -f --overwrite python@3.13
brew install libomp
brew link libomp --force

Note added: Homebrew ≥4.6 rejects installing formulae from raw paths.

`eng/helix.proj`

macOS x64 only:
- Set DYLD_LIBRARY_PATH so Helix can find libomp.dylib.
- Copy only /usr/local/opt/libomp/lib/libomp.dylib into the publish folder.
- Remove copying of libiomp5.dylib (not present with libomp from Homebrew).
- Add install-name fix so binaries reference @loader_path/libomp.dylib.

Why

Hosted macOS runners changed: raw formula paths are blocked, and only libomp is available via core.
Ensures reliable dependency install and payload runtime linking.
Other platforms (Linux, Windows, macOS arm64) are unaffected.

Testing

Reproduced failure on osx.13.amd64.open queue.
With these changes:
- Dependency install step completes successfully.
- libomp.dylib is present in publish folder.
- Helix payload runs with DYLD_LIBRARY_PATH set correctly.
Validated in a test run: both macOS x64 Debug/Release proceed past dependency install and build succeeds.

Risk / Impact

Low: scoped only to macOS x64 build dependencies and Helix payload.
No product code changes, only CI infra adjustments.

Additional Notes

Linux and Windows jobs were already green.
If maintainers prefer llvm over libomp as the OpenMP provider, happy to adjust.

PR Checklist

CLA signed
Tests: N/A (CI infra only)
Docs: N/A (no public-facing change)
Breaking change: No
Linked issue: MacOS_x64 Debug/Release build jobs fail in MachineLearning-CI: “Bash exited with code '1' – Install MacOS build dependencies” #7509

azure-pipelines · 2025-09-18T07:57:43Z

Commenter does not have sufficient privileges for PR 7510 in repo dotnet/machinelearning

codecov · 2025-09-18T08:39:28Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 69.00%. Comparing base (fb39755) to head (1390b39).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #7510      +/-   ##
==========================================
- Coverage   69.01%   69.00%   -0.01%     
==========================================
  Files        1482     1482              
  Lines      273999   273999              
  Branches    28258    28258              
==========================================
- Hits       189093   189075      -18     
- Misses      77520    77536      +16     
- Partials     7386     7388       +2

Flag	Coverage Δ
Debug	`69.00% <ø> (-0.01%)`	⬇️
production	`63.29% <ø> (-0.01%)`	⬇️
test	`89.46% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.
see 5 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

azure-pipelines · 2025-09-18T12:02:34Z

Commenter does not have sufficient privileges for PR 7510 in repo dotnet/machinelearning

asp2286 · 2025-09-26T00:51:01Z

@artl93 @ericstj Could you please review this PR? Thank you!

…ustom formula path

skyRoma · 2025-09-30T14:02:47Z

I really need these changes, thanks!

ericstj · 2025-09-30T21:00:11Z

I tried to simplify a bit where x64 and M1 were the same. Let's see if this will also resolve the M1 failure.

asp2286 · 2025-10-01T02:40:17Z

/ba-g Mac-OS unrelated failures

…ebrew libomp, adjust Helix payload) (#7510) * CI(macOS x64): use Homebrew libomp + copy only libomp.dylib; remove custom formula path * Fix indenting and make M1 and x64 consistent * Simplify helix.proj --------- Co-authored-by: Eric StJohn <ericstj@microsoft.com>

* Update dependencies from https://github.com/dotnet/arcade build 20250516.2 Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25266.2 * update cmake mac (#7443) * update cmake mac * cmake specific version * remove specific version * cmake policy version minimum * Update build/ci/job-template.yml * Try adding -DCMAKE_POLICY_VERSION_MINIMUM=3.5 to libomp.rb * Update job-template.yml * Update cmake_minimum_required --------- Co-authored-by: Eric StJohn <ericstj@microsoft.com> * Update dependencies from https://github.com/dotnet/arcade build 20250521.1 Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25271.1 * Update dependencies from https://github.com/dotnet/arcade build 20250602.2 Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25302.2 * Update dependencies from https://github.com/dotnet/arcade build 20250625.4 Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25325.4 * Update dependencies from https://github.com/dotnet/arcade build 20250716.1 Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25366.1 * Update dependencies from https://github.com/dotnet/arcade build 20250730.1 Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25380.1 * Update dependencies from https://github.com/dotnet/arcade build 20250807.2 Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25407.2 * Update dependencies from https://github.com/dotnet/arcade build 20250815.3 Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25415.3 * Update dependencies from https://github.com/dotnet/arcade build 20250828.3 Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25428.3 * Update dependencies from https://github.com/dotnet/arcade build 20250910.1 Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25460.1 * Update dependencies from https://github.com/dotnet/arcade build 20250912.4 On relative base path root Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25462.4 * Update dependencies from https://github.com/dotnet/arcade build 20250915.2 On relative base path root Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25465.2 * Update dependencies from https://github.com/dotnet/arcade build 20250923.2 On relative base path root Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25473.2 * Update dependencies from https://github.com/dotnet/arcade build 20250925.3 On relative base path root Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25475.3 * Update dependencies from https://github.com/dotnet/arcade build 20251003.3 On relative base path root Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25503.3 * Update dependencies from https://github.com/dotnet/arcade build 20251015.2 On relative base path root Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25515.2 * macOS x64 CI: fix dependency install and OpenMP runtime copy (use Homebrew libomp, adjust Helix payload) (#7510) * CI(macOS x64): use Homebrew libomp + copy only libomp.dylib; remove custom formula path * Fix indenting and make M1 and x64 consistent * Simplify helix.proj --------- Co-authored-by: Eric StJohn <ericstj@microsoft.com> * Update Windows image, fix mac build (#7515) * Fix new compilation errors with latest SDK * Fix additional breaks introduced with new SDK * Update Helix ubuntu arm32 container (#7410) * Update Helix ubuntu arm32 container * Switch to debian-12-helix-arm32v7 * Revert back to ubuntu-24.04-helix-arm32v7 * Update helixQueue to use Ubuntu 22.04 on ARM32 * Add linux-arm baseline for Summarize.txt --------- Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com> Co-authored-by: Michael Sharp <51342856+michaelgsharp@users.noreply.github.com> Co-authored-by: Eric StJohn <ericstj@microsoft.com> Co-authored-by: Alexey Sokolov <asp2286@gmail.com>

dotnet-policy-service Bot added the community-contribution label Sep 18, 2025

asp2286 marked this pull request as draft September 18, 2025 12:03

asp2286 marked this pull request as ready for review September 18, 2025 12:03

asp2286 force-pushed the fix/macos-x64-libomp branch from 6c0c6a4 to 47bb847 Compare September 18, 2025 12:06

ericstj reviewed Sep 26, 2025

View reviewed changes

Comment thread build/ci/job-template.yml

ericstj reviewed Sep 26, 2025

View reviewed changes

Comment thread build/ci/job-template.yml Outdated

asp2286 force-pushed the fix/macos-x64-libomp branch 2 times, most recently from e44de68 to eb0ee80 Compare September 28, 2025 16:50

build-analysis Bot mentioned this pull request Sep 28, 2025

Code coverage build fails to instrument file in use #7417

Open

CI(macOS x64): use Homebrew libomp + copy only libomp.dylib; remove c…

d282b6a

…ustom formula path

asp2286 force-pushed the fix/macos-x64-libomp branch from eb0ee80 to d282b6a Compare September 29, 2025 01:39

skyRoma approved these changes Sep 30, 2025

View reviewed changes

Merge branch 'dotnet:main' into fix/macos-x64-libomp

3a674f2

ericstj reviewed Sep 30, 2025

View reviewed changes

Comment thread build/ci/job-template.yml Outdated

Fix indenting and make M1 and x64 consistent

e9dd7eb

ericstj reviewed Sep 30, 2025

View reviewed changes

Comment thread eng/helix.proj

ericstj approved these changes Sep 30, 2025

View reviewed changes

Simplify helix.proj

1390b39

asp2286 force-pushed the fix/macos-x64-libomp branch from f0ce5ba to 1390b39 Compare October 1, 2025 02:50

ericstj merged commit a5997af into dotnet:main Oct 1, 2025
22 of 25 checks passed

asp2286 deleted the fix/macos-x64-libomp branch October 2, 2025 07:45

github-actions Bot locked and limited conversation to collaborators Nov 1, 2025

Navigation Menu

Uh oh!

macOS x64 CI: fix dependency install and OpenMP runtime copy (use Homebrew libomp, adjust Helix payload) - #7510

macOS x64 CI: fix dependency install and OpenMP runtime copy (use Homebrew libomp, adjust Helix payload)#7510
ericstj merged 4 commits into
dotnet:mainfrom
asp2286:fix/macos-x64-libomp

asp2286 commented Sep 18, 2025

azure-pipelines Bot commented Sep 18, 2025

codecov Bot commented Sep 18, 2025 •

edited

Loading

azure-pipelines Bot commented Sep 18, 2025

asp2286 commented Sep 26, 2025

Uh oh!

Uh oh!

skyRoma commented Sep 30, 2025

Uh oh!

ericstj commented Sep 30, 2025

Uh oh!

asp2286 commented Oct 1, 2025

Uh oh!

Labels

3 participants

Uh oh!

Conversation

asp2286 commented Sep 18, 2025

PR: macOS x64 CI: fix dependency install and OpenMP runtime copy

Summary

Changes

build/ci/job-template.yml

eng/helix.proj

Why

Testing

Risk / Impact

Additional Notes

PR Checklist

azure-pipelines Bot commented Sep 18, 2025

codecov Bot commented Sep 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

azure-pipelines Bot commented Sep 18, 2025

asp2286 commented Sep 26, 2025

Uh oh!

Uh oh!

skyRoma commented Sep 30, 2025

Uh oh!

ericstj commented Sep 30, 2025

Uh oh!

asp2286 commented Oct 1, 2025

Uh oh!

Labels

3 participants

`build/ci/job-template.yml`

`eng/helix.proj`

codecov Bot commented Sep 18, 2025 •

edited

Loading