pythonPackages.watchdog 2.0.0 is broken on darwin

[dhess]

Describe the bug

963299d upgraded pythonPackages.watchdog to 2.0.0, but this version is broken on darwin:

...
creating build/temp.macosx-10.6-x86_64-3.8
creating build/temp.macosx-10.6-x86_64-3.8/src
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -DWATCHDOG_VERSION_STRING="2.0.0" -DWATCHDOG_VERSION_MAJOR=2 -DWATCHDOG_VERSION_MINOR=0 -DWATCHDOG_VERSION_BUILD=0 -I/nix/store/5vn5f6slfg1nw8q0bhapd3vgk9j7559i-python3-3.8.7/include/python3.8 -c src/watchdog_fsevents.c -o build/temp.macosx-10.6-x86_64-3.8/src/watchdog_fsevents.o -std=c99 -pedantic -Wall -Wextra -fPIC -Wno-nullability-completeness -Wno-nullability-extension -Wno-newline-eof -Wno-error=unused-command-line-argument
clang-7: warning: argument unused during compilation: '-fno-strict-overflow' [-Wunused-command-line-argument]
src/watchdog_fsevents.c:191:25: error: use of undeclared identifier 'kFSEventStreamEventFlagItemCloned'
FLAG_PROPERTY(IsCloned, kFSEventStreamEventFlagItemCloned)
                        ^
src/watchdog_fsevents.c:404:56: error: use of undeclared identifier 'kFSEventStreamEventExtendedDataPathKey'
        cf_path = CFDictionaryGetValue(path_info_dict, kFSEventStreamEventExtendedDataPathKey);
                                                       ^
src/watchdog_fsevents.c:405:57: error: use of undeclared identifier 'kFSEventStreamEventExtendedFileIDKey'
        cf_inode = CFDictionaryGetValue(path_info_dict, kFSEventStreamEventExtendedFileIDKey);
                                                        ^
src/watchdog_fsevents.c:585:40: error: use of undeclared identifier 'kFSEventStreamCreateFlagUseExtendedData'
                                     | kFSEventStreamCreateFlagUseExtendedData
                                       ^
4 errors generated.
error: command 'clang' failed with exit status 1

It appears this is because nixpkgs' macOS SDK is 10.12, but the symbols that cause problems in 2.0.0 are only available in 10.13+:

https://developer.apple.com/documentation/coreservices/kfseventstreameventextendeddatapathkey

[FRidh]

Issue regarding newer SDK #101229.

[ldeck]

I'm also seeing watchdog as a blocker for installing some packages.

e.g., installing was-sam-cli is broken:

% nix-env -i aws-sam-cli --show-trace
installing 'aws-sam-cli-1.23.0'
error: while evaluating the attribute 'drvPath' at /nix/store/39lgkp36x5qqpbi6rjcwg85lrc791ymc-nixpkgs-21.05pre287067.0d337eb6b77/nixpkgs/lib/customisation.nix:163:7:
while evaluating the attribute 'propagatedBuildInputs' of the derivation 'aws-sam-cli-1.23.0' at /nix/store/39lgkp36x5qqpbi6rjcwg85lrc791ymc-nixpkgs-21.05pre287067.0d337eb6b77/nixpkgs/pkgs/stdenv/generic/make-derivation.nix:201:11:
while evaluating the attribute 'out.outPath' at /nix/store/39lgkp36x5qqpbi6rjcwg85lrc791ymc-nixpkgs-21.05pre287067.0d337eb6b77/nixpkgs/lib/customisation.nix:156:13:
while evaluating the attribute 'handled' at /nix/store/39lgkp36x5qqpbi6rjcwg85lrc791ymc-nixpkgs-21.05pre287067.0d337eb6b77/nixpkgs/pkgs/stdenv/generic/check-meta.nix:301:7:
while evaluating 'handleEvalIssue' at /nix/store/39lgkp36x5qqpbi6rjcwg85lrc791ymc-nixpkgs-21.05pre287067.0d337eb6b77/nixpkgs/pkgs/stdenv/generic/check-meta.nix:188:38, called from /nix/store/39lgkp36x5qqpbi6rjcwg85lrc791ymc-nixpkgs-21.05pre287067.0d337eb6b77/nixpkgs/pkgs/stdenv/generic/check-meta.nix:302:14:
Package ‘python3.8-watchdog-2.0.3’ in /nix/store/39lgkp36x5qqpbi6rjcwg85lrc791ymc-nixpkgs-21.05pre287067.0d337eb6b77/nixpkgs/pkgs/development/python-modules/watchdog/default.nix:42 is marked as broken, refusing to evaluate.

a) To temporarily allow broken packages, you can use an environment variable
   for a single invocation of the nix tools.

     $ export NIXPKGS_ALLOW_BROKEN=1

b) For `nixos-rebuild` you can set
  { nixpkgs.config.allowBroken = true; }
in configuration.nix to override this.

c) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowBroken = true; }
to ~/.config/nixpkgs/config.nix.

Any work-around or fix? @lo1tuma, @cillianderoiste.

 % nix-shell -p nix-info --run "nix-info -m"
these paths will be fetched (1.46 MiB download, 7.88 MiB unpacked):
  /nix/store/3kz8p0rh7h260ky00ysjsg7q07jn41i2-bash-interactive-4.4-p23-dev
  /nix/store/6v47rygwaahny2wsmn9avnczf0sd2dn3-DarwinTools-1
  /nix/store/8c94id8pana0v7wq5ipvam6sfjwg8w1w-bash-interactive-4.4-p23-doc
  /nix/store/ff1d4zkny91w0cg8jdzfm6rlvcmpldwf-bash-interactive-4.4-p23-info
  /nix/store/gqvdmb1q5llvhccqm8ap7fxidhxfh67b-bash-interactive-4.4-p23-man
  /nix/store/i8dxjfdkf4ipnbnqr7ws4q4hjqxmiszq-bash-interactive-4.4-p23
  /nix/store/mcj5lk0cqds210nnspppynfwxnqf9ilw-readline-7.0p5
  /nix/store/sbazpkxvzplf2yfxcvq7q0f1v6qmfiva-nix-info
copying path '/nix/store/8c94id8pana0v7wq5ipvam6sfjwg8w1w-bash-interactive-4.4-p23-doc' from 'https://cache.nixos.org'...
copying path '/nix/store/6v47rygwaahny2wsmn9avnczf0sd2dn3-DarwinTools-1' from 'https://cache.nixos.org'...
copying path '/nix/store/ff1d4zkny91w0cg8jdzfm6rlvcmpldwf-bash-interactive-4.4-p23-info' from 'https://cache.nixos.org'...
copying path '/nix/store/gqvdmb1q5llvhccqm8ap7fxidhxfh67b-bash-interactive-4.4-p23-man' from 'https://cache.nixos.org'...
copying path '/nix/store/sbazpkxvzplf2yfxcvq7q0f1v6qmfiva-nix-info' from 'https://cache.nixos.org'...
copying path '/nix/store/mcj5lk0cqds210nnspppynfwxnqf9ilw-readline-7.0p5' from 'https://cache.nixos.org'...
copying path '/nix/store/i8dxjfdkf4ipnbnqr7ws4q4hjqxmiszq-bash-interactive-4.4-p23' from 'https://cache.nixos.org'...
copying path '/nix/store/3kz8p0rh7h260ky00ysjsg7q07jn41i2-bash-interactive-4.4-p23-dev' from 'https://cache.nixos.org'...
 - system: `"x86_64-darwin"`
 - host os: `Darwin 20.3.0, macOS 10.16`
 - multi-user?: `no`
 - sandbox: `no`
 - version: `nix-env (Nix) 2.3.10`
 - channels(ldeck): `"nixpkgs-21.05pre287067.0d337eb6b77"`
 - nixpkgs: `/Users/ldeck/.nix-defexpr/channels/nixpkgs`

[kutyel]

Still happens with [email protected]:

error: Package ‘python3.8-watchdog-2.1.1’ in /nix/store/p6wjywj33a1nj7lnbsb43kar52d6dmvy-nixpkgs-src/pkgs/development/python-modules/watchdog/default.nix:42 is marked as broken, refusing to evaluate.

a) To temporarily allow broken packages, you can use an environment variable
   for a single invocation of the nix tools.

     $ export NIXPKGS_ALLOW_BROKEN=1

b) For `nixos-rebuild` you can set
  { nixpkgs.config.allowBroken = true; }
in configuration.nix to override this.

c) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowBroken = true; }
to ~/.config/nixpkgs/config.nix.

(use '--show-trace' to show detailed location information)
make: *** [nix-shell] Error 1

[dhess]

Yeah, unfortunately, I think nothing is going to change on this issue until someone has time to upgrade the macOS SDK (#101229)

[domenkozar]

That's happening at #121055 as part of https://opencollective.com/nix-macos

[domenkozar]

What if the package uses sdk_11 for now?

[mweinelt]

It needs https://developer.apple.com/documentation/coreservices/1455361-fseventstreameventflags/kfseventstreameventflagitemcloned, so maybe fine.

[toonn]

Talking to @LnL7, this doesn't seem so simple. The 11 SDK is only available through the Aarch64 stdenv IIUC. So maybe this package doesn't need to be marked broken for Aarch64 Darwin, could someone using native Nix on an M1 mac confirm this?

Making it available on x86_64 properly would entail as much work as bumping the SDK to 10.13 does because we have as much as possible depend on the Apple source releases rather than the SDK. So I don't think this'd be an easy workaround.

@thefloweringash would be better able to judge how practical the idea is though.

[toonn]

Here's what I tried, from a checkout of master:

> NIXPKGS_ALLOW_BROKEN=1 nix-build --no-out-link -E 'let pkgs = import ./. {}; apple_sdk_11_0 = pkgs.callPackage pkgs/os-specific/darwin/apple-sdk-11.0 {}; in pkgs.python3Packages.watchdog.override {inherit (apple_sdk_11_0.frameworks) CoreServices; }'
... [Elided]
The following paths did not match any rewrite rule
  /usr/lib/libobjc.A.dylib
... [Elided]

Errors occurred during rewrite of /nix/store/30f0jc58f542qlli3zachx2ncx17jgfa-apple-framework-CoreFoundation-11.0.0/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation.tbd                                                   
Rewrite config:
Exact mappings:                                                                 
Errors occurred during rewrite of /nix/store/6dax4h9wc21h0a2h5bmqwkpyr4sgxx25-apple-framework-CoreFoundation-11.0.0/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation.tbd
Rewrite config:                                                                 
Exact mappings:                                                                 
Prefix mappings:                                                                
  - /System/Library/Frameworks/CoreFoundation.framework/ -> /nix/store/6dax4h9wc21h0a2h5bmqwkpyr4sgxx25-apple-framework-CoreFoundation-11.0.0/Library/Frameworks/CoreFoundation.framework/              
Required prefix: /nix/store             
Check existence of resulting paths: yes

Prefix mappings:
  - /System/Library/Frameworks/CoreFoundation.framework/ -> /nix/store/30f0jc58f542qlli3zachx2ncx17jgfa-apple-framework-CoreFoundation-11.0.0/Library/Frameworks/CoreFoundation.framework/
Required prefix: /nix/store
Check existence of resulting paths: yes

The following paths did not match any rewrite rule
  /usr/lib/libobjc.A.dylib
The following paths did not match any rewrite rule
  /usr/lib/libobjc.A.dylib
libc++abi.dylib: terminating with uncaught exception of type rewrite_errors: Rewriting failed
libc++abi.dylib: terminating with uncaught exception of type rewrite_errors: Rewriting failed
/nix/store/vnazyva3nawmvmhsxl8d2698wqg3743l-stdenv-darwin/setup: line 1329: 4818
3 Abort trap: 6           rewrite-tbd -p /System/Library/Frameworks/CoreFoundation.framework/:$out/Library/Frameworks/CoreFoundation.framework/ -r /nix/store "$tbd"
/nix/store/vnazyva3nawmvmhsxl8d2698wqg3743l-stdenv-darwin/setup: line 1329: 4818
2 Abort trap: 6           rewrite-tbd -p /System/Library/Frameworks/CoreFoundation.framework/:$out/Library/Frameworks/CoreFoundation.framework/ -r /nix/store "$tbd"
post-installation fixup
builder for '/nix/store/y8kvzqx49krhb8mvas81y5kb8d4m5fpq-apple-framework-CoreFoundation-11.0.0.drv' failed with exit code 134

I suspect the problem is trying to rewrite-tbd with the 10.12 darwin-stubs, which could explain missing dylibs. The usual 10.12 apple_sdk expression explicitly depends on darwin-stubs but apple_sdk_11_0 does not so I'm not able to override it. I'm not sure why this is, @thefloweringash?

[toonn]

Maybe it's using impure stubs from the host? Would be cool if someone running Big Sur could confirm or deny whether the above works for them.

[davegallant]

Maybe it's using impure stubs from the host? Would be cool if someone running Big Sur could confirm or deny whether the above works for them.

I ran into the same issue on Big Sur.

[AlexMoutonNoble]

That's happening at #121055 as part of https://opencollective.com/nix-macos

Just coming into this but I see this has been merged? Is there anything to be done here?

Thanks
A

[toonn]

@AlexMoutonNoble, sorry for causing confusion. That PR became just a bump of LLVM and the bit that was merged is just the small bootstrap-tools bump the work depends on. So this has no impact on this issue.
I continue to work on both getting LLVM updated and the Apple SDK.

You can keep an eye on #116341 or the Nix ❤️ macOS Monthly Discourse thread for updates.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nix-macos-monthly/12330/8

[stale]

I marked this as stale due to inactivity. → More info

[amitavaghosh1]

Package ‘python3.9-watchdog-2.1.6’ in /nix/store/16znkxs25xghynrdrrcf3asczxh3hdm3-nixpkgs-22.05pre346850.9789ed45b75/nixpkgs/pkgs/development/python-modules/watchdog/default.nix:52 is marked as broken, refusing to evaluate.

[kubukoz]

Hi all, is there a recommended workaround for this?

[yipengsun]

I used to create a Python virtualenv with my flake.nix/shell.nix, then just install watchdog directly with pip inside that virtualenv:

I mainly use this for mkdocs, and it WAS working correctly.

          buildInputs = with pythonPackages; [
            # Python requirements (enough to get a virtualenv going).
            virtualenvwrapper
            numpy 
          ];

          shellHook = ''
            # Allow the use of wheels.
            SOURCE_DATE_EPOCH=$(date +%s)
            VENV=./.virtualenv

            if test ! -d $VENV; then
              virtualenv $VENV
            fi
            source $VENV/bin/activate

            # Allow for the environment to pick up packages installed with virtualenv
            export PYTHONPATH=$VENV/${python.sitePackages}/:$PYTHONPATH
        '';

I don't have a mac currently, so can't say if it still works, but you can give it a try.