FCS is trying to load a reference assembly on mono.

[matthid]

I really have no idea where that comes from.
I'm just trying to build a new FAKE version, but apparently FCS tries to load a reference assembly which is never a good idea (they should be used for compiling only).

Repro steps

On a travis like system (Mono JIT compiler version 4.8.1 (Stable 4.8.1.0/22a39d7 Wed Apr 12 12:00:40 UTC 2017))

1. git clone https://github.com/matthid/FAKE.git
2. git checkout -b fix_bug 7b5a00b67a364a3d3efb0ca0bfe9b266f2209b77
3. ./build.sh

This builds FAKE and tries to run it with the build script (a bootstrapping sanity check if the compiled binary works)

Expected behavior

The build succeeds.

Actual behavior

FAKE.exe Information: 0 : Reactor: enqueue ParseAndCheckProject /tmp/tmp779dac93.fsproj, length 0
FAKE.exe Information: 0 : Reactor: --> ParseAndCheckProject /tmp/tmp779dac93.fsproj, remaining 0, mem 67, gc2 6
Considering _appConfigKey fsharp-compiler-location which has value ''
Fail: Failed to compute key in OpenILModuleReaderAfterReadingAllBytes cache. Falling back to uncached.
Fail: Failed to compute key in OpenILModuleReaderAfterReadingAllBytes cache. Falling back to uncached.
Fail: Failed to compute key in OpenILModuleReaderAfterReadingAllBytes cache. Falling back to uncached.
Fail: Failed to compute key in OpenILModuleReaderAfterReadingAllBytes cache. Falling back to uncached.
Fail: Failed to compute key in OpenILModuleReaderAfterReadingAllBytes cache. Falling back to uncached.
FAKE.exe Information: 0 : Reactor: <-- ParseAndCheckProject /tmp/tmp779dac93.fsproj, remaining 0, took 817.641ms
FAKE.exe Information: 0 : Reactor: receiving..., remaining 0, mem 95, gc2 8
Fail: Bug seen in compiler: System.BadImageFormatException: 
File name: '/usr/lib/mono/4.5-api/Mono.Cecil.dll'
  at (wrapper managed-to-native) System.Reflection.Assembly:LoadFrom (string,bool)
  at System.Reflection.Assembly.LoadFrom (System.String assemblyFile) [0x00000] in <dbb16e0bacdc4a0f87478e401bc29b6c>:0 
  at Microsoft.FSharp.Compiler.AbstractIL.Internal.Library+Shim+DefaultFileSystem.Microsoft-FSharp-Compiler-AbstractIL-Internal-Library-Shim-IFileSystem-AssemblyLoadFrom (System.String fileName) [0x00001] in <5903835dddab8ea7a74503835d830359>:0 
  at Microsoft.FSharp.Compiler.AbstractIL.ILRuntimeWriter.convTypeRefAux (Microsoft.FSharp.Compiler.AbstractIL.ILRuntimeWriter+cenv cenv, Microsoft.FSharp.Compiler.AbstractIL.IL+ILTypeRef tref) [0x000b7] in <5903835dddab8ea7a74503835d830359>:0 
  at Microsoft.FSharp.Compiler.AbstractIL.ILRuntimeWriter.convTypeSpec (Microsoft.FSharp.Compiler.AbstractIL.ILRuntimeWriter+cenv cenv, Microsoft.FSharp.Compiler.AbstractIL.ILRuntimeWriter+emEnv emEnv, System.Boolean preferCreated, Microsoft.FSharp.Compiler.AbstractIL.IL+ILTypeSpec tspec) [0x0000a] in <5903835dddab8ea7a74503835d830359>:0 
  at Microsoft.FSharp.Compiler.AbstractIL.ILRuntimeWriter.convTypeAux (Microsoft.FSharp.Compiler.AbstractIL.ILRuntimeWriter+cenv cenv, Microsoft.FSharp.Compiler.AbstractIL.ILRuntimeWriter+emEnv emEnv, System.Boolean preferCreated, Microsoft.FSharp.Compiler.AbstractIL.IL+ILType typ) [0x00155] in <5903835dddab8ea7a74503835d830359>:0 
  at Microsoft.FSharp.Compiler.AbstractIL.ILRuntimeWriter.emitLocal (Microsoft.FSharp.Compiler.AbstractIL.ILRuntimeWriter+cenv cenv, Microsoft.FSharp.Compiler.AbstractIL.ILRuntimeWriter+emEnv emEnv, System.Reflection.Emit.ILGenerator ilG, Microsoft.FSharp.Compiler.AbstractIL.IL+ILLocal local) [0x00001] in <5903835dddab8ea7a74503835d830359>:0 
  at Microsoft.FSharp.Compiler.AbstractIL.ILRuntimeWriter+localBs@1302.Invoke (Microsoft.FSharp.Compiler.AbstractIL.IL+ILLocal local) [0x00001] in <5903835dddab8ea7a74503835d830359>:0 
  at Microsoft.FSharp.Compiler.AbstractIL.ILRuntimeWriter.emitILMethodBody (Microsoft.FSharp.Compiler.AbstractIL.ILRuntimeWriter+cenv cenv, System.Reflection.Emit.ModuleBuilder modB, Microsoft.FSharp.Compiler.AbstractIL.ILRuntimeWriter+emEnv emEnv, System.Reflection.Emit.ILGenerator ilG, Microsoft.FSharp.Compiler.AbstractIL.IL+ILMethodBody ilmbody) [0x0005d] in <5903835dddab8ea7a74503835d830359>:0 
  at Microsoft.FSharp.Compiler.AbstractIL.ILRuntimeWriter.emitMethodBody[a,b] (Microsoft.FSharp.Compiler.AbstractIL.ILRuntimeWriter+cenv cenv, System.Reflection.Emit.ModuleBuilder modB, Microsoft.FSharp.Compiler.AbstractIL.ILRuntimeWriter+emEnv emEnv, Microsoft.FSharp.Core.FSharpFunc`2[T,TResult] ilG, b _name, Microsoft.FSharp.Compiler.AbstractIL.IL+ILLazyMethodBody mbody) [0x00044] in <5903835dddab8ea7a74503835d830359>:0 
  at Microsoft.FSharp.Compiler.AbstractIL.ILRuntimeWriter.buildMethodPass3 (Microsoft.FSharp.Compiler.AbstractIL.ILRuntimeWriter+cenv cenv, Microsoft.FSharp.Compiler.AbstractIL.IL+ILTypeRef tref, System.Reflection.Emit.ModuleBuilder modB, System.Reflection.Emit.TypeBuilder typB, Microsoft.FSharp.Compiler.AbstractIL.ILRuntimeWriter+emEnv emEnv, Microsoft.FSharp.Compiler.AbstractIL.IL+ILMethodDef mdef) [0x001b4] in <5903835dddab8ea7a74503835d830359>:0 
  at Microsoft.FSharp.Compiler.AbstractIL.ILRuntimeWriter+buildTypeDefPass3@1818.Invoke (Microsoft.FSharp.Compiler.AbstractIL.IL+ILMethodDef mdef) [0x00001] in <5903835dddab8ea7a74503835d830359>:0 
  at Microsoft.FSharp.Collections.SeqModule.Iterate[T] (Microsoft.FSharp.Core.FSharpFunc`2[T,TResult] action, System.Collections.Generic.IEnumerable`1[T] source) [0x0002d] in <5893d081904cf4daa745038381d09358>:0 
  at Microsoft.FSharp.Compiler.AbstractIL.ILRuntimeWriter.buildTypeDefPass3 (Microsoft.FSharp.Compiler.AbstractIL.ILRuntimeWriter+cenv cenv, Microsoft.FSharp.Collections.FSharpList`1[T] nesting, System.Reflection.Emit.ModuleBuilder modB, Microsoft.FSharp.Compiler.AbstractIL.ILRuntimeWriter+emEnv emEnv, Microsoft.FSharp.Compiler.AbstractIL.IL+ILTypeDef tdef) [0x00054] in <5903835dddab8ea7a74503835d830359>:0 
  at Microsoft.FSharp.Compiler.AbstractIL.ILRuntimeWriter+emEnv@1828-7.Invoke (Microsoft.FSharp.Compiler.AbstractIL.ILRuntimeWriter+emEnv emEnv, Microsoft.FSharp.Compiler.AbstractIL.IL+ILTypeDef tdef) [0x00001] in <5903835dddab8ea7a74503835d830359>:0 
  at Microsoft.FSharp.Collections.ListModule.loop@219-29[T,TState] (Microsoft.FSharp.Core.OptimizedClosures+FSharpFunc`3[T1,T2,TResult] f, TState s, Microsoft.FSharp.Collections.FSharpList`1[T] xs) [0x00019] in <5893d081904cf4daa745038381d09358>:0 
  at Microsoft.FSharp.Collections.ListModule.Fold[T,TState] (Microsoft.FSharp.Core.FSharpFunc`2[T,TResult] folder, TState state, Microsoft.FSharp.Collections.FSharpList`1[T] list) [0x00014] in <5893d081904cf4daa745038381d09358>:0 
  at Microsoft.FSharp.Compiler.AbstractIL.ILRuntimeWriter.buildTypeDefPass3 (Microsoft.FSharp.Compiler.AbstractIL.ILRuntimeWriter+cenv cenv, Microsoft.FSharp.Collections.FSharpList`1[T] nesting, System.Reflection.Emit.ModuleBuilder modB, Microsoft.FSharp.Compiler.AbstractIL.ILRuntimeWriter+emEnv emEnv, Microsoft.FSharp.Compiler.AbstractIL.IL+ILTypeDef tdef) [0x0016d] in <5903835dddab8ea7a74503835d830359>:0 
  at Microsoft.FSharp.Compiler.AbstractIL.ILRuntimeWriter.buildModuleTypePass3 (Microsoft.FSharp.Compiler.AbstractIL.ILRuntimeWriter+cenv cenv, System.Reflection.Emit.ModuleBuilder modB, Microsoft.FSharp.Compiler.AbstractIL.ILRuntimeWriter+emEnv emEnv, Microsoft.FSharp.Compiler.AbstractIL.IL+ILTypeDef tdef) [0x00007] in <5903835dddab8ea7a74503835d830359>:0 
  at Microsoft.FSharp.Compiler.AbstractIL.ILRuntimeWriter+emEnv@1984-10.Invoke (Microsoft.FSharp.Compiler.AbstractIL.ILRuntimeWriter+emEnv emEnv, Microsoft.FSharp.Compiler.AbstractIL.IL+ILTypeDef tdef) [0x00001] in <5903835dddab8ea7a74503835d830359>:0 
  at Microsoft.FSharp.Collections.ListModule.loop@219-29[T,TState] (Microsoft.FSharp.Core.OptimizedClosures+FSharpFunc`3[T1,T2,TResult] f, TState s, Microsoft.FSharp.Collections.FSharpList`1[T] xs) [0x00019] in <5893d081904cf4daa745038381d09358>:0 
  at Microsoft.FSharp.Collections.ListModule.Fold[T,TState] (Microsoft.FSharp.Core.FSharpFunc`2[T,TResult] folder, TState state, Microsoft.FSharp.Collections.FSharpList`1[T] list) [0x00014] in <5893d081904cf4daa745038381d09358>:0 
  at Microsoft.FSharp.Compiler.AbstractIL.ILRuntimeWriter.buildModuleFragment (Microsoft.FSharp.Compiler.AbstractIL.ILRuntimeWriter+cenv cenv, Microsoft.FSharp.Compiler.AbstractIL.ILRuntimeWriter+emEnv emEnv, System.Reflection.Emit.AssemblyBuilder asmB, System.Reflection.Emit.ModuleBuilder modB, Microsoft.FSharp.Compiler.AbstractIL.IL+ILModuleDef m) [0x00128] in <5903835dddab8ea7a74503835d830359>:0 
  at Microsoft.FSharp.Compiler.AbstractIL.ILRuntimeWriter.emitModuleFragment (Microsoft.FSharp.Compiler.AbstractIL.IL+ILGlobals ilg, Microsoft.FSharp.Compiler.AbstractIL.ILRuntimeWriter+emEnv emEnv, System.Reflection.Emit.AssemblyBuilder asmB, System.Reflection.Emit.ModuleBuilder modB, Microsoft.FSharp.Compiler.AbstractIL.IL+ILModuleDef modul, System.Boolean debugInfo, Microsoft.FSharp.Core.FSharpFunc`2[T,TResult] resolveAssemblyRef, Microsoft.FSharp.Core.FSharpFunc`2[T,TResult] tryFindSysILTypeRef) [0x0000e] in <5903835dddab8ea7a74503835d830359>:0 
  at Microsoft.FSharp.Compiler.Interactive.Shell+FsiDynamicCompiler.ProcessInputs (Microsoft.FSharp.Compiler.AbstractIL.Internal.Library+CompilationThreadToken ctok, Microsoft.FSharp.Compiler.ErrorLogger+ErrorLogger errorLogger, Microsoft.FSharp.Compiler.Interactive.Shell+FsiDynamicCompilerState istate, Microsoft.FSharp.Collections.FSharpList`1[T] inputs, System.Boolean showTypes, System.Boolean isIncrementalFragment, System.Boolean isInteractiveItExpr, Microsoft.FSharp.Collections.FSharpList`1[T] prefixPath) [0x00306] in <5903835dddab8ea7a74503835d830359>:0 
  at Microsoft.FSharp.Compiler.Interactive.Shell+FsiDynamicCompiler.EvalParsedSourceFiles (Microsoft.FSharp.Compiler.AbstractIL.Internal.Library+CompilationThreadToken ctok, Microsoft.FSharp.Compiler.ErrorLogger+ErrorLogger errorLogger, Microsoft.FSharp.Compiler.Interactive.Shell+FsiDynamicCompilerState istate, Microsoft.FSharp.Collections.FSharpList`1[T] inputs) [0x00088] in <5903835dddab8ea7a74503835d830359>:0 
  at Microsoft.FSharp.Compiler.Interactive.Shell+FsiDynamicCompiler.EvalSourceFiles (Microsoft.FSharp.Compiler.AbstractIL.Internal.Library+CompilationThreadToken ctok, Microsoft.FSharp.Compiler.Interactive.Shell+FsiDynamicCompilerState istate, Microsoft.FSharp.Compiler.Range+range m, Microsoft.FSharp.Collections.FSharpList`1[T] sourceFiles, Microsoft.FSharp.Compiler.Lexhelp+LexResourceManager lexResourceManager, Microsoft.FSharp.Compiler.ErrorLogger+ErrorLogger errorLogger) [0x00173] in <5903835dddab8ea7a74503835d830359>:0 
  at [email protected] (Microsoft.FSharp.Compiler.Interactive.Shell+FsiDynamicCompilerState istate) [0x0074b] in <5903835dddab8ea7a74503835d830359>:0 
  at Microsoft.FSharp.Compiler.Interactive.Shell+FsiInteractionProcessor.InteractiveCatch[b] (Microsoft.FSharp.Compiler.ErrorLogger+ErrorLogger errorLogger, Microsoft.FSharp.Core.FSharpFunc`2[T,TResult] f, b istate) [0x0001f] in <5903835dddab8ea7a74503835d830359>:0 

Known workarounds

I have no idea.

Related information

Interestingly we updated FCS in the main branch as well and there we don't have this problem.
In the main branch we use an other version of Mono.Cecil and not the latest prerelease one which supports netstandard (which we need for the netcore version obviously).

To be honest I'm lost there. I cannot reproduce the error with FSI either. The debugging information was retrieved by a debug build from @cloudRoutine which is from the master branch. This initially happened on 12.0.5 which wouldn't really print anything but internal error, see #762

Is this maybe related to #759 (which I'm seeing in my log as well) or #731 (which has the reference assembly path as well in its error messages)

Note: The Error: System.NotSupportedException is probably a follow-up error when trying to save the broken assembly

Any help is appreciated... (The debugging version is here and available via the https://ci.appveyor.com/nuget/customnuspecs nuget feed)

[dsyme]

@matthid Yes, we should never be looking in 4.5-api when running scripting code. The problem is here https://github.com/fsharp/FSharp.Compiler.Service/blob/4a231d9404233b9e921e6a3f34885b0b36272fb2/src/fsharp/CompileOps.fs#L2850

This case has come up before dotnet/fsharp#2823

If you look at that code carefully, on Mono we currently look in the runtime root first. But then we look in the 4.5-api directory, which is fine for compilation but very incorrect for scripting code.

I think the simplest fix is to do much as we do on non-Mono, i.e. this:

        if runningOnMono then 
            [ let runtimeRoot = System.Runtime.InteropServices.RuntimeEnvironment.GetRuntimeDirectory()
              let runtimeRootWithoutSlash = runtimeRoot.TrimEnd('/', '\\')
              let api = runtimeRootWithoutSlash + "-api"
                match tcConfig.resolutionEnvironment with
                | ReferenceResolver.RuntimeLike ->
                    yield runtimeRoot // The defaut FSharp.Core is found in lib/mono/4.5
                | _ -> 
                    ... all the stuff we have now

[dsyme]

@matthid Here's the proposed fix: dotnet/fsharp#2949

Feel free to cherry-pick it here and try it out?

[dsyme]

@matthid You will need these commits if you do

2f78fc2
4c57bdd
f4f1408
0e8b9d0

[matthid]

First: Thanks for the quick help!

To be honest now I'm really curious why FAKE 4 (where we updated to fcs 12.0.5 as well) works. And I couldn't reproduce this with the fsi in the fsharp.compiler.tools package.
So i guess the error was only part of some builds and only triggered on special occasions?

I used the test.fsx in this commit: fsprojects/FAKE@e9eea07)

Testing this soon ;)

[matthid]

Ok I think I figured it out:
The issue was only happening when the reference was found indirectly. For some reason fsi prefers the indirect resolution from the runtime directory to any direct reference given via #r. I think the current fix works but we probably really should prefer what we explicitly write into the script?
We might need to slightly differentiate though. There are some assemblies (like Mono.Posix) which are never allowed to be loaded from anywhere but the runtime directory. So I would suggest this:

#I "some/path"
#r "Mono.Posix" // always prefer runtime directory

#r "./Mono.Posix.dll" // always prefer the local path

I actually was hit by this before. What I'm suggesting is that it works as documented (see #313).
Currently this doesn't work properly for indirect references.

Even more interesting: fsprojects/FAKE@18ee098 This commit seems to workaround the problem. I assumed loading the Mono.Cecil reference before would make it work. However when you compare that with the original reported problem from @haf (causiq/RingBuffer@a3282d7#diff-879359970577bc1dba6b3bc3fc59e6d2) He referenced Mono.Cecil before Expecto (which is indirectly using it), so it doesn't only depend on the referenced order but also on the order of the code using it.

@haf I'd assume adding some code which uses Mono.Cecil before using Expecto would be a workaround for now (in addition to correctly order the references like you already have)

[matthid]

I'd assume there are some additional problems to the one just fixed. But we will see when I reference these issues next time :)

For now I have a workaround and I'll wait if @cloudRoutine can make a build of https://github.com/matthid/FSharp.Compiler.Service/commits/fix_fcs_763 for testing (I cannot get it to build MSB4057: C:\PROJ\FSharp.Compiler.Service\src\fsharp\FSharp.Compiler.Service.MSBuild.v12\FSharp.Compiler.Service.MSBuild.v12.fsproj(0,0): The target "Build" does not exist in the project. and I don't want to hunt down some random msbuild issue either).

I really think the fix works, but I'm not planning to do a real FAKE 4 release (which has the same problem) with a custom FCS build anyway so my netcore branch can wait for a FCS release as well :)

[dsyme]

The issue was only happening when the reference was found indirectly. For some reason fsi prefers the indirect resolution from the runtime directory to any direct reference given via #r. I think the current fix works but we probably really should prefer what we explicitly write into the script?

dotnet/fsharp#2887 looks relevant here. That fix has now been integrated into this repo

[dsyme]

BTW the fix to avoid the API directory is being integrated here: #767

[dsyme]

There are some assemblies (like Mono.Posix) which are never allowed to be loaded from anywhere but the runtime directory. So I would suggest this...

I didn't know that. It makes things really hard when there are rules like this lurking around.

[matthid]

I didn't know that. It makes things really hard when there are rules like this lurking around.

@dsyme Yeah so much time wasted because of all of this ;). I can understand that it might make things different.
But really I don't see a reason why we couldn't use the same reference assemblies for compiling (to keep the compilation logic as is) and then later when loading the assembly we use the last "best" match from the script we currently are trying to interpret. These might not be the same at all.

I don't know the codebase but it might be different to replace "better" matching assemblies once we know them, but we could at least trigger a warning when we see that a .dll was explicitly referenced and we implicitly loaded another one already (we could even tell the user why we loaded the other one implicitly so that he can re-order if needed).

Later I don't really know why it depends on the code as well (and not only on the reference order). This smells like another bug :)

Hope this makes at least some kind of sense :)

[matthid]

Note also that I didn't suggested to make an exception for Mono.Posix. If you use #r "./path/Mono.Posix.dll" we can probably see that as user error.
I think the mono guys did all of this according to the docs (#313 (comment)) we just are not very good in following this at the moment on Linux when running scripts.

[matthid]

Yes dotnet/fsharp#2863 is most likely related. I'd assume its just another symptom of how our current way of doing this is wrong :).

[dsyme]

@matthid When you get the chance could you validate that the repro steps now pass for latest published FCS?