Exception of type 'Microsoft.FSharp.Compiler.Tast+InternalUndefinedItemRef' when generating reference docs for an assembly that uses the Json type provider.

[talbottmike]

I'm using FSharp.Formatting as implemented in the ProjectScaffold to generate reference documentation. When attempting to generate reference documentation, I'm getting an exception when MetadataFormat.Generate is called. By process of elimination I found that removing all JsonProviders from my assembly resolves the issue. However, when I re-include a JsonProvider in my project the error returns. I'm not sure that this is an issue with FSharp.Formatting, but any help is appreciated. Thanks

Here's the function from the scaffold generate.fsx where the error occurs.

// Build API reference from XML comments
let buildReference () =
  CleanDir (output @@ "reference")
  MetadataFormat.Generate
    ( binaries, output @@ "reference", layoutRootsAll.["en"],
      parameters = ("root", root)::info,
      sourceRepo = githubLink @@ "tree/master",
      sourceFolder = __SOURCE_DIRECTORY__ @@ ".." @@ "..",
      publicOnly = true,libDirs = libDirs )

Here's the exception.

Microsoft.FSharp.Compiler.ErrorLogger+ReportedError: Exception of type 'Microsoft.FSharp.Compiler.Tast+InternalUndefinedItemRef' was thrown.
   at Microsoft.FSharp.Compiler.ErrorLogger.ErrorLoggerExtensions.ErrorLogger.Error[b](ErrorLogger x, Exception exn) in C:\GitHub\fsharp\FSharp.Compiler.Service\src\fsharp\ErrorLogger.fs:line 321
   at Microsoft.FSharp.Compiler.Tast.EntityRef.get_Deref() in C:\GitHub\fsharp\FSharp.Compiler.Service\src\fsharp\tast.fs:line 2585
   at Microsoft.FSharp.Compiler.Tastops.stripTyEqnsA(TcGlobals g, Boolean canShortcut, TType ty) in C:\GitHub\fsharp\FSharp.Compiler.Service\src\fsharp\tastops.fs:line 614
   at Microsoft.FSharp.Compiler.Tastops.extensionInfoOfTy(TcGlobals g, TType ty) in C:\GitHub\fsharp\FSharp.Compiler.Service\src\fsharp\tastops.fs:line 1441
   at Microsoft.FSharp.Compiler.Tastops.metadataOfTy(TcGlobals g, TType ty) in C:\GitHub\fsharp\FSharp.Compiler.Service\src\fsharp\tastops.fs:line 1466
   at Microsoft.FSharp.Compiler.Infos.GetImmediateIntrinsicPropInfosOfType(FSharpOption`1 optFilter, AccessorDomain ad, TcGlobals g, ImportMap amap, range m, TType typ) in C:\GitHub\fsharp\FSharp.Compiler.Service\src\fsharp\infos.fs:line 3242
   at <StartupCode$FSharp-Compiler-Service>[email protected](IEnumerable`1& next) in C:\GitHub\fsharp\FSharp.Compiler.Service\src\fsharp\vs\Symbols.fs:line 410
   at Microsoft.FSharp.Core.CompilerServices.GeneratedSequenceBase`1.MoveNextImpl()
   at Microsoft.FSharp.Core.CompilerServices.GeneratedSequenceBase`1.System-Collections-IEnumerator-MoveNext()
   at Microsoft.FSharp.Collections.SeqModule.ToList[T](IEnumerable`1 source)
   at <StartupCode$FSharp-Compiler-Service>[email protected](Unit unitVar0) in C:\GitHub\fsharp\FSharp.Compiler.Service\src\fsharp\vs\Symbols.fs:line 396
   at Microsoft.FSharp.Compiler.SourceCodeServices.Impl.protect[a](FSharpFunc`2 f) in C:\GitHub\fsharp\FSharp.Compiler.Service\src\fsharp\vs\Symbols.fs:line 36
   at Microsoft.FSharp.Compiler.SourceCodeServices.FSharpEntity.get_MembersFunctionsAndValues() in C:\GitHub\fsharp\FSharp.Compiler.Service\src\fsharp\vs\Symbols.fs:line 395
   at [email protected](IEnumerable`1& next) in c:\Tomas\Public\tpetricek\FSharp.Formatting\src\FSharp.MetadataFormat\Main.fs:line 0
   at Microsoft.FSharp.Core.CompilerServices.GeneratedSequenceBase`1.MoveNextImpl()
   at Microsoft.FSharp.Core.CompilerServices.GeneratedSequenceBase`1.System-Collections-IEnumerator-MoveNext()
   at Microsoft.FSharp.Collections.SeqModule.ToList[T](IEnumerable`1 source)
   at [email protected](String cat, IDictionary`2 cmds, Comment comment) in c:\Tomas\Public\tpetricek\FSharp.Formatting\src\FSharp.MetadataFormat\Main.fs:line 898
   at FSharp.MetadataFormat.Reader.readCommentsInto[a](ReadingContext ctx, String xmlDoc, FSharpFunc`2 f) in c:\Tomas\Public\tpetricek\FSharp.Formatting\src\FSharp.MetadataFormat\Main.fs:line 788
   at FSharp.MetadataFormat.Reader.readType(ReadingContext ctx, FSharpEntity typ) in c:\Tomas\Public\tpetricek\FSharp.Formatting\src\FSharp.MetadataFormat\Main.fs:line 883
   at [email protected](ReadingContext ctx, FSharpEntity typ) in c:\Tomas\Public\tpetricek\FSharp.Formatting\src\FSharp.MetadataFormat\Main.fs:line 876
   at Microsoft.FSharp.Collections.IEnumerator.choose@202.System-Collections-IEnumerator-MoveNext()
   at Microsoft.FSharp.Collections.SeqModule.ToList[T](IEnumerable`1 source)
>    at FSharp.MetadataFormat.Reader.readModulesAndTypes(ReadingContext ctx, IEnumerable`1 entities) in c:\Tomas\Public\tpetricek\FSharp.Formatting\src\FSharp.MetadataFormat\Main.fs:line 877
   at [email protected](String cat, IDictionary`2 cmd, Comment comment) in c:\Tomas\Public\tpetricek\FSharp.Formatting\src\FSharp.MetadataFormat\Main.fs:line 940
   at FSharp.MetadataFormat.Reader.readCommentsInto[a](ReadingContext ctx, String xmlDoc, FSharpFunc`2 f) in c:\Tomas\Public\tpetricek\FSharp.Formatting\src\FSharp.MetadataFormat\Main.fs:line 788
   at FSharp.MetadataFormat.Reader.readModule(ReadingContext ctx, FSharpEntity modul) in c:\Tomas\Public\tpetricek\FSharp.Formatting\src\FSharp.MetadataFormat\Main.fs:line 934
   at [email protected](ReadingContext ctx, FSharpEntity modul) in c:\Tomas\Public\tpetricek\FSharp.Formatting\src\FSharp.MetadataFormat\Main.fs:line 875
   at Microsoft.FSharp.Collections.IEnumerator.choose@202.System-Collections-IEnumerator-MoveNext()
   at Microsoft.FSharp.Collections.SeqModule.ToList[T](IEnumerable`1 source)
   at FSharp.MetadataFormat.Reader.readModulesAndTypes(ReadingContext ctx, IEnumerable`1 entities) in c:\Tomas\Public\tpetricek\FSharp.Formatting\src\FSharp.MetadataFormat\Main.fs:line 877
   at FSharp.MetadataFormat.Reader.readNamespace(ReadingContext ctx, String ns, IEnumerable`1 entities) in c:\Tomas\Public\tpetricek\FSharp.Formatting\src\FSharp.MetadataFormat\Main.fs:line 955
   at [email protected](Tuple`2 tupledArg) in c:\Tomas\Public\tpetricek\FSharp.Formatting\src\FSharp.MetadataFormat\Main.fs:line 995
   at [email protected](b& )
   at Microsoft.FSharp.Collections.IEnumerator.MapEnumerator`1.System-Collections-IEnumerator-MoveNext()
   at Microsoft.FSharp.Collections.SeqModule.ToList[T](IEnumerable`1 source)
   at FSharp.MetadataFormat.Reader.readAssembly(FSharpAssembly assembly, Boolean publicOnly, String xmlFile, FSharpOption`1 sourceFolderRepo, FSharpFunc`2 urlRangeHighlight, Boolean markDownComments, IUrlHolder urlMap, String codeFormatCompilerArgs) in c:\Tomas\Public\tpetricek\FSharp.Formatting\src\FSharp.MetadataFormat\Main.fs:line 998
   at <StartupCode$FSharp-MetadataFormat>[email protected](Tuple`2 _arg2) in c:\Tomas\Public\tpetricek\FSharp.Formatting\src\FSharp.MetadataFormat\Main.fs:line 1121
   at Microsoft.FSharp.Collections.ListModule.chooseAllAcc[a,b](FSharpFunc`2 f, FSharpList`1 xs, FSharpList`1 acc)
   at FSharp.MetadataFormat.MetadataFormat.Generate(FSharpList`1 dllFiles, String outDir, FSharpList`1 layoutRoots, FSharpOption`1 parameters, FSharpOption`1 namespaceTemplate, FSharpOption`1 moduleTemplate, FSharpOption`1 typeTemplate, FSharpOption`1 xmlFile, FSharpOption`1 sourceRepo, FSharpOption`1 sourceFolder, FSharpOption`1 publicOnly, FSharpOption`1 libDirs, FSharpOption`1 otherFlags, FSharpOption`1 markDownComments, FSharpOption`1 urlRangeHighlight, FSharpOption`1 assemblyReferences) in c:\Tomas\Public\tpetricek\FSharp.Formatting\src\FSharp.MetadataFormat\Main.fs:line 1116
   at <StartupCode$FSI_0004>.$FSI_0004.main@() in C:\Workspaces\Personal\TalBot\docs\tools\generate.fsx:line 143

[talbottmike]

Any thoughts on this? I'm still trying to work through it. Tried using the omit tag on the members that use the JsonProvider but I'm still getting the same message.

Reading through some of the other issues, I see there might be problems with using different versions of the Compiler.Service and VSPowerTools.
I did notice that in my paket.lock file there are to different versions of the compiler service listed. I think that's ok though since the FSharp.Formatting one is >=
FSharp.Compiler.Service (0.0.90)
FSharp.Formatting (2.9.10)
FSharp.Compiler.Service (>= 0.0.87)
FSharpVSPowerTools.Core (1.8.0)
FSharpVSPowerTools.Core (1.8.0)

[talbottmike]

Tried another test.

• Started a new project with the scaffold and built reference docs
• Added a type provider and tried re-building reference docs

The first step succeeds but the second fails with the same message I've been getting in my project.

Here's the JsonProvider.

type MessageType = JsonProvider<"""{"type": "hello","subtype": "message"}""">

[tpetricek]

Thanks for producing an easy-to-reproduce scenario!

This looks like an F# Compiler Service issue (http://github.com/fsharp/FSharp.Compiler.Service/), or perhaps we're not calling the F# Compiler Service correctly... Let me cc @dsyme to see if he has any ideas on what might be causing this.

[dsyme]

There seems to be a steady stream of issues involving FCS and FSharp.Formatting, especially in the area of assembly references and other compiler configuration flags. I think FSharp.Formatting is exposed to this more than other tooling because of two things

• It constructs its own F# compiler option flags and references (by cracking a script) rather than getting them via cracking a project file
• It uses a secondary in-process F# interactive instance to evaluate stuff during documentation generation, in the context of an existing fsi.exe F# interactive execution of the generate.fsx script. There's definitely potential for problems or gremlins here.

To get to the bottom of this kind of issue we really need much better visibility of how FSharp.Formatting is actually instantiating and using FCS. For example, it would be very helpful if FSharp.Formatting printed out the exact command line option set being passed to create the FSharpChecker and also to FsiEvaluationSession. @tpetricek do you think you could add these diagnostics?

On the specific issue above - I'm not really sure - it could be a missing FSharp.Data reference meaning the provided symbols can't be linked. Whatever comes we should add more tests to FSharp.Compiler.Service which drives the API in a way that is equivalent to the way that FSHarp.Formatting is driving it. Some systematic testing in FCS would help flush out bugs here.

[tpetricek]

Yes, adding some logging so that we can easily diagnose these kinds of errors would be nice. I currently have a few other things that I need to finish soon. Can you perhaps open a separate issue for this, so that we can move the diagnostics-related discussion there? (And add a couple of pointers in case someone else wanted to contribute.)

[talbottmike]

Thanks for the responses. I'm coming from a C# background and have been using my project the last few months as a way of learning F#. The community has been a great help. I'd be interesting in contributing if there are some pointers and it's something that can be done at my skill level.

[matthid]

I would be willing to contribute to this as well (improving the overall exception code to make it easier to reproduce in FCS), I have already improved the exception messages within the F# scripting code of FSF, so this would be the next logical thing to do. It's OK if @talbottmike want's to take it though, as he was faster :). Currently I'm still busy anyway so I would start with this in a couple of weeks at best.

[talbottmike]

@matthid I'm not sure where to start so don't let me get in the way. I will go take a look at your commits to FSF to see what I can learn.

[talbottmike]

@matthid I looked at some history and you're better equipped to help than I am. I'm out of my depth here. I'd like to find something to contribute to but need to start closer to beginner level. I could do a lot of learning here but don't know how much value I could contribute until I've done some more learning.

[dsyme]

CLosing really old bugs we likely aren't going to investigate directly now things have moved on