Further work on FCS API

docs/compiler-guide.md

@@ -72,7 +72,10 @@ The following are the key phases and high-level logical operations of the F# com
 
 * _Parsing_. Accepts a token stream and produces an AST per the grammar in the F# Language Specification.
 
-* _Resolving references_. See [ReferenceResolver.fs](https://github.com/dotnet/fsharp/blob/master/src/fsharp/ReferenceResolver.fs) for the abstract definition of compiler reference resolution. See [LegacyMSBuildReferenceResolver.fs](https://github.com/dotnet/fsharp/blob/master/src/fsharp/LegacyMSBuildReferenceResolver.fs) for reference resolution used by the .NET Framework F# compiler when running on .NET Framework. See [SimulatedMSBuildReferenceResolver.fs](https://github.com/dotnet/fsharp/blob/master/src/fsharp/SimulatedMSBuildReferenceResolver.fs) when not using the .NET Framework F# compiler. See [Microsoft.DotNet.DependencyManager](https://github.com/dotnet/fsharp/tree/master/src/fsharp/Microsoft.DotNet.DependencyManager) for reference resolution and package management used in `fsi`.
+* _Resolving references_. For .NET SDK generally references are resolved explicitly by external tooling. 
+   There is a legacy aspect to this if references use old .NET Framework references including for
+   scripting.  See [ReferenceResolver.fs](https://github.com/dotnet/fsharp/blob/master/src/fsharp/ReferenceResolver.fs) for the abstract definition of compiler reference resolution. See [LegacyMSBuildReferenceResolver.fs](https://github.com/dotnet/fsharp/blob/master/src/fsharp/LegacyMSBuildReferenceResolver.fs) for reference resolution used by the .NET Framework F# compiler when running on .NET Framework. See [SimulatedMSBuildReferenceResolver.fs](https://github.com/dotnet/fsharp/blob/master/src/fsharp/SimulatedMSBuildReferenceResolver.fs) when not using the .NET Framework F# compiler. 
+   See [Microsoft.DotNet.DependencyManager](https://github.com/dotnet/fsharp/tree/master/src/fsharp/Microsoft.DotNet.DependencyManager) for reference resolution and package management used in `fsi`.
 
 * _Importing referenced .NET binaries_, see [import.fsi](https://github.com/dotnet/fsharp/blob/master/src/fsharp/import.fsi)/[import.fs](https://github.com/dotnet/fsharp/blob/master/src/fsharp/import.fs). Accepts file references and produces a Typed Tree node for each referenced assembly, including information about its type definitions (and type forwarders if any).
 
@@ -556,6 +559,24 @@ decide not to bother continuing with the computation (it drops it on the floor)
 
 The second can be interrupted via having `isResultObsolete` to the F# Compiler Service API return true.
 
+### The F# Compiler Service Public Surface Area
+
+The "intended" FCS API is the parts under the namespaces
+
+* FSharp.Compiler.SourceCodeServices.* (analysis, compilation, tooling, lexing)
+* FSharp.Compiler.Interactive.Shell.*  (scripting support)
+* FSharp.Compiler.AbstractIL.*  (for ILAssemblyReader hook for Rider)
+* FSharp.Compiler.SyntaxTree.*  (direct access to full untyped tree)
+
+These sections are generally designed with F#/.NET design conventions (e.g. types in namespaces, not modules, no nesting of modules etc.)
+and we will continue to iterate to make this so.
+
+In contrast, the public parts of the compiler directly under `FSharp.Compiler.*` and `FSharp.AbstractIL.*` are
+"incidental" and not really designed for public use apart from the hook for Jet Brains Rider
+(Aside: In theory all these other parts could be renamed to FSharp.Compiler.Internal though there's no need to do that right now).  
+These internal parts tend to be implemented with the "module containing lots of stuff in one big file" approach for layers of the compiler.
+
+
 ### The F# Compiler Service Operations Queue
 
 See [F# Compiler Service Queue](fsharp-compiler-service-queue.md).

docs/fcs/compiler.fsx

@@ -30,6 +30,7 @@ First, we need to reference the libraries that contain F# interactive service:
 #r "FSharp.Compiler.Service.dll"
 open System.IO
 open FSharp.Compiler.SourceCodeServices
+open FSharp.Compiler.Text
 
 // Create an interactive checker instance 
 let checker = FSharpChecker.Create()

docs/fcs/corelib.fsx

@@ -4,6 +4,25 @@
 Compiler Services: Notes on FSharp.Core.dll
 =================================================
 
+Versions of FSharp.Core involved in the operation of FSharp.Compiler.Service
+---------------------------------------------
+
+There are three versions of FSharp.Core relevant to the operation of FSharp.Compiler.Service:
+
+1. **The FSharp.Compiler.Service.dll static reference to FSharp.Core** - The FCS DLL and nuget have a static minbound dependency on FSharp.Core.
+
+   This is just a normal .NET dependency like any other, it expresses the minimum surface area of FSharp.Core that the implementation of FSharp.Compiler.Service (and any components that depend on it) needs.  It could be a reference to a reference assembly if we supported that.  In theory this could be very low and all is cool - if we could implement FCS in terms of FSharp.Core 2.0.0.0 then that could be the minbound (indeed in theory we could implement FCS pretty almost without any use of FSharp.Core functionality at all, though obviously we don't)
+
+   In practice this is 0-2 versions behind latest FSharp.Core.
+
+2. **The runtime reference to FSharp.Core in a tool, application or test suite that includes FSharp.Compiler.Service** - This is the actual version of FSharp.Core used when, say, fsc.exe or devenv.exe or fsi.exe or fsdocs.exe runs.
+
+   This must be at least as high as (1) and is usually the very latest FSharp.Core available (in or out of repo tree).  This is important to the operation of the FCS-based tool because it is used for execution of scripts, and the default compilation reference for scripts.  If scripts are going to use a particular language feature then this must be sufficient to support the language feature
+
+3. **The FSharp.Core reference in a compilation or analysis being processed by FSharp.Compiler.Service**.
+
+   This can be anything - 2.0.0.0, 4.0.0.0 or 5.0.0 or whatever.  For script compilation and execution is is the same as (2).  It must be sufficient to support language features used in the compilation.
+
 Shipping an FSharp.Core with your application
 ---------------------------------------------
 
@@ -13,32 +32,6 @@ include a copy of FSharp.Core.dll as part of your application.
 For example, if you build a ``HostedCompiler.exe``, you will normally place an FSharp.Core.dll (say 4.3.1.0) alongside
 your ``HostedCompiler.exe``.
 
-Binding redirects for your application
---------------------------------------
-
-The FSharp.Compiler.Service.dll component depends on FSharp.Core 4.4.0.0.  Normally your application will target
-a later version of FSharp.Core, and you may need a [binding redirect](https://docs.microsoft.com/en-us/dotnet/framework/configure-apps/redirect-assembly-versions) to ensure
-that other versions of FSharp.Core forward to the final version of FSharp.Core.dll your application uses.
-Binding redirect files are normally generated automatically by build tools. If not, you can use one like this
-(if your tool is called ``HostedCompiler.exe``, the binding redirect file is called ``HostedCompiler.exe.config``)
-
-Some other dependencies may also need to be reconciled and forwarded.
-
-    <?xml version="1.0" encoding="utf-8" ?>
-    <configuration>
-        <runtime>
-          <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
-            <dependentAssembly>
-              <assemblyIdentity name="FSharp.Core" publicKeyToken="b03f5f7f11d50a3a" culture="neutral"/>
-              <bindingRedirect oldVersion="2.0.0.0-4.4.0.0" newVersion="4.4.1.0"/>
-            </dependentAssembly>
-            <dependentAssembly>
-              <assemblyIdentity name="System.Collections.Immutable" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
-              <bindingRedirect oldVersion="1.0.0.0-1.2.0.0" newVersion="1.2.1.0" />
-            </dependentAssembly>
-          </assemblyBinding>
-        </runtime>
-    </configuration>
 
 Which FSharp.Core and .NET Framework gets referenced in compilation?
 --------------------------------------
@@ -89,6 +82,7 @@ Summary
 
 In this design note we have discussed three things:
 
+- the versions of FSharp.Core relevant to the operation of FSharp.Compiler.Service.dll
 - which FSharp.Core.dll is used to run your compilation tools
 - how  to configure binding redirects for the FSharp.Core.dll used to run your compilation tools
 - which FSharp.Core.dll and/or framework assemblies are  referenced during the checking and compilations performed by your tools.

docs/fcs/editor.fsx

@@ -128,7 +128,6 @@ identifier (the other option lets you get tooltip with full assembly location wh
 
 *)
 // Get tag of the IDENT token to be used as the last argument
-open FSharp.Compiler
 let identToken = FSharpTokenTag.Identifier
 
 // Get tool tip at the specified location

docs/fcs/filesystem.fsx

@@ -21,9 +21,10 @@ open System
 open System.IO
 open System.Collections.Generic
 open System.Text
-open FSharp.Compiler.AbstractIL.Internal.Library
+open FSharp.Compiler.SourceCodeServices
+open FSharp.Compiler.Text
 
-let defaultFileSystem = Shim.FileSystem
+let defaultFileSystem = FileSystem
 
 let fileName1 = @"c:\mycode\test1.fs" // note, the path doesn't exist
 let fileName2 = @"c:\mycode\test2.fs" // note, the path doesn't exist
@@ -91,15 +92,14 @@ let B = File1.A + File1.A"""
             defaultFileSystem.AssemblyLoad assemblyName 
 
 let myFileSystem = MyFileSystem()
-Shim.FileSystem <- MyFileSystem() 
+FileSystem <- MyFileSystem() 
 
 (**
 
 Doing a compilation with the FileSystem 
 ---------------------------------------
 
 *)
-open FSharp.Compiler.SourceCodeServices
 
 let checker = FSharpChecker.Create()
 

docs/fcs/interactive.fsx

@@ -31,8 +31,9 @@ First, we need to reference the libraries that contain F# interactive service:
 *)
 
 #r "FSharp.Compiler.Service.dll"
-open FSharp.Compiler.SourceCodeServices
 open FSharp.Compiler.Interactive.Shell
+open FSharp.Compiler.SourceCodeServices
+open FSharp.Compiler.Text
 
 (**
 To communicate with F# interactive, we need to create streams that represent input and
@@ -227,7 +228,6 @@ based on the declarations executed so far.
 
 You can also request declaration list information, tooltip text and symbol resolution:
 *)
-open FSharp.Compiler
 
 // get a tooltip
 checkResults.GetToolTipText(1, 2, "xxx + xx", ["xxx"], FSharpTokenTag.IDENT)

docs/fcs/ja/corelib.fsx

@@ -13,27 +13,6 @@ FSharp.Compiler.Service.dll を利用するアプリケーションまたはプ
 
 動的コンパイルや動的実行を行う場合、FSharp.Core.optdata と FSharp.Core.sigdata も含める必要があるかもしれませんが、これらについては下記の指針をご覧ください。
 
-あなたのアプリケーションにリダイレクトをバインドする
-----------------------------------------------------
-
-FSharp.Compiler.Service.dll コンポーネントは FSharp.Core 4.3.0.0 に依存しています。通例、あなたのアプリケーションはこれより後のバージョンの FSharp.Core をターゲットにしており、FSharp.Core 4.3.0.0 をあなたのアプリケーションで用いる FSharp.Core.dll の最終バージョンにちゃんと転送させるように[バインド リダイレクト](https://msdn.microsoft.com/ja-jp/library/7wd6ex19(v=vs.110).aspx)が必要になるでしょう。バインド リダイレクト ファイルは通常ビルドツールによって自動的に生成されます。そうでない場合、下記のようなファイル(あなたのツールが ``HostedCompiler.exe`` という名前で、バインド リダイレクト ファイルが ``HostedCompiler.exe.config`` という名前の場合)を使うことが出来ます。
-
-    <?xml version="1.0" encoding="utf-8" ?>
-    <configuration>
-        <runtime>
-          <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
-            <dependentAssembly>
-              <assemblyIdentity name="FSharp.Core" publicKeyToken="b03f5f7f11d50a3a" culture="neutral"/>
-              <bindingRedirect oldVersion="2.0.0.0-4.3.0.0" newVersion="4.3.1.0"/>
-            </dependentAssembly>
-            <dependentAssembly>
-              <assemblyIdentity name="System.Collections.Immutable" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
-              <bindingRedirect oldVersion="1.0.0.0-1.2.0.0" newVersion="1.2.1.0" />
-            </dependentAssembly>
-          </assemblyBinding>
-        </runtime>
-    </configuration>
-
 どの FSharp.Core と .NET フレームワークがコンパイル時に参照される？
 --------------------------------------
 

docs/fcs/ja/editor.fsx

@@ -138,7 +138,6 @@ let checkFileResults =
 
 *)
 // 最後の引数に指定する、IDENTトークンのタグを取得
-open FSharp.Compiler
 
 // 特定の位置におけるツールチップを取得
 let tip = checkFileResults.GetToolTipText(4, 7, inputLines.[1], ["foo"], FSharpTokenTag.Identifier)

docs/fcs/ja/filesystem.fsx

@@ -17,13 +17,12 @@ FileSystemの設定
 以下の例ではディスクからの読み取りを行うような実装をファイルシステムに設定しています:
 *)
 #r "FSharp.Compiler.Service.dll"
-open System
 open System.IO
-open System.Collections.Generic
 open System.Text
-open FSharp.Compiler.AbstractIL.Internal.Library
+open FSharp.Compiler.SourceCodeServices
+open FSharp.Compiler.Text
 
-let defaultFileSystem = Shim.FileSystem
+let defaultFileSystem = FileSystem
 
 let fileName1 = @"c:\mycode\test1.fs" // 注意: 実際には存在しないファイルのパス
 let fileName2 = @"c:\mycode\test2.fs" // 注意: 実際には存在しないファイルのパス
@@ -91,7 +90,7 @@ let B = File1.A + File1.A"""
             defaultFileSystem.AssemblyLoad assemblyName 
 
 let myFileSystem = MyFileSystem()
-Shim.FileSystem <- MyFileSystem() 
+FileSystem <- MyFileSystem() 
 
 (**
 

docs/fcs/ja/interactive.fsx

@@ -43,6 +43,7 @@ F# Interactiveの開始
 
 #r "FSharp.Compiler.Service.dll"
 open FSharp.Compiler.SourceCodeServices
+open FSharp.Compiler.Text
 open FSharp.Compiler.Interactive.Shell
 
 (**
@@ -244,7 +245,6 @@ checkResults.Errors.Length // 1
 要求することもできます:
 
 *)
-open FSharp.Compiler
 
 // ツールチップを取得する
 checkResults.GetToolTipText(1, 2, "xxx + xx", ["xxx"], FSharpTokenTag.IDENT) 

docs/fcs/ja/tokenizer.fsx

@@ -22,6 +22,7 @@ F#のソースコードに対して、トークナイザは
 *)
 #r "FSharp.Compiler.Service.dll"
 open FSharp.Compiler.SourceCodeServices
+open FSharp.Compiler.Text
 (**
 すると `FSharpSourceTokenizer` のインスタンスを作成できるようになります。
 このクラスには2つの引数を指定します。
@@ -32,7 +33,7 @@ open FSharp.Compiler.SourceCodeServices
 ファイル名はソースコードの位置を特定する場合にのみ指定する必要があります
 (存在しないファイル名でも指定できます):
 *)
-let sourceTok = FSharpSourceTokenizer([], "C:\\test.fsx")
+let sourceTok = FSharpSourceTokenizer([], Some "C:\\test.fsx")
 (**
 `sourceTok` オブジェクトを使用することでF#ソースコードの各行を
 (繰り返し)トークン化することができます。

docs/fcs/tokenizer.fsx

@@ -21,6 +21,7 @@ To use the tokenizer, reference `FSharp.Compiler.Service.dll` and open the
 *)
 #r "FSharp.Compiler.Service.dll"
 open FSharp.Compiler.SourceCodeServices
+open FSharp.Compiler.Text
 (**
 Now you can create an instance of `FSharpSourceTokenizer`. The class takes two 
 arguments - the first is the list of defined symbols and the second is the

docs/fcs/untypedtree.fsx

@@ -84,7 +84,7 @@ code](https://github.com/fsharp/fsharp/blob/master/src/fsharp/ast.fs#L464).
 
 The relevant parts are in the following namespace:
 *)
-open FSharp.Compiler.Ast
+open FSharp.Compiler.SyntaxTree
 (**
 
 When processing the AST, you will typically write a number of mutually recursive functions
@@ -126,7 +126,8 @@ options). In the following, we only show how to handle `if .. then ..` and `let
 /// Walk over an expression - if expression contains two or three 
 /// sub-expressions (two if the 'else' branch is missing), let expression
 /// contains pattern and two sub-expressions
-let rec visitExpression = function
+let rec visitExpression e = 
+  match e with
   | SynExpr.IfThenElse(cond, trueBranch, falseBranchOpt, _, _, _, _) ->
       // Visit all sub-expressions
       printfn "Conditional:"