Switch to the new Mono.Unix nuget #5971
Conversation
|
Note that it uses a branch of |
|
The many tests fail because And there doesn't seem to be a good way to make hello gentle people, I'm running into the situation described here dotnet/msbuild#1887 https://gist.github.com/grendello/5a1f796b17cff10ba7f7d0e8ba0c0310#file-gistfile1-txt-L184-L186 - this is the tree of the nuget one of my thoughts was to use
|
grendello
force-pushed
the
new-mono.posix
branch
3 times, most recently
from
b5737fc to
ea3075b
Compare
src/Xamarin.Android.Build.Tasks/Tasks/PrependEnvironmentVariable.cs
Outdated
Show resolved
Hide resolved
|
This looks like a much simpler approach :D 👍 👍 on and have a 🦖 too cos 🦖 make everything better :D |
grendello
changed the title
|
monodroid diff: https://github.com/xamarin/monodroid/compare/76c04cd15eca7afc269a6d26296e9d2db6f79be2...1f2ce156245ef1bf63ec8014882d283b3224216b What's potentially odd is that the Legacy I believe that "Package size difference" means that the new My guess is that the I need to figure out how to validate this guess. |
|
Verification? I installed current main, 11.3.99.58: https://artprodscussu1.artifacts.visualstudio.com/A011b8bdf-6d56-4f87-be0d-0092136884d9/0bdbc590-a062-4c3f-b0f6-9383f67865ee/_apis/artifact/cGlwZWxpbmVhcnRpZmFjdDovL2RldmRpdi9wcm9qZWN0SWQvMGJkYmM1OTAtYTA2Mi00YzNmLWIwZjYtOTM4M2Y2Nzg2NWVlL2J1aWxkSWQvNDg0OTg4Mi9hcnRpZmFjdE5hbWUvaW5zdGFsbGVycy11bnNpZ25lZA2/content?format=file&subPath=%2Fxamarin.android-11.3.99.58.pkg I then built I also downloaded the (hopefully) equivalent "Success", The 195KB "Package size difference" is not explained by any entry differences. This looks "similar enough". What about the At this point I cast about for tools which will tell me offsets of data, on the assumption that the "extra size" may be due to inserted padding between entries, or something. I come across zipdetails, and find a difference, though I don't understand it: It appears, via
This is where things confuse me: the length of the field [sic] at 09C7EA is 0x0DF07F for main, 0x0E9870 for this PR. What is the payload at 09C7EA? It's Yes …the same uncompressed size, but different compressed sizes! I think we're onto something here: the compressed size of Is It looks like the new libZipSharp compresses |
|
It seems the difference in |
|
So, I checked whether using compression level 9 would make a difference and it did not. |
|
I will create a new version of LibZipSharp with some further tweaks to see whether we can get the old compression ratio back with |
Context: dotnet/android#5971 (comment) Also, attempt to tweak `zlib-ng` to use the best compression possible, by sacrificing speed.
Context: dotnet/android#5971 (comment) Also, attempt to tweak `zlib-ng` to use the best compression possible, by sacrificing speed.
I'm not sure. A few additional things that worry me:
Back on (2), if I run a "side-by-side" comparison of
Then we hit native libs (main vs pr):
The more I look, the more worried I am about how entries are being compressed. Returning to standard zlib increasingly looks preferrable. |
Context: dotnet/android#5971 (comment) Context: dotnet/android#5971 (comment) Context: zlib-ng/zlib-ng#850 Context: https://github.com/zlib-ng/zlib-ng/wiki/Deflate-config-comparison `zlib-ng` is mostly about speed and efficiency, however it achieves its goal by sometimes sacrificing the compression ratio which then affects Xamarin.Android, the main LibZipSharp consumer. Add a new `zlib` submodule and switch to using `zlib` by default instead of `zlib-ng`, while retaining `zlib-ng` as an option.
Context: dotnet/android#5971 (comment) Context: dotnet/android#5971 (comment) Context: zlib-ng/zlib-ng#850 Context: https://github.com/zlib-ng/zlib-ng/wiki/Deflate-config-comparison `zlib-ng` is mostly about speed and efficiency, however it achieves its goal by sometimes sacrificing the compression ratio which then affects Xamarin.Android, the main LibZipSharp consumer. Add a new `zlib` submodule and switch to using `zlib` by default instead of `zlib-ng`, while retaining `zlib-ng` as an option.
Context: dotnet/android#5971 (comment) Context: dotnet/android#5971 (comment) Context: zlib-ng/zlib-ng#850 Context: https://github.com/zlib-ng/zlib-ng/wiki/Deflate-config-comparison `zlib-ng` is mostly about speed and efficiency, however it achieves its goal by sometimes sacrificing the compression ratio which then affects Xamarin.Android, the main LibZipSharp consumer. Add a new `zlib` submodule and switch to using `zlib` by default instead of `zlib-ng`, while retaining `zlib-ng` as an option.
grendello
force-pushed
the
new-mono.posix
branch
from
4e0f357 to
82a0070
Compare
`Xamarin.Android` uses the Mono's Mono.Posix assembly on Unix machines to perform tasks not possible with BCL classes, provided so far by the `Mono.Posix.NETStandard` nuget. Unfortunately, that nuget doesn't work with `dotnet` and so the `Mono.Posix` code has recently been extracted from the Mono repository and placed in its own, from which the new `Mono.Unix` nuget is built and published. This commit switches to `Mono.Unix` in Xamarin.Android and also updates a number of submodules which likewise migrated from `Mono.Posix.NETStandard` to `Mono.Unix` Additionally, LibZipSharp is updated as well, since it now uses `Mono.Unix` `Mono.Unix` no longer uses the older `libMonoPosixHelper` dynamic library, replaced by a new `libMono.Unix` one. This change, however, broke a number of tests since the `Mono.Unix.dll` assembly was no longer able to find its companion native shared library. The reason is that `Mono.Posix.NETStandard` used the "standard" helper library which was also part of the Mono distribution that we use on all the Unix machines. However, that meant the helper library from `Mono.Posix.NETStandard` was never used, instead the Mono copy was loaded and everything worked as expected. `Mono.Unix`'s new native helper library, however, must be taken from the nuget and both Mono and dotnet runtimes must be told where from to load the library once a p/invoke into it is encountered in managed code. The native library is copied from the nuget to the referencing application's output directory and it should be loaded from there. This proved to be easy for the "legacy" Mono - a simple `dllmap` configuration is provided and everything works as it should. With `dotnet` however, dllmap doesn't work. `dotnet` has instead a number (5 I think) mechanisms to configure where the native libraries can be found. Unfortunately, the mechanisms either require that a main executable of the application calls the APIs on entry (e.g. in the `Main`) method or that a JSON configuration file is provided for the application, telling the runtime where the native libraries reside. In case of `Xamarin.Android.Build.Tasks` there is no main executable we can configure, since it works in the msbuild context, providing tasks and utilities to build Xamarin.Android apps. In such instance, `dotnet` could be persuaded to find the libraries by calling one of the 5 APIs. The problem with this approach, however, is that this action would have to be performed at **every** possible entry point to the `Xamarin.Android.Build.Tasks` assembly, since any of them could be used as the first one. While certainly possible, it would be both fragile and an unnecessary maintenance burden. Instead, a simpler (albeit a bit kludgy) solution was chosen. The `Xamarin.Android.Build.Tasks` build process now takes care of generating a fat (multi-architecture) binary for macOS hosts (including `x86-64` and `arm64` architectures) using the `lipo` utility, then it copies the resulting binary to the same directory where `Xamarin.Android.Build.Tasks.dll` and `Mono.Unix.dll` live. The Linux shared library is also copied to the same location. The `dotnet` runtime is able to find and load shared libraries that are in the same directory as the assembly that needs them and everything works as expected.
grendello
force-pushed
the
new-mono.posix
branch
from
82a0070 to
5ea632f
Compare
Use Mono.Unix NuGet package
Changes: https://github.com/xamarin/monodroid/compare/76c04cd15eca7afc269a6d26296e9d2db6f79be2...1f2ce156245ef1bf63ec8014882d283b3224216b
* xamarin/monodroid@1f2ce1562: [tools/msbuild] only run _GetPrimaryCpuAbi for Fast Dev (#1208)
* xamarin/monodroid@691310ede: Bump android-sdk-installer to use Mono.Unix (#1207)
* xamarin/monodroid@48843fcb2: [tools/msbuild] <GetPrimaryCpuAbi/> selects backup RIDs for .NET 6 (#1205)
* xamarin/monodroid@4af48f54b: [tools/fastdev] Add error checking when writing data to disk. (#1204)
* xamarin/monodroid@65b7b2dd4: [tools/fastdev] Rework Unix timestamp calculation code in xamarin.find. (#1202)
* xamarin/monodroid@401f170e9: [build] fix `dotnet tool install` command (#1203)
* xamarin/monodroid@a879b250b: Bump to xamarin/xamarin-android@032d840, xamarin/androidtools@355d015 (#1199)
* xamarin/monodroid@9f9ee378c: [tools/msbuild] Missing translations for XA0135 (#1198)
Changes: https://github.com/xamarin/xamarin-android-tools/compare/683f37508b56c76c24b3287a5687743438625341...49936d60ce808d706f9e4c1ace71c2fe9c604733
* xamarin/xamarin-android-tools@49936d6: Merge pull request #124 from xamarin/update-libzipsharp
* xamarin/xamarin-android-tools@ef78dfc: Bump LibZipSharp to 2.0.0-alpha6
* xamarin/xamarin-android-tools@e3d708c: [BaseTasks] fix `\`-delimited paths on macOS (#122)
* xamarin/xamarin-android-tools@bdcf899: Reference the new Mono.Unix nuget (#123)
* xamarin/xamarin-android-tools@90d7621: [BaseTasks] add ABI detection for RIDs (#121)
* xamarin/xamarin-android-tools@79e3b97: [JdkInfo] handle invalid XML from /usr/libexec/java_home (#120)
* xamarin/xamarin-android-tools@81519fe: Add SECURITY.md (#119)
Xamarin.Android uses the Mono's Mono.Posix assembly on Unix machines
to perform tasks not possible with BCL classes, provided by the
[`Mono.Posix.NETStandard` NuGet][0]. The `Mono.Posix` source has been
extracted into the [mono/mono.posix repo][1], which is used to build
the new [`Mono.Unix` NuGet package][2].
Update the xamarin/xamarin-android repo -- and various dependencies --
to use the `Mono.Unix` package instead of `Mono.Posix.NETStandard`.
This includes the [`Xamarin.LibZipSharp` NuGet][3], as of
xamarin/LibZipSharp@cf5e33c6.
`Mono.Unix` no longer uses the older `libMonoPosixHelper` dynamic
library, replaced by a new `libMono.Unix` native library.
Unfortunately, this change broke a number of tests since the
`Mono.Unix.dll` assembly was no longer able to find its companion
native shared library. While the `Mono.Posix.NETStandard` NuGet
package provides the `libMonoPosixHelper` native library, in practice
the *actual* `libMonoPosixHelper` that was used was the "system"
library included with the system mono.
`Mono.Unix`'s new native helper library, however, must be taken from
the NuGet and both the Mono and dotnet runtimes must be told where to
load the library from once a P/Invoke into `Mono.Unix` is encountered
in managed code. The native library is copied from the NuGet to the
referencing application's output directory and it should be loaded from
there. This proved to be easy for the "legacy" Mono: a simple
[`dllmap` configuration][4] and everything works as it should.
With `dotnet` however, dllmap doesn't work. `dotnet` has instead a
number of mechanisms to configure where the native libraries can be
found (5 I think). Unfortunately, the mechanisms either require that
a main executable of the application calls the APIs on entry (e.g. in
the `Main()` method) or that a JSON configuration file is provided for
the application, telling the runtime where the native libraries reside.
In case of `Xamarin.Android.Build.Tasks` there is no main executable we
can configure, since it works in the MSBuild context, providing tasks
and utilities to build Xamarin.Android apps. In this instance, `dotnet`
could be persuaded to find the libraries by calling one of the 5 APIs.
The problem with this approach, however, is that this action would have
to be performed at **every** possible entry point to the
`Xamarin.Android.Build.Tasks` assembly, since any of them could be used
as the first one. While certainly possible, it would be both fragile
and an unnecessary maintenance burden. Instead, a simpler (albeit a
bit kludgy) solution was chosen: the `src/Xamarin.Android.Build.Tasks`
build process now takes care of generating a fat (multi-architecture)
binary for macOS hosts (including `x86-64` and `arm64` architectures)
using the `lipo` utility, then it copies the resulting binary to the
same directory where `Xamarin.Android.Build.Tasks.dll` and
`Mono.Unix.dll` live. The Linux shared library is also copied to the
same location. The `dotnet` runtime is able to find and load shared
libraries that are in the same directory as the assembly that needs
them and everything works as expected.
[0]: https://www.nuget.org/packages/Mono.Posix.NETStandard/5.20.1-preview
[1]: https://github.com/mono/mono.posix
[2]: https://www.nuget.org/packages/Mono.Unix/
[3]: https://www.nuget.org/packages/Xamarin.LibZipSharp
[4]: https://www.mono-project.com/docs/advanced/pinvoke/dllmap/ |
Xamarin.Androiduses the Mono's Mono.Posix assembly on Unix machinesto perform tasks not possible with BCL classes, provided so far by the
Mono.Posix.NETStandardnuget. Unfortunately, that nuget doesn't workwith
dotnetand so theMono.Posixcode has recently been extractedfrom the Mono repository and placed in its own, from which the new
Mono.Unixnuget is built and published.This commit switches to
Mono.Unixin Xamarin.Android and also updatesa number of submodules which likewise migrated from
Mono.Posix.NETStandardtoMono.UnixAdditionally, LibZipSharp is updated as well, since it now uses
Mono.UnixMono.Unixno longer uses the olderlibMonoPosixHelperdynamiclibrary, replaced by a new
libMono.Unixone. This change, however,broke a number of tests since the
Mono.Unix.dllassembly was no longerable to find its companion native shared library. The reason is that
Mono.Posix.NETStandardused the "standard" helper library which wasalso part of the Mono distribution that we use on all the Unix machines.
However, that meant the helper library from
Mono.Posix.NETStandardwasnever used, instead the Mono copy was loaded and everything worked as
expected.
Mono.Unix's new native helper library, however, must be taken from thenuget and both Mono and dotnet runtimes must be told where from to load
the library once a p/invoke into it is encountered in managed code. The
native library is copied from the nuget to the referencing application's
output directory and it should be loaded from there. This proved to be
easy for the "legacy" Mono - a simple
dllmapconfiguration is providedand everything works as it should.
With
dotnethowever, dllmap doesn't work.dotnethas instead anumber (5 I think) mechanisms to configure where the native libraries
can be found. Unfortunately, the mechanisms either require that a main
executable of the application calls the APIs on entry (e.g. in the
Main) method or that a JSON configuration file is provided for theapplication, telling the runtime where the native libraries reside. In
case of
Xamarin.Android.Build.Tasksthere is no main executable we canconfigure, since it works in the msbuild context, providing tasks and
utilities to build Xamarin.Android apps. In such instance,
dotnetcould be persuaded to find the libraries by calling one of the 5 APIs.
The problem with this approach, however, is that this action would have
to be performed at every possible entry point to the
Xamarin.Android.Build.Tasksassembly, since any of them could be usedas the first one. While certainly possible, it would be both fragile
and an unnecessary maintenance burden. Instead, a simpler (albeit a bit
kludgy) solution was chosen. The
Xamarin.Android.Build.Tasksbuildprocess now takes care of generating a fat (multi-architecture) binary
for macOS hosts (including
x86-64andarm64architectures) using thelipoutility, then it copies the resulting binary to the samedirectory where
Xamarin.Android.Build.Tasks.dllandMono.Unix.dlllive. The Linux shared library is also copied to the same location.
The
dotnetruntime is able to find and load shared libraries that arein the same directory as the assembly that needs them and everything
works as expected.