Compile for iOS #569

AlexandrGraschenkov · 2015-05-13T14:02:39Z

I need to compile this library for iPhone. Target architecture is arm64 and armv7. I set default clang compiler to use and arm64 arch.

make CC=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang ARCH=arm64

It fails with error:
make[1]: *** No rule to make target "../kernel/arm64/amax.S", needed by "samax_k.o". Stop.

It's successfully compile if I do not set ARCH=arm64. Can you give advice how to compile library with arm64 and armv7 support.

Also if I add arch flag to compiler, so compile going like so:

Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang -O2 -arch arm64 -miphoneos-version-min=5.0 -DMAX_STACK_ALLOC=2048 -DEXPRECISION  -Wall -m64 -DF_INTERFACE_GFORT -fPIC -DSMP_SERVER -DNO_WARMUP -DMAX_CPU_NUMBER=8 -DASMNAME=_saxpy -DASMFNAME=_saxpy_ -DNAME=saxpy_ -DCNAME=saxpy -DCHAR_NAME=\"saxpy_\" -DCHAR_CNAME=\"saxpy\" -DNO_AFFINITY -I.. -I. -UDOUBLE  -UCOMPLEX -c axpy.c -o saxpy.o

I start get error like this.

/usr/include/sys/cdefs.h:680:2: error: Unsupported architecture
/usr/include/sys/cdefs.h:680:2#error Unsupported architecture:
/usr/include/sys/cdefs.h:680:2: error: Unsupported architecture
/usr/include/sys/cdefs.h:680:2: error: Unsupported architecture
unknown type name '__uint32_t'; did you mean '__uint128_t'?
error: unknown type name '__darwin_va_list'
error: unknown type name '__darwin_size_t'
// and others

Thank you.

The text was updated successfully, but these errors were encountered:

martin-frbg · 2015-05-13T14:48:02Z

Closed ticket #523 looks relevant. Also please try the develop branch if you are currently using one of the releases - I believe some ARM-related fixes went in after 0.2.14

xianyi · 2015-05-13T14:59:06Z

@AlexandrGraschenkov , I didn't try to compile OpenBLAS for iOS before.

AlexandrGraschenkov · 2015-05-13T16:39:24Z

Thank you for reply. I've try to make TARGET=ARMV8 but get error on compile:

symm.c:388:36: error: use of undeclared identifier 'QGEMM_DEFAULT_P'
gemm.c:397:37: error: use of undeclared identifier 'QGEMM_DEFAULT_P'
gemm.c:397:46: error: use of undeclared identifier 'QGEMM_DEFAULT_Q'
// and so on

With make TARGET=ARMV8 ARCH=arm64 more code get compiled but still get errors:

../kernel/arm64/../arm/scal.c:55:2: error: non-void function 'sscal_k' should return a value [-Wreturn-type]

After change to return 0; get errors:

../kernel/arm64/sgemm_kernel_4x4.S:759:19: error: unknown token in expression
        add sp,sp,#-(5*16)
                  ^
../kernel/arm64/sgemm_kernel_4x4.S:760:19: error: brackets expression not supported on this target
        stp d8,d9,[sp,#(0*16)]
                  ^
../kernel/arm64/sgemm_kernel_4x4.S:761:21: error: brackets expression not supported on this target
        stp d10,d11,[sp,#(1*16)]
                    ^
../kernel/arm64/sgemm_kernel_4x4.S:762:21: error: brackets expression not supported on this target
        stp d12,d13,[sp,#(2*16)]
                    ^
../kernel/arm64/sgemm_kernel_4x4.S:763:21: error: brackets expression not supported on this target
        stp d14,d15,[sp,#(3*16)]
                    ^
../kernel/arm64/sgemm_kernel_4x4.S:764:21: error: brackets expression not supported on this target
        stp d16,d17,[sp,#(4*16)]
                    ^
../kernel/arm64/sgemm_kernel_4x4.S:766:22: error: unexpected token in argument list
        mov x12, v0.d[0]
// and so on

martin-frbg · 2015-05-13T20:09:06Z

Googling for the "brackets expression..." error message finds some cases where the cause appeared to be mis-detection of the target architecture, e.g.
project-imas/security-check#3
(I assume you did issue "make clean" between your various attempts ?)

tchernitski · 2015-06-15T06:59:12Z

Hi Alexander,

Have you solved the compilation problem? I have exactly the same issue.

AlexandrGraschenkov · 2015-06-15T13:00:34Z

Hi @tchernitski
I'm not solved the problem. I compile one project witch uses Open-BLAS. I decide to complie project without library, and change code usage of Open-BLAS to Accelerate framework functions. But project successfully compile for iOS, and I stop dig into this problem.

tchernitski · 2015-06-16T06:19:24Z

Thanks for answer, but I would like to use something that is less platform-dependend

sstepashka · 2016-04-01T10:41:48Z

Hello,

Do you have any progress of this issue?

tchernitski · 2016-04-01T12:48:44Z

Hi,
I replaced OpenBlas by Eigen in my project

ivan-ushakov · 2017-02-26T18:13:41Z

Hello,
Not sure if this still actual but here is how I did build for ARM64 arch.

In file c_check add following line for $cross_suffix.

 $cross_suffix = "";

 if (dirname($compiler_name) ne ".") {
     $cross_suffix .= dirname($compiler_name) . "/";
 }

 if (basename($compiler_name) =~ /([^\s]*-)(.*)/) {
     $cross_suffix .= $1;
 }

 $cross_suffix = "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/";

You need this because we will provide CC with arguments and Perl function for extracting directory doesn't work well.

Create file with following content.

 TOOLCHAIN_PATH=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
 SYSROOT_PATH=/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS10.2.sdk
 make TARGET=ARMV8 BINARY=64 HOSTCC=clang CC="$TOOLCHAIN_PATH/clang -isysroot $SYSROOT_PATH -arch arm64" NOFORTRAN=1 libs

Now you can build library.

martin-frbg · 2017-02-26T18:44:21Z

Thank you very much for this. I wonder if your addition to the c_check file can be made conditional on something like
if ( $data =~ /OS_DARWIN/ && $data =~ /ARCH_ARM/)
which should allow adding it to the current code without breaking other platforms ? Then users would only need to create your build script (which I think could be added as "quickbuild.ios", if the toolchain path can be expected to be fairly constant). Unfortunately I do not have any Apple hardware to try this myself.

ivan-ushakov · 2017-02-28T18:37:50Z

I guess we need to find how get directory name from path like this:

/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS10.2.sdk -arch arm64

Functions like dirname() or basename() don't work well for such path.

martin-frbg · 2017-02-28T19:37:34Z

I understand the sysroot path will contain a version number and/or target platform name, so your build script would always need to be edited by the user. Can we rely on the toolchain path that you put in c_check to be constant ? Then at least the user would not have to change any files from OpenBLAS if that line was present by default - less chance to get something wrong.

martin-frbg · 2017-03-01T10:25:46Z

Copied your text to the wiki as https://github.com/xianyi/OpenBLAS/wiki/How-to-build-OpenBLAS-for-iPhone-iOS

sstepashka · 2017-03-01T14:59:25Z

Will your solution work for armv7?

ivan-ushakov · 2017-03-01T18:26:24Z

@sstepashka
No, I tried to build for armv7 arch but got error about CPU/FPU features.

tamworth · 2017-03-05T13:28:48Z

hi, all, I build the develop branches to use the solution, it has no error, but just generate .a library(the size only 3m) ，when I use "make" to build the x86 library，it generate 2 library(.a and .dylib, the size both more than 10m), I want to use the dylib
by the way, it has a warning:"clang: warning: using sysroot for 'iPhoneOS' but targeting 'MacOSX'"
is it right?

ccy022364 · 2017-03-07T11:31:18Z

@tamworth Did you build for armv7?

sstepashka · 2017-03-07T12:03:11Z

I've found Accelerate.framework with BLAS functions. Maybe you shouldn't use OpenBLAS for iOS?
See https://developer.apple.com/reference/accelerate/blas

ccy022364 · 2017-03-07T12:08:50Z

Thanks,I kown Accelerate.framework.Now we want to test the Openblas and Accelerate.framework,which one is faster.

tamworth · 2017-03-07T15:50:16Z

@ccy022364 I build for arm64

martin-frbg · 2017-03-07T16:03:13Z

@tamworth googling for the error message led me to https://curl.haxx.se/mail/lib-2016-10/0094.html which seems to suggest you may have "-mmacosx-version-min" or similar set somewhere in your default clang options (but I know nothing about cross-compiling for iOS on OSX)
And if someone of you who tried to build this for armv7 could post the error message (about wrong or missing CPU features apparently ?) maybe this will provide a clue for the ARM experts here

ccy022364 · 2017-03-08T10:14:32Z

When I use arm64 library,it failed with error,
Undefined symbols for architecture arm64:
"_sgemm_kernel", referenced from:
_sgemm_nn in libopenblas_armv8p-r0.2.20.dev.a(sgemm_nn.o)
_sgemm_nt in libopenblas_armv8p-r0.2.20.dev.a(sgemm_nt.o)
_inner_thread in libopenblas_armv8p-r0.2.20.dev.a(sgemm_thread_nn.o)
_inner_thread in libopenblas_armv8p-r0.2.20.dev.a(sgemm_thread_nt.o)
_inner_thread in libopenblas_armv8p-r0.2.20.dev.a(sgemm_thread_tn.o)
_inner_thread in libopenblas_armv8p-r0.2.20.dev.a(sgemm_thread_tt.o)
_sgemm_tn in libopenblas_armv8p-r0.2.20.dev.a(sgemm_tn.o)
...
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

thank you

tamworth · 2017-03-10T16:27:31Z

@martin-frbg Thank you for your reply, it works, this is my script:

SYSROOT_PATH=/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS10.2.sdk
make clean
make TARGET=ARMV8 BINARY=64 HOSTCC=clang CC="$TOOLCHAIN_PATH/clang -isysroot $SYSROOT_PATH -arch arm64 -miphoneos-version-min=8.0 -O2" NOFORTRAN=1  libs

tamworth · 2017-03-11T14:41:32Z

@ccy022364 This is my test project, It working on arm64 device https://github.com/tamworth/BLASCompare

In this project, I use same function to make a test between openblas and Accelerate.framewrok, It seems that openblas is far slower than Accelerate.framewrok, I'm a newbie of openblas, I don't know the results whether it's right

ashwinyes · 2017-03-11T14:54:20Z

Try building OpenBLAS for TARGET=CORTEXA57 and compare the the performance on arm64.

Also what APIs are you comparing??

tamworth · 2017-03-11T18:30:41Z

@ashwinyes I comparing with Accelerate.framewrok, it's Apple's cblas library,
I run 10000000 times on iPhone 5S with:

    double A[6] = {1.0,2.0,1.0,-3.0,4.0,-1.0};
    double B[6] = {1.0,2.0,1.0,-3.0,4.0,-1.0};
    double C[9] = {.5,.5,.5,.5,.5,.5,.5,.5,.5};
    cblas_dgemm(CblasColMajor, CblasNoTrans, CblasTrans,3,3,2,1,A, 3, B, 3,2,C,3);

It took 9s on openblas, and it only took 2s on Accelerate.framewrok

I have try to build withTARGET=CORTEXA57, It caused so many error, And I'm not sure the CORTEXA57 is applicable to iPhone

ashwinyes · 2017-03-11T18:38:18Z

Thanks for testing.

CORTEXA57 should work on all arm64 processors having Advanced SIMD support which I believe should be there for iphone.

Could you share the compile errors that you are seeing for CORTEXA57 target ??

FYI. TARGET=ARM64 just uses the generic C implementations. So not much performance is to be expected. On actual cortexa57 processor, TARGET=CORTEXA57 is 3-4x times faster than TARGET=ARM64 if i remember correctly.

ashwinyes · 2017-03-11T18:46:23Z

Also I believe the matrix sizes that you are using is very small and not appropriate for benchmarking OpenBLAS as it is optimized for larger matrices.

Would request you to try with larger matrix sizes (starting from 100x100) with lesser number if iterations.

tamworth · 2017-03-12T04:25:18Z

@ashwinyes Here is the error message

<instantiation>:4:21: error: unexpected token in argument list
 beq KERNEL_F1_NEXT_\@
                    ^
<instantiation>:1:1: note: while in macro instantiation
KERNEL_F1
^
../kernel/arm64/nrm2.S:150:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:7:27: error: unexpected token in argument list
 bge KERNEL_F1_SCALE_GE_X_\@
                          ^
<instantiation>:1:1: note: while in macro instantiation
KERNEL_F1
^
../kernel/arm64/nrm2.S:150:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:13:19: error: unexpected token in argument list
 b KERNEL_F1_NEXT_\@
                  ^
<instantiation>:1:1: note: while in macro instantiation
KERNEL_F1
^
../kernel/arm64/nrm2.S:150:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:14:22: error: unknown token in expression
KERNEL_F1_SCALE_GE_X_\@:
                     ^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS10.2.sdk -arch arm64 -miphoneos-version-min=8.0 -O2 -c -O2 -DMAX_STACK_ALLOC=2048 -Wall -DF_INTERFACE_GFORT -fPIC -DNO_LAPACK -DNO_LAPACKE -DSMP_SERVER -DNO_WARMUP -DMAX_CPU_NUMBER=4 -march=armv8-a+crc+crypto+fp+simd -mtune=cortex-a57 -DASMNAME=_saxpby_k -DASMFNAME=_saxpby_k_ -DNAME=saxpby_k_ -DCNAME=saxpby_k -DCHAR_NAME=\"saxpby_k_\" -DCHAR_CNAME=\"saxpby_k\" -DNO_AFFINITY -I.. -UDOUBLE  -UCOMPLEX -UCOMPLEX -UDOUBLE ../kernel/arm64/../arm/axpby.c -o saxpby_k.o
<instantiation>:1:1: note: while in macro instantiation
KERNEL_F1
^
../kernel/arm64/nrm2.S:150:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:14:22: error: invalid operand
KERNEL_F1_SCALE_GE_X_\@:
                     ^
<instantiation>:1:1: note: while in macro instantiation
KERNEL_F1
^
../kernel/arm64/nrm2.S:150:2: note: while in macro instantiation
 KERNEL_F8
 ^
../kernel/arm64/nrm2.S:87:16: error: unknown token in expression
KERNEL_F1_NEXT_\@:
               ^
<instantiation>:1:1: note: while in macro instantiation
KERNEL_F1
^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
../kernel/arm64/nrm2.S:87:16: error: invalid operand
KERNEL_F1_NEXT_\@:
               ^
<instantiation>:1:1: note: while in macro instantiation
KERNEL_F1
^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:4:21: error: unexpected token in argument list
 beq KERNEL_F1_NEXT_\@
                    ^
<instantiation>:2:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:7:27: error: unexpected token in argument list
 bge KERNEL_F1_SCALE_GE_X_\@
                          ^
<instantiation>:2:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:13:19: error: unexpected token in argument list
 b KERNEL_F1_NEXT_\@
                  ^
<instantiation>:2:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:14:22: error: unknown token in expression
KERNEL_F1_SCALE_GE_X_\@:
                     ^
<instantiation>:2:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:14:22: error: invalid operand
KERNEL_F1_SCALE_GE_X_\@:
                     ^
<instantiation>:2:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
../kernel/arm64/nrm2.S:87:16: error: unknown token in expression
KERNEL_F1_NEXT_\@:
               ^
<instantiation>:2:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
../kernel/arm64/nrm2.S:87:16: error: invalid operand
KERNEL_F1_NEXT_\@:
               ^
<instantiation>:2:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:4:21: error: unexpected token in argument list
 beq KERNEL_F1_NEXT_\@
                    ^
<instantiation>:3:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:7:27: error: unexpected token in argument list
 bge KERNEL_F1_SCALE_GE_X_\@
                          ^
<instantiation>:3:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:13:19: error: unexpected token in argument list
 b KERNEL_F1_NEXT_\@
                  ^
<instantiation>:3:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:14:22: error: unknown token in expression
KERNEL_F1_SCALE_GE_X_\@:
                     ^
<instantiation>:3:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:14:22: error: invalid operand
KERNEL_F1_SCALE_GE_X_\@:
                     ^
<instantiation>:3:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
../kernel/arm64/nrm2.S:87:16: error: unknown token in expression
KERNEL_F1_NEXT_\@:
               ^
<instantiation>:3:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
../kernel/arm64/nrm2.S:87:16: error: invalid operand
KERNEL_F1_NEXT_\@:
               ^
<instantiation>:3:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:4:21: error: unexpected token in argument list
 beq KERNEL_F1_NEXT_\@
                    ^
<instantiation>:4:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:7:27: error: unexpected token in argument list
 bge KERNEL_F1_SCALE_GE_X_\@
                          ^
<instantiation>:4:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:13:19: error: unexpected token in argument list
 b KERNEL_F1_NEXT_\@
                  ^
<instantiation>:4:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:14:22: error: unknown token in expression
KERNEL_F1_SCALE_GE_X_\@:
                     ^
<instantiation>:4:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:14:22: error: invalid operand
KERNEL_F1_SCALE_GE_X_\@:
                     ^
<instantiation>:4:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
../kernel/arm64/nrm2.S:87:16: error: unknown token in expression
KERNEL_F1_NEXT_\@:
               ^
<instantiation>:4:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
../kernel/arm64/nrm2.S:87:16: error: invalid operand
KERNEL_F1_NEXT_\@:
               ^
<instantiation>:4:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:4:21: error: unexpected token in argument list
 beq KERNEL_F1_NEXT_\@
                    ^
<instantiation>:5:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:7:27: error: unexpected token in argument list
 bge KERNEL_F1_SCALE_GE_X_\@
                          ^
<instantiation>:5:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:13:19: error: unexpected token in argument list
 b KERNEL_F1_NEXT_\@
                  ^
<instantiation>:5:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:14:22: error: unknown token in expression
KERNEL_F1_SCALE_GE_X_\@:
                     ^
<instantiation>:5:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:14:22: error: invalid operand
KERNEL_F1_SCALE_GE_X_\@:
                     ^
<instantiation>:5:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
../kernel/arm64/nrm2.S:87:16: error: unknown token in expression
KERNEL_F1_NEXT_\@:
               ^
<instantiation>:5:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
../kernel/arm64/nrm2.S:87:16: error: invalid operand
KERNEL_F1_NEXT_\@:
               ^
<instantiation>:5:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:4:21: error: unexpected token in argument list
 beq KERNEL_F1_NEXT_\@
                    ^
<instantiation>:6:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:7:27: error: unexpected token in argument list
 bge KERNEL_F1_SCALE_GE_X_\@
                          ^
<instantiation>:6:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:13:19: error: unexpected token in argument list
 b KERNEL_F1_NEXT_\@
                  ^
<instantiation>:6:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:14:22: error: unknown token in expression
KERNEL_F1_SCALE_GE_X_\@:
                     ^
<instantiation>:6:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:14:22: error: invalid operand
KERNEL_F1_SCALE_GE_X_\@:
                     ^
<instantiation>:6:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
../kernel/arm64/nrm2.S:87:16: error: unknown token in expression
KERNEL_F1_NEXT_\@:
               ^
<instantiation>:6:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
../kernel/arm64/nrm2.S:87:16: error: invalid operand
KERNEL_F1_NEXT_\@:
               ^
<instantiation>:6:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:4:21: error: unexpected token in argument list
 beq KERNEL_F1_NEXT_\@
                    ^
<instantiation>:7:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:7:27: error: unexpected token in argument list
 bge KERNEL_F1_SCALE_GE_X_\@
                          ^
<instantiation>:7:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:13:19: error: unexpected token in argument list
 b KERNEL_F1_NEXT_\@
                  ^
<instantiation>:7:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:14:22: error: unknown token in expression
KERNEL_F1_SCALE_GE_X_\@:
                     ^
<instantiation>:7:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:14:22: error: invalid operand
KERNEL_F1_SCALE_GE_X_\@:
                     ^
<instantiation>:7:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^                            pplications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS10.2.sdk -arch arm64 -miphoneos-version-min=8.0 -O2 -c -O2 -DMAX_STACK_ALLOC=2048 -Wall -DF_INTERFACE_GFORT -fPIC -DNO_LAPACK -DNO_LAPACKE -DSMP_SERVER -DNO_WARMUP -DMAX_CPU_NUMBER=4 -march=armv8-a+crc+crypto+fp+simd -mtune=cortex-a57 -DASMNAME=_sgemv_n -DASMFNAME=_sgemv_n_ -DNAME=sgemv_n_ -DCNAME=sgemv_n -DCHAR_NAME=\"sgemv_n_\" -DCHAR_CNAME=\"sgemv_n\" -DNO_AFFINITY -I.. -UDOUBLE  -UCOMPLEX -UDOUBLE -UCOMPLEX  -UTRANS ../kernel/arm64/gemv_n.S -o sgemv_n.o
31merror: unknown token in expression
KERNEL_F1_NEXT_\@:
               ^
<instantiation>:7:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
../kernel/arm64/nrm2.S:87:16: error: invalid operand
KERNEL_F1_NEXT_\@:
               ^
<instantiation>:7:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:4:21: error: unexpected token in argument list
 beq KERNEL_F1_NEXT_\@
                    ^
<instantiation>:8:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:7:27: error: unexpected token in argument list
 bge KERNEL_F1_SCALE_GE_X_\@
                          ^
<instantiation>:8:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:13:19: error: unexpected token in argument list
 b KERNEL_F1_NEXT_\@
                  ^
<instantiation>:8:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:14:22: error: unknown token in expression
KERNEL_F1_SCALE_GE_X_\@:
                     ^
<instantiation>:8:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:14:22: error: invalid operand
KERNEL_F1_SCALE_GE_X_\@:
                     ^
<instantiation>:8:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
../kernel/arm64/nrm2.S:87:16: error: unknown token in expression
KERNEL_F1_NEXT_\@:
               ^
<instantiation>:8:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
../kernel/arm64/nrm2.S:87:16: error: invalid operand
KERNEL_F1_NEXT_\@:
               ^
<instantiation>:8:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:4:21: error: unexpected token in argument list
 beq KERNEL_F1_NEXT_\@
                    ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:159:2: note: while in macro instantiation
 KERNEL_F1
 ^
<instantiation>:7:27: error: unexpected token in argument list
 bge KERNEL_F1_SCALE_GE_X_\@
                          ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:159:2: note: while in macro instantiation
 KERNEL_F1
 ^
<instantiation>:13:19: error: unexpected token in argument list
 b KERNEL_F1_NEXT_\@
                  ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:159:2: note: while in macro instantiation
 KERNEL_F1
 ^
<instantiation>:14:22: error: unknown token in expression
KERNEL_F1_SCALE_GE_X_\@:
                     ^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS10.2.sdk -arch arm64 -miphoneos-version-min=8.0 -O2 -c -O2 -DMAX_STACK_ALLOC=2048 -Wall -DF_INTERFACE_GFORT -fPIC -DNO_LAPACK -DNO_LAPACKE -DSMP_SERVER -DNO_WARMUP -DMAX_CPU_NUMBER=4 -march=armv8-a+crc+crypto+fp+simd -mtune=cortex-a57 -DASMNAME=_sgemv_t -DASMFNAME=_sgemv_t_ -DNAME=sgemv_t_ -DCNAME=sgemv_t -DCHAR_NAME=\"sgemv_t_\" -DCHAR_CNAME=\"sgemv_t\" -DNO_AFFINITY -I.. -UDOUBLE  -UCOMPLEX -UDOUBLE -UCOMPLEX  -DTRANS  ../kernel/arm64/gemv_t.S -o sgemv_t.o
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:159:2: note: while in macro instantiation
 KERNEL_F1
 ^
<instantiation>:14:22: error: invalid operand
KERNEL_F1_SCALE_GE_X_\@:
                     ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:159:2: note: while in macro instantiation
 KERNEL_F1
 ^
../kernel/arm64/nrm2.S:87:16: error: unknown token in expression
KERNEL_F1_NEXT_\@:
               ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:159:2: note: while in macro instantiation
 KERNEL_F1
 ^
../kernel/arm64/nrm2.S:87:16: error: invalid operand
KERNEL_F1_NEXT_\@:
               ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:159:2: note: while in macro instantiation
 KERNEL_F1
 ^
make[1]: *** [snrm2_k.o] Error 1
make[1]: *** Waiting for unfinished jobs....
make: *** [libs] Error 1

I build script with:

TOOLCHAIN_PATH=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
SYSROOT_PATH=/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS10.2.sdk
make clean
make TARGET=CORTEXA57 BINARY=64 HOSTCC=clang CC="$TOOLCHAIN_PATH/clang -isysroot $SYSROOT_PATH -arch arm64 -miphoneos-version-min=8.0 -O2" NOFORTRAN=1  libs

Thank you

ashwinyes · 2017-03-12T04:38:56Z

Thanks for the logs.

Looks like the only files that are causing the errors are kernel/arm64/nrm2.S and kernel/arm64/znrm2.S. clang dont seem to understand the @ token in macros used to create unique labels.

You can comment out the following lines in kernel/arm64/KERNEL.CORTEXA57 and do a clean make to get rid of the compilation errors.
SNRM2KERNEL = nrm2.S
DNRM2KERNEL = nrm2.S
CNRM2KERNEL = znrm2.S
ZNRM2KERNEL = znrm2.S

Thanks

tamworth · 2017-03-12T05:44:20Z

@ashwinyes Thank you for your reply, The library is successful compile with your solution, But when I run with Xcode, It caused some error:

Undefined symbols for architecture arm64:
  "_dgemm_incopy", referenced from:
      _inner_thread in libopenblas_cortexa57p-r0.2.20.dev.a(dgemm_thread_tn.o)
      _inner_thread in libopenblas_cortexa57p-r0.2.20.dev.a(dgemm_thread_tt.o)
      _dgemm_tn in libopenblas_cortexa57p-r0.2.20.dev.a(dgemm_tn.o)
      _dgemm_tt in libopenblas_cortexa57p-r0.2.20.dev.a(dgemm_tt.o)
  "_dgemm_otcopy", referenced from:
      _dgemm_nt in libopenblas_cortexa57p-r0.2.20.dev.a(dgemm_nt.o)
      _inner_thread in libopenblas_cortexa57p-r0.2.20.dev.a(dgemm_thread_nt.o)
      _inner_thread in libopenblas_cortexa57p-r0.2.20.dev.a(dgemm_thread_tt.o)
      _dgemm_tt in libopenblas_cortexa57p-r0.2.20.dev.a(dgemm_tt.o)
  "_dgemm_oncopy", referenced from:
      _dgemm_nn in libopenblas_cortexa57p-r0.2.20.dev.a(dgemm_nn.o)
      _inner_thread in libopenblas_cortexa57p-r0.2.20.dev.a(dgemm_thread_nn.o)
      _inner_thread in libopenblas_cortexa57p-r0.2.20.dev.a(dgemm_thread_tn.o)
      _dgemm_tn in libopenblas_cortexa57p-r0.2.20.dev.a(dgemm_tn.o)
  "_dgemm_itcopy", referenced from:
      _dgemm_nn in libopenblas_cortexa57p-r0.2.20.dev.a(dgemm_nn.o)
      _dgemm_nt in libopenblas_cortexa57p-r0.2.20.dev.a(dgemm_nt.o)
      _inner_thread in libopenblas_cortexa57p-r0.2.20.dev.a(dgemm_thread_nn.o)
      _inner_thread in libopenblas_cortexa57p-r0.2.20.dev.a(dgemm_thread_nt.o)
  "_dgemm_kernel", referenced from:
      _dgemm_nn in libopenblas_cortexa57p-r0.2.20.dev.a(dgemm_nn.o)
      _dgemm_nt in libopenblas_cortexa57p-r0.2.20.dev.a(dgemm_nt.o)
      _inner_thread in libopenblas_cortexa57p-r0.2.20.dev.a(dgemm_thread_nn.o)
      _inner_thread in libopenblas_cortexa57p-r0.2.20.dev.a(dgemm_thread_nt.o)
      _inner_thread in libopenblas_cortexa57p-r0.2.20.dev.a(dgemm_thread_tn.o)
      _inner_thread in libopenblas_cortexa57p-r0.2.20.dev.a(dgemm_thread_tt.o)
      _dgemm_tn in libopenblas_cortexa57p-r0.2.20.dev.a(dgemm_tn.o)
      ...
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

ashwinyes · 2017-03-12T06:32:08Z

Now this is a tougher problem.

All the above undefined references are implemented in assembly file for TARGET=CORTEXA57, but are implemented in C files for TARGET=ARMV8. So, I think the linker is unable to find any symbols implemented in assembly files.

I believe you will face the same issue if you try to use SGEMM in TARGET=ARMV8 as its implemented in assembly file.

Looks like OpenBLAS is lacking support for iOS compilation and this needs to be added. I don't have much idea how compilation/linking works in iOS. So will take time for me to figure out as I dont even have a development environment for iOS.

If any iOS experts are here, they will be able to figure this out quickly.

Thanks

ashwinyes · 2017-03-12T08:39:56Z

A quick fix that you can try is in the following branch.
https://github.com/ashwinyes/OpenBLAS/commits/ios_experiment_20170312
The above is based on http://stackoverflow.com/questions/28109826/arm64-using-gas-on-ios

Also, please note that the iOS ARM64 ABI has some divergences from the Standard Procedure Call Standard for the ARM 64-bit Architecture (AAPCS64) . Please refer below link.
https://developer.apple.com/library/content/documentation/Xcode/Conceptual/iPhoneOSABIReference/Articles/ARM64FunctionCallingConventions.html

This may result in unexpected errors as OpenBLAS assembly implementations for ARM64 does not follow the iOS ABI.

Thanks

martin-frbg · 2017-03-12T11:46:01Z

I wonder if this is only a problem with clang's built-in assembler, i.e. if adding "-no-integrated-as" to the CFLAGS would help in any way ?

tamworth · 2017-03-13T14:22:23Z

@martin-frbg No matter ARMV8 or CORTEXA57 also error caused When I used -no-integrated-as.
If I used CORTEXA57 and didn't comment out like this: ashwinyes@db92f4a#diff-6b7ad97b93aa3fd3e259f94aafba00fb

the error is:

/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/as: can't specifiy -Q with -arch arm64
clang: error: assembler command failed with exit code 1 (use -v to see invocation)
make[1]: *** [scopy.o] Error 1
make[1]: *** Waiting for unfinished jobs....
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/as: can't specifiy -Q with -arch arm64
clang: error: assembler command failed with exit code 1 (use -v to see invocation)
make[1]: *** [sscal.o] Error 1
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/as: can't specifiy -Q with -arch arm64
clang: error: assembler command failed with exit code 1 (use -v to see invocation)
make[1]: *** [sswap.o] Error 1
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/as: can't specifiy -Q with -arch arm64
clang: error: assembler command failed with exit code 1 (use -v to see invocation)
make[1]: *** [saxpy.o] Error 1
make: *** [libs] Error 1

tamworth · 2017-03-13T14:43:53Z

@ashwinyes I successfully build and run in ARMV8.
I used to print all functions with 'nm' in ARMV8 and CORTEXA57, Not any undefined symbols function in ARMV8, but it can be found in CORTEXA57 like this:

U _dgemm_incopy
U _dgemm_otcopy

ashwinyes · 2017-03-13T14:58:42Z

Please see the stackoverflow link that I posted in my previous comment.

iOS has an additional underscore at the beginning of the function name which is not followed in the arm64 assembly files. Please try the fix that I posted which adds this underscore to the function names in the assembly files.

ashwinyes · 2017-03-17T04:26:20Z

@tamworth Did you get the opportunity to test the code at https://github.com/ashwinyes/OpenBLAS/commits/ios_experiment_20170312

If yes, please let me know whether it worked for you.

Thanks

brightbetter · 2017-08-16T03:26:06Z

@ashwinyes @tamworth Is it successful?

Thanks

panic-milan · 2017-11-16T12:35:48Z

Hi, I am trying to make Openblas library to work on my project on IOS arm64 target:

@ccy022364 Did you manage to resolve your issue:

Showing All Messages
ld: symbol(s) not found for architecture arm64

"_sgemm_kernel", referenced from ...

I have following issue:
My project uses dlib with BLAS but linking fails with the same message.

Weird is that I used https://github.com/tamworth/BLASCompare project and Openblas library that I built behaves the same way as the one in that XCode project, it works without issues (also prebuilt Openblas library from that project fails to link in my project). Does anyone knows what might be the issue.

supertrouper · 2021-05-15T03:19:43Z

Hello,
Not sure if this still actual but here is how I did build for ARM64 arch.
In file c_check add following line for $cross_suffix.
 $cross_suffix = "";

 if (dirname($compiler_name) ne ".") {
     $cross_suffix .= dirname($compiler_name) . "/";
 }

 if (basename($compiler_name) =~ /([^\s]*-)(.*)/) {
     $cross_suffix .= $1;
 }

 $cross_suffix = "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/";
You need this because we will provide CC with arguments and Perl function for extracting directory doesn't work well.
Create file with following content.
 TOOLCHAIN_PATH=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
 SYSROOT_PATH=/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS10.2.sdk
 make TARGET=ARMV8 BINARY=64 HOSTCC=clang CC="$TOOLCHAIN_PATH/clang -isysroot $SYSROOT_PATH -arch arm64" NOFORTRAN=1 libs
Now you can build library.

Thank you so much

* Created Installation Guide (markdown) * Updated quick installation (markdown) * Updated Home (markdown) * Updated Document (markdown) * Updated Document (markdown) * Updated Document (markdown) * Created Installation Guide (markdown) * Created Home (markdown) * Init version * Updated OpenBLAS Wiki (markdown) * Updated OpenBLAS Wiki (markdown) * Updated OpenBLAS Wiki (markdown) * Updated Document (markdown) * Updated Installation Guide (markdown) * Updated Installation Guide (markdown) * Created Download (markdown) * Created Faq (markdown) * Updated Faq (markdown) * Updated FAQ * Created How to use OpenBLAS in Microsoft Visual Studio (markdown) * Updated Document (markdown) * Updated Faq (markdown) * Updated Faq (markdown) * Updated How to use OpenBLAS in Microsoft Visual Studio (markdown) * Updated Faq (markdown) * Updated OpenBLAS Wiki (markdown) * Updated Home (markdown) * Updated How to use OpenBLAS in Microsoft Visual Studio (markdown) * Created How to generate import library for MingW (markdown) * Updated Document (markdown) * Updated Faq (markdown) * Updated How to use OpenBLAS in Microsoft Visual Studio (markdown) * Updated How to use OpenBLAS in Microsoft Visual Studio (markdown) * Build instrunctions for FreeBSD * Updated How to use OpenBLAS in Microsoft Visual Studio (markdown) * Updated How to use OpenBLAS in Microsoft Visual Studio (markdown) * Updated Installation Guide (markdown) * Updated Faq (markdown) * Updated How to use OpenBLAS in Microsoft Visual Studio (markdown) * Updated How to use OpenBLAS in Microsoft Visual Studio (markdown) * minor edits * Updated How to use OpenBLAS in Microsoft Visual Studio (markdown) * Updated How to use OpenBLAS in Microsoft Visual Studio (markdown) * Updated How to use OpenBLAS in Microsoft Visual Studio (markdown) * Updated Faq (markdown) * Installation instructions for Windows * Updated Faq (markdown) * G77 conventions no longer needed with GCC 4.7+ * Updated Home (markdown) * Document why issue 168 occurred. * Updated Home (markdown) * Created Publications (markdown) * Updated Home (markdown) * Updated Document (markdown) * Updated Faq (markdown) * Updated Download (markdown) * Updated Publications (markdown) * Updated Faq (markdown) * Updated Document (markdown) * Revert 7580d38ffad37e6613e6304707aaaa681f3d78c2 ... b1bd4ff37d2106bbd5c4730a08dbb789cc44e7d4 * Created Mailing List (markdown) * Updated Mailing List (markdown) * Updated Mailing List (markdown) * Updated Home (markdown) * Updated Document (markdown) * Updated Publications (markdown) * Updated Download (markdown) * Updated Faq (markdown) * Updated Home (markdown) * Updated Faq (markdown) * Updated Home (markdown) * Revert b69f1417cdf8820be046cc27a2b96b42a25bc3a3 ... 90a227c317c3572ced943461ac3a252c40790f44 on Home * Updated Home (markdown) * Updated Publications (markdown) * Updated Faq (markdown) * Updated How to use OpenBLAS in Microsoft Visual Studio (markdown) * We already ensure the stack alignment in Makefile.system for Win32. * Updated Faq (markdown) * Updated Faq (markdown) * Updated Publications (markdown) * Created Donation (markdown) * Updated Home (markdown) * Updated Document (markdown) * Updated Faq (markdown) * Updated Publications (markdown) * Updated Download (markdown) * Updated Mailing List (markdown) * Updated Donation (markdown) * Updated Download (markdown) * Updated Donation (markdown) * Updated Donation (markdown) * Updated Donation (markdown) * Updated Donation (markdown) * Updated Home (markdown) * Updated Faq (markdown) * Updated Download (markdown) * Updated Home (markdown) * Updated Home (markdown) * Add new entry for static linking and pthread. * Fix named anchors (see http://stackoverflow.com/questions/5319754/cross-reference-named-anchor-in-markdown/7335259#7335259) * Created Related packages that use OpenBLAS (markdown) * Updated Related packages that use OpenBLAS (markdown) * Updated Related packages that use OpenBLAS (markdown) * Updated How to use OpenBLAS in Microsoft Visual Studio (markdown) * Updated Document (markdown) * Created To-do List (markdown) * Updated To do List (markdown) * Updated Fixed optimized kernels To do List (markdown) * Fix English idiom * Remove trailing whitespace * Updated Fixed optimized kernels To do List (markdown) * Updated Fixed optimized kernels To do List (markdown) * Updated Fixed optimized kernels To do List (markdown) * Updated Fixed optimized kernels To do List (markdown) * Updated Faq (markdown) * Updated How to use OpenBLAS in Microsoft Visual Studio (markdown) * Updated How to use OpenBLAS in Microsoft Visual Studio (markdown) * Updated Related packages that use OpenBLAS (markdown) * Updated Related packages that use OpenBLAS (markdown) * Created Machine List (markdown) * Updated Document (markdown) * Updated Installation Guide (markdown) * Created User Manual (markdown) * Updated User Manual (markdown) * Updated Document (markdown) * Updated User Manual (markdown) * Updated User Manual (markdown) * Updated User Manual (markdown) * Updated User Manual (markdown) * Updated Related packages that use OpenBLAS (markdown) * Updated Faq (markdown) * Updated Related packages that use OpenBLAS (markdown) * Updated Machine List (markdown) * Updated Related packages that use OpenBLAS (markdown) * Updated Related packages that use OpenBLAS (markdown) * Add a note about building in QEMU * Updated Home (markdown) * Updated Faq (markdown) * update for allocating too many meory error. * Updated Faq (markdown) * Updated Faq (markdown) * Updated Installation Guide (markdown) * Updated Faq (markdown) * Init function doc * Updated Document (markdown) * Updated User Manual (markdown) * Updated User Manual (markdown) * Created How to build OpenBLAS for Android (markdown) * Updated How to build OpenBLAS for Android (markdown) * Updated Home (markdown) * Part of the description is really no clear, I add some more information, so it would be easier for VS user to fix the problems facing them. * Created Developer manual (markdown) * Updated Document (markdown) * Updated How to use OpenBLAS in Microsoft Visual Studio (markdown) * a typo, download ** frome -> download from * Updated Faq (markdown) * English (minor edit) * Updated Developer manual (markdown) * Updated Developer manual (markdown) * Updated Developer manual (markdown) * Updated Machine List (markdown) * Updated Developer manual (markdown) * Updated Developer manual (markdown) * Updated How to build OpenBLAS for Android (markdown) * Updated How to build OpenBLAS for Android (markdown) * Updated How to use OpenBLAS in Microsoft Visual Studio (markdown) * Updated How to use OpenBLAS in Microsoft Visual Studio (markdown) * issue#842 * Updated How to build OpenBLAS for Android (markdown) * Updated How to build OpenBLAS for Android (markdown) * Updated How to build OpenBLAS for Android (markdown) * Updated How to build OpenBLAS for Android (markdown) * Added FC for building with Fortran * Change link for the Intel MKL documentation * Updated User Manual (markdown) * Updated User Manual (markdown) * Added MIPS build instructions from OpenMathLib#949 * use TARGET_CFLAGS and TARGET_LDFLAGS instead of CFLAGS and LDFLAGS for linking OpenBLAS on ARMv7 * Add Windows updates (msys2,mingw/w64 merger), Android/MIPS pointers, qemu hint * Building libs & netlib targets to prevent errors in tests * Recipes not targets (for make) * Making only libs, not netlib (which also contains link/run tests...) * Copied from instructions by Ivan Ushakov, originally posted in OpenMathLib#569 * Updated How to build OpenBLAS for iPhone iOS (markdown) * Updated Faq (markdown) * Created How to build OpenBLAS for iPhone iOS (markdown) * error code (0xc000007b) was missing a character * Updated How to build OpenBLAS for iPhone iOS (ARMv8) (markdown) * Updated How to build OpenBLAS for iPhone iOS (ARMv8) (markdown) * Revert 7e9dd0ebf079e002e3aa831fa671fde3e8cfad81...8d105c7be8cd447482f61e0295c0c146f5314eb5 on How to build OpenBLAS for iPhone iOS * Add guide on how to reversibly supplant Ubuntu LTS libblas.so.3 * typo * Updated How to use OpenBLAS in Microsoft Visual Studio (markdown) * Updated User Manual (markdown) * Updated Faq (markdown) * Updated Download (markdown) * Add perl to pacman package list * Fixed formatting on general questions * Copied from issue OpenMathLib#1136 * Added instructions for building for Windows UWP. * To clear confusions vs super-fat-binaries that dont exist. * Updated How to use OpenBLAS in Microsoft Visual Studio (markdown) * Update for 0.2.20 (full builds, ARMv7 softfp support, newer NDKs using CLANG) * Updated How to build OpenBLAS for Android (markdown) * Fix some formatting issues * Updated How to build OpenBLAS for Android (markdown) * Updated How to use OpenBLAS in Microsoft Visual Studio (markdown) * Updated How to use OpenBLAS in Microsoft Visual Studio (markdown) * Updated How to build OpenBLAS for Android (markdown) * Created Precompiled installation packages (markdown) * Updated Precompiled installation packages (markdown) * Example - debian? * Mention (and link to) distribution-specific packages * Updated Installation Guide (markdown) * OpenSuSE (13.2, SLE included) * Updated Precompiled installation packages (markdown) * Updated Precompiled installation packages (markdown) * Make it look consistent. * Fedora+EPEL // maybe rpmbuild is too heavy * Updated Precompiled installation packages (markdown) * Updated Precompiled installation packages (markdown) * Updated How to use OpenBLAS in Microsoft Visual Studio (markdown) * Updated Precompiled installation packages (markdown) * fix toolchain argument in armv8 clang build as per OpenMathLib#1337 * add note about stdio.h not found error * Add flang instructions * Use the SVG Travis badge * homebrew option for OSX * Promote native MSVC builds with LLVM * Updated How to use OpenBLAS in Microsoft Visual Studio (markdown) * Updated How to use OpenBLAS in Microsoft Visual Studio (markdown) * Updated How to use OpenBLAS in Microsoft Visual Studio (markdown) * Updated How to use OpenBLAS in Microsoft Visual Studio (markdown) * Direct people to the appropriate instructions * Add link to the Goto paper * Add CMAKE_BUILD_TYPE * Add note about having to specify AR on a Mac, from issue 1435 * Mention requirement to build a standalone toolchain in the clang section as well * added 'perl' to conda install command * homebrew/science was deprecated. This tap is now empty as all its formulae were migrated. * Added hint for "expected identifier" error message to mingw section following OpenMathLib#1503 * Revert 9161c3b54281131e892dec739d888f35e6c59cf3...03f879be0c9e6a55705bc7efd5ee193299e04029 on How to use OpenBLAS in Microsoft Visual Studio * Revert to recommending mingw-w64 from sf.net and add note about issue 1503 * Updated How to use OpenBLAS in Microsoft Visual Studio (markdown) * Update MSVC installation procedure with info from OpenMathLib#1521 * Add downgrade option for msys2 mingw compiler issue as suggested by econwang in OpenMathLib#1503 * Add note about static linking bug with NDK 16 and API>22 * Updated Precompiled installation packages (markdown) * Updated Precompiled installation packages (markdown) * Updated Faq (markdown) * OBS is renamed and deep link format changed. Apparently recent SLE includes rpm by default too. * Add links to Conda-Forge and to staticfloat's builds for Julia * Mention _64 suffix appended to Julia builds with INTERFACE64 (issue 1617) * Fix unwanted markdown italicization * Add instruction to change to the generic sgemmkernel implementation from issue 1531 * Added hint about stack size requirements for running lapack-test from PR 1645; fixed markup of section headings * Add link to RvdG's publications page as a non-paywalled source of the "Goto paper" * Add section about non-suitability of the IBM XL compiler on POWER8 * Mention cmake version requirement in view or recent issues with link failures in utest etc. * Replace outdated entry for Sandybridge support with more general section on AVX512, Ryzen and GPU * Mention Apple Accelerate here as iOS build issue tickets usually die as soon as someone points out this option to the questioner. * Add section about unexpectedly using an older pre-installed version of the shared library (issue 1822) * fix markup of new entry * Mention perl and C compiler as prerequisites on the build host * Save WIP page * Updated Notes on parallelism and OpenBLAS (markdown) * Updated Notes on parallelism and OpenBLAS (markdown) * Updated Notes on parallelism and OpenBLAS (markdown) * Updated [WIP] Notes on parallelism and OpenBLAS (markdown) * Updated [WIP] Notes on parallelism and OpenBLAS (markdown) * Updated [WIP] Notes on parallelism and OpenBLAS (markdown) * Destroyed [WIP] Notes on parallelism and OpenBLAS (markdown) * Updated Faq (markdown) * Add small note on AVX512 for CentOS/RHEL section. * document the extension functions * formatting * Updated How to use OpenBLAS in Microsoft Visual Studio (markdown) * Updated Download (markdown) * Add brief general usage information from issue 1925 * Add link to Pete Warden blog article on GEMM rather than just deep-linking to a diagram from it * Document some of the less useful parameters from param.h * Updated Installation Guide (markdown) * Done with OpenMathLib#2089 * Add note about changed library names for update-alternatives on Debian/Ubuntu * Updated Home (markdown) * Add note about using OpenBLAS with CUDA_HPL 2.3 from issue OpenMathLib#909 * Fix typos in previous commit * Add pdb instructions fir cross-builds * Add note about generic QEMU CPUID clashing with existing P2(MMX) * typo * typo * C code syntax highlight * Updated multithreading section to introduce option USE_LOCKING (issue 2164) * Updated How to build OpenBLAS for iPhone iOS (ARMv8) (markdown) * Updated How to build OpenBLAS for iPhone iOS (ARMv8) (markdown) * Clarify Miniconda/cmake install instructions and redact outdated note about msys2 * Document cmake install step * Updated How to build OpenBLAS for Android (markdown) * Add solution for programs that look for libblas.so/liblapack.so * Add entry for powersaving modes on ARM boards (from issue 2540) * Add suggestion for speed problems on big.little systems from issue 2589 * Convert the ARMV8 big.little tidbit to a separate topic and update it with more details from the issue ticket * Add entry about problems caused by using the raw cblas.h (issue 2593) * complete quote symbol around CPATH environment variable * Updated How to use OpenBLAS in Microsoft Visual Studio (markdown) * Updated How to use OpenBLAS in Microsoft Visual Studio (markdown) * Add note about running conda activate when working in a VS window (from issue 2637) * Add note about (not) compiling with -fbounds-check (ticket 2657) * Add entry about compile-time NUM_THREADS setting (issue 2678) * Added some sketchy description of adding cpuids for autodetection, adding targets and architectures * Markup and typo fixes * Add openblas_set_affinity from PR 2547 * Created _Footer (markdown) * Destroyed _Footer (markdown) * Add LAPACK-like SHGEMM to document the "official" status of the SH prefix * fix formatting of latest addition * Move outdated instructions for gcc-based NDK versions to the bottom, add hint about x86 builds * Add help for cpuid recognition failure * Update source tree layout & mention extraneous cpu paramerts * Updated How to use OpenBLAS in Microsoft Visual Studio (markdown) * Explain why pure VS builds are slower, and highlight that they do not support DYNAMIC_ARCH * Updated How to use OpenBLAS in Microsoft Visual Studio (markdown) * Mention fortran requirement and incompatibility of ifort with msvc * preliminary page for understanding the build system, needs a lot more work and input from more knowledgeable people than me * Updated Build system overview (markdown) * Updated WIP Build system overview (community made) (markdown) * add information for HOSTCC, HOST_CFLAGS * Added alternative script which was tested on OSX with latest NDK * added link to targets list * Updated WIP Build system overview (community made) (markdown) * Updated WIP Build system overview (community made) (markdown) * Updated WIP Build system overview (community made) (markdown) * Updated WIP Build system overview (community made) (markdown) * Updated WIP Build system overview (community made) (markdown) * Updated WIP Build system overview (community made) (markdown) * added script for x86_64 architecture * Updated WIP Build system overview (community made) (markdown) * Updated WIP Build system overview (community made) (markdown) * updated link to FLAME publications list * Created How to use OpenBLAS on Cortex-M (markdown) * Updated How to use OpenBLAS on Cortex M (markdown) * Updated Precompiled installation packages (markdown) * Updated How to use OpenBLAS on Cortex M (markdown) * Updated How to use OpenBLAS on Cortex M (markdown) * Updated How to use OpenBLAS on Cortex M (markdown) * Update source layout graph and start a short section on benchmarking to collect various pointers from the issue tracker * Add workaround for building with CMAKE on OSX * Use actual small headings to fix... weird bullet indent shit * Oops * Updated Faq (markdown) * Updated Faq (markdown) * Updated How to generate import library for MingW (markdown) * Updated How to generate import library for MinGW (markdown) * Updated How to generate import library for MinGW (markdown) * Updated How to generate import library for MinGW (markdown) * Updated How to generate import library for MinGW (markdown) * Updated How to generate import library for MinGW (markdown) * Updated How to generate import library for MinGW (markdown) * Updated How to generate import library for MinGW (markdown) * Updated How to generate import library for MinGW (markdown) * explicitly set CMAKE_MT to replace the new cmake default llvm-mt (failing) * Add -Wl,-rpath,/your_path/OpenBLAS/lib option to gcc linker line in "Link shared library" section + explanation for why it is needed/can be omitted. Also make note that -lgfortran not needed if only making LAPACKE calls. * Add note explaining that build flags passed to make should also be passed to make install * give example of install error * Describe how to build openblas library for win/arm64 targets * Add Xen to the existing entry for QEMU/KVM based on issue 3445 * Updated Download (markdown) * Updated Installation Guide (markdown) * Updated Installation Guide (markdown) * Revert b8da0e8523b898a2206d1e2fe99dbfb4ebb0ffa8...bc55aade759d2f925689b000828da249e1fc6a1a on Installation Guide * Revert b0c9a2ee060b8dd0b46b4c58375ef2a743c0363a...cecf8cf67963bd77a0bb97086e3a457a4cee11ff on Download * Revert bc55aade759d2f925689b000828da249e1fc6a1a...134894a0f09a0e92eef1b9a5c9e63f459d2db55e on Installation Guide * Add NDK23B example * Makes iOS build more robust * Double -isysroot * Bump up required devtoolset version for AVX-512 intrinsics. * Updated Installation Guide (markdown) * Updated How to build OpenBLAS for Windows on ARM64 (markdown) * Revert b8da0e8523b898a2206d1e2fe99dbfb4ebb0ffa8...75bba70832f8765faee693931c4a9e3eb6c84d98 on Installation Guide * Revert 75bba70832f8765faee693931c4a9e3eb6c84d98...d171e711a5cd8026b2eb507b249b5e51fa28b2a2 on Installation Guide * restore Windows link after malicious edit * Revert 1bcb03dcef85c675aace7f0a755d5aa36ec46eca...f732906434146b1a1ee82abe944a6d51d8f43b81 on Installation Guide * restore Windows link after malicious edit * Updated Installation Guide (markdown) * Bump up AVX-512 devtoolset because of identified packaging issues * Updated How to use OpenBLAS in Microsoft Visual Studio (markdown) * n-dash html entity instead of - * Updated How to use OpenBLAS in Microsoft Visual Studio (markdown) * Updated How to use OpenBLAS in Microsoft Visual Studio (markdown) * Add the bfloat16 functions * mention AXPBY * Update building for Apple M1 * Updated How to build OpenBLAS for Windows on ARM64 (markdown) * Created How to build OpenBLAS for macOS M1 / arm64 (markdown) * Updated How to use OpenBLAS in Microsoft Visual Studio (markdown) * Updated How to use OpenBLAS in Microsoft Visual Studio (markdown) * Updated How to use OpenBLAS in Microsoft Visual Studio (markdown) * Updated How to use OpenBLAS in Microsoft Visual Studio (markdown) * Updated How to use OpenBLAS in Microsoft Visual Studio (markdown) * Add NO_AVX2 build hint for OSX Docker Desktop/xhyve (issues 2194 and 2244) * Mention the ELF offset/address bug from binutils 2.38 ld * moved issue 665 (sparse matrix/vector support) to a faq entry * Update and simplify based on CI experience and 3741 * Updated Download (markdown) * Updated How to build OpenBLAS for Windows on ARM64 (markdown) * Revert 0dcee87d486028fbd88c603853cdcae810e025c6...bf3d15e74d42b0b01618b4beb7b9d658fb905118 on Download * Revert a02f9e470f8e26eda1b8d8601ad2486557721ccf...c862aeb3492c29b487858d43c93676855b60a1f2 on How to build OpenBLAS for Windows on ARM64 * Updated How to use OpenBLAS in Microsoft Visual Studio (markdown) * Revert 9db97d11d88c801e8c5e9b8d6cc85fb44e5bca61...d2eb48810f3ecc1680900581473005f79c394ca4 on How to use OpenBLAS in Microsoft Visual Studio * start with the smallest configs, Appveyor and Cirrus * Updated CI jobs overview (markdown) * Add Azure CI * Add github workflows * Add the crossbuild parts of the dynamic_arch workflow * remove trailing separator * Add FreeBSD/Cirrus * Add ILP64 jobs on Cirrus * Add C910V and the OSUOSL Jenkins jobs (currently configured for my fork) * Updated Installation Guide (markdown) * Expand section on precompiled windows binaries to mention INTERFACE64=0 option * Remove reference to buildbot (domain reregistered to someone else, issue 4148

martin-frbg mentioned this issue Sep 18, 2017

How to build openblas for iOS? #1305

Closed

martin-frbg mentioned this issue Oct 31, 2017

How to build OpenBLAS for iPhone iOS (ARMV7) #1340

Closed

martin-frbg mentioned this issue Apr 20, 2018

Compile for iOS ARMV8 #1531

Closed

martin-frbg mentioned this issue Jul 18, 2018

How to build openblas for iOS? #1687

Closed

L1onKing mentioned this issue Sep 30, 2019

Build OpenBLAS 0.3.6 for iOS #2275

Closed

martin-frbg mentioned this issue Apr 2, 2020

Make ARMV7 compile with xcode and add a CI job for it #2537

Merged

martin-frbg closed this as completed in #2537 Apr 2, 2020

Compile for iOS #569

Compile for iOS #569

Comments

AlexandrGraschenkov commented May 13, 2015

martin-frbg commented May 13, 2015

xianyi commented May 13, 2015

AlexandrGraschenkov commented May 13, 2015

martin-frbg commented May 13, 2015

tchernitski commented Jun 15, 2015

AlexandrGraschenkov commented Jun 15, 2015

tchernitski commented Jun 16, 2015

sstepashka commented Apr 1, 2016

tchernitski commented Apr 1, 2016

ivan-ushakov commented Feb 26, 2017 • edited Loading

martin-frbg commented Feb 26, 2017

ivan-ushakov commented Feb 28, 2017 • edited Loading

martin-frbg commented Feb 28, 2017

martin-frbg commented Mar 1, 2017

sstepashka commented Mar 1, 2017

ivan-ushakov commented Mar 1, 2017

tamworth commented Mar 5, 2017 • edited Loading

ccy022364 commented Mar 7, 2017

sstepashka commented Mar 7, 2017

ccy022364 commented Mar 7, 2017

tamworth commented Mar 7, 2017

martin-frbg commented Mar 7, 2017

ccy022364 commented Mar 8, 2017

tamworth commented Mar 10, 2017

tamworth commented Mar 11, 2017 • edited Loading

ashwinyes commented Mar 11, 2017

tamworth commented Mar 11, 2017

ashwinyes commented Mar 11, 2017 • edited Loading

ashwinyes commented Mar 11, 2017

tamworth commented Mar 12, 2017

ashwinyes commented Mar 12, 2017

tamworth commented Mar 12, 2017

ashwinyes commented Mar 12, 2017

ashwinyes commented Mar 12, 2017

martin-frbg commented Mar 12, 2017

tamworth commented Mar 13, 2017

tamworth commented Mar 13, 2017 • edited Loading

ashwinyes commented Mar 13, 2017

ashwinyes commented Mar 17, 2017

brightbetter commented Aug 16, 2017

panic-milan commented Nov 16, 2017 • edited Loading

supertrouper commented May 15, 2021

ivan-ushakov commented Feb 26, 2017 •

edited

Loading

ivan-ushakov commented Feb 28, 2017 •

edited

Loading

tamworth commented Mar 5, 2017 •

edited

Loading

tamworth commented Mar 11, 2017 •

edited

Loading

ashwinyes commented Mar 11, 2017 •

edited

Loading

tamworth commented Mar 13, 2017 •

edited

Loading

panic-milan commented Nov 16, 2017 •

edited

Loading