Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add bfloat16 support for numpy arrays <> tflm tensors #2759

Closed
jaeyoo opened this issue Nov 12, 2024 · 0 comments
Closed

Add bfloat16 support for numpy arrays <> tflm tensors #2759

jaeyoo opened this issue Nov 12, 2024 · 0 comments
Assignees
@jaeyoo

Comments

@jaeyoo
Copy link
Member

jaeyoo commented Nov 12, 2024

ml_dtypes is a standard ml data type library used in Jax. ml_dtypes provides blfoat16 as NPY_USERDEF (user defined type in Numpy). Even though we have more dtypes in ml_dtypes (e.g. int2, float8, etc.) we can start supporting bfloat16 first.
@jaeyoo jaeyoo mentioned this issue Nov 12, 2024
Merged
mergify bot pushed a commit that referenced this issue Nov 12, 2024
@jaeyoo
Update numpy_utils.cc to support ml_dtypes.bfloat16 (#2758)
26ada36 
This commit adds NPY_USERDEF for ml_dtypes.bfloat16 only. Other types are not supported yet.

BUG=#2759
BUG=2759
ddavis-2015 added a commit to ddavis-2015/tflite-micro that referenced this issue Nov 25, 2024
@ddavis-2015
Squashed commit of the following:
defad29 
commit 414a55a
Author: ddavis-2015 <[email protected]>
Date:   Thu Nov 21 18:15:40 2024 -0800

    last merge missed xtensa.inc file

commit 1110543
Author: ddavis-2015 <[email protected]>
Date:   Thu Nov 21 18:06:45 2024 -0800

    Squashed commit of the following:

    commit 300751d
    Author: ddavis-2015 <[email protected]>
    Date:   Wed Nov 20 12:03:16 2024 -0800

        Update to latest Cadence code.  Int8 any bitwidth on normal quant axis updated.

    commit 0a49b2a
    Author: ddavis-2015 <[email protected]>
    Date:   Wed Nov 20 10:50:30 2024 -0800

        Add input tensor CRC to Generic Benchmark application.
        Only use -O3 -LNO:simd with the xtensa decompress.cc target.

    commit 83dafce
    Author: ddavis-2015 <[email protected]>
    Date:   Mon Nov 18 17:23:58 2024 -0800

        cleanup

    commit 2f8cead
    Author: ddavis-2015 <[email protected]>
    Date:   Mon Nov 18 13:29:33 2024 -0800

        Revert FakeMicroContext changes for alternate profiler.
        Add default alternate profiler implementation to MicroContext.

    commit fddf003
    Author: ddavis-2015 <[email protected]>
    Date:   Mon Nov 18 12:29:26 2024 -0800

        Fix typo.

    commit ae6a207
    Author: ddavis-2015 <[email protected]>
    Date:   Mon Nov 18 12:29:06 2024 -0800

        Implement alternate profiler for MicroInterpreter.
        Enable use of alternate profiler by decompression code.
        Enable use of alternate profiler by Generic Benchmark application.

    commit 5e1a1c9
    Author: ddavis-2015 <[email protected]>
    Date:   Sun Nov 10 18:24:02 2024 -0800

        changes to make the memory planner debug output easier to interpret

    commit f651c88
    Author: ddavis-2015 <[email protected]>
    Date:   Sun Nov 10 04:27:29 2024 -0800

        single pending ops queue
        process pending ops recursively

    commit cfd9890
    Author: ddavis-2015 <[email protected]>
    Date:   Sun Nov 10 00:26:14 2024 -0800

        expand model_facade
        redo var handle tracking

    commit 7776cda
    Author: ddavis-2015 <[email protected]>
    Date:   Tue Nov 5 13:24:58 2024 -0800

        remove [[maybe_unused]]

    commit 40e7530
    Author: ddavis-2015 <[email protected]>
    Date:   Mon Nov 4 11:17:43 2024 -0800

        fix arena

    commit 0d889e0
    Author: ddavis-2015 <[email protected]>
    Date:   Sat Nov 2 14:18:55 2024 -0700

        Fix MicroProfiler bug with ClearEvents().
        Add pre-inference profiling to the Generic Benchmark.

commit bd04eb2
Author: ddavis-2015 <[email protected]>
Date:   Wed Nov 20 18:40:42 2024 -0800

    Pre-rebase empty commit

commit 661ad5e
Author: Ryan Kuester <[email protected]>
Date:   Thu Nov 14 11:31:23 2024 -0600

    feat(compression): update model compression tools

commit 6eac102
Author: Ryan Kuester <[email protected]>
Date:   Thu Nov 14 00:31:56 2024 -0600

    feat: add model compression draft

    Add a feature-complete draft of the model compression feature in
    a single commit. This commit isn't intended for merging into
    mainline; it exists solely for reviewers and testers to take an
    early look.

commit f91dd91
Author: Ryan Kuester <[email protected]>
Date:   Wed Nov 13 16:46:20 2024 -0600

    build(codegen): suppress noise in console output (tensorflow#2742)

    build(codegen): suppress noise in console output

    Redirect output from the code_generator binary so that when it's
    used within the build system, it doesn't print unexpected,
    distracting noise to the console. Generally, compiler or
    generator commands don't print output unless there's an error.

    This reverts "build(codegen): suppress noise in console output
    (tensorflow#2708)", commit d249577, which attempted to fix the same
    problem by adding a --quiet flag. A subsequent upgrade to the
    Tensorflow Python package caused new noise that wasn't possible
    to suppress with the flag.

    BUG=description

commit 182c8c7
Author: Adrian Lundell <[email protected]>
Date:   Tue Nov 12 23:34:09 2024 +0100

    Update CMSIS-NN Transpose conv function call (tensorflow#2760)

    Also removes OPTIMIZE_FOR_SIZE compiler option as the new implementation is more space efficient.

    BUG=CMSIS-NN update

commit d7c42b9
Author: suleshahid <[email protected]>
Date:   Tue Nov 12 12:40:15 2024 -0800

    Add SQRT to Requantize Flatbuffer. (tensorflow#2756)

    BUG=Add more support for tool.

commit 26ada36
Author: Jae H. Yoo <[email protected]>
Date:   Tue Nov 12 12:14:54 2024 -0800

    Update numpy_utils.cc to support ml_dtypes.bfloat16 (tensorflow#2758)

    This commit adds NPY_USERDEF for ml_dtypes.bfloat16 only. Other types are not supported yet.

    BUG=tensorflow#2759
    BUG=2759

commit 79c3fde
Author: Ryan Kuester <[email protected]>
Date:   Tue Nov 12 13:40:59 2024 -0600

    ci(bazel): improve logging of bazel CI tests (tensorflow#2749)

    Improve the logging of Bazel commands run in CI by setting several
    options that turn off, e.g., progress messages intended for interactive
    shells, and turn on, e.g., more verbose output when something fails.

    BUG=tensorflow#2742

commit 9245002
Author: Ryan OShea <[email protected]>
Date:   Wed Nov 6 20:18:28 2024 +0100

    Fix issue with transposing shape in CMSIS-NN batch matmul (tensorflow#2741)

    BUG=tensorflow#2740

commit 4bb78c7
Author: Ryan Kuester <[email protected]>
Date:   Tue Nov 5 19:11:45 2024 -0600

    fix(create_tflm_tree): remove recent tests from exported tree (tensorflow#2751)

    Remove the recently added span_test.cc and static_vector_test.cc from
    the files exported by the create_tflm_tree.py project generation process
    by adding them to the list of tests in the Makefile. Unit tests are not
    meant to be included in exported trees; they may include files that are
    not exported.

    This change also ensures that these tests are included when `make test`
    is run.

    BUG=fixes tensorflow#2718

commit 45cd79b
Author: Måns Nilsson <[email protected]>
Date:   Tue Nov 5 23:13:20 2024 +0100

    Replace CoreDebug with DCB (tensorflow#2746)

    BUG=The CMSIS CoreDebug macro is deprecated.

commit 694d250
Author: Ryan Kuester <[email protected]>
Date:   Tue Nov 5 15:48:52 2024 -0600

    chore: remove obsolete ci/temp_patches (tensorflow#2744)

    chore: remove obsolete ci/temp_patches

    Remove ci/temp_patches, which was obsoleted in 23f608f once it
    was no longer used by the sync script. It should have been
    deleted then.

    Remove it not only to clean up dead code, but because it contains
    a reference to `micro_copts`, which is about to be refactored
    away, and we don't want to leave stray references to it in the
    tree.

    BUG=tensorflow#2636

commit 8eb6b23
Author: Ryan Kuester <[email protected]>
Date:   Fri Nov 1 17:10:05 2024 -0500

    build(bazel): add integrity check to nnlib_hifi4 download (tensorflow#2743)

    build(bazel): add integrity check to nnlib_hifi4 download

    Add an integrity check to the http_archive() download of nnlib_hifi4 in
    order to make the build more hermetic, reduce the security risk that a
    remote file changes, and silence the noisy warning on the console during
    the build:

        DEBUG: Rule 'nnlib_hifi4' indicated that a canonical reproducible
        form can be obtained by modifying arguments integrity[....]

    BUG=description

commit 5a46ab0
Author: Måns Nilsson <[email protected]>
Date:   Fri Nov 1 20:57:00 2024 +0100

    Add kernels optimized for size flag to FC and SVDF (tensorflow#2734)

    The kernels optimized for size flag provides an alternative implementation where size is prioritized over latency.

    For size option (speed option is default) it means the CMSIS-NN kernels are calculating kernel sums during inference.

    BUG=no bug but this will let users prioritize speed vs size even more

commit 389e775
Author: suleshahid <[email protected]>
Date:   Tue Oct 29 13:57:29 2024 -0700

    Fix upstream sync (tensorflow#2728)

    * .

    * nl add

    * Sync from upstream TF.

    * fix include

    ---------

    Co-authored-by: TFLM-bot <[email protected]>

commit e440f0a
Author: Måns Nilsson <[email protected]>
Date:   Mon Oct 28 20:16:43 2024 +0100

    Update CMSIS third party download (tensorflow#2716)

    CMSIS is updated from CMSIS_5 to CMSIS_6.

    BUG=It is recommended to upgrade from CMSIS_5 to CMSIS_6

commit 1b975d9
Author: suleshahid <[email protected]>
Date:   Mon Oct 28 11:27:23 2024 -0700

    Update deprecated mergify rules (tensorflow#2732)

    * .

    * nl add

    * update mergify

commit 78e5467
Author: Ryan Kuester <[email protected]>
Date:   Fri Oct 25 18:27:21 2024 -0500

    fix(python): add compatibility with NumPy 2 (tensorflow#2733)

    Upgrade the pinned packages used in the Bazel build environment to make
    the Python extension module compatible with NumPy 2 in addition to NumPy
    1. Modules built against NumPy 1 are not compatible with runtime
    environments that use NumPy 2, but modules built against NumPy 2 should
    be compatible with both.

    The python_requirements.txt defining the Bazel build environment is was
    upgraded via the command:

        bazel run //third_party:python_requirements.update -- --upgrade

    See third_party/python_requirements.in for details.

    This change upgraded a number of other packages as well, notably
    Tensorflow, from 2.17 to 2.18. In fact, Tensorflow 2.18 is also required
    for compatibility with NumPy 2.

    Also update the path to the header files within the NumPy wheel, used by
    the :numpy_cc_deps target. Our method of including header files shipped
    in Python packages is sensitive to the internal details of those
    packages, and NumPy moved their headers during this major version
    change.

    This fixes tensorflow#2731, in which //python/tflite_micro:whl_test began failing.
    :whl_test installs the newly built tflite_micro package, which packages
    our Python extension module, and its dependencies into a clean virtual
    environment and performs tests. The tflite_micro package has unversioned
    dependencies on the tensorflow and numpy packages. Within the last 24
    hours, tensorflow in PyPI was upgraded from 2.17 to 2.18. The 2.18
    release upgraded tensorflow's versioned dependency on numpy from 1 to 2,
    forcing the environment created by :whl_test to use NumPy 2 instead of
    NumPy 1; thereby exposing the incompatibility of the extension module
    that was built against NumPy 1, and causing :whl_test to fail.

    BUG=tensorflow#2731

commit e86d97b
Author: RJ Ascani <[email protected]>
Date:   Mon Oct 7 10:36:26 2024 -0700

    Replace rascani with suleshahid on OWNERS (tensorflow#2715)

    BUG=none

commit e3f6dc1
Author: David Davis <[email protected]>
Date:   Thu Oct 3 10:45:00 2024 -0700

    Compression documentation (tensorflow#2711)

    @tensorflow/micro

    Add documentation describing some compression/decompression internals and makefile build procedures.

    bug=tensorflow#2710

commit b3967a9
Author: Ryan Kuester <[email protected]>
Date:   Wed Oct 2 13:36:01 2024 -0500

    style: add .style.yapf to control yapf styling of Python code (tensorflow#2709)

    Add a .style.yapf file so yapf can be used to style Python code without
    passing the project's style via command line option. Remove the
    corresponding patch to pigweed's call to yapf, used by CI, and instead
    let it too rely on .style.yapf. Remove the developer documentation's
    instruction to use the command line option.

    BUG=description

commit d249577
Author: Ryan Kuester <[email protected]>
Date:   Tue Oct 1 16:16:45 2024 -0500

    build(codegen): suppress noise in console output (tensorflow#2708)

    Add a --quiet option to the code_generator binary so that when it's used
    within the build system, it doesn't print unexpected, distracting noise
    to the console. Generally, compiler or generator commands don't print
    output unless there's an error.

    BUG=description
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jaeyoo jaeyoo

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ddavis-2015 @jaeyoo