taimen android 8.1 deodexing issues

[Furniel]

Tested taimen-opm1.171019.011-factory-2df1c1cb deodexing and got two errors during framework deodexing:

[DEBUG] [11132] 2017/12/20 20:22:23 (vdexExtractor.c:206 main) Processing 'services.vdex'
[DEBUG] [11132] 2017/12/20 20:22:23 (vdex.c:64 vdex_isVersionValid) Vdex version '010' detected
------ Vdex Header Info ------
magic header & version      : vdex-010
number of dex files         : 1 (1)
dex size (overall)          : 85b388 (8762248)
verifier dependencies size  : 120f8 (73976)
verifier dependencies offset: 85b3a4 (8762276)
quickening info size        : a72 (2674)
quickening info offset      : 86d49c (8836252)
dex files info              :
  [0] location checksum : 7c35823a (2083881530)
---- EOF Vdex Header Info ----
[DEBUG] [11132] 2017/12/20 20:22:23 (vdex.c:110 vdex_GetNextDexFileData) Processing first Dex file at offset:0x1c
[DEBUG] [11132] 2017/12/20 20:22:23 (dex.c:327 dex_isValidDexMagic) Dex version '037' detected
[ERROR] [11132] 2017/12/20 20:22:23 (vdex_backend_v10.c:451 vdex_process_v10) Unexpected checksum (89f9fc35 vs fa7bb981) - failed to unquicken Dex file
[DEBUG] [11132] 2017/12/20 20:22:23 (vdex.c:217 vdex_process) Took 78 ms to process Vdex file
[ERROR] [11132] 2017/12/20 20:22:23 (vdexExtractor.c:262 main) Failed to process Dex files - skipping 'services.vdex'
[INFO] 0 out of 1 Vdex files have been processed
[INFO] 0 Dex files have been extracted in total
[INFO] Extracted Dex files are available in '.'

and

[INFO] Processing 1 file(s) from wifi-service.vdex
[DEBUG] [2284] 2017/12/20 20:28:09 (vdexExtractor.c:206 main) Processing 'wifi-service.vdex'
[DEBUG] [2284] 2017/12/20 20:28:09 (vdex.c:64 vdex_isVersionValid) Vdex version '010' detected
------ Vdex Header Info ------
magic header & version      : vdex-010
number of dex files         : 1 (1)
dex size (overall)          : 1cffac (1900460)
verifier dependencies size  : 34f0 (13552)
verifier dependencies offset: 1cffc8 (1900488)
quickening info size        : 74 (116)
quickening info offset      : 1d34b8 (1914040)
dex files info              :
  [0] location checksum : a8dc636b (2833015659)
---- EOF Vdex Header Info ----
[DEBUG] [2284] 2017/12/20 20:28:09 (vdex.c:110 vdex_GetNextDexFileData) Processing first Dex file at offset:0x1c
[DEBUG] [2284] 2017/12/20 20:28:09 (dex.c:327 dex_isValidDexMagic) Dex version '037' detected
[ERROR] [2284] 2017/12/20 20:28:09 (vdex_backend_v10.c:451 vdex_process_v10) Unexpected checksum (c684b9f3 vs f02f3db5) - failed to unquicken Dex file
[DEBUG] [2284] 2017/12/20 20:28:09 (vdex.c:217 vdex_process) Took 31 ms to process Vdex file
[ERROR] [2284] 2017/12/20 20:28:09 (vdexExtractor.c:262 main) Failed to process Dex files - skipping 'wifi-service.vdex'
[INFO] 0 out of 1 Vdex files have been processed
[INFO] 0 Dex files have been extracted in total
[INFO] Extracted Dex files are available in '.'

files.zip

[anestisb]

Thanks for reporting. Will have a look at them tomorrow.

[Furniel]

Tested taimen apps deodexing, everything ok except some google apps(vdexExtractor fails with same type of error):

CalculatorGooglePrebuilt.vdex
Chrome.vdex
GoogleCamera.vdex
Maps.vdex
PrebuiltDeskClockGoogle.vdex
PrebuiltGmail.vdex
YouTube.vdex
services.vdex
wifi-service.vdex
GoogleContacts.vdex
GoogleDialer.vdex
Phonesky.vdex
SettingsGoogle.vdex

log.txt
vdexFiles

[anestisb]

Thanks. The more affected files the easier to spot the pattern. Will start digging later today.

[anestisb]

Oh boy that was painful. I've concluded that the problem is in the way the mentioned bytecode files have been compiled. More specifically, for some reason the compiler backend that generated the oat / vdex files, is shuffling the class list.

If you run the following you will see that the class list is not alphabetically ordered by class descriptor as expected.

vdexExtractor -i ~/Desktop/vdex/GoogleDialer.vdex -f --no-unquicken --dis | grep "class #" | cut -d "'" -f2

Furthermore if you compare the previous list against the original dex file it has been generated from, you can further confirm the shuffling problem.

dexdump GoogleDialer/classes.dex | grep "Class descriptor  :" | cut -d "'" -f2

Finally, to even further verify that this is indeed the problem, you can run the libart oatdump utility against the problematic oat files to also notice the out-of-order issue. It should be also noted that the unquickening logic currently in AOSP master is also affected by the problematic cases.

So of course when the vdex extractor tool decompiles the out-of-ordered version, the checksum of the generated file will not match the original (ordered) one. However, all methods are properly decompiled since the QuickeningInfo offsets are correct for the out-of-order version.

The ART runtime has no problem dealing with the out-of-order version since the resources have been pre-verified and inserted into the loader's class list.

I think the issue is also related to the non-deterministic behaviour of the ART compiler that has been recently reported by the @thestinger while working the reproducible builds feature for the CopperheadOS.

So to wrap-up, there is nothing I can do from this tool's perspective to work with these shuffled files. Putting the classes back to the correct order will require a massive change of backward fix-ups in the extracted dex files. Something which I'm not planning to do.

[anestisb]

Btw I plan to raise this to the Google tracker since I don't believe it's an intended compiler behaviour.

[Furniel]

Thanks for your research. I tested android 8.1 roms deodexing for all other devices(angler, bullhead, marlin, ryu, sailfish, walleye) and they are all affected(in all cases, the list of files is the same - interesting why only this files are affected). Maybe this will help with your report to Google.

[ingbrzy]

Same errors for MIUI on Android 8.1...
services.jar and wifi-service.jar failed to deodex from vdex files.. they failed even with oatdump from API 27... What the hell is Google doing to the dev community...

btw what is services.jar.prof file for?

framework-miui.zip

[ale8530]

@ingbrzy I also saw your zip, changed Google files again :(.

@anestisb what do you think about it?

[anestisb]

As mentioned in #3 (comment) I can't really do anything. The reordering doesn't appear to be deterministic or based on something we can calculate and revert. But even if it was possible, the backwards repair process if I modify the dexClassDef list requires significant code additions.

I've examined a few samples from the list you've provided and it appears that the files can be properly decompiled (method implementations compared to original DEX), although the class list has a different order thus the checksum doesn't verify the output file.

A possible hack-around would be to iterate with the disassembler the decompiled file and if no errors occur (all offsets & index restrictions are ok) force a CRC repair and mark the file as valid. Or export an additional command-line arg to ignore CRC errors and force a repair. Then its user's responsibility to verify the sanity of the output file.

In general I would expect that if a quickened file is decompiled successfully it should be functionally equivalent to the original DEX file (ART runtime is running the same process). However, the only way to prove this assumption is via the original CRC. If we hack around it, that guarantee is over.

All suggestions are welcome.

PS: I'm temporarily re-opening the issue due to this recent discussion.

[anestisb]

Decided to implement a CRC error ignore flag so users will not have to comment code to keep out-of-order decompiled files. For most cases it is expected that the decompiled Dex can be normally processed from decompilers and disassemblers (JEB2, dexdump, dex2jar, etc.).

[xuuhaoo]

@anestisb Is there anything update? and how to ignore CRC check?

[anestisb]

@xuuhaoo you can ignore CRC errors with "--ignore-crc-error" flag