Parse Error: Library does not support fixed length elements with length >= 2147483648

[stl-steve-moore]

Describe the bug

The problem is with DICOM files where the pixel data is not compressed and the length field for element (7FE0,0010) is >= 2147483648. That field length is supposed to be a 32 bit unsigned number, but it would appear as though the library is treating it as a signed value. The DCM4CHE library throws an exception when trying to parse such an image.

To Reproduce

The attached zip has some scripts and XML files to demonstrate a very close problem that is in the underlying DCM4CHE software. The inflate.sh script creates an image where the pixel length is 1GB; that works. It then creates one where the pixel length is 2GB - 1. That works (hmmm, ignore the problem with odd number of bytes; the software does not blow up). When the script tries to create an image where the pixel length is 2GB, the DCM4CHE software throws a fault.

If you run this script, it will leave large files in your /tmp folder.

Expected behavior
Library should treat the length value in the DICOM metadata as an unsigned 32 bit number.
Understood that the JVM needs enough memory to support this.

Screenshots
None

Desktop (please complete the following information):

• MacOS, but others as well

Smartphone (please complete the following information):
NA

Additional context
We use DCM4CHE2 in our system, but the problem also exists in DCM4CHE5.

Because the pixel data is not compressed, the length field for the pixel data gives the absolute length from beginning to end. If the pixel data had been compressed, that length field would have been 0xffffffff and there would have been multiple compressed fragments within that DICOM element (7FE0,0010). Not sure what XNAT would have done with that, but that is not the issue reported in this ticket.

No PHI in the scripts or data files in the attached zip. Original DICOM file used to generate the XML came from TCIA.

[stl-steve-moore]

2G-problem.zip

[gunterze]

Pixeldata >= 2GiB can't be allocated in memory. But parsing it with

File f = ...;
DicomInputStream in = new DicomInputStream(f);
in.setIncludeBulkData(IncludeBulkData.URI);
Attributes data = in.readDataset();

will work since dcm4che/dcm4che#1266 .
E.g.:

$ storescp -b11112
17:46:38.482 INFO  - Start TCP Listener on 0.0.0.0/0.0.0.0:11112

$ storescu -cSTORESCP@localhost:11112 /tmp/2G.dcm 
Native OpenCV library has been found in /home/gunter/dcm4che/lib/linux-x86-64/libopencv_java.so
Scanning files to send
.
Scanned 1 files in 0.026s (=26ms/file)
17:47:06.328 INFO  - Initiate connection from 0.0.0.0/0.0.0.0:0 to localhost:11112
17:47:06.335 INFO  - Established connection Socket[addr=localhost/127.0.0.1,port=11112,localport=43013]
...
17:47:06.352 INFO  - STORESCU->STORESCP(1) << 1:C-STORE-RQ[pcid=5, prior=0
  cuid=1.2.840.10008.5.1.4.1.1.2 - CT Image Storage
  iuid=1.3.6.1.4.1.9328.50.4.763 - ?
  tsuid=1.2.840.10008.1.2 - Implicit VR Little Endian]
17:47:06.352 DEBUG - STORESCU->STORESCP(1) << 1:C-STORE-RQ Command:
(0000,0002) UI [1.2.840.10008.5.1.4.1.1.2] AffectedSOPClassUID
(0000,0100) US [1] CommandField
(0000,0110) US [1] MessageID
(0000,0700) US [0] Priority
(0000,0800) US [0] CommandDataSetType
(0000,1000) UI [1.3.6.1.4.1.9328.50.4.763] AffectedSOPInstanceUID

17:47:06.368 DEBUG - STORESCU->STORESCP(1) << 1:C-STORE-RQ Dataset sending...
17:47:08.282 DEBUG - STORESCU->STORESCP(1) << 1:C-STORE-RQ Dataset sent
17:47:08.721 INFO  - STORESCU->STORESCP(1) >> 1:C-STORE-RSP[pcid=5, status=0H
  cuid=1.2.840.10008.5.1.4.1.1.2 - CT Image Storage
  iuid=1.3.6.1.4.1.9328.50.4.763 - ?
  tsuid=1.2.840.10008.1.2 - Implicit VR Little Endian]
17:47:08.721 DEBUG - STORESCU->STORESCP(1) >> 1:C-STORE-RSP Command:
(0000,0002) UI [1.2.840.10008.5.1.4.1.1.2] AffectedSOPClassUID
(0000,0100) US [32769] CommandField
(0000,0120) US [1] MessageIDBeingRespondedTo
(0000,0800) US [257] CommandDataSetType
(0000,0900) US [0] Status
(0000,1000) UI [1.3.6.1.4.1.9328.50.4.763] AffectedSOPInstanceUID

.17:47:08.722 INFO  - STORESCU->STORESCP(1) << A-RELEASE-RQ
17:47:08.722 DEBUG - STORESCU->STORESCP(1): enter state: Sta7 - Awaiting A-RELEASE-RP PDU
17:47:08.722 INFO  - STORESCU->STORESCP(1) >> A-RELEASE-RP
17:47:08.722 INFO  - STORESCU->STORESCP(1): close Socket[addr=localhost/127.0.0.1,port=11112,localport=43013]
17:47:08.722 DEBUG - STORESCU->STORESCP(1): enter state: Sta1 - Idle
Sent 1 objects (=2,048.002MB) in 2.371s (=863.771MB/s)

17:47:06.335 INFO  - Accept connection Socket[addr=/127.0.0.1,port=43013,localport=11112]
17:47:06.343 DEBUG - /127.0.0.1:11112<-/127.0.0.1:43013(1): enter state: Sta2 - Transport connection open
17:47:06.348 INFO  - STORESCP<-STORESCU(1) >> A-ASSOCIATE-RQ
...
17:47:06.385 INFO  - STORESCP<-STORESCU(1) >> 1:C-STORE-RQ[pcid=5, prior=0
  cuid=1.2.840.10008.5.1.4.1.1.2 - CT Image Storage
  iuid=1.3.6.1.4.1.9328.50.4.763 - ?
  tsuid=1.2.840.10008.1.2 - Implicit VR Little Endian]
17:47:06.385 DEBUG - STORESCP<-STORESCU(1) >> 1:C-STORE-RQ Command:
(0000,0002) UI [1.2.840.10008.5.1.4.1.1.2] AffectedSOPClassUID
(0000,0100) US [1] CommandField
(0000,0110) US [1] MessageID
(0000,0700) US [0] Priority
(0000,0800) US [0] CommandDataSetType
(0000,1000) UI [1.3.6.1.4.1.9328.50.4.763] AffectedSOPInstanceUID

17:47:06.402 DEBUG - STORESCP<-STORESCU(1) >> 1:C-STORE-RQ Dataset receiving...
17:47:06.413 INFO  - STORESCP<-STORESCU(1): M-WRITE ./1.3.6.1.4.1.9328.50.4.7635849699475061559238.part
17:47:08.719 INFO  - STORESCP<-STORESCU(1): M-RENAME ./1.3.6.1.4.1.9328.50.4.7635849699475061559238.part to ./1.3.6.1.4.1.9328.50.4.763
17:47:08.720 INFO  - STORESCP<-STORESCU(1) << 1:C-STORE-RSP[pcid=5, status=0H
  cuid=1.2.840.10008.5.1.4.1.1.2 - CT Image Storage
  iuid=1.3.6.1.4.1.9328.50.4.763 - ?
  tsuid=1.2.840.10008.1.2 - Implicit VR Little Endian]
17:47:08.720 DEBUG - STORESCP<-STORESCU(1) << 1:C-STORE-RSP Command:
(0000,0002) UI [1.2.840.10008.5.1.4.1.1.2] AffectedSOPClassUID
(0000,0100) US [32769] CommandField
(0000,0120) US [1] MessageIDBeingRespondedTo
(0000,0800) US [257] CommandDataSetType
(0000,0900) US [0] Status
(0000,1000) UI [1.3.6.1.4.1.9328.50.4.763] AffectedSOPInstanceUID

$ dcmdump 1.3.6.1.4.1.9328.50.4.763 | tail
1656: (0028,1051) DS #8 [400\1500] WindowWidth
1672: (0028,1052) DS #6 [-1024] RescaleIntercept
1686: (0028,1053) DS #2 [1] RescaleSlope
1696: (0028,1055) LO #16 [WINDOW1\WINDOW2] WindowCenterWidthExplanation
1720: (0032,1060) LO #16 [*CT Colonography] RequestedProcedureDescription
1744: (0040,A124) UI #0 [] UID
1752: (0088,0140) UI #0 [] StorageMediaFileSetUID
1760: (3006,0024) UI #0 [] ReferencedFrameOfReferenceUID
1768: (3006,00C2) UI #0 [] RelatedFrameOfReferenceUID
1776: (7FE0,0010) OW #2147483648 [0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\