Creating Raw Histograms From Digital Camera Raw Files

  1. Contents

Click Here, the Mystery of the Canon DSLR Combing Artifacts in Raw Histograms is Solved!

The Raw Histogram Creator Program

I wrote a quick-and-dirty program to create raw histograms from Canon (and others?) digital camera raw files. It is a command line (console) Windows application - no installation is required. The source code is available here if you are interested.

Usage:
rawhistogram.exe [-?][-help] file
	-?, -help	Gives a help message
	file		PGM input file created with dcraw.

The input file must be a raw file processed into 16 bit raw PGM by using dcraw. For example if you have a Canon raw file called IMG_0005.CR2 first use dcraw:

dcraw -D -4 -t 0 -o 0 -v IMG_0005.CR2

this will create IMG_0005.pgm. Note that the options are critical to get the right (16 bit raw) conversion - you can omit "-o 0 -v" if you want but the rest are required. Now run:

rawhistogram IMG_0005.pgm

The result will be a (text) file called IMG_0005.csv that contains the raw histogram data. This file can easily be opened in a spreadsheet program (e.g. Excel) to plot the RGGB histograms.

To try the raw histogram creator (v.0.93 updated to handle 14 bit cameras and create statistics) on your own raw files you can download this file:
findcombedraws.zip
it contains rawhistorgam.exe, dcraw.exe and a batch file - findcombedraws.bat - that helps in processing many raw files automatically and searching for combed ones. Please refer to the included README.TXT to find out how to use the batch file.

The version of dcraw I used is compiled from this version of the source and Little cms version 1.15.

Raw Histograms From Canon 30D and Other DSLR Cameras

Using the above program I created raw histograms of some photographs shot with a Canon 30D DSLR. Below you can find the small/fine in camera JPEG files, the CSV files created by rawhistogram.exe and plots made from them. The interesting thing to note are the combing artifacts in the histograms of some of the 30D and 5D photos which should not be there if the data was "pure unmodified raw" as it came from the A/D converters of the camera. For a discussion of the phenomenon see this thread in the DPReview forums. Also I would like to thank the people at the Open Photography Forums who helped a lot in trying to figure out what causes this behaviour. There are some theories as to what is causing the combing in the conclusion.

In the list below the combed photos are marked with a bold "Lens" field.

Canon EOS 30D Photos

Click here for a deteiled test of the 30D.

On the 30D at ISO 100 the pixels saturate before the A/D converters; the saturation level is 3398 raw counts out of the possible 4096. See IMG_1829 for an example, see also the Canon EOS 1D Mark III below. Another interesting fact is that the intermediate 1/3 stop ISO settings of the camera (i.e. ISO 125, 160, 250, 320, 500, 640, 1000 and 1250) are implemented in the firmware by setting the sensor amplifiers for the closest full stop and (mathematically) scaling the raw data up or down. Therefore all 1/3 stop ISOs are combed and this is not a mystery.
Photo Lens Shooting conditions Raw histogram plot Raw Histogram file
IMG_0005 EF-S18-55mm f/3.5-5.6 ISO 1600, EC 0, 1/200 f/4.5, flash, FEC 0 IMG_0005.png IMG_0005.csv
IMG_0008 EF-S18-55mm f/3.5-5.6 ISO 100, EC 0, 1/13 f/22 IMG_0008.png IMG_0008.csv
IMG_0010 EF-S18-55mm f/3.5-5.6 ISO 1600, EC 0, 1/200 f/29 IMG_0010.png IMG_0010.csv
IMG_0011 EF-S18-55mm f/3.5-5.6 ISO 100, EC 0, 1/13 f/29 IMG_0011.png IMG_0011.csv
IMG_0541 EF100mm f/2 USM ISO 100, EC 0, 1/400 f/2 IMG_0541.png IMG_0541.csv
IMG_0843 EF100mm f/2 USM ISO 100, EC +1/3, 1/500 f/2 IMG_0843.png IMG_0843.csv
IMG_0903 EF50mm f/1.8 ISO 800, EC +1/3, 1/250 f/1.8, flash, FEC 0 IMG_0903.png IMG_0903.csv
IMG_0947 EF100mm f/2 USM ISO 800, EC +2, 1/40 f/2 IMG_0947.png IMG_0947.csv
IMG_0963 EF50mm f/1.8 ISO 800, EC +2/3, 1/125 f/1.8 IMG_0963.png IMG_0963.csv
IMG_1753 EF50mm f/1.8 ISO 100, EC 0, 1/80 f/22, 2006-07-22 15:35:14 IMG_1753.png, IMG_1753_zoom.png IMG_1753.csv
IMG_1754 EF50mm f/1.8 ISO 125, EC 0, 1/125 f/22, 2006-07-22 15:35:20 IMG_1754.png, IMG_1754_zoom.png IMG_1754.csv
IMG_1755 EF50mm f/1.8 ISO 160, EC 0, 1/125 f/22, 2006-07-22 15:35:24 IMG_1755.png, IMG_1755_zoom.png IMG_1755.csv
IMG_1829 EF50mm f/1.8 ISO 100, EC +1/3, 1/30 f/8, 2006-07-23 14:33:38 IMG_1829.png IMG_1829.csv
IMG_1830 EF50mm f/1.8 ISO 100, EC +1/3, 1/640 f/1.8, 2006-07-23 14:34:26 IMG_1830.png IMG_1830.csv
Canon EOS 20D Photos
Photo Lens Shooting conditions Raw histogram plot Raw Histogram file Source of the photo
IMG_0119 16.0 - 35.0mm ISO 100, EC 0, 1/125 f/5.6 IMG_0119.png IMG_0119.csv 20D review at DPReview??
IMG_9505 24.0 - 70.0mm ISO 100, EC 0, 1/400 f/7.1 IMG_9505.png IMG_9505.csv
Canon EOS 300D (EOS DIGITAL REBEL) Photos
Photo Lens Shooting conditions Raw histogram plot Raw Histogram file
CRW_7394 24.0 - 70.0mm ISO 100, EC 0, 1/317.9 f/8 CRW_7394.png CRW_7394.csv
Canon EOS D60 Photos
Photo Lens Shooting conditions Raw histogram plot Raw Histogram file Source of the photo
CRW_3057 28.0 - 75.0mm ISO 100, EC 0, 1/201.7 f/5 CRW_3057.png CRW_3057.csv http://www.aim-dtp.net/aim/digicam/dcraw/
Canon EOS 350D (EOS DIGITAL REBEL XT) Photos
On the 350D the black point is calibrated at raw level 256 unlike the 300D, D60, 20D, 30D and 5D where it is at 128.
Photo Lens Shooting conditions Raw histogram plot Raw Histogram file Source of the photo
XTI_6755 18.0 - 55.0mm ISO 100, 1/25 f/8, flash, FEC 0 XTI_6755.png XTI_6755.csv A very good friend
XTI_6784 18.0 - 55.0mm ISO 100, 1/8 f/16 XTI_6784.png XTI_6784.csv A very good friend
XTI_6787 18.0 - 55.0mm ISO 100, 1/8 f/16 XTI_6787.png XTI_6787.csv A very good friend
XTI_6833 18.0 - 55.0mm ISO 100, 1/8 f/36 XTI_6833.png XTI_6833.csv A very good friend
Canon EOS 5D Photos
Photo Lens Shooting conditions Raw histogram plot Raw Histogram f/ile Source of the photo
5DI_0187 EF50mm f/1.4 USM ISO 1600, EC 0, 1/100 f/1.4 5DI_0187.png 5DI_0187.csv Mike Robbins
5DI_1285 50.0mm ISO 1600, EC 0, 1/20 f/1.8 5DI_1285.png 5DI_1285.csv Mike Robbins
5DI_2259 EF50mm f/1.4 USM ISO 1600, EC 0, 1/30 f/1.8 5DI_2259.png 5DI_2259.csv Mike Robbins
5DI_2272 EF50mm f/1.4 USM ISO 1600, EC 0, 1/30 f/1.8 5DI_2272.png 5DI_2272.csv Mike Robbins
5DI_2278 EF50mm f/1.4 USM ISO 1600, EC 0, 1/30 f/1.8 5DI_2278.png 5DI_2278.csv Mike Robbins
5DI_2303 EF50mm f/1.4 USM ISO 1600, EC 0, 1/30 f/1.8 5DI_2303.png 5DI_2303.csv Mike Robbins
Canon EOS 1D Mark III Photos

Click here for a deteiled test of the Mark III.

This camera has 14bit analog to digital converters - notice the scale on the histograms! The raw data has a potential range from 0 to 16383 compared to the 0-4095 range on all previous DSLRs with 12bit A/D converters. In addition the black point appears to be calibrated at 1024, while on the 300D, D60, 20D, 30D and 5D it is at 128 and on the 350D it is at 256. Like some previous models, at ISO 100 the pixels saturate before the A/D converters; the saturation level is 15280 counts out of the possible 16383 (see also IMG_1829 from the 30D above).
Photo Lens Shooting conditions Raw histogram plot Raw Histogram file Source of the photo
Y0B0B0327 50mm ISO 100, 1/200 f/5.6 Y0B0B0327.png Y0B0B0327.csv Imaging Resource
Y0B0B0430 50mm ISO 100, 1/200 f/6.4 Y0B0B0430.png Y0B0B0430.csv Imaging Resource
Y0B0B0494 50mm ISO 100, 1/200 f/6.4 Y0B0B0494.png Y0B0B0494.csv Imaging Resource

Note: You can see the full EXIF information from some of the JPEG files.
Note: There are two green lines in the histograms - G1 and G2 - this is because there are two green filters in each 2x2 set of pixels in the bayer matrix. The two lines overlap almost perfectly most of the time of course.
Note: The photographs were not chosen for their artistic merit :-)
Note: Only the 30D photos are mine the rest are collected from either people I know or the Web.

Results From Detailed Analysis of a Big Collection of Canon 30D Raw Files

The mystery of the combing artifacts is solved. This section is kept only for "historic" reference"

To try to understand what is causing the combing I analyzed my whole collection of raw files. Here are the results.

LensPhotos totalCombed photos%CombedNon Comed ISOsCombed ISOs
EF50mm f/1.841912830.55%100, 200, 400, 800, 1600100, 200, 400, 800, 1600
EF100mm f/2 USM45212728.10%100, 200, 400, 800, 1600100, 200, 400, 800
EF24-85mm f/3.5-4.5 USM3487110.32%100, 200, 400, 800, 1600500
EF-S18-55mm f/3.5-5.63960.00%100, 200, 400, 800, 1600
Sigma 170-500mm1100.00%100, 200, 400, 800
Tokina 12-24mm70.00%200, 400, 800
Tamron 200-500mm460.00%100, 200
No lens (pinhole body cap or LensBaby)430.00%
Total45411383.04%

The combed files from the EF24-85mm f/3.5-4.5 USM are all at ISO 500 and that is always combed on the 30D, so they are not a mystery.

LensEF50mm f/1.8EF100mm f/2 USMEF24-85mm f/3.5-4.5 USMEF-S18-55mm f/3.5-5.6Sigma 170-500mmTokina 12-24mmTamron 200-500mm
f/TotalCombed%TotalCombed%TotalCombed%TotalCombed%TotalCombed%TotalCombed%TotalCombed%
Total41912830.55%45212728.10%3487110.32%396110746
1.812811589.84%
2601321.67%16312777.91%
2.82247
3.21
3.52430610.33%1
4118219610.51%13
4.5264720.31%5
571131
5.6183169154631
6.32106999
7.1881
837156791723320
92143171
101326722
113453671.31%88913
13122921
1448621
164367419
2226121
292
328

Conclusion Regarding the Combing Artifacts

The mystery of the combing artifacts is solved. This section is kept only for "historic" reference"

From the above and some results reported by other people - like the combed 5D examples above - the following conclusions can be drawn:

To sum up, the combing does not appear to be random but the contributing factors are not entirely clear at this point.