Images aren't automatically being processed

[midori-ao]

I followed your directions for Digital Ocean, Ubuntu with Docker installed. I scp'ed the image to my server but noticed that nothing was displaying.. so I manually ran process_images.sh and got this error:

./process_images.sh: 6: cd: can't cd to /opt/deepdream
./process_images.sh: 10: cd: can't cd to /opt/deepdream/inputs
/root/clouddream/deepdream/inputs
File is input.jpg
Deepdream input.jpg
cp: cannot stat ‘inputs/input.jpg’: No such file or directory
pwd is /root/clouddream/deepdream/inputs
python: can't open file 'deepdream_single.py': [Errno 2] No such file or directory
cp: cannot stat ‘output.jpg’: No such file or directory
./process_images.sh: 10: cd: can't cd to /opt/deepdream/inputs
/root/clouddream/deepdream
File is 6031848_1547091962_98large.jpg
Deepdream 6031848_1547091962_98large.jpg
pwd is /root/clouddream/deepdream
Traceback (most recent call last):
  File "deepdream_single.py", line 3, in <module>
    import numpy as np
ImportError: No module named numpy
cp: cannot stat ‘output.jpg’: No such file or directory

Is there something I'm doing wrong here? Sorry, I'm new with Linux, Python and such.

[quantombone]

You should run ./stop.sh to stop the docker containers.
Then run ./start.sh to start the three main processes.
The command docker ps will show you which Docker containers are running, and there should be three of them.

Finally, the command process_images.sh is meant to be running inside the Docker container. If you're new to Docker, don't worry. The start.sh script should do all of this for you, but feel free to poke inside.

[midori-ao]

I just did ./stop.sh, did ./start.sh, saw that 3 docker containers are running, and scp'd another image into the server, but I still don't see any images in /output/. Is this related to the file paths, or not having the dependency numpy?

[radames]

yes, same issue here, however I can see python process using almost 100% of the CPU, I guess we don't see outputs because it still slowly process through CPU?

[radames]

so the script process_files.sh is trying to access input images from cd /opt/deepdream/inputs but technically this mus be inside the docks, how to make sure when we add the images to the input folder inside clouddream it goes to this virtual folder into the docker? I'm very new to the docker system

[quantombone]

If you want to see what's happening, there is a log file written to log.html. This will contain the outputs of the python program:
http://deepdream.vision.ai/log.html

In addition you can see which image is currently being processed by visiting
http://deepdream.vision.ai/input.jpg

And yes, process_files.sh must be ran inside of the Docker container. If you look at start.sh you'll see a line which looks like this:

docker run --name deepdream-compute -v `pwd`/deepdream:/opt/deepdream -d visionai/clouddream /bin/bash -c "cd /opt/deepdream && ./process_images.sh 2>&1 > log.html"

This will start a new Docker container from the visionai/clouddream image. It will take the /deepdream folder from the host machine and mount it inside the container as /opt/deepdream. And finally it will CD into this directory and call process_image.sh.

I also made some minor changes, so it's a good idea to do a git pull

[radames]

Makes sense, the log.html is been update, I'm getting the

/opt/deepdream File is Photo Deepdream Photo Error Code is 137 File is on Deepdream on

error code 137, do you know what it can be?

[quantombone]

It seems something is wrong with the python script. This is usually a problem with the image. Try making sure the file is called something like "PhotoDeepdream.jpg" and not "Photo Deepdream" with spaces. If that works, then it might be an issue with spaces in the filenames.

[midori-ao]

I did git pull origin master, ran start.sh, checked docker ps, ran enter.sh, did cat log.html and got this output:

Just created outputs/./6031848_1547091962_98large.jpg
Deepdream ./6031848_1547091962_98large.jpg
Error Code is 137

[quantombone]

What happens if you use the file command? When I run

file image-00001.jpg

on one of my images I get:

image-00001.jpg: JPEG image data, JFIF standard 1.02, comment: "Lavc55.70.100"

[radames]

in my case
$ file f.jpg f.jpg: JPEG image data, JFIF standard 1.01

[quantombone]

What happens if you do:

cd ~/clouddream
./stop.sh
cp ~/problemimage.jpg ~/clouddream/deepdream/input.jpg
./enter.sh
#now you are inside a Docker container which has all the Caffe/Python dependencies
cd /opt/deepdream
python deepdream.py

You will not get a more thorough log of what might be happening incorrectly. In my example, the input image is bad and I get the following at the end of my log:

Traceback (most recent call last):
  File "deepdream.py", line 101, in <module>
    img = PIL.Image.open('input.jpg')
  File "/usr/lib/python2.7/dist-packages/PIL/Image.py", line 2028, in open
    raise IOError("cannot identify image file")
IOError: cannot identify image file

If I do this with a successful image, It starts to process the image and shows output like:

0 0 conv2/3x3 (178, 146, 3)
0 1 conv2/3x3 (178, 146, 3)
0 2 conv2/3x3 (178, 146, 3)

[radames]

I was playing inside the Docker container and tried the script in python, seems like the image is right but then I've got this error
I0704 19:18:36.805817 64 net.cpp:248] Memory required for data: 545512320
and makes sense I'm using the 512mb RAM droplet

so I don't think it's generating the output.jpg and then the Error Code is 137
I'll try to increase the RAM

[quantombone]

Well I'm glad this was resolved as a memory issue. I'm playing with a 4GB droplet and it works.

I'll run some tests to see what the minimum drop size is, but for now 512MB is not enough.

[radames]

Yes, it's processing now, so it needs more than the tiny 512MB of the basic droplets!!
thanks for that! and happy to start learning the Docker system!

[quantombone]

I just added a script called ./youtube.sh which uses youtube-dl to get a youtube movie and ffmpeg to get the frames. It shows how simple it is to use somebody else's prebuilt Docker images without having to install/configure the old-fashioned way.

[radames]

Amazing!!

[midori-ao]

I increased my RAM to 1GB and I was able to finally get 1 image output! Now looking into how I can get bigger resolution outputs. edit just found it on the README. Thanks for the feedback guys!

[radames]

Just FYI 1GB it's also been to little if you increase the maxwidth to 600px for example

[radames]

Need 2GB of RAM to process an output image of 1000px width

[midori-ao]

Yeah.. I just ran into that problem unfortunately.

[gpfransen]

Hi all,

When trying to use any models using rectified linear unit (relu) I receive the following error;

KeyError: u'conv2/relu_3x3_reduce'
root@0e1e62948dd7:/opt/deepdream# Logging before InitGoogleLogging() is written to STDERR
bash: syntax error near unexpected token `('

Last part of log;
I0707 16:47:46.279047 30 net.cpp:247] Network initialization done.
I0707 16:47:46.279052 30 net.cpp:248] Memory required for data: 545512320
I0707 16:47:46.407258 30 upgrade_proto.cpp:626] Successfully upgraded file specified using deprecated V1LayerParameter
/usr/lib/python2.7/dist-packages/scipy/ndimage/interpolation.py:532: UserWarning: From scipy 0.13.0, the output shape of zoom() is calculated with round() instead of int() - for these inputs the size of the returned array has changed.
"the returned array has changed.", UserWarning)
Traceback (most recent call last):
File "deepdream.py", line 114, in
frame = deepdream(net, frame, end=json_data['layer'])
File "deepdream.py", line 84, in deepdream
make_step(net, end=end, clip=clip, **step_params)
File "deepdream.py", line 48, in make_step
dst = net.blobs[end]

Now did I change the picture size up to 50x50 pixels over different degradations and the memory message looks to be static?

Now is my main question has anybody got relu working on CPU?
I am currently running it on E5-4600 v3 with 56 GiB ram.
(yes I know, CUDA but don't have that available atm so have to do with resources available.)

Thanks!
Guillaume

[quantombone]

Can you paste your settings.json file?

[gpfransen]

Sure,

I have tested this with;

{
"maxwidth" : 5000,
"layer" : "conv2/relu_3x3_reduce"
}

and

{
"maxwidth" : 5000,
"layer" : "conv1/relu_7x7"
}

When I change it to a non relu value it works perfectly :)

[quantombone]

Ok thanks. I took a deeper look and it seems that the relu layers aren't valid, just like you mentioned.

(Pdb) net.blobs.keys()
['data', 'conv1/7x7_s2', 'pool1/3x3_s2', 'pool1/norm1', 'conv2/3x3_reduce', 'conv2/3x3', 'conv2/norm2', 'pool2/3x3_s2', 'pool2/3x3_s2_pool2/3x3_s2_0_split_0', 'pool2/3x3\
_s2_pool2/3x3_s2_0_split_1', 'pool2/3x3_s2_pool2/3x3_s2_0_split_2', 'pool2/3x3_s2_pool2/3x3_s2_0_split_3', 'inception_3a/1x1', 'inception_3a/3x3_reduce', 'inception_3a/3\
x3', 'inception_3a/5x5_reduce', 'inception_3a/5x5', 'inception_3a/pool', 'inception_3a/pool_proj', 'inception_3a/output', 'inception_3a/output_inception_3a/output_0_spli\
t_0', 'inception_3a/output_inception_3a/output_0_split_1', 'inception_3a/output_inception_3a/output_0_split_2', 'inception_3a/output_inception_3a/output_0_split_3', 'inc\
eption_3b/1x1', 'inception_3b/3x3_reduce', 'inception_3b/3x3', 'inception_3b/5x5_reduce', 'inception_3b/5x5', 'inception_3b/pool', 'inception_3b/pool_proj', 'inception_3\
b/output', 'pool3/3x3_s2', 'pool3/3x3_s2_pool3/3x3_s2_0_split_0', 'pool3/3x3_s2_pool3/3x3_s2_0_split_1', 'pool3/3x3_s2_pool3/3x3_s2_0_split_2', 'pool3/3x3_s2_pool3/3x3_s\
2_0_split_3', 'inception_4a/1x1', 'inception_4a/3x3_reduce', 'inception_4a/3x3', 'inception_4a/5x5_reduce', 'inception_4a/5x5', 'inception_4a/pool', 'inception_4a/pool_p\
roj', 'inception_4a/output', 'inception_4a/output_inception_4a/output_0_split_0', 'inception_4a/output_inception_4a/output_0_split_1', 'inception_4a/output_inception_4a/\
output_0_split_2', 'inception_4a/output_inception_4a/output_0_split_3', 'inception_4b/1x1', 'inception_4b/3x3_reduce', 'inception_4b/3x3', 'inception_4b/5x5_reduce', 'in\
ception_4b/5x5', 'inception_4b/pool', 'inception_4b/pool_proj', 'inception_4b/output', 'inception_4b/output_inception_4b/output_0_split_0', 'inception_4b/output_inceptio\
n_4b/output_0_split_1', 'inception_4b/output_inception_4b/output_0_split_2', 'inception_4b/output_inception_4b/output_0_split_3', 'inception_4c/1x1', 'inception_4c/3x3_r\
educe', 'inception_4c/3x3', 'inception_4c/5x5_reduce', 'inception_4c/5x5', 'inception_4c/pool', 'inception_4c/pool_proj', 'inception_4c/output', 'inception_4c/output_inc\
eption_4c/output_0_split_0', 'inception_4c/output_inception_4c/output_0_split_1', 'inception_4c/output_inception_4c/output_0_split_2', 'inception_4c/output_inception_4c/\
output_0_split_3', 'inception_4d/1x1', 'inception_4d/3x3_reduce', 'inception_4d/3x3', 'inception_4d/5x5_reduce', 'inception_4d/5x5', 'inception_4d/pool', 'inception_4d/p\
ool_proj', 'inception_4d/output', 'inception_4d/output_inception_4d/output_0_split_0', 'inception_4d/output_inception_4d/output_0_split_1', 'inception_4d/output_inceptio\
n_4d/output_0_split_2', 'inception_4d/output_inception_4d/output_0_split_3', 'inception_4e/1x1', 'inception_4e/3x3_reduce', 'inception_4e/3x3', 'inception_4e/5x5_reduce'\
, 'inception_4e/5x5', 'inception_4e/pool', 'inception_4e/pool_proj', 'inception_4e/output', 'pool4/3x3_s2', 'pool4/3x3_s2_pool4/3x3_s2_0_split_0', 'pool4/3x3_s2_pool4/3x\
3_s2_0_split_1', 'pool4/3x3_s2_pool4/3x3_s2_0_split_2', 'pool4/3x3_s2_pool4/3x3_s2_0_split_3', 'inception_5a/1x1', 'inception_5a/3x3_reduce', 'inception_5a/3x3', 'incept\
ion_5a/5x5_reduce', 'inception_5a/5x5', 'inception_5a/pool', 'inception_5a/pool_proj', 'inception_5a/output', 'inception_5a/output_inception_5a/output_0_split_0', 'incep\
tion_5a/output_inception_5a/output_0_split_1', 'inception_5a/output_inception_5a/output_0_split_2', 'inception_5a/output_inception_5a/output_0_split_3', 'inception_5b/1x\
1', 'inception_5b/3x3_reduce', 'inception_5b/3x3', 'inception_5b/5x5_reduce', 'inception_5b/5x5', 'inception_5b/pool', 'inception_5b/pool_proj', 'inception_5b/output', '\
pool5/7x7_s1', 'loss3/classifier', 'prob']

In addition the dropout layers are also not valid. I will update the README to list only the valid layers. Thanks for catching this!

[gpfransen]

Thank you for putting together an easy docker! :)

I bet you have more experience with neural networking than I do, any idea what can cause this?

It looks like RELU layers are supported on CPU; http://caffe.berkeleyvision.org/tutorial/layers.html
Just expect it to be very slow :)

[quantombone]

The ReLU layer is there, but the Caffe layers description for ReLU states:
It also supports in-place computation, meaning that the bottom and the top blob could be the same to preserve memory consumption.

Additionally if you peek inside the tmp.prototxt file for the GoogleNet model, you'll see:

layer {
  name: "conv1/relu_7x7"
  type: "ReLU"
  bottom: "conv1/7x7_s2"
  top: "conv1/7x7_s2"
}

Which confirms that the bottom and top layer are the same. This just makes the network more memory efficient. I suspect it could be possible to re-train the entire network, but I doubt you'll get a lot of value from doing this.

[gpfransen]

Learning every minute! :)

Browsing the tmp now and makes sense :)

Thanks a million!

[quantombone]

You're welcome. I updated the README so it now only lists the layers which have actual data. I still haven't tested all of the layers listed, but at least dropout and relu are gone from the list.