My fork might be of interest

[hamstah]

Hi,

Sent you an email a few weeks ago as I started working on my fork, but not sure you received it.
I've heavily changed your original code for my needs, and I thought it might be worth discussing how we could work together to make more people benefit from them.

My fork is at https://github.com/hamstah/clouddream

Some of the changes I introduced

• There is a web based manager that allows the uploading of new images by url/file upload
• You can browse the input/output of the file from there
• You can process any of the input images with parameters specified by the user (maxwidth, n_inter, n_octave etc), so you can have multiple outputs from the same image with different parameters
• Processing is ordered and done by workers, you can see the queue and jobs in progress (you can have multiple workers in parallel). I had issues with the original project where some image would fail and it would get stuck in a loop trying to process them.
• Use of docker compose to make it easier to spin the containers and scale the number of workers

It's obviously work in progress and I'm no docker expert, but I think it provides a good foundation to extend.

I'm thinking of moving the project out of the fork and into its own repo, as it's almost a complete rewrite.

Would be great to hear your thoughts.

I have a version of it running on a digital ocean droplet, I can give you access if you want to check it out.

[itsTeknas]

I tried your fork on google cloud,
I got some errors... pls look into it
Sorry had to do this here, your repository doesn't have a Issues tab.

root@dreamer:/home/blackcurrant_apps/clouddream# docker-compose up
Recreating clouddream_redis_1...
Recreating clouddream_manager_1...
Recreating clouddream_compute_1...
Recreating clouddream_nginx_1...
Attaching to clouddream_redis_1, clouddream_manager_1, clouddream_compute_1, clouddream_nginx_1
redis_1 | .
redis_1 | .-__ ''-._ redis_1 | _.- .. ''-._ Redis 3.0.3 (00000000/0) 64 bit
redis_1 | .-.-``.``\/ _.,_ ''-._ redis_1 | ( ' , .- | , ) Running in standalone mode redis_1 | |-.-...- __...-.-._|'.-'| Port: 6379
redis_1 | |-._ ._ / .-' | PID: 1
redis_1 |-._ -.-./ _.-' _.-' redis_1 | |-.-._-.**.-' .-'.-'|
redis_1 | | -._-. .-'.-' | http://redis.io
redis_1 | -._-.-.**.-'_.-' _.-' redis_1 | |-.-._-.**.-' .-'.-'|
redis_1 | | -._-._ .-'.-' |
redis_1 | -._-.-.**.-'_.-' _.-' redis_1 | -. -.__.-' _.-' redis_1 |-._ _.-'
redis_1 | `-.__.-'
redis_1 |
redis_1 | 1:M 09 Aug 16:40:11.947 # Server started, Redis version 3.0.3
redis_1 | 1:M 09 Aug 16:40:11.969 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
redis_1 | 1:M 09 Aug 16:40:11.969 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
redis_1 | 1:M 09 Aug 16:40:11.969 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
redis_1 | 1:M 09 Aug 16:40:11.970 * DB loaded from append only file: 0.001 seconds
redis_1 | 1:M 09 Aug 16:40:11.970 * The server is now ready to accept connections on port 6379
compute_1 | 16:40:15 RQ worker u'rq:worker:4d7af1871d2d.11' started, version 0.5.4
compute_1 | 16:40:15
compute_1 | 16:40:15 *** Listening on compute...
manager_1 | [2015-08-09 16:40:15 +0000] [1] [INFO] Starting gunicorn 19.3.0
manager_1 | [2015-08-09 16:40:15 +0000] [1] [INFO] Listening at: http://0.0.0.0:8000 (1)
manager_1 | [2015-08-09 16:40:15 +0000] [1] [INFO] Using worker: sync
manager_1 | [2015-08-09 16:40:15 +0000] [14] [INFO] Booting worker with pid: 14
manager_1 | [2015-08-09 16:40:18 +0000] [14] [ERROR] Exception in worker process:
manager_1 | Traceback (most recent call last):
manager_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 507, in spawn_worker
manager_1 | worker.init_process()
manager_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 118, in init_process
manager_1 | self.wsgi = self.app.wsgi()
manager_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/base.py", line 67, in wsgi
manager_1 | self.callable = self.load()
manager_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 65, in load
manager_1 | return self.load_wsgiapp()
manager_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
manager_1 | return util.import_app(self.app_uri)
manager_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/util.py", line 355, in import_app
manager_1 | import(module)
manager_1 | File "/opt/manager/manager/init.py", line 1, in
manager_1 | from app import app
manager_1 | File "/opt/manager/manager/app.py", line 18, in
manager_1 | from models import (
manager_1 | File "/opt/manager/manager/models.py", line 7, in
manager_1 | from utils import create_thumbnail
manager_1 | ImportError: No module named utils
manager_1 | Traceback (most recent call last):
manager_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 507, in spawn_worker
manager_1 | worker.init_process()
manager_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 118, in init_process
manager_1 | self.wsgi = self.app.wsgi()
manager_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/base.py", line 67, in wsgi
manager_1 | self.callable = self.load()
manager_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 65, in load
Attaching to clouddream_redis_1, clouddream_manager_1, clouddream_compute_1, clouddream_nginx_1
manager_1 | return self.load_wsgiapp()
manager_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
manager_1 | return util.import_app(self.app_uri)
manager_1 | File "/usr/local/lib/python2.7/dist-packages/gunicorn/util.py", line 355, in import_app
manager_1 | import(module)
manager_1 | File "/opt/manager/manager/init.py", line 1, in
manager_1 | from app import app
manager_1 | File "/opt/manager/manager/app.py", line 18, in
manager_1 | from models import (
manager_1 | File "/opt/manager/manager/models.py", line 7, in
manager_1 | from utils import create_thumbnail
manager_1 | ImportError: No module named utils
manager_1 | [2015-08-09 16:40:18 +0000] [14] [INFO] Worker exiting (pid: 14)
manager_1 | [2015-08-09 16:40:18 +0000] [1] [INFO] Shutting down: Master
manager_1 | [2015-08-09 16:40:18 +0000] [1] [INFO] Reason: Worker failed to boot.
clouddream_manager_1 exited with code 3
Gracefully stopping... (press Ctrl+C again to force)
Stopping clouddream_nginx_1...
Stopping clouddream_compute_1...
Stopping clouddream_redis_1...
Exception in thread Thread-1 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
File "/code/build/docker-compose/out00-PYZ.pyz/threading", line 810, in __bootstrap_inner
File "/code/build/docker-compose/out00-PYZ.pyz/threading", line 763, in run
File "/code/build/docker-compose/out00-PYZ.pyz/compose.cli.multiplexer", line 41, in _enqueue_output
File "/code/build/docker-compose/out00-PYZ.pyz/compose.cli.log_printer", line 62, in _make_log_generator
File "/code/build/docker-compose/out00-PYZ.pyz/compose.container", line 155, in wait
File "/code/build/docker-compose/out00-PYZ.pyz/docker.utils.decorators", line 20, in wrapped
File "/code/build/docker-compose/out00-PYZ.pyz/docker.client", line 826, in wait
File "/code/build/docker-compose/out00-PYZ.pyz/docker.clientbase", line 83, in _post
File "/code/build/docker-compose/out00-PYZ.pyz/requests.sessions", line 508, in post
File "/code/build/docker-compose/out00-PYZ.pyz/requests.sessions", line 465, in request
File "/code/build/docker-compose/out00-PYZ.pyz/requests.sessions", line 573, in send
File "/code/build/docker-compose/out00-PYZ.pyz/requests.adapters", line 414, in send
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'error'
root@dreamer:/home/blackcurrant_apps/clouddream#

[hamstah]

Hey

Thanks for trying it, I forgot to add a file, which I've now added.
Also added issues to the repo.

[itsTeknas]

@hamstah Great ! will try again !