Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support Swarm Services #224

Open
ehazlett opened this issue Jul 10, 2017 · 23 comments
Open

Support Swarm Services #224

ehazlett opened this issue Jul 10, 2017 · 23 comments

Comments

@ehazlett
Copy link
Owner

This will add support for Swarm services.
@jcmcote
Copy link

jcmcote commented Jul 13, 2017

I'm running Interlock 1.4 with docker 17.05.0-ce and nginx 1.11.10. Both Interlock and nginx are running on the swarm manager.

I configure my services using docker compose file version 3 and deploy them with docker stack deploy.

I successfully configured Interlock so that any container started on the manager node gets added to the nginx configuration and I'm able to access the service via the nginx gateway.

However if the container starts on one of the worker nodes Interlock does not detect it and does not create any routes for it.

Is this a know issue?

Note that when I run
docker -H tcp://192.168.1.100:2375 events
I only see events from manager node not the worker nodes. Expected behaviour?
thanks!

@byrnedo
Copy link

byrnedo commented Jul 14, 2017

Any idea when this support would be added?

This was referenced Jul 18, 2017
Closed
Closed
@jdelamar jdelamar mentioned this issue Aug 7, 2017
Closed
@jcmcote
Copy link

jcmcote commented Aug 16, 2017

I have looked at the code for the server.go file

In that file there is a poller function called runPoller. This function uses the docker SDK ContainerList function. However this function only returns containers on the current docker daemon not the containers in the cluster of swarm nodes.

containers, err := s.client.ContainerList(context.Background(), opts)

So I do not understand how Interlock can add routes for services which are deployed on other nodes. Is this functionality not supported ? Do the containers exposed via nginx (the load balancer) need to run on the same host where Interlock is running?

Using Java docker SDK I was able to list the swarm services, find the tasks for these services. Could Interlock use the tasks label instead of using the labels on docker containers like is done in the runPoller function? Would such a patch be welcomed?

Thanks

@ehazlett
Copy link
Owner Author

Thanks for the interest!

Yes, you could use the tasks however the user would need to create a service using the --container-label option instead of service labels --label. If you wanted to use the service labels, you would need to get the service info and then use that.

Another side note is that if the service is on it's own network you would need to make sure that the proxy container (nginx, etc) gets attached to the service network otherwise it won't be able to get to the backend.

@jcmcote
Copy link

jcmcote commented Aug 16, 2017 via email

@byrnedo
Copy link

byrnedo commented Aug 16, 2017 •
edited
Loading

Just to chime in: not 100% sure what you mean about the polling but AFAIK the managers events are now swarm-wide so one can watch those to discover new services/tasks.

@ehazlett
Copy link
Owner Author

I can either use the service labels or container labels. However I think I
will first try with service labels, if present dig down to the task to find
their specific IP, then dig down to the containers to find the ports they
expose via DockerFile.

Be careful as this will have a lot of API calls which will impact service at large scale. FYI, /services and /tasks returns a summary that has most of the info needed and is a lot cheaper.

@jcmcote
Copy link

jcmcote commented Aug 16, 2017 via email

@jcmcote
Copy link

jcmcote commented Aug 16, 2017 via email

@byrnedo
Copy link

byrnedo commented Aug 16, 2017 via email

@jcmcote
Copy link

jcmcote commented Aug 16, 2017 via email

@byrnedo
Copy link

byrnedo commented Aug 16, 2017 via email

@byrnedo
Copy link

byrnedo commented Aug 16, 2017 via email

@jcmcote
Copy link

jcmcote commented Aug 16, 2017 via email

@byrnedo
Copy link

byrnedo commented Aug 17, 2017

I think a possibly better way might be to do a dns lookup on the tasks.<service-dns-alias>, that should give the actual underlying ips to each task. This makes more sense when using with Prometheus for instance where you want to be able to poll each individual container.

@byrnedo
Copy link

byrnedo commented Aug 17, 2017 •
edited
Loading

I suppose you could just get that from the tasks endpoint but the dns lookup will probably be less demanding on the docker daemon and possibly a lot faster.

@jcmcote
Copy link

jcmcote commented Aug 23, 2017

Turns out I do have the IP address in the tasks. What I don't have are the port. If I declare my service like this
version: '3'

networks:
mynet:

services:

app:
image: ehazlett/docker-demo:latest
deploy:
replicas: 3
networks:
- mynet
ports:
- 8080
labels:
- "interlock.hostname=test"
- "interlock.domain=local"
- "interlock.context_root=/test"
- "interlock.port=8080"
- "interlock.network=ga_mynet"

Then I do see mappings in the service but those maybe only apply to the VIP. But I would actually rather not even specify the ports mapping and only rely on the interlock.port label. This works fine because the container does listen to port 8080 on the overlay network at the various container IP. Since I know the IP from the task and the port from the service label I'll be able to generate this nginx configuration

upstream ctxtest.local__test {
zone ctxtest.local__test_backend 64k;
server 10.0.0.3:8080;
server 10.0.0.8:8080;
server 10.0.0.10:8080;

}

I think its the best I can do, since I can't seem to be able to get at the container exposed port (the one EXPOSED via Dockerfile in the image)

@jcmcote
Copy link

jcmcote commented Aug 27, 2017 via email

@jcmcote
Copy link

jcmcote commented Sep 1, 2017

I have a working proof of concept which uses docker swarm services/tasks to configure a load balancer like nginx/haproxy. I will submit a pull request shortly.

However I've notice there's another pull request made by @ehazlett #186 which seems to also be related to creating routes based on information from swarm services and tasks. I'm a little confuse. Is that pull request still valid. It seems old..

Can you shed some light?
Thanks
Jean-Claude

@ehazlett
Copy link
Owner Author

ehazlett commented Sep 1, 2017

Yes there has been quite a bit of work towards Swarm support in that branch but you are correct that it has stalled. I have been trying to work on it but there are a few other things going. Feel free to open a PR -- I would rather review early than have you write a bunch and not get merged :)

@jcmcote
Copy link

jcmcote commented Sep 1, 2017 via email

@ehazlett
Copy link
Owner Author

ehazlett commented Sep 1, 2017

Hey it might be easier to discuss via chat as there are quite a few changes in that branch. Do you have a preferred chat and would you be up for a quick sync? I'm on the Docker Community Slack if you are there (ehazlett)

@jcmcote
Copy link

jcmcote commented Oct 18, 2017

I notice my implementation had an issue. It cannot detect when containers are stopped on other nodes. This is currently a limitation of the docker events. From the node you are listening from say the manager node you only receive container events from that manager not the other worker nodes.

To work around this issue I'm using the poller and doing a diff of the task states which I have access to from the manager node. This way I know when a container on a worker node is down.

This all works well. I'm now testing with two separate stacks deployed to the same swarm. Ideally I'd like to be able to deploy multiple stacks representing various staging branches from our build host so these stacks should work independently.

My question to you is: Would it be a good idea to use the stack membership to detect task changes and to generate the nginx configuration.

I know in a previous comment you said to make sure container IPs are only added to the nginx configuration if the given container and nginx are part of the same network.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@ehazlett @byrnedo @jcmcote