Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve the docker-compose environment #461

Merged
merged 12 commits into from
Oct 29, 2020
Merged

Improve the docker-compose environment #461

merged 12 commits into from
Oct 29, 2020

Conversation

ankon
Copy link
Contributor

@ankon ankon commented Oct 20, 2020

Bunch of fixes and improvements I needed to make the docker-compose environment work smoothly

Note-worthy:

  • 42487c1 removes the odd IP address that was hard-coded. I couldn't find a reference to this anywhere else, but I may have missed something here.
  • I think it would be good to tie the confluentinc/* images to a specific version of the Confluent Platform, to avoid unintentional breakages such as the one fixed by a15f93b

ankon added 12 commits October 20, 2020 11:48
@ankon
Configure the SELinux labeling for all volumes
4e9bb80 
This is required to run the docker-compose environments on a SELinux-enabled environment
such as Fedora 32.

Note that ':z' (shared) is only used for the app directory itself. while ':Z' (private) is used
for all other volume mounts.
@ankon
Remove the obsolete webpack container
0ea8949
@ankon
Remove the reference to the non-existing 'connects-plugins' directory
6045111 
This aligns this compose file with the main docker-compose.yml, and brings the Kafka connect
container closer to actually starting up.
@ankon
Remove references to the IP 127.11.8.17
42487c1 
There is nothing here or anywhere that ensures this IP actually exists, and at least
with using podman on Fedora this breaks accessing any of the containers.
@ankon
Mark the kafkacat container as depends_on the kafka container
ea991b7
@ankon
Explicitly disable the new zookeeper HTTP "Admin server"
a15f93b 
This is new in zookeeper 3.5 (which is used by the newer confluentinc images), and the default
port 8080 conflicts with akhq, preventing that from starting up.
@ankon
Explicitly and consistently quote the environment in the docker-com…
43f0876 
…pose YAML files

YAML is sometimes misleading, and especially in environment variables this can be problematic: These
MUST be strings, and numbers such as `0` would actually get interpreted as empty strings.

By using single quotes everywhere this should be less likely to happen accidentally when modifying
these files ... as long as one sticks to the "keep the existing style rule" :)
@ankon
Assign a broker id to the Kafka brokers
1410912 
This should make stopping and starting a docker-compose environment
possible without losing all data in it.
@ankon
Fix the configuration for the connect server
0109bea
@ankon
Fix the path to the kafka data to match the non-dev server configuration
a99b842
@ankon
Provide some akhq configuration for the development version
29d4ebf
@ankon
Configure the authorizer on the Kafka cluster to prevent errors when …
d9f37b8 
…using the `ACLs` tab
@tchiotludo
Copy link
Owner

The special 127.11.8.17 is a hack on linux that allow to listen on same port with multiple ip (mostly multiple docker in my case).
Just add in /etc/hosts an alias, and don't bother with with available port on your machine.
Since I don't use anymore the docker-compose this don't mind to remove this

for my knowledge: zookeeper-data:/var/lib/zookeeper/data:Z
What is the Z you add everywhere ?

@ankon
Copy link
Contributor Author

ankon commented Oct 22, 2020

Since I don't use anymore the docker-compose this don't mind to remove this

Hmm. In that case I would actually propose to drop that completely. For development I suspect it should be easy enough to just run the two parts separately in your IDE (JVM backend + webpack dev server), and just providing a basic docker-compose setup for zookeeper + kafka should then be enough.

Just add in /etc/hosts an alias, and don't bother with with available port on your machine.

Fair enough, could do that and add it to the documentation. I mainly don't like spending time working out why the documentation for development setup won't work when I come back to a project after a while -- I want to get to doing stuff! :)

for my knowledge: zookeeper-data:/var/lib/zookeeper/data:Z
What is the Z you add everywhere ?

This is telling docker how it should handle SELinux labels on the directories. 'z' means "allow this directory be shared by multiple containers if needed", while 'Z' means "only allow this one container to access it". If SELinux isn't enabled then these flags won't do anything.

From docker-run(1):

Labeling systems like SELinux require that proper labels are placed on volume content mounted into a container. Without a label, the security system might prevent the processes running inside the container from using the content. By default, Docker does not change the labels set by the OS.

To change the label in the container context, you can add either of two suffixes :z or :Z to the volume mount. These suffixes tell Docker to relabel file objects on the shared volumes. The z option tells Docker that two containers share the volume content. As a result, Docker labels the content with a shared content label. Shared volume labels allow all containers to read/write content. The Z option tells Docker to label the content with a private unshared label. Only the current container can use a private volume.

@tchiotludo tchiotludo merged commit 6ba38f2 into tchiotludo:dev Oct 29, 2020
@tchiotludo
Copy link
Owner

Thanks for all the detailed clarification, I don't have such knowledge on Docker (SELinux is a word I that I don't take time to look at).
I also think an backend server and a npm run start is fine but people may prefer docker-compose so I merge it for this guys.

@Grishnackh
Copy link
Contributor

@tchiotludo Could you please point out what you do to get AKHQ locally working for development purposes? When I start the project locally (per gradle task application/run or as IntelliJ IDEA build configuration of type Application on class org.akhq.App, I get the following output:

Task :App.main()
2021-02-08 21:26:59,766 INFO main i.m.runtime.Micronaut Startup completed in 2506ms. Server Running: http://localhost:8080
2021-02-08 21:27:48,743 INFO r-thread-1 org.akhq.log.access [Date: 2021-02-08T21:27:48.3583598+01:00] [Duration: 416 ms] [Url: GET /] [Status: 307] [Ip: /127.0.0.1] [User: -]
2021-02-08 21:27:48,788 INFO pGroup-1-3 org.akhq.log.access [Date: 2021-02-08T21:27:48.7863468+01:00] [Duration: 1 ms] [Url: GET /ui] [Status: 200] [Ip: /127.0.0.1] [User: -]
2021-02-08 21:27:48,893 INFO pGroup-1-3 org.akhq.log.access [Date: 2021-02-08T21:27:48.8899247+01:00] [Duration: 3 ms] [Url: GET /ui/ui/static/css/2.7caccc14.chunk.css] [Status: 200] [Ip: /127.0.0.1] [User: -]
2021-02-08 21:27:48,914 INFO pGroup-1-5 org.akhq.log.access [Date: 2021-02-08T21:27:48.9143521+01:00] [Duration: 0 ms] [Url: GET /ui/ui/static/js/2.a6ffe02f.chunk.js] [Status: 200] [Ip: /127.0.0.1] [User: -]
2021-02-08 21:27:48,919 INFO pGroup-1-4 org.akhq.log.access [Date: 2021-02-08T21:27:48.9192378+01:00] [Duration: 0 ms] [Url: GET /ui/ui/static/css/main.91579e51.chunk.css] [Status: 200] [Ip: /127.0.0.1] [User: -]
2021-02-08 21:27:48,923 INFO pGroup-1-6 org.akhq.log.access [Date: 2021-02-08T21:27:48.9221696+01:00] [Duration: 0 ms] [Url: GET /ui/ui/static/js/main.223782d5.chunk.js] [Status: 200] [Ip: /127.0.0.1] [User: -]
2021-02-08 21:27:48,968 INFO pGroup-1-3 org.akhq.log.access [Date: 2021-02-08T21:27:48.9671205+01:00] [Duration: 0 ms] [Url: GET /ui/ui/favicon.ico] [Status: 200] [Ip: /127.0.0.1] [User: -]

When I open the link for AKHQ, a blank/white screen is shown. The developer console output is as follows:

Task :App.main()
2021-02-08 21:26:59,766 INFO main i.m.runtime.Micronaut Startup completed in 2506ms. Server Running: http://localhost:8080
2021-02-08 21:27:48,743 INFO r-thread-1 org.akhq.log.access [Date: 2021-02-08T21:27:48.3583598+01:00] [Duration: 416 ms] [Url: GET /] [Status: 307] [Ip: /127.0.0.1] [User: -]
2021-02-08 21:27:48,788 INFO pGroup-1-3 org.akhq.log.access [Date: 2021-02-08T21:27:48.7863468+01:00] [Duration: 1 ms] [Url: GET /ui] [Status: 200] [Ip: /127.0.0.1] [User: -]
2021-02-08 21:27:48,893 INFO pGroup-1-3 org.akhq.log.access [Date: 2021-02-08T21:27:48.8899247+01:00] [Duration: 3 ms] [Url: GET /ui/ui/static/css/2.7caccc14.chunk.css] [Status: 200] [Ip: /127.0.0.1] [User: -]
2021-02-08 21:27:48,914 INFO pGroup-1-5 org.akhq.log.access [Date: 2021-02-08T21:27:48.9143521+01:00] [Duration: 0 ms] [Url: GET /ui/ui/static/js/2.a6ffe02f.chunk.js] [Status: 200] [Ip: /127.0.0.1] [User: -]
2021-02-08 21:27:48,919 INFO pGroup-1-4 org.akhq.log.access [Date: 2021-02-08T21:27:48.9192378+01:00] [Duration: 0 ms] [Url: GET /ui/ui/static/css/main.91579e51.chunk.css] [Status: 200] [Ip: /127.0.0.1] [User: -]
2021-02-08 21:27:48,923 INFO pGroup-1-6 org.akhq.log.access [Date: 2021-02-08T21:27:48.9221696+01:00] [Duration: 0 ms] [Url: GET /ui/ui/static/js/main.223782d5.chunk.js] [Status: 200] [Ip: /127.0.0.1] [User: -]
2021-02-08 21:27:48,968 INFO pGroup-1-3 org.akhq.log.access [Date: 2021-02-08T21:27:48.9671205+01:00] [Duration: 0 ms] [Url: GET /ui/ui/favicon.ico] [Status: 200] [Ip: /127.0.0.1] [User: -]

What am I doing wrong, what is missing?

@tchiotludo
Copy link
Owner

The better is to start:

  • the backend with intellij running org.akhq.App
  • the front end with npm run start

After this the ui is working on localhost:3000

@Grishnackh
Copy link
Contributor

Thank you @tchiotludo, this worked. I consider this a very valuable information. Don't you think it would be good to have this instructions noted in the development chapter of the README?

@tchiotludo
Copy link
Owner

Agree with that, @Grishnackh fell free to add a PR about that 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@ankon @tchiotludo @Grishnackh