Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[grid][java] browser containers provisioned in dynamic grid can get hostconfig from node-docker #13804

Merged
merged 3 commits into from
Apr 12, 2024

Conversation

VietND96
Copy link
Member

@VietND96 VietND96 commented Apr 11, 2024

User description

Thanks for contributing to Selenium!
A PR well described will help maintainers to quickly review and merge it

Before submitting your PR, please check our contributing guidelines.
Avoid large PRs, help reviewers by making them as simple and short as possible.

Description

Motivation and Context

Fix: SeleniumHQ/docker-selenium#2143
Fix: SeleniumHQ/docker-selenium#2098
Fix: SeleniumHQ/docker-selenium#1812
Fix: SeleniumHQ/docker-selenium#1642

Docker compose file to deploy node-docker looks like

services:
  node-docker:
    image: ${NAMESPACE}/node-docker:${TAG}
    user: root
    volumes:
      - ./videos:/opt/selenium/assets
      - ./videos/config.toml:/opt/bin/config.toml
      - /var/run/docker.sock:/var/run/docker.sock
      - ./videos/Downloads:/home/seluser/Downloads
    dns:
      - 8.8.8.8
      - 8.8.4.4
    dns_search: selenium-grid.local
    extra_hosts:
      - "prod.domain.com:${HOST_IP}"
    depends_on:
      - selenium-hub
    environment:
      - SE_EVENT_BUS_HOST=selenium-hub
      - SE_EVENT_BUS_PUBLISH_PORT=4442
      - SE_EVENT_BUS_SUBSCRIBE_PORT=4443
      - SE_NODE_ENABLE_MANAGED_DOWNLOADS=true
      - SE_LOG_LEVEL=${LOG_LEVEL}

config.toml looks like

[docker]
configs = [
    "${NAMESPACE}/standalone-firefox:${TAG}", '{"browserName": "firefox", "platformName": "linux"}',
    "${NAMESPACE}/standalone-chrome:${TAG}", '{"browserName": "chrome", "platformName": "linux"}',
    "${NAMESPACE}/standalone-edge:${TAG}", '{"browserName": "MicrosoftEdge", "platformName": "linux"}'
    ]

host-config-keys = ["Dns", "DnsOptions", "DnsSearch", "ExtraHosts", "Binds"]

url = "http://127.0.0.1:2375"
video-image = "${NAMESPACE}/video:${VIDEO_TAG}"

[node]
enable-managed-downloads = true
override-max-sessions = true
max-sessions = 4

With the config key host-config-keys under section [docker] in a config.toml file (or CLI option --docker-host-config-keys). Users can specify a list of docker host configuration keys that should be passed to browser containers.

Valid key names for Docker host config can be found in the Docker API documentation or via the command docker inspect the node-docker container.

Steps to debug

Set the log-level to FINE, and you are able to see the Container config looks like

10:15:38.988 DEBUG [DockerSessionFactory.createBrowserContainer] - Container config: ContainerConfig{image=Image{summary=org.openqa.selenium.docker.internal.ImageSummary@3bcbb589}, portBindings={}, envVars={SE_EVENT_BUS_PUBLISH_PORT=4442, SE_EVENT_BUS_HOST=selenium-hub, SE_REJECT_UNSUPPORTED_CAPS=false, SE_EVENT_BUS_SUBSCRIBE_PORT=4443, TZ=UTC, SE_NODE_ENABLE_MANAGED_DOWNLOADS=false, SE_OTEL_TRACES_EXPORTER=otlp, SE_BIND_HOST=false, SE_OTEL_JAVA_GLOBAL_AUTOCONFIGURE_ENABLED=true, SE_OTEL_SERVICE_NAME=selenium-node-docker, SE_LOG_LEVEL=FINE}, volumeBinds={}, networkName=tests_default, devices=[], autoRemove=true, shmSize=2147483648, hostConfig={ExtraHosts=[prod.domain.com:10.1.0.178], Binds=[/home/runner/work/docker-selenium/docker-selenium/tests/videos:/opt/selenium/assets:rw, /home/runner/work/docker-selenium/docker-selenium/tests/videos/config.toml:/opt/bin/config.toml:rw, /var/run/docker.sock:/var/run/docker.sock:rw, /home/runner/work/docker-selenium/docker-selenium/tests/videos/Downloads:/home/seluser/Downloads:rw], Dns=[8.8.8.8, 8.8.4.4], DnsSearch=[selenium-grid.local]}}

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist

  • I have read the contributing document.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I have added tests to cover my changes.
  • All new and existing tests passed.

Type

enhancement


Description

  • Added support for Docker host configurations in Selenium Grid's Docker integration, allowing for more granular control over browser containers.
  • Introduced hostConfig in ContainerConfig and ContainerInfo to manage and apply Docker container configurations.
  • Added a new flag --docker-host-config-keys to specify which host configuration keys should be passed to browser containers.
  • Enhanced DockerSessionFactory to apply specified host configurations to browser containers, improving flexibility in container management.

Changes walkthrough

Relevant files
Enhancement
ContainerConfig.java
Enhance ContainerConfig with Host Configuration Support   

java/src/org/openqa/selenium/docker/ContainerConfig.java

  • Added support for hostConfig in ContainerConfig to manage Docker
    container configurations.
  • Introduced a new constructor and a method getHostConfig to filter and
    apply host configurations based on provided keys.
  • +34/-0   
    ContainerInfo.java
    Include Host Config in ContainerInfo                                         

    java/src/org/openqa/selenium/docker/ContainerInfo.java

  • Added hostConfig map to store Docker host configurations.
  • Updated constructor to include hostConfig and added a getter method
    for it.
  • +11/-1   
    InspectContainer.java
    Update InspectContainer to Fetch HostConfig                           

    java/src/org/openqa/selenium/docker/v1_41/InspectContainer.java

  • Modified apply method to include hostConfig in ContainerInfo
    instantiation.
  • +2/-1     
    DockerFlags.java
    Add Docker Host Config Keys Flag                                                 

    java/src/org/openqa/selenium/grid/node/docker/DockerFlags.java

  • Introduced --docker-host-config-keys flag to specify Docker container
    host configuration keys.
  • +12/-0   
    DockerOptions.java
    Implement Host Config Keys Handling in DockerOptions         

    java/src/org/openqa/selenium/grid/node/docker/DockerOptions.java

  • Added handling for hostConfigKeys to filter and apply specific Docker
    host configurations.
  • +12/-1   
    DockerSessionFactory.java
    Enhance DockerSessionFactory with Host Config Management 

    java/src/org/openqa/selenium/grid/node/docker/DockerSessionFactory.java

  • Added hostConfig and hostConfigKeys to manage Docker host
    configurations for browser containers.
  • Updated createBrowserContainer method to apply host configurations
    based on the provided keys.
  • +14/-4   

    PR-Agent usage:
    Comment /help on the PR to get a list of all available PR-Agent tools and their descriptions

    @VietND96 VietND96 marked this pull request as ready for review April 11, 2024 21:20
    Copy link
    Contributor

    PR Description updated to latest commit (a73ba2f)

    Copy link
    Contributor

    PR Review

    ⏱️ Estimated effort to review [1-5]

    4, because the PR involves multiple changes across various files that integrate new functionality for handling host configurations in Docker containers. The changes are spread across the creation, configuration, and inspection of containers, which requires a thorough understanding of the existing Docker integration within the Selenium Grid. Additionally, the introduction of new methods and modifications to existing ones necessitate a detailed review to ensure compatibility and adherence to project standards.

    🧪 Relevant tests

    No

    🔍 Possible issues

    Possible Bug: The method getHostConfig in DockerSessionFactory.java non-null asserts hostConfig and hostConfigKeys, but getDockerHostConfig in DockerOptions.java can potentially return null, which could lead to a NullPointerException.

    Inconsistency: The handling of hostConfig in ContainerConfig.java merges the provided hostConfig map with itself, which seems redundant and could be a mistake. This occurs in the toJson method where copyMap.putAll(this.hostConfig); is called.

    Performance Concern: The use of streams and filters in ContainerConfig.java for getHostConfig method might introduce performance overhead, especially for large configurations. It's worth considering optimizing this part of the code.

    🔒 Security concerns

    No


    ✨ Review tool usage guide:

    Overview:
    The review tool scans the PR code changes, and generates a PR review which includes several types of feedbacks, such as possible PR issues, security threats and relevant test in the PR. More feedbacks can be added by configuring the tool.

    The tool can be triggered automatically every time a new PR is opened, or can be invoked manually by commenting on any PR.

    • When commenting, to edit configurations related to the review tool (pr_reviewer section), use the following template:
    /review --pr_reviewer.some_config1=... --pr_reviewer.some_config2=...
    
    [pr_reviewer]
    some_config1=...
    some_config2=...
    

    See the review usage page for a comprehensive guide on using this tool.

    Copy link
    Contributor

    PR Code Suggestions

    CategorySuggestions                                                                                                                                                       
    Best practice
    Use ImmutableMap for initializing hostConfig in the constructor.

    Consider using ImmutableMap instead of HashMap for initializing hostConfig in the
    constructor to ensure immutability of the configuration maps. This change enhances thread
    safety and predictability of the ContainerConfig instances.

    java/src/org/openqa/selenium/docker/ContainerConfig.java [56]

    -this(image, portBindings, envVars, volumeBinds, devices, networkName, shmSize, new HashMap<>());
    +this(image, portBindings, envVars, volumeBinds, devices, networkName, shmSize, ImmutableMap.of());
     
    Possible issue
    Add null check for hostConfig in getHostConfig method.

    To avoid potential NullPointerException, add a null check for hostConfig parameter in the
    getHostConfig method before streaming and collecting keys.

    java/src/org/openqa/selenium/docker/ContainerConfig.java [142-146]

    -configKeys.stream()
    -        .filter(hostConfig::containsKey)
    -        .filter(key -> hostConfig.get(key) != null)
    -        .collect(Collectors.toMap(key -> key, hostConfig::get));
    +if (hostConfig != null) {
    +    return configKeys.stream()
    +            .filter(hostConfig::containsKey)
    +            .filter(key -> hostConfig.get(key) != null)
    +            .collect(Collectors.toMap(key -> key, hostConfig::get));
    +}
    +return Collections.emptyMap();
     
    Handle potential null hostConfig in ContainerInfo constructor.

    Ensure hostConfig is safely handled when it's null. The constructor currently requires it
    to be non-null, but there might be cases where hostConfig is not provided. Consider
    allowing hostConfig to be nullable or provide a default empty map.

    java/src/org/openqa/selenium/docker/ContainerInfo.java [44]

    -this.hostConfig = Require.nonNull("Host config", hostConfig);
    +this.hostConfig = hostConfig != null ? hostConfig : Collections.emptyMap();
     
    Enhancement
    Simplify merging logic in toJson method for hostConfig.

    The logic to merge hostConfig maps in toJson method seems redundant and potentially
    incorrect as it copies the map onto itself. Consider simplifying this by directly using
    ImmutableMap.copyOf(hostConfig) if not empty, or just skip adding to the final map if
    hostConfig is empty.

    java/src/org/openqa/selenium/docker/ContainerConfig.java [206-209]

     if (this.hostConfig != null && !this.hostConfig.isEmpty()) {
    -  Map<String, Object> copyMap = new HashMap<>(hostConfig);
    -  copyMap.putAll(this.hostConfig);
    -  hostConfig = ImmutableMap.copyOf(copyMap);
    +  hostConfig = ImmutableMap.copyOf(this.hostConfig);
     }
     
    Ensure hostConfigKeys are utilized in createBrowserContainer.

    When creating a new ContainerConfig instance in createBrowserContainer, it's important to
    ensure that hostConfig and hostConfigKeys are properly utilized. Currently, the method
    signature was changed to include hostConfig, but the implementation does not show how
    hostConfigKeys is used. Ensure that hostConfigKeys are utilized to filter or modify the
    hostConfig before passing it to ContainerConfig.

    java/src/org/openqa/selenium/grid/node/docker/DockerSessionFactory.java [297]

    -.getHostConfig(hostConfig, hostConfigKeys);
    +.getHostConfig(filterHostConfig(hostConfig, hostConfigKeys), hostConfigKeys);
     

    ✨ Improve tool usage guide:

    Overview:
    The improve tool scans the PR code changes, and automatically generates suggestions for improving the PR code. The tool can be triggered automatically every time a new PR is opened, or can be invoked manually by commenting on a PR.

    • When commenting, to edit configurations related to the improve tool (pr_code_suggestions section), use the following template:
    /improve --pr_code_suggestions.some_config1=... --pr_code_suggestions.some_config2=...
    
    [pr_code_suggestions]
    some_config1=...
    some_config2=...
    

    See the improve usage page for a comprehensive guide on using this tool.

    @VietND96 VietND96 force-pushed the docker branch 2 times, most recently from 08787b9 to d5971ea Compare April 12, 2024 04:58
    @VietND96
    Copy link
    Member Author

    @diemol, could you please review this?

    Copy link
    Member

    @diemol diemol left a comment

    Choose a reason for hiding this comment

    The reason will be displayed to describe this comment to others. Learn more.

    Thank you, @VietND96!

    Just a comment in one of the changes.

    Signed-off-by: Viet Nguyen Duc <[email protected]>
    @VietND96 VietND96 requested a review from diemol April 12, 2024 10:51
    Copy link
    Member

    @diemol diemol left a comment

    Choose a reason for hiding this comment

    The reason will be displayed to describe this comment to others. Learn more.

    Thank you, @VietND96!

    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment