Nomadproject.io missing documentation about Docker driver `image` behavior

wimax-grapl · September 10, 2021, 2:00am

Hi there!

I just discovered an undocumented (and, for my team, useful) behavior of Nomad’s Docker driver that I’d love to see documented on Drivers: Docker | Nomad | HashiCorp Developer .

According to Hashicorp dev jrasell, in this topic:

Nomad looks locally first for Docker images before attempting to pull from Docker Hub.

This is not mentioned in the online documentation, which says conversely:

By default it will be fetched from Docker Hub

Would love to get this cleared up. Thank you!

fhemberger · September 10, 2021, 11:31am

Nomad delegates image handling to the Docker daemon, which default behavior is to check the existence of an image locally before pulling from Docker Hub (unless you run docker run --pull always, which corresponds to force_pull = true in Nomad’s job file). So the pulling strategy is not really a feature of Nomad itself.

wimax-grapl · September 10, 2021, 7:45pm

It looks like Nomad does have its own specific behavior around ‘latest’ that is not specified by Docker:

github.com

hashicorp/nomad/blob/main/drivers/docker/driver.go#L537

    
      
          // loading it from the file system
          func (d *Driver) createImage(task *drivers.TaskConfig, driverConfig *TaskConfig, client *docker.Client) (string, error) {
          	image := driverConfig.Image
          	repo, tag := parseDockerImage(image)
          
          
	// We're going to check whether the image is already downloaded. If the tag
          	// is "latest", or ForcePull is set, we have to check for a new version every time so we don't
          	// bother to check and cache the id here. We'll download first, then cache.
          	if driverConfig.ForcePull {
          		d.logger.Debug("force pulling image instead of inspecting local", "image_ref", dockerImageRef(repo, tag))
          	} else if tag != "latest" {
          		if dockerImage, _ := client.InspectImage(image); dockerImage != nil {
          			// Image exists so just increment its reference count
          			d.coordinator.IncrementImageReference(dockerImage.ID, image, task.ID)
          			return dockerImage.ID, nil
          		}
          	}
          
          
	// Load the image if specified
          	if driverConfig.LoadImage != "" {
          		return d.loadImage(task, driverConfig, client)

Basically:
tag ‘latest’ causes a force pull, and that’s not documented (and not default Docker behavior).

jrasell · September 13, 2021, 7:09am

Hi @wimax-grapl and @fhemberger, thanks for the discussion on this topic. I feel this is something worth documenting on the Nomad site so readers don’t have to search for this detail in multiple places.

@wimax-grapl would you be able to raise an issue request against the Nomad repository so we can track this?

Thanks,
jrasell and the Nomad team

angrycub · September 13, 2021, 4:42pm

This behavior is described in the documentation for the image attribute for the Docker task driver.

If the tag is omitted or equal to latest the driver will always try to pull the image.

Is it a question of formatting that information so that it’s more visible?

wimax-grapl · September 13, 2021, 7:14pm

I guess ultimately my problem is with the previous sentence, “By default it will be fetched from Docker Hub”. The true behavior is that "By default it will be fetched from the host’s Docker daemon, unless the tag is latest".

Anyway - yes, I’ll raise an issue. Thanks!