sam_oneal/pulumi-docker-build
1
0
Fork 0
Code Actions Packages Releases Wiki Activity

Forklift buildx provider

Browse Source
This commit is contained in:
Bryce Lampe
2024-03-25 11:40:33 -07:00
parent 2b348f84e4
commit d50d156bd8
349 changed files with 61549 additions and 1141 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1796
provider/internal/embed/image-examples.md Normal file
View File

File diff suppressed because it is too large Load Diff

226
provider/internal/embed/image-migration.md Normal file
View File

@@ -0,0 +1,226 @@
## Migrating v3 and v4 Image resources
The `buildx.Image` resource provides a superset of functionality over the `Image` resources available in versions 3 and 4 of the Pulumi Docker provider.
Existing `Image` resources can be converted to `build.Image` resources with minor modifications.
### Behavioral differences
There are several key behavioral differences to keep in mind when transitioning images to the new `buildx.Image` resource.
#### Previews
Version `3.x` of the Pulumi Docker provider always builds images during preview operations.
This is helpful as a safeguard to prevent "broken" images from merging, but users found the behavior unnecessarily redundant when running previews and updates locally.
Version `4.x` changed build-on-preview behavior to be opt-in.
By default, `v4.x` `Image` resources do _not_ build during previews, but this behavior can be toggled with the `buildOnPreview` option.
Some users felt this made previews in CI less helpful because they no longer detected bad images by default.
The default behavior of the `buildx.Image` resource has been changed to strike a better balance between CI use cases and manual updates.
By default, Pulumi will now only build `buildx.Image` resources during previews when it detects a CI environment like GitHub Actions.
Previews run in non-CI environments will not build images.
This behavior is still configurable with `buildOnPreview`.
#### Push behavior
Versions `3.x` and `4.x` of the Pulumi Docker provider attempt to push images to remote registries by default.
They expose a `skipPush: true` option to disable pushing.
The `buildx.Image` resource matches the Docker CLI's behavior and does not push images anywhere by default.
To push images to a registry you can include `push: true` (equivalent to Docker's `--push` flag) or configure an `export` of type `registry` (equivalent to Docker's `--output type=registry`).
Like Docker, if an image is configured without exports you will see a warning with instructions for how to enable pushing, but the build will still proceed normally.
#### Secrets
Version `3.x` of the Pulumi Docker provider supports secrets by way of the `extraOptions` field.
Version `4.x` of the Pulumi Docker provider does not support secrets.
The `buildx.Image` resource supports secrets but does not require those secrets to exist on-disk or in environment variables.
Instead, they should be passed directly as values.
(Please be sure to familiarize yourself with Pulumi's [native secret handling](https://www.pulumi.com/docs/concepts/secrets/).)
Pulumi also provides [ESC](https://www.pulumi.com/product/esc/) to make it easier to share secrets across stacks and environments.
#### Caching
Version `3.x` of the Pulumi Docker provider exposes `cacheFrom: bool | { stages: [...] }`.
It builds targets individually and pushes them to separate images for caching.
Version `4.x` exposes a similar parameter `cacheFrom: { images: [...] }` which pushes and pulls inline caches.
Both versions 3 and 4 require specific environment variables to be set and deviate from Docker's native caching behavior.
This can result in inefficient builds due to unnecessary image pulls, repeated file transfers, etc.
The `buildx.Image` resource delegates all caching behavior to Docker.
`cacheFrom` and `cacheTo` options (equivalent to Docker's `--cache-to` and `--cache-from`) are exposed and provide additional cache targets, such as local disk, S3 storage, etc.
#### Outputs
Versions `3.x` and `4.x` of the provider exposed a `repoDigest` output which was a fully qualified tag with digest.
In `4.x` this could also be a single sha256 hash if the image wasn't pushed.
Unlike earlier providers the `buildx.Image` resource can push multiple tags.
As a convenience, it exposes a `ref` output consisting of a tag with digest as long as the image was pushed.
If multiple tags were pushed this uses one at random.
If you need more control over tag references you can use the `digest` output, which is always a single sha256 hash as long as the image was exported somewhere.
#### Tag deletion and refreshes
Versions 3 and 4 of Pulumi Docker provider do not delete tags when the `Image` resource is deleted, nor do they confirm expected tags exist during `refresh` operations.
The `buidx.Image` will query your registries during `refresh` to ensure the expected tags exist.
If any are missing a subsequent `update` will push them.
When a `buildx.Image` is deleted, it will _attempt_ to also delete any pushed tags.
Deletion of remote tags is not guaranteed because not all registries support the manifest `DELETE` API (`docker.io` in particular).
Manifests are _not_ deleted in the same way during updates -- to do so safely would require a full build to determine whether a Pulumi operation should be an update or update-replace.
Use the [`retainOnDelete: true`](https://www.pulumi.com/docs/concepts/options/retainondelete/) option if you do not want tags deleted.
### Example migration
Examples of "fully-featured" `v3` and `v4` `Image` resources are shown below, along with an example `buildx.Image` resource showing how they would look after migration.
The `v3` resource leverages `buildx` via a `DOCKER_BUILDKIT` environment variable and CLI flags passed in with `extraOption`.
After migration, the environment variable is no longer needed and CLI flags are now properties on the `buildx.Image`.
In almost all cases, properties of `buildx.Image` are named after the Docker CLI flag they correspond to.
The `v4` resource is less functional than its `v3` counterpart because it lacks the flexibility of `extraOptions`.
It it is shown with parameters similar to the `v3` example for completeness.
{{% examples %}}
## Example Usage
{{% example %}}
### v3/v4 migration
```typescript
// v3 Image
const v3 = new docker.Image("v3-image", {
imageName: "myregistry.com/user/repo:latest",
localImageName: "local-tag",
skipPush: false,
build: {
dockerfile: "./Dockerfile",
context: "../app",
target: "mytarget",
args: {
MY_BUILD_ARG: "foo",
},
env: {
DOCKER_BUILDKIT: "1",
},
extraOptions: [
"--cache-from",
"type=registry,myregistry.com/user/repo:cache",
"--cache-to",
"type=registry,myregistry.com/user/repo:cache",
"--add-host",
"metadata.google.internal:169.254.169.254",
"--secret",
"id=mysecret,src=/local/secret",
"--ssh",
"default=/home/runner/.ssh/id_ed25519",
"--network",
"host",
"--platform",
"linux/amd64",
],
},
registry: {
server: "myregistry.com",
username: "username",
password: pulumi.secret("password"),
},
});
// v3 Image after migrating to buildx.Image
const v3Migrated = new docker.buildx.Image("v3-to-buildx", {
tags: ["myregistry.com/user/repo:latest", "local-tag"],
push: true,
dockerfile: {
location: "./Dockerfile",
},
context: {
location: "../app",
},
targets: ["mytarget"],
buildArgs: {
MY_BUILD_ARG: "foo",
},
cacheFrom: [{ registry: { ref: "myregistry.com/user/repo:cache" } }],
cacheTo: [{ registry: { ref: "myregistry.com/user/repo:cache" } }],
secrets: {
mysecret: "value",
},
addHosts: ["metadata.google.internal:169.254.169.254"],
ssh: {
default: ["/home/runner/.ssh/id_ed25519"],
},
network: "host",
platforms: ["linux/amd64"],
registries: [{
address: "myregistry.com",
username: "username",
password: pulumi.secret("password"),
}],
});
// v4 Image
const v4 = new docker.Image("v4-image", {
imageName: "myregistry.com/user/repo:latest",
skipPush: false,
build: {
dockerfile: "./Dockerfile",
context: "../app",
target: "mytarget",
args: {
MY_BUILD_ARG: "foo",
},
cacheFrom: {
images: ["myregistry.com/user/repo:cache"],
},
addHosts: ["metadata.google.internal:169.254.169.254"],
network: "host",
platform: "linux/amd64",
},
buildOnPreview: true,
registry: {
server: "myregistry.com",
username: "username",
password: pulumi.secret("password"),
},
});
// v4 Image after migrating to buildx.Image
const v4Migrated = new docker.buildx.Image("v4-to-buildx", {
tags: ["myregistry.com/user/repo:latest"],
push: true,
dockerfile: {
location: "./Dockerfile",
},
context: {
location: "../app",
},
targets: ["mytarget"],
buildArgs: {
MY_BUILD_ARG: "foo",
},
cacheFrom: [{ registry: { ref: "myregistry.com/user/repo:cache" } }],
cacheTo: [{ registry: { ref: "myregistry.com/user/repo:cache" } }],
addHosts: ["metadata.google.internal:169.254.169.254"],
network: "host",
platforms: ["linux/amd64"],
registries: [{
address: "myregistry.com",
username: "username",
password: pulumi.secret("password"),
}],
});
```
{{% /example %}}

408
provider/internal/embed/index-examples.md Normal file
View File

@@ -0,0 +1,408 @@
{{% examples %}}
## Example Usage
{{% example %}}
### Multi-platform registry caching
```typescript
import * as pulumi from "@pulumi/pulumi";
import * as docker from "@pulumi/docker";
const amd64 = new docker.buildx.Image("amd64", {
cacheFrom: [{
registry: {
ref: "docker.io/pulumi/pulumi:cache-amd64",
},
}],
cacheTo: [{
registry: {
mode: docker.buildx.image.CacheMode.Max,
ref: "docker.io/pulumi/pulumi:cache-amd64",
},
}],
context: {
location: "app",
},
platforms: [docker.buildx.image.Platform.Linux_amd64],
tags: ["docker.io/pulumi/pulumi:3.107.0-amd64"],
});
const arm64 = new docker.buildx.Image("arm64", {
cacheFrom: [{
registry: {
ref: "docker.io/pulumi/pulumi:cache-arm64",
},
}],
cacheTo: [{
registry: {
mode: docker.buildx.image.CacheMode.Max,
ref: "docker.io/pulumi/pulumi:cache-arm64",
},
}],
context: {
location: "app",
},
platforms: [docker.buildx.image.Platform.Linux_arm64],
tags: ["docker.io/pulumi/pulumi:3.107.0-arm64"],
});
const index = new docker.buildx.Index("index", {
sources: [
amd64.ref,
arm64.ref,
],
tag: "docker.io/pulumi/pulumi:3.107.0",
});
export const ref = index.ref;
```
```python
import pulumi
import pulumi_docker as docker
amd64 = docker.buildx.Image("amd64",
cache_from=[docker.buildx.CacheFromArgs(
registry=docker.buildx.CacheFromRegistryArgs(
ref="docker.io/pulumi/pulumi:cache-amd64",
),
)],
cache_to=[docker.buildx.CacheToArgs(
registry=docker.buildx.CacheToRegistryArgs(
mode=docker.buildx.image.CacheMode.MAX,
ref="docker.io/pulumi/pulumi:cache-amd64",
),
)],
context=docker.buildx.BuildContextArgs(
location="app",
),
platforms=[docker.buildx.image.Platform.LINUX_AMD64],
tags=["docker.io/pulumi/pulumi:3.107.0-amd64"])
arm64 = docker.buildx.Image("arm64",
cache_from=[docker.buildx.CacheFromArgs(
registry=docker.buildx.CacheFromRegistryArgs(
ref="docker.io/pulumi/pulumi:cache-arm64",
),
)],
cache_to=[docker.buildx.CacheToArgs(
registry=docker.buildx.CacheToRegistryArgs(
mode=docker.buildx.image.CacheMode.MAX,
ref="docker.io/pulumi/pulumi:cache-arm64",
),
)],
context=docker.buildx.BuildContextArgs(
location="app",
),
platforms=[docker.buildx.image.Platform.LINUX_ARM64],
tags=["docker.io/pulumi/pulumi:3.107.0-arm64"])
index = docker.buildx.Index("index",
sources=[
amd64.ref,
arm64.ref,
],
tag="docker.io/pulumi/pulumi:3.107.0")
pulumi.export("ref", index.ref)
```
```csharp
using System.Collections.Generic;
using System.Linq;
using Pulumi;
using Docker = Pulumi.Docker;
return await Deployment.RunAsync(() =>
{
var amd64 = new Docker.Buildx.Image("amd64", new()
{
CacheFrom = new[]
{
new Docker.Buildx.Inputs.CacheFromArgs
{
Registry = new Docker.Buildx.Inputs.CacheFromRegistryArgs
{
Ref = "docker.io/pulumi/pulumi:cache-amd64",
},
},
},
CacheTo = new[]
{
new Docker.Buildx.Inputs.CacheToArgs
{
Registry = new Docker.Buildx.Inputs.CacheToRegistryArgs
{
Mode = Docker.Buildx.Image.CacheMode.Max,
Ref = "docker.io/pulumi/pulumi:cache-amd64",
},
},
},
Context = new Docker.Buildx.Inputs.BuildContextArgs
{
Location = "app",
},
Platforms = new[]
{
Docker.Buildx.Image.Platform.Linux_amd64,
},
Tags = new[]
{
"docker.io/pulumi/pulumi:3.107.0-amd64",
},
});
var arm64 = new Docker.Buildx.Image("arm64", new()
{
CacheFrom = new[]
{
new Docker.Buildx.Inputs.CacheFromArgs
{
Registry = new Docker.Buildx.Inputs.CacheFromRegistryArgs
{
Ref = "docker.io/pulumi/pulumi:cache-arm64",
},
},
},
CacheTo = new[]
{
new Docker.Buildx.Inputs.CacheToArgs
{
Registry = new Docker.Buildx.Inputs.CacheToRegistryArgs
{
Mode = Docker.Buildx.Image.CacheMode.Max,
Ref = "docker.io/pulumi/pulumi:cache-arm64",
},
},
},
Context = new Docker.Buildx.Inputs.BuildContextArgs
{
Location = "app",
},
Platforms = new[]
{
Docker.Buildx.Image.Platform.Linux_arm64,
},
Tags = new[]
{
"docker.io/pulumi/pulumi:3.107.0-arm64",
},
});
var index = new Docker.Buildx.Index("index", new()
{
Sources = new[]
{
amd64.Ref,
arm64.Ref,
},
Tag = "docker.io/pulumi/pulumi:3.107.0",
});
return new Dictionary<string, object?>
{
["ref"] = index.Ref,
};
});
```
```go
package main
import (
"github.com/pulumi/pulumi-docker/sdk/v4/go/docker/buildx"
"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)
func main() {
pulumi.Run(func(ctx *pulumi.Context) error {
amd64, err := buildx.NewImage(ctx, "amd64", &buildx.ImageArgs{
CacheFrom: buildx.CacheFromArray{
&buildx.CacheFromArgs{
Registry: &buildx.CacheFromRegistryArgs{
Ref: pulumi.String("docker.io/pulumi/pulumi:cache-amd64"),
},
},
},
CacheTo: buildx.CacheToArray{
&buildx.CacheToArgs{
Registry: &buildx.CacheToRegistryArgs{
Mode: buildx.CacheModeMax,
Ref: pulumi.String("docker.io/pulumi/pulumi:cache-amd64"),
},
},
},
Context: &buildx.BuildContextArgs{
Location: pulumi.String("app"),
},
Platforms: buildx.PlatformArray{
buildx.Platform_Linux_amd64,
},
Tags: pulumi.StringArray{
pulumi.String("docker.io/pulumi/pulumi:3.107.0-amd64"),
},
})
if err != nil {
return err
}
arm64, err := buildx.NewImage(ctx, "arm64", &buildx.ImageArgs{
CacheFrom: buildx.CacheFromArray{
&buildx.CacheFromArgs{
Registry: &buildx.CacheFromRegistryArgs{
Ref: pulumi.String("docker.io/pulumi/pulumi:cache-arm64"),
},
},
},
CacheTo: buildx.CacheToArray{
&buildx.CacheToArgs{
Registry: &buildx.CacheToRegistryArgs{
Mode: buildx.CacheModeMax,
Ref: pulumi.String("docker.io/pulumi/pulumi:cache-arm64"),
},
},
},
Context: &buildx.BuildContextArgs{
Location: pulumi.String("app"),
},
Platforms: buildx.PlatformArray{
buildx.Platform_Linux_arm64,
},
Tags: pulumi.StringArray{
pulumi.String("docker.io/pulumi/pulumi:3.107.0-arm64"),
},
})
if err != nil {
return err
}
index, err := buildx.NewIndex(ctx, "index", &buildx.IndexArgs{
Sources: pulumi.StringArray{
amd64.Ref,
arm64.Ref,
},
Tag: pulumi.String("docker.io/pulumi/pulumi:3.107.0"),
})
if err != nil {
return err
}
ctx.Export("ref", index.Ref)
return nil
})
}
```
```yaml
description: Multi-platform registry caching
name: registry-caching
outputs:
ref: ${index.ref}
resources:
amd64:
properties:
cacheFrom:
- registry:
ref: docker.io/pulumi/pulumi:cache-amd64
cacheTo:
- registry:
mode: max
ref: docker.io/pulumi/pulumi:cache-amd64
context:
location: app
platforms:
- linux/amd64
tags:
- docker.io/pulumi/pulumi:3.107.0-amd64
type: docker:buildx/image:Image
arm64:
properties:
cacheFrom:
- registry:
ref: docker.io/pulumi/pulumi:cache-arm64
cacheTo:
- registry:
mode: max
ref: docker.io/pulumi/pulumi:cache-arm64
context:
location: app
platforms:
- linux/arm64
tags:
- docker.io/pulumi/pulumi:3.107.0-arm64
type: docker:buildx/image:Image
index:
properties:
sources:
- ${amd64.ref}
- ${arm64.ref}
tag: docker.io/pulumi/pulumi:3.107.0
type: docker:buildx/image:Index
runtime: yaml
```
```java
package generated_program;
import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.pulumi.docker.buildx.Image;
import com.pulumi.docker.buildx.ImageArgs;
import com.pulumi.docker.buildx.inputs.CacheFromArgs;
import com.pulumi.docker.buildx.inputs.CacheFromRegistryArgs;
import com.pulumi.docker.buildx.inputs.CacheToArgs;
import com.pulumi.docker.buildx.inputs.CacheToRegistryArgs;
import com.pulumi.docker.buildx.inputs.BuildContextArgs;
import com.pulumi.docker.buildx.Index;
import com.pulumi.docker.buildx.IndexArgs;
import java.util.List;
import java.util.ArrayList;
import java.util.Map;
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Paths;
public class App {
public static void main(String[] args) {
Pulumi.run(App::stack);
}
public static void stack(Context ctx) {
var amd64 = new Image("amd64", ImageArgs.builder()
.cacheFrom(CacheFromArgs.builder()
.registry(CacheFromRegistryArgs.builder()
.ref("docker.io/pulumi/pulumi:cache-amd64")
.build())
.build())
.cacheTo(CacheToArgs.builder()
.registry(CacheToRegistryArgs.builder()
.mode("max")
.ref("docker.io/pulumi/pulumi:cache-amd64")
.build())
.build())
.context(BuildContextArgs.builder()
.location("app")
.build())
.platforms("linux/amd64")
.tags("docker.io/pulumi/pulumi:3.107.0-amd64")
.build());
var arm64 = new Image("arm64", ImageArgs.builder()
.cacheFrom(CacheFromArgs.builder()
.registry(CacheFromRegistryArgs.builder()
.ref("docker.io/pulumi/pulumi:cache-arm64")
.build())
.build())
.cacheTo(CacheToArgs.builder()
.registry(CacheToRegistryArgs.builder()
.mode("max")
.ref("docker.io/pulumi/pulumi:cache-arm64")
.build())
.build())
.context(BuildContextArgs.builder()
.location("app")
.build())
.platforms("linux/arm64")
.tags("docker.io/pulumi/pulumi:3.107.0-arm64")
.build());
var index = new Index("index", IndexArgs.builder()
.sources(
amd64.ref(),
arm64.ref())
.tag("docker.io/pulumi/pulumi:3.107.0")
.build());
ctx.export("ref", index.ref());
}
}
```
{{% /example %}}
{{% /examples %}}