Add dev workflow README, improve other readmes

05-example-web-application/README.md

@@ -3,9 +3,9 @@
 ![](./readme-assets/app-screenshot.png)
 
 ## Minimal 3 tier web application
-- React frontend
+- **React frontend:** Uses react query to load data from the two apis and display the result
-- Node JS and Golang APIs
+- **Node JS and Golang APIs:** Both have `/` and `/ping` endpoints. `/` queries the Database for the current time, and `/ping` returns `pong`
-- Postgres Database
+- **Postgres Database:** An empty PostgreSQL database with no tables or data. Used to show how to set up connectivity. The API applications execute `SELECT NOW() as now;` to determine the current time to return.
 
 ![](./readme-assets/tech-stack.png)
 

10-interacting-with-docker-objects/README.md

@@ -8,45 +8,93 @@ You should:
 
 ## Images
 
-1) ls
+`docker image COMMAND`:
-2) build (https://docs.docker.com/engine/reference/commandline/build/)
+```
-3) tag
+  build       Build an image from a Dockerfile (`docker build` is the same as `docker image build`)
-4) pull
+  history     Show the history of an image
-5) push
+  import      Import the contents from a tarball to create a filesystem image
-6) rm
+  inspect     Display detailed information on one or more images
-7) prune
+  load        Load an image from a tar archive or STDIN
-8) save
+  ls          List images
-9) docker scan <image> (snyk security scan, also show trivy)
+  prune       Remove unused images
+  pull        Pull an image or a repository from a registry
+  push        Push an image or a repository to a registry
+  rm          Remove one or more images
+  save        Save one or more images to a tar archive (streamed to STDOUT by default)
+  tag         Create a tag TARGET_IMAGE that refers to SOURCE_IMAGE
+```
+
+### Scanning Images
+
+Not a `docker image` subcommand, but still something you do with images:
+
+```
+docker scan IMAGE
+```
+
+***Note:*** You can also use a 3rd party scanner such as Trivy (https://github.com/aquasecurity/trivy)
+
+### Signing Images
+
+Another protection against software supply chain attacks is the ability to uniquely sign specific image tags to ensure an image was created by the entity who signed it.
+
+```
+docker trust sign IMAGE:TAG
+docker trust inspect --pretty IMAGE:TAG
+```
 
 ## Containers
 
-1) ls
+`docker container COMMAND`:
-2) run
+
-3) start
+```
-4) attach
+  attach      Attach local standard input, output, and error streams to a running container
-5) exec
+  commit      Create a new image from a container's changes
-6) logs
+  cp          Copy files/folders between a container and the local filesystem
-7) top
+  create      Create a new container
-8) cp
+  diff        Inspect changes to files or directories on a container's filesystem
-9) stop
+  exec        Run a command in a running container
-10) kill
+  export      Export a container's filesystem as a tar archive
-11) prune
+  inspect     Display detailed information on one or more containers
-12) export
+  kill        Kill one or more running containers
+  logs        Fetch the logs of a container
+  ls          List containers
+  pause       Pause all processes within one or more containers
+  port        List port mappings or a specific mapping for the container
+  prune       Remove all stopped containers
+  rename      Rename a container
+  restart     Restart one or more containers
+  rm          Remove one or more containers
+  run         Run a command in a new container
+  start       Start one or more stopped containers
+  stats       Display a live stream of container(s) resource usage statistics
+  stop        Stop one or more running containers
+  top         Display the running processes of a container
+  unpause     Unpause all processes within one or more containers
+  update      Update configuration of one or more containers
+  wait        Block until one or more containers stop, then print their exit codes
+```
 
 ## Volumes
 
-1) ls
+`docker volume COMMAND`:
-2) create
+```
-3) inspect
+  create      Create a volume
-4) rm
+  inspect     Display detailed information on one or more volumes
-5) prune
+  ls          List volumes
+  prune       Remove all unused local volumes
+  rm          Remove one or more volumes
+```
 
 ## Networks
 
-1) ls
+`docker network COMMAND`:
-2) create
+```
-3) inspect
+  connect     Connect a container to a network
-4) connect
+  create      Create a network
-5) disconnect
+  disconnect  Disconnect a container from a network
-6) rm
+  inspect     Display detailed information on one or more networks
-7) prune
+  ls          List networks
+  prune       Remove all unused networks
+  rm          Remove one or more networks
+```

11-development-workflow/Makefile

@@ -1,5 +1,6 @@
 DEV_COMPOSE_FILE=docker-compose-dev.yml
 DEBUG_COMPOSE_FILE=docker-compose-debug.yml
+TEST_COMPOSE_FILE=docker-compose-test.yml
 
 ### DOCKER COMPOSE COMMANDS
 
@@ -26,14 +27,9 @@ compose-down:
 ###
 
 DOCKERCONTEXT_DIR:=../05-example-web-application/
-DOCKERFILE_DIR:=../10-development-workflow/
+DOCKERFILE_DIR:=../11-development-workflow/
-
-.PHONY: docker-build-all
-docker-build-all:
-	docker build -t api-node -f ${DOCKERFILE_DIR}/api-node/Dockerfile.dev ${DOCKERCONTEXT_DIR}/api-node/
-	docker build -t api-golang -f ${DOCKERFILE_DIR}/api-golang/Dockerfile.dev ${DOCKERCONTEXT_DIR}/api-golang/
 
 .PHONY: run-tests
 run-tests:
-	docker run -t api-golang go test -v ./...
+	docker compose -f $(DEV_COMPOSE_FILE) -f $(TEST_COMPOSE_FILE) run --build api-golang
-	docker run -it api-node npm run test
+	docker compose -f $(DEV_COMPOSE_FILE) -f $(TEST_COMPOSE_FILE) run --build api-node

11-development-workflow/README.md

@@ -0,0 +1,47 @@
+# Development Workflow
+
+## Development Environment
+
+Because we are running our application within containers, we need a way to quickly iterate and make changes to them. Some of our tactics in `06-building-container-images` help here (e.g. protecting the layer cache) so that images build quickly, but we can do better.
+
+We want our development environment to have the following attributes:
+
+1) **Easy/simple to set up:** Using docker compose, we can define the entire environment with a single yaml file. To get started, team members can issue a single command `make compose-up-build` or `make compose-up-build-debug` depending if they want to run the debugger or not.
+
+2) **Ability to iterate without rebuilding the container image:** In order to avoid having to rebuild the container image with every single change, we can use a bind mount to mount the code from our host into the container filesystem. For example:
+
+```yml
+      - type: bind
+        source: ../05-example-web-application/api-node/
+        target: /usr/src/app/
+```
+
+3) **Automatic reloading of the application:** 
+   - <ins>*React Client:*</ins> We are using Vite for the react client which handles this handles this automatically
+   - <ins>*Node API:*</ins> We added nodemon as a development dependency and specify the Docker CMD to use it
+   - <ins>*Golang API:*</ins> We added a utility called `air` (https://github.com/cosmtrek/air) within `Dockerfile.dev` which watches for changes and rebuild the app automatically.
+
+4) **Use a debugger:**
+   - <ins>*React Client:*</ins> For a react app, you can use the browser developer tools + extensions to debug. I did include `react-query-devtools` to help debug react query specific things. It is also viewed from within the browser.
+   - <ins>*Node API:*</ins> To enable debugging for a NodeJS application we can run the app with the `--inspect` flag. The debug session can then be accessed via a websocket on port `9229`. The additional considerations in this case are to specify that the debugger listen for requests from 0.0.0.0 (any) and to publish port `9229` from the container to localhost.
+   - <ins>*Golang API:*</ins> To enable remote debugging for a golang application I installed a tool called delve (https://github.com/go-delve/delve) within `./api-golang/Dockerfile.dev`. We then override the command used to run the container to use this tool (see: `docker-compose-debug.yml`)
+     
+      ---
+
+      These modifications to the configuration (overridden commands + port publishing) are specified in `docker-compose-debug.yml`. By passing both `docker-compose-dev.yml` AND `docker-compose-debug.yml` to the `docker compose up` command (See: `make compose-up-debug-build`) Docker combines the two files, taking the config from the latter and overlaying it onto the former.
+
+      Both `./api-golang/README.md` and `./api-node/README.md` show a launch.json configuration you can use to connnect to these remote debuggers using VSCode. The key setting is `substitutePath` such that you can set breakpoints on your local system that get recognized within the container.
+
+5) **Executing tests:** We also need the ability to execute our test suites within containers. Again, we can create a custom `docker-compose-test.yml` overlay which modifies the container commands to execute our tests. To build the api images and execute their tests, you can execute `make run-tests` which will use the `test` compose file along with the `dev` compose file to do so.
+
+## Continuous Integration
+
+See `.github/workflows/image-ci.yml` for a basic GitHub Action workflow that builds, scans, tags, and pushes a container image.
+
+It leverages a few publicly available actions from the marketplace:
+1) https://github.com/marketplace/actions/docker-metadata-action (generates tags for the container images)
+2) https://github.com/marketplace/actions/docker-login (logs into DockerHub)
+3) https://github.com/marketplace/actions/build-and-push-docker-images (builds and pushes the images)
+4) https://github.com/marketplace/actions/aqua-security-trivy (scans the images for vulnerabilities)
+
+If you want to build out more advanced CI workflows I recommend looking at Bret Fisher's `Automation with Docker for CI/CD Workflows` repo (https://github.com/BretFisher/docker-cicd-automation). It has many great examples of the types of things you might want to do with Docker in a CI/CD pipeline!

11-development-workflow/api-golang/Dockerfile.dev

@@ -1,7 +1,4 @@
-# Pin specific version for stability
+# Using  bullseye instead of alpine because debugger didnt work in alpine
-# using  bullseye instead of alpine because of:
-##  runtime/cgo
-## cgo: C compiler "gcc" not found: exec: "gcc": executable file not found in $PATH
 FROM golang:1.19-bullseye
 
 WORKDIR /app 
@@ -12,7 +9,6 @@ RUN go install github.com/cosmtrek/air@latest
 # Install delve for debugging
 RUN go install github.com/go-delve/delve/cmd/dlv@latest
 
-# Copy only files required to install dependencies (better layer caching)
 COPY go.mod go.sum ./
 
 RUN go mod download

11-development-workflow/api-node/Dockerfile.dev

@@ -1,25 +1,15 @@
-# Pin specific version for stability
-# Use alpine for reduced image size
 FROM node:19.4-alpine as dev
 
-# Specify working directory other than /
 WORKDIR /usr/src/app
 
-# Copy only files required to install
-# dependencies (better layer caching)
 COPY package*.json ./
 
-# Install only production dependencies
-# Use cache mount to speed up install of existing dependencies
 RUN --mount=type=cache,target=/usr/src/app/.npm \
   npm set cache /usr/src/app/.npm && \
   npm install
 
-# Copy remaining source code AFTER installing dependencies. 
-# Again, copy only the necessary files
 COPY . .
 
-# Indicate expected port
 EXPOSE 3000
 
 CMD [ "npm", "run", "dev" ]

11-development-workflow/docker-compose-dev.yml

@@ -18,7 +18,7 @@ services:
   api-node:
     build:
       context: ../05-example-web-application/api-node/
-      dockerfile: ../../10-development-workflow/api-node/Dockerfile.dev
+      dockerfile: ../../11-development-workflow/api-node/Dockerfile.dev
       target: dev
     volumes:
       - type: bind
@@ -37,7 +37,7 @@ services:
   api-golang:
     build:
       context: ../05-example-web-application/api-golang/
-      dockerfile: ../../10-development-workflow/api-golang/Dockerfile.dev
+      dockerfile: ../../11-development-workflow/api-golang/Dockerfile.dev
     volumes:
       - type: bind
         source: ../05-example-web-application/api-golang/

11-development-workflow/docker-compose-test.yml

@@ -0,0 +1,13 @@
+# Overlay configuration to enable debuggers
+services:
+  api-node:
+    command: 
+      - "npm"
+      - "run"
+      - "test"
+  api-golang:
+    command: 
+      - "go"
+      - "test"
+      - "-v"
+      - "./..."