Skip to content
Snippets Groups Projects
  1. Aug 29, 2018
  2. Aug 14, 2018
    • Dduvall's avatar
      Bump Blubber config version to v3 · b6c9e31f
      Dduvall authored
      Change-Id: I91cac046976cf88d513487e27e1b52bf77a6f974
      b6c9e31f
    • Dduvall's avatar
      Refactor builder to support file requirements and run pre-install · 374976d4
      Dduvall authored
      The builder configuration has proven useful for supporting generic
      pre-entrypoint commands such as dependency managers not otherwise
      supported by specific Blubber configuration. Adding additional
      `builder.requirements` config expands support for such commands by
      allowing the user to specify files that should be copied into the image
      before the builder command runs.
      
      To support this extra configuration, `builder` had to be changed from a
      simple string to a mapping. The builder command must now by given as
      `builder.command`.
      
      The pattern of creating parent directories, copying files, and executing
      one or more commands prior to the entrypoint has become a common
      one. Some of the implementation of this pattern was moved from
      `PythonConfig` into shared build macros `build.SortFilesByDir` and
      `build.SyncFiles`. All config types that must have requirements files
      copied over independently of the entire source tree (`PythonConfig`,
      `BuilderConfig`, `NodeConfig`) now delegate to these functions.
      
      Change-Id: I67f33034f22cee2851ec866cfb07ab20c23eba8c
      374976d4
  3. Aug 10, 2018
    • Dduvall's avatar
      Test Blubber in CI using Blubber · e7ce38ca
      Dduvall authored
      Added a `blubber.yaml` that defines a single test variant and entry
      point for use with the new generic `blubber-test` CI job (see
      I31b367f2ab8b897fa988e70222f70d84186a631b).
      
      Bug: T200452
      Change-Id: I60b23e1bca8486e43ebf0f62198e85833f86b921
      e7ce38ca
  4. Aug 09, 2018
    • Dduvall's avatar
      Makefile lint rule now tests gofmt output and runs go vet correctly · d8d27730
      Dduvall authored
      The `lint` rule was previous surfacing purely whitespace output of
      `gofmt` as a failure and running `go tool vet` instead of `go vet` (the
      former accepts source files as arguments, the latter takes package
      names).
      
      A real lint error in `config/node.go` was also fixed.
      
      Change-Id: I7f13dcbb45a448f1ecac87e97527a7f4ed4d2c48
      d8d27730
  5. Aug 02, 2018
  6. Aug 01, 2018
    • Dduvall's avatar
      Provide Makefile rules for running linters and tests · 44dc7d91
      Dduvall authored
      With the move to Gerrit and away from Phabricator/Arcanist, we lost an
      easy way to run linters. New rules have been added to the `Makefile`,
      `lint`, `unit`, and `test` that run linters, unit tests, and both
      linters and unit tests, respectively.
      
      Bug: T200452
      Change-Id: I0742daaa14389841d88f13eba47dee07ac127cf2
      44dc7d91
    • Hashar's avatar
      Add .gitreview file · 1fe58ac9
      Hashar authored
      Change-Id: I4b6a2ece0f0096c49ab4dba06f358e794941f94d
      1fe58ac9
  7. Jul 31, 2018
    • Dduvall's avatar
      Change import paths from Phabricator to Gerrit · 80936122
      Dduvall authored
      All import paths and other references to the previously Blubber repo in
      Phabricator have been changed to point to the new Gerrit project URL.
      
      Note that building or testing against this change will require you to
      move your working directory to `$GOPATH/src/gerrit.wikimedia.org/r/blubber`.
      
      Contribution documentation was updated to reflect the changes, and the
      `.arcvendor` submodule has been removed.
      
      Bug: T200452
      Change-Id: I3ab23e420b2825e86e2bd7c9b3da9d4de23edaa1
      80936122
  8. Jul 26, 2018
    • Thcipriani's avatar
      Add slash to directory path if copying > 1 file · aba163aa
      Thcipriani authored
      Summary:
      Ran into a Docker error when attempting to build a node project with
      more than one requirements files:
      
          When using COPY with more than one source file, the destination must
          be a directory and end with a /
      
      Reviewers: dduvall, #release-engineering-team
      
      Reviewed By: dduvall, #release-engineering-team
      
      Tags: #release-engineering-team
      
      Differential Revision: https://phabricator.wikimedia.org/D1080
      aba163aa
  9. May 31, 2018
  10. May 25, 2018
  11. May 24, 2018
  12. May 20, 2018
  13. May 19, 2018
    • Alexandros Kosiaris's avatar
      Add a pip install step · 1b6a7003
      Alexandros Kosiaris authored and Dduvall's avatar Dduvall committed
      Summary:
      We want to also install the wheels, not just generate them. To do that
      we add a pip install -t <target> command. For this to work however, we
      need to install into /opt/lib/python and amend the environment
      accordingly to set PYTHONPATH and PATH in order for the libraries and
      binaries to be accessible.
      
      Reviewers: dduvall, thcipriani, #release-engineering-team
      
      Tags: #release-engineering-team
      
      Differential Revision: https://phabricator.wikimedia.org/D1057
      1b6a7003
  14. Apr 30, 2018
    • Dduvall's avatar
      Ensure non-interactive front-end for APT · 71f24e2b
      Dduvall authored
      Summary:
      Ensure that `DEBIAN_FRONTEND=noninteractive` is set before injecting
      `apt-get install` instructions.
      
      Test Plan: Run unit tests.
      
      Reviewers: thcipriani, mmodell, hashar, demon, #release-engineering-team
      
      Reviewed By: thcipriani, #release-engineering-team
      
      Tags: #release-engineering-team
      
      Differential Revision: https://phabricator.wikimedia.org/D1032
      71f24e2b
  15. Apr 05, 2018
    • Dduvall's avatar
      Refactor validation tests without YAML parsing · ea364b2f
      Dduvall authored
      Summary:
      Since `config.Validate` was changed to take any interface as an
      argument, many of the validation tests can be refactored in a way that
      avoids having to parse the full config context in YAML and instead
      validates each specific config struct directly. The new test pattern is
      simpler and less prone to future breakage should unrelated parts of the
      overall config change.
      
      Tests that rely on root config context were left unchanged.
      
      Test Plan: Run `go test ./...`.
      
      Reviewers: thcipriani, demon, hashar, mmodell, #release-engineering-team
      
      Reviewed By: thcipriani, #release-engineering-team
      
      Tags: #release-engineering-team
      
      Differential Revision: https://phabricator.wikimedia.org/D1023
      ea364b2f
    • Dduvall's avatar
      Introduce strict/versioned config parsing · 26b99845
      Dduvall authored
      Summary:
      Introduced a `version` config field that must be specified and match
      `config.CurrentVersion`.
      
      Changed `config.ReadConfig` to use `yaml.UnmarshalStrict` to ensure that
      errors are surfaced when unknown/bad fields are present in the given
      YAML config. A smaller `config.VersionConfig` is now unmarshaled first
      to prevalidate the new `version` field before the entire config is
      parsed.
      
      Fixes T191460
      
      Test Plan:
      Run `go test ./...`. Run `blubber` against some configuration containing
      invalid fields and ensure that it surfaces a YAML error.
      
      Reviewers: thcipriani, demon, hashar, mmodell, mobrovac, #release-engineering-team
      
      Reviewed By: thcipriani, #release-engineering-team
      
      Tags: #release-engineering-team
      
      Maniphest Tasks: T191460
      
      Differential Revision: https://phabricator.wikimedia.org/D1021
      26b99845
  16. Apr 04, 2018
  17. Mar 28, 2018
    • Dduvall's avatar
      Provide a CONTRIBUTING.md guide · 73ee493d
      Dduvall authored
      Summary:
      To help new developers get on board, let's provide a `CONTRIBUTING.md`
      doc.
      
      Test Plan: Proofread with your eyeballs.
      
      Reviewers: thcipriani, demon, hashar, mmodell, zeljkofilipin, greg, Jrbranaa, #release-engineering-team
      
      Reviewed By: thcipriani, hashar, #release-engineering-team
      
      Tags: #release-engineering-team
      
      Differential Revision: https://phabricator.wikimedia.org/D993
      73ee493d
  18. Mar 23, 2018
  19. Mar 22, 2018
    • Dduvall's avatar
      Merge branch 'master' into debian · 5787a390
      Dduvall authored
      5787a390
    • Dduvall's avatar
      Release v0.3.0 · c085b6f5
      Dduvall authored
      Summary: Bump the version to v0.3.0.
      
      Test Plan: Run `make && blubber --version` and verify the output.
      
      Reviewers: thcipriani, hashar, demon, #release-engineering-team
      
      Reviewed By: thcipriani, #release-engineering-team
      
      Tags: #release-engineering-team
      
      Differential Revision: https://phabricator.wikimedia.org/D1014
      v0.3.0
      c085b6f5
    • Thcipriani's avatar
      Policy example should work on blubber.example.yaml · 877e93a3
      Thcipriani authored
      Test Plan:
      Run:
          blubber -p policy.example.yaml blubber.example.yaml production
      
      Be happy it works.
      
      Reviewers: dduvall, #release-engineering-team
      
      Reviewed By: dduvall, #release-engineering-team
      
      Tags: #release-engineering-team
      
      Differential Revision: https://phabricator.wikimedia.org/D1013
      877e93a3
    • Thcipriani's avatar
      No need to export NewValidator · 0595606a
      Thcipriani authored
      Summary:
      It occurred to me while looking at code in another patch that there is
      no need to have this function be public.
      
      Test Plan: `go test ./...`
      
      Reviewers: dduvall, #release-engineering-team
      
      Reviewed By: dduvall, #release-engineering-team
      
      Tags: #release-engineering-team
      
      Differential Revision: https://phabricator.wikimedia.org/D1012
      0595606a
    • Dduvall's avatar
      Provide a `runs.insecurely` to be used with test variants · cffb77e9
      Dduvall authored
      Summary:
      Use cases involving running of test suites and doc generation require
      more liberal ownership and read/write permission to application files.
      When `runs.insecurely` is set to `true`, the effective runtime user will
      be `lives.as`, the same user that owns the application files and
      installed dependencies.
      
      D999 is a complement to this change to allow restrictions on this and
      other potentially sensitive configuration.
      
      Depends on D999, D1002
      
      Test Plan: Run `go test ./...`.
      
      Reviewers: thcipriani, mmodell, hashar, #release-engineering-team, demon
      
      Reviewed By: thcipriani, #release-engineering-team
      
      Tags: #release-engineering-team
      
      Differential Revision: https://phabricator.wikimedia.org/D1003
      cffb77e9
    • Dduvall's avatar
      Fix ownership on artifact copies · 50c57939
      Dduvall authored
      Summary:
      The implementation of D984 did not include enforcing ownership for
      `build.CopyFrom` instruction and so artifacts copied from one image to
      another via `copies:` were problematically owned as root.
      
      In order to fix this behavior:
       1. `config.ArtifactConfig` `build.CopyFrom` instructions are now
          injected duration `build.PhaseInstall`
       2. `config.VariantConfig` calls `build.ApplyUser` for these artifact
          instructions as well using the `runs.as` user
       3. `build.CopyAs` was refactored to wrap any `build.Instruction` which
          should only really be used with `build.Copy` or `build.CopyFrom`.
      
      Test Plan:
      Run `go test ./...`. Run `blubber` against configuration with a variant that
      uses `copies` and verify that the `COPY --from` instructions also include a
      `--chown` flag.
      
      Reviewers: thcipriani, mmodell, hashar, #release-engineering-team, demon
      
      Reviewed By: thcipriani, #release-engineering-team
      
      Tags: #release-engineering-team
      
      Differential Revision: https://phabricator.wikimedia.org/D1002
      50c57939
  20. Mar 19, 2018
    • Dduvall's avatar
      Allow for configuration policies · eb9b69dd
      Dduvall authored
      Summary:
      Implements a rough interface for validating configuration against
      arbitrary policy rules. Policies are provided as YAML and passed via the
      command line as file paths or remote URIs.
      
      The format of policies is:
      
          enforcements:
            - path: <path>
              rule: <rule>
      
      Where `<path>` is a YAML-ish path to a config field and `<rule>` is any
      expression our config validator understands (expressions built in by the
      validator library and custom tags defined in `config.validation.go`).
      
      Example policy:
      
          enforcements:
            - path: variants.production.base
              rule: oneof=debian:jessie debian:stretch
            - path: variants.production.runs.as
              rule: ne=foo
            - path: variants.production.node.dependencies
              rule: isfalse
      
      Command flag parsing was implemented in `main.go` to support the new
      `--policy=uri` flag and improve existing handling of `--version` and the
      usage statement.
      
      Test Plan: Run `go test ./...`.
      
      Reviewers: thcipriani, demon, hashar, mmodell, #release-engineering-team
      
      Reviewed By: thcipriani, #release-engineering-team
      
      Tags: #release-engineering-team
      
      Differential Revision: https://phabricator.wikimedia.org/D999
      eb9b69dd
  21. Mar 06, 2018
    • Dduvall's avatar
      Support Python projects · 6896e655
      Dduvall authored
      Summary:
      A new root and variant `python` config field is provided with two new
      fields below, `version` and `requirements`.
      
      The former, `version`, should specify the Python executable to use when
      executing related package installation commands and ostensibly the same
      executable that will be used to run the application.
      
      The latter, `requirements`, should specify all pip requirements files
      such that a compiler that supports layered filesystems (e.g. Docker) can
      output separate instructions that will invalidate cache layers for
      changes to those files independently of changes to the rest of the
      codebase.
      
      Python related instructions will be generated only if either `version`
      or `requirements` are given.
      
      Fixes T186545
      
      Test Plan: Run `go test ./...`.
      
      Reviewers: thcipriani, hashar, demon, #release-engineering-team
      
      Reviewed By: thcipriani, #release-engineering-team
      
      Tags: #release-engineering-team
      
      Maniphest Tasks: T186545
      
      Differential Revision: https://phabricator.wikimedia.org/D976
      6896e655
    • Thcipriani's avatar
      Makefile: install to global GOPATH with correct -ldflags · b790283b
      Thcipriani authored
      Summary:
      Since we now use ldflags to set global variables it's helpful to
      automate via Make. Using a single global GOPATH seems to jive more with
      the best-practice established in the go community[0] and make
      testing/code review easier.
      
      [0]. <https://github.com/golang/go/wiki/GOPATH#use-a-single-gopath>
      
      Test Plan: run make
      
      Reviewers: dduvall, #release-engineering-team
      
      Reviewed By: dduvall, #release-engineering-team
      
      Tags: #release-engineering-team
      
      Differential Revision: https://phabricator.wikimedia.org/D998
      b790283b
    • Dduvall's avatar
      Generalize instructions for entrypoint and working directory · 0902d688
      Dduvall authored
      Summary:
      Introduce new `build.EntryPoint` and `build.WorkingDirectory`
      instructions to allow configuration to inject them instead of hard
      coding their generation in the Docker compiler.
      
      Simplified the Docker compiler to simply iterate over build phases as
      returned by a new function `build.Phases()`.
      
      Depends on D990
      
      Test Plan: Run `go test ./...`.
      
      Reviewers: thcipriani, demon, hashar, #release-engineering-team
      
      Reviewed By: thcipriani, #release-engineering-team
      
      Tags: #release-engineering-team
      
      Differential Revision: https://phabricator.wikimedia.org/D991
      0902d688
    • Dduvall's avatar
      Simplify Docker instruction compilation · 8fa191f0
      Dduvall authored
      Summary:
      After adding a number of new instructions, it seemed there was a lot of
      redundant implementation between concrete types. This refactor uses just
      one concrete unexported type and a few options for determining the
      output format instead of one struct type per possible Docker
      instruction.
      
      The exported `docker.Instruction` interface and signature of
      `docker.NewInstruction` is unchanged with this refactor. Unit tests for
      Docker instruction compilation were simplified to not make type
      assertions but were otherwise left to ensure no regression.
      
      Depends on D984
      
      Test Plan: Run `go test ./...`.
      
      Reviewers: thcipriani, demon, hashar, mmodell, #release-engineering-team
      
      Reviewed By: thcipriani, #release-engineering-team
      
      Tags: #release-engineering-team
      
      Differential Revision: https://phabricator.wikimedia.org/D990
      8fa191f0