diff options
Diffstat (limited to 'content/versioning-proposal.article')
| -rw-r--r-- | content/versioning-proposal.article | 289 |
1 files changed, 289 insertions, 0 deletions
diff --git a/content/versioning-proposal.article b/content/versioning-proposal.article new file mode 100644 index 0000000..20c4426 --- /dev/null +++ b/content/versioning-proposal.article @@ -0,0 +1,289 @@ +A Proposal for Package Versioning in Go +26 Mar 2018 +Tags: tools, versioning + +Russ Cox + +* Introduction + +Eight years ago, the Go team introduced `goinstall` +(which led to `go`get`) +and with it the decentralized, URL-like import paths +that Go developers are familiar with today. +After we released `goinstall`, one of the first questions people asked +was how to incorporate version information. +We admitted we didn’t know. +For a long time, we believed that the problem of package versioning +would be best solved by an add-on tool, +and we encouraged people to create one. +The Go community created many tools with different approaches. +Each one helped us all better understand the problem, +but by mid-2015 it was clear that there were now too many solutions. +We needed to adopt a single, official tool. + +After a community discussion started at GopherCon in July 2015 and continuing into the fall, +we all believed the answer would be to follow the package versioning approach +exemplified by Rust’s Cargo, with tagged semantic versions, +a manifest, a lock file, and a +[[https://research.swtch.com/version-sat][SAT solver]] to decide which versions to use. +Sam Boyer led a team to create Dep, which followed this rough plan, +and which we intended to serve as the model for `go` command integration. +But as we learned more about the implications of the Cargo/Dep approach, +it became clear to me that Go would benefit from changing +some of the details, especially concerning backwards compatibility. + +* The Impact of Compatibility + +The most important new feature of +[[https://blog.golang.org/preview-of-go-version-1][Go 1]] +was not a language feature. +It was Go 1’s emphasis on backwards compatibility. +Until that point we’d issued stable release +snapshots approximately monthly, +each with significant incompatible changes. +We observed significant acceleration in interest and adoption +immediately after the release of Go 1. +We believe that the +[[https://golang.org/doc/go1compat.html][promise of compatibility]] +made developers feel much more comfortable relying on +Go for production use +and is a key reason that Go is popular today. +Since 2013 the +[[https://golang.org/doc/faq][Go FAQ]] +has encouraged package developers to provide their own +users with similar expectations of compatibility. +We call this the _import_compatibility_rule_: +“If an old package and a new package have the same import path, +the new package must be backwards compatible with the old package.” + +Independently, +[[http://semver.org/][semantic versioning]] +has become the _de_facto_ +standard for describing software versions in many language communities, +including the Go community. +Using semantic versioning, later versions are expected to be +backwards-compatible with earlier versions, +but only within a single major version: +v1.2.3 must be compatible with v1.2.1 and v1.1.5, +but v2.3.4 need not be compatible with any of those. + +If we adopt semantic versioning for Go packages, +as most Go developers expect, +then the import compatibility rule requires that +different major versions must use different import paths. +This observation led us to _semantic_import_versioning_, +in which versions starting at v2.0.0 include the major +version in the import path: `my/thing/v2/sub/pkg`. + +A year ago I strongly believed that whether to include +version numbers in import paths was largely a matter of taste, +and I was skeptical that having them was particularly elegant. +But the decision turns out to be a matter not of taste but of logic: +import compatibility and semantic versioning together require +semantic import versioning. +When I realized this, the logical necessity surprised me. + +I was also surprised to realize that +there is a second, independent logical route to +semantic import versioning: +[[https://talks.golang.org/2016/refactor.article][gradual code repair]] +or partial code upgrades. +In a large program, it’s unrealistic to expect all packages in the program +to update from v1 to v2 of a particular dependency at the same time. +Instead, it must be possible for some of the program to keep using v1 +while other parts have upgraded to v2. +But then the program’s build, and the program’s final binary, +must include both v1 and v2 of the dependency. +Giving them the same import path would lead to confusion, +violating what we might call the _import_uniqueness_rule_: +different packages must have different import paths. +The only way to have +partial code upgrades, import uniqueness, _and_ semantic versioning +is to adopt +semantic import versioning as well. + +It is of course possible to build systems that use semantic versioning +without semantic import versioning, +but only by giving up either partial code upgrades or import uniqueness. +Cargo allows partial code upgrades by +giving up import uniqueness: +a given import path can have different meanings +in different parts of a large build. +Dep ensures import uniqueness by +giving up partial code upgrades: +all packages involved in a large build must find +a single agreed-upon version of a given dependency, +raising the possibility that large programs will be unbuildable. +Cargo is right to insist on partial code upgrades, +which are critical to large-scale software development. +Dep is equally right to insist on import uniqueness. +Complex uses of Go’s current vendoring support can violate import uniqueness. +When they have, the resulting problems have been quite challenging +for both developers and tools to understand. +Deciding between partial code upgrades +and import uniqueness +requires predicting which will hurt more to give up. +Semantic import versioning lets us avoid the choice +and keep both instead. + +I was also surprised to discover how much +import compatibility simplifies version selection, +which is the problem of deciding which package versions to use for a given build. +The constraints of Cargo and Dep make version selection +equivalent to +[[https://research.swtch.com/version-sat][solving Boolean satisfiability]], +meaning it can be very expensive to determine whether +a valid version configuration even exists. +And then there may be many valid configurations, +with no clear criteria for choosing the “best” one. +Relying on import compatibility can instead let Go use +a trivial, linear-time algorithm +to find the single best configuration, which always exists. +This algorithm, +which I call +[[https://research.swtch.com/vgo-mvs][_minimal_version_selection_]], +in turn eliminates the need for separate lock and manifest files. +It replaces them with a single, short configuration file, +edited directly by both developers and tools, +that still supports reproducible builds. + +Our experience with Dep demonstrates the impact of compatibility. +Following the lead of Cargo and earlier systems, +we designed Dep to give up import compatibility +as part of adopting semantic versioning. +I don’t believe we decided this deliberately; +we just followed those other systems. +The first-hand experience of using Dep helped us +better understand exactly how much complexity +is created by permitting incompatible import paths. +Reviving the import compatibility rule +by introducing semantic import versioning +eliminates that complexity, +leading to a much simpler system. + +* Progress, a Prototype, and a Proposal + +Dep was released in January 2017. +Its basic model—code tagged with +semantic versions, along with a configuration file that +specified dependency requirements—was +a clear step forward from most of the Go vendoring tools, +and converging on Dep itself was also a clear step forward. +I wholeheartedly encouraged its adoption, +especially to help developers get used to thinking about Go package versions, +both for their own code and their dependencies. +While Dep was clearly moving us in the right direction, I had lingering concerns +about the complexity devil in the details. +I was particularly concerned about Dep +lacking support for gradual code upgrades in large programs. +Over the course of 2017, I talked to many people, +including Sam Boyer and the rest of the +package management working group, +but none of us could see any clear way to reduce the complexity. +(I did find many approaches that added to it.) +Approaching the end of the year, +it still seemed like SAT solvers and unsatisfiable builds +might be the best we could do. + +In mid-November, trying once again to work through +how Dep could support gradual code upgrades, +I realized that our old advice about import compatibility +implied semantic import versioning. +That seemed like a real breakthrough. +I wrote a first draft of what became my +[[https://research.swtch.com/vgo-import][semantic import versioning]] +blog post, +concluding it by suggesting that Dep adopt the convention. +I sent the draft to the people I’d been talking to, +and it elicited very strong responses: +everyone loved it or hated it. +I realized that I needed to work out more of the +implications of semantic import versioning +before circulating the idea further, +and I set out to do that. + +In mid-December, I discovered that import compatibility +and semantic import versioning together allowed +cutting version selection down to [[https://research.swtch.com/vgo-mvs][minimal version selection]]. +I wrote a basic implementation to be sure I understood it, +I spent a while learning the theory behind why it was so simple, +and I wrote a draft of the post describing it. +Even so, I still wasn’t sure the approach would be practical +in a real tool like Dep. +It was clear that a prototype was needed. + +In January, I started work on a simple `go` command wrapper +that implemented semantic import versioning +and minimal version selection. +Trivial tests worked well. +Approaching the end of the month, +my simple wrapper could build Dep, +a real program that made use of many versioned packages. +The wrapper still had no command-line interface—the fact that +it was building Dep was hard-coded in a few string constants—but +the approach was clearly viable. + +I spent the first three weeks of February turning the +wrapper into a full versioned `go` command, `vgo`; +writing drafts of a +[[https://research.swtch.com/vgo][blog post series introducing `vgo`]]; +and discussing them with +Sam Boyer, the package management working group, +and the Go team. +And then I spent the last week of February finally +sharing `vgo` and the ideas behind it with the whole Go community. + +In addition to the core ideas of import compatibility, +semantic import versioning, and minimal version selection, +the `vgo` prototype introduces a number of smaller +but significant changes motivated by eight years of +experience with `goinstall` and `go`get`: +the new concept of a [[https://research.swtch.com/vgo-module][Go module]], +which is a collection of packages versioned as a unit; +[[https://research.swtch.com/vgo-repro][verifiable and verified builds]]; +and +[[https://research.swtch.com/vgo-cmd][version-awareness throughout the `go` command]], +enabling work outside `$GOPATH` +and the elimination of (most) `vendor` directories. + +The result of all of this is the [[https://golang.org/design/24301-versioned-go][official Go proposal]], +which I filed last week. +Even though it might look like a complete implementation, +it’s still just a prototype, +one that we will all need to work together to complete. +You can download and try the `vgo` prototype from [[golang.org/x/vgo][https://golang.org/x/vgo]], +and you can read the +[[https://research.swtch.com/vgo-tour][Tour of Versioned Go]] +to get a sense of what using `vgo` is like. + +* The Path Forward + +The proposal I filed last week is exactly that: an initial proposal. +I know there are problems with it that the Go team and I can’t see, +because Go developers use Go in many clever ways that we don’t know about. +The goal of the proposal feedback process is for us all to work together +to identify and address the problems in the current proposal, +to make sure that the final implementation that ships in a future +Go release works well for as many developers as possible. +Please point out problems on the [[https://golang.org/issue/24301][proposal discussion issue]]. +I will keep the +[[https://golang.org/issue/24301#issuecomment-371228742][discussion summary]] +and +[[https://golang.org/issue/24301#issuecomment-371228664][FAQ]] +updated as feedback arrives. + +For this proposal to succeed, the Go ecosystem as a +whole—and in particular today’s major Go projects—will need to +adopt the import compatibility rule and semantic import versioning. +To make sure that can happen smoothly, +we will also be conducting user feedback sessions +by video conference with projects that have questions about +how to incorporate the new versioning proposal into their code bases +or have feedback about their experiences. +If you are interested in participating in such a session, +please email Steve Francia at spf@golang.org. + +We’re looking forward to (finally!) providing the Go community with a single, official answer +to the question of how to incorporate package versioning into `go`get`. +Thanks to everyone who helped us get this far, and to everyone who will help us going forward. +We hope that, with your help, we can ship something that Go developers will love. |