6

I'm in need of a dependency manager that is not tied to a particular language or build system. I've looked into several excellent tools (Gradle, Bazel, Hunter, Biicode, Conan, etc.), but none satisfy my requirements (see below). I've also used Git Submodules and Mercurial Subrepos.

My needs are well described in a presentation by Daniel Pfeifer at Meeting C++ 2014. To summarize the goals of this dependency tool (discussed @18:55 of the linked video):

  • Not just a package manager
  • Supports pre-built or source dependencies
  • Can download or find locally - no unnecessary downloads
  • Fetches using a variety of methods (i.e. download, or VCS clones, etc.)
  • Integrated with the system installer - can check if lib is installed
  • No need to adapt source code in any way
  • No need to adapt the build system
  • Cross-platform

Further requirements or clarifications I would add:

  • Suitable for third-party and/or versioned dependencies, but also capable of specifying non-versioned and/or co-developed dependencies (probably specified by a git/mercurial hash or tag).
  • Provides a mechanism to override the specified fetching behavior to use some alternate dependency version of my choosing.
  • No need to manually set up a dependency store. I'm not opposed to a central dependency location as a way to avoid redundant or circular dependencies. However, we need the simplicity of cloning a repo and executing some top-level build script that invokes the dependency manager and builds everything.
  • Despite the requirement that I should not have to modify my build system, obviously some top-level build must wield the dependency manager and then feed those dependencies to the individual builds. The requirement means that the individual builds should not be aware of the dependency manager. For example, if using CMake for a C++ package, I should not need to modify its CMakeLists.txt to make special functional calls to locate dependencies. Rather, the top-level build manager should invoke the dependency manager to retrieve the dependencies and then provide arguments CMake can consume in traditional ways (i.e find_package or add_subdirectory). In other words, I should always have the option of manually doing the work of the top-level build and dependency manager and the individual build should not know the difference.

Nice-to-have:

  • A way to interrogate the dependency manager after-the-fact to find where a dependency was placed. This would allow me to create VCS hooks to automatically update the hash in dependency metadata of co-developed source repo dependencies. (Like submodules or subrepos do).
Ken
  • 533
  • 6
  • 10
  • 1
    I think your requisites are too much for any C/C++ package manager, might be extremely difficult to achieve. Conan might be the closest one, provide several of them, and be close to others, but yes, it won't fully satisfy your needs as described. If you want more details or discuss features, just contact. – drodri Aug 22 '16 at 18:05
  • @drodri - Thanks. I'll contact directly. To reiterate, I'm looking for more than a C/C++ package manager. What I'd like is a dependency manager that can collect a heterogeneous set of dependencies. So the top-level build manager could be responsible for fetching, configuring, and building Go or Rust or Sphinx docs, etc. – Ken Aug 22 '16 at 23:03
  • I see. Just some pointers, as you talk about rust and go, just in case. Some conan-rust integration: http://blog.conan.io/2016/06/23/Rust-cargo-and-Conan-C_and_C++-package-manager-integration.html. How conan handle go-lang: http://docs.conan.io/en/latest/examples/go.html. – drodri Aug 22 '16 at 23:17
  • @drodri - Apparently I can't PM in SO - were you suggesting I contact Conan support? I'm still new to Conan. It looks great for package management, but not well suited to a larger dependency management concept. The clear non-starter: the requirement that users manually install package recipes and binaries in a local or remote store. This is appropriate for relatively static packages, but is unworkable if the dependency is a co-developed repo and the version or hash the dependent links to is updated frequently. This is beyond the intended scope of Conan, from what I can tell. – Ken Aug 23 '16 at 17:02
  • Github issues: https://github.com/conan-io/conan, Twitter: https://twitter.com/conan_io, email: info@conan.io. Also (rarely) freenode #conan channel. About the non-starter, there are already some users managing large systems (like >500 packages), having some performance issues: https://github.com/conan-io/conan/issues/245 – drodri Aug 23 '16 at 17:16

1 Answers1

5

After thoroughly searching the available technologies, comparing against package managers in various languages (i.e. npm), and even having a run at my own dependency manager tool, I have settled on Conan. After diving deep into Conan, I find that it satisfies most of my requirements out of the box and is readily extensible.

Prior to looking into Conan, I saw BitBake as the model of what I was looking for. However, it is linux only and is heavily geared toward embedded linux distros. Conan has essentially the same recipe features as bb and is truly cross-platform

Here are my requirements and what I found with Conan:

  • Not just a package manager
  • Supports pre-built or source dependencies

Conan supports classic release or dev dependencies and also allows you to package source. If binaries with particular configurations/settings do not exist in the registry (or "repository", in Conan parlance), a binary will be built from source.

  • Can download or find locally - no unnecessary downloads
  • Integrated with the system installer - can check if lib is installed

Conan maintains a local registry as a cache. So independent projects that happen to share dependencies don't need to redo expensive downloads and builds.

Conan does not prevent you from finding system packages instead of the declared dependencies. If you write your build script to be passed prefix paths, you can change the path of individual dependencies on the fly.

  • Fetches using a variety of methods (i.e. download, or VCS clones, etc.)

Implementing the source function of the recipe gives full control over how a dependency is fetched. Conan supports the recipes that do the download/clone of source or can "snapshot" the source, packaging it with the recipe itself.

  • No need to adapt source code in any way
  • No need to adapt the build system

Conan supports a variety of generators to make dependencies consumable by your chosen build system. The agnosticism from a particular build system is Conan's real win and ultimately what makes dependency management from the likes of Bazel, Buckaroo, etc. cumbersome.

  • Cross-platform Python. Check.

  • Suitable for third-party and/or versioned dependencies, but also capable of specifying non-versioned and/or co-developed dependencies (probably specified by a git/mercurial hash or tag).

Built with semver in mind, but can use any string identifier as version. Additionally has user and channel to act as namespaces for package versions.

  • Provides a mechanism to override the specified fetching behavior to use some alternate dependency version of my choosing.

You can prevent the fetch of a particular dependency by not including it in the install command. Or you can modify or override the generated prefix info to point to a different location on disk.

  • No need to manually set up a dependency store. I'm not opposed to a central dependency location as a way to avoid redundant or circular dependencies. However, we need the simplicity of cloning a repo and executing some top-level build script that invokes the dependency manager and builds everything. Despite the requirement that I should not have to modify my build system, obviously some top-level build must wield the dependency manager and then feed those dependencies to the individual builds. The requirement means that the individual builds should not be aware of the dependency manager. For example, if using CMake for a C++ package, I should not need to modify its CMakeLists.txt to make special functional calls to locate dependencies. Rather, the top-level build manager should invoke the dependency manager to retrieve the dependencies and then provide arguments CMake can consume in traditional ways (i.e find_package or add_subdirectory). In other words, I should always have the option of manually doing the work of the top-level build and dependency manager and the individual build should not know the difference.

Conan caches dependencies in a local registry. This is seamless. The canonical pattern you'll see in Conan's documentation is to add some Conan-specific calls in your build scripts, but this can be avoided. Once again, if you write your build scripts to consumer prefix paths and/or input arguments, you can pass the info in and not use Conan at all. I think the Conan CMake generators could use a little work to make this more elegant. As a fallback, Conan lets me write my own generator.

  • A way to interrogate the dependency manager after-the-fact to find where a dependency was placed. This would allow me to create VCS hooks to automatically update the hash in dependency metadata of co-developed source repo dependencies. (Like submodules or subrepos do).

The generators point to these locations. And with the full capability of Python, you can customize this to your heart's content.

Currently co-developing dependent projects is the biggest question mark for me. Meaning, I don't know if Conan has something out of the box to make tracking commits easy, but I'm confident the hooks are in there to add this customization.

Other things I found in Conan:

  • Conan provides the ability to download or build toolchains that I need during development. It uses Python virtualenv to make enabling/disabling these custom environments easy without polluting my system installations.
Ken
  • 533
  • 6
  • 10