repository enhancements

While the support is fairly recent, I wanted to highlight some of the recent portage improvements for repository supported that were released in/before These are configurable per repository via modifying metadata/layout.conf for that repository.

  • thin-manifests = [ true | false ]
    Defaults to false if unspecified.
    For repositories that are distributed via git, the VCS provides via it’s sha1 internals guarantees of the content. This means the non-distfile manifest checksums are redundant; if enabled, this disables non-distfile validation and turns off generation of those checksums when creating manifests. Pretty much if you’re got a git vcs repo, you likely want this enabled unless you’re paranoid about someone having a sha1 pre-image attack they’re sitting on.
  • use-manifests = [ false | true | strict ]
    Defaults to strict if unspecified.
    This provides per repository control as to whether manifests should be used/generated; if set to false, manifest usage and generation is disabled for that repository.
    If set to true, this directs the package manager to use manifest data if available, but to not consider it a failure if a manifest file is missing. Additionally, if set to true, the package manager will generate manifests by default. This mode is primarily of use for migrating a repository that lacked manifests, to using/requiring manifests.
    Finally, if set to strict, manifests are generated/required and it’s considered a failure if the data isn’t available. Generally speaking, there rarely is any reason to set this option to anything other than strict.
  • cache-format = [ pms | md5-dict ]
    Defaults to pms.
    As it sounds- this is a directive which cache format this repository uses for any pregenerated cache distributed with the tree. Currently there are two formats; pms, the standard metadata/cache format that has the following restrictions- should not be used for any repository that has specified masters, and cannot be used if the repository is distributed in a fashion that doesn’t preserve mtime (git for example, doesn’t preserve mtime). Not a great cache format frankly, but for where it’s usable it suffices and is well supported; main limitation is that it has no real validation built into it beyond an mtime check of the cache file in comparison to the ebuild; as such, it’s impossible to validate if the eclass has changed since thus precluding from using it in an overlay/repository w/ master setup. As said, it’s historical, works well for the main repository but has definite flaws.
    The new kid on the block is md5-dict, which is a bit of a hack, but has useful properties. It enabled, it lives at metadata/md5-cache; it’s basically a flat_hash cache (the format used at /var/cache/edb/dep/*), just using md5 rather than mtime for validation. Specifically, this means you can distribute this cache via git, and means you can safely use it for overlays/repositories with masters specified; it carries enough validation information to detect if the cache entry is stale, in which case the manager regenerates as necessary.
    Down the line, I intend to design a format explicitly optimized for pregenerated usage- both reducing the inode requirements of existing pregenerated caches, and speeding it up. In the interim, md5-dict is probably what you’re after unless we’re talking about the literal gentoo repository itself (which must remain pms format).

I’d like to note that this functionality was contributed by the chromium-os project, one of the multiple gentoo derivatives; in addition, cookies should be sent to Zac for cleanup/fixing of the cache-format functionality (the refactoring enabling it was a bit touchy getting right).

Beyond those features, sign-manifests was added (unless you have a good reason, leave it enabled), and manifest-hashes was added (again, unless you have a good reason, leave it alone right now). I expect in the next week or two for an additional feature to appear to explicitly mark a repository if it’s using PMS incompliant package.mask as a directory.

At this point, as stated this functionality is in portage; for pkgcore, thin-manifests are supported, and the rest will be addressed in the next release or two.


4 Responses to repository enhancements

  1. dleverton says:

    I don’t much like “cache-format = pms” – seems to imply that PMS will never specify any other format than the current one. Maybe calling it flat_list would make more sense, since it’s already called that internally?

  2. ferringb says:

    The semantics of what a pms cache is shouldn’t require people to know that it’s flat_list, or flat_hash, or etc. Should just call it pms, pms-1, pms-2, etc, which is why the naming is that way.

  3. swan says:

    use-manifests=true is useful for what we’re doing

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: