blob_stores

package
v0.0.0-...-ed7c31d Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Feb 28, 2026 License: MIT Imports: 40 Imported by: 0

Documentation

Index

Constants

View Source
const (
	CopyResultStateUnknown = copyResultState(iota)
	CopyResultStateSuccess
	CopyResultStateNilRemoteBlobStore
	CopyResultStateMissingLocally
	CopyResultStateExistsLocally
	CopyResultStateExistsLocallyAndRemotely
	CopyResultStateError
)

Variables

This section is empty.

Functions

func MakeBlobStore

func MakeBlobStore(
	envDir env_dir.Env,
	configNamed blob_store_configs.ConfigNamed,
	blobStores BlobStoreMap,
) (store domain_interfaces.BlobStore, err error)

NOTE: blobStores parameter added to support inventory archive's loose-blob-store-id resolution. This couples MakeBlobStore to the store map, which may not scale well if more store types need cross-references. If this becomes a problem, switch to two-pass initialization: first pass creates all stores without cross-refs, second pass wires them up.

TODO describe base path agnostically

func MakeSSHAgent

func MakeSSHAgent(
	ctx interfaces.ActiveContext,
	uiPrinter ui.Printer,
) (sshAgent agent.ExtendedAgent, err error)

func MakeSSHClientForExplicitConfig

func MakeSSHClientForExplicitConfig(
	ctx interfaces.ActiveContext,
	uiPrinter ui.Printer,
	config blob_store_configs.ConfigSFTPConfigExplicit,
) (sshClient *ssh.Client, err error)

TODO refactor `blob_store_configs.ConfigSFTP` for ssh-client-specific methods

func MakeSSHClientFromSSHConfig

func MakeSSHClientFromSSHConfig(
	ctx interfaces.ActiveContext,
	uiPrinter ui.Printer,
	config blob_store_configs.ConfigSFTPUri,
) (sshClient *ssh.Client, err error)

func VerifyBlob

func VerifyBlob(
	ctx errors.Context,
	blobStore domain_interfaces.BlobStore,
	expected domain_interfaces.MarklId,
	progressWriter io.Writer,
) (err error)

TODO offer options like just checking the existence of the blob, getting its size, or full verification

Types

type ArchiveIndex

type ArchiveIndex interface {
	AllArchiveEntryChecksums() map[string][]string // archiveChecksum -> []blobIdString
}

ArchiveIndex is implemented by blob stores backed by archive files. It exposes the in-memory index for listing archives and their blob IDs.

type BlobDeleter

type BlobDeleter interface {
	DeleteBlob(id domain_interfaces.MarklId) error
}

BlobDeleter is implemented by blob stores that support removing individual blobs by their content address. Used by Pack to delete loose blobs after they have been safely written to an archive.

type BlobStoreInitialized

type BlobStoreInitialized struct {
	blob_store_configs.ConfigNamed
	domain_interfaces.BlobStore
}

func MakeRemoteBlobStore

func MakeRemoteBlobStore(
	envDir env_dir.Env,
	configNamed blob_store_configs.ConfigNamed,
) (blobStore BlobStoreInitialized)

func (BlobStoreInitialized) GetBlobStore

func (blobStoreInitialized BlobStoreInitialized) GetBlobStore() domain_interfaces.BlobStore

type BlobStoreMap

type BlobStoreMap = map[string]BlobStoreInitialized

func MakeBlobStoreMap

func MakeBlobStoreMap(blobStores ...BlobStoreInitialized) BlobStoreMap

func MakeBlobStores

func MakeBlobStores(
	ctx interfaces.ActiveContext,
	envDir env_dir.Env,
	directoryLayout directory_layout.BlobStore,
) (blobStores BlobStoreMap)

TODO pass in custom UI context for printing TODO consolidated envDir and ctx arguments

type CopyResult

type CopyResult struct {
	BlobId domain_interfaces.MarklId // may not be nil
	// contains filtered or unexported fields
}

func CopyBlobIfNecessary

func CopyBlobIfNecessary(
	env env_ui.Env,
	dst domain_interfaces.BlobStore,
	src domain_interfaces.BlobStore,
	expectedDigest domain_interfaces.MarklId,
	extraWriter io.Writer,
	hashType domain_interfaces.FormatHash,
) (copyResult CopyResult)

func CopyReaderToWriter

func CopyReaderToWriter(
	ctx errors.Context,
	dst domain_interfaces.BlobWriter,
	src io.Reader,
	expected domain_interfaces.MarklId,
	extraWriter io.Writer,
	heartbeats func(time time.Time),
	pulse time.Duration,
) (copyResult CopyResult)

func (CopyResult) Exists

func (copyResult CopyResult) Exists() bool

func (CopyResult) GetBytesWrittenAndState

func (copyResult CopyResult) GetBytesWrittenAndState() (int64, CopyResultState)

func (CopyResult) GetError

func (copyResult CopyResult) GetError() error

func (CopyResult) IsError

func (copyResult CopyResult) IsError() bool

func (CopyResult) IsMissing

func (copyResult CopyResult) IsMissing() bool

func (*CopyResult) SetBlobExistsLocally

func (copyResult *CopyResult) SetBlobExistsLocally()

func (*CopyResult) SetBlobExistsLocallyAndRemotely

func (copyResult *CopyResult) SetBlobExistsLocallyAndRemotely()

func (*CopyResult) SetBlobMissingLocally

func (copyResult *CopyResult) SetBlobMissingLocally()

func (*CopyResult) SetError

func (copyResult *CopyResult) SetError(err error)

func (CopyResult) String

func (copyResult CopyResult) String() string

type CopyResultState

type CopyResultState interface {
	// contains filtered or unexported methods
}

type DeletionPrecondition

type DeletionPrecondition interface {
	CheckBlobsSafeToDelete(
		blobs interfaces.SeqError[domain_interfaces.MarklId],
	) error
}

DeletionPrecondition checks whether blobs are safe to delete from the loose store. The default implementation always returns nil (safe). Future implementations can verify off-host replication before allowing deletion.

func NopDeletionPrecondition

func NopDeletionPrecondition() DeletionPrecondition

type Multi

type Multi struct {
	// contains filtered or unexported fields
}

func (Multi) HasBlob

func (parentStore Multi) HasBlob(id domain_interfaces.MarklId) bool

func (Multi) MakeBlobReader

func (parentStore Multi) MakeBlobReader(
	id domain_interfaces.MarklId,
) (domain_interfaces.BlobReader, error)

func (Multi) MakeBlobWriter

func (parentStore Multi) MakeBlobWriter(
	marklHashType domain_interfaces.FormatHash,
) (domain_interfaces.BlobWriter, error)

type PackOptions

type PackOptions struct {
	// Context supports cancellation of the pack operation via signals or
	// memory-exhaustion monitoring. When nil, packing runs without
	// cancellation support.
	Context interfaces.ActiveContext

	// DeleteLoose causes loose blobs to be deleted after they have been
	// packed into the archive and the archive has been validated.
	DeleteLoose bool

	// DeletionPrecondition is checked before any loose blobs are deleted.
	// When nil, deletion proceeds without additional checks.
	DeletionPrecondition DeletionPrecondition

	// BlobFilter restricts packing to only the specified blob IDs. When nil,
	// all loose blobs not yet in the archive are packed.
	BlobFilter map[string]domain_interfaces.MarklId

	// MaxPackSize overrides the configured max pack size when non-zero.
	MaxPackSize uint64

	// SkipMissingBlobs causes unreadable loose blobs to be skipped with a
	// TAP comment instead of aborting the pack. When false, an unreadable
	// blob emits a not-ok test point and stops packing.
	SkipMissingBlobs bool

	// Delta enables delta compression during packing.
	Delta bool

	// TapWriter emits phase-level TAP test points during packing. When nil,
	// packing is silent (backward compatible for unit tests).
	TapWriter *tap.Writer
}

PackOptions controls the behavior of the Pack operation.

type PackableArchive

type PackableArchive interface {
	Pack(options PackOptions) error
}

PackableArchive is implemented by blob stores that support packing loose blobs into archive files.

Source Files

  • archive_index.go
  • blob_deleter.go
  • copy.go
  • copy_result.go
  • copy_result_state.go
  • copyresultstate_string.go
  • deletion_precondition.go
  • initialized.go
  • main.go
  • multi.go
  • pack.go
  • pack_parallel.go
  • pack_v0.go
  • pack_v1.go
  • store_inventory_archive.go
  • store_inventory_archive_v1.go
  • store_local_hash_bucketed.go
  • store_remote_sftp.go
  • util_local.go
  • util_ssh.go
  • verification.go

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL