Documentation
¶
Index ¶
- Variables
- func DisableExtension(name string) error
- func EnableExtension(name string) error
- func GenerateProfile(tablePath string) error
- func InitConfig(configFile string) error
- func InstallExtension(name string) error
- func IsExtensionEnabled(name string) bool
- func RunQualityChecks(tablePath string) error
- func StartServer(host string, port int) error
- func UninstallExtension(name string) error
- func UpdateConfig(updates map[string]interface{}) error
- type Checkpoint
- type ColumnStats
- type Config
- type DeltaTable
- type Extension
- type ExtensionsConfig
- type ProfileStats
- type Profiler
- func (p *Profiler) GenerateProfile() (*TableProfile, error)
- func (p *Profiler) GetColumnProfile(columnName string) (*ProfileStats, error)
- func (p *Profiler) GetPartitionInfo() (map[string]string, error)
- func (p *Profiler) GetTableSize() (int64, error)
- func (p *Profiler) PrintProfile(profile *TableProfile)
- type Reader
- func (r *Reader) Close() error
- func (r *Reader) GetLastModified() time.Time
- func (r *Reader) GetPartitionSchema() []string
- func (r *Reader) GetPartitionStats(partition string) (int64, error)
- func (r *Reader) GetSchema() map[string]string
- func (r *Reader) GetStats() *TableStats
- func (r *Reader) GetTablePath() string
- func (r *Reader) GetVersion() int64
- func (r *Reader) ListPartitions() ([]string, error)
- func (r *Reader) ReadAll() (io.ReadCloser, error)
- func (r *Reader) ReadAllParsed() ([]map[string]interface{}, error)
- func (r *Reader) ReadPartition(partition string) (io.ReadCloser, error)
- type Server
- type ServerConfig
- type TableProfile
- type TableStats
- type TelemetryConfig
- type TelemetryEvent
- type TelemetryManager
Constants ¶
This section is empty.
Variables ¶
var Telemetry = &TelemetryManager{Enabled: false, Endpoint: ""}
Helper: No-op singleton if not enabled
Functions ¶
func DisableExtension ¶
DisableExtension removes an extension from the enabled list
func EnableExtension ¶
EnableExtension adds an extension to the enabled list
func GenerateProfile ¶
GenerateProfile creates a profile for a Delta table
func InstallExtension ¶
InstallExtension installs a new extension
func IsExtensionEnabled ¶
IsExtensionEnabled checks if an extension is enabled
func RunQualityChecks ¶
RunQualityChecks performs data quality checks on a Delta table
func StartServer ¶
StartServer initializes and starts the API server
func UninstallExtension ¶
UninstallExtension removes an installed extension
func UpdateConfig ¶
UpdateConfig updates the configuration with new values
Types ¶
type Checkpoint ¶
type Checkpoint struct {
Version int64
Timestamp time.Time
FileCount int64
FilePaths []string
Schema map[string]string
Partitions []string
}
Checkpoint represents a Delta table checkpoint
type ColumnStats ¶
type ColumnStats struct {
NullCount int64
DistinctCount int64
MinValue interface{}
MaxValue interface{}
AvgValue float64
}
ColumnStats represents statistics for a single column
type Config ¶
type Config struct {
Server ServerConfig `mapstructure:"server"`
Extensions ExtensionsConfig `mapstructure:"extensions"`
Telemetry TelemetryConfig `mapstructure:"telemetry"`
}
Config represents the application configuration
type DeltaTable ¶
type DeltaTable struct {
Path string
Version int64
LastModified time.Time
PartitionSchema []string
Schema map[string]string
Stats *TableStats
LogFiles []string
Checkpoint *Checkpoint
}
DeltaTable represents a Delta table
type Extension ¶
Extension represents a nessi extension
func ListExtensions ¶
ListExtensions returns all available extensions
type ExtensionsConfig ¶
type ExtensionsConfig struct {
Enabled []string `mapstructure:"enabled"`
Path string `mapstructure:"path"`
}
ExtensionsConfig represents the extensions configuration
func GetExtensionsConfig ¶
func GetExtensionsConfig() ExtensionsConfig
GetExtensionsConfig returns the extensions configuration
type ProfileStats ¶
type ProfileStats struct {
ColumnName string
DataType string
TotalCount int64
NullCount int64
DistinctCount int64
MinValue interface{}
MaxValue interface{}
AvgValue float64
Histogram map[string]int64
}
ProfileStats represents the statistics for a column in a Delta table
type Profiler ¶
type Profiler struct {
// contains filtered or unexported fields
}
Profiler handles the profiling of Delta tables
func NewProfiler ¶
NewProfiler creates a new Profiler instance
func (*Profiler) GenerateProfile ¶
func (p *Profiler) GenerateProfile() (*TableProfile, error)
GenerateProfile creates a complete profile for the Delta table
func (*Profiler) GetColumnProfile ¶
func (p *Profiler) GetColumnProfile(columnName string) (*ProfileStats, error)
GetColumnProfile returns detailed statistics for a specific column
func (*Profiler) GetPartitionInfo ¶
GetPartitionInfo returns information about table partitions
func (*Profiler) GetTableSize ¶
GetTableSize returns the total size of the Delta table
func (*Profiler) PrintProfile ¶
func (p *Profiler) PrintProfile(profile *TableProfile)
PrintProfile prints the table profile in a human-readable format
type Reader ¶
type Reader struct {
// contains filtered or unexported fields
}
Reader handles reading from Delta tables
func (*Reader) GetLastModified ¶
GetLastModified returns the last modification time
func (*Reader) GetPartitionSchema ¶
GetPartitionSchema returns the partition schema
func (*Reader) GetPartitionStats ¶
GetPartitionStats returns statistics for a specific partition
func (*Reader) GetStats ¶
func (r *Reader) GetStats() *TableStats
GetStats returns the table statistics
func (*Reader) GetTablePath ¶
GetTablePath returns the path of the table being read.
func (*Reader) GetVersion ¶
GetVersion returns the current table version
func (*Reader) ListPartitions ¶
ListPartitions returns all available partitions
func (*Reader) ReadAll ¶
func (r *Reader) ReadAll() (io.ReadCloser, error)
ReadAll reads all data from the table
func (*Reader) ReadAllParsed ¶
ReadAllParsed reads all data from the table and parses it into a structured format
func (*Reader) ReadPartition ¶
func (r *Reader) ReadPartition(partition string) (io.ReadCloser, error)
ReadPartition reads data from a specific partition
type ServerConfig ¶
ServerConfig represents the server configuration
func GetServerConfig ¶
func GetServerConfig() ServerConfig
GetServerConfig returns the server configuration
type TableProfile ¶
type TableProfile struct {
TablePath string
TotalRows int64
TotalColumns int
LastModified time.Time
ColumnStats map[string]*ProfileStats
TableSize int64
PartitionInfo map[string]string
}
TableProfile represents the complete profile of a Delta table
type TableStats ¶
type TableStats struct {
NumFiles int64
NumRecords int64
TotalSize int64
PartitionCounts map[string]int64
ColumnStats map[string]*ColumnStats
}
TableStats represents statistics about a Delta table
type TelemetryConfig ¶
type TelemetryConfig struct {
Enabled bool `mapstructure:"enabled"`
Endpoint string `mapstructure:"endpoint"`
}
TelemetryConfig represents telemetry settings in config Add this to your main config struct
Telemetry TelemetryConfig `mapstructure:"telemetry"`
func GetTelemetryConfig ¶
func GetTelemetryConfig() TelemetryConfig
GetTelemetryConfig returns the telemetry configuration
type TelemetryEvent ¶
type TelemetryEvent struct {
Timestamp time.Time `json:"timestamp"`
Command string `json:"command"`
Args []string `json:"args"`
Success bool `json:"success"`
Error string `json:"error,omitempty"`
Version string `json:"version"`
}
TelemetryEvent represents a single telemetry event Extend fields as needed E.g., add Command, Args, Duration, Error, etc.
type TelemetryManager ¶
TelemetryManager handles sending telemetry events Usage: pkg.TelemetryManager.SendEvent(...)
func NewTelemetryManager ¶
func NewTelemetryManager(cfg TelemetryConfig) *TelemetryManager
NewTelemetryManager creates a TelemetryManager from config
func (*TelemetryManager) SendEvent ¶
func (tm *TelemetryManager) SendEvent(event TelemetryEvent)
SendEvent sends a telemetry event if enabled
Directories
¶
| Path | Synopsis |
|---|---|
|
api
|
|
|
aws
Package aws is a generated GoMock package.
|
Package aws is a generated GoMock package. |
|
Package datalake provides functionality for reading and writing Delta tables
|
Package datalake provides functionality for reading and writing Delta tables |
|
Package freshness provides functionality for tracking data freshness and SLA compliance.
|
Package freshness provides functionality for tracking data freshness and SLA compliance. |
|
example
command
|
|
|
Package security provides license validation and security features
|
Package security provides license validation and security features |