README
¶
Got.
Simple and fast concurrent downloader.
Installation ❘ CLI Usage ❘ Module Usage ❘ License
Comparison
Comparison in cloud server:
[root@centos-nyc-12 ~]# time got -o /tmp/test -c 20 http://www.ovh.net/files/1Gio.dat
URL: http://www.ovh.net/files/1Gio.dat done!
real 0m8.832s
user 0m0.203s
sys 0m3.176s
[root@centos-nyc-12 ~]# time curl http://www.ovh.net/files/1Gio.dat --output /tmp/test1
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1024M 100 1024M 0 0 35.6M 0 0:00:28 0:00:28 --:--:-- 34.4M
real 0m28.781s
user 0m0.379s
sys 0m1.970s
Installation
Download and install the latest release:
# go to tmp dir.
cd /tmp
# Download latest version.
curl -sfL https://git.io/getgot | sh
# Make the binary executable.
chmod +x /tmp/bin/got
# Move the binary to your PATH
sudo mv /tmp/bin/got /usr/bin/got
Or Go ahead compile it yourself:
go get github.com/melbahja/got/cmd/got
Or from the AUR
Install got for the latest release version or got-git for the latest development version.
Note: these packages are not maintained by melbahja
Command Line Tool Usage
Simple usage:
got https://example.com/file.mp4
You can specify destination path:
got -o /path/to/save https://example.com/file.mp4
You can download multiple URLs and save them to directory:
got --dir /path/to/dir https://example.com/file.mp4 https://example.com/file2.mp4
You can download multiple URLs from a file:
got --dir /path/to/dir -f urls.txt
You can pipe multiple URLs:
cat urls.txt | got --dir /path/to/dir
Docs for available flags:
got help
Module Usage
You can use Got to download large files in your go code, the usage is simple as the CLI tool:
package main
import "github.com/melbahja/got"
func main() {
g := got.New()
err := g.Download("http://localhost/file.ext", "/path/to/save")
if err != nil {
// ..
}
}
For more see GoDocs.
How It Works?
Got takes advantage of the HTTP range requests support in servers RFC 7233, if the requested URL server supports partial content Got split the file into chunks, then starts downloading and merging them into the destinaton file concurrently.
License
Got is provided under the MIT License © Mohammed El Bahja.
Documentation
¶
Index ¶
- Constants
- Variables
- func GetDefaultClient() *http.Client
- func GetFilename(URL string) string
- func NewRequest(ctx context.Context, method, URL string, options ...func(*http.Request)) (req *http.Request, err error)
- type Chunk
- type Download
- func (d Download) AvgSpeed() uint64
- func (d Download) Context() context.Context
- func (d *Download) DownloadChunk(ctx context.Context, c Chunk, dest *os.File) error
- func (d Download) GetInfo() (size uint64, rangeable bool, err error)
- func (d *Download) Init() (err error)
- func (d Download) IsRangeable() bool
- func (d *Download) RunProgress(fn ProgressFunc)
- func (d Download) SetOptions(opts ...func(req *http.Request))
- func (d Download) Size() uint64
- func (d Download) Speed() uint64
- func (d *Download) Start() error
- func (d Download) TotalCost() time.Duration
- func (d Download) TotalSize() uint64
- func (d *Download) Write(b []byte) (int, error)
- type Got
- type ProgressFunc
Examples ¶
Constants ¶
const DefaulUserAgent = "Got/1.0"
DefaulUserAgent is the default Got user agent to send http requests.
Variables ¶
var ChunkPool = &sync.Pool{ New: func() interface{} { return new(Chunk) }, }
ChunkPool helps in multi *Download files.
var DefaultFileName = "got.output"
DefaultFileName is the fallback name for GetFilename.
var ErrDownloadAborted = errors.New("Operation aborted")
ErrDownloadAborted - When download is aborted by the OS before it is completed, ErrDownloadAborted will be triggered
Functions ¶
func GetDefaultClient ¶
GetDefaultClient returns Got default http.Client
func GetFilename ¶
GetFilename it returns default file name from a URL.
Types ¶
type Chunk ¶
type Chunk struct {
// Chunk start pos.
Start uint64
// Chunk end pos.
End uint64
// Path name where this chunk downloaded.
Path string
// Done to check is this chunk downloaded.
Done chan struct{}
}
Chunk is a partial content range.
type Download ¶
type Download struct {
Client *http.Client
Concurrency uint
URL, Dest string
Interval, ChunkSize, MinChunkSize, MaxChunkSize uint64
StopProgress bool
RequestOptions []func(*http.Request)
// contains filtered or unexported fields
}
Download holds downloadable file config and infos.
Example ¶
// Just for testing
destPath := createTemp()
defer clean(destPath)
ctx := context.Background()
dl := got.NewDownload(ctx, testUrl, destPath)
// Init
if err := dl.Init(); err != nil {
fmt.Println(err)
}
// Start download
if err := dl.Start(); err != nil {
fmt.Println(err)
}
fmt.Println("Done")
Output: Done
func NewDownload ¶
NewDownload returns new *Download with context.
func (*Download) DownloadChunk ¶
DownloadChunk downloads a file chunk.
func (*Download) Init ¶
Init set defaults and split file into chunks and gets Info, you should call Init before Start
func (Download) IsRangeable ¶
IsRangeable returns file server partial content support state.
func (*Download) RunProgress ¶
func (d *Download) RunProgress(fn ProgressFunc)
RunProgress runs ProgressFunc based on Interval and updates lastSize.
func (Download) SetOptions ¶
SetOptions set http.Request options
type Got ¶
type Got struct {
ProgressFunc
Client *http.Client
// contains filtered or unexported fields
}
Got holds got download config.
Example ¶
// Just for testing
destPath := createTemp()
defer clean(destPath)
g := got.New()
err := g.Download(testUrl, destPath)
if err != nil {
log.Fatal(err)
return
}
fmt.Println("done")
Output: done
Example (WithContext) ¶
// Just for testing
destPath := createTemp()
defer clean(destPath)
ctx := context.Background()
g := got.NewWithContext(ctx)
err := g.Download(testUrl, destPath)
if err != nil {
log.Fatal(err)
return
}
fmt.Println("done")
Output: done
Example (WithContext_withOption) ¶
// Just for testing
destPath := createTemp()
defer clean(destPath)
ctx := context.Background()
// add some referer
referer := func(req *http.Request) {
req.Header.Set("Referer", "http://127.0.0.1/")
}
g := got.NewWithContextWithReqOptions(ctx, referer)
err := g.Download(testUrl, destPath)
if err != nil {
log.Fatal(err)
return
}
fmt.Println("done")
Output: done
func NewWithContext ¶
NewWithContext wants Context and returns *Got with default http client.
type ProgressFunc ¶
type ProgressFunc func(d *Download)
ProgressFunc to show progress state, called by RunProgress based on interval.
Source Files
Directories