Limiting bandwidth of http get

Question

I'm a beginner to golang.

Is there any way to limit golang's http.Get() bandwidth usage? I found this: http://godoc.org/code.google.com/p/mxk/go1/flowcontrol, but I'm not sure how to piece the two together. How would I get access to the http Reader?

score 15 · Answer 1 · answered Jan 10 '15 at 16:55

Thirdparty packages have convenient wrappers. But if you interested in how things work under the hood - it's quite easy.

package main

import (
    "io"
    "net/http"
    "os"
    "time"
)

var datachunk int64 = 500       //Bytes
var timelapse time.Duration = 1 //per seconds

func main() {
    responce, _ := http.Get("http://google.com")
    for range time.Tick(timelapse * time.Second) {
        _, err :=io.CopyN(os.Stdout, responce.Body, datachunk)
        if err!=nil {break}
    }
}

Nothing magic.

score 5 · Accepted Answer · answered Jan 10 '15 at 00:43

5

There is an updated version of the package on github

You use it by wrapping an io.Reader

Here is a complete example which will show the homepage of Google veeeery sloooowly.

This wrapping an interface to make new functionality is very good Go style, and you'll see a lot of it in your journey into Go.

package main

import (
    "io"
    "log"
    "net/http"
    "os"

    "github.com/mxk/go-flowrate/flowrate"
)

func main() {
    resp, err := http.Get("http://google.com")
    if err != nil {
        log.Fatalf("Get failed: %v", err)
    }
    defer resp.Body.Close()

    // Limit to 10 bytes per second
    wrappedIn := flowrate.NewReader(resp.Body, 10)

    // Copy to stdout
    _, err = io.Copy(os.Stdout, wrappedIn)
    if err != nil {
        log.Fatalf("Copy failed: %v", err)
    }
}

answered Jan 10 '15 at 00:43

Nick Craig-Wood

48,112
10
112
118

Looks great. What about checkin if a limit has been reached such as 5MB? Would I just use b:=make([]byte, 5000000); wrappedIn.Read(b);? – John Jan 10 '15 at 03:01
1

No you would wrap the `io.Reader` in an [io.LimitedReader](http://golang.org/pkg/io/#LimitedReader). – Nick Craig-Wood Jan 10 '15 at 13:48
2

This limits the speed at which you read from the kernel's TCP buffer, not directly the speed at which you read from remote. The local TCP buffer will fill, the network will be idle while you read (slowly) from the buffer, then it will burst again to refill the buffer. For a large download you're averaging 10 bytes/sec from remote, but for a small download you're just consuming it slowly, with no difference in network activity. To limit net activity see https://unix.stackexchange.com/questions/28198/how-to-limit-network-bandwidth – Graham King Nov 14 '17 at 18:25

Limiting bandwidth of http get

2 Answers2

Linked