go语言一些开源的package拾遗
Table of Contents
- 日志相关
- 解析命令行参数的工具 gopkg.in/alecthomas/kingpin.v2
- github.com/spf13/cobra 命令行相关
- 读取配置文件 相关(json ,yaml,环境变量,etcd,命令行参数)
- github.com/dgryski/go-jump
- github.com/pquerna/ffjson
- github.com/tools/godep
- github.com/gogo/protobuf/proto
- github.com/stretchr/testify/assert
- leveldb 的hashmap cache与LRU 实现可以参考使用
- leveldb 的memdb 实现分析
- 乱入之一个位操作的小技巧
- 乱入一段 UUID 生成算法（从fabric/common/util来）
- github.com/eapache/queue
- github.com/fsouza/go-dockerclient docker的go客户端
- github.com/gocraft/web 一个轻量http框架
- github.com/mitchellh/mapstructure
在看一些golang 的开源项目的时候 ，看到他们用到的一些第三方库，感觉还不错
记下来以备后用,看多少加多少
最近在看https://github.com/hyperledger/fabric
日志相关
github.com/cihub/seelog
github.com/op/go-logging
可以支持多种backend ,包括seelog 、文件等
可以实现按模块定制日志输出级别（通过正则匹配package名实现）
使用示例 github.com/hyperledger/fabric/common/flogging/logging.go
解析命令行参数的工具 gopkg.in/alecthomas/kingpin.v2
轻松实现以下格式的命令行参数
usage: main [<flags>] <command> [<args> …]
Flags:
–help Show context-sensitive help (also try –help-long and –help-man).
-d, –dir="." directory of data
Commands:
help [<command>…]
Show help.
accountlist [<n>]
get all account List.
short
get short info about our chainblock system.
account <name>
get account info.
blocklist
get all blocklist.
block <name>
get block info.
var (
dir = kingpin.Flag("dir", "directory of data").Short('d').Default(".").String()
accountListCMD = kingpin.Command("accountlist", "get all account List.")
accountListLimit = accountListCMD.Arg("n", "limit account count").Int()
shortCMD = kingpin.Command("short", "get short info about our chainblock system.")
accountCMD = kingpin.Command("account", "get account info.")
accountName = accountCMD.Arg("name", "account name").Required().String()
blockListCMD = kingpin.Command("blocklist", "get all blocklist.")
blockCMD = kingpin.Command("block", "get block info.")
blockName = blockCMD.Arg("name", "block name").Required().String()
)
func main() {
kingpin.Parse()
switch kingpin.MustParse(kingpin.Parse(), nil) {
// Register user
case shortCMD.FullCommand():
printChain(*dir)
return
case accountListCMD.FullCommand():
printAccountList(*dir, *accountListLimit)
return
case blockListCMD.FullCommand():
printBlockList(*dir)
return
case blockCMD.FullCommand():
if blockName == nil || *blockName == "" {
fmt.Printf("please give an account name like this:%s %s %s\n", os.Args[0], os.Args[1], "block{blockid}")
return
}
printBlock(*dir, *blockName)
return
case accountCMD.FullCommand():
if accountName == nil || *accountName == "" {
fmt.Printf("please give an account name like this:%s %s %s\n", os.Args[0], os.Args[1], "account{accountid}")
return
}
printAccount(*dir, *accountName)
return
}
github.com/spf13/cobra 命令行相关
可以参考 https://studygolang.com/articles/7588
似乎是可以用这个工具帮你生成一部分代码,然后去填充每个具体的命令的实现
demo
package main
import (
"fmt"
"strings"
"github.com/spf13/cobra"
)
func main() {
var echoTimes int
var cmdPrint = &cobra.Command{
Use: "print [string to print]",
Short: "Print anything to the screen",
Long: `print is for printing anything back to the screen.
For many years people have printed back to the screen.
`,
Run: func(cmd *cobra.Command, args []string) {
fmt.Println("Print: " + strings.Join(args, " "))
},
}
var cmdEcho = &cobra.Command{
Use: "echo [string to echo]",
Short: "Echo anything to the screen",
Long: `echo is for echoing anything back.
Echo works a lot like print, except it has a child command.
`,
Run: func(cmd *cobra.Command, args []string) {
fmt.Println("Print: " + strings.Join(args, " "))
},
}
var cmdTimes = &cobra.Command{
Use: "times [# times] [string to echo]",
Short: "Echo anything to the screen more times",
Long: `echo things multiple times back to the user by providing
a count and a string.`,
Run: func(cmd *cobra.Command, args []string) {
for i := 0; i < echoTimes; i++ {
fmt.Println("Echo: " + strings.Join(args, " "))
}
},
}
cmdTimes.Flags().IntVarP(&echoTimes, "times", "t", 1, "times to echo the input")
var rootCmd = &cobra.Command{Use: "app"}
rootCmd.AddCommand(cmdPrint, cmdEcho)
cmdEcho.AddCommand(cmdTimes)
rootCmd.Execute()
}
读取配置文件 相关(json ,yaml,环境变量,etcd,命令行参数)
github.com/spf13/viper
可以设定默认值，以各种方式读取配置，各个配置可以相互覆盖,甚至可以通过e,tcd、Consul等方式进行读取
也自以可选的进行加密
github.com/dgryski/go-jump
golang 使用memcached 集群时有用到一致性哈希算法，这是一种实现
纯计算 内在占用少
github.com/pquerna/ffjson
json decode encode 加速
github.com/tools/godep
go 包版本管理工具,
github.com/gogo/protobuf/proto
protobuf decode encode 加速
可以实现为每条协议 生成 Marshaler UnMarshaler Size 等函数
生成减少一次内在copy ,并且序列化反序列化的代码在编译期就生成 ，速度提升
default: @mkdir -p ../pb/ @awk -v "n=line-number"\ -v 'line1=import "github.com/gogo/protobuf/gogoproto/gogo.proto"; '\ -v 'line2=option (gogoproto.marshaler_all) = true;'\ -v 'line3=option (gogoproto.sizer_all) = true;'\ -v 'line4=option (gogoproto.unmarshaler_all) = true;'\ '(NR==2) { print line1;print line2;print line3;print line4 } 1' base.proto>base2.proto protoc --proto_path=$$GOPATH/src/github.com/gogo/protobuf/protobuf:$$GOPATH/src/:. --gogo_out=. base2.proto @mv base2.pb.go ../pb/base.pb.go @rm -f base2.proto cp -f base.proto ../pb/base_bak.proto
github.com/stretchr/testify/assert
写测试用的工具 demo:
assert.True(t, true, "should be true")
leveldb 的hashmap cache与LRU 实现可以参考使用
https://github.com/syndtr/goleveldb/blob/master/leveldb/cache/cache.go
目前似乎key 只支持uint64,不支持string
支持namespace,即一个 namespace 和key 两个参数才惟一确定一条数据
hash 算法全用 murmur
hash 算法将namespace与key 与seed 算出一个hash值（uint32）,
然后根据hash值的低位4字节 分配到一个桶内（bucket）,后期随着桶内元素的增加，会增加桶的数量
重新进行hash,
type mNode struct {
buckets []unsafe.Pointer // []*mBucket n个桶桶
mask uint32
pred unsafe.Pointer // *mNode
resizeInProgess int32
overflow int32
growThreshold int32
shrinkThreshold int32
}
type mBucket struct {
mu sync.Mutex
node []*Node
frozen bool
}
// Node is a 'cache node'.
type Node struct {
r *Cache
hash uint32
ns, key uint64
mu sync.Mutex
size int
value Value // 真正存数据
ref int32
onDel []func()
CacheData unsafe.Pointer
}
demo
c := cache.NewCache(nil)
h := c.Get(1, 11, func() (int, cache.Value) { return 5, "hello" }) // namespace=1,key==11
fmt.Println(h.Value().(string))
h.Release() // 用完就得release()
h = c.Get(1, 11, nil)
fmt.Println(h.Value().(string))
h.Release() // 用完就得release()
var c *cache.Cache
c = cache.NewCache(cache.NewLRU(20)) // 这种情况支持lru的hash ,当占用内存达到20时，将会采用lru算法清除最久没使用的数据
for i := 0; i < 1000; i++ {
c.Get(1, uint64(i), func() (int, cache.Value) { return 4, "world" + strconv.Itoa(i) }).Release()
}
如果不需要namespace ,即所有的key在同一个namespace下，
则以用NamespaceGetter包装一下,以后Get 只需要 传key 不需要传namespace
var c *cache.Cache
c = cache.NewCache(cache.NewLRU(20)) // 这种情况支持lru的hash ,当占用内存达到20时，将会采用lru算法清除最久没使用的数据
geter:=cache.NamespaceGetter{Cache: c, NS: 0}
geter.Get(11, func() (int, cache.Value) { return 5, "hello" }).Release() // namespace=1,key==11
geter.Get(11).Release()
leveldb 的memdb 实现分析
关于跳表可以参考 http://blog.sina.com.cn/s/blog_72995dcc01017w1t.html,%E4%B8%8D%E5%86%8D
跳表是一种随机化的数据结构，目前开源软件 Redis 和 LevelDB 都有用到它，
它的效率和红黑树以及 AVL 树不相上下O(logn)，但跳表的原理相当简单，只要你能熟练操作链表，
就能轻松实现一个 SkipList
主要实现使用skiplist 跳转来实现
type DB struct {
cmp comparer.BasicComparer
rnd *rand.Rand
mu sync.RWMutex
kvData []byte // key and value 都 append到此，通过偏移量来定位key value
nodeData []int // 记录key value 的偏移量等信息
prevNode [tMaxHeight]int
maxHeight int
n int
kvSize int
}
优点 ，内存连续，GC压力应该小一点，缺点似乎单从Put Get 来看，并不如golang 自带的map 快
当然leveldb可能有其他方面的考虑，如iterator等
这个benchmark中， 所有带2的都是RWMutex+map 的实现，其他则是goleveldb.memdb的
可以看出来，memdb比map 大概慢一个数量级,rwMutex+map能满足你需求的时候 就不要考虑memdb了
BenchmarkPut2 1000000 1059 ns/op
BenchmarkPutRandom2 20000000 81.0 ns/op
BenchmarkGet2 10000000 110 ns/op
BenchmarkGetRandom2 3000000 462 ns/op
BenchmarkPut 1000000 1584 ns/op
BenchmarkPutRandom 1000000 2969 ns/op
BenchmarkGet 1000000 1863 ns/op
BenchmarkGetRandom 1000000 3831 ns/op
乱入之一个位操作的小技巧
声明位常量的时候可以这样声明
type Strict uint
const (
StrictManifest Strict = 1 << iota // 1
StrictJournalChecksum // 2
StrictJournal // 4
)
乱入一段 UUID 生成算法（从fabric/common/util来）
// GenerateBytesUUID returns a UUID based on RFC 4122 returning the generated bytes
func GenerateBytesUUID() []byte {
uuid := make([]byte, 16)
_, err := io.ReadFull(rand.Reader, uuid)
if err != nil {
panic(fmt.Sprintf("Error generating UUID: %s", err))
}
// variant bits; see section 4.1.1
uuid[8] = uuid[8]&^0xc0 | 0x80
// version 4 (pseudo-random); see section 4.1.3
uuid[6] = uuid[6]&^0xf0 | 0x40
return uuid
}
// GenerateIntUUID returns a UUID based on RFC 4122 returning a big.Int
func GenerateIntUUID() *big.Int {
uuid := GenerateBytesUUID()
z := big.NewInt(0)
return z.SetBytes(uuid)
}
// GenerateUUID returns a UUID based on RFC 4122
func GenerateUUID() string {
uuid := GenerateBytesUUID()
return idBytesToStr(uuid)
}
func idBytesToStr(id []byte) string {
return fmt.Sprintf("%x-%x-%x-%x-%x", id[0:4], id[4:6], id[6:8], id[8:10], id[10:])
}
github.com/eapache/queue
一个简单的非并发安全的queue(非常简单)
github.com/fsouza/go-dockerclient docker的go客户端
demo:
import (
"fmt"
"github.com/fsouza/go-dockerclient"
)
func main() {
endpoint := "unix:///var/run/docker.sock"
client, _ := docker.NewClient(endpoint)
imgs, _ := client.ListImages(docker.ListImagesOptions{All: false})
for _, img := range imgs {
fmt.Println("ID: ", img.ID)
fmt.Println("RepoTags: ", img.RepoTags)
fmt.Println("Created: ", img.Created)
fmt.Println("Size: ", img.Size)
fmt.Println("VirtualSize: ", img.VirtualSize)
fmt.Println("ParentId: ", img.ParentID)
}
}
github.com/gocraft/web 一个轻量http框架
package main
import (
"github.com/gocraft/web"
"fmt"
"net/http"
"strings"
)
type Context struct {
HelloCount int
}
func (c *Context) SetHelloCount(rw web.ResponseWriter, req *web.Request, next web.NextMiddlewareFunc) {
c.HelloCount = 3
next(rw, req)
}
func (c *Context) SayHello(rw web.ResponseWriter, req *web.Request) {
fmt.Fprint(rw, strings.Repeat("Hello ", c.HelloCount), "World!")
}
func main() {
router := web.New(Context{}). // Create your router
Middleware(web.LoggerMiddleware). // Use some included middleware
Middleware(web.ShowErrorsMiddleware). // ...
Middleware((*Context).SetHelloCount). // Your own middleware!
Get("/", (*Context).SayHello) // Add a route
http.ListenAndServe("localhost:3000", router) // Start the server!
}
github.com/mitchellh/mapstructure
实现 了 将一个 map[string]interface{} 转成结构体的功能
比如遇到以下问题： 一串json ,在未解析json某些字段之前，并不知道具体还有哪些字段
则可以将json 先转成map[string]interface{},然后判断之后利用此包再转成具体的struct