我有this project尝试在 Go 中同时运行无限的大查询。父项目全部是Python。我需要能够跟踪查询结果,就像在 map 中一样。
Input:
{
'reports_portal': 'select * from reports_portal',
'billing_portal': 'select * from billing_portal',
}
output:
{
'reports_portal': [23, 123, 5234, 632],
'billing_portal': [23, 123, 5234, 632],
}
等等
这些大查询需要异步运行,因为它们非常慢(从 UI 角度来看,SRE 等待 15-30 秒才能得到结果。
我首先尝试将项目异步写入 map :
package main
import (
"fmt"
)
func add_to_map(m map[string] string, word string) map[string]string {
added_word := word + " plus more letters"
m[word] = added_word
return m
}
func main() {
words_map := make(map[string]string)
words := []string{"giraffe", "cat", "dog", "turtle"}
for _, this_word := range words {
go add_to_map(words_map, this_word)
}
fmt.Println(words_map)
}
爆炸如下:
$ go run try_asynchronous.go
fatal error: concurrent map writes
goroutine 7 [running]:
runtime.throw(0x10b3b96, 0x15)
/usr/local/Cellar/go/1.8.1/libexec/src/runtime/panic.go:596 +0x95 fp=0xc420032eb8 sp=0xc420032e98
runtime.mapassign(0x109ad20, 0xc420016270, 0xc420032fa0, 0x10b3268)
/usr/local/Cellar/go/1.8.1/libexec/src/runtime/hashmap.go:499 +0x667 fp=0xc420032f58 sp=0xc420032eb8
main.add_to_map(0xc420016270, 0x10b1ba0, 0x3, 0x0)
/tmp/golang-w-python/try_asynchronous.go:10 +0xa3 fp=0xc420032fc0 sp=0xc420032f58
runtime.goexit()
/usr/local/Cellar/go/1.8.1/libexec/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420032fc8 sp=0xc420032fc0
created by main.main
/tmp/golang-w-python/try_asynchronous.go:19 +0xc8
goroutine 1 [runnable]:
fmt.(*pp).fmtString(0xc42001e0c0, 0x10b1f52, 0x7, 0x76)
/usr/local/Cellar/go/1.8.1/libexec/src/fmt/print.go:424 +0x1a2
fmt.(*pp).printValue(0xc42001e0c0, 0x10953c0, 0xc42000e260, 0x98, 0x76, 0x1)
/usr/local/Cellar/go/1.8.1/libexec/src/fmt/print.go:729 +0x27aa
fmt.(*pp).printValue(0xc42001e0c0, 0x109ad20, 0xc420016270, 0x15, 0x76, 0x0)
/usr/local/Cellar/go/1.8.1/libexec/src/fmt/print.go:750 +0x103d
fmt.(*pp).printArg(0xc42001e0c0, 0x109ad20, 0xc420016270, 0x76)
/usr/local/Cellar/go/1.8.1/libexec/src/fmt/print.go:682 +0x217
fmt.(*pp).doPrintln(0xc42001e0c0, 0xc420045f28, 0x1, 0x1)
/usr/local/Cellar/go/1.8.1/libexec/src/fmt/print.go:1138 +0xa1
fmt.Fprintln(0x1108140, 0xc42000c018, 0xc420045f28, 0x1, 0x1, 0xc420045ef0, 0xc420045ee0, 0x1087218)
/usr/local/Cellar/go/1.8.1/libexec/src/fmt/print.go:247 +0x5c
fmt.Println(0xc420045f28, 0x1, 0x1, 0x10b1e6f, 0x6, 0x0)
/usr/local/Cellar/go/1.8.1/libexec/src/fmt/print.go:257 +0x57
main.main()
/tmp/golang-w-python/try_asynchronous.go:21 +0x132
exit status 2
基于需要一次运行多个查询并尝试按名称跟踪结果,我希望在异步期间写入映射。但是 fatal error :并发映射写入
说你不能。
我听不懂
- 为什么不
- 我应该做什么同时运行这些大查询。
编辑:
我拥有的最接近的返回结果的东西不是异步的:
package main
import (
"fmt"
"math/rand"
"sync"
"time"
)
var mutex sync.Mutex
var wg sync.WaitGroup
func random_sleep() {
r := rand.Intn(3000)
time.Sleep(time.Duration(r) * time.Millisecond)
}
func add_to_map(m map[string] string, word string) {
defer wg.Done()
added_word := word + " plus more letters"
mutex.Lock()
defer mutex.Unlock()
fmt.Println("Before sleep")
random_sleep()
m[word] = added_word
fmt.Println("Added word %v", word)
}
func main() {
words_map := make(map[string]string)
words := []string{"giraffe", "cat", "dog", "turtle"}
for _, this_word := range words {
wg.Add(1)
go add_to_map(words_map, this_word)
}
wg.Wait()
fmt.Println(words_map)
}
结果是错误的:
cchilders:~/work_projects/metricsportal/golang_integration (feature/golang-query)
$ go run try_async.go
Before sleep
Added word %v turtle
Before sleep
Added word %v cat
Before sleep
Added word %v giraffe
Before sleep
Added word %v dog
map[dog:dog plus more letters turtle:turtle plus more letters cat:cat plus more letters giraffe:giraffe plus more letters]
cchilders:~/work_projects/metricsportal/golang_integration (feature/golang-query)
$ go run try_async.go
Before sleep
Added word %v turtle
Before sleep
Added word %v cat
Before sleep
Added word %v giraffe
Before sleep
Added word %v dog
map[dog:dog plus more letters turtle:turtle plus more letters cat:cat plus more letters giraffe:giraffe plus more letters]
结果应该非常快,不超过 3 秒(我认为随机的最大值):
Expectation -
Before sleep
Before sleep
Before sleep
Before sleep
Added word %v cat
Added word %v giraffe
Added word %v turtle
Added word %v dog
最佳答案
好吧,让我澄清一些事情并帮助你。
您不需要从此处返回修改后的 map ,因为您的 函数获取对 map 的引用而不是它的副本。 (让我们忽略这个事实 您完全忽略了返回值)
func add_to_map(m map[string] string, word string) map[string]string {
added_word := word + " plus more letters"
m[word] = added_word
return m
}
下一步是您需要同步对 map 的访问。您可以使用 mutex为此。
import "sync"
var mutex sync.Mutex //glabal variable but can be created as local also
func add_to_map(m map[string] string, word string) {
added_word := word + " plus more letters"
// here you can do long to compute task and calculate result
// calc here
mutex.Lock() //result ready lock mutex
defer mutex.Unlock() // unlock mutex when we return from function
m[word] = added_word // result write to shared map
}
请注意,在 Go 1.9 中会有 Concurrent Map类型。
编辑:
您需要等待所有 go 例程完成,因为您的 main()
现在在它们之前完成。您可以使用WaitGroup来做到这一点
package main
import (
"fmt"
"sync"
)
var mutex sync.Mutex
var wg sync.WaitGroup
func add_to_map(m map[string] string, word string) {
defer wg.Done()
added_word := word + " plus more letters"
// do heavy work here
//
mutex.Lock()
defer mutex.Unlock()
m[word] = added_word
}
func main() {
words_map := make(map[string]string)
words := []string{"giraffe", "cat", "dog", "turtle"}
for _, this_word := range words {
wg.Add(1)
go add_to_map(words_map, this_word)
}
wg.Wait()
fmt.Println(words_map)
}
关于python - 运行Go异步操作并写入map,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/45378537/