Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Data race in native histogram #1605

Closed
krajorama opened this issue Aug 31, 2024 · 1 comment · Fixed by #1608
Closed

Data race in native histogram #1605

krajorama opened this issue Aug 31, 2024 · 1 comment · Fixed by #1608
Assignees
@krajorama
Labels
bug

Comments

@krajorama
Copy link
Member

krajorama commented Aug 31, 2024 •
edited
Loading

Version github.com/prometheus/client_golang v1.20.0

Capacity for exemplars is called without lock here:

client_golang/prometheus/histogram.go

Line 1685 in dbf72fc

if cap(n.exemplars) == 0 {

Which causes the race with

client_golang/prometheus/histogram.go

Line 1704 in dbf72fc

n.exemplars = append(n.exemplars[:nIdx], append([]*dto.Exemplar{e}, n.exemplars[nIdx:]...)...)

See logs from Mimir PR CI run:

22:12:09 query-frontend: ==================
22:12:09 query-frontend: WARNING: DATA RACE
22:12:09 query-frontend: Read at 0x00c001bba3d8 by goroutine 78:
22:12:09 query-frontend: github.com/prometheus/client_golang/prometheus.(*nativeExemplars).addExemplar()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/github.com/prometheus/client_golang/prometheus/histogram.go:1685 +0x57
22:12:09 query-frontend: github.com/prometheus/client_golang/prometheus.(*histogram).updateExemplar()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/github.com/prometheus/client_golang/prometheus/histogram.go:1140 +0x116
22:12:09 query-frontend: github.com/prometheus/client_golang/prometheus.(*histogram).ObserveWithExemplar()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/github.com/prometheus/client_golang/prometheus/histogram.go:770 +0x69
22:12:09 query-frontend: github.com/grafana/dskit/instrument.ObserveWithExemplar()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/github.com/grafana/dskit/instrument/instrument.go:76 +0x1a2
22:12:09 query-frontend: github.com/grafana/dskit/middleware.observe()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/github.com/grafana/dskit/middleware/grpc_instrumentation.go:34 +0x205
22:12:09 query-frontend: github.com/grafana/dskit/middleware.StreamServerInstrumentInterceptor.func1()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/github.com/grafana/dskit/middleware/grpc_instrumentation.go:58 +0x18b
22:12:09 query-frontend: google.golang.org/grpc.getChainStreamHandler.func1()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/google.golang.org/grpc/server.go:1525 +0x118
22:12:09 query-frontend: github.com/opentracing-contrib/go-grpc.OpenTracingStreamServerInterceptor.func1()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/github.com/opentracing-contrib/go-grpc/server.go:114 +0x4ae
22:12:09 query-frontend: google.golang.org/grpc.getChainStreamHandler.func1()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/google.golang.org/grpc/server.go:1525 +0x118
22:12:09 query-frontend: github.com/grafana/dskit/middleware.GRPCServerLog.StreamServerInterceptor()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/github.com/grafana/dskit/middleware/grpc_logging.go:87 +0xbd
22:12:09 query-frontend: github.com/grafana/dskit/middleware.GRPCServerLog.StreamServerInterceptor-fm()
22:12:09 query-frontend: <autogenerated>:1 +0xbd
22:12:09 query-frontend: google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/google.golang.org/grpc/server.go:1516 +0xdc
22:12:09 query-frontend: google.golang.org/grpc.(*Server).processStreamingRPC()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/google.golang.org/grpc/server.go:1680 +0x1f8e
22:12:09 query-frontend: google.golang.org/grpc.(*Server).handleStream()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/google.golang.org/grpc/server.go:1794 +0x17c4
22:12:09 query-frontend: google.golang.org/grpc.(*Server).serveStreams.func2.1()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/google.golang.org/grpc/server.go:1029 +0x158
22:12:09 query-frontend: google.golang.org/grpc.(*Server).serverWorker()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/google.golang.org/grpc/server.go:637 +0x32
22:12:09 query-frontend: google.golang.org/grpc.(*Server).initServerWorkers.gowrap1()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/google.golang.org/grpc/server.go:650 +0x33
22:12:09 query-frontend: Previous write at 0x00c001bba3d8 by goroutine 73:
22:12:09 query-frontend: github.com/prometheus/client_golang/prometheus.(*nativeExemplars).addExemplar()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/github.com/prometheus/client_golang/prometheus/histogram.go:1704 +0x54b
22:12:09 query-frontend: github.com/prometheus/client_golang/prometheus.(*histogram).updateExemplar()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/github.com/prometheus/client_golang/prometheus/histogram.go:1140 +0x116
22:12:09 query-frontend: github.com/prometheus/client_golang/prometheus.(*histogram).ObserveWithExemplar()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/github.com/prometheus/client_golang/prometheus/histogram.go:770 +0x69
22:12:09 query-frontend: github.com/grafana/dskit/instrument.ObserveWithExemplar()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/github.com/grafana/dskit/instrument/instrument.go:76 +0x1a2
22:12:09 query-frontend: github.com/grafana/dskit/middleware.observe()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/github.com/grafana/dskit/middleware/grpc_instrumentation.go:34 +0x205
22:12:09 query-frontend: github.com/grafana/dskit/middleware.StreamServerInstrumentInterceptor.func1()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/github.com/grafana/dskit/middleware/grpc_instrumentation.go:58 +0x18b
22:12:09 query-frontend: google.golang.org/grpc.getChainStreamHandler.func1()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/google.golang.org/grpc/server.go:1525 +0x118
22:12:09 query-frontend: github.com/opentracing-contrib/go-grpc.OpenTracingStreamServerInterceptor.func1()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/github.com/opentracing-contrib/go-grpc/server.go:114 +0x4ae
22:12:09 query-frontend: google.golang.org/grpc.getChainStreamHandler.func1()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/google.golang.org/grpc/server.go:1525 +0x118
22:12:09 query-frontend: github.com/grafana/dskit/middleware.GRPCServerLog.StreamServerInterceptor()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/github.com/grafana/dskit/middleware/grpc_logging.go:87 +0xbd
22:12:09 query-frontend: github.com/grafana/dskit/middleware.GRPCServerLog.StreamServerInterceptor-fm()
22:12:09 query-frontend: <autogenerated>:1 +0xbd
22:12:09 query-frontend: google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/google.golang.org/grpc/server.go:1516 +0xdc
22:12:09 query-frontend: google.golang.org/grpc.(*Server).processStreamingRPC()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/google.golang.org/grpc/server.go:1680 +0x1f8e
22:12:09 query-frontend: google.golang.org/grpc.(*Server).handleStream()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/google.golang.org/grpc/server.go:1794 +0x17c4
22:12:09 query-frontend: google.golang.org/grpc.(*Server).serveStreams.func2.1()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/google.golang.org/grpc/server.go:1029 +0x158
22:12:09 query-frontend: google.golang.org/grpc.(*Server).serverWorker()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/google.golang.org/grpc/server.go:637 +0x32
22:12:09 query-frontend: google.golang.org/grpc.(*Server).initServerWorkers.gowrap1()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/google.golang.org/grpc/server.go:650 +0x33
22:12:09 query-frontend: Goroutine 78 (running) created at:
22:12:09 query-frontend: google.golang.org/grpc.(*Server).initServerWorkers()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/google.golang.org/grpc/server.go:650 +0x219
22:12:09 query-frontend: google.golang.org/grpc.NewServer()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/google.golang.org/grpc/server.go:682 +0xdda
22:12:09 query-frontend: github.com/grafana/dskit/server.newServer()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/github.com/grafana/dskit/server/server.go:453 +0x3a9e
22:12:09 query-frontend: github.com/grafana/dskit/server.New()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/github.com/grafana/dskit/server/server.go:243 +0x97
22:12:09 query-frontend: github.com/grafana/mimir/pkg/mimir.(*Mimir).initServer()
22:12:09 query-frontend: /__w/mimir/mimir/pkg/mimir/modules.go:325 +0x3c4
22:12:09 query-frontend: github.com/grafana/mimir/pkg/mimir.(*Mimir).initServer-fm()
22:12:09 query-frontend: <autogenerated>:1 +0x33
22:12:09 query-frontend: github.com/grafana/dskit/modules.(*Manager).initModule()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/github.com/grafana/dskit/modules/modules.go:136 +0x35a
22:12:09 query-frontend: github.com/grafana/dskit/modules.(*Manager).InitModuleServices()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/github.com/grafana/dskit/modules/modules.go:108 +0x14a
22:12:09 query-frontend: github.com/grafana/mimir/pkg/mimir.(*Mimir).Run()
22:12:09 query-frontend: /__w/mimir/mimir/pkg/mimir/mimir.go:851 +0x72c
22:12:09 query-frontend: main.main()
22:12:09 query-frontend: /__w/mimir/mimir/cmd/mimir/main.go:227 +0x1dd6
22:12:09 query-frontend: Goroutine 73 (running) created at:
22:12:09 query-frontend: google.golang.org/grpc.(*Server).initServerWorkers()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/google.golang.org/grpc/server.go:650 +0x219
22:12:09 query-frontend: google.golang.org/grpc.NewServer()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/google.golang.org/grpc/server.go:682 +0xdda
22:12:09 query-frontend: github.com/grafana/dskit/server.newServer()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/github.com/grafana/dskit/server/server.go:453 +0x3a9e
22:12:09 query-frontend: github.com/grafana/dskit/server.New()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/github.com/grafana/dskit/server/server.go:243 +0x97
22:12:09 query-frontend: github.com/grafana/mimir/pkg/mimir.(*Mimir).initServer()
22:12:09 query-frontend: /__w/mimir/mimir/pkg/mimir/modules.go:325 +0x3c4
22:12:09 query-frontend: github.com/grafana/mimir/pkg/mimir.(*Mimir).initServer-fm()
22:12:09 query-frontend: <autogenerated>:1 +0x33
22:12:09 query-frontend: github.com/grafana/dskit/modules.(*Manager).initModule()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/github.com/grafana/dskit/modules/modules.go:136 +0x35a
22:12:09 query-frontend: github.com/grafana/dskit/modules.(*Manager).InitModuleServices()
22:12:09 query-frontend: /__w/mimir/mimir/vendor/github.com/grafana/dskit/modules/modules.go:108 +0x14a
22:12:09 query-frontend: github.com/grafana/mimir/pkg/mimir.(*Mimir).Run()
22:12:09 query-frontend: /__w/mimir/mimir/pkg/mimir/mimir.go:851 +0x72c
22:12:09 query-frontend: main.main()
22:12:09 query-frontend: /__w/mimir/mimir/cmd/mimir/main.go:227 +0x1dd6
22:12:09 query-frontend: ==================
@dosubot dosubot bot added the bug label Aug 31, 2024
@krajorama krajorama self-assigned this Aug 31, 2024
@krajorama krajorama mentioned this issue Aug 31, 2024
Closed
krajorama added a commit to krajorama/client_golang that referenced this issue Aug 31, 2024
@krajorama
Use a trivial solution to prometheus#1605
87d0504 
Signed-off-by: György Krajcsovits <[email protected]>
This was referenced Aug 31, 2024
Closed
Merged
Merged
krajorama added a commit to krajorama/client_golang that referenced this issue Sep 4, 2024
@krajorama
Use a trivial solution to prometheus#1605
dc819ce 
Signed-off-by: György Krajcsovits <[email protected]>
@krajorama
Copy link
Member Author

fixed

amberpixels pushed a commit to amberpixels/prometheus_client_golang that referenced this issue Nov 29, 2024
@krajorama @amberpixels
Use a trivial solution to prometheus#1605
d31ec24 
Signed-off-by: György Krajcsovits <[email protected]>
Signed-off-by: Eugene <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@krajorama krajorama

Labels
bug
Projects
None yet
Milestone
No milestone
1 participant
@krajorama