[Bug]: [benchmark][cluster] milvus insert timeout and flush raise error: err: context deadline exceeded #14707

wangting0128 · 2021-12-31T13:15:39Z

Is there an existing issue for this?

I have searched the existing issues

Environment

- Milvus version: master-20211231-9baa6e8
- Deployment mode(standalone or cluster): cluster
- SDK version(e.g. pymilvus v2.0.0rc2): 2.0.0rc9.dev22
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

client pod: benchmark-no-clean-98c2h-3511807547

client.log

client logs:

[2021-12-31 11:07:07,079] [    INFO] - {'total_time': 5276.6751, 'rps': 18951.3277, 'ni_time': 2.6383} (milvus_benchmark.runners.base:151)
[2021-12-31 11:07:07,192] [   DEBUG] - Start flush. (milvus_benchmark.runners.locust:428)
[2021-12-31 12:06:20,495] [   ERROR] - 
Addr [benchmark-no-clean-98c2h-1-milvus.qa-milvus.svc.cluster.local:19530] get_flush_state
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
	status = StatusCode.UNKNOWN
	details = "err: context deadline exceeded
, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:46 github.com/milvus-io/milvus/internal/util/trace.StackTrace
/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:231 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase).ReCall
/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:458 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetFlushState
/go/src/github.com/milvus-io/milvus/internal/proxy/impl.go:3567 github.com/milvus-io/milvus/internal/proxy.(*Proxy).GetFlushState
/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:573 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).GetFlushState
/go/src/github.com/milvus-io/milvus/internal/proto/milvuspb/milvus.pb.go:6356 github.com/milvus-io/milvus/internal/proto/milvuspb._MilvusService_GetFlushState_Handler.func1
/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/tracing/opentracing/server_interceptors.go:38 github.com/grpc-ecosystem/go-grpc-middleware/tracing/opentracing.UnaryServerInterceptor.func1
/go/src/github.com/milvus-io/milvus/internal/proto/milvuspb/milvus.pb.go:6358 github.com/milvus-io/milvus/internal/proto/milvuspb._MilvusService_GetFlushState_Handler
/go/pkg/mod/google.golang.org/[email protected]/server.go:1286 google.golang.org/grpc.(*Server).processUnaryRPC
/go/pkg/mod/google.golang.org/[email protected]/server.go:1609 google.golang.org/grpc.(*Server).handleStream
"
	debug_error_string = "{"created":"@1640952380.494975777","description":"Error received from peer ipv4:10.96.54.119:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"err: context deadline exceeded\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:46 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:231 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:458 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetFlushState\n/go/src/github.com/milvus-io/milvus/internal/proxy/impl.go:3567 github.com/milvus-io/milvus/internal/proxy.(*Proxy).GetFlushState\n/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:573 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).GetFlushState\n/go/src/github.com/milvus-io/milvus/internal/proto/milvuspb/milvus.pb.go:6356 github.com/milvus-io/milvus/internal/proto/milvuspb._MilvusService_GetFlushState_Handler.func1\n/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/tracing/opentracing/server_interceptors.go:38 github.com/grpc-ecosystem/go-grpc-middleware/tracing/opentracing.UnaryServerInterceptor.func1\n/go/src/github.com/milvus-io/milvus/internal/proto/milvuspb/milvus.pb.go:6358 github.com/milvus-io/milvus/internal/proto/milvuspb._MilvusService_GetFlushState_Handler\n/go/pkg/mod/google.golang.org/[email protected]/server.go:1286 google.golang.org/grpc.(*Server).processUnaryRPC\n/go/pkg/mod/google.golang.org/[email protected]/server.go:1609 google.golang.org/grpc.(*Server).handleStream\n","grpc_status":2}"
>
	{'API start': '2021-12-31 12:06:02.993281', 'RPC start': '2021-12-31 12:06:02.993307', 'RPC error': '2021-12-31 12:06:20.495448'} (pymilvus.client.grpc_handler:84)
[2021-12-31 12:06:20,497] [   ERROR] - 
Addr [benchmark-no-clean-98c2h-1-milvus.qa-milvus.svc.cluster.local:19530] _wait_for_flushed
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
	status = StatusCode.UNKNOWN
	details = "err: context deadline exceeded
, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:46 github.com/milvus-io/milvus/internal/util/trace.StackTrace
/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:231 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase).ReCall
/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:458 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetFlushState
/go/src/github.com/milvus-io/milvus/internal/proxy/impl.go:3567 github.com/milvus-io/milvus/internal/proxy.(*Proxy).GetFlushState
/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:573 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).GetFlushState
/go/src/github.com/milvus-io/milvus/internal/proto/milvuspb/milvus.pb.go:6356 github.com/milvus-io/milvus/internal/proto/milvuspb._MilvusService_GetFlushState_Handler.func1
/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/tracing/opentracing/server_interceptors.go:38 github.com/grpc-ecosystem/go-grpc-middleware/tracing/opentracing.UnaryServerInterceptor.func1
/go/src/github.com/milvus-io/milvus/internal/proto/milvuspb/milvus.pb.go:6358 github.com/milvus-io/milvus/internal/proto/milvuspb._MilvusService_GetFlushState_Handler
/go/pkg/mod/google.golang.org/[email protected]/server.go:1286 google.golang.org/grpc.(*Server).processUnaryRPC
/go/pkg/mod/google.golang.org/[email protected]/server.go:1609 google.golang.org/grpc.(*Server).handleStream
"
	debug_error_string = "{"created":"@1640952380.494975777","description":"Error received from peer ipv4:10.96.54.119:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"err: context deadline exceeded\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:46 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:231 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:458 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetFlushState\n/go/src/github.com/milvus-io/milvus/internal/proxy/impl.go:3567 github.com/milvus-io/milvus/internal/proxy.(*Proxy).GetFlushState\n/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:573 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).GetFlushState\n/go/src/github.com/milvus-io/milvus/internal/proto/milvuspb/milvus.pb.go:6356 github.com/milvus-io/milvus/internal/proto/milvuspb._MilvusService_GetFlushState_Handler.func1\n/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/tracing/opentracing/server_interceptors.go:38 github.com/grpc-ecosystem/go-grpc-middleware/tracing/opentracing.UnaryServerInterceptor.func1\n/go/src/github.com/milvus-io/milvus/internal/proto/milvuspb/milvus.pb.go:6358 github.com/milvus-io/milvus/internal/proto/milvuspb._MilvusService_GetFlushState_Handler\n/go/pkg/mod/google.golang.org/[email protected]/server.go:1286 google.golang.org/grpc.(*Server).processUnaryRPC\n/go/pkg/mod/google.golang.org/[email protected]/server.go:1609 google.golang.org/grpc.(*Server).handleStream\n","grpc_status":2}"
>
	{'API start': '2021-12-31 11:07:07.279619', 'RPC start': '2021-12-31 11:07:07.279663', 'RPC error': '2021-12-31 12:06:20.497280'} (pymilvus.client.grpc_handler:84)
[2021-12-31 12:06:20,498] [   ERROR] - 
Addr [benchmark-no-clean-98c2h-1-milvus.qa-milvus.svc.cluster.local:19530] flush
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
	status = StatusCode.UNKNOWN
	details = "err: context deadline exceeded
, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:46 github.com/milvus-io/milvus/internal/util/trace.StackTrace
/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:231 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase).ReCall
/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:458 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetFlushState
/go/src/github.com/milvus-io/milvus/internal/proxy/impl.go:3567 github.com/milvus-io/milvus/internal/proxy.(*Proxy).GetFlushState
/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:573 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).GetFlushState
/go/src/github.com/milvus-io/milvus/internal/proto/milvuspb/milvus.pb.go:6356 github.com/milvus-io/milvus/internal/proto/milvuspb._MilvusService_GetFlushState_Handler.func1
/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/tracing/opentracing/server_interceptors.go:38 github.com/grpc-ecosystem/go-grpc-middleware/tracing/opentracing.UnaryServerInterceptor.func1
/go/src/github.com/milvus-io/milvus/internal/proto/milvuspb/milvus.pb.go:6358 github.com/milvus-io/milvus/internal/proto/milvuspb._MilvusService_GetFlushState_Handler
/go/pkg/mod/google.golang.org/[email protected]/server.go:1286 google.golang.org/grpc.(*Server).processUnaryRPC
/go/pkg/mod/google.golang.org/[email protected]/server.go:1609 google.golang.org/grpc.(*Server).handleStream
"
	debug_error_string = "{"created":"@1640952380.494975777","description":"Error received from peer ipv4:10.96.54.119:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"err: context deadline exceeded\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:46 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:231 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:458 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetFlushState\n/go/src/github.com/milvus-io/milvus/internal/proxy/impl.go:3567 github.com/milvus-io/milvus/internal/proxy.(*Proxy).GetFlushState\n/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:573 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).GetFlushState\n/go/src/github.com/milvus-io/milvus/internal/proto/milvuspb/milvus.pb.go:6356 github.com/milvus-io/milvus/internal/proto/milvuspb._MilvusService_GetFlushState_Handler.func1\n/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/tracing/opentracing/server_interceptors.go:38 github.com/grpc-ecosystem/go-grpc-middleware/tracing/opentracing.UnaryServerInterceptor.func1\n/go/src/github.com/milvus-io/milvus/internal/proto/milvuspb/milvus.pb.go:6358 github.com/milvus-io/milvus/internal/proto/milvuspb._MilvusService_GetFlushState_Handler\n/go/pkg/mod/google.golang.org/[email protected]/server.go:1286 google.golang.org/grpc.(*Server).processUnaryRPC\n/go/pkg/mod/google.golang.org/[email protected]/server.go:1609 google.golang.org/grpc.(*Server).handleStream\n","grpc_status":2}"
>
	{'API start': '2021-12-31 11:07:07.193281', 'RPC start': '2021-12-31 11:07:07.193305', 'RPC error': '2021-12-31 12:06:20.497973'} (pymilvus.client.grpc_handler:84)
[2021-12-31 12:06:20,498] [   ERROR] - <_MultiThreadedRendezvous of RPC that terminated with:
	status = StatusCode.UNKNOWN
	details = "err: context deadline exceeded
, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:46 github.com/milvus-io/milvus/internal/util/trace.StackTrace
/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:231 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase).ReCall
/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:458 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetFlushState
/go/src/github.com/milvus-io/milvus/internal/proxy/impl.go:3567 github.com/milvus-io/milvus/internal/proxy.(*Proxy).GetFlushState
/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:573 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).GetFlushState
/go/src/github.com/milvus-io/milvus/internal/proto/milvuspb/milvus.pb.go:6356 github.com/milvus-io/milvus/internal/proto/milvuspb._MilvusService_GetFlushState_Handler.func1
/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/tracing/opentracing/server_interceptors.go:38 github.com/grpc-ecosystem/go-grpc-middleware/tracing/opentracing.UnaryServerInterceptor.func1
/go/src/github.com/milvus-io/milvus/internal/proto/milvuspb/milvus.pb.go:6358 github.com/milvus-io/milvus/internal/proto/milvuspb._MilvusService_GetFlushState_Handler
/go/pkg/mod/google.golang.org/[email protected]/server.go:1286 google.golang.org/grpc.(*Server).processUnaryRPC
/go/pkg/mod/google.golang.org/[email protected]/server.go:1609 google.golang.org/grpc.(*Server).handleStream
"
	debug_error_string = "{"created":"@1640952380.494975777","description":"Error received from peer ipv4:10.96.54.119:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"err: context deadline exceeded\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:46 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:231 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:458 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetFlushState\n/go/src/github.com/milvus-io/milvus/internal/proxy/impl.go:3567 github.com/milvus-io/milvus/internal/proxy.(*Proxy).GetFlushState\n/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:573 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).GetFlushState\n/go/src/github.com/milvus-io/milvus/internal/proto/milvuspb/milvus.pb.go:6356 github.com/milvus-io/milvus/internal/proto/milvuspb._MilvusService_GetFlushState_Handler.func1\n/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/tracing/opentracing/server_interceptors.go:38 github.com/grpc-ecosystem/go-grpc-middleware/tracing/opentracing.UnaryServerInterceptor.func1\n/go/src/github.com/milvus-io/milvus/internal/proto/milvuspb/milvus.pb.go:6358 github.com/milvus-io/milvus/internal/proto/milvuspb._MilvusService_GetFlushState_Handler\n/go/pkg/mod/google.golang.org/[email protected]/server.go:1286 google.golang.org/grpc.(*Server).processUnaryRPC\n/go/pkg/mod/google.golang.org/[email protected]/server.go:1609 google.golang.org/grpc.(*Server).handleStream\n","grpc_status":2}"
> (milvus_benchmark.main:118)
[2021-12-31 12:06:20,507] [   ERROR] - Traceback (most recent call last):
  File "main.py", line 87, in run_suite
    runner.prepare(**cases[0])
  File "/src/milvus_benchmark/runners/locust.py", line 429, in prepare
    self.milvus.flush()
  File "/src/milvus_benchmark/client.py", line 49, in wrapper
    result = func(*args, **kwargs)
  File "/src/milvus_benchmark/client.py", line 251, in flush
    self._milvus.flush([tmp_collection_name], _async=_async, timeout=timeout)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/stub.py", line 48, in handler
    raise e
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/stub.py", line 42, in handler
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/stub.py", line 983, in flush
    return handler.flush(collection_names, timeout, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 85, in handler
    raise e
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 67, in handler
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 1104, in flush
    _check()
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 1092, in _check
    self._wait_for_flushed(segment_ids)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 85, in handler
    raise e
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 67, in handler
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 1077, in _wait_for_flushed
    flush_ret = self.get_flush_state(segment_ids, timeout=timeout, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 85, in handler
    raise e
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 67, in handler
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 1057, in get_flush_state
    response = future.result()
  File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 744, in result
    raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
	status = StatusCode.UNKNOWN
	details = "err: context deadline exceeded
, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:46 github.com/milvus-io/milvus/internal/util/trace.StackTrace
/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:231 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase).ReCall
/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:458 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetFlushState
/go/src/github.com/milvus-io/milvus/internal/proxy/impl.go:3567 github.com/milvus-io/milvus/internal/proxy.(*Proxy).GetFlushState
/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:573 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).GetFlushState
/go/src/github.com/milvus-io/milvus/internal/proto/milvuspb/milvus.pb.go:6356 github.com/milvus-io/milvus/internal/proto/milvuspb._MilvusService_GetFlushState_Handler.func1
/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/tracing/opentracing/server_interceptors.go:38 github.com/grpc-ecosystem/go-grpc-middleware/tracing/opentracing.UnaryServerInterceptor.func1
/go/src/github.com/milvus-io/milvus/internal/proto/milvuspb/milvus.pb.go:6358 github.com/milvus-io/milvus/internal/proto/milvuspb._MilvusService_GetFlushState_Handler
/go/pkg/mod/google.golang.org/[email protected]/server.go:1286 google.golang.org/grpc.(*Server).processUnaryRPC
/go/pkg/mod/google.golang.org/[email protected]/server.go:1609 google.golang.org/grpc.(*Server).handleStream
"
	debug_error_string = "{"created":"@1640952380.494975777","description":"Error received from peer ipv4:10.96.54.119:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"err: context deadline exceeded\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:46 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:231 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:458 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetFlushState\n/go/src/github.com/milvus-io/milvus/internal/proxy/impl.go:3567 github.com/milvus-io/milvus/internal/proxy.(*Proxy).GetFlushState\n/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:573 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).GetFlushState\n/go/src/github.com/milvus-io/milvus/internal/proto/milvuspb/milvus.pb.go:6356 github.com/milvus-io/milvus/internal/proto/milvuspb._MilvusService_GetFlushState_Handler.func1\n/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/tracing/opentracing/server_interceptors.go:38 github.com/grpc-ecosystem/go-grpc-middleware/tracing/opentracing.UnaryServerInterceptor.func1\n/go/src/github.com/milvus-io/milvus/internal/proto/milvuspb/milvus.pb.go:6358 github.com/milvus-io/milvus/internal/proto/milvuspb._MilvusService_GetFlushState_Handler\n/go/pkg/mod/google.golang.org/[email protected]/server.go:1286 google.golang.org/grpc.(*Server).processUnaryRPC\n/go/pkg/mod/google.golang.org/[email protected]/server.go:1609 google.golang.org/grpc.(*Server).handleStream\n","grpc_status":2}"
>
 (milvus_benchmark.main:119)

Expected Behavior

argo task: benchmark-no-clean-98c2h

test yaml:
client-configmap: client-random-locust-100m-ddl-r8-w2
server-configmap: server-cluster-8c64m-nfs

server:

NAME                                                              READY   STATUS      RESTARTS   AGE     IP             NODE                      NOMINATED NODE   READINESS GATES
benchmark-no-clean-98c2h-1-etcd-0                                 1/1     Running     0          4h37m   10.97.17.82    qa-node014.zilliz.local   <none>           <none>
benchmark-no-clean-98c2h-1-etcd-1                                 1/1     Running     0          4h37m   10.97.16.122   qa-node013.zilliz.local   <none>           <none>
benchmark-no-clean-98c2h-1-etcd-2                                 1/1     Running     0          4h37m   10.97.17.83    qa-node014.zilliz.local   <none>           <none>
benchmark-no-clean-98c2h-1-milvus-datacoord-cbdc95945-q4zmf       1/1     Running     0          4h37m   10.97.19.211   qa-node016.zilliz.local   <none>           <none>
benchmark-no-clean-98c2h-1-milvus-datanode-856f766d4b-s4rz8       1/1     Running     1          4h37m   10.97.12.137   qa-node015.zilliz.local   <none>           <none>
benchmark-no-clean-98c2h-1-milvus-indexcoord-5ff87b899b-f9d9r     1/1     Running     0          4h37m   10.97.12.138   qa-node015.zilliz.local   <none>           <none>
benchmark-no-clean-98c2h-1-milvus-indexnode-75db5cdcf9-klc5c      1/1     Running     2          4h37m   10.97.19.212   qa-node016.zilliz.local   <none>           <none>
benchmark-no-clean-98c2h-1-milvus-proxy-6dfdf94cd8-ddmm9          1/1     Running     0          4h37m   10.97.10.88    qa-node008.zilliz.local   <none>           <none>
benchmark-no-clean-98c2h-1-milvus-querycoord-696788c94c-j8qdd     1/1     Running     0          4h37m   10.97.10.89    qa-node008.zilliz.local   <none>           <none>
benchmark-no-clean-98c2h-1-milvus-querynode-54b95dc8d5-6mppm      1/1     Running     0          4h37m   10.97.16.121   qa-node013.zilliz.local   <none>           <none>
benchmark-no-clean-98c2h-1-milvus-rootcoord-79cc745989-hktnt      1/1     Running     0          4h37m   10.97.12.139   qa-node015.zilliz.local   <none>           <none>
benchmark-no-clean-98c2h-1-minio-0                                1/1     Running     0          4h37m   10.97.16.120   qa-node013.zilliz.local   <none>           <none>
benchmark-no-clean-98c2h-1-minio-1                                1/1     Running     0          4h37m   10.97.12.140   qa-node015.zilliz.local   <none>           <none>
benchmark-no-clean-98c2h-1-minio-2                                1/1     Running     0          4h37m   10.97.12.141   qa-node015.zilliz.local   <none>           <none>
benchmark-no-clean-98c2h-1-minio-3                                1/1     Running     0          4h37m   10.97.12.142   qa-node015.zilliz.local   <none>           <none>
benchmark-no-clean-98c2h-1-pulsar-autorecovery-755cfc69f6-z4q5n   1/1     Running     0          4h37m   10.97.4.221    qa-node002.zilliz.local   <none>           <none>
benchmark-no-clean-98c2h-1-pulsar-bastion-6ccc487c78-fs9fs        1/1     Running     0          4h37m   10.97.8.134    qa-node006.zilliz.local   <none>           <none>
benchmark-no-clean-98c2h-1-pulsar-bookkeeper-0                    1/1     Running     0          4h37m   10.97.4.222    qa-node002.zilliz.local   <none>           <none>
benchmark-no-clean-98c2h-1-pulsar-bookkeeper-1                    1/1     Running     0          4h36m   10.97.4.223    qa-node002.zilliz.local   <none>           <none>
benchmark-no-clean-98c2h-1-pulsar-bookkeeper-2                    1/1     Running     0          4h35m   10.97.13.131   qa-node010.zilliz.local   <none>           <none>
benchmark-no-clean-98c2h-1-pulsar-broker-85d9c84779-f4blq         1/1     Running     1          4h37m   10.97.8.135    qa-node006.zilliz.local   <none>           <none>
benchmark-no-clean-98c2h-1-pulsar-proxy-66645b8c59-cjtj5          2/2     Running     0          4h37m   10.97.11.192   qa-node009.zilliz.local   <none>           <none>
benchmark-no-clean-98c2h-1-pulsar-zookeeper-0                     1/1     Running     0          4h37m   10.97.9.95     qa-node007.zilliz.local   <none>           <none>
benchmark-no-clean-98c2h-1-pulsar-zookeeper-1                     1/1     Running     0          4h37m   10.97.9.96     qa-node007.zilliz.local   <none>           <none>
benchmark-no-clean-98c2h-1-pulsar-zookeeper-2                     1/1     Running     0          4h36m   10.97.9.97     qa-node007.zilliz.local   <none>           <none>
benchmark-no-clean-98c2h-1-pulsar-zookeeper-metadata-92nls        0/1     Completed   0          4h37m   10.97.8.133    qa-node006.zilliz.local   <none>           <none>

Steps To Reproduce

1、create collation
2、build index of ivf_sq8
3、insert 100 million vectors <- raise error
4、flush collection <- raise error

Anything else?

No response

The text was updated successfully, but these errors were encountered:

yanliang567 · 2022-01-04T02:39:39Z

/assign @sunby

sunby · 2022-01-17T02:08:07Z

@yanliang567 This issue is duplicaged with #14747

sunby · 2022-01-17T02:08:34Z

/assign @wangting0128
Please verify it.

wangting0128 added kind/bug Issues or changes related a bug priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. test/benchmark benchmark test labels Dec 31, 2021

wangting0128 assigned yanliang567 Dec 31, 2021

yanliang567 added this to the 2.0.0-GA milestone Jan 2, 2022

yanliang567 assigned xiaofan-luan and unassigned yanliang567 Jan 2, 2022

wangting0128 changed the title ~~[Bug]: [benchmark][standalone] milvus insert timeout and flush raise error: err: context deadline exceeded~~ [Bug]: [benchmark][cluster] milvus insert timeout and flush raise error: err: context deadline exceeded Jan 4, 2022

sre-ci-robot assigned sunby Jan 4, 2022

yanliang567 added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jan 4, 2022

sre-ci-robot assigned wangting0128 Jan 17, 2022

yanliang567 unassigned sunby and xiaofan-luan Jan 17, 2022

wangting0128 closed this as completed Jan 19, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug]: [benchmark][cluster] milvus insert timeout and flush raise error: err: context deadline exceeded #14707

[Bug]: [benchmark][cluster] milvus insert timeout and flush raise error: err: context deadline exceeded #14707

wangting0128 commented Dec 31, 2021 •

edited

Loading

yanliang567 commented Jan 4, 2022

sunby commented Jan 17, 2022

sunby commented Jan 17, 2022

[Bug]: [benchmark][cluster] milvus insert timeout and flush raise error: err: context deadline exceeded #14707

[Bug]: [benchmark][cluster] milvus insert timeout and flush raise error: err: context deadline exceeded #14707

Comments

wangting0128 commented Dec 31, 2021 • edited Loading

Is there an existing issue for this?

Environment

Current Behavior

Expected Behavior

Steps To Reproduce

Anything else?

yanliang567 commented Jan 4, 2022

sunby commented Jan 17, 2022

sunby commented Jan 17, 2022

wangting0128 commented Dec 31, 2021 •

edited

Loading