Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WriteAPI Out Of Memory #163

Closed
tietang opened this issue Jul 28, 2020 · 2 comments · Fixed by #175
Closed

WriteAPI Out Of Memory #163

tietang opened this issue Jul 28, 2020 · 2 comments · Fixed by #175
Assignees
Labels
bug Something isn't working
Milestone

Comments

@tietang
Copy link

tietang commented Jul 28, 2020

Each time the method is called, a new instance is created and a slice of writeAPIs is placed.
So as you run the slice the capacity gets larger and larger and it doesn't get released.


 func (c *clientImpl) WriteAPI(org, bucket string) api.WriteAPI {
	w := api.NewWriteAPI(org, bucket, c.httpService, c.options.writeOptions)
	c.writeAPIs = append(c.writeAPIs, w)
	return w
}

Here is the actual running pprof:


Showing nodes accounting for 6.56GB, 99.41% of 6.60GB total
Dropped 20 nodes (cum <= 0.03GB)
Showing top 10 nodes out of 18
      flat  flat%   sum%        cum   cum%
    6.56GB 99.41% 99.41%     6.57GB 99.55%  github.com/influxdata/influxdb-client-go/api.NewWriteAPI

  1. Each call creates a new one
  2. And it's put into the slice

Expected behavior:
Instances can be reused or resources can be released exactly when they are used

Actual behavior:
Cannot reuse and cannot release resources

Specifications:

  • Client Version: Cannot reuse and cannot release resources
  • InfluxDB Version: 2.0
  • Platform: mac OSX & CentOS

Pooling is recommended for managing WriteAPI.

@vlastahajek
Copy link
Contributor

vlastahajek commented Jul 28, 2020

@tietang, the expected best practice is to get WriteAPI for defined org and bucket and keep the instance.

However, for sure, the better implementation would be to keep WriteAPI in a map with the key org_bucket.

@vlastahajek vlastahajek added this to the v1.5.0 milestone Jul 28, 2020
@vlastahajek vlastahajek added the bug Something isn't working label Jul 28, 2020
@vlastahajek vlastahajek self-assigned this Jul 28, 2020
@tietang
Copy link
Author

tietang commented Aug 9, 2020

May be it is a good practice to use pools to manage writeAPI.
Here's my solution:

import "github.com/silenceper/pool"

config := pool.Config{
		InitialCap: 5,  //资源池初始连接数
		MaxIdle:    20, //最大空闲连接数
		MaxCap:     30, //最大并发连接数
		//Ping:       ping,
		//连接最大空闲时间,超过该时间的连接 将会关闭,可避免空闲时连接EOF,自动失效的问题
		IdleTimeout: 15 * time.Minute,
	}



	api := &WriteAPIPool{}
	config.Close = api.poolClose
	config.Factory = api.poolFactory
	p, err := pool.NewChannelPool(&config)



func poolFactory() (interface{}, error) {
	return influx.WriteAPI(), nil
}
func poolClose(v interface{}) (err error) {
	v.(api.WriteAPI).Close()
	return nil
}

vlastahajek added a commit to bonitoo-io/influxdb-client-go that referenced this issue Aug 11, 2020
vlastahajek added a commit to bonitoo-io/influxdb-client-go that referenced this issue Aug 11, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants