Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Check the driver re-plugin in dynamic mode #544

Closed
powerkimhub opened this issue Dec 6, 2021 · 4 comments
Closed

Check the driver re-plugin in dynamic mode #544

powerkimhub opened this issue Dec 6, 2021 · 4 comments

Comments

@powerkimhub
Copy link
Member

  • 현재 server 운영 중에 driver의 동적 plugin은 정상 동작하여 활용 중에 있음
  • 동작 중인 driver patch 후 다시 plugin은 정상 동작하지 않고 있음( reported by @jihoon-seo )
@powerkimhub
Copy link
Member Author

[cf] https://go.dev/src/plugin/plugin.go

// When a plugin is first opened, the init functions of all packages not
// already part of the program are called. The main function is not run.
// A plugin is only initialized once, and cannot be closed.
//
// Currently plugins are only supported on Linux, FreeBSD, and macOS.

@powerkimhub
Copy link
Member Author

powerkimhub commented Mar 2, 2022 •
edited
Loading

[현황]

  • new driver(new.so) 동적 적용을 위해서 memory에 load된 old driver(old.so)의 close(unload)가 필요한데,
    • 현재는 제공하지 않음.
  • load된 library(old.so)의 close를 제공한다고 해도 old driver로 동작 중인 상태이므로 불안정해질 수 있음

[Plan]

  • 사용중인 old driver는 사용될 때까지 유지하고, new driver를 신규 등록해서 신규 사용에만 사용하도록 제공
  • 사용자가 old driver 사용하지 않음을 판단하고, 사용자에 의해 관련 정보를 삭제

[Try-1] shared library build 시 name 변경으로 new plugin으로 초기화 유도 => Fail

  • ref) https://stackoverflow.com/questions/70033200/plugin-already-loaded

  • 결과: old driver 기능을 개선한 new driver 신규 등록시, plugin already loaded 에러 발생

    [CLOUD-BARISTA].[ERROR]: 2022-03-02 02:27:48 CloudDriverHandler_dynamic.go:60, github.com/cloud-barista/cb-spider/cloud-control-manager.getCloudDriver() - plugin.Open: plugin.Open("/tmp/mini/cb-spider/cloud-driver-libs/mock-driver-v1.1.so"): plugin already loaded (previous failure)

  • 분석: new.so library file은 open은 시도되지만, shared library Symbol 충돌로 load 되지 못함

    • library name 보다는 내부 함수, 변수 정의 등이 중요

    • 참고: old driver(v1.0)과 new driver(v1.1) 비교 정보

      $ readelf -aW mock-driver-v1.0.so |grep CloudDriver
  4301: 000000000061c4c0     0 OBJECT  GLOBAL DEFAULT   27 plugin/unnamed-7d7c2352821deff2075e80f998bc8eb0b0751afb.CloudDriver
  7297: 000000000061c4c0     0 OBJECT  GLOBAL DEFAULT   27 plugin/unnamed-7d7c2352821deff2075e80f998bc8eb0b0751afb.CloudDriver

$ readelf -aW mock-driver-v1.1.so |grep CloudDriver
  4301: 000000000061c4c0     0 OBJECT  GLOBAL DEFAULT   27 plugin/unnamed-7d7c2352821deff2075e80f998bc8eb0b0751afb.CloudDriver
  7298: 000000000061c4c0     0 OBJECT  GLOBAL DEFAULT   27 plugin/unnamed-7d7c2352821deff2075e80f998bc8eb0b0751afb.CloudDriver

[위 출력 정보 Header 참고]
Symbol table '.dynsym' contains 9528 entries:
   Num:    Value          Size Type    Bind   Vis      Ndx Name
     0: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT  UND
     1: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND seteuid@GLIBC_2.2.5 (2)
     2: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND abort@GLIBC_2.2.5 (2)
     3: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND __fprintf_chk@GLIBC_2.3.4 (3)

[추가 참고]
$  readelf -aW mock-driver-v1.0.so |grep CreateVPC
  4089: 000000000027ec20   197 FUNC    GLOBAL DEFAULT   12 github.com/cloud-barista/cb-spider/cloud-control-manager/cloud-driver/drivers/mock/connect.(*MockConnection).CreateVPCHandler
  7857: 000000000027c8e0  1170 FUNC    GLOBAL DEFAULT   12 github.com/cloud-barista/cb-spider/cloud-control-manager/cloud-driver/drivers/mock/resources.(*MockVPCHandler).CreateVPC
  3339: 000000000027c8e0  1170 FUNC    LOCAL  DEFAULT   12 local.github.com/cloud-barista/cb-spider/cloud-control-manager/cloud-driver/drivers/mock/resources.(*MockVPCHandler).CreateVPC
  3353: 000000000027ec20   197 FUNC    LOCAL  DEFAULT   12 local.github.com/cloud-barista/cb-spider/cloud-control-manager/cloud-driver/drivers/mock/connect.(*MockConnection).CreateVPCHandler
  3592: 00000000004f17c0    32 OBJECT  LOCAL  DEFAULT   21 github.com/cloud-barista/cb-spider/cloud-control-manager/cloud-driver/drivers/mock/connect.(*MockConnection).CreateVPCHandler.stkobj
  3626: 00000000004f1980    32 OBJECT  LOCAL  DEFAULT   21 github.com/cloud-barista/cb-spider/cloud-control-manager/cloud-driver/drivers/mock/resources.(*MockVPCHandler).CreateVPC.stkobj
  6192: 000000000027c8e0  1170 FUNC    GLOBAL DEFAULT   12 github.com/cloud-barista/cb-spider/cloud-control-manager/cloud-driver/drivers/mock/resources.(*MockVPCHandler).CreateVPC
  9338: 000000000027ec20   197 FUNC    GLOBAL DEFAULT   12 github.com/cloud-barista/cb-spider/cloud-control-manager/cloud-driver/drivers/mock/connect.(*MockConnection).CreateVPCHandler


$ readelf -aW mock-driver-v1.1.so |grep CreateVPC
  4089: 000000000027ecc0   197 FUNC    GLOBAL DEFAULT   12 github.com/cloud-barista/cb-spider/cloud-control-manager/cloud-driver/drivers/mock/connect.(*MockConnection).CreateVPCHandler
  7857: 000000000027c920  1253 FUNC    GLOBAL DEFAULT   12 github.com/cloud-barista/cb-spider/cloud-control-manager/cloud-driver/drivers/mock/resources.(*MockVPCHandler).CreateVPC
  3339: 000000000027c920  1253 FUNC    LOCAL  DEFAULT   12 local.github.com/cloud-barista/cb-spider/cloud-control-manager/cloud-driver/drivers/mock/resources.(*MockVPCHandler).CreateVPC
  3353: 000000000027ecc0   197 FUNC    LOCAL  DEFAULT   12 local.github.com/cloud-barista/cb-spider/cloud-control-manager/cloud-driver/drivers/mock/connect.(*MockConnection).CreateVPCHandler
  3592: 00000000004f17a0    32 OBJECT  LOCAL  DEFAULT   21 github.com/cloud-barista/cb-spider/cloud-control-manager/cloud-driver/drivers/mock/connect.(*MockConnection).CreateVPCHandler.stkobj
  3626: 00000000004f4520    56 OBJECT  LOCAL  DEFAULT   21 github.com/cloud-barista/cb-spider/cloud-control-manager/cloud-driver/drivers/mock/resources.(*MockVPCHandler).CreateVPC.stkobj
  6193: 000000000027c920  1253 FUNC    GLOBAL DEFAULT   12 github.com/cloud-barista/cb-spider/cloud-control-manager/cloud-driver/drivers/mock/resources.(*MockVPCHandler).CreateVPC
  9339: 000000000027ecc0   197 FUNC    GLOBAL DEFAULT   12 github.com/cloud-barista/cb-spider/cloud-control-manager/cloud-driver/drivers/mock/connect.(*MockConnection).CreateVPCHandler

    • 참고

@powerkimhub
Copy link
Member Author

[Try-2] new driver source plugin path 변경 (ex: mock-plugin => mock-plugin2)

[시험방법]

$ tree mock-plugin               # create: cloud-driver-libs/mock-driver-v1.0.so
mock-plugin
|-- MockDriver-lib.go
`-- build_driver_lib.sh

$ tree mock-plugin2              # create: cloud-driver-libs/mock-driver-v1.0.so
mock-plugin2/
|-- MockDriver-lib.go
`-- build_driver_lib-new.sh

[시험결과]

  • new driver library file Open은 plugin already loaded 에러 없이 성공이지만,
  • 메모리에 이미 적재된 old driver의 기존 공유 객체들이 호출되어 변경 내용이 반영되지 않음
  • 운영 중인 기존 driver의 일부 기능만 개선하여,
  • 서버 동작 중에 on-line patch를 제공하기 위해서는 별도의 방안이 필요할 것으로 보임

@powerkimhub
Copy link
Member Author

powerkimhub commented Mar 2, 2022 •
edited
Loading

[결론/운영가이드]

  • 현재 Go runtime은 동일 plugin에 대해서 1회만 초기화 되며, close가 불가능함(On-line시)
  • Go plugin 기능에서 close 또는 reopen 등 메모리에 load된 plugin에 대한 update 방법이 제공되지 않는다면,
  • 운영 중인 기존 driver의 on-line patch는 불가(또는 특수한 방법 도안이 필요함)
  • 반드시 On-line patch가 필요하지 않다면, driver patch => server restart 방법을 사용하고,
  • 꼭 필요하다면, 신규 드라이버처럼 소스 트리를 신규 구성하여 신규 드라이버처럼 on-line 등록 및 활용 가능
    • on-line(dynamic) re-plugin은 불가

@jihoon-seo

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@powerkimhub