fix: pause lifecyle changes and add drained status

[kohlisid]

fixes #2027
fixes #1991

DrainedOnPause field is added here

1. It will be set when the pipeline is moved to a Paused state after a successful drain has been completed
2. It will always be reset whenever phase is moved back to running

[kohlisid]

1. Pipeline created before ISB Service, and desiredPhase=Paused

skohli@macos-JQWR9T560R numaflow % kubectl get pl
NAME              PHASE    VERTICES   AGE   MESSAGE
simple-pipeline   Failed   3          13s   ISB Service not found.

We keep the phase as failed and do not try to pause
#1992 (comment)

[codecov]

Codecov Report

Attention: Patch coverage is 53.84615% with 6 lines in your changes missing coverage. Please review.

Project coverage is 61.89%. Comparing base (8fc99bb) to head (ec3adf9).

Files with missing lines	Patch %	Lines
pkg/apis/numaflow/v1alpha1/pipeline_types.go	0.00%	4 Missing ⚠️
pkg/reconciler/pipeline/controller.go	77.77%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2028      +/-   ##
==========================================
+ Coverage   61.85%   61.89%   +0.04%     
==========================================
  Files         318      318              
  Lines       28979    28986       +7     
==========================================
+ Hits        17924    17940      +16     
+ Misses      10098    10096       -2     
+ Partials      957      950       -7     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[kohlisid]

 drainedOnPause: true
  lastUpdated: "2024-09-05T07:36:36Z"
  mapUDFCount: 1
  message: Pipeline paused
  observedGeneration: 5
  phase: Paused
  reduceUDFCount: 0
  sinkCount: 1
  sourceCount: 1
  udfCount: 1
  vertexCount: 3
skohli@macos-JQWR9T560R numaflow % kubectl get pl                        
NAME              PHASE    VERTICES   AGE   MESSAGE
simple-pipeline   Paused   3          8h    Pipeline paused

[kohlisid]

  lastUpdated: "2024-09-05T15:36:32Z"
  mapUDFCount: 1
  observedGeneration: 6
  phase: Running
  reduceUDFCount: 0
  sinkCount: 1
  sourceCount: 1
  udfCount: 1
  vertexCount: 3
skohli@macos-JQWR9T560R numaflow % kubectl get pl                        
NAME              PHASE     VERTICES   AGE   MESSAGE
simple-pipeline   Running   3          8h   

[kohlisid]

skohli@macos-JQWR9T560R numaflow %   kubectl patch pl simple-pipeline --type=merge --patch '{"spec": {"lifecycle": {"desiredPhase": "Paused"}}}' 

pipeline.numaflow.numaproj.io/simple-pipeline patched (no change)
skohli@macos-JQWR9T560R numaflow % kubectl get pl                                                                                               
NAME              PHASE    VERTICES   AGE   MESSAGE
simple-pipeline   Failed   3          11s   ISB Service not found.

[kohlisid]

skohli@macos-JQWR9T560R numaflow %   kubectl patch pl simple-pipeline --type=merge --patch '{"spec": {"lifecycle": {"desiredPhase": "Paused"}}}'

pipeline.numaflow.numaproj.io/simple-pipeline patched
skohli@macos-JQWR9T560R numaflow % kubectl get po                                                                                               
NAME                               READY   STATUS      RESTARTS   AGE
simple-pipeline-cln-ac4416-l6xds   0/1     Completed   0          24s
skohli@macos-JQWR9T560R numaflow % kubectl get pl
NAME              PHASE    VERTICES   AGE   MESSAGE
simple-pipeline   Failed   3          16s   ISB Service not found.

[kohlisid]

  drainedOnPause: false
  lastUpdated: "2024-09-05T16:48:56Z"
  mapUDFCount: 1
  observedGeneration: 4
  phase: Running
  reduceUDFCount: 0
  sinkCount: 1
  sourceCount: 1
  udfCount: 1
  vertexCount: 3
skohli@macos-JQWR9T560R numaflow % kubectl get pl                        
NAME              PHASE     VERTICES   AGE    MESSAGE
simple-pipeline   Running   3          6m9s  

[juliev0]

So, I just noticed that isLifecycleChange() == true in the case that the oldPhase was Paused but the desiredPhase==Running. I haven't read enough of the other code yet to confirm that that's okay here ?

[kohlisid]

That should be a resume pipeline case and I kept it like that for it to be handled by the
resume-logic
So the replicas will be scaled up there

[juliev0]

Suggested change

	// This should be happening in call cases to ensure a clean initialization regardless of the lifecycle phase
	// This should be happening in all cases to ensure a clean initialization regardless of the lifecycle phase

[juliev0]

I assume that's what you meant?

[whynowy]

Merging this, will address it in another PR.