Need a Route prober to confirm effectiveness of VirtualService changes

[tcnghia]

Expected Behavior

When we report that a Route is Ready, it should be accessible through Ingress.

Actual Behavior

However, after we create a VirtualService there is some delay before it routes are working and currently the v1alpha3 API doesn't provide a good way to know when (istio/istio#882).

As a stop-gap we could add a special route in our VirtualService that always return the Route/VirtualSerice version. Then when Routes change we can add a prober Pod to probe all ingress pods to make sure that all ingress Pods are seeing updated version of the Routes and mark them with IngressReady condition.

This way users (and integration tests) can start relying on Route.Status more. When istio/istio#882 is fixed we simply replacing these probers by the newly provided mechanism.

Additional Info

This will also remove some of the Polling/Probing noted in #1178

[lichuqiang]

I've seen a lot of folks (including myself) claiming that they need a way to confirm a VirtualService is indeed ready to serve. But I failed to see progress in istio/istio#882.

So maybe it's time to revisit this? I'd like to pick this up if you've no objection.

[lichuqiang]

/assign

[lichuqiang]

I've got some POC code for IngressGateway side probe here in my repo: https://github.com/lichuqiang/serving/tree/prober

Will consider to post an official PR later if I'm sure we can get rid of this issue on activator side.

[tcnghia]

/cc @greghaynes