PR #8546 - 05-27 10:19

Job: hypershift
FAILURE
← Back to Test Grid

Test Summary

337
Total Tests
278
Passed
15
Failed
44
Skipped

Failed Tests

TestNodePool
0s
TestNodePool/HostedCluster0
1h57m44.03s
hypershift_framework.go:530: Successfully created hostedcluster e2e-clusters-kg77l/node-pool-mbzgj in 2m52s hypershift_framework.go:179: Summarizing unexpected conditions for HostedCluster node-pool-mbzgj util.go:3135: Failed to wait for HostedCluster e2e-clusters-kg77l/node-pool-mbzgj to have valid conditions in 2s: context deadline exceeded eventually.go:225: observed *v1beta1.HostedCluster e2e-clusters-kg77l/node-pool-mbzgj invalid at RV 104019 after 2s: eventually.go:228: - incorrect condition: wanted Available=True, got Available=False: EtcdWaitingForQuorum(Waiting for etcd to reach quorum) eventually.go:228: - incorrect condition: wanted DataPlaneConnectionAvailable=Unknown, got DataPlaneConnectionAvailable=False: ReconciliationError(Couldn't list konnectivity-agent PODs in kube-system namespace: the server was unable to return a response in the time allotted, but may still be processing the request (get pods)) eventually.go:228: - incorrect condition: wanted ControlPlaneConnectionAvailable=Unknown, got ControlPlaneConnectionAvailable=False: KASAccessFailed(Data plane to control plane connection is not available: no successful connectivity check recorded) eventually.go:228: - incorrect condition: wanted EtcdAvailable=True, got EtcdAvailable=False: EtcdWaitingForQuorum(Waiting for etcd to reach quorum) eventually.go:228: - incorrect condition: wanted Degraded=False, got Degraded=True: UnavailableReplicas([azure-cloud-controller-manager deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-oauth-apiserver deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas]) hypershift_framework.go:349: skipping postTeardown() hypershift_framework.go:327: skipping teardown, already called
TestNodePool/HostedCluster0/Main
150ms
util.go:304: Successfully waited for kubeconfig to be published for HostedCluster e2e-clusters-kg77l/node-pool-mbzgj in 25ms util.go:321: Successfully waited for kubeconfig secret to have data in 25ms util.go:383: Successfully waited for a successful connection to the guest API server in 25ms
TestNodePool/HostedCluster0/Main/TestMirrorConfigs
45m9.12s
nodepool_mirrorconfigs_test.go:60: Starting test MirrorConfigsTest nodepool_test.go:369: Validating CAPI v1 condition message bubbling during machine provisioning nodepool_test.go:505: CAPI bubbling confirmed for AllNodesHealthy: Status=False, Reason=WaitingForNodeRef nodepool_test.go:505: CAPI bubbling confirmed for AllMachinesReady: Status=False, Reason=WaitingForInfrastructure eventually.go:259: Failed to get **v1.Node: the server was unable to return a response in the time allotted, but may still be processing the request (get nodes) eventually.go:259: Failed to get **v1.Node: Get "https://api-node-pool-mbzgj.aks-e2e.hypershift.azure.devcluster.openshift.com:443/api/v1/nodes?labelSelector=hypershift.openshift.io%2FnodePool%3Dnode-pool-mbzgj-test-mirrorconfigs": context deadline exceeded util.go:578: Failed to wait for 1 nodes to become ready for NodePool e2e-clusters-kg77l/node-pool-mbzgj-test-mirrorconfigs in 45m0s: context deadline exceeded eventually.go:384: observed invalid **v1.Node state after 45m0s eventually.go:401: - observed **v1.Node collection invalid: expected 1 nodes, got 0
TestNodePool/HostedCluster0/Main/TestNTOMachineConfigAppliedInPlace
45m9.35s
nodepool_nto_machineconfig_test.go:67: Starting test NTOMachineConfigRolloutTest nodepool_test.go:369: Validating CAPI v1 condition message bubbling during machine provisioning nodepool_test.go:505: CAPI bubbling confirmed for AllNodesHealthy: Status=False, Reason=WaitingForNodeRef nodepool_test.go:505: CAPI bubbling confirmed for AllMachinesReady: Status=False, Reason=WaitingForInfrastructure eventually.go:259: Failed to get **v1.Node: the server was unable to return a response in the time allotted, but may still be processing the request (get nodes) eventually.go:259: Failed to get **v1.Node: Get "https://api-node-pool-mbzgj.aks-e2e.hypershift.azure.devcluster.openshift.com:443/api/v1/nodes?labelSelector=hypershift.openshift.io%2FnodePool%3Dnode-pool-mbzgj-test-ntomachineconfig-inplace": context deadline exceeded util.go:578: Failed to wait for 2 nodes to become ready for NodePool e2e-clusters-kg77l/node-pool-mbzgj-test-ntomachineconfig-inplace in 45m0s: context deadline exceeded eventually.go:384: observed invalid **v1.Node state after 45m0s eventually.go:401: - observed **v1.Node collection invalid: expected 2 nodes, got 1
TestNodePool/HostedCluster0/Main/TestNTOMachineConfigGetsRolledOut
45m9.34s
nodepool_nto_machineconfig_test.go:67: Starting test NTOMachineConfigRolloutTest nodepool_test.go:369: Validating CAPI v1 condition message bubbling during machine provisioning nodepool_test.go:505: CAPI bubbling confirmed for AllNodesHealthy: Status=False, Reason=WaitingForNodeRef nodepool_test.go:505: CAPI bubbling confirmed for AllMachinesReady: Status=False, Reason=WaitingForInfrastructure eventually.go:259: Failed to get **v1.Node: the server was unable to return a response in the time allotted, but may still be processing the request (get nodes) eventually.go:259: Failed to get **v1.Node: Get "https://api-node-pool-mbzgj.aks-e2e.hypershift.azure.devcluster.openshift.com:443/api/v1/nodes?labelSelector=hypershift.openshift.io%2FnodePool%3Dnode-pool-mbzgj-test-ntomachineconfig-replace": context deadline exceeded util.go:578: Failed to wait for 2 nodes to become ready for NodePool e2e-clusters-kg77l/node-pool-mbzgj-test-ntomachineconfig-replace in 45m0s: context deadline exceeded eventually.go:384: observed invalid **v1.Node state after 45m0s eventually.go:401: - observed **v1.Node collection invalid: expected 2 nodes, got 0
TestNodePool/HostedCluster0/Main/TestNTOPerformanceProfile
45m9.31s
nodepool_nto_performanceprofile_test.go:59: Starting test NTOPerformanceProfileTest nodepool_test.go:369: Validating CAPI v1 condition message bubbling during machine provisioning nodepool_test.go:505: CAPI bubbling confirmed for AllNodesHealthy: Status=False, Reason=WaitingForNodeRef nodepool_test.go:505: CAPI bubbling confirmed for AllMachinesReady: Status=False, Reason=WaitingForInfrastructure eventually.go:259: Failed to get **v1.Node: the server was unable to return a response in the time allotted, but may still be processing the request (get nodes) eventually.go:259: Failed to get **v1.Node: Get "https://api-node-pool-mbzgj.aks-e2e.hypershift.azure.devcluster.openshift.com:443/api/v1/nodes?labelSelector=hypershift.openshift.io%2FnodePool%3Dnode-pool-mbzgj-test-ntoperformanceprofile": context deadline exceeded util.go:578: Failed to wait for 1 nodes to become ready for NodePool e2e-clusters-kg77l/node-pool-mbzgj-test-ntoperformanceprofile in 45m0s: context deadline exceeded eventually.go:384: observed invalid **v1.Node state after 45m0s eventually.go:401: - observed **v1.Node collection invalid: expected 1 nodes, got 0
TestNodePool/HostedCluster0/Main/TestNodePoolInPlaceUpgrade
54m28.44s
nodepool_upgrade_test.go:99: starting test NodePoolUpgradeTest nodepool_test.go:369: Validating CAPI v1 condition message bubbling during machine provisioning nodepool_test.go:505: CAPI bubbling confirmed for AllNodesHealthy: Status=False, Reason=WaitingForNodeRef nodepool_test.go:505: CAPI bubbling confirmed for AllMachinesReady: Status=False, Reason=WaitingForInfrastructure util.go:578: Successfully waited for 1 nodes to become ready for NodePool e2e-clusters-kg77l/node-pool-mbzgj-test-inplaceupgrade in 9m18.05s nodepool_test.go:429: Successfully waited for NodePool e2e-clusters-kg77l/node-pool-mbzgj-test-inplaceupgrade to have correct status in 25ms nodepool_upgrade_test.go:177: Validating all Nodes have the synced labels and taints nodepool_upgrade_test.go:180: Successfully waited for NodePool e2e-clusters-kg77l/node-pool-mbzgj-test-inplaceupgrade to have version 5.0.0-0.ci-2026-05-26-183200 in 50ms nodepool_upgrade_test.go:195: Validating NodesInfo is populated with version 5.0.0-0.ci-2026-05-26-183200 before upgrade nodepool_upgrade_test.go:196: Successfully waited for NodePool e2e-clusters-kg77l/node-pool-mbzgj-test-inplaceupgrade to have NodesInfo with version 5.0.0-0.ci-2026-05-26-183200 in 25ms nodepool_upgrade_test.go:218: Updating NodePool image. Image: registry.build01.ci.openshift.org/ci-op-kmqvdmpl/release@sha256:f6774081ceb8c63ee5bd0bec1b181521be15456de94204e134dfeb42deedc397 nodepool_upgrade_test.go:225: Successfully waited for NodePool e2e-clusters-kg77l/node-pool-mbzgj-test-inplaceupgrade to start the upgrade in 3.025s nodepool_upgrade_test.go:238: Failed to wait for NodePool e2e-clusters-kg77l/node-pool-mbzgj-test-inplaceupgrade to have version 5.0.0-0.ci-2026-05-27-101955-test-ci-op-kmqvdmpl-latest in 45m0s: context deadline exceeded eventually.go:225: observed *v1beta1.NodePool e2e-clusters-kg77l/node-pool-mbzgj-test-inplaceupgrade invalid at RV 57585 after 45m0s: eventually.go:228: - wanted version 5.0.0-0.ci-2026-05-27-101955-test-ci-op-kmqvdmpl-latest, got 5.0.0-0.ci-2026-05-26-183200 eventually.go:228: - incorrect condition: wanted UpdatingVersion=False, got UpdatingVersion=True: AsExpected(Nodepool update in progress. Target Config version: 36a60305. Total Nodes: 1. Upgraded: 0) nodepool_upgrade_test.go:238: *v1beta1.NodePool e2e-clusters-kg77l/node-pool-mbzgj-test-inplaceupgrade conditions: nodepool_upgrade_test.go:238: AutoscalingEnabled=False: AsExpected nodepool_upgrade_test.go:238: UpdateManagementEnabled=True: AsExpected nodepool_upgrade_test.go:238: ValidReleaseImage=True: AsExpected(Using release image: registry.build01.ci.openshift.org/ci-op-kmqvdmpl/release@sha256:f6774081ceb8c63ee5bd0bec1b181521be15456de94204e134dfeb42deedc397) nodepool_upgrade_test.go:238: ValidArchPlatform=True: AsExpected nodepool_upgrade_test.go:238: ReconciliationActive=True: ReconciliationActive(Reconciliation active on resource) nodepool_upgrade_test.go:238: SupportedVersionSkew=True: AsExpected(Release image version is valid) nodepool_upgrade_test.go:238: ValidMachineConfig=True: AsExpected nodepool_upgrade_test.go:238: UpdatingConfig=False: AsExpected nodepool_upgrade_test.go:238: UpdatingVersion=True: AsExpected(Nodepool update in progress. Target Config version: 36a60305. Total Nodes: 1. Upgraded: 0) nodepool_upgrade_test.go:238: ValidGeneratedPayload=True: AsExpected(Payload generated successfully) nodepool_upgrade_test.go:238: ReachedIgnitionEndpoint=True: AsExpected nodepool_upgrade_test.go:238: AllMachinesReady=True: AsExpected(All is well) nodepool_upgrade_test.go:238: AllNodesHealthy=True: AsExpected(All is well) nodepool_upgrade_test.go:238: ValidPlatformConfig=True: AsExpected(All is well) nodepool_upgrade_test.go:238: ValidTuningConfig=True: AsExpected nodepool_upgrade_test.go:238: UpdatingPlatformMachineTemplate=False: AsExpected nodepool_upgrade_test.go:238: AutorepairEnabled=False: AsExpected nodepool_upgrade_test.go:238: Ready=True: AsExpected
TestNodePool/HostedCluster0/Main/TestNodePoolPrevReleaseN3
45m9.31s
nodepool_prev_release_test.go:33: Starting NodePoolPrevReleaseCreateTest. nodepool_test.go:369: Validating CAPI v1 condition message bubbling during machine provisioning nodepool_test.go:505: CAPI bubbling confirmed for AllNodesHealthy: Status=False, Reason=WaitingForNodeRef nodepool_test.go:505: CAPI bubbling confirmed for AllMachinesReady: Status=False, Reason=WaitingForInfrastructure eventually.go:259: Failed to get **v1.Node: the server was unable to return a response in the time allotted, but may still be processing the request (get nodes) eventually.go:259: Failed to get **v1.Node: Get "https://api-node-pool-mbzgj.aks-e2e.hypershift.azure.devcluster.openshift.com:443/api/v1/nodes?labelSelector=hypershift.openshift.io%2FnodePool%3Dnode-pool-mbzgj-kmw7x": context deadline exceeded util.go:578: Failed to wait for 1 nodes to become ready for NodePool e2e-clusters-kg77l/node-pool-mbzgj-kmw7x in 45m0s: context deadline exceeded eventually.go:384: observed invalid **v1.Node state after 45m0s eventually.go:401: - observed **v1.Node collection invalid: expected 1 nodes, got 0
TestNodePool/HostedCluster0/Main/TestNodePoolReplaceUpgrade
45m9.41s
nodepool_upgrade_test.go:99: starting test NodePoolUpgradeTest nodepool_test.go:369: Validating CAPI v1 condition message bubbling during machine provisioning nodepool_test.go:505: CAPI bubbling confirmed for AllNodesHealthy: Status=False, Reason=WaitingForNodeRef nodepool_test.go:505: CAPI bubbling confirmed for AllMachinesReady: Status=False, Reason=WaitingForInfrastructure eventually.go:259: Failed to get **v1.Node: the server was unable to return a response in the time allotted, but may still be processing the request (get nodes) eventually.go:259: Failed to get **v1.Node: Get "https://api-node-pool-mbzgj.aks-e2e.hypershift.azure.devcluster.openshift.com:443/api/v1/nodes?labelSelector=hypershift.openshift.io%2FnodePool%3Dnode-pool-mbzgj-test-replaceupgrade": context deadline exceeded util.go:578: Failed to wait for 1 nodes to become ready for NodePool e2e-clusters-kg77l/node-pool-mbzgj-test-replaceupgrade in 45m0s: context deadline exceeded eventually.go:384: observed invalid **v1.Node state after 45m0s eventually.go:401: - observed **v1.Node collection invalid: expected 1 nodes, got 0
TestNodePool/HostedCluster0/Main/TestNodepoolMachineconfigGetsRolledout
45m9.43s
nodepool_machineconfig_test.go:54: Starting test NodePoolMachineconfigRolloutTest nodepool_test.go:369: Validating CAPI v1 condition message bubbling during machine provisioning nodepool_test.go:505: CAPI bubbling confirmed for AllNodesHealthy: Status=False, Reason=WaitingForNodeRef nodepool_test.go:505: CAPI bubbling confirmed for AllMachinesReady: Status=False, Reason=WaitingForInfrastructure eventually.go:259: Failed to get **v1.Node: the server was unable to return a response in the time allotted, but may still be processing the request (get nodes) eventually.go:259: Failed to get **v1.Node: Get "https://api-node-pool-mbzgj.aks-e2e.hypershift.azure.devcluster.openshift.com:443/api/v1/nodes?labelSelector=hypershift.openshift.io%2FnodePool%3Dnode-pool-mbzgj-test-machineconfig": context deadline exceeded util.go:578: Failed to wait for 1 nodes to become ready for NodePool e2e-clusters-kg77l/node-pool-mbzgj-test-machineconfig in 45m0s: context deadline exceeded eventually.go:384: observed invalid **v1.Node state after 45m0s eventually.go:401: - observed **v1.Node collection invalid: expected 1 nodes, got 0
TestNodePool/HostedCluster0/Main/TestRollingUpgrade
45m9.11s
nodepool_test.go:369: Validating CAPI v1 condition message bubbling during machine provisioning nodepool_test.go:505: CAPI bubbling confirmed for AllNodesHealthy: Status=False, Reason=WaitingForNodeRef nodepool_test.go:505: CAPI bubbling confirmed for AllMachinesReady: Status=False, Reason=WaitingForInfrastructure eventually.go:259: Failed to get **v1.Node: the server was unable to return a response in the time allotted, but may still be processing the request (get nodes) eventually.go:259: Failed to get **v1.Node: Get "https://api-node-pool-mbzgj.aks-e2e.hypershift.azure.devcluster.openshift.com:443/api/v1/nodes?labelSelector=hypershift.openshift.io%2FnodePool%3Dnode-pool-mbzgj-test-rolling-upgrade": context deadline exceeded util.go:578: Failed to wait for 2 nodes to become ready for NodePool e2e-clusters-kg77l/node-pool-mbzgj-test-rolling-upgrade in 45m0s: context deadline exceeded eventually.go:384: observed invalid **v1.Node state after 45m0s eventually.go:401: - observed **v1.Node collection invalid: expected 2 nodes, got 0
TestUpgradeControlPlane
33m13.87s
control_plane_upgrade_test.go:25: Starting control plane upgrade test. FromImage: registry.build01.ci.openshift.org/ci-op-kmqvdmpl/release@sha256:d48456f6889f1268389d1b066d562c9e9673b76e85bab0f21a1105b04d2f3f49, toImage: registry.build01.ci.openshift.org/ci-op-kmqvdmpl/release@sha256:f6774081ceb8c63ee5bd0bec1b181521be15456de94204e134dfeb42deedc397 hypershift_framework.go:530: Successfully created hostedcluster e2e-clusters-sm877/control-plane-upgrade-xt9jn in 3m21s hypershift_framework.go:179: Summarizing unexpected conditions for HostedCluster control-plane-upgrade-xt9jn util.go:3135: Successfully waited for HostedCluster e2e-clusters-sm877/control-plane-upgrade-xt9jn to have valid conditions in 25ms hypershift_framework.go:349: skipping postTeardown() hypershift_framework.go:327: skipping teardown, already called
TestUpgradeControlPlane/ValidateHostedCluster
18m13.01s
util.go:304: Successfully waited for kubeconfig to be published for HostedCluster e2e-clusters-sm877/control-plane-upgrade-xt9jn in 2m15.05s util.go:321: Successfully waited for kubeconfig secret to have data in 25ms eventually.go:105: Failed to get *v1.SelfSubjectReview: Post "https://api-control-plane-upgrade-xt9jn.aks-e2e.hypershift.azure.devcluster.openshift.com:443/apis/authentication.k8s.io/v1/selfsubjectreviews": EOF util.go:383: Successfully waited for a successful connection to the guest API server in 19.45s util.go:578: Successfully waited for 2 nodes to become ready in 6m57.075s util.go:615: Successfully waited for HostedCluster e2e-clusters-sm877/control-plane-upgrade-xt9jn to rollout in 8m30.05s util.go:3135: Successfully waited for HostedCluster e2e-clusters-sm877/control-plane-upgrade-xt9jn to have valid conditions in 50ms
TestUpgradeControlPlane/ValidateHostedCluster/EnsureNoCrashingPods
160ms
util.go:824: Container etcd in pod etcd-0 has a restartCount > 0 (1)