PR #8280 - 07-15 11:03

Job: hypershift
FAILURE
← Back to Test Grid

Test Summary

122
Total Tests
81
Passed
21
Failed
20
Skipped

Failed Tests

TestAutoscaling
34m34.39s
hypershift_framework.go:530: Successfully created hostedcluster e2e-clusters-z2s5s/autoscaling-l9mcb in 2m28s hypershift_framework.go:179: Summarizing unexpected conditions for HostedCluster autoscaling-l9mcb util.go:3158: Failed to wait for HostedCluster e2e-clusters-z2s5s/autoscaling-l9mcb to have valid conditions in 2s: context deadline exceeded eventually.go:225: observed *v1beta1.HostedCluster e2e-clusters-z2s5s/autoscaling-l9mcb invalid at RV 35427 after 2s: eventually.go:228: - incorrect condition: wanted ControlPlaneConnectionAvailable=True, got ControlPlaneConnectionAvailable=False: KASAccessFailed(Data plane to control plane connection is not available: no successful connectivity check recorded) eventually.go:228: - incorrect condition: wanted ClusterVersionProgressing=False, got ClusterVersionProgressing=True: ClusterOperatorsNotAvailable(Unable to apply 5.0.0-0.ci-2026-07-15-111627-test-ci-op-nwrlf9nh-latest: some cluster operators are not available) eventually.go:228: - incorrect condition: wanted AWSEndpointAvailable=True, got AWSEndpointAvailable=False: AWSError(operation error Route 53: ChangeResourceRecordSets, exceeded maximum number of attempts, 3, https response error StatusCode: 400, RequestID: 64782838-8692-40a9-93e1-a134ed12872d, api error Throttling: Rate exceeded) eventually.go:228: - incorrect condition: wanted ClusterVersionAvailable=True, got ClusterVersionAvailable=False: FromClusterVersion eventually.go:228: - incorrect condition: wanted ExternalDNSReachable=True, got ExternalDNSReachable=False: ExternalDNSHostNotReachable(lookup api-autoscaling-l9mcb.service.ci.hypershift.devcluster.openshift.com on 172.29.0.10:53: no such host) eventually.go:228: - incorrect condition: wanted ClusterVersionSucceeding=True, got ClusterVersionSucceeding=False: ClusterOperatorsNotAvailable(Cluster operators console, dns, image-registry, ingress, insights, kube-storage-version-migrator, monitoring, openshift-samples, service-ca are not available) hypershift_framework.go:349: skipping postTeardown() hypershift_framework.go:327: skipping teardown, already called
TestAutoscaling/ValidateHostedCluster
11m3.01s
util.go:305: Successfully waited for kubeconfig to be published for HostedCluster e2e-clusters-z2s5s/autoscaling-l9mcb in 1m3s util.go:322: Successfully waited for kubeconfig secret to have data in 0s eventually.go:105: Failed to get *v1.SelfSubjectReview: Post "https://api-autoscaling-l9mcb.service.ci.hypershift.devcluster.openshift.com:443/apis/authentication.k8s.io/v1/selfsubjectreviews": dial tcp: lookup api-autoscaling-l9mcb.service.ci.hypershift.devcluster.openshift.com on 172.30.0.10:53: no such host util.go:384: Failed to wait for a successful connection to the guest API server in 10m0s: context deadline exceeded
TestCreateCluster
43m11.55s
create_cluster_test.go:47: Sufficient zones available for InfrastructureAvailabilityPolicy HighlyAvailable hypershift_framework.go:518: failed to create cluster, tearing down: failed to create infra: failed to tag hosted zone: operation error Route 53: ChangeTagsForResource, https response error StatusCode: 400, RequestID: 3ce50278-5f74-436a-8c48-b558246e27b1, api error Throttling: Rate exceeded failed to get rate limit token, retry quota exceeded, 3 available, 5 requested fixture.go:417: Failed saving machine console logs; this is nonfatal: failed to get machine console logs: failed to get hostedcluster: hostedclusters.hypershift.openshift.io "create-cluster-27xth" not found fixture.go:425: Failed to dump machine journals; this is nonfatal: no SSH secret specified for cluster, cannot dump journals hypershift_framework.go:591: Destroyed cluster. Namespace: e2e-clusters-7bwnf, name: create-cluster-27xth
TestCreateClusterCustomConfig
43m18.94s
hypershift_framework.go:518: failed to create cluster, tearing down: failed to create infra: failed to create hosted zone: operation error Route 53: CreateHostedZone, https response error StatusCode: 400, RequestID: 79c98191-a315-475c-9bd8-b638feed34bc, api error Throttling: Rate exceeded failed to get rate limit token, retry quota exceeded, 2 available, 5 requested fixture.go:417: Failed saving machine console logs; this is nonfatal: failed to get machine console logs: failed to get hostedcluster: hostedclusters.hypershift.openshift.io "custom-config-rwg5m" not found fixture.go:425: Failed to dump machine journals; this is nonfatal: no SSH secret specified for cluster, cannot dump journals hypershift_framework.go:591: Destroyed cluster. Namespace: e2e-clusters-q77sg, name: custom-config-rwg5m
TestCreateClusterHABreakGlassCredentials
42m14.91s
hypershift_framework.go:530: Successfully created hostedcluster e2e-clusters-cfxbz/ha-break-glass-creds-8q5kv in 12m52s hypershift_framework.go:179: Summarizing unexpected conditions for HostedCluster ha-break-glass-creds-8q5kv util.go:3158: Failed to wait for HostedCluster e2e-clusters-cfxbz/ha-break-glass-creds-8q5kv to have valid conditions in 2s: context deadline exceeded eventually.go:225: observed *v1beta1.HostedCluster e2e-clusters-cfxbz/ha-break-glass-creds-8q5kv invalid at RV 49077 after 2s: incorrect condition: wanted ExternalDNSReachable=True, got ExternalDNSReachable=False: ExternalDNSHostNotReachable(lookup api-ha-break-glass-creds-8q5kv.service.ci.hypershift.devcluster.openshift.com on 172.29.0.10:53: no such host) hypershift_framework.go:349: skipping postTeardown() hypershift_framework.go:327: skipping teardown, already called
TestCreateClusterHABreakGlassCredentials/ValidateHostedCluster
10m57.01s
util.go:305: Successfully waited for kubeconfig to be published for HostedCluster e2e-clusters-cfxbz/ha-break-glass-creds-8q5kv in 57s util.go:322: Successfully waited for kubeconfig secret to have data in 0s eventually.go:105: Failed to get *v1.SelfSubjectReview: Post "https://api-ha-break-glass-creds-8q5kv.service.ci.hypershift.devcluster.openshift.com:443/apis/authentication.k8s.io/v1/selfsubjectreviews": dial tcp: lookup api-ha-break-glass-creds-8q5kv.service.ci.hypershift.devcluster.openshift.com on 172.30.0.10:53: no such host util.go:384: Failed to wait for a successful connection to the guest API server in 10m0s: context deadline exceeded
TestCreateClusterPrivate
24m4.16s
hypershift_framework.go:518: failed to create cluster, tearing down: failed to create infra: operation error Route 53: ListResourceRecordSets, exceeded maximum number of attempts, 21, https response error StatusCode: 400, RequestID: d53c8fbb-b1b8-4cf7-9a36-3a59c999cda3, api error Throttling: Rate exceeded fixture.go:417: Failed saving machine console logs; this is nonfatal: failed to get machine console logs: failed to get hostedcluster: hostedclusters.hypershift.openshift.io "private-skmjl" not found fixture.go:425: Failed to dump machine journals; this is nonfatal: no SSH secret specified for cluster, cannot dump journals hypershift_framework.go:591: Destroyed cluster. Namespace: e2e-clusters-54rh6, name: private-skmjl
TestCreateClusterProxy
1h28m16.74s
hypershift_framework.go:530: Successfully created hostedcluster e2e-clusters-vhk8v/proxy-ws994 in 25m23s hypershift_framework.go:179: Summarizing unexpected conditions for HostedCluster proxy-ws994 util.go:3158: Failed to wait for HostedCluster e2e-clusters-vhk8v/proxy-ws994 to have valid conditions in 2s: context deadline exceeded eventually.go:225: observed *v1beta1.HostedCluster e2e-clusters-vhk8v/proxy-ws994 invalid at RV 63350 after 2s: eventually.go:228: - incorrect condition: wanted ClusterVersionSucceeding=True, got ClusterVersionSucceeding=False: ClusterOperatorsNotAvailable(Cluster operators console, dns, image-registry, ingress, insights, kube-storage-version-migrator, monitoring, node-tuning, openshift-samples, service-ca, storage are not available) eventually.go:228: - incorrect condition: wanted ClusterVersionAvailable=True, got ClusterVersionAvailable=False: FromClusterVersion eventually.go:228: - incorrect condition: wanted ClusterVersionProgressing=False, got ClusterVersionProgressing=True: ClusterOperatorsNotAvailable(Unable to apply 5.0.0-0.ci-2026-07-15-111627-test-ci-op-nwrlf9nh-latest: some cluster operators are not available) eventually.go:228: - incorrect condition: wanted ControlPlaneConnectionAvailable=True, got ControlPlaneConnectionAvailable=Unknown: NoWorkerNodesAvailable(No worker nodes available to verify control plane connectivity) eventually.go:228: - incorrect condition: wanted ExternalDNSReachable=True, got ExternalDNSReachable=False: ExternalDNSHostNotReachable(lookup api-proxy-ws994.service.ci.hypershift.devcluster.openshift.com on 172.29.0.10:53: no such host) eventually.go:228: - incorrect condition: wanted DataPlaneConnectionAvailable=True, got DataPlaneConnectionAvailable=Unknown: NoWorkerNodesAvailable(No worker nodes available) hypershift_framework.go:349: skipping postTeardown() hypershift_framework.go:327: skipping teardown, already called
TestCreateClusterProxy/ValidateHostedCluster
10m39.01s
util.go:305: Successfully waited for kubeconfig to be published for HostedCluster e2e-clusters-vhk8v/proxy-ws994 in 39s util.go:322: Successfully waited for kubeconfig secret to have data in 0s eventually.go:105: Failed to get *v1.SelfSubjectReview: Post "https://api-proxy-ws994.service.ci.hypershift.devcluster.openshift.com:443/apis/authentication.k8s.io/v1/selfsubjectreviews": dial tcp: lookup api-proxy-ws994.service.ci.hypershift.devcluster.openshift.com on 172.30.0.10:53: no such host util.go:384: Failed to wait for a successful connection to the guest API server in 10m0s: context deadline exceeded
TestCreateClusterRequestServingIsolation
47m44.21s
requestserving.go:105: Created request serving nodepool clusters/6b1398d0e5-mgmt-reqserving-pk7hc requestserving.go:105: Created request serving nodepool clusters/6b1398d0e5-mgmt-reqserving-7v7ft requestserving.go:113: Created non request serving nodepool clusters/6b1398d0e5-mgmt-non-reqserving-8srgl requestserving.go:113: Created non request serving nodepool clusters/6b1398d0e5-mgmt-non-reqserving-9h9rh requestserving.go:113: Created non request serving nodepool clusters/6b1398d0e5-mgmt-non-reqserving-72vpm util.go:583: Successfully waited for 1 nodes to become ready for NodePool clusters/6b1398d0e5-mgmt-reqserving-pk7hc in 4m3s util.go:583: Successfully waited for 1 nodes to become ready for NodePool clusters/6b1398d0e5-mgmt-reqserving-7v7ft in 21s util.go:583: Successfully waited for 1 nodes to become ready for NodePool clusters/6b1398d0e5-mgmt-non-reqserving-8srgl in 0s util.go:583: Successfully waited for 1 nodes to become ready for NodePool clusters/6b1398d0e5-mgmt-non-reqserving-9h9rh in 6s util.go:583: Successfully waited for 1 nodes to become ready for NodePool clusters/6b1398d0e5-mgmt-non-reqserving-72vpm in 18s create_cluster_test.go:299: Sufficient zones available for InfrastructureAvailabilityPolicy HighlyAvailable hypershift_framework.go:518: failed to create cluster, tearing down: failed to create infra: operation error Route 53: ListResourceRecordSets, https response error StatusCode: 400, RequestID: 5f5fbf16-3ff4-4ded-b114-cb795ebe8807, api error Throttling: Rate exceeded failed to get rate limit token, retry quota exceeded, 2 available, 5 requested requestserving.go:132: Tearing down custom nodepool clusters/6b1398d0e5-mgmt-reqserving-pk7hc requestserving.go:132: Tearing down custom nodepool clusters/6b1398d0e5-mgmt-reqserving-7v7ft requestserving.go:132: Tearing down custom nodepool clusters/6b1398d0e5-mgmt-non-reqserving-8srgl requestserving.go:132: Tearing down custom nodepool clusters/6b1398d0e5-mgmt-non-reqserving-9h9rh requestserving.go:132: Tearing down custom nodepool clusters/6b1398d0e5-mgmt-non-reqserving-72vpm fixture.go:417: Failed saving machine console logs; this is nonfatal: failed to get machine console logs: failed to get hostedcluster: hostedclusters.hypershift.openshift.io "request-serving-isolation-lhpcp" not found fixture.go:425: Failed to dump machine journals; this is nonfatal: no SSH secret specified for cluster, cannot dump journals hypershift_framework.go:591: Destroyed cluster. Namespace: e2e-clusters-9w8wz, name: request-serving-isolation-lhpcp
TestKarpenter
44m29.49s
hypershift_framework.go:518: failed to create cluster, tearing down: failed to create infra: operation error Route 53: ChangeResourceRecordSets, https response error StatusCode: 400, RequestID: ecded8d9-24bc-4a6a-a964-13297a8bc610, api error Throttling: Rate exceeded failed to get rate limit token, retry quota exceeded, 4 available, 5 requested fixture.go:417: Failed saving machine console logs; this is nonfatal: failed to get machine console logs: failed to get hostedcluster: hostedclusters.hypershift.openshift.io "karpenter-jjjpz" not found fixture.go:425: Failed to dump machine journals; this is nonfatal: no SSH secret specified for cluster, cannot dump journals hypershift_framework.go:591: Destroyed cluster. Namespace: e2e-clusters-9rxvr, name: karpenter-jjjpz
TestKarpenterUpgradeControlPlane
1h59m58.74s
hypershift_framework.go:518: failed to create cluster, tearing down: failed to create infra: operation error Route 53: ListResourceRecordSets, exceeded maximum number of attempts, 21, https response error StatusCode: 400, RequestID: 6932be51-cb09-47c3-b98b-135268ac15dc, api error Throttling: Rate exceeded
TestKarpenterUpgradeControlPlane/Teardown
1h37m55.47s
fixture.go:417: Failed saving machine console logs; this is nonfatal: failed to get machine console logs: failed to get hostedcluster: hostedclusters.hypershift.openshift.io "karpenter-upgrade-control-plane-pvl6m" not found fixture.go:425: Failed to dump machine journals; this is nonfatal: no SSH secret specified for cluster, cannot dump journals hypershift_framework.go:576: Failed to destroy cluster, will retry: failed to destroy IAM: context canceled fixture.go:417: Failed saving machine console logs; this is nonfatal: failed to get machine console logs: failed to get hostedcluster: client rate limiter Wait returned an error: context canceled fixture.go:425: Failed to dump machine journals; this is nonfatal: no SSH secret specified for cluster, cannot dump journals hypershift_framework.go:589: Failed to destroy cluster: context canceled util.go:213: Failed to delete namespace: e2e-clusters-5v6nf, will retry: client rate limiter Wait returned an error: context canceled hypershift_framework.go:605: Failed to delete test namespace: failed to delete namespace: context canceled fixture.go:417: Failed saving machine console logs; this is nonfatal: failed to get machine console logs: failed to get hostedcluster: client rate limiter Wait returned an error: context canceled fixture.go:425: Failed to dump machine journals; this is nonfatal: no SSH secret specified for cluster, cannot dump journals
TestNodePool
0s
TestNodePool/HostedCluster0
31m42.03s
hypershift_framework.go:530: Successfully created hostedcluster e2e-clusters-725r8/node-pool-ww927 in 33s hypershift_framework.go:179: Summarizing unexpected conditions for HostedCluster node-pool-ww927 util.go:3158: Failed to wait for HostedCluster e2e-clusters-725r8/node-pool-ww927 to have valid conditions in 2s: context deadline exceeded eventually.go:225: observed *v1beta1.HostedCluster e2e-clusters-725r8/node-pool-ww927 invalid at RV 30379 after 2s: incorrect condition: wanted ExternalDNSReachable=True, got ExternalDNSReachable=False: ExternalDNSHostNotReachable(lookup api-node-pool-ww927.service.ci.hypershift.devcluster.openshift.com on 172.29.0.10:53: no such host) hypershift_framework.go:349: skipping postTeardown() hypershift_framework.go:327: skipping teardown, already called
TestNodePool/HostedCluster0/ValidateHostedCluster
10m51.01s
util.go:305: Successfully waited for kubeconfig to be published for HostedCluster e2e-clusters-725r8/node-pool-ww927 in 51s util.go:322: Successfully waited for kubeconfig secret to have data in 0s eventually.go:105: Failed to get *v1.SelfSubjectReview: Post "https://api-node-pool-ww927.service.ci.hypershift.devcluster.openshift.com:443/apis/authentication.k8s.io/v1/selfsubjectreviews": dial tcp: lookup api-node-pool-ww927.service.ci.hypershift.devcluster.openshift.com on 172.30.0.10:53: no such host eventually.go:105: Failed to get *v1.SelfSubjectReview: context deadline exceeded util.go:384: Failed to wait for a successful connection to the guest API server in 10m0s: context deadline exceeded
TestNodePool/HostedCluster2
34m47.44s
hypershift_framework.go:530: Successfully created hostedcluster e2e-clusters-5ffdk/node-pool-cbjkq in 1m52s hypershift_framework.go:179: Summarizing unexpected conditions for HostedCluster node-pool-cbjkq util.go:3158: Failed to wait for HostedCluster e2e-clusters-5ffdk/node-pool-cbjkq to have valid conditions in 2s: context deadline exceeded eventually.go:225: observed *v1beta1.HostedCluster e2e-clusters-5ffdk/node-pool-cbjkq invalid at RV 33340 after 2s: eventually.go:228: - incorrect condition: wanted ExternalDNSReachable=True, got ExternalDNSReachable=False: ExternalDNSHostNotReachable(lookup api-node-pool-cbjkq.service.ci.hypershift.devcluster.openshift.com on 172.29.0.10:53: no such host) eventually.go:228: - incorrect condition: wanted AWSEndpointAvailable=True, got AWSEndpointAvailable=False: AWSError(operation error Route 53: ChangeResourceRecordSets, exceeded maximum number of attempts, 3, https response error StatusCode: 400, RequestID: 539b808c-bbd6-43b1-95a3-dc2437bd604b, api error Throttling: Rate exceeded) hypershift_framework.go:349: skipping postTeardown() hypershift_framework.go:327: skipping teardown, already called
TestNodePool/HostedCluster2/ValidateHostedCluster
10m42.01s
util.go:305: Successfully waited for kubeconfig to be published for HostedCluster e2e-clusters-5ffdk/node-pool-cbjkq in 42s util.go:322: Successfully waited for kubeconfig secret to have data in 0s eventually.go:105: Failed to get *v1.SelfSubjectReview: Post "https://api-node-pool-cbjkq.service.ci.hypershift.devcluster.openshift.com:443/apis/authentication.k8s.io/v1/selfsubjectreviews": dial tcp: lookup api-node-pool-cbjkq.service.ci.hypershift.devcluster.openshift.com on 172.30.0.10:53: no such host eventually.go:105: Failed to get *v1.SelfSubjectReview: context deadline exceeded util.go:384: Failed to wait for a successful connection to the guest API server in 10m0s: context deadline exceeded
TestNodePoolAutoscalingScaleFromZero
25m33.24s
hypershift_framework.go:518: failed to create cluster, tearing down: failed to create infra: operation error Route 53: ListResourceRecordSets, exceeded maximum number of attempts, 21, https response error StatusCode: 400, RequestID: 4840eb62-ecf2-4180-aa7f-8a69fc85ca85, api error Throttling: Rate exceeded fixture.go:417: Failed saving machine console logs; this is nonfatal: failed to get machine console logs: failed to get hostedcluster: hostedclusters.hypershift.openshift.io "scale-from-zero-85jzk" not found fixture.go:425: Failed to dump machine journals; this is nonfatal: no SSH secret specified for cluster, cannot dump journals hypershift_framework.go:591: Destroyed cluster. Namespace: e2e-clusters-zb548, name: scale-from-zero-85jzk
TestUpgradeControlPlane
48m55.17s
control_plane_upgrade_test.go:25: Starting control plane upgrade test. FromImage: registry.build01.ci.openshift.org/ci-op-nwrlf9nh/release@sha256:622e3aad9f1316badf4e7c2069a15bb6e480d3a5783727abee78031623fec157, toImage: registry.build01.ci.openshift.org/ci-op-nwrlf9nh/release@sha256:52cfe6831dfc534ca617219f86a755d112ef5b7abd1c6b66fc8267c0df3fbdf4 hypershift_framework.go:530: Successfully created hostedcluster e2e-clusters-xsf6j/control-plane-upgrade-6r5kq in 31m14s hypershift_framework.go:179: Summarizing unexpected conditions for HostedCluster control-plane-upgrade-6r5kq util.go:3158: Failed to wait for HostedCluster e2e-clusters-xsf6j/control-plane-upgrade-6r5kq to have valid conditions in 2s: context deadline exceeded eventually.go:225: observed *v1beta1.HostedCluster e2e-clusters-xsf6j/control-plane-upgrade-6r5kq invalid at RV 73407 after 2s: eventually.go:228: - incorrect condition: wanted AWSEndpointAvailable=True, got AWSEndpointAvailable=False: AWSError(operation error Route 53: ChangeResourceRecordSets, exceeded maximum number of attempts, 3, https response error StatusCode: 400, RequestID: 96149482-9f2c-4411-9b19-7aeea646e993, api error Throttling: Rate exceeded) eventually.go:228: - incorrect condition: wanted ClusterVersionSucceeding=True, got ClusterVersionSucceeding=False: ClusterOperatorsNotAvailable(Cluster operators console, dns, image-registry, ingress, insights, kube-storage-version-migrator, monitoring, node-tuning, openshift-samples, service-ca, storage are not available) eventually.go:228: - incorrect condition: wanted ExternalDNSReachable=True, got ExternalDNSReachable=False: ExternalDNSHostNotReachable(lookup api-control-plane-upgrade-6r5kq.service.ci.hypershift.devcluster.openshift.com on 172.29.0.10:53: no such host) eventually.go:228: - incorrect condition: wanted ClusterVersionAvailable=True, got ClusterVersionAvailable=False: FromClusterVersion eventually.go:228: - incorrect condition: wanted ClusterVersionProgressing=False, got ClusterVersionProgressing=True: ClusterOperatorsNotAvailable(Unable to apply 5.0.0-0.ci-2026-07-10-071205: some cluster operators are not available) eventually.go:228: - incorrect condition: wanted DataPlaneConnectionAvailable=True, got DataPlaneConnectionAvailable=Unknown: NoWorkerNodesAvailable(No worker nodes available) hypershift_framework.go:349: skipping postTeardown() hypershift_framework.go:327: skipping teardown, already called
TestUpgradeControlPlane/ValidateHostedCluster
11m0.01s
util.go:305: Successfully waited for kubeconfig to be published for HostedCluster e2e-clusters-xsf6j/control-plane-upgrade-6r5kq in 1m0s util.go:322: Successfully waited for kubeconfig secret to have data in 0s eventually.go:105: Failed to get *v1.SelfSubjectReview: Post "https://api-control-plane-upgrade-6r5kq.service.ci.hypershift.devcluster.openshift.com:443/apis/authentication.k8s.io/v1/selfsubjectreviews": dial tcp: lookup api-control-plane-upgrade-6r5kq.service.ci.hypershift.devcluster.openshift.com on 172.30.0.10:53: no such host eventually.go:105: Failed to get *v1.SelfSubjectReview: context deadline exceeded util.go:384: Failed to wait for a successful connection to the guest API server in 10m0s: context deadline exceeded