Uploaded image for project: 'DC/OS'
  1. DC/OS
  2. DCOS_OSS-5138

Flaky test - download_dcos_installer fails with "Operation too slow"

        Details

        • Epic Link:
        • Sprint:
          DevProd Team Sprint 47
        • Story Points:
          1
        • Transition Due Date:
        • Parent Initiative:
          D2IQ-52727 - [DC/OS] Enhance the DC/OS Testing Utility Library to enable Local Development
        • Product (inherited):
          DC/OS

          Description

          This is download_dcos_installer from DC/OS Launch.
          I have linked issues that I think may be related.

          See https://teamcity.mesosphere.io/viewLog.html?buildId=1866761&buildTypeId=DcOs_Enterprise_Test_Inte_AwsOnpremWStaticBackendAndSecurityStrictGroup4&tab=buildLog&_focus=929

          [04:02:06][Step 8/11] < HTTP/1.1 200 OK
[04:02:06][Step 8/11] < x-amz-id-2: mcV77CD1h+yCsAPb7cwSn8cYoXSdCkH7ULGBizLGb/f5/UtmCah8vn9gpJort78PvoKa19/SqdQ=
[04:02:06][Step 8/11] < x-amz-request-id: F5510059FF8F6604
[04:02:06][Step 8/11] < Date: Tue, 07 May 2019 04:02:07 GMT
[04:02:06][Step 8/11] < x-amz-replication-status: COMPLETED
[04:02:06][Step 8/11] < Last-Modified: Tue, 07 May 2019 01:33:39 GMT
[04:02:06][Step 8/11] < ETag: "1ca69ed9733610a84e79253dd773076e"
[04:02:06][Step 8/11] < Cache-Control: no-cache
[04:02:06][Step 8/11] < x-amz-version-id: BtTK__NGlcN.Fzyjvk55kEVcIKng_WqZ
[04:02:06][Step 8/11] < Accept-Ranges: bytes
[04:02:06][Step 8/11] < Content-Type: binary/octet-stream
[04:02:06][Step 8/11] < Content-Length: 1465837659
[04:02:06][Step 8/11] < Server: AmazonS3
[04:02:06][Step 8/11] < 
[04:02:06][Step 8/11] { [data not shown]
[04:03:12][Step 8/11] * Operation too slow. Less than 100000 bytes/sec transferred the last 60 seconds
[04:03:12][Step 8/11] * Closing connection 20
[04:03:12][Step 8/11] curl: (28) Operation too slow. Less than 100000 bytes/sec transferred the last 60 seconds
[04:03:12][Step 8/11] [2019-05-07 04:03:12,717|dcos_launch.platforms.onprem|ERROR]: Download failed!
[04:03:12][Step 8/11] Traceback (most recent call last):
[04:03:12][Step 8/11]   File "dcos_launch/platforms/onprem.py", line 184, in download_dcos_installer
[04:03:12][Step 8/11]   File "site-packages/dcos_test_utils/ssh_client.py", line 57, in command
[04:03:12][Step 8/11]   File "subprocess.py", line 708, in run
[04:03:12][Step 8/11] subprocess.CalledProcessError: Command '['ssh', '-p', '22', '-oConnectTimeout=10', '-oStrictHostKeyChecking=no', '-oUserKnownHostsFile=/dev/null', '-oLogLevel=ERROR', '-oBatchMode=yes', '-oPasswordAuthentication=no', '-oControlPath=/teamcity/temp/buildTmp/tmp5j01j8pv', '-oControlMaster=auto', 'centos@52.12.26.113', 'curl', '-fLsSv', '--retry', '20', '-Y', '100000', '-y', '60', '--create-dirs', '-o', '/home/centos/dcos_generate_config.sh', 'https://s3.amazonaws.com/downloads.mesosphere.io/dcos-enterprise/testing/pull/5614/commit/70b849bcc84be36121ba8233244246c0a065cdfe/dcos_generate_config.ee.sh']' returned non-zero exit status 28
[04:03:12][Step 8/11] [3168] Failed to execute script cli
[04:03:12][Step 8/11] Traceback (most recent call last):
[04:03:12][Step 8/11]   File "dcos_launch/cli.py", line 113, in <module>
[04:03:12][Step 8/11]   File "dcos_launch/cli.py", line 105, in main
[04:03:12][Step 8/11]   File "dcos_launch/cli.py", line 71, in do_main
[04:03:12][Step 8/11]   File "dcos_launch/onprem.py", line 189, in install_dcos
[04:03:12][Step 8/11]   File "dcos_launch/platforms/onprem.py", line 134, in prepare_bootstrap
[04:03:12][Step 8/11]   File "site-packages/retrying.py", line 49, in wrapped_f
[04:03:12][Step 8/11]   File "site-packages/retrying.py", line 212, in call
[04:03:12][Step 8/11]   File "site-packages/retrying.py", line 247, in get
[04:03:12][Step 8/11]   File "site-packages/six.py", line 693, in reraise
[04:03:12][Step 8/11]   File "site-packages/retrying.py", line 200, in call
[04:03:12][Step 8/11]   File "dcos_launch/platforms/onprem.py", line 184, in download_dcos_installer
[04:03:12][Step 8/11]   File "site-packages/dcos_test_utils/ssh_client.py", line 57, in command
[04:03:12][Step 8/11]   File "subprocess.py", line 708, in run
[04:03:12][Step 8/11] subprocess.CalledProcessError: Command '['ssh', '-p', '22', '-oConnectTimeout=10', '-oStrictHostKeyChecking=no', '-oUserKnownHostsFile=/dev/null', '-oLogLevel=ERROR', '-oBatchMode=yes', '-oPasswordAuthentication=no', '-oControlPath=/teamcity/temp/buildTmp/tmp5j01j8pv', '-oControlMaster=auto', 'centos@52.12.26.113', 'curl', '-fLsSv', '--retry', '20', '-Y', '100000', '-y', '60', '--create-dirs', '-o', '/home/centos/dcos_generate_config.sh', 'https://s3.amazonaws.com/downloads.mesosphere.io/dcos-enterprise/testing/pull/5614/commit/70b849bcc84be36121ba8233244246c0a065cdfe/dcos_generate_config.ee.sh']' returned non-zero exit status 28
[04:03:12][Step 8/11] Process exited with code 255
[04:03:13][Step 8/11] Process exited with code 255
[04:03:13][Step 8/11] Step Launch Cluster (Command Line) failed

            Attachments

              Activity

                People

                • Assignee:
                  cartergawron Carter Gawron
                  Reporter:
                  adamdangoor Adam Dangoor (Inactive)
                  Team:
                  Dev Prod Team
                  Watchers:
                  Adam Dangoor (Inactive), Alex Rukletsov (Inactive), Branden Rolston, Carter Gawron, Charles Provencher, Daniel Baker, Deepak Goel, Gastón Kleiman (Inactive), Gauri Powale, Gilbert Song, Greg Mann, Gustav Paul, Jan-Philip Gehrcke (Inactive), Jonathan Giddy, Julian Gieseke, Justin Barrick, Mergebot, Orlando Hohmeier (Inactive), Philip Norman (Inactive), Senthil Kumaran, Sergey Urbanovich (Inactive), Till Toenshoff, Tim Weidner (Inactive)
                • Watchers:
                  23 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: