Uploaded image for project: 'DC/OS'
  1. DC/OS
  2. DCOS_OSS-2041

Integration Test Failure: test_applications.py::test_if_marathon_app_can_be_deployed_with_mesos_containerizer

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: High
    • Resolution: Done
    • Affects Version/s: DC/OS 1.10.2
    • Fix Version/s: DC/OS 1.9.0, DC/OS 1.10.0
    • Component/s: platform
    • Labels:
      None
    • Sprint:
      Security Team Sprint 21

      Description

      Affects 1.10.2 (latest stable)

      Integration tests are failing on the latest master of dcos-docker (probably not due to dcos-docker changes):

      The log shows one 500 error from adminrouter/dcos-oauth on a marathon request:

      Jan 09 01:19:00 dcos-docker-master1 adminrouter.sh[2795]: 2018/01/09 01:19:00 [notice] 2800#0: *208 [lua] service.lua:188: resolve_via_mesos_state(): Resolved via Mesos state-summary, service id: `marathon`, client: 172.17.0.2, server: master.mesos, request: "GET /marathon/v2/apps/integration-test-fa3607fb68c94af2897624f8a5f6e31a?embed=apps.lastTaskFailure&embed=apps.counts HTTP/1.1", host: "leader.mesos"
      Jan 09 01:19:00 dcos-docker-master1 adminrouter.sh[2795]: 2018/01/09 01:19:00 [notice] 2800#0: *208 [lua] common.lua:121: validate_jwt(): UID from valid JWT: `albert@bekstil.net`, client: 172.17.0.2, server: master.mesos, request: "GET /marathon/v2/apps/integration-test-fa3607fb68c94af2897624f8a5f6e31a?embed=apps.lastTaskFailure&embed=apps.counts HTTP/1.1", host: "leader.mesos"
      Jan 09 01:19:00 dcos-docker-master1 dcos-oauth[1053]: time="2018-01-09T01:19:00Z" level=info msg="HTTP request received" method=GET uri="/acs/api/v1/users/albert@bekstil.net"
      Jan 09 01:19:00 dcos-docker-master1 java[1008]: [myid:] INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /127.0.0.1:42878
      Jan 09 01:19:00 dcos-docker-master1 java[1008]: [myid:] INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@883] - Processing ruok command from /127.0.0.1:42878
      Jan 09 01:19:00 dcos-docker-master1 java[1008]: [myid:] INFO [Thread-180:NIOServerCnxn@1044] - Closed socket connection for client /127.0.0.1:42878 (no session established for client)
      Jan 09 01:19:00 dcos-docker-master1 java[1008]: [myid:] INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /127.0.0.1:42880
      Jan 09 01:19:00 dcos-docker-master1 java[1008]: [myid:] INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@883] - Processing srvr command from /127.0.0.1:42880
      Jan 09 01:19:00 dcos-docker-master1 java[1008]: [myid:] INFO [Thread-181:NIOServerCnxn@1044] - Closed socket connection for client /127.0.0.1:42880 (no session established for client)
      Jan 09 01:19:00 dcos-docker-master1 marathon[1969]: [INFO] [01/09/2018 01:19:00.959] [kamon-akka.actor.default-dispatcher-9] [akka://kamon/user/kamon-jmx-reporter/kamon-jmx-actor] Message [akka.dispatch.ContextWrapper] from Actor[akka://kamon/user/kamon-jmx-reporter#-1910396533] to Actor[akka://kamon/user/kamon-jmx-reporter/kamon-jmx-actor#-1653294748] was not delivered. [9] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
      Jan 09 01:19:01 dcos-docker-master1 dcos-oauth[1053]: Recv loop terminated: err=read tcp 127.0.0.1:41578->127.0.0.1:2181: i/o timeout
      Jan 09 01:19:01 dcos-docker-master1 dcos-oauth[1053]: Send loop terminated: err=<nil>
      Jan 09 01:19:01 dcos-docker-master1 adminrouter.sh[2795]: 2018/01/09 01:19:01 [error] 2800#0: *208 [lua] open.lua:54: access_service_endpoint(): Unexpected response from IAM: 500 while sending to client, client: 172.17.0.2, server: master.mesos, request: "GET /marathon/v2/apps/integration-test-fa3607fb68c94af2897624f8a5f6e31a?embed=apps.lastTaskFailure&embed=apps.counts HTTP/1.1", host: "leader.mesos"
      Jan 09 01:19:01 dcos-docker-master1 dcos-oauth[1053]: time="2018-01-09T01:19:01Z" level=info msg="Zookeeper error" method=GET uri="/acs/api/v1/users/albert@bekstil.net"
      Jan 09 01:19:01 dcos-docker-master1 java[1008]: [myid:] WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@368] - caught end of stream exception
      EndOfStreamException: Unable to read additional data from client sessionid 0x160d87d083d0004, likely client has closed socket
      at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
      at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
      at java.lang.Thread.run(Thread.java:745)
      Jan 09 01:19:01 dcos-docker-master1 java[1008]: [myid:] INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1044] - Closed socket connection for client /127.0.0.1:41578 which had sessionid 0x160d87d083d0004
      Jan 09 01:19:01 dcos-docker-master1 nginx[2800]: dcos-docker-master1 nginx: 172.17.0.2 - - [09/Jan/2018:01:19:01 +0000] "GET /marathon/v2/apps/integration-test-fa3607fb68c94af2897624f8a5f6e31a?embed=apps.lastTaskFailure&embed=apps.counts HTTP/1.1" 500 199 "-" "python-requests/2.10.0"
      

      (full log attached)

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                adamdangoor Adam Dangoor (Inactive)
                Reporter:
                karl Karl Isenberg (Inactive)
                Team:
                DELETE Security Team
                Watchers:
                Adam Dangoor (Inactive), Jan-Philip Gehrcke (Inactive), Karl Isenberg (Inactive), Pawel Rozlach
              • Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Zendesk Support

                    NextupJiraPlusStatus

                    Error rendering 'slack.nextup.jira:nextup-jira-plus-status'. Please contact your JIRA administrators.