Uploaded image for project: 'DC/OS'
  1. DC/OS
  2. DCOS_OSS-697

Spartan CPU demands are Too High at High Scale

    Details

    • Sprint:
      Networking Team 1.9 Sprint 8, Networking Team 1.10 Sprint 1, Networking Team 1.10 Sprint 4
    • Story Points:
      5

      Description

      On a 1k agent cluster with 45k tasks running with no real network activity other than DCOS components, Spartan was consistently the 2nd highest entry on `top` running around 80%.  This is on the leader.mesos node.

      According to Deepak Goel navstar and minuteman refresh DNS entries of spartan every 30 secs.

      This seems like a high cpu demand for a background process.

      It should be noted that the reason we were debugging the resource demands of this node is we were getting slow response times with HTTP 500 errors and timeouts.  The demands of this process could exasperate the ability of other critical components on the node.

      top snap

      13484 root 20 0 4737312 3.083g 20608 S 162.7 21.0 1583:21 /opt/mesosphere/packages/mesos--aec8113f8cb8a1d4a6a8e4b2df621c1391b95050/bin/mesos-master 
      10833 root 20 0 3768384 384460 4644 S 83.5 2.5 662:06.36 /opt/mesosphere/packages/spartan--5ba4837686ec300782bbff81e7452d4322ad524e/spartan/erts-8.2.2/bin/beam.smp -Bd -A 100 -K true -P 256000 -C multi_time_warp -c true -- -root /opt/mesosphere/packages/spartan--5ba4837686ec300782bbff81e7452d4322ad524e/spartan -progname opt/mesosphere/active/spartan+
      6654 nobody 20 0 150920 10412 5848 R 36.5 0.1 0:04.24 nginx: worker process

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                sergeyurbanovich Sergey Urbanovich (Inactive)
                Reporter:
                ken Ken Sipe (Inactive)
              • Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Zendesk Support