UMTS - Troubleshooting with Wireshark

• Wireshark Menu Bar
  • Preferences for Gb, Iu-ps, Iu-cs, Iub, S1, Gn/Gp/Gi ivnterfaces
  • Verification of Settings of important Telecom Protocol's

• Start Tracing IP-connection (Network Interface Card's)
  • Capture Interface(s) for Monitoring

• Wireshark Preference Settings
  
  • Timestamp options, Colorizing two different IP (e.g. FTP) Connections in same Logfile
    
• Adding customized Column's for enhanced analysis
  
  • Text-export or CSV-export allows further post-processing in Excel

• Export of Logfiles as Textfile, CSV-file, PCAP-file
• Reviewing important TCP/IP Fundamentals
  • Overview of IP, TCP and UDP header
  • TCP Slow Start and Congestion Avoidance
  • TCP Connection Establishment & Release
    • 3-way handshake, Receive Window Size of Client is critical, Reset a Connection
  • Overview of important TCP-Parameters
    • Window Size, Round Trip Time, Maximum Segment Size, Maximum Transfer Unit, Socket Parameter
  • Bandwidth Delay Product: Throughput = Window Size / RTT
    • Specifics of a Wireless System like GPRS, UMTS or LTE:
      - Throughput should be preferably limited by the UE's Capability's
      - The E-t-E-RTT cannot fall below a minimum value but increases easily
      - Window Size of TCP-client (e.g. laptop) should be set to a certain value matching the RTT of the system
  • Concurrent Download and Upload leads to lower throughput
    • Upload throttles down the download due to Windows-PPP issue (no prioritization of TCP-ACK's),
      Linux computer are able to prioritize the sending of TCP-ACK's before TCP-data are sent
  • Impact of Duplicate ACK's & Fast Retransmissions on Subscriber Throughput
  • Pro's and Con's of Selective Acknowledgments
    • Issue of lower layer retransmissions while Selective ACK's invoke TCP-Retransmissions resulting in “double” resp. unnecessary retransmissions, Advantage when UTRAN or PCU do not stall the forwarding of TCP-ACK's to the TCP-Server and thus allow out-of-sequence delivery of TCP-ACK's if they hang in retransmissions due to RLC-AM
      A TCP aware UTRAN or PCU would be beneficial for the throughput if TCP-ACK's are handled preferred
• Quick Logfile Analysis using “Expert Info”
  • Analysis of Example Log's

For every new TCP/IP Packet the Client sends a Duplicate ACK pointing out that a certain older TCP/IP-packet is still missing. All the newer incoming TCP/IP-packets have to be buffered until the very missing/dropped TCP/IP Packet is re-transmitted by the Server and properly received by the Client.

• Issues with Buffer Size in Client and Network Nodes are discussed
• IO-Graph in Wireshark (for quick throughput analysis)
• Detailed Throughput and RTT Analysis
  • Determining Slow Start and Retransmission using various TCP Stream Graph's

Troubleshooting of RAN and Core Errors

• Failure & Drops in GPRS/EGPRS
  • Filtering in Wireshark

• GPRS Connection Hang-up with Drop

• Low Throughput in EGPRS
• EGPRS Throughput per cell/BVCI, faulty GPRS Dial-up software (dashboard issue with LCP protocol)
• GPRS Suspend after 3G?2G HO but no Suspend ACK
• Attach Accept but no Attach Complete from UE
  • Ciphering Key mismatch between UE and SGSN

Failure and Drops in UMTS

• Core Network causes Duplicate ACK's and Fast Retransmissions
  • Out of Sequence Delivery of TCP-Packets on Gn/Gp- or Gi-interface
    • UE sends Duplicates ACK's in uplink; UTRAN assures In-Sequence Delivery through RLC-AM and HARQ;

• GGSN discards sometimes IP-packets
  • Delta-measurement between Gi (incoming) and Gn (outgoing) of IP-Packets; Internet Server may stick with Congestion Avoidance only after TCP-Retransmission due to packet loss
• RLC-AM In-Sequence and Out-of-Sequence Delivery Configuration
  • For Rel. 99 bearers, RLC-AM out-of-sequence delivery is counter-productive as it causes Duplicate ACK's. Bearers mapped on Rel. 5/6 HS-DSCH or Rel. 6 E-DCH provide almost in-sequence deliver due to HARQ underneath unless during HSPA Serving Cell Changes and if RLC-AM is configured for out-of-sequence delivery!
• SRNC can delay TCP/IP packets mapped on HS-DSCH unnecessary
  • Iub HS-DSCH Frame Protocol (user plane) trace required to compare with Iu-ps user plane timing
• Modem/Data-card hangs-up (UE software bug)
  • One or more TCP Retransmissions for same TCP-Packet but no ACK from TCP-client, then UE performs sudden GPRS ATTACH due to Switch-off-on, PDP Context Activation and “normal” download resumes

Failure and Drops in LTE

• S1-MME Signaling Issues
  • UE lost in E-UTRAN (uplink drop)

• UE drops due to downlink out-of-sync
  • RRC Re-establishment Procedure Scenario
• Tracking Area Update after failed RRC Re-establishment
  • Network Paging in case of downlink Data Delivery, As long as the TCP Timers in UE/Client are not fired, TCP triggers the Connection Establishment with E-UTRAN again (same in GPRS and UMTS), UE keeps its IP-address after drop or being in Idle Mode until Detach or “Deactivate PDP Context”
• X2AP Signaling Issues
  • Handover Signaling
    • Meaning of snStatusTransfer, Why are there duplicate X2AP messages visible?

• How to trace the complete signaling of a single UE on S1-MME?

SCTP does not distinguish the Signaling for different Users/UE's between eNodeB and MME. Thus S1AP in eNodeB assigns upon connection establishment (e.g. with ATTACH REQUEST) a unique number called eNB-UE-S1AP-ID. In response the MME allocates his unique number MME-UE-S1AP-ID and mirrors back the number which eNodeB has assigned for the call. Thus from that moment on the UE/User is uniquely identified by two numbers and eNodeB refers to the MME-ID when sending some S1AP-message to the MME where as the MME refers to eNB-ID when sending some message for the specific UE to eNodeB.

• How to identify the User Plane Connection (S1-U) of eNB ? SGW for a certain UE?

Low Throughput Troubleshooting in HSPA Networks

• Iub Flow Control Issues
  
  • Bad HS-DSCH Flow Control

• E-DCH FP throttles uplink Throughput
  
  • Example for a bunch of corrupted E-DCH frames. The last correct received packet on Iub is having sequence number 291210. After that it takes 2.5s until the next non-corrupted packet is received on Iub:

• Combined Trace Analysis of Air-interface (Uu) and Wireshark
  
  • HS-SCCH Scheduling Analysis

• Serving Grant Monitoring for low HSUPA Throughput
  • SIR-Target in UTRAN and UE TX Power limitation

• TCP/IP ACK counting on Client Laptop
  • (typically a Windows PC ack's only every 2nd TCP-frame) downlink packets arrive too slow although Client-PC ack's nicely every 2nd TCP-frame <=> self-clocking principle of the TCP-connection. A TCP-Server in Slow Start mode would double the number of TCP-frames for every received TCP-ACK