Garage & Updates

Skype Public API has two built-in methods for 3rd party applications to communicate with each-other: streams and datagrams. As The API reference states, datagrams are a non-guaranteed packet delivery method, vaguely similar to UDP while streams are a continuous, guaranteed delivery and packet order method, similar to TCP/IP connections. Except that there has been a long-standing problem with API streams apparently not really delivering the packets in the same order.

This phenomenon was particularly easy to reproduce with short streams of small packets. Pumping a string of 10 packets of 100 bytes each, pretty much always resulted in packets coming out from the stream in reverse order.

After long search, we finally found the root cause of this problem and while it appears to be too difficult to fix it in a hurry, we can at least offer firstly an explanation of what is going on and secondly a (hopefully temporary) workaround.

]]>Firstly, the good news is – the API itself is working fine. The problem occurs when the receiver part of an application uses Skype4Com library. A Skype4Com receiver works by handling OnApplicationReceived events. On this event, the handler uses IStream.Read method to retrieve the data from the stream and then process it.

The “occasionally random packet order” phenomenon is caused by two facts. Firstly, IStream.Read is for practical purposes not instantaneous. Secondly, the event handler is re-entrant. When a string of packets arrive, the next packet may trigger OnApplicationReceived event while the previous packet is still in process of being read and processed. This causes the event handler to essentially go into recursion. And as with all recursions, the processing out of it will occur in reverse order.

For purpose of visualization, lets take following example.

A stream of five packets is sent out from transmitter, packets are numbered 1, 2, 3, 4 and 5. Now, lets examine what may well happen on receiver end.

1. First packet arrives. OnApplicationReceived event is fired.
2. Application handles the event, reads data and processes it. Everything is still fine.
3. Second packet arrives. Again, OnApplicationReceived event is fired.
4. Application handles the event.. starts reading the data from stream, but alas..
5. Third packet arrives before the reading/proccessing of the second packet is complete.
6. Your application launches its event handler once more, going into recursion.
7. 4th packet arrives, recursion is now 3 instances deep.
8. Processing of packet no. 4 gets finished and is processed.
9. Processing of packet no. 3 gets finished and is processed.
10. Processing of packet no. 2 gets finished and is processed. The event handler is now out of recursion.
11. Packet no. 5 is received, event handler gets fired for the last time.

The apparent order of packets from receiver looks like this: 1, 4, 3, 2, 5.

Now the obvious solution to solve this problem would be to implement a packet-by-packet protocol in your application. Send one packet at a time from transmitter, report back with an ACK message from receiver, then send next packet. But hey, where’s the fun in that and wouldn’t it make streams as such pretty much useless? After all, their only excuse and advantage over datagrams is supposed to be the guaranteed order, guaranteed delivery being mostly myth in any case.

Another (and somewhat more challenging) method is to make your !OnApplicationReceived event handler re-entry compliant. To do this you would have to detect when the handler goes into recursion and then dynamically buffer the received data until the handler digs itself out of recursion. Then process the data out of your buffer in correct order and clear the buffer. Until next recursion.

As writing re-entry compliant handlers is somewhat less trivial than your normal linear programming, here’s two examples (Delphi and VBScript).

//---------------------------------------------------------------
// This example uses TStrings list as dynamic buffer.
// Note that you need to initialize both Buffer and RecursionLevel variables.

Var	Buffer: TStrings;

	RecursionLevel: Integer;

Procedure TForm1.SkypeApplicationReceiving (

	Sender: TObject; const 

	pApp: IApplication; const 

	pStreams: IApplicationStreamCollection);

Var S 		: String;

    I, Z 	: Integer;

Begin

 if pStreams.Count > 0 Then

 Begin

// Exiting handler if the stram contains no data,

  if pStreams[1].DataLength = 0 Then Exit;

 

  Inc(RecursionLevel);

	

// Appending empty string slot at the end of the buffer 

// and storing it's position in Z 

  Buffer.Append('');

  Z := Buffer.Count - 1;

 

// Reading data out of stream. The following is the most likely place 

// your handler will stall and cause recursive re-entries

  S := pStreams.Item[1].Read;

  Inc(ReadCounter);

  Buffer[Z] := S;

// This is where we process the data. First we check if we are in recursion (RecursionLevel > 1)

// If we are in recursion, we do nothing - just accumulate received data in dynamic buffer.

// When we are out of recursion tho, the data gets processed out of Buffer in FIFO order.

// After each processed string, corresponding Buffer item is deleted.

  if RecursionLevel = 1 Then

  Begin

    Z := Buffer.Count - 1;

    For I := 0 to Z do

    Begin

// This is where the data processing should go. 

// In this example, we just log 1st 5 characters and recursion depth.

      Log('SenderID: ' + Copy(Buffer[0], 1, 5) + ' Recursion depth: ' + IntToStr(Z-I));

      Buffer.Delete(0);

    End;

  End;

  Dec(RecursionLevel);

 End;

End;

NB! Note that if the handler above goes really deep into recursion then you will eventually end up with stack overflow. If you start getting stack overflows with the example above, try running it without Delphi IDE.

Here is the VBScript example:

    Dim arrBuffer
    Dim nLevel : nLevel = 0

    Public Sub Skype_ApplicationReceiving(ByRef aApp, ByVal aStreams)

      If aStreams.Count Then
        nLevel = nLevel + 1
        If nLevel = 1 Then
          arrBuffer = Array()
          ReDim arrBuffer(1)
        Else
          ReDim PRESERVE arrBuffer(nLevel)
        End If

        arrBuffer(nLevel-1) = aStreams.Item(1).Read
        nLevel = nLevel - 1

        If nLevel=0 Then
          For i=0 to UBound(arrBuffer)-1
            WScript.Echo arrBuffer(i)
          Next
          arrBuffer = Null
        End If
      End If

    End Sub