Bulk Transport API and Protocols over UDT

Programmers' Guide 

(for btap-scho version 0.3)

  1. Introduction
  2. Bulk transport API
  3. Hello, World!
  4. Using FAST and GTP
  5. Errors
  6. Socket options
  7. Related links

1. Introduction

The bulk transport API is a specification drafted by the Bulk Transport Working Group at Internet2, which defines a set of functions for performing bulk data transfer.  The API closely resembles that of the 4.4BSD socket interface.  So, programmers who are familiar with socket programming should feel right at home when using the API. In contrast to the traditional socket interface which uses TCP/UDP as the transport, the bulk transport API is intended to be layered on some advanced transport protocols that can deliver better performance in networks with high bandwidth-delay products.

This implementation of the API (btap-scho) is based on the UDT library.  From a programmers' point of view, there are two differences between this API and that of UDT.  First, this API is compatible to the C programming language (note: UDT is written in C++).  "Traditional" socket programs written in C can therefore be modified to use the UDT library with little effort.  Second, programs that use the API may be "linked" with any data transfer libraries that conform to the API specification, which can bring better flexibility and facilitate experiments and performance optimization.

2. Bulk transport API

2.1 Data structures

btap-scho defines two data structures, X_SOCKET and X_STATS.  X_SOCKET is the handle of a socket, whose function is similar to that of the traditional socket descriptor.  The structure includes several fields; but programmers may access only x_stats and error in their codes since the other fields are for internal use and subject to change.

typedef struct X_SOCKET_t {
        /* private; not to be accessed by user */
        int sock;                       /* to be typecasted to UDTSOCKET */
        int type;                       /* SOCK_STREAM or SOCK_DGRAM */
        unsigned int select_mark;       /* conditions for select() */
        unsigned int select_result;     /* result from select() */

        /* group of sockets, used in GTP */
        void *grp;                      /* pointer to the GTP object */
        struct X_SOCKET_t **skts;       /* X_SOCKET's in the group */
        int grp_size;                   /* group size */

        /* public */
        X_STATS x_stats;                /* stats of this socket */
        int error;                      /* default to 0 */
} X_SOCKET;

An X_STATS structure is included in each X_SOCKET, which records the statistics related to that socket instance.  Programmers may retrieve an updated version of x_stats by calling x_sockstats().

typedef struct X_STATS_t {
        /* incremental statistics since last call to x_sockstats() */
        double send_rate;       /* in Mbps */
        double recv_rate;       /* in Mbps */
        float p_retransmit;     /* percentage of retransmission */
        double rtt;             /* in milliseconds */

        /* statistics since socket creation */
        unsigned long long sentbytes;   /* in bytes */
        unsigned long long recvbytes;   /* in bytes */
        double avg_sent_rate;
        double avg_recv_rate;
        float avg_p_retransmit;
} X_STATS;

2.2 API

The API specifies the following functions.  The purpose of each function might best be realized by looking at the "counterpart" in the traditional socket interface; they are essentially analogous.  You may refer to the API specification for the detailed usage of the functions.

Function Return type "Socket" counterpart
Socket creation/deletion
x_socket(int socktype, X_SOCKET *skt) X_SOCKET * socket()
x_close(X_SOCKET *skt) int close()
Connection establishment
x_connect(X_SOCKET *skt, const struct sockaddr *addr, int len) int connect()
x_bind(X_SOCKET *skt, const struct sockaddr *addr, int len) int bind()
x_listen(X_SOCKET *skt, int backlog) int listen()
x_accept(X_SOCKET *skt, struct sockaddr *addr, int *len) X_SOCKET * accept()
Socket options
x_setsockopt(X_SOCKET *skt, int optname, void *optval, int optlen) int setsockopt()
x_getsockopt(X_SOCKET *skt, int optname, void *optval, int *optlen) int getsockopt()
Socket statistics
x_sockstats(X_SOCKET *skt) X_STATS * -
Errors
x_sockerror(X_SOCKET *skt) int errno
x_errortext(int err) const char * strerror()
Data transfer
x_sendfile(X_SOCKET *skt, int fd, off_t offset, size_t size) size_t sendfile()
x_send(X_SOCKET *skt, const void *buf, size_t len) size_t write()
x_recvfile(X_SOCKET *skt, int fd, off_t offset, size_t size) size_t -
x_recv(X_SOCKET *skt, void *buf, size_t len) size_t read()
Select family
x_selectmark(X_SOCKET *skt, int mark) int FD_SET()
x_selecttest(X_SOCKET *skt) int FD_ISSET()
x_select(int len, X_SOCKET **skts, struct timeval *timeout) int select()

Please refer to the original API specification about the detailed usage of these functions; complete lists of errors and optnames are included below for reference.

3. Hello, World!

Any program using the API has to include <c_udt.h>, and add the -I and -L flags during the compilation to specify the path to the patched and compiled UDT (please refer to the compilation instructions for details).  The following program is a typical "client" program, which sends a message "Hello, World!" to a "server" program.

/* client.c, sending a "Hello, World!" to server.c */
#include <c_udt.h>

#define PORT 9000
#define STRLEN 200

int main(void) {

    X_SOCKET *sock;
    struct sockaddr_in serv_addr;
    char buffer[STRLEN];

    strcpy(buffer, "Hello, World!");
    sock = x_socket(SOCK_DGRAM, NULL);

    if (x_sockerror(X_LAST_ERROR)) {
        printf("Cannot get x_socket: %s\n",
               x_errortext(x_sockerror(X_LAST_ERROR)));
        exit(1);
    }

    serv_addr.sin_family = AF_INET;
    serv_addr.sin_port = htons(PORT);
    inet_pton(AF_INET, "127.0.0.1", &serv_addr.sin_addr);
    memset(&(serv_addr.sin_zero), '\0', 8);

    if (x_connect(sock, (struct sockaddr *)&serv_addr, sizeof(serv_addr))) {
        printf("Cannot connect: %s\n", x_errortext(x_sockerror(sock)));
        exit(1);
    }

    if (x_send(sock, buffer, STRLEN) != STRLEN) {
        printf("Send failed: %s\n", x_errortext(x_sockerror(sock)));
        exit(1);
    }

    x_close(sock);
    return 0;
}

The server program is as below.

/* server.c, receiving a "Hello, World!" from client.c */
#include <c_udt.h>

#define PORT 9000
#define STRLEN 200

int main(void) {

    X_SOCKET *sock, *newsock;
    struct sockaddr_in my_addr, remote_addr;
    int len;
    char buffer[200];

    sock = x_socket(SOCK_DGRAM, NULL);

    if (x_sockerror(X_LAST_ERROR)) {
        printf("Cannot get x_socket: %s\n",
               x_errortext(x_sockerror(X_LAST_ERROR)));
        exit(1);
    }

    my_addr.sin_family = AF_INET;
    my_addr.sin_port = htons(PORT);
    my_addr.sin_addr.s_addr = INADDR_ANY;
    memset(&(my_addr.sin_zero), '\0', 8);

    if (x_bind(sock, (struct sockaddr *)&my_addr, sizeof(my_addr)) ||
        x_listen(sock, 10)) {
        printf("Cannot bind/listen: %s\n",
               x_errortext(x_sockerror(sock)));
        exit(1);
    }
    newsock = x_accept(sock, (struct sockaddr *)&remote_addr, &len);

    printf("New connection: %s:%d\n", inet_ntoa(remote_addr.sin_addr),
           ntohs(remote_addr.sin_port));

    if (x_recv(newsock, buffer, STRLEN) != STRLEN) {
        printf("Receive failed: %s\n",
               x_errortext(x_sockerror(newsock)));
        exit(1);
    }
    printf("Server has got a message: %s\n", buffer);

    x_close(newsock);
    x_close(sock);
    return 0;
}

As shown in the above example, it is very much similar to traditional socket programming.  You may now read the API specification to learn about the usage of the other API calls.

4. Using FAST and GTP

btap-scho includes three additional function calls that create sockets based on the FAST and GTP congestion control mechanisms.  If you would like to use FAST, simply replace your x_socket() call with x_fastsocket().
X_SOCKET *x_fastsocket(int socktype, X_SOCKET *skt)

A socket created by x_fastsocket() can be closed by using x_close().

Using GTP is a little more complicated.  btap-scho provides the following two functions for creating/closing GTP sockets.

X_SOCKET **x_gtpsocket(int socktype, int n, int linkspeed)
int x_gtpclose(X_SOCKET **skts, int n)

x_gtpsocket() creates a group of n sockets that share a link with a speed of linkspeed (in Mbps), and returns an array of pointers, each pointing to a new X_SOCKET structure.  The array of sockets can be closed by calling x_gtpclose().  It should be noted that in a multipoint-to-point scenario (that's where GTP can be used), the receiver should create a group of n sockets (each connecting to a distant sender), while each sender  should create a group of only 1 socket (the sender should still call x_gtpsocket() instead of x_socket() in order to use the GTP congestion control mechanism.).

5. Errors

You may call x_sockerror() and x_errortext() to retrieve the error numbers and their text descriptions, respectively.  If an X_SOCKET is inputted to these functions, the API would return the socket-specific errors, which are listed below.  If X_LAST_ERROR is specified, the API would try to return the current system-wide error which is managed by UDT.  Due to this reason, you may want to use x_errortext() instead of x_sockerror() for X_LAST_ERROR in order to obtain the description of an error.  For details on UDT's errors, please refer to the UDT error code list.

Error number Name of error Description
7001 X_UNKNOWN Unknown error
7002 X_BADTYPE Invalid socket type
7003 X_MALLOC malloc() failed
7004 X_CCFAILED Error in customizing congestion control
7005 X_INVAL Invalid parameters
7006 X_BADSOCK Invalid socket
7007 X_NONBLOCKCLOSE Non-blocking socket with UDT_LINGER set
7008 X_NOEXIST Server does not exist
7009 X_REJECTED Connection attempt rejected
7010 X_CONNECTED Socket already connected
7011 X_BOUND Socket already bound to a certain address
7012 X_BADADDR Invalid or unavailable address
7013 X_NOBIND Socket is not bound
7014 X_NOTSTREAM Socket is not of type SOCK_STREAM
7015 X_NOACCEPT Socket is not listening
7016 X_NONBLOCKACCEPT Non-blocking socket and no connection available
7017 X_DTRACE -DTRACE was not enabled in Makefile
7018 X_NOTCONN Socket is not connected
7019 X_BROKEN Connection broken
7020 X_WOULDBLOCK Socket would block
7021 X_BADOPT Option value or option length is invalid
7022 X_BADFD Error using mmap() on fd
7023 X_SEEK Error seeking/writing on fd
7024 X_OVERLAP An overlapped recv is in progress

6. Socket options

The optnames accepted by get/setsockopt() and their definitions are listed below.

Optname Description Type of optval Note
X_MSS maximum packet size int in bytes
X_SNDSYN blocking mode of send int 0 => blocking, !=0 => nonblocking
X_RCVSYN blocking mode of recv int 0 => blocking, !=0 => nonblocking
X_FC maximum flow window size int in number of packets
X_SNDBUF UDT max. buffer size for send int in bytes
X_RCVBUF UDT max. buffer size for recv int in bytes
X_LINGER linger time on close struct linger see bits/socket.h
X_UDPSNDBUF UDP buffer size for send int in bytes
X_UDPRCVBUF UDP buffer size for recv int in bytes
X_QTTL TTL of messages waiting for retransmission int in msec

7. Related links