Ryan Dahl ryan@joyent.com - -
This document assumes you are familiar with how non-blocking socket I/O -is done in Unix. - -
The syscall
-select
- is available in Windows
-but select processing is O(n) in the number of file descriptors
-unlike the modern constant-time multiplexers like epoll which makes select
-unacceptable for high-concurrency servers.
-This document will describe how high-concurrency programs are
-designed in Windows.
-
-
-Instead of epoll -or -kqueue, -Windows has its own I/O multiplexer called -I/O completion ports -(IOCPs). IOCPs are the objects used to poll -overlapped I/O -for completion. IOCP polling is constant time (REF?). - -
-The fundamental variation is that in a Unix you generally ask the kernel to
-wait for state change in a file descriptor's readability or writablity. With
-overlapped I/O and IOCPs the programmers waits for asynchronous function
-calls to complete.
-For example, instead of waiting for a socket to become writable and then
-using send(2)
-on it, as you commonly would do in a Unix, with overlapped I/O you
-would rather WSASend()
-the data and then wait for it to have been sent.
-
-
Unix non-blocking I/O is not beautiful. A principle abstraction in Unix
-is the unified treatment of many things as files (or more precisely as file
-descriptors). write(2), read(2), and
-close(2) work with TCP sockets just as they do on regular
-files. Well—kind of. Synchronous operations work similarly on different
-types of file descriptors but once demands on performance drive you to world of
-O_NONBLOCK various types of file descriptors can act quite
-different for even the most basic operations. In particular,
-regular file system files do not support non-blocking operations.
-
-(Disturbingly no man page mentions this rather important fact.)
-
-For example, one cannot poll on a regular file FD for readability expecting
-it to indicate when it is safe to do a non-blocking read.
-
-Regular file are always readable and read(2) calls
-always have the possibility of blocking the calling thread for an
-unknown amount of time.
-
-
POSIX has defined an
- asynchronous interface for some operations but implementations for
-many Unixes have unclear status. On Linux the
-aio_* routines are implemented in userland in GNU libc using
-pthreads.
-io_submit(2)
-does not have a GNU libc wrapper and has been reported to be very slow and
- possibly blocking. Solaris
- has real kernel AIO but it's unclear what its performance
-characteristics are for socket I/O as opposed to disk I/O.
-Contemporary high-performance Unix socket programs use non-blocking
-file descriptors with a I/O multiplexer—not POSIX AIO.
-Common practice for accessing the disk asynchronously is still done using custom
-userland thread pools—not POSIX AIO.
-
-
Windows IOCPs does support both sockets and regular file I/O which
-greatly simplifies the handling of disks. For example,
-ReadFileEx()
-operates on both.
-As a first example let's look at how ReadFile() works.
-
-
-typedef void* HANDLE; - -BOOL ReadFile(HANDLE file, - void* buffer, - DWORD numberOfBytesToRead, - DWORD* numberOfBytesRead, - OVERLAPPED* overlapped); -- -
-The function has the possibility of executing the read synchronously
-or asynchronously. A synchronous operation is indicated by
-returning 0 and WSAGetLastError()
-returning WSA_IO_PENDING.
-When ReadFile() operates asynchronously the
-the user-supplied OVERLAPPED*
-is a handle to the incomplete operation.
-
-
-typedef struct {
- unsigned long* Internal;
- unsigned long* InternalHigh;
- union {
- struct {
- WORD Offset;
- WORD OffsetHigh;
- };
- void* Pointer;
- };
- HANDLE hEvent;
-} OVERLAPPED;
-
-
-To poll on the completion of one of these functions,
-use an IOCP, overlapped->hEvent, and
-GetQueuedCompletionStatus().
-
-To demonstrate the use of GetQueuedCompletionStatus() an
-example of connecting to localhost at port 8000 is presented.
-
-
-char* buffer[200];
-WSABUF b = { buffer, 200 };
-size_t bytes_recvd;
-int r, total_events;
-OVERLAPPED overlapped;
-HANDLE port;
-
-port = CreateIoCompletionPort(INVALID_HANDLE_VALUE, NULL, NULL, 0);
-if (!port) {
- goto error;
-}
-
-
-r = WSARecv(socket, &b, 1, &bytes_recvd, NULL, &overlapped, NULL);
-
-CreateIoCompletionPort(port, &overlapped.hEvent,
-
-if (r == 0) {
- if (WSAGetLastError() == WSA_IO_PENDING) {
- /* Asynchronous */
- GetQueuedCompletionStatus()
-
-
- if (r == WAIT_TIMEOUT) {
- printf("Timeout\n");
- } else {
-
- }
-
-
- } else {
- /* Error */
- printf("Error %d\n", WSAGetLastError());
- }
-} else {
- /* Synchronous */
- printf("read %ld bytes from socket\n", bytes_recvd);
-}
-
-
-
-
-
-Writing code that can take advantage of the best worlds on across Unix operating -systems and Windows is very difficult, requiring one to understand intricate -APIs and undocumented details from many different operating systems. There -are several projects which have made attempts to provide an abstraction -layer but in the author's opinion, none are completely satisfactory. - -
Marc Lehmann's
- libev and
- libeio.
- libev is the perfect minimal abstraction of the Unix I/O multiplexers. It
- includes several helpful tools like ev_async, which is for
- asynchronous notification, but the main piece is the ev_io,
- which informs the user about the state of file descriptors. As mentioned
- before, in general it is not possible to get state changes for regular
- files—and even if it were the write(2) and
- read(2) calls do not guarantee that they won't block.
- Therefore libeio is provided for calling various disk-related
- syscalls in a managed thread pool. Unfortunately the abstraction layer
- which libev targets is not appropriate for IOCPs—libev works strictly
- with file descriptors and does not the concept of a socket.
- Furthermore users on Unix will be using libeio for file I/O which is not
- ideal for porting to Windows. On windows libev currently uses
- select()—which is limited to 64 file descriptors per
- thread.
-
-
libevent. - Somewhat bulkier than libev with code for RPC, DNS, and HTTP included. - Does not support file I/O. - libev was - created after Lehmann evaluated - libevent and rejected it—it's interesting to read his reasons - why. A - major rewrite was done for version 2 to support Windows IOCPs but anecdotal - evidence suggests that it is still not working correctly. - -
Boost - ASIO. It basically does what you want on Windows and Unix for - sockets. That is, epoll on Linux, kqueue on Macintosh, IOCPs on Windows. - It does not support file I/O. In the author's opinion is it too large - for a not extremely difficult problem (~300 files, ~12000 semicolons). - - - -
-Almost every socket operation that you're familiar with has an -overlapped counter-part. The following section tries to pair Windows -overlapped I/O syscalls with non-blocking Unix ones. - - -
_setmaxstdio().)
-
-
-send(2), write(2)WSASend(),
-WriteFileEx()
-recv(2), read(2)WSARecv(),
-ReadFileEx()
-connect(2)ConnectEx()
-
-
-Non-blocking connect() is has difficult semantics in
-Unix. The proper way to connect to a remote host is this: call
-connect(2) while it returns
-EINPROGRESS poll on the file descriptor for writablity.
-Then use
-
int error; -socklen_t len = sizeof(int); -getsockopt(fd, SOL_SOCKET, SO_ERROR, &error, &len);-A zero
error indicates that the connection succeeded.
-(Documented in connect(2) under EINPROGRESS
-on the Linux man page.)
-accept(2)AcceptEx()
-sendfile(2)TransmitFile()
-
- The exact API of sendfile(2) on Unix has not been agreed
-on yet. Each operating system does it slightly different. All
-sendfile(2) implementations (except possibly FreeBSD?) are blocking
-even on non-blocking sockets.
-
shutdown(2), graceful close, half-duplex connectionsDisconnectEx()
-
-close(2)closesocket()
-SOCKET.
-sockaddr
- bind()
- getsockname()
-AF_Unix
- domain sockets. AF_Unix sockets exist in the file system
-often looking like
-/tmp/pipename- -Windows named pipes have a path, but they are not directly part of the file -system; instead they look like - -
\\.\pipe\pipename- - -
socket(AF_Unix, SOCK_STREAM, 0), bind(2), listen(2)CreateNamedPipe()
-
-Use FILE_FLAG_OVERLAPPED, PIPE_TYPE_BYTE,
-PIPE_NOWAIT.
-
send(2), write(2)WriteFileEx()
-recv(2), read(2)ReadFileEx()
-connect(2)CreateNamedPipe()
-accept(2)ConnectNamedPipe()
--In Unix file system files are not able to use non-blocking I/O. There are -some operating systems that have asynchronous I/O but it is not standard and -at least on Linux is done with pthreads in GNU libc. For this reason -applications designed to be portable across different Unixes must manage a -thread pool for issuing file I/O syscalls. - -
-The situation is better in Windows: true overlapped I/O is available when -reading or writing a stream of data to a file. - -
write(2)WriteFileEx()
-
-Solaris's event completion ports has true in-kernel async writes with aio_write(3RT) -
read(2)ReadFileEx()
-
-Solaris's event completion ports has true in-kernel async reads with aio_read(3RT) -
It is (usually?) possible to poll a Unix TTY file descriptor for
-readability or writablity just like a TCP socket—this is very helpful
-and nice. In Windows the situation is worse, not only is it a completely
-different API but there are not overlapped versions to read and write to the
-TTY. Polling for readability can be accomplished by waiting in another
-thread with RegisterWaitForSingleObject().
-
-
read(2)ReadConsole()
-and
-ReadConsoleInput()
-do not support overlapped I/O and there are no overlapped
-counter-parts. One strategy to get around this is
-RegisterWaitForSingleObject(&tty_wait_handle, tty_handle, - tty_want_poll, NULL, INFINITE, WT_EXECUTEINWAITTHREAD | - WT_EXECUTEONLYONCE)-which will execute
tty_want_poll() in a different thread.
-You can use this to notify the calling thread that
-ReadConsoleInput() will not block.
-write(2)WriteConsole()
-is also blocking but this is probably acceptable.
-tcsetattr(3)SetConsoleMode()
--tips -
GetAddrInfoEx() function. It seems Asynchronous Procedure Calls must be used instead.
-Windows Sockets 2
--IOCP: -
OVERLAPPED Structure
- GetOverlappedResult()
- HasOverlappedIoCompleted()
- CancelIoEx()
- — cancels an overlapped operation.
- WSASend()
-WSARecv()
-ConnectEx()
-TransmitFile()
-— an async sendfile() for windows.
-WSADuplicateSocket()
-— describes how to share a socket between two processes.
-_setmaxstdio()
-— something like setting the maximum number of file decriptors
-and setrlimit(3)
-AKA ulimit -n. Note the file descriptor limit on windows is
-2048.
--APC: -
DNSQuery()
-— General purpose DNS query function like res_query() on Unix.
-Pipe functions
-CreateNamedPipe
-CallNamedPipe
-— like accept is for Unix pipes.
-ConnectNamedPipe
-WaitForMultipleObjectsEx is pronounced "wait for multiple object sex".
-
-Also useful:
-Introduction
- to Visual C++ for Unix Users
-
-