Filling the gap between blocking I/O and NIO

A non-blocking NIO Channel and a blocking InputStream have inevitable impedance mismatch. Because an InputStream is supposed to block for every read operation unless there’s some data available in the buffer, any InputStream-based decoder implementation can’t be used with a non-blocking NIO application right away.

A common workaround is to prepend a length field for each message so you can wait until you read a whole message before calling InputStream.read(). However, this turns your NIO application incompatible with a legacy blocking I/O application because the legacy application doesn’t prepend a length field at all. You might have managed to modify the legacy application to prepend a length field, but we know it’s not always the case. We need something to fill this gap between two I/O paradigms.

An ObjectInput/OutputStream-based blocking I/O network applications are the most common case because it was considered to be the easiest quick-and-dirty solution for intranet Java object exchange. It’s as simple as wrapping an InputStream of a Socket with an ObjectInputStream (i.e. in = new ObjectInputStream(socket.getInputStream());).

How can we implement a NIO network application to be interoperable with those legacy applications without any modification? It was considered to be impossible… until today!

I’ve just released a new milestone of Netty which addresses the issue I described above. It provides CompatibleObjectEncoder and CompatibleObjectDecoder, which retains interoperability with the legacy ObjectInput/OutputStream-based socket applications.

You will also find you can do the same for any kind of InputStream implementations with Netty’s ReplayingDecoder with fairly small amount of effort, which means you can shift the paradigm of your complicated blocking protocol client/server to more scalable non-blocking paradigm while retaining most legacy code.

The excitement of ReplayingDecoder doesn’t stop here. It also allows you to implement a non-blocking decoder in a blocking paradigm. In a non-blocking paradigm, you always had to check if there’s enough data in the buffer, like the following:

public boolean decode(ByteBuffer in) {
  if (in.remaining() < 4) {
    return false;
  }

  // Read the length header.
  int position = in.position();
  int length = in.getInt();
  if (in.remaining() < length) {
    in.position(position);
    return false;
  }

  // Read the body.
  byte[] data = new byte[length];
  in.get(data);
  ...
  return true;
}

With ReplayingDecoder, you don’t need to check the availability of the input buffer at all:

public void decode(ByteBuffer in) {
  // Read the length header.
  int length = in.getInt();

  // Read the body.
  byte[] data = new byte[length];
  in.get(data);
  ...
}

How could this work? ReplayingDecoder uses a sort of continuation technique. It rewinds the buffer position to the beginning when there’s not enough data in the buffer automatically and calls decode() again (i.e. replays the decode) when more data is received from a remote peer.

You might think this is pretty inefficient, but it turned out to be very efficient in most cases. Higher throughput means lower chance of replay because we will receive more than one message for a single input buffer (often dozens). Consequently, most messages will be decoded in one shot without a replay. In case of slow connection, it will be less than optimal but you won’t see much difference because it’s already slow because the connection itself is slow. Just compare the code complexity of the two paradigms. I’d definitely go for the latter.

A problem with recent RSS feeds in gleamynode.net

Brett Porter kindly pointed me out that there’s some problem with the RSS feed of gleamynode.net. I have updated FeedBurner information along with the meta tags to make sure the same problem doesn’t occur again. Also, please make sure that you are using the FeedBurner RSS feed if you are subscribed to this blog. Apologies.

Brett Porter가 친절하게도 gleamynode.net의 RSS 피드에 문제가 있다는 사실을 지적해 주었습니다. 메타 태그와 피드버너 정보를 수정해서 같은 문제가 다시 일어나지 않도록 조치했습니다. 또, 만약 이 블로그를 구독중이시라면 피드버너 RSS 주소를 사용하고 있는지 확인해 주시길 바랍니다. 불편을 끼쳐 드려 죄송합니다.

네티 한국어 사용자 그룹을 개설했습니다.

원문은 여기

미나 프로젝트를 떠난 지도 상당한 시간이 흘렀습니다. 네티 프로젝트 를 시작하고 처음부터 모든 것을 다시 작성하느라 바쁘게 보낸 몇 달 간이었습니다. 이제 버그 리포트도 들어 오고 커뮤니티가 형성되어 가는 과정이어서 흐뭇합니다.

그동안 국문으로 여러분들께 미나에 대한 지원을 하고 싶었지만 재단 정책을 포함한 여러 가지 이유로 하지 못해 아쉬운 점이 많았는데요. 네티에서는 별도로 네티 한국어 사용자 그룹 을 운영하여 더 많은 분들의 편의를 도모해 보기로 하였습니다. 네티 프로젝트가 아직 초기 단계이니만큼 많은 질문과 논의점, 버그가 있으리라고 생각됩니다. 언제든지 자유롭게 이 그룹을 통해 메시지를 남겨 주시면 감사하겠습니다.

단, 공식 영문 네티 사용자 그룹 및 이슈 트래커는 전 세계인이 함께 보는 자리이므로 항상 영문을 사용하셔야 한다는 점 양지해 주시길 부탁드리겠습니다.

그럼 앞으로 많은 메시지가 이 그룹에서 오고 가기를 기원해 봅니다.

Using Fail2Ban to refuse brute-force attacks

I have been using a quick and dirty shell script to update /etc/hosts.deny file when brute-force attack flows into my server. It was pretty effective but was not effective enough to block the break-in attempts immediately. Today, I found a better solution – Fail2Ban. It scans the system log files and bans brute-force attacks for a certain period.

Most examples use iptables, but I always prefer /etc/hosts.deny and I don’t even care about unbanning once a host is banned. Therefore, I added the following to /etc/fail2ban/jail.conf:

[ssh-hostsdeny]
enabled  = true
filter   = sshd
action   = hostsdeny-nounban
           mail-whois[name=SSH, [email protected]]
logpath  = /var/log/messages

[ssh-ddos-hostsdeny]
enabled  = true
filter   = sshd-ddos
action   = hostsdeny-nounban
           mail-whois[name=SSH-DDoS, [email protected]]
logpath  = /var/log/messages

Please note that I defined a new action called hostsdeny-nounban, which doesn’t unban the attacker’s IP address once banned It’s /etc/fail2ban/action.d/hostsdeny-nounban.conf:

[Definition]
actionstart =
actionstop =
actioncheck =
actionban = IP=<ip> && grep -q "ALL: $IP" <file> || echo "ALL: $IP" >> <file>
actionunban =

[Init]
file = /etc/hosts.deny

For more information, I’d recommend you to read the Gentoo HOWTO fail2ban.

A nice quote about computer programming

From Larry O’Brien and Bruce Eckel in Thinking in C#:

Computer programming is tremendous fun. Like music, it is a skill that derives from an unknown blend of innate talent and constant practice. Like drawing, it can be shaped to a variety of ends – commercial, artistic, and pure entertainment. Programmers have a well-deserved reputation for working long hours but are rarely credited with being driven by creative fevers. Programmers talk about software development on weekends, vacations, and over meals not because they lack imagination, but because their imagination reveals worlds that others cannot see.
컴퓨터 프로그래밍은 너무나 재미있다. 그것은 음악과 마찬가지로 내면에 간직된 재능과 지속적인 훈련이 미묘한 방식으로 혼합된 결과이다. 그리고 그 결과는 그림과 마찬가지로 상업적 비즈니스, 예술, 혹은 순수한 엔터테인먼트, 어느 것으로도 나타날 수 있다.  프로그래머들은 엄청나게 많은 시간을 일하는 것으로 정평이 나 있는데, 그들이 창조적인 열정 때문에 그렇게 한다는 사실을 아는 사람은 별로 없다. 프로그래머들은 주말에, 휴가에서, 식사를 하면서 계속 소프트웨어 개발에 대한 이야기를 나눈다.  상상력이 부족해서 그런 것이 아니다. 그들의 상상력은 다른 사람들이 보지 못하는 멋진 세계를 드러내기 때문이다.