한국 미나 사용자께 드리는 네티로의 전환 권고

미나 프로젝트에서 손을 뗀 후, 지난 약 3개월간 네티 프로젝트 에 상당한 시간을 할애하였고, 그 결과 현재 API 와 구현 모두 안정화 단계에 접어들었습니다.

적절한 문서화, 공개 가능한 성능 테스트 보고서 작성 및 웹 사이트 컨텐츠 보강을 마치는 대로 최초의 안정 버전이 릴리즈될 예정입니다. 지금까지 적어도 2 주에 한 번씩은 릴리즈를 해 왔고, 앞으로 2 주 내로 릴리즈 계획이 되어 있으므로, 미나처럼 불필요하게 프로젝트의 릴리즈가 늦어지는 일은 없을 것으로 예상합니다.

네티 안정 버전의 릴리즈와 동시에 미나 프로젝트에 대한 한국어 지원에 할애하는 시간을 많이 줄일 예정이니, 양해 부탁드리겠습니다. (상황에 따라 답장을 못 보내 드릴 수도 있습니다.) 따라서 모든 미나 관련 문의는 공식 커뮤니티를 통해 해 주시면 감사하겠습니다.

네티 프로젝트는 제가 직접 운영하는 한국어 사용자 그룹을 갖고 있습니다. 따라서 네티 프로젝트에 대한 모든 문의 사항 및 피드 백은 네티 한국어 사용자 그룹 을 통해 해 주시면 성심 성의껏 회신하도록 하겠습니다.

만약 진행중이신 프로젝트가 초기 단계에 있다면 네티로의 전환을 심각하게 고려하시기를 권합니다. 부가 기능 (예: JMX) 면에서 부족함이 있으나, 스루풋, 레이턴시, 유연성, 안정성 등 대부분의 면에서 비교할 수 없는 경험이 되리라 확신합니다.

Netty reaches 75%+ test coverage. What's next?

Netty 3.0.0.CR3 was my first serious attempt to achieve high test coverage by writing many test cases in my life, and it was very exciting for me. I was able to fix more than a dozen bugs with them.

EclEmma was very helpful to review all the source code and to achieve high test coverage. The byte code instrumented by Emma ran much faster than other open source test coverage tools, yet providing reliable numbers.

Granted that both the API and implementation of Netty became stable enough, I’m going to focus on writing JavaDoc and user guide for the next week. The first GA is expected in a couple weeks – please stay tuned!

As always, your feed back is more than appreciated.

Netty moves its nest from GoogleCode to JBoss.org with its first release candidate

Finally, I have managed to move Netty to JBoss.org and announce the first release candidate. If you were using Google Groups, you need to subscribe again.

Here’s the new URL:

and here are the related announcement messages:

For those who don’t know what Netty is yet — The Netty project is an effort to provide an asynchronous · event-driven network application framework for rapid development of maintainable high-performance · high-scalability protocol servers and clients, including its related out-of-the-box protocol extensions and tool suite. To somewhat oversimplify, it’s a framework that allows you to write a NIO client and server very easily.

Using GNOME Evolution with a huge IMAP mailbox

Update: The latest development snapshot of Opera 9.5 has full support for virtual folder with powerful filter which supports a regular expression. Let’s give it a try! Adding GPG signing and better multiple identity support would be an icing on the cake.

I’ve been struggling with various mail clients to deal with my huge IMAP mailbox which contains about 200k messages. I didn’t want to split it into more than one. It’s not my job but an e-mail client’s job; see what GMail does!

Although Opera Mail does a great job on dealing with such a big IMAP mailbox, it lacks a couple critical features which make me have a difficulty in organizing my messages. I have missed Evolution virtual folders especially.

However, all other GUI/Web-based e-mail clients than Opera Mail are very poor at dealing with a huge IMAP mailbox. It seems like IMAP support is not a high priority task for most e-mail clients in the world. The latest alpha version of RoundCube seems to be very efficient but it lacks too many features at this moment.

Before I give up again and stick to Opera Mail, I decided to give a try to various IMAP-to-Maildir synchronization tools – isync and OfflineIMAP. Long time agao, I had a bad experience with OfflineIMAP and it seems like it doesn’t work with my account yet. By contrast, isync was indeed a great tool which does its job without any problem.

One problem with isync is that its configuration file format is somewhat undocumented, so I’d like to share my settings:

# ~/.mbsyncrc
MaildirStore Local
Path ~/.maildir
Inbox ~/.maildir

IMAPStore Remote
Host <hostname>
User <username>
Pass <password>
CertificateFile ~/.mbsync.crt

Channel Local-Remote
Master :Remote:
Slave :Local:
Create Slave
Sync All

You also need to set up your crontab:http://en.wikipedia.org/wiki/Vixie_cron#Modern_versions to run mbsync:http://isync.sourceforge.net/mbsync.html periodically because Evolution doesn’t provide an option to execute a command before refreshing the local Maildir. Here’s my crontab:

*/5 * * * * pgrep -f "^(/usr/bin/)?evolution" > /dev/null && mbsync -a -q

Please note that I used pgrep to make sure mbsync runs only when Evolution is running.

Finally, I succeeded to make Evolution run pretty fast with my IMAP mailbox, although it’s a kind of workaround. However, I don’t think this is a ugly hack. Considering that Opera Mail stores my all messages in its local storage, what Evolution does for the local Maildir is very similar to what Opera Mail does. It creates a full text index for all messages and maintains the index for every message operation. What’s missing is immediate mailbox synchronization based on IMAP notification, and it shouldn’t be difficult to be integrated into Evolution codebase IMHO.

PS: I had a difficulty searching for isync because Apple has a product with the same name. The actual executable of isync is mbsync, so try this google search.

Filling the gap between blocking I/O and NIO

A non-blocking NIO Channel and a blocking InputStream have inevitable impedance mismatch. Because an InputStream is supposed to block for every read operation unless there’s some data available in the buffer, any InputStream-based decoder implementation can’t be used with a non-blocking NIO application right away.

A common workaround is to prepend a length field for each message so you can wait until you read a whole message before calling InputStream.read(). However, this turns your NIO application incompatible with a legacy blocking I/O application because the legacy application doesn’t prepend a length field at all. You might have managed to modify the legacy application to prepend a length field, but we know it’s not always the case. We need something to fill this gap between two I/O paradigms.

An ObjectInput/OutputStream-based blocking I/O network applications are the most common case because it was considered to be the easiest quick-and-dirty solution for intranet Java object exchange. It’s as simple as wrapping an InputStream of a Socket with an ObjectInputStream (i.e. in = new ObjectInputStream(socket.getInputStream());).

How can we implement a NIO network application to be interoperable with those legacy applications without any modification? It was considered to be impossible… until today!

I’ve just released a new milestone of Netty which addresses the issue I described above. It provides CompatibleObjectEncoder and CompatibleObjectDecoder, which retains interoperability with the legacy ObjectInput/OutputStream-based socket applications.

You will also find you can do the same for any kind of InputStream implementations with Netty’s ReplayingDecoder with fairly small amount of effort, which means you can shift the paradigm of your complicated blocking protocol client/server to more scalable non-blocking paradigm while retaining most legacy code.

The excitement of ReplayingDecoder doesn’t stop here. It also allows you to implement a non-blocking decoder in a blocking paradigm. In a non-blocking paradigm, you always had to check if there’s enough data in the buffer, like the following:

public boolean decode(ByteBuffer in) {
  if (in.remaining() < 4) {
    return false;
  }

  // Read the length header.
  int position = in.position();
  int length = in.getInt();
  if (in.remaining() < length) {
    in.position(position);
    return false;
  }

  // Read the body.
  byte[] data = new byte[length];
  in.get(data);
  ...
  return true;
}

With ReplayingDecoder, you don’t need to check the availability of the input buffer at all:

public void decode(ByteBuffer in) {
  // Read the length header.
  int length = in.getInt();

  // Read the body.
  byte[] data = new byte[length];
  in.get(data);
  ...
}

How could this work? ReplayingDecoder uses a sort of continuation technique. It rewinds the buffer position to the beginning when there’s not enough data in the buffer automatically and calls decode() again (i.e. replays the decode) when more data is received from a remote peer.

You might think this is pretty inefficient, but it turned out to be very efficient in most cases. Higher throughput means lower chance of replay because we will receive more than one message for a single input buffer (often dozens). Consequently, most messages will be decoded in one shot without a replay. In case of slow connection, it will be less than optimal but you won’t see much difference because it’s already slow because the connection itself is slow. Just compare the code complexity of the two paradigms. I’d definitely go for the latter.

A problem with recent RSS feeds in gleamynode.net

Brett Porter kindly pointed me out that there’s some problem with the RSS feed of gleamynode.net. I have updated FeedBurner information along with the meta tags to make sure the same problem doesn’t occur again. Also, please make sure that you are using the FeedBurner RSS feed if you are subscribed to this blog. Apologies.

Brett Porter가 친절하게도 gleamynode.net의 RSS 피드에 문제가 있다는 사실을 지적해 주었습니다. 메타 태그와 피드버너 정보를 수정해서 같은 문제가 다시 일어나지 않도록 조치했습니다. 또, 만약 이 블로그를 구독중이시라면 피드버너 RSS 주소를 사용하고 있는지 확인해 주시길 바랍니다. 불편을 끼쳐 드려 죄송합니다.

네티 한국어 사용자 그룹을 개설했습니다.

원문은 여기

미나 프로젝트를 떠난 지도 상당한 시간이 흘렀습니다. 네티 프로젝트 를 시작하고 처음부터 모든 것을 다시 작성하느라 바쁘게 보낸 몇 달 간이었습니다. 이제 버그 리포트도 들어 오고 커뮤니티가 형성되어 가는 과정이어서 흐뭇합니다.

그동안 국문으로 여러분들께 미나에 대한 지원을 하고 싶었지만 재단 정책을 포함한 여러 가지 이유로 하지 못해 아쉬운 점이 많았는데요. 네티에서는 별도로 네티 한국어 사용자 그룹 을 운영하여 더 많은 분들의 편의를 도모해 보기로 하였습니다. 네티 프로젝트가 아직 초기 단계이니만큼 많은 질문과 논의점, 버그가 있으리라고 생각됩니다. 언제든지 자유롭게 이 그룹을 통해 메시지를 남겨 주시면 감사하겠습니다.

단, 공식 영문 네티 사용자 그룹 및 이슈 트래커는 전 세계인이 함께 보는 자리이므로 항상 영문을 사용하셔야 한다는 점 양지해 주시길 부탁드리겠습니다.

그럼 앞으로 많은 메시지가 이 그룹에서 오고 가기를 기원해 봅니다.

Using Fail2Ban to refuse brute-force attacks

I have been using a quick and dirty shell script to update /etc/hosts.deny file when brute-force attack flows into my server. It was pretty effective but was not effective enough to block the break-in attempts immediately. Today, I found a better solution – Fail2Ban. It scans the system log files and bans brute-force attacks for a certain period.

Most examples use iptables, but I always prefer /etc/hosts.deny and I don’t even care about unbanning once a host is banned. Therefore, I added the following to /etc/fail2ban/jail.conf:

[ssh-hostsdeny]
enabled  = true
filter   = sshd
action   = hostsdeny-nounban
           mail-whois[name=SSH, [email protected]]
logpath  = /var/log/messages

[ssh-ddos-hostsdeny]
enabled  = true
filter   = sshd-ddos
action   = hostsdeny-nounban
           mail-whois[name=SSH-DDoS, [email protected]]
logpath  = /var/log/messages

Please note that I defined a new action called hostsdeny-nounban, which doesn’t unban the attacker’s IP address once banned It’s /etc/fail2ban/action.d/hostsdeny-nounban.conf:

[Definition]
actionstart =
actionstop =
actioncheck =
actionban = IP=<ip> && grep -q "ALL: $IP" <file> || echo "ALL: $IP" >> <file>
actionunban =

[Init]
file = /etc/hosts.deny

For more information, I’d recommend you to read the Gentoo HOWTO fail2ban.

A nice quote about computer programming

From Larry O’Brien and Bruce Eckel in Thinking in C#:

Computer programming is tremendous fun. Like music, it is a skill that derives from an unknown blend of innate talent and constant practice. Like drawing, it can be shaped to a variety of ends – commercial, artistic, and pure entertainment. Programmers have a well-deserved reputation for working long hours but are rarely credited with being driven by creative fevers. Programmers talk about software development on weekends, vacations, and over meals not because they lack imagination, but because their imagination reveals worlds that others cannot see.
컴퓨터 프로그래밍은 너무나 재미있다. 그것은 음악과 마찬가지로 내면에 간직된 재능과 지속적인 훈련이 미묘한 방식으로 혼합된 결과이다. 그리고 그 결과는 그림과 마찬가지로 상업적 비즈니스, 예술, 혹은 순수한 엔터테인먼트, 어느 것으로도 나타날 수 있다.  프로그래머들은 엄청나게 많은 시간을 일하는 것으로 정평이 나 있는데, 그들이 창조적인 열정 때문에 그렇게 한다는 사실을 아는 사람은 별로 없다. 프로그래머들은 주말에, 휴가에서, 식사를 하면서 계속 소프트웨어 개발에 대한 이야기를 나눈다.  상상력이 부족해서 그런 것이 아니다. 그들의 상상력은 다른 사람들이 보지 못하는 멋진 세계를 드러내기 때문이다.