Search My Blog

ruby (4) web (4) ruby on rails (3) security (3) GPG (2) OpenPGP (2) RFC (2) linux (2) rails (2) shell (2) sysadmin (2) Exchange (1) GIT. (1) IMAP (1) RCS (1) SSH (1) SVN (1) bundle (1) cURL (1) command line (1) crack (1) css (1) developer (1) email (1) fail (1) hack (1) http (1) mac (1) network (1) password (1) regular expression (1) script (1) subversion (1) terminal (1) textmate (1) tip (1) vim (1)

Monday, January 9, 2012

Monkeypatching the Ruby IMAP class to build a client for Exchange email servers not compliant with RFC standards

When trying to use Ruby Class Net::IMAP to automate some searching/arranging/deleting tasks for my office email I came across some Exchange responses that are not strictly compliant with the IMAP standard (currently 4rev1, described in RFC3501)

After loading the standard IMAP class
ruby-1.9.2-p290 :001 > require 'net/imap'
=> true

we can connect to the IMAP server by instantiating an Net::IMAP object that describes the type of connection used (IP address, TCP port and whether we should use SSL to encrypt the communication).
My company uses IMAP over SSL (port 993/tcp) with a 'PLAIN' authentication, so let's start with:
ruby-1.9.2-p290 :002 > imap= Net::IMAP.new( 'outlook.h3g.it', :port=>'imaps', :ssl=>true )
=> #<Net::IMAP:0x0000010109e118 @mon_owner=nil, @mon_count=0, @mon_mutex=#<Mutex:0x0000010109e0c8>, @host="outlook.h3g.it", @port="imaps", @tag_prefix="RUBY", @tagno=0, @parser=#<Net::IMAP::ResponseParser:0x0000010109e028 @str="* OK IMAP4 019 Ready\r\n", @pos=22, @lex_state=:EXPR_BEG, @token=nil, @flag_symbols={}>, @sock=#<OpenSSL::SSL::SSLSocket:0x0000010109d678>, @usessl=true, @responses={}, @tagged_responses={}, @response_handlers=[], @tagged_response_arrival=#<MonitorMixin::ConditionVariable:0x0000010109bb70 @monitor=#<Net::IMAP:0x0000010109e118 ...>, @cond=#<ConditionVariable:0x0000010109bb48 @waiters=[], @waiters_mutex=#<Mutex:0x0000010109baf8>>>, @continuation_request_arrival=#<MonitorMixin::ConditionVariable:0x0000010109bad0 @monitor=#<Net::IMAP:0x0000010109e118 ...>, @cond=#<ConditionVariable:0x0000010109baa8 @waiters=[], @waiters_mutex=#<Mutex:0x0000010109ba58>>>, @idle_done_cond=nil, @logout_command_tag=nil, @debug_output_bol=true, @exception=nil, @greeting=#<struct Net::IMAP::UntaggedResponse name="OK", data=#<struct Net::IMAP::ResponseText code=nil, text="IMAP4 019 Ready">, raw_data="* OK IMAP4 019 Ready\r\n">, @client_thread=#<Thread:0x00000100892c80 run>, @receiver_thread=#<Thread:0x0000010109afb8 run>>

So far so good, but when I introduce myself to the server I receive an error:
ruby-1.9.2-p290 :003 > imap.authenticate( 'PLAIN', my_name, my_password )
Net::IMAP::ResponseParseError: unexpected token CRLF (expected SPACE)
from /usr/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/net/imap.rb:3235:in `parse_error'
from /usr/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/net/imap.rb:3087:in `match'
from /usr/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/net/imap.rb:2058:in `continue_req'
from /usr/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/net/imap.rb:2045:in `response'
from /usr/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/net/imap.rb:1973:in `parse'
from /usr/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/net/imap.rb:1124:in `get_response'
from /usr/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/net/imap.rb:1036:in `receive_responses'
from /usr/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/net/imap.rb:1023:in `block in initialize'

It seems that our client is expecting a T_SPACE while server is sending T_CRLF. Interesting to say, Microsoft states that Exchange is "compatible with RFC3501" but also that "(AUTH=PLAIN not supported)" (http://technet.microsoft.com/en-us/library/ff848256.aspx). Why?? It seems to me that once again they are not able to understand a clear (A)BNF specification on a standard RFC:

RFC3501: INTERNET MESSAGE ACCESS PROTOCOL - VERSION 4rev1
9. Formal Syntax
response = *(continue-req / response-data) response-done
continue-req = "+" SP (resp-text / base64) CRLF
response-data = "*" SP (resp-cond-state / resp-cond-bye / mailbox-data / message-data / capability-data) CRLF
response-done = response-tagged / response-fatal
response-fatal = "*" SP resp-cond-bye CRLF ; Server closes connection immediately
response-tagged = tag SP resp-cond-state CRLF

RFC clearly says that a SP(ace) is needed after the "+" sign, as correctly defined in the 'continue_req' method:
require 'net/imap'
module Net
class IMAP
class ResponseParser
def continue_req
match(T_PLUS)
match(T_SPACE)
return ContinuationRequest.new(resp_text, @str)
end
end
end
end

however we can try to fix (but I should say 'break') it to suit the Exchange server's needs, deleting the 'match(T_SPACE)' statement:
require 'net/imap'
module Net
class IMAP
class ResponseParser
def continue_req
match(T_PLUS)
return ContinuationRequest.new(resp_text, @str)
end #def continue_req
end #class ResponseParser
end #class IMAP
end #module Net

And then try again:
ruby-1.9.2-p290 :004 > imap.authenticate( 'PLAIN', my_name, my_password )
=> #, raw_data="RUBY0001 OK AUTHENTICATE completed.\r\n">

And now it's ok.
But testing some other IMAP command we soon face another error:
ruby-1.9.2-p290 :005 > imap.noop
=> #, raw_data="RUBY0002 OK NOOP completed.\r\n">

ruby-1.9.2-p290 :006 > imap.status( 'INBOX', 'MESSAGES' )
Net::IMAP::ResponseParseError: unexpected token SPACE (expected CRLF)
from /usr/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/net/imap.rb:3235:in `parse_error'
from /usr/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/net/imap.rb:3087:in `match'
from /usr/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/net/imap.rb:2051:in `response'
from /usr/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/net/imap.rb:1973:in `parse'
from /usr/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/net/imap.rb:1124:in `get_response'
from /usr/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/net/imap.rb:1036:in `receive_responses'
from /usr/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/net/imap.rb:1023:in `block in initialize'

Exchange server (sometimes?) adds an extra space before CRLF (again violating the RFC command syntax listed above).
This time we need to change the response method:
require 'net/imap'
module Net
class IMAP
class ResponseParser
def response
token = lookahead
case token.symbol
when T_PLUS
result = continue_req
when T_STAR
result = response_untagged
else
result = response_tagged
end
match(T_CRLF)
match(T_EOF)
return result
end
end
end
end

adding a 'match(T_SPACE) if lookahead.symbol == T_SPACE' just before the CRLF match:
require 'net/imap'
module Net
class IMAP
class ResponseParser
def response
token = lookahead
case token.symbol
when T_PLUS
result = continue_req
when T_STAR
result = response_untagged
else
result = response_tagged
end
match(T_SPACE) if lookahead.symbol == T_SPACE
match(T_CRLF)
match(T_EOF)
return result
end #def response
end #class ResponseParser
end #class IMAP
end #module Net

I've tested this 'monkeypatched' IMAP class long enough to say that fortunately this is all we need to make our client work... does it means that Microsoft is really shifting towards standards-based technology? Bah!

No comments:

Post a Comment

If you find this useful please leave a feedback :)