Documentation

Not much really... The program is quite alpha quality and so de documentation. The proggy has no way to install it (it just runs from the directory where you untar it) and it sure is full of bugs. Though it WORKS FOR ME(tm).

1. The Things
   
2. Requirements
   
3. Installation
   
4. KIPT
   
   • Running the program
     
   • Configuring KIPT
     
   • A Simple Example
     
   • A Not So Simple Example
     
   • A Truly Complex Example
     
5. mpserver
   
6. smtpclient
   
   • Setting up postfix
   • Configuring the SMTP client

And the documentation for the 0.2 release is still there.

The Things

Currently the system is made up of three pieces. The most important is of course kipt which does all the proxying (DNS and HTTP). It can use the stream multiplexer called mpserver, which you'll have to install on a machine outside of your LAN. Of course its use is not obligatory, but it can greatly improve the performance, by allowing you to avoid ssh authentication for each new connection. And finally there's smtpclient, a simple SMTP sender program to use it from postfix (or possibly from any other MTA) for sending your mails.

Requirements

The proxy is written in scheme and you will have to install the PLT Scheme environment to be able to use it. Fortunately it runs on nearly everything. I'm writing the proggy on FreeBSD but it should run without modifications on any *BSD, Linux, quite a few unix variants, and even on Windows.

You will also need the simple connect utility if you plan to use ssh and you can use dante's socksify to send unsuspecting programs like telnet or ftp trough it.

Installation

For now, just untar it somewhere and leave there everything. To start it, create your ~/.kipt-config and optionally your smtpclient.conf and set up postfix, and start (as root if you want the nameserver on the port 53) .../kipt-0.4/kipt. If started in debug mode it will print out a lots of information (incoming requests, multiplexer messages, etc.), so redirect it somewhere if you care. Don't forget to have mzscheme somewhere in your path.

KIPT

KIPT is the main program and it does all the DNS and HTTP proxying.

Running the program

To start it, just run the script kipt in the directory where you unpacked the distribution. You'll have to run it as root if you want the nameserver on port 53, and you'll have to create your ~/.kipt-config.

It accepts only one command-line argument, -F config-file, with which you can change the configuration file it uses. If you don't specify it, it defaults to ~/.kipt-config.

Configuring KIPT

The program takes only one command line argument: -F config-file, which defaults to .kipt-config in the user's home directory. The syntax of this file is that of scheme but the semantics are somewhat different.

(define NAME VALUE)

(define NAME LIST)

You can use it to assign the VALUE to NAME. In the rest of the file you can use the new name and it will be replaced with VALUE in a macro-like manner.

(router-rules RULE RULE...)

Use it to set up rules for opening tcp connections. Each RULE has the following structure: (DESCRIPTION HOST PORTS CONNECTION-METHOD), where DESCRIPTION is a single string, HOST is a string containing a regular expression to match the hostname and PORTS is a list of either single port numbers or ranges in the form of (MIN MAX). The HOST and PORTS values are used to match against a destination hostname (or ip) and port number to connect to and either of them can be the value #t (true), in which case it matches everything, and the rules are tested in order of appearence. Connection requests not matched by any rule will be responded with a normal connect syscall. The possible connection methods are:

• direct
  to make a normal connect syscall

• (proxy)
  to call the internal proxy with a CONNECT request (and thus sending the request through the rule-set of it)

• (subprocess command args...)
  to start a subprocess specified by the first argument, with the rest of the arguments as its command line, and use its input and output as communication ports. The arguments must be strings, and you can use %HOST and %PORT as the destination address and port. To escape a % sign, write it twice (%%).

  To run an ssh process like

  /usr/bin/ssh -o ProxyCommand="connect -H PROXY-HOST:PROXY-PORT %h %p" 
      -l USER -p FRIENDY-PORT FRIENDLY-HOST
      /usr/bin/nc TARGET-HOST TARGET-PORT
  

  with PROXY-HOST:PROXY-PORT being the address where kipt is listening (here, 127.0.0.1:8080), FRIENDLY-HOST and FRIENDLY-PORT is the address of sshd on a machine outside of your LAN (like unix.rulez.org:443), USER is the username of your account there, and TARGET-HOST and TARGET-PORT is the destination address, the command will be (subprocess "/usr/bin/ssh" "-o" "ProxyCommand=connect -H -P 127.0.0.1:8080 %%h %%p" "-l" "kizome" "-p" "443" "unix.rulez.org" "/usr/bin/nc" "%HOST" "%PORT").

• (multiplexer command args...)
  to start up the mpserver and use it for outgoing connections. The arguments are the same as for the subprocess method. You can use a keyword here, ping, to specify an interval in seconds after which it'll generate a PING message to avoid the closing of the connection by the proxy or firewall (see A Truly Complex Example).

(proxy-rules RULE RULE...)

With this you can control the HTTP proxy. Each RULE has the structure (DESCRIPTION METHODS SCHEME HOST PORTS CONNECTION-METHOD), where the DESCRIPTION, HOST and PORTS fields are the same as for the router-rules, METHODS is a list of HTTP methods like get, put, post, head, connect, etc., and SCHEME is a regular expression to match the scheme part of the requested URL. As for the router-rules, any of the METHOD, SCHEME, HOST and PORTS fields can be #t to match everything. The possible connection methods are:

• direct
  to connect to the target host directly (by the connection method selected by the router-rules)

• (proxy HOST PORT)
  to connect through a proxy at HOST:PORT.

• (msproxy HOST PORT NT-DOMAIN NT-USER NT-HOST NT-PASSWORD)
  to connect to the MS proxy at HOST:PORT and authenticate as NT-USER on NT-HOST in NT-DOMAIN, with password NT-PASSWORD, using the NTLM authentication scheme. The fields should describe a user with rights to access the given resources, and they don't have to be the same what you use to login to the windows PCs on the site.

• (or alt1 alt2...)
  Try the alternatives in order and accept the first one which can access the destination. The alternatives can be any of the valid connection methods (direct, proxy or msproxy). Proxies fail if the response is not within the range between 200 and 400, and direct connection fails if it can't connect to the destination.

(dns-rules RULE RULE...)

These are used by the DNS proxy to decide whom to ask about the host in question. The structure of the rules are (DESCRIPTION HOST SERVER-IP) and HOST is a regular expression to match with the NAME part of the DNS query messages (so for PTR queries about the ip A.B.C.D it should be matched with D.C.B.A.in-addr.arpa).

Special Variables

There are also some special variables that you can set with define.

• debug
  Possible values are #t (true) or #f (false). Set it to true (#t) if you want to see diagnostic messages on the program's standard output.

• anonftp-pwd
  Anonymous FTP password. You must specify it if you plan to use the FTP proxy as there's no sensible default value for it.

• dns-proxy/udp
  Set it to false to disable the UDP listener of the dns-proxy, or to a list of (IP PORT) to make it listen on the given port of the given ip, e.g. ("127.0.0.1" 53). With a value of "127.0.0.1" as the IP, you disable listening to connections not originating on your machine, and with a value of #f, it'll listen on every interface.

• dns-proxy/tcp
  It's the same as dns-proxy/udp but for the TCP listener.

• http-proxy
  With this you can specify the IP and the port where the proxy will listen to incoming connections. Its behaviour is the same as of dns-proxy/udp.

A Simple Example

Let's suppose you're behind a MS proxy which gives you access to nearly everything what you care about, but it needs you to authenticate yourself NTLM authentication. And once you're at it, you also want to set it up so you don't have to use those proxy auto-configuration files or can use browsers with no PAC support without having to change the setup every time you want to access some local servers.

Here the local net is something like 10.0.0.0/8, and the domain is intra.my.company.com.

; turn on debug mode for a lots of diagnostic messages on stdout
(define debug #t)

; ip of the ms proxy and the port on which it listens
(define master-proxy-host "10.x.y.z")
(define master-proxy-port 8080)

; the nt domain name, your machine's name as the nt server knows it,
; your user in that domain, and the password. these should be the things
; the proxy expects to let you go, not neccessarily the ones with what
; you log in to the windows machines.
(define nt-domain "BLADOM")
(define nt-host "MYHOST")
(define nt-user "KIZOME")
(define nt-password "secret")

; here we just define a proxy connection using the msproxy function.
; it takes the information specified above, and creates a proxy
; function which responds to the requests by opening a connection
; to the proxy, doing the NTLM authentication and sending it the request.
(define master-proxy (msproxy master-proxy-host master-proxy-port
                              nt-domain nt-user nt-host nt-password))

; these are the rules for the proxy. they're tested in order and the
; first match is selected. the first two rules check for local targets
; (ip 10.x.x.x or *.intra.my.company.com) and go for them directly,
; and the last rule matches everything else and tries to go through
; the MS proxy using the connection method defined above.
(proxy-rules
 ("Local/IP" #t #t "^10\\.[0-9]+\\.[0-9]+\\.[0-9]+$" #t direct)
 ("Local"    #t #t "^[^.]+(\\.intra\\.my\\.company\\.com)?$" #t direct)
 ("Internet" #t #t #t master-proxy))

; and finally the servers. we don't need dns proxying here, so we just
; switch them off.
(define dns-proxy/udp #f)
(define dns-proxy/tcp #f)
; the http proxy will listen only on the loopback address, so no
; connections from other hosts allowed. if you want it to listen on
; all interfaces, you can change it to #f (false).
(define http-proxy ("127.0.0.1" local-proxy-port))

A Not So Simple Example

This is just like my config was before the version 0.3. With this, I can download anything with http scheme on port 80 or with ftp scheme on port 21 using the GET method of the proxy except when the site has 'download' in its name, and I can make connections on the port 443, and I have an sshd listening on port 443 on a machine of a friend of mine.

; These are the same as in the simple example above
(define debug #t)

(define master-proxy-host "10.x.y.z")
(define master-proxy-port 8080)

(define nt-domain "BLADOM")
(define nt-host "MYHOST")
(define nt-user "KIZOME")
(define nt-password "secret")

(define master-proxy (msproxy master-proxy-host master-proxy-port
                              nt-domain nt-user nt-host nt-password))

; here are the real name servers to use. one local to the site and one
; external.
(define local-dns "10.a.b.c")
(define external-dns "195.e.f.g")

; and here is the port our proxy will listen on for incoming
; connections
(define local-proxy-port 8080)

; this is a connection method to be used in the router-rules ruleset
; using an ssh subprocess to open raw (not http) connections by
; connecting to a friendly machine with an sshd listening on a port on
; which outgoing CONNECT requests are allowed on the master-proxy (in
; our case, 443)
(define my-way-out
        (subprocess "/usr/bin/ssh"
                    "-o" "ProxyCommand=connect -H -P 127.0.0.1:8080 %%h %%p"
                    "-l" "kizome" "-p" "443" "unix.rulez.org"
                    "/usr/bin/nc" "%HOST" "%PORT"))

; you should understand these rules without major problems by now. the
; first two rules match the local hosts by IP and by name. the third
; one matches everything on the port 443 not matched by the first two
; and sends them through the proxy, and the last one sends the rest
; through ssh.
(router-rules
 ("Local/IP"        "^10\\.[0-9]+\\.[0-9]+\\.[0-9]+$" #t direct)
 ("Local"           "^[^.]+(\\.intra\\.my\\.company\\.com)?)$" #t direct)
 ("HTTPS"           #t (443) (proxy))
 ("Everything else" #t #t my-way-out))

; the proxy rules. here again the first to make the proxy connect directly
; to the hosts local to the site. it also connects directly to hosts with
; the word 'download' in their name, but because of the router-rules
; above, it will fire up an ssh which in turn connects to the proxy
; trying to go out on the port 443. the fourth rule is for ftp and it
; only supports the get method on port 21, and it will ask the master
; proxy for the document. the fifth rule is the same for http but supporting
; more methods, and the last one is for the connections through the port 443.
; everything not matched by these rules will be connected directly (through
; the router ruleset).
(proxy-rules
 ("Local/IP"                #t #t "^10\\.[0-9]+\\.[0-9]+\\.[0-9]+$" #t direct)
 ("Local"                   #t #t "^[^.]+(\\.intra\\.my\\.company\\.com)?$" #t direct)
 ("Things named 'download'" (get put post head) "^http$" "download" #t direct)
 ("FTP"                     (get) "^ftp$" #t (21) master-proxy)
 ("Standard HTTP"           (get put post head) "^http$" #t (80) master-proxy)
 ("HTTPS"                   (connect) #t #t (443) master-proxy))

; the DNS rules. again, the first two is for the machines local to the
; site (note that here we have no IP to match but the in-addr.arpa
; space. the last rule will connect for information to a name server
; on the 'net using the router ruleset
(dns-rules
 ("Local"         "^[^.]+(\\.intra\\.my\\.company\\.com)?$" local-dns)
 ("Local/Reverse" "^[0-9]+\\.[0-9]+\\.[0-9]+\\.10\\.in-addr\\.arpa$" local-dns)
 ("External"      #t external-dns))

; then we just start up the servers
(define listen-on-address "127.0.0.1")
(define dns-proxy/udp (listen-on-address 53))
(define dns-proxy/tcp (listen-on-address 53))
(define http-proxy (listen-on-address local-proxy-port))

A Truly Complex Example

This is just like my configuration as of now. It uses the same ports as the previous example, but it uses the stream multiplexer, and alternative connection methods so that I can access sites disabled in the local proxy without knowing about it, or sites unreachables by the local proxy but usables by the external machine.

Here I've only commented the new things.

; These are the same as in the simple example above
(define debug #t)

(define master-proxy-host "10.x.y.z")
(define master-proxy-port 8080)

(define nt-domain "BLADOM")
(define nt-host "MYHOST")
(define nt-user "KIZOME")
(define nt-password "secret")

(define master-proxy (msproxy master-proxy-host master-proxy-port
                              nt-domain nt-user nt-host nt-password))

(define local-dns "10.a.b.c")
(define external-dns "195.e.f.g")

(define local-proxy-port 8080)

; Set the anonymous FTP password to your email address
(define anonftp-pwd "kizome@unix.r0x.org")

; this is the multiplexer definition to be used in the router ruleset.
; notice the use of ping: 60. this is an optional keyword
; argument, and leaving it out or specifying it as ping: #f
; results in no PING messages. also notice, that it calls ssh which in
; turn tries to connect to the external machine on port 443, and this is
; handled by the proxy-rule (or master-proxy direct), where the direct part
; is routed through the same multiplexer. of course it will not work when
; starting up the multiplexer itself, so it's better work through the
; proxy... later though it will route the connections through the multiplexer
; if they can not go through the proxy.
(define my-way-out
        (multiplexer ping: 60
                     "/usr/bin/ssh"
                     "-o" "ProxyCommand=connect -H -P 127.0.0.1:8080 %h %p"
                     "-l" "kizome" "-p" "443" "unix.rulez.org"
                     "/usr/bin/mzscheme" "-i" "~/kipt/mpserver"))

(router-rules
 ("Local/IP"        "^10\\.[0-9]+\\.[0-9]+\\.[0-9]+$" #t direct)
 ("Local"           "^[^.]+(\\.intra\\.my\\.company\\.com)?)$" #t direct)
 ("HTTPS"           #t (443) (proxy))
 ("Everything else" #t #t my-way-out))

; the proxy rules. here i have two alternatives for the standard HTTP and
; HTTPS connections, one through the master proxy and the other through the
; multiplexer (as direct connections to the outside get routed through it).
; Note that here I route all FTP traffic through the direct connection
; method which possibly causes connecting to the FTP server through the
; external machine.
(proxy-rules
 ("Local/IP"                #t #t "^10\\.[0-9]+\\.[0-9]+\\.[0-9]+$" #t direct)
 ("Local"                   #t #t "^[^.]+(\\.intra\\.my\\.company\\.com)?$" #t direct)
 ("Things named 'download'" (get put post head) "^http$" "download" #t direct)
 ("FTP"                     (get) "^ftp$" #t (21) direct)
 ("Standard HTTP"           (get put post head) "^http$" #t (80) (or master-proxy direct))
 ("HTTPS"                   (connect) #t #t (443) (or master-proxy direct)))

(dns-rules
 ("Local"         "^[^.]+(\\.intra\\.my\\.company\\.com)?$" local-dns)
 ("Local/Reverse" "^[0-9]+\\.[0-9]+\\.[0-9]+\\.10\\.in-addr\\.arpa$" local-dns)
 ("External"      #t external-dns))

; here we set the servers to listen on all interfaces
(define listen-on-address #f)
(define dns-proxy/udp (listen-on-address 53))
(define dns-proxy/tcp (listen-on-address 53))
(define http-proxy (listen-on-address local-proxy-port))

mpserver

mpserver is the stream multiplexer server. It should be installed on an external machine (together with mzscheme). It does not accept command line arguments, and it is not intented to be run manually.

smtpclient

smtpclient is an SMTP sender program to be used with postfix. It understands the same config file as KIPT, but if used together with KIPT, it should be quite simple.

Command line

The program excepts three flags and the list of recipients on the command line. The flags are -d domain to specify the domain from which it will appear to send the mail, -f from to specify the sender, and -s server to specify the server's name. This name will be looked up first for an MX record, and connected directly if none was found. The rest of the arguments must be raw email addresses.

It also accepts an optional -F config-file flag, so you can override the location and name of the config file, which defaults to ~/.smtpclient.conf. Be careful not to turn on the debug mode in this file, as it will probably confuse the MTA.

Setting up postfix

To use the smtpclient with postfix (to be able to send mails through the proxy), you have to change the master.cf file of postfix. Look for the line

smtp      unix  -       -       n       -       -       smtp

and change it to

smtp      unix  -       n       n       -       -       pipe
  user=youruser:yourgroup argv=/usr/bin/mzscheme -i yourhome/kipt-0.4/smtpclient \
  -F yourhome/smtpclient.conf \
  -f $sender -s $nexthop -d your.organization.com ${recipient}

(the last three lines go to a single line in master.cf) and that's all. Don't forget to reload postfix, and be careful, as it seems that pipe is ignoring the non-zero exit code (which usually means an unsuccesful attempt to send the mail).

Configuring the SMTP client

As it understands the same configuration options as KIPT, there is not much to explain. The most important thing to keep in mind is that simply accessing everything through the proxy provided by a running KIPT (instead of keeping all the routes duplicated here) makes things a lot easier and probably faster too.

Here is my configuration:

(define debug #f)

(define dns "127.0.0.1")
(define proxy-port 8080)
(define master-proxy (proxy "127.0.0.1" proxy-port))

(router-rules
 ("Everything through proxy" #t #t (proxy)))

(proxy-rules
 ("Everything through proxy" #t #t #t #t master-proxy))

(dns-rules
 ("Everything" #t dns))