excess.org

Ian Ward

Consulting
Boxkite Inc.
Software
CKAN contributor/tech lead
PyRF primary contributor
Urwid author
Speedometer author

Presentations
Contributing to Open Source
IASA E-Summit, 2014-05-16
Urwid Applications
2012-11-14
Urwid Intro
2012-01-22
Unfortunate Python
2011-12-19
Django 1.1
2009-05-16

Writing
Moving to Python 3
2011-02-17
Article Tags

Home

Ian Ward's email:
first name at this domain

wardi on OFTC, freenode and github

Locations of visitors to this page

File Tosser 1.0

Purpose

Copying files from one computer to another over a network should be a simple matter, yet the majority of tools complicate this task unnecessarily. File Tosser is a collection of scripts and code examples that demonstrate a very simple approach to serving files across a network.

File Tosser is not responsible for any file meta-data, and is unaware of hard links, symlinks and devices. If you need to send this sort of thing use tar(1) on your files first, or find another protocol.

Background

One of my favorite things about Linux is that it has let me recycle old boxen by turning them into firewalls. A few of the firewalls I've created have been only barely fast enough to keep up with ssh(1) connections running at cable-modem speed, and were limiting my scp(1) transfers on the LAN to a few hundred kbps. A good solution for moving the occasional file to or from it over the LAN is netcat - nc(1):

  1. On the sending computer:

    nc -l -p 12345 < file.to.send
    
  2. On the receiving computer:

    nc sending.computer 12345 > destination.file
    

This solution is good because it only requires a working TCP stack at both ends, however when I need to copy multiple files or copy files at as part of a cron job, this just won't cut it. "it" the original tosser was written to fill this role. It can be set to run from inetd as a normal user, and at less than 10 lines of shell script it is easy to verify that it is not a security risk.

Later I started using the original tosser on a remote box with an unreliable connection, launched from an sshd(8) authorized_keys file. Eventually I noticed that I was wasting time and cycles verifying RSA keys for each file when sending lots of small files. I was also having to re-send entire files when the connection failed during a transfer, wasting time and bandwidth. "ft" the improved tosser was written to solve these problems. ft is still pretty light at less than 30 lines of shell script.

File Tosser Servers

File Tosser Protocols

Things should be made as simple as possible – but no simpler.

it protocol:

  1. Client connects to server
  2. Client: Full path to file +CR
  3. Server: Complete contents of file (if file is in the access list)
  4. Server closes connection

ft protocol:

  1. Client connects to server
  2. Client: ":persist" +CR
    If client sends anything else, jump to "it protocol" step 2 to process input.
  3. (optional) Client: ":seek" +CR+ byte offset as decimal number +CR
  4. Client: Full path to file +CR
  5. If file is not in the access list server closes connection
  6. Server: File size as decimal number +CR
  7. Server: Contents of file starting from offset to the end of the file If no offset was specified for this file send complete contents of the file
  8. Client continues from step 3 or closes connection

Both protocols require an 8-bit clean connection like the one provided by ssh(1) or nc(1). telnet(1) is not 8-bit clean and will not work. Neither protocol can handle file names with embedded CRs.. but only a moron would do that.

Creating a File Tosser Server

  1. Set up a user account for the server

    adduser --disabled-password tosser
    

    Make sure the user has the required access to read all the files you want to serve, and execute the server script.

  2. Create an access list of files you want to serve

    The format is up to you. You could use a simple list of files:

    find -type f /path/to/shared/files >/home/tosser/access.list
    

    or a list with file lengths:

    find -type f /path/to/shared/files | while read fn; do echo `wc -c "$fn"`  
    done >/home/tosser/access.list
    

    or even md5sums:

    find -type f /path/to/shared/files | while read fn; do echo `md5sum "$fn"`  
    done >/home/tosser/access.list
    

    It makes sense to include the path to the access list in the access list so that a client can download the list of files it has access to (and perhaps verify files' sizes or md5sums)

  3. Choose a server and customize its settings

    Open the server script in your favorite text editor and change the following: If you are using a simple list of files:

    ACCESS_LIST="/bin/cat /home/tosser/access.list"
    

    If you included file lengths or md5sums:

    ACCESS_LIST="/usr/bin/cut -d' ' -f 2- /home/tosser/access.list"
    
  4. Trigger the server from inetd or ssh

    If you want to provide access to the server over a simple TCP connection with only network-level security, add a line like the following to /etc/inetd.conf:

    12345 stream tcp nowait tosser /usr/sbin/tcpd /path/to/server
    

    Use whatever port is convenient.

    If you want real security, generate some ssh keys for the clients that will be accessing your server and add a line for each client to /home/tosser/.ssh/authorized_keys like the following:

    command="/path/to/server",no-port-forwarding,no-X11-forwarding (ssh public key)
    

    to allow ssh(1) RSA login but keep password login disabled you need to edit /etc/shadow and change the tosser line that starts with:

    tosser:!: ...
    

    to:

    tosser:*: ...
    

File Tosser Clients

I don't use anything fancy at the moment, although I may write a tosser browser with my Urwid UI library and Speedometer code at some point.

To download the latest index of files from an inetd(8) tosser server:

echo /home/tosser/access.list | nc host port > access.list

or from an sshd(8) tosser server:

echo /home/tosser/access.list | ssh -i tosser_key tosser@host > access.list

To fetch a file from an inetd(8) tosser server:

echo /path/to/file_i_want | nc port host > dest_file

or from an sshd(8) tosser server:

echo /path/to/file_i_want | ssh -i tosser_key tosser@host > dest_file

Credits

Zygo Blaxell - for the original idea I expanded upon