Command Line FTP Client

Automated file back up can help businesses improve efficiency. You do not need to manually copy company related files to your storage device, however.

FTP file hosting offers secure storage and automatic backups using a backup software. This is a cost-effective service that is easy to implement. You can also use the service's API and integrate it with your business software or CRM. This allows you to automatically upload data to an online server. Having a centralized storage server allows other employees from other departments or branches to access company data. Even if your system crashes, you can still access your files or customer records using a command line FTP client on another computer with Internet connection. It is also a cheaper solution because you do not need to maintain and purchase new hardware. You just need to pay for the monthly service fee for the storage space plan.

SmartFile offers effective storage solutions to help improve your business. We have several storage plans available to suit your needs. Sign up for a 30-day free trial by doing so, you'll have access to all the features the service offers.

File Transfer Protocol is a protocol service that allows files to be transferred from computers to networks and vice versa. The exchange of files can occur between user accounts, various computers and online software archives. You can access this service using graphical FTP clients where you log in using a username and password or you can access it via your web browser by entering the URL. Most computer operating systems today come with a FTP client. If you are comfortable using a file transfer protocol client, all you need to do is to enter the FTP Server Commands you need.

From your computer, you can transfer files just by entering a set of commands in the command prompt line. Take note that you will enter the commands into the local machine (the computer you are logged on to) to connect to the remote machine. These commands like “abor” (abort file transfer) and “quit” (terminate connection) all correspond to particular actions in the network; FTP is the mediator between the two machines and acts like an interpreter that makes sure certain actions are executed in the remote machine. Basically, the user sends FTP commands to be received by another network to facilitate the transfer of files.

SmartFile can take care of all your file management needs. Take advantage of our free 14-day trial today.

Managing a business can take a lot of your time, and it is easy to forget things you have to do such as making sure your files are always updated, keeping track of bills, purchases and client requests. A simple mistake can make or break your business. This is the reason it's important to have a system that can organize and keep your files updated. 

Investing on an effective file management service will greatly help keep your business organized. If your business depends heavily on online transactions, make sure to get a web based file manager you can rely on. There are numerous online file management systems on the market today. If you don't choose the one that provides all your business management needs, you'll just be throwing money away. Get a service that will give you your money's worth.

Signing up for SmartFile's service gives you more than just online storage for your files. With our user-friendly interface, you won't have to learn numerous command lines and codes. You will be able to easily integrate our service to your own system and access your account from a web based control panel. Our FTP service also allows you to manage and share your files safely through secure file encryption. Sign up for a free 14 day trial to see how easy it is to manage your business with our service.

    On a few occassions I have been asked "Why do I need an FTP client? Can't I do everyting with my FTP site?".  The SmartFile application has many features, but some are better handled by a FTP client. Here are some advantages to using an FTP client.

• Allows you to transfer multiple files as well as directories
• The ability to resume a transfer is the connection is lost
• The ablilty to add items to a "queue" to be uploaded/downloaded
• Many FTP clients have the ability to schedule transfers
• No size limitation on single transfers (browsers only allow up to 2 GB)
• Many clients have scripting capabilities through command line
• Most clients have a synchronizing utility
• Faster transfers then HTTP

Those are just a few of the advantages of using an FTP client. While the clients help make transfers easier they are not without their drawbacks. Here are some of the disadvantages of using and FTP client.

• Usernames, passwords and files are sent in clear text
• filtering active FTP connections is difficult on your local machine (passive is preferred)
• servers can be spoofed to send data to a random port on an unintended computer

In those cases you can see where using the web interface would eliminate these issues. Both FTP clients and the FTP site have their advantages and disadvantages. Fortunately you are able to use them hand in hand depending on your specific needs.

As browsers continue to advance more and more of these features will be worked in. In the future the need for a third party client may be unnecessary, but for now they are really nice to have.

    We have already gone through a few of that FTP programs that we have tested with SmartFile. The newest program I have tested is Auto FTP Manager. The look of the program is pretty standard issue; however, it does offer several unique features.

The first is the server-to-server connection. Most FTP clients only allow you to connect from your local PC to the server. In this case you would need to connect to the server you are requesting files from and download them to your local machine. Once downloaded, you could connect to the second server and upload the files. This can be very time consuming depending on the amount of files you have. Auto FTP allows you to make a connection to server 2 from server 1 and copy files directly between them without making a local copy. This is my favorite feature of the program.

Next is the ability to set up profiles and use rules and filters to automate the file transfer process. An example is that I can create rules so that when I upload files to the server, it will automatically delete them from my local machine when it is finished. Also once the transfer is complete I can move the files to a specific backup folder on the server. This can all be set up to be done automatically everytime I transfer files.

It also has the capabilities to schedule transfers. Using its command line interface and the integrated scheduler it can run as a scheduled task and automatically upload files based on the name, modification date, or file type. In that respect it can almost run as a backup utility. It can run at specific times (hourly, daily, etc.) to FTP files automatically.

There are several other useful features Auto FTP offers. There are only two drawbacks that I see. The first is that it runs on Windows only. It doesnt have a multi platform install like some other FTP programs. The second is the supported protocols. While it does support FTP and FTPS, that is where it stops. It does not have the option to use FTPES which is the secure protol SmartFile uses for FTP.

Overall this is a really good ftp program. I would definately recommed this for a Windows users needing a substantial FTP program.

In a previous post, I explained why PASV mode is the preferred method to connect to an FTP server. That is all well and good, but how do you actually USE PASV mode?

The FTP client that ships with Windows does not support PASV mode. Google will tell you that it does, by using the raw command. However, this is not true, you cannot perform PASV connections using the Windows FTP client. It can only do active connections. So if you are behind a firewall that does not allow this, how do you do command line FTP?

Command line FTP is useful for scheduled operations often performed from a script. On Windows, this might be a batch file executed by the Windows Task Scheduler. Well, if you wish to automate a file transfer in this way, you will almost certainly have to download a third party FTP client that supports PASV data channels.

One FTP client that is available for this task is ncFTP. For uploading or downloading a file inside a script, you can use the ncftpget and ncftpput programs respectively.

For example, to download a file from an FTP server using PASV mode, you can use ncftpget with the following options.

ncftpget -F -u <username> -p <password> <site name>.smartfile.com /path/to/remote C:pathtolocal

SmartFile provides a backup client for Windows. However, if you have Linux servers, it is just as important to back them up as well. Since SmartFile provides FTP access to your space, this task can be easily accomplished with some tools you likely already have installed.

This article will detail the steps to perform a simple, safe, encrypted backup directly to the SmartFile servers. At the end of the article a script will be provided that you can simply install onto your system to perform nightly backups.

Required Tools

Tar is historically for creating Tape ARchives. Thus backing up to a tape would usually involve using tar. Tar has many features including compression and is great for performing backups of Linux systems. Not only can it write to a tape, but also to a file on disk. Further, it can write the archive to stdout, so it can be fed into another program.

OpenSSL is an open source library and command line application that is capable of performing myriad encryption tasks. It is basically the swiss army knife of encryption for Linux systems. For our purposes, we will use it to encrypt our backup file before sending it to the FTP server. By default openssl will read input from stdin and output to stdout. This is perfect for our purposes.

cURL is a network client that is URL driven. It allows uploading or downloading to or from FTP or HTTP servers. For us, the main feature that cURL provides is that you can stream data directly to a file on an FTP server. Let me explain, while most FTP clients will allow you to upload a file from your file system to an FTP server this requires that the file you wish to send to the FTP server already exist on your disk. What is wanted for our backup is a way to “stream” the backup file directly to the FTP server without touching the local disk. cURL provides this with the -T option. If -T is passed – as the file, then the file data is read from stdin.

Now that we are familiar with the tools, let’s take a look at how we will use them all together. Linux allows multiple commands to be chained together by piping the output (stdout) of one command on to the input (stdin) of another command. The | or pipe character is used for this purpose. Thus at a high level, we will be doing the following.

tar | openssl | curl

Tar will create the backup of our system, openssl will then encrypt that backup and curl will transfer it to the FTP server, all without creating any temporary files that we would otherwise need to be cleaned up later.

All that remains is to determine what parameters each of the above commands needs to be given to get the behavior we want.

Tar – Parameters.

To create an archive, you use the c option. To compress the archive using Bzip2, you use the j option. Since we want to back up the entire system, our tar command thus far is.

tar cj /

By omitting any option to save the archive to disk, tar will by default output it to stdout. This allows us to pass the archive data to the next program in our chain without saving it to disk.

There are certain directories within your Linux system that should not be backed up. Some examples are:

• /proc – The proc file system is provided by the Linux kernel and contains information about running programs.
• /sys – The sys file system is provided by the kernel and contains information about hardware.
• /dev – The dev file system consists of device nodes, which represent Linux device drivers.

Backing up the above directories would be folly, as they are provided by the kernel, and some of them (/dev/zero) are actually infinite in size. So, the second set of parameters we will pass to tar will exclude these file systems.
tar cj / --exclude=/proc --exclude=/dev --exclude=/sys

You may also wish to exclude /mnt, as generally you will have other file systems mounted there. These may be remote file systems that are already being backed up via other means. Of course, /mnt may contain file systems that you wish to back up. Your system configuration will dictate your choice here.

OpenSSL – Parameters.

We want openssl to perform encryption, thus we pass it the enc option. Also, I have opted to use the aes-256 algorithm in cbc mode, so we must pass that as well. Finally, openssl requires a key to perform our encryption. This key will be derived from a passphrase, this derivative procedure will use a salt value, so we also provide that option. We will store the passphrase in a file, so that openssl can retrieve it from that file.

openssl enc -aes-256-cbc -salt -pass file:/etc/backup-key

And we can create the key by doing the following.

echo 'This is my backup key!' > /etc/backup-key
chmod 400 /etc/backup-key

Of course, you are well-advised to use something other than the example key above.

cURL – Parameters.

Now, the final step in our backup procedure is to actually transfer the file to SmartFile. We will do this using cURL and the FTP protocol. cURL is driven by URLs, so we must provide one.

curl ftp://www.smartfile.com/backup/

This tells curl to connect to www.smartfile.com and move into the backup directory. However, if the backup directory does not exist, curl will fail. Thus we will ask curl to create it for us if it does not exist.

curl --ftp-create-dirs ftp://www.smartfile.com/backup/

Now, as I alluded to before, we want curl to upload the data that it receives from it’s stdin. This is achieved by using the -T option like so.

curl --ftp-create-dirs -T - ftp://www.smartfile.com/backup/

If we want to use SSL, there are a couple of other options to provide. I suggest skipping SSL if you are already encrypting the backup file. However, if you want to use SSL, you would use the following parameters.

curl --ftp-create-dirs --ftp-ssl --ftp-ssl-reqd --insecure -T - ftp://www.smartfile.com/backup/

We are almost done, the final bit of information that curl needs is a username and password. We could have provided it as part of the URL, but that would expose our credentials to anyone snooping on the machine while the backup is running. It is safer to place the credentials into a file and instruct curl to retrieve them from the file. cURL is capable of doing this using a .netrc file. You can create the .netrc file like so.

echo machine www.smartfile.com login <username> password <password> > ~/.netrc
chmod 400 ~/.netrc

Of course, replace <username> and <password> with your username and password respectively. Now we instruct cURL to use our new .netrc file.

curl --ftp-create-dirs --ftp-ssl --ftp-ssl-reqd --insecure --netrc -T - ftp://www.smartfile.com/backup/

Putting it all together.

Now that you understand the basic building blocks of our backup to FTP solution. Please allow me provide you with a working script. This script was written and tested on CentOS 5.4. Some of the utilities used are out-of-date, for example, the version of curl available from the CentOS repositories uses some deprecated options, on other distributions, you may need to make modifications to these options. You will need to edit the configuration section of the script if you want to customize the behavior.

To install and use this backup script follow the steps below.

Download the script in the following location and ensure it is executable.
wget http://www.smartfile.com/downloads/smartfile-backup.sh -O /usr/local/bin/smartfile-backup.sh
chmod +x /usr/local/bin/smartfile-backup.sh
Customize the configuration section.
Create your key and .netrc files as directed above.
Finally, schedule it to run with cron. The example below will run at midnight every night.
crontab -e
0 0 * * * /usr/local/bin/smartfile-backup.sh
You can also run the script manually to ensure it works properly.

/bin/bash -x /usr/local/bin/smartfile-backup.sh

Restoring from a backup.

To restore the backup, or to retrieve files from the backup you can follow the steps below.

1. Download the backup file.
2. Decrypt the backup file.
3. Use tar to extract what you need.

Download the backup file.

You can either use the SmartFile web interface or FTP to retrieve the file.

Decrypt the backup file.

You can use OpenSSL to decrypt the file. The following command line would do the trick.

openssl enc -d -aes-256-cbc -salt -pass pass:'This is my backup key!' -in full-2010-06-03.tar.bz2 -out full-2010-06-03.tar.bz2.dec

Use tar to extract what you need.

You can either extract the entire archive or a portion of it. Below are commands to perform either task. For more information, read the tar man page..

mkdir /tmp/restore
tar xjf full-2010-06-03.tar.bz2.dec -C /tmp/restore

mkdir /tmp/restore
tar xjf full-2010-06-03.tar.bz2.dec -C /tmp/restore /path/to/file

** Note **
You may receive the following warning during extraction:

bzip2: (stdin): trailing garbage after EOF ignored

This seems harmless, you can get rid of it by either writing the archive to disk before transfer or using gzip instead of bzip2. The archive still decompresses fine, but tar is apparently outputting some additional garbage when using bzip2 and outputting to stdout. I personally still using bzip2 and stdout, as the advantages (greater compression ratio, no temp disk space required) outweigh the disadvantages.