Lurgle.Transfer - A standardised SSH.NET and FluentFTP implementation with extra goodies!

Standardising transfers

I mentioned some time back that I had developed an internal app for critical file transfers. It's quite a useful application that has developed over time to support most of our critical file transfers. The functionality that it uses has been predominantly based around SFTP transfers using SSH.NET, and it has a number of capabilities, including:

• unattended upload and download of files to/from multiple configured destinations
• compression of upload files to Gzip, ZIP of all files, and individual ZIP for each file
• conversion of PDF files to lower or higher versions
• detailed logging (using Lurgle.Logging)
• email alerting (using Lurgle.Alerting)
• archival of source upload files
• cleanup of old archive files
• integrated exit codes

This internal application used a library that I had created to handle a majority of these functions. That library was well suited to become a 'common' transfer library for use in our other applications.

Conversion to a common library

When I started work on the necessary conversion to create Lurgle.Transfer, I also reviewed older code that had been the predecessor of the transfer app. This code included a capability of configuring either FTP (using FluentFTP) or SFTP connections. Although SFTP using SSH is far preferable to FTP, it's true that there are times that FTP is the lowest common denominator and you need it. An example might be downloading from a third party FTP site and then uploading via SFTP over SSH. 

Hence - apps using Lurgle.Transfer can be configured to either use FTP or SFTP connections. The library abstracts the transfer methods so that the same calls can work for both FTP and SFTP, and return coherent results for either case. 

As the PDF conversion functionality uses a licensed library, and is really outside the scope of requirements for Lurgle.Transfer, I haven't included this in the conversion.

Configuration

Lurgle.Transfer uses a prescriptive approach for configuration using the TransferConfig and TransferDestination classes, and as with other Lurgle libraries, can be configured using constructors or via app.config. 

TransferConfig is a simple class that contains only a few config key value pairs.

    <add key="AppName" value="Test" />

    <!-- Global paths that can be overridden on a per-destination basis-->
    <add key="SourcePath" value="C:\Transfer\Upload" />
    <add key="DestPath" value="C:\Transfer\Download" />
    <add key="DoArchive" value="false" />
    <add key="ArchivePath" value="C:\Transfer\Archive" />
    <add key="ArchiveDays" value="30" />

    <!-- Provides the key names for each destination, and the order to execute the transfers-->
    <add key="TransferDestinations" value="SftpUpload,SftpDownload,FtpUpload,FtpDownload" />

AppName is a universal for Lurgle libraries. If using app.config, AppName will be read in for Lurgle.Logging, Lurgle.Alerting, and Lurgle.Transfer - or automatically determined from the executable.

SourcePath, DestPath, DoArchive, ArchivePath, and ArchiveDays are all configurations that can be set here, as global values, and optionally overridden at a per-destination level.

TransferDestinations has two functions. The first is to tell Lurgle.Transfer what TransferDestination configurations look for, and the second is to allow configuration of what order to execute them in. The format is a comma-delimited string, and if using app.config, each value should correlate to a group of keyvalue pairs that are prefixed with the same name. 

For example, a simple constructor-based configuration of an SFTP upload might look like:

            Dictionary<string, TransferDestination> destinations = new Dictionary<string, TransferDestination>
            {
                {"Test",
                new TransferDestination(name: "Test", transferType: TransferType.Upload,
                    transferMode: TransferMode.Sftp, destination: "Test", authMode: TransferAuth.Password,
                    bufferSize: 262144, server: "127.0.0.1", port: 22, userName: "TestUser", password: "TestPassword",
                    remotePath: "/upload", sourcePath: "C:\\Test")}
            };
            Transfers.SetConfig(new TransferConfig(transferDestinations: destinations));
            Transfers.Init();

while an app.config upload configuration might look like:

    <add key="SftpUploadName" value="Upload via SFTP" />
    <add key="SftpUploadTransferType" value="Upload" />
    <add key="SftpUploadTransferMode" value="SFTP" />
    <add key="SftpUploadAuthMode" value="Password" />
    <add key="SftpUploadBufferSize" value="262144" />
    <add key="SftpUploadServer" value="192.168.1.181" />
    <add key="SftpUploadPort" value="22" />
    <add key="SftpUploadUsePassive" value="false" />
    <add key="SftpUploadRemotePath" value="upload" />
    <add key="SftpUploadSourcePath" value="C:\Transfer\Upload"/>
    <add key="SftpUploadUserName" value="FtpTest" />
    <add key="SftpUploadPassword" value="Password1" />
    <add key="SftpUploadRetryCount" value="3" />
    <add key="SftpUploadRetryDelay" value="10" />

You will note that the app.config upload configuration has a prefix to the properties, and this is because the TransferDestination configuration is used to configure multiple destinations as a comma-delimited string. This is shown in the constructor-based configuration, where we have configured that the TransferDestination that is named Test will be used.

You can specify as many TransferDestinations as you would like.

Configuration Properties

The full set of properties available for TransferDestination is extensive. Here's an extract from the LurgleTest app that was created to provide a sample implementation.

    <!-- Destination configurations-->
    <add key="FtpUploadName" value="Upload via FTP" />
    <add key="FtpUploadTransferType" value="Upload" />
    <add key="FtpUploadTransferMode" value="FTP" />
    <add key="FtpUploadAuthMode" value="Password" />
    <!-- Used with Certificate or Both authmodes-->
    <!--<add key="FtpUploadCertPath" value=""/>-->
    <add key="FtpUploadBufferSize" value="262144" />
    <add key="FtpUploadServer" value="192.168.1.181" />
    <add key="FtpUploadPort" value="21" />
    <!-- This will only affect FTP transfer modes -->
    <add key="FtpUploadUsePassive" value="false" />
    <add key="FtpUploadRemotePath" value="upload" />
    <!-- Per destination overrides for Source, Dest, and Archive Path-->
    <add key="FtpUploadSourcePath" value="C:\Transfer\Upload" />
    <!-- Used for download-->
    <!--<add key="FtpUploadDestPath" value="C:\Transfer\Download"/>-->
    <!-- Unused for LurgleTest. These work with the Files.ArchiveFiles method-->
    <!--<add key="DoArchive" value="False"/>
    <add key="ArchivePath" value="C:\Transfer\Archive"/>
    <add key="ArchiveDays" value="30"/>-->
    <add key="FtpUploadUserName" value="FtpTest" />
    <add key="FtpUploadPassword" value="Password1" />
    <add key="FtpUploadRetryCount" value="3" />
    <add key="FtpUploadRetryDelay" value="10" />
    <!-- Only useful for debugging-->
    <!--<add key="FtpUploadRetryTest" value="false" />
    <add key="FtpUploadRetryFailAll" value="false" />
    <add key="FtpUploadRetryFailConnect" value="false" />-->
    <!--Enable and configure proxy-->
    <!--<add key="FtpUploadUseProxy" value="false" />
    <add key="FtpUploadProxyType" value="Http" />
    <add key="FtpUploadProxyServer" value="" />
    <add key="FtpUploadProxyPort" value="8081" />
    <add key="FtpUploadProxyUser" value="" />
    <add key="FtpUploadProxyPassword" value="" />-->
    <!-- Unused for LurgleTest. These work with the Files.CompressFiles method -->
    <!--<add key="FtpUploadCompressType" value="gzip" />
    <add key="FtpUploadZipPrefix" value="" />-->
    <!-- Unused for LurgleTest. These allow for an alert to be defined on success/failure, which can be used in conjunction with Lurgle.Alerting-->
    <!--<add key="FtpUploadMailTo" value="[email protected]" />
    <add key="FtpUploadMailToError" value="[email protected]" />
    <add key="FtpUploadMailIfError" value="true" />
    <add key="FtpUploadMailIfSuccess" value="false" />-->
    <!-- Used with download. 0 means any age files, greater than 0 provides a limit to the age of files-->
    <!--<add key="FtpUploadDownloadDays" value="1" />-->

These keys relate to functionality in the in-house transfer app, and are retained for compatibility. 

Sample Implementation

While LurgleTest in the Lurgle.Transfer repository provides a sample of implementation, a more simplistic example that uses all functionality would be:

var destinations = new Dictionary<string, TransferDestination>
            {
                {"Test",
                    new TransferDestination(name: "Test", transferType: TransferType.Upload,
                        transferMode: TransferMode.Ftp, destination: "Test", authMode: TransferAuth.Password,
                        bufferSize: 262144, server: "127.0.0.1", port: 21, userName: "FtpTest", password: "Password1",
                        remotePath: "upload", sourcePath: "C:\\Transfer\\Upload", doArchive: true, archivePath: "C:\\Transfer\\Archive", archiveDays: 30)}
            };
            
            Transfers.SetConfig(new TransferConfig(transferDestinations: destinations));
            Transfers.Init();
            var ftransfer = new FileTransfer(destinations["Test"]);
            var files = Files.CompressFiles(ftransfer.TransferConfig, CompressType.Gzip);
            ftransfer.Connect();
            ftransfer.SendFiles(files.DestFiles, true, true);
            ftransfer.Disconnect();
            Files.DeleteCompressedFiles(ftransfer.TransferConfig, CompressType.Gzip, files.DestFiles);
            Files.CleanArchivedFiles(ftransfer.TransferConfig);
            Files.ArchiveFiles(ftransfer.TransferConfig);

Of course, in this example we aren't paying attention to the result of anything, but it's worth noting that we provide functions for compression and cleanup of the compressed files, as well as archival of files and cleanup based on age of archived files.

Get it now!

There is a lot of power and capability in Lurgle.Transfer, and the methods provide detailed results that can be used in your code logic and in logging and alerting.

Download Lurgle.Transfer now, and check out the other Lurgles!