DataKitchen DataOps Documention

FTP

Get and put data to and from Recipe Nodes via FTP.

Tool Documentation

Source & Sink Category

FTP Data Sources and Sinks are of the file-based category.

Type Codes

FTP
Type Code

Data Source

DKDataSource_FTP

Data Sink

DKDataSink_FTP

Connection Properties

Field Name
Scope
Field Type
Required?
Description

hostname

Source/Sink

HostInfo

Required

A valid host name / IP address.

passive

Source/Sink

Boolean

Optional

Boolean flag that enables/disables passive mode. Disabled by default.

password

Source/Sink

Password

Required

FTP service password.

port

Source/Sink

TCPPort

Required

FTP port.

username

Source/Sink

String

Required

FTP service user name.

Define at Kitchen Level

{
    "ftpconfig": {
        "passive": "use kitchen override",
        "username": "use kitchen override",
        "password": "use kitchen override",
        "hostname": "use kitchen override",
        "port": "use kitchen override"
    }
}
{
    "recipeoverrides": {
        "ftpconfig": {
            "passive": "#{vault://ftp/passive}",
            "username": "#{vault://ftp/username}",
            "password": "#{vault://ftp/password}",
            "hostname": "#{vault://ftp/hostname}",
            "port": "#{vault://ftp/port}"
        }
    }
}

Expanded Syntax

{
    "type": "DKDataSource_FTP",
    "name": "ftp_datasource",
    "passive": "{{ftpconfig.passive}}",
    "username": "{{ftpconfig.username}}",
    "password": "{{ftpconfig.password}}",
    "hostname": "{{ftpconfig.hostname}}",
    "port": "{{ftpconfig.port}}",
    "keys": {},
    "tests": {}
}
{
    "type": "DKDataSink_FTP",
    "name": "ftp_datasink",
    "passive": "{{ftpconfig.passive}}",
    "username": "{{ftpconfig.username}}",
    "password": "{{ftpconfig.password}}",
    "hostname": "{{ftpconfig.hostname}}",
    "port": "{{ftpconfig.port}}",
    "keys": {},
    "tests": {}
}

Condensed Syntax

Check Your Syntax

Do not use quotes for your condensed connection configuration variables.

{
    "type": "DKDataSource_FTP",
    "name": "ftp_datasource",
    "config": {{ftpconfig}},
    "keys": {},
    "tests": {}
}
{
    "type": "DKDataSink_FTP",
    "name": "ftp_datasink",
    "config": {{ftpconfig}},
    "keys": {},
    "tests": {}
}

Local Connection

FTP file contents may be viewed locally by configuring connections with file-transfer applications like Transmit.

Transmit 5 also supports FTP with Implicit SSL and FTP with TLS/SSL connection protocols.

Transmit 5 also supports FTP with Implicit SSL and FTP with TLS/SSL connection protocols.

Optional Properties

Pre-existing Path Requirement

FTP data sinks can only put files to existing output directories. They cannot create subfolders during processing. Other file-based data sinks, such as S3 or Google Cloud Storage, have the underlying functionality to create folders automatically.

Field Name
Scope
Field Type
Description

cache

Key

Boolean

Caches the data of a given data source. Other data sources of the same type and Key name may leveraged cached data so as to prevent repetitive retrievals. This feature does not presently support OrderRun resume and is thus not recommended.

complete

Key

Boolean

decrypt-key

Key (Sources)

String

The key used to decrypt a file.

decrypt-passphrase

Key (Sources)

String

The key passphrase used to decrypt a file.

encrypt-key

Key (Sinks)

String

Specifies the key used to encrypt a file.

file-key

Key

String

Denotes the file being picked/pushed for explicit keys. The path is built using key/file-key.
Specifies either the file name or the full path of the file. See use-only-file-key.

set-runtime-vars

Source/Sink, Key

Dictionary

Used to declare runtime variables set equal to built-in variables. Can be applied at the Source/Sink level or the Key level.

use-only-file-key

Key

Boolean

Optional, with a default value of false. When set to true, it uses only the value file-key as path/file spec to reference a file, ignoring the key. Applied at the Source/Sink level when using wildcards. For explicitly specified Keys, used at the Key-level for each Key.

Wildcards

Field Name
Scope
Field Type
Description

wildcard

Source/Sink

String

Specifies a glob wildcard expression to pick/push a set of arbitrary files that match the declared wildcard expression. Wildcards apply only to a single directory; they are not recursive. Use multiple Data Sources or Data Sinks if pulling or pushing files across multiple directories.

wildcard-key-prefix

Source/Sink

String

Specifies the path prefix for the given wildcard expression. When using wildcard mappings on DataMapper node, it specifies the prefix with the base path where files will be stored.

Built-in Runtime Variables

Built-in Runtime Variable
Scope
Description

key_count

Source/Sink

Exports the total count of Keys in a Data Source or Data Sink, regardless of whether a Key is wildcard-generated or explicitly configured by a user.

key_files

Source/Sink

Exports a list of file names with paths for all the files that match the wildcard.

key_names

Source/Sink

Exports a list of key names generated by the wildcard.

row_count

Key

Exports the line count of any kind of text file (txt, csv, json, etc.) associated to a key. If a header is present, it is included in the row count value.

size

Key

Exports the size of a file associated to a key, in bytes.

md5

Key

Exports the MD5 hash of a file associated to a key.

sha

Key

Exports the SHA hash of a file associated to a key.

Data Source Example

{
    "type": "DKDataSource_FTP",
    "name": "ftp_datasource",
    "config": {{ftpconfig}},
    "keys": {},
    "tests": {}
}


Data Sink Example

Example: Push a .csv File to FTP via an Explicit Key and Generate Runtime Variables for Tests

UI form configuration above, underlying JSON configuration below

UI form configuration above, underlying JSON configuration below

{
    "type": "DKDataSink_FTP",
    "name": "ftp_datasink",
    "username": "{{ftpconfig.username}}",
    "password": "{{ftpconfig.password}}",
    "hostname": "{{ftpconfig.host}}",
    "port": "{{ftpconfig.port}}",
    "set-runtime-vars": {
        "key_count": "ftp_sink_key_count", 
        "key_files": "ftp_sink_key_files", 
        "key_names": "ftp_sink_key_names"
    },
    "keys": {
        "key1": {
            "file-key": "ftp-test/sample-recipe-datamapper-salesforce-ftp.csv", 
            "use-only-file-key": true, 
            "set-runtime-vars": {
                "md5": "ftp_sink_md5", 
                "sha": "ftp_sink_sha", 
                "size": "ftp_sink_size", 
                "row_count": "ftp_sink_row_count"
            }

        }

    }
, 
    "tests": {
    
    }

}

Updated about a month ago


FTP


Get and put data to and from Recipe Nodes via FTP.

Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.