DataKitchen DataOps Documention

Action Nodes

Nodes that connect solely to Data Sources.

For the specific case of Action Nodes, Data Sources are intended to perform operations on sources that do not require the retrieval of data. Action Nodes are thus convenient to use when creating database schemas, execute scheduled stored procedures, etc. Actions Nodes do not leverage Data Sinks.

Description.json

{
    "type": "DKNode_Action",
    "description": "[YOUR DESCRIPTION HERE]"
}

Notebook.json

The notebook.json file is optional for Action Nodes.

Data Source Actions

Each Action Node also contains an /actions subdirectory containing Data Source configuration files.

Examples

Example1

{
    "type": "DKNode_Action",
    "description": "Sample action node, with postgres datasource."
}
{
    "name": "postgresql_datasource", 
    "type": "DKDataSource_PostgreSQL", 
    "config" : {{samples_postgres_config or {}}},
    "set-runtime-vars" : {
        "key_count": "global_key_count"
    },
    "keys": {
        "create_table": {
            "target-field": "sql", 
            "resource-file": "postgres/create.sql", 
            "query-type": "execute_scalar",
            "set-runtime-vars" : {
                "result" : "create_table_result"
            }
        },
        "insert_data": {
            "target-field": "sql", 
            "resource-file": "postgres/insert.sql", 
            "query-type": "execute_scalar",
            "set-runtime-vars" : {
                "result" : "insert_data_result"
            }            
        },
        "my_query": {
            "sql": "select * from {{samples_postgres_config.schema}}.{{redshifttablename}};", 
            "query-type": "execute_query",
            "set-runtime-vars" : {
                "column_count": "my_query_column_count",
                "row_count": "my_query_row_count"
            }
        }

    }
, 
    "tests": {
        "test-create-table": {
            "test-variable": "create_table", 
            "type": "test-contents-as-integer", 
            "applies-to-keys": ["create_table"], 
            "action": "stop-on-error", 
            "keep-history": "True", 
            "test-logic": "create_table == 0"
        },
        "test-insert-data": {
            "test-variable": "insert_data", 
            "type": "test-contents-as-integer", 
            "applies-to-keys": ["insert_data"], 
            "action": "stop-on-error", 
            "test-logic": "insert_data == 2"
        },
        "test-runtime_variable_create_table_result": {
            "test-variable": "create_table_result", 
            "action": "stop-on-error", 
            "test-logic": "create_table_result == 0"
        },
        "test-runtime_variable_insert_data_result": {
            "test-variable": "insert_data_result", 
            "action": "stop-on-error", 
            "test-logic": "insert_data_result == 2"
        },
        "test-runtime_variable_my_query_column_count": {
            "test-variable": "my_query_column_count", 
            "action": "stop-on-error", 
            "test-logic": "my_query_column_count == 2"
        },
        "test-runtime_variable_my_query_row_count": {
            "test-variable": "my_query_row_count", 
            "action": "stop-on-error", 
            "test-logic": "my_query_row_count == 2"
        },
        "test-runtime_variable_global_key_count": {
            "test-variable": "global_key_count", 
            "action": "stop-on-error", 
            "test-logic": "global_key_count == 3"
        }
    }

}

(source)

Example2

Where a data source and a data sink are tightly integrated, like with the case of AWS S3 and AWS Redshift, an Action Node can be used rather than a DataMapper as the "mappings" are handled by those services by default (here with the "COPY" SQL statement).

{
    "type": "DKNode_Action",
    "description": "Sample Action Node with an S3 Data Source and a pseudo Redshift Data Sink.  Because S3 and Redshift are tightly integrated one can move data between them without the need of a DataMapper Node with a Postgresql/Redshift Data Sink.
}
{
    "type": "DKDataSource_S3",
    "name": "s3_datasource",
    "config": {{s3config}},
    "keys": {},
    "tests": {}
}

(source)

Updated 4 months ago


Action Nodes


Nodes that connect solely to Data Sources.

Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.