dkutils.datakitchen_api.datakitchen_client module¶

class dkutils.datakitchen_api.datakitchen_client.DataKitchenClient(username, password, base_url=None, kitchen=None, recipe=None, variation=None, is_api_token=False)[source]¶

Bases: object

Client object for invoking DataKitchen API calls. If the API call requires a kitchen, recipe, and/or variation, you must set those fields on the client instance prior to invoking those methods (or set them via the constructor when a class instance is created).

Parameters

username (str) – Username to authenticate and obtain a session token via the DataKitchen login API.
password (str) – Password to authenticate and obtain a session token via the DataKitchen login API.
base_url (str, optional) – Base DataKitchen Platform URL
kitchen (str, optional) – Kitchen to use in API requests
recipe (str, optional) – Recipe to use in API requests
variation (str, optional) – Variation to use in API requests
is_api_token (bool, optional) – Indicates whether username and possword are an api token pair

__init__(username, password, base_url=None, kitchen=None, recipe=None, variation=None, is_api_token=False)[source]¶

Client object for invoking DataKitchen API calls. If the API call requires a kitchen, recipe, and/or variation, you must set those fields on the client instance prior to invoking those methods (or set them via the constructor when a class instance is created).

Parameters

username (str) – Username to authenticate and obtain a session token via the DataKitchen login API.
password (str) – Password to authenticate and obtain a session token via the DataKitchen login API.
base_url (str, optional) – Base DataKitchen Platform URL
kitchen (str, optional) – Kitchen to use in API requests
recipe (str, optional) – Recipe to use in API requests
variation (str, optional) – Variation to use in API requests
is_api_token (bool, optional) – Indicates whether username and possword are an api token pair

add_kitchen_staff(new_kitchen_staff)[source]¶

Adds the given email addresses to the kitchen staff for the current kitchen. Email addresses that are already part of the existing kitchen staff will be ignored.

Parameters

new_kitchen_staff (list) – A list containing the email addresses to be added to the kitchen staff

Raises

HTTPError – If the request fails.
ValueError – If the kitchen attribute is not set

compare_overrides(other=None)[source]¶

Compare the overrides in the current kitchen to those of the specified kitchen. If other is None then the comparison is done against the parent kitchen

Raises

HTTPError – If the request fails.
ValueError – If the kitchen attribute is not set If the name of the specified kitchen doesn’t match any available kitchen

Returns

A DictionaryCompparator which can be used to get the results of the comparison

Return type

DictionaryComparator

create_and_monitor_orders(orders_details, sleep_secs, duration_secs, max_concurrent=None, stop_on_error=False)[source]¶

Create the specified orders and wait for them to complete (or timeout after the specified duration_secs).

Parameters

orders_details (list) –
List of order_details, each of which contains the kitchen, recipe, and variation of the order to create, as well as an optional dictionary of parameters. An example order_details item is of the form:
```
{
    "kitchen": "IM_Development",
    "recipe": "Demo_Recipe_Replay",
    "variation": "Process_Daily_Data",
    "parameters": {
        "DT": "20200529"
    }
}
```
sleep_secs (int) – Number of seconds to sleep in between loop executions.
duration_secs (int) – Max duration in seconds after which the loop will exit.
max_concurrent (integer or None) – Max number of orders to kick off concurrently. If None, all orders will be kicked off concurrently.
stop_on_error (boolean) – If True, any order run failure will prevent new order runs from being created. Otherwise, if False, all orders are submitted regardless of any failures.

Returns

List of 3 lists: [completed orders, active orders, queued orders]. If all the orders completed in the provided duration_secs, then active orders and queued orders will be empty lists. The completed orders and active orders lists contain the orders_details entries with order_id, order_run_id, and order_status fields appended to each entry. These additional fields will all be populated in the completed orders list. However, the order_run_id and order_run_status fields may be None if the timeout occurred before the order run id for the created order was obtained. The final queued orders list contains a copy of the order_details entries that never made it out of the queue, with no additional fields added. An example order_details entry for the completed orders and active orders array will be of the form:

{
    "kitchen": "IM_Development",
    "recipe": "Demo_Recipe_Replay",
    "variation": "Process_Daily_Data",
    "parameters": {
        "DT": "20200529"
    }
    "order_id": "dc90de6e-a12b-11ea-8ba5-96323db651da",
    "order_run_id": "f9b107a8-a12b-11ea-a95a-72553da658b3",
    "order_run_status": "COMPLETED_SERVING",
}

Return type

list

create_kitchen(name: str, description: str = '')[source]¶

Create a child kitchen off of self.kitchen and return a Kitchen object for the newly created kitchen.

Parameters

name (str) – New kitchen name
description (str) – New kitchen description

Return type

Kitchen

create_order(parameters={})[source]¶

Create a new order. Kitchen, recipe and variation attributes must be set prior to invoking this method.

Parameters: parameters (dict, optional) – Dictionary of variable overrides (default is empty dictionary)
Raises: HTTPError – If the request fails.
Returns: Response object
Return type: requests.Response

delete_order(order_id)[source]¶

Delete an Order. Kitchen attribute must be set prior to invoking this method.

Parameters: order_id (str) – ID of the order being deleted.
Raises: HTTPError – If the request fails.
Returns: Response object
Return type: requests.Response

delete_order_run(order_run_id)[source]¶

Delete an Order Run. Kitchen attribute must be set prior to invoking this method.

Parameters: order_run_id (str) – ID of the order run being deleted.
Raises: HTTPError – If the request fails.
Returns: Response object
Return type: requests.Response

get_kitchen()[source]¶: Create and return a kitchen object for use in making kitchen related API requests.

get_kitchen_staff()[source]¶

Returns a list containing the email addresses of the kitchen staff for the current kitchen

Raises

HTTPError – If the request fails.
ValueError – If the kitchen attribute is not set If the name in the given kitchen_info does not match that of the current kitchen

Returns

A list containing the email address of the kitchen staff

Return type

list

get_kitchens()[source]¶

Retrieve a list of all kitchen names:

Returns: List of all kitchen names.
Return type: list

get_order_run_details(order_run_id, include_logs=False, include_servingjson=False, include_summary=False, include_testresults=False, include_timingresults=False)[source]¶

Retrieve the details of an order run.

Parameters

order_run_id (str) – Order run id for which to retrieve details
include_logs (boolean) – Include logs in response (Default: False)
include_servingjson (boolean) – Include servings in response (Default: False)
include_summary (boolean) – Include summary information in response (Default: False)
include_testresults (boolean) – Include test results in response (Default: False)
include_timingresults (boolean) – Include timing results in response (Default: False)

Raises

HTTPError – If the request fails.

Returns

Order run details of the form:

{
    "order_id": "ca789c92-8bb6-11ea-883f-46ee3c6afcbf",
    "hid": "cd463d80-8bb6-11ea-97c5-8a10ccb96113",
    "recipe_id": "ce7b696e-8bb6-11ea-97c5-8a10ccb96113",
    "status": "SERVING_ERROR",
    "variation_name": "sub_workflow",
    "recipe_name": "Sub_Workflow",
    "run_time_variables": {
        "CAT": "CLS",
        "CurrentKitchen": "Add_Real_Data_and_Infrastructure",
        "CurrentOrderId": "ca789c92-8bb6-11ea-883f-46ee3c6afcbf",
        "CurrentOrderRunId": "cd463d80-8bb6-11ea-97c5-8a10ccb96113",
        "CurrentVariation": "sub_workflow",
        "DH": "12",
        "DT": "20200426",
        ...
        ...
        ...
        "tableauConfig": {
            "username": "",
            "password": "",
            "url_login": "",
            "url": "",
            "selenium_url": "",
            "content_url": ""
        }
    },
    "orderrun_status": "Error in OrderRun",
    "resumed_by": null,
    "scheduled_start_time": 1588342778543,
    "variation": {},
    "state": "unknown DKNodeStatus"
}

Return type

dict

get_order_run_status(order_run_id)[source]¶

Retrieve the status of the provided order run. If the order run isn’t found, return None.

Parameters: order_run_id (str) – Order run for which to retrieve status
Returns: One of PLANNED_SERVING, ACTIVE_SERVING, COMPLETED_SERVING, STOPPED_SERVING, SERVING_ERROR, SERVING_RERAN, or None if the order run is not found.
Return type: str or none

get_order_runs(order_id)[source]¶

Retrieve all the order runs associated with the provided order. The kitchen attribute must be set prior to invoking this method.

Parameters

order_id (str) – Order id for which to retrieve order runs

Returns

None if no order runs are found. Otherwise, return a list of order run details. Each order run details entry is of the form:

{
    "order_id": "71d8a966-38e0-11ea-8cf9-a6bbea194887",
    "status": "COMPLETED_SERVING",
    "orderrun_status": "OrderRun Completed",
    "hid": "a8978ddc-83c7-11ea-88ba-9a815c325cee",
    "variation_name": "dk_agent_checker_run_hourly",
    "timings": {
      "start-time": 1587470413845,
      "end-time": 1587470432441,
      "duration": 18596
    }
}

Return type

list or None

get_order_status(time_period_hours=None, order_id_regex=None, order_status=None, order_run_status=None, order_run_count=3)[source]¶

Retrieve the order (and associated order runs) status details based on the applied filters. To filter based on recipe and variation, ensure these properties are set on this client. Alternatively, set recipe and/or variation to None to remove filtering based on these properties.

Parameters

time_period_hours (int, optional) – Limit retrieved orders to those that started less than the provided number of hours ago.
order_id_regex (string, optional) – Filter retrieved orders based on this provided order id regular expression
order_status (string, optional) – Filter retrieved orders with this provided order status
order_run_status (string, optional) – Filter retrieved order run with this provided order run status
order_run_count (int, optional) – Limit the number of retrieved order runs (default is 3)

Raises

HTTPError – If the request fails.
ValueError – If the kitchen attribute is not set

Returns

order_runs –

A list of order runs sorted from most recent start-time to oldest in the form:

[
    {
        'hid': '66152846-e0ee-11ea-8280-12128c919b99',
        'order_id': 'a8f9ac78-e0ed-11ea-b3f4-56a20effdf97',
        'orderrun_status': 'OrderRun Completed',
        'status': 'COMPLETED_SERVING',
        'timings': {
            'duration': 22261,
            'end-time': 1597712533033,
            'start-time': 1597712510772
        }
    },
    {
        'hid': '3f57a7d0-e0ec-11ea-a65f-5edd64ee22f9',
        'order_id': '3942feda-e0ec-11ea-a00e-56a20effdf97',
        'orderrun_status': 'OrderRun Completed',
        'status': 'COMPLETED_SERVING',
        'timings': {
            'duration': 22676,
            'end-time': 1597711609459,
            'start-time': 1597711586783
        },
        'variation_name': 'variation1'
    }
]

Return type

list

get_orders(time_period_hours=None, order_id_regex=None, order_status=None, order_run_status=None, order_run_count=3)[source]¶

Retrieve a dictionary of orders and their associated order runs based on the applied filters. To filter based on recipe and variation, ensure these properties are set on this client. Alternatively, set recipe and/or variation to None to remove filtering based on these properties.

Parameters

time_period_hours (int, optional) – Limit retrieved orders to those that started less than the provided number of hours ago.
order_id_regex (string, optional) – Filter retrieved orders based on this provided order id regular expression
order_status (string, optional) – Filter retrieved orders with this provided order status
order_run_status (string, optional) – Filter retrieved order run with this provided order run status
order_run_count (int, optional) – Limit the number of retrieved order runs (default is 3)

Raises

HTTPError – If the request fails.
ValueError – If the kitchen attribute is not set

Returns

orders –

Dictionary of orders and their associated order runs in the form:

{
    "customer": "DKImplementation",
    "kitchenname": "New_Customer_Training",
    "orders": [
        {
            "hid": "d1cea220-8c0a-11eb-8d5d-5acebe1b7ce3",
            "input_settings": {
                "customer_git_name": "im",
                "customer_git_org": "DKImplementation",
                "customer_name": "Implementation",
                "email": "user@gmail.com",
                "email_verified": true,
                "graph-setting": [
                    [
                        "Train_Model"
                    ]
                ],
                "login": "user@gmail.com",
                "parameters": {
                    "edges": [],
                    "nodes": [
                        "Train_Model"
                    ]
                },
                "role": "ADMIN",
                "schedule": "now",
                "user_id": "auth0|5d3b2624888ba60cf584c9d3"
            },
            "order-status": "ORDER_ERROR",
            "recipe": "Training_Sales_Forecast",
            "schedule": "now",
            "variation": "train_walmart_sales_forecasting_model"
        }
    ],
    "repo": "im",
    "servings": {
        "d1cea220-8c0a-11eb-8d5d-5acebe1b7ce3": {
            "servings": [
                {
                    "hid": "d5ccbef2-8c0a-11eb-bfad-1a3acc706153",
                    "order_id": "d1cea220-8c0a-11eb-8d5d-5acebe1b7ce3",
                    "orderrun_status": "Error in OrderRun",
                    "status": "SERVING_ERROR",
                    "timings": {
                        "duration": 120439,
                        "end-time": 1616526493384,
                        "start-time": 1616526372945
                    },
                    "variation_name": "train_walmart_sales_forecasting_model"
                }
            ],
            "total": 1
        }
    },
    "total-orders": 1
}

Return type

dict

get_override_names_that_do_not_exist(override_names)[source]¶

Given a set of override names that could exist in the kitchen, return a set of the names of any that do not exist in the kitchen.

Parameters

override_names (set) – A set of override names that could exist in the kitchen

Raises

HTTPError – If the request fails.
ValueError – If the kitchen attribute is not set If the set of overrides is empty

Returns

The set of any of given override names to do not exist in the kitchen. If all of the override names exist then an empty set will be returned.

Return type

set

get_override_names_that_exist(override_names)[source]¶

Given a set of override names that could exist in the kitchen, return a set of the names of that exist in the kitchen.

Parameters

override_names (set) – A set of override names that could exist in the kitchen

Raises

HTTPError – If the request fails.
ValueError – If the kitchen attribute is not set If the set of overrides is empty

Returns

The set of any of given override names that exist in the kitchen. If none of the override names exist then an empty set will be returned.

Return type

set

get_overrides(override_names=None)[source]¶

Returns a dictionary containing the overrides for the current kitchen or if override_names is specified a subset of overrides

Raises

HTTPError – If the request fails.
ValueError – If the kitchen attribute is not set If the name in the given kitchen_info does not match that of the current kitchen If any of the names in override_names are not available in the current kitchen

Returns

A dictionary containing the overrides

Return type

dict

get_recipe(recipe_files=None, include_recipe_tree=False)[source]¶

Retrieve the file structure and contents of a recipe. Using the default arguments, all files and their contents will be returned. To return specific file contents, provide those file paths in the recipe_files list. To return a subset of file contents, but the entire recipe files hierarchy, set the include_recipe_tree to True.

Parameters

recipe_files (list or None) – None, to return the contents of all files, or a list of file paths (relative to the root of the recipe) to return contents of specific files.
include_recipe_tree (boolean) – When set to True and used in conjunction with a non-empty list of recipe_files, a recipe-tree key is added to the returned dictionary. This added field includes a hierarchy of all files in a recipe and allows a user to return a subset of file contents.

Raises

HTTPError – If the request fails.
ValueError – If the kitchen or recipe attribute is not set, or if recipe_files is an empty list.

Returns

Dictionary of recipe files (for this example, recipe_files=[‘description.json’] and include_recipe_tree==True) of the form:

{
    "ORIG_HEAD": "ef1d7cd1ed6e6371e9ea964b74e90f61de38fe68",
    "recipe-tree": {
        "Test": {
            "Test": [
                {
                    "filename": "description.json",
                    "json": "{    "description": "",     "recipe-emails...
                    "sha": "20ceab58e77be39ccbfcd1cfae52fc6710eb47e5",
                    "type": "blob",
                    "url": "https://github.com/api/v3/repos/DataKitchen...
                },
                {
                    "filename": "variables.json",
                    "sha": "73b22809acd0c96f981f33bf9e7936780a5df397",
                    "type": "blob",
                    "url": "https://github.com/api/v3/repos/DataKitchen...
                },
                {
                    "filename": "variations.json",
                    "sha": "ba744dc4669fb44f1200cc86770772433c73ddf8",
                    "type": "blob",
                    "url": "https://github.com/api/v3/repos/DataKitchen...
                }
            ],
            "Test/resources": [
                {
                    "filename": "README.txt",
                    "sha": "6966c84b6474c42b28a51b3546ec27a5733abffa",
                    "type": "blob",
                    "url": "https://github.com/api/v3/repos/DataKitchen...
                }
            ]
        }
    },
    "recipes": {
        "Test": {
            "Test": [
                {
                    "filename": "description.json",
                    "json": "{    "description": "",     "recipe-emails...
                    "sha": "20ceab58e77be39ccbfcd1cfae52fc6710eb47e5",
                    "type": "blob",
                    "url": "https://github.com/api/v3/repos/DataKitchen...
                }
            ]
        }
    }
}

Return type

dict

get_recipes()[source]¶

Returns a list of recipe names.

Raises

HTTPError – If the request fails.
ValueError – If the kitchen attribute is not set If the current user is not in the kitchen_staff

Returns

A list containing the recipe names. For example:

[ 'Demo_Azure', 'Demo_NiFi' ]

Return type

list

get_variations()[source]¶

Returns a dictionary that contains the variation-list from the variations.json file

Raises

HTTPError – If the request fails.
ValueError – If the recipe attribute is not set

Returns

A dictionary containing the information from the variations.json file. For example:

{ "some_variation": {
    "description": "A descript of the variation",
    "graph-setting": "a_graph_setting"
    }
}

Return type

dict

monitor_order_run(sleep_secs, duration_secs, order_run_id)[source]¶

Wait for the specified order run to complete and return completion status when finished. If the order run takes > duration_secs, return None.

Parameters

sleep_secs (int) – Number of seconds to sleep in between loop executions.
duration_secs (int) – Max duration in seconds after which the loop will exit.
order_run_id (str) – Order run for which to wait until it’s completed

Returns

One of PLANNED_SERVING, ACTIVE_SERVING, COMPLETED_SERVING, STOPPED_SERVING, SERVING_ERROR, SERVING_RERAN, or None if the order run is not found.

Return type

str or none

monitor_order_runs(sleep_secs, duration_secs, order_run_ids)[source]¶

Wait for the specified order runs to complete and return completion status when finished. If the order runs take > duration_secs, return None.

Parameters

sleep_secs (int) – Number of seconds to sleep in between loop executions.
duration_secs (int) – Max duration in seconds after which the loop will exit.
order_run_ids (dict) – Dictionary keyed by order run id and valued by kitchen for the order runs to wait for completion.

Returns

Dictionary keyed by order run id and valued by one of PLANNED_SERVING, ACTIVE_SERVING, COMPLETED_SERVING, STOPPED_SERVING, SERVING_ERROR, SERVING_RERAN, or None if the order run is not found.

Return type

dict or none

resume_and_monitor_orders(order_runs_details, sleep_secs, duration_secs, max_concurrent=None, stop_on_error=False)[source]¶

Resume the specified order runs and wait for them to complete (or timeout after the specified duration_secs).

Parameters

order_runs_details (list) –
List of order_runs_details, each of which contains the kitchen and order run id. An example order_run_details item is of the form:
```
{
    "kitchen": "IM_Development",
    "order_run_id": "84e77b50-a1f3-11ea-aaaf-521d4744de4c"
}
```
sleep_secs (int) – Number of seconds to sleep in between loop executions.
duration_secs (int) – Max duration in seconds after which the loop will exit.
max_concurrent (integer or None) – Max number of orders to kick off concurrently. If None, all orders will be kicked off concurrently.
stop_on_error (boolean) – If True, any order run failure will prevent new order runs from being created. Otherwise, if False, all orders are submitted regardless of any failures.

Returns

List of 3 lists: [completed orders, active orders, queued orders]. If all the orders completed in the provided duration_secs, then active orders and queued orders will be empty lists. The completed orders and active orders lists contain the orders_details entries with order_id, order_run_id, and order_status fields appended to each entry. These additional fields will all be populated in the completed orders list. However, the order_run_id and order_run_status fields may be None if the timeout occurred before the order run id for the created order was obtained. The final queued orders list contains a copy of the order_details entries that never made it out of the queue, with no additional fields added. An example order_details entry for the completed orders and active orders array will be of the form:

{
    "kitchen": "IM_Development",
    "order_id": "dc90de6e-a12b-11ea-8ba5-96323db651da",
    "order_run_id": "f9b107a8-a12b-11ea-a95a-72553da658b3",
    "order_run_status": "COMPLETED_SERVING",
}

Return type

list

resume_order_run(order_run_id)[source]¶

Resume a failed order run. Kitchen attribute must be set prior to invoking this method.

Parameters: order_run_id (str) – Failed order run id to resume.
Raises: HTTPError – If the request fails.
Returns: Response object
Return type: requests.Response

set_kitchen(kitchen)[source]¶

set_recipe(recipe)[source]¶

set_variation(variation)[source]¶

update_kitchen_staff(kitchen_staff)[source]¶

Updates the kitchen staff for the current kitchen

Parameters

list – A list containing the email addresses of the kitchen staff

Raises

HTTPError – If the request fails.
ValueError – If the kitchen attribute is not set If the current user is not in the kitchen_staff

update_kitchen_vault(prefix, vault_token, vault_url='https://vault2.datakitchen.io:8200', private=False, inheritable=True)[source]¶

Updates the custom vault configuration for a Kitchen.

Parameters

prefix (str) – Prefix specifying the vault location (e.g. Implementation/dev).
vault_token (str) – Token for the custom vault.
vault_url (str, optional) – Vault URL (default: https://vault2.datakitchen.io:8200).
private (str, optional) – Set to True if this is a private vault service, otherwise set to False (default: False)
inheritable (str, optional) – Set to True if this vault should be inherited by child kitchens, otherwise set to false (default: True).

Raises

HTTPError – If the request fails.

Returns

Response object

Return type

requests.Response

update_overrides(overrides)[source]¶

Updates the overrides for the current kitchen

Parameters

dict – A dictionary containing the overrides

Raises

HTTPError – If the request fails.
ValueError – If the kitchen attribute is not set If the name in the given kitchen_info does not match that of the current kitchen

property kitchen¶

property recipe¶

property variation¶

dkutils.datakitchen_api.datakitchen_client.create_using_context(context='default', kitchen=None, recipe=None, variation=None)[source]¶

This is a factory method that can be used to create a client using the context created by DKCloudCommand

Parameters

context (str, optional) – The name of a context created by DKCloudCommand
kitchen (str, optional) – Kitchen to use in API requests
recipe (str, optional) – Recipe to use in API requests
variation (str, optional) – Variation to use in API requests

Returns

Client object for invoking DataKitchen API calls

Return type

DataKitchenClient

dkutils.datakitchen_api.datakitchen_client.ensure_and_get_kitchen(kitchen, kitchens)[source]¶

Ensure the provided kitchens dictionary contains the provided kitchen and if so, return its kitchen info. Otherwise, raise an exception.

Parameters

kitchen (str) – Name of the kitchen
kitchens (dict) – Dictionary keyed by kitchen name and valued by a dictionary of kitchen info

Raises

ValueError – If provided kitchen name doesn’t exist in the provided kitchens dictionary

Returns

Kitchen info

Return type

dict