Certain fault categories have unique requirements, from a permissions and set up (or configuration) perspective.
HCE recommends you create a dedicated pod security policy (PSP) or equivalent that you can map to the transient chaos pods or service account.
Internally, the load fault leverages Locust (support for other tools is a part of the roadmap). The internal load engine uses a Python script to define the API calls that should be part of the load profile. The script is embedded within a ConfigMap that is referenced by the chaos pods during execution. Go to locust prerequisites for detailed steps
Create an IAM role on the cloud account that is mapped to the appropriate policy.
Your (cloud account user) credentials must be embedded with a Kubernetes secret before executing the fault.
You can create a superset AWS policy that allows executing all the fault types supported by HCE.
This table lists the permissions required to execute Kubernetes fault (node-level and pod-level).
| Pod-level faults | Mode (Scopes of chaos agent) | Permissions required | 
|---|
| Pod delete * | Namespaced, Cluster | pod = [create, delete, get, list, patch, deletecollection, update]events = [create, get, list, patch, update]configMaps = NApods/log = [get, list, watch]pods/exec = NAdeployments, statefulsets = [get, list]replicasets, daemonsets = [get, list]networkpolicies = NAjobs = [create, delete, get, list, deletecollection]chaosEngines, chaosExperiments, chaosResults = [create, delete, get, list, patch, update]secrets = NApod eviction = NAnodes = NA*
 | 
| Container kill | Namespaced, Cluster | pod = [create, delete, get, list, patch, deletecollection, update]events = [create, get, list, patch, update]configMaps = NApods/log = [get, list, watch]pods/exec = NAdeployments, statefulsets = [get, list]replicasets, daemonsets = [get, list]networkpolicies = NAjobs = [create, delete, get, list, deletecollection]chaosEngines, chaosExperiments, chaosResults = [create, delete, get, list, patch, update]secrets = NApod eviction = NAnodes = NA*
 | 
| Disk fill | Namespaced, Cluster | pod = [create, delete, get, list, patch, deletecollection, update]events = [create, get, list, patch, update]configMaps = NApods/log = [get, list, watch]pods/exec = NAdeployments, statefulsets = [get, list]replicasets, daemonsets = [get, list]networkpolicies = NAjobs = [create, delete, get, list, deletecollection]chaosEngines, chaosExperiments, chaosResults = [create, delete, get, list, patch, update]secrets = NApod eviction = NAnodes = NA*
 | 
| Pod API block | Namespaced, Cluster | pod = [create, delete, get, list, patch, deletecollection, update]events = [create, get, list, patch, update]configMaps = NApods/log = [get, list, watch]pods/exec = NAdeployments, statefulsets = [get, list]replicasets, daemonsets = [get, list]networkpolicies = NAjobs = [create, delete, get, list, deletecollection]chaosEngines, chaosExperiments, chaosResults = [create, delete, get, list, patch, update]secrets = NApod eviction = NAnodes = NA*
 | 
| Pod API latency | Namespaced, Cluster | pod = [create, delete, get, list, patch, deletecollection, update]events = [create, get, list, patch, update]configMaps = NApods/log = [get, list, watch]pods/exec = NAdeployments, statefulsets = [get, list]replicasets, daemonsets = [get, list]networkpolicies = NAjobs = [create, delete, get, list, deletecollection]chaosEngines, chaosExperiments, chaosResults = [create, delete, get, list, patch, update]secrets = NApod eviction = NAnodes = NA*
 | 
| Pod API modify body | Namespaced, Cluster | pod = [create, delete, get, list, patch, deletecollection, update]events = [create, get, list, patch, update]configMaps = NApods/log = [get, list, watch]pods/exec = NAdeployments, statefulsets = [get, list]replicasets, daemonsets = [get, list]networkpolicies = NAjobs = [create, delete, get, list, deletecollection]chaosEngines, chaosExperiments, chaosResults = [create, delete, get, list, patch, update]secrets = NApod eviction = NAnodes = NA*
 | 
| Pod API modify header | Namespaced, Cluster | pod = [create, delete, get, list, patch, deletecollection, update]events = [create, get, list, patch, update]configMaps = NApods/log = [get, list, watch]pods/exec = NAdeployments, statefulsets = [get, list]replicasets, daemonsets = [get, list] networkpolicies = NAjobs = [create, delete, get, list, deletecollection]chaosEngines, chaosExperiments, chaosResults = [create, delete, get, list, patch, update]secrets = NApod eviction = NAnodes = NA*
 | 
| Pod API status code | Namespaced, Cluster | pod = [create, delete, get, list, patch, deletecollection, update]events = [create, get, list, patch, update]configMaps = NApods/log = [get, list, watch]pods/exec = NAdeployments, statefulsets = [get, list]replicasets, daemonsets = [get, list]networkpolicies = NAjobs = [create, delete, get, list, deletecollection]chaosEngines, chaosExperiments, chaosResults = [create, delete, get, list, patch, update]secrets = NApod eviction = NAnodes = NA*
 | 
| Pod autoscaler * | Namespaced, Cluster | pod = [create, delete, get, list, patch, deletecollection, update]events = [create, get, list, patch, update]configMaps = NApods/log = [get, list, watch]pods/exec = NAdeployments, statefulsets = [get, list, patch, update]replicasets, daemonsets = NAnetworkpolicies = NAjobs = [create, delete, get, list, deletecollection]chaosEngines, chaosExperiments, chaosResults = [create, delete, get, list, patch, update]secrets = NApod eviction = NAnodes = NA*
 | 
| Pod CPU hog exec * | Namespaced, Cluster | pod = [create, delete, get, list, patch, deletecollection, update]events = [create, get, list, patch, update]configMaps = NApods/log = [get, list, watch]pods/exec = [get, list, create]deployments, statefulsets = [get, list]replicasets, daemonsets = [get, list]networkpolicies = NAjobs = [create, delete, get, list, deletecollection]chaosEngines, chaosExperiments, chaosResults = [create, delete, get, list, patch, update]secrets = NApod eviction = NAnodes = NA*
 | 
| Pod CPU hog | Namespaced, Cluster | pod = [create, delete, get, list, patch, deletecollection, update]events = [create, get, list, patch, update]configMaps = NApods/log = [get, list, watch]pods/exec = NAdeployments, statefulsets = [get, list]replicasets, daemonsets = [get, list]networkpolicies = NAjobs = [create, delete, get, list, deletecollection]chaosEngines, chaosExperiments, chaosResults = [create, delete, get, list, patch, update]secrets = NApod eviction = NAnodes = NA*
 | 
| Pod DNS error | Namespaced, Cluster | pod = [create, delete, get, list, patch, deletecollection, update]events = [create, get, list, patch, update]configMaps = NApods/log = [get, list, watch]pods/exec = NAdeployments, statefulsets = [get, list]replicasets, daemonsets = [get, list]networkpolicies = NAjobs = [create, delete, get, list, deletecollection]chaosEngines, chaosExperiments, chaosResults = [create, delete, get, list, patch, update]secrets = NApod eviction = NAnodes = NA*
 | 
| Pod DNS spoof | Namespaced, Cluster | pod = [create, delete, get, list, patch, deletecollection, update]events = [create, get, list, patch, update]configMaps = NApods/log = [get, list, watch]pods/exec = NAdeployments, statefulsets = [get, list]replicasets, daemonsets = [get, list]networkpolicies = NAjobs = [create, delete, get, list, deletecollection]chaosEngines, chaosExperiments, chaosResults = [create, delete, get, list, patch, update]secrets = NApod eviction = NAnodes = NA*
 | 
| Pod HTTP latency | Namespaced, Cluster | pod = [create, delete, get, list, patch, deletecollection, update]events = [create, get, list, patch, update]configMaps = NApods/log = [get, list, watch]pods/exec = NAdeployments, statefulsets = [get, list]replicasets, daemonsets = [get, list]networkpolicies = NAjobs = [create, delete, get, list, deletecollection]chaosEngines, chaosExperiments, chaosResults = [create, delete, get, list, patch, update]secrets = NApod eviction = NAnodes = NA*
 | 
| Pod HTTP modify body | Namespaced, Cluster | pod = [create, delete, get, list, patch, deletecollection, update]events = [create, get, list, patch, update]configMaps = NApods/log = [get, list, watch]pods/exec = NAdeployments, statefulsets = [get, list]replicasets, daemonsets = [get, list]networkpolicies = NAjobs = [create, delete, get, list, deletecollection]chaosEngines, chaosExperiments, chaosResults = [create, delete, get, list, patch, update]secrets = NApod eviction = NAnodes = NA*
 | 
| Pod HTTP modify header | Namespaced, Cluster | pod = [create, delete, get, list, patch, deletecollection, update]events = [create, get, list, patch, update]configMaps = NApods/log = [get, list, watch]pods/exec = NAdeployments, statefulsets = [get, list]replicasets, daemonsets = [get, list]networkpolicies = NAjobs = [create, delete, get, list, deletecollection]chaosEngines, chaosExperiments, chaosResults = [create, delete, get, list, patch, update]secrets = NApod eviction = NAnodes = NA*
 | 
| Pod HTTP reset peer | Namespaced, Cluster | pod = [create, delete, get, list, patch, deletecollection, update]events = [create, get, list, patch, update]configMaps = NApods/log = [get, list, watch]pods/exec = NAdeployments, statefulsets = [get, list]replicasets, daemonsets = [get, list]networkpolicies = NAjobs = [create, delete, get, list, deletecollection]chaosEngines, chaosExperiments, chaosResults = [create, delete, get, list, patch, update]secrets = NApod eviction = NAnodes = NA*
 | 
| Pod HTTP status code | Namespaced, Cluster | pod = [create, delete, get, list, patch, deletecollection, update]events = [create, get, list, patch, update]configMaps = NApods/log = [get, list, watch]pods/exec = NAdeployments, statefulsets = [get, list]replicasets, daemonsets = [get, list]networkpolicies = NAjobs = [create, delete, get, list, deletecollection]chaosEngines, chaosExperiments, chaosResults = [create, delete, get, list, patch, update]secrets = NApod eviction = NAnodes = NA*
 | 
| Pod IO attribute override | Namespaced, Cluster | pod = [create, delete, get, list, patch, deletecollection, update]events = [create, get, list, patch, update]configMaps = NApods/log = [get, list, watch]pods/exec = NAdeployments, statefulsets = [get, list]replicasets, daemonsets = [get, list]networkpolicies = NAjobs = [create, delete, get, list, deletecollection]chaosEngines, chaosExperiments, chaosResults = [create, delete, get, list, patch, update]secrets = NApod eviction = NAnodes = NA*
 | 
| Pod IO error | Namespaced, Cluster | pod = [create, delete, get, list, patch, deletecollection, update]events = [create, get, list, patch, update]configMaps = NApods/log = [get, list, watch]pods/exec = NAdeployments, statefulsets = [get, list]replicasets, daemonsets = [get, list]networkpolicies = NAjobs = [create, delete, get, list, deletecollection]chaosEngines, chaosExperiments, chaosResults = [create, delete, get, list, patch, update]secrets = NApod eviction = NAnodes = NA*
 | 
| Pod IO latency | Namespaced, Cluster | pod = [create, delete, get, list, patch, deletecollection, update]events = [create, get, list, patch, update]configMaps = NApods/log = [get, list, watch]pods/exec = NAdeployments, statefulsets = [get, list]replicasets, daemonsets = [get, list]networkpolicies = NAjobs = [create, delete, get, list, deletecollection]chaosEngines, chaosExperiments, chaosResults = [create, delete, get, list, patch, update]secrets = NApod eviction = NAnodes = NA*
 | 
| Pod IO stress | Namespaced, Cluster | pod = [create, delete, get, list, patch, deletecollection, update]events = [create, get, list, patch, update]configMaps = NApods/log = [get, list, watch]pods/exec = NAdeployments, statefulsets = [get, list]replicasets, daemonsets = [get, list]networkpolicies = NAjobs = [create, delete, get, list, deletecollection]chaosEngines, chaosExperiments, chaosResults = [create, delete, get, list, patch, update]secrets = NApod eviction = NAnodes = NA*
 | 
| Pod memory hog exec * | Namespaced, Cluster | pod = [create, delete, get, list, patch, deletecollection, update]events = [create, get, list, patch, update]configMaps = NApods/log = [get, list, watch]pods/exec = [get, list, create]deployments, statefulsets = [get, list]replicasets, daemonsets = [get, list]networkpolicies = NAjobs = [create, delete, get, list, deletecollection]chaosEngines, chaosExperiments, chaosResults = [create, delete, get, list, patch, update]secrets = NApod eviction = NAnodes = NA*
 | 
| Pod memory hog | Namespaced, Cluster | pod = [create, delete, get, list, patch, deletecollection, update]events = [create, get, list, patch, update]configMaps = NApods/log = [get, list, watch]pods/exec = NAdeployments, statefulsets = [get, list]replicasets, daemonsets = [get, list]networkpolicies = NAjobs = [create, delete, get, list, deletecollection]chaosEngines, chaosExperiments, chaosResults = [create, delete, get, list, patch, update]secrets = NApod eviction = NAnodes = NA*
 | 
| Pod network corruption | Namespaced, Cluster | pod = [create, delete, get, list, patch, deletecollection, update]events = [create, get, list, patch, update]configMaps = NApods/log = [get, list, watch]pods/exec = NAdeployments, statefulsets = [get, list]replicasets, daemonsets = [get, list]networkpolicies = NAjobs = [create, delete, get, list, deletecollection]chaosEngines, chaosExperiments, chaosResults = [create, delete, get, list, patch, update]secrets = NApod eviction = NAnodes = NA*
 | 
| Pod network duplication | Namespaced, Cluster | pod = [create, delete, get, list, patch, deletecollection, update]events = [create, get, list, patch, update]configMaps = NApods/log = [get, list, watch]pods/exec = NAdeployments, statefulsets = [get, list]replicasets, daemonsets = [get, list]networkpolicies = NAjobs = [create, delete, get, list, deletecollection]chaosEngines, chaosExperiments, chaosResults = [create, delete, get, list, patch, update]secrets = NApod eviction = NAnodes = NA*
 | 
| Pod network latency | Namespaced, Cluster | pod = [create, delete, get, list, patch, deletecollection, update]events = [create, get, list, patch, update]configMaps = NApods/log = [get, list, watch]pods/exec = NAdeployments, statefulsets = [get, list]replicasets, daemonsets = [get, list]networkpolicies = NAjobs = [create, delete, get, list, deletecollection]chaosEngines, chaosExperiments, chaosResults = [create, delete, get, list, patch, update]secrets = NApod eviction = NAnodes = NA*
 | 
| Pod network loss | Namespaced, Cluster | pod = [create, delete, get, list, patch, deletecollection, update]events = [create, get, list, patch, update]configMaps = NApods/log = [get, list, watch]pods/exec = NAdeployments, statefulsets = [get, list]replicasets, daemonsets = [get, list]networkpolicies = NAjobs = [create, delete, get, list, deletecollection]chaosEngines, chaosExperiments, chaosResults = [create, delete, get, list, patch, update]secrets = NApod eviction = NAnodes = NA*
 | 
| Pod network partition * | Namespaced, Cluster | pod = [create, delete, get, list, patch, deletecollection, update]events = [create, get, list, patch, update]configMaps = NApods/log = [get, list, watch]pods/exec = NAdeployments, statefulsets = [get, list]replicasets, daemonsets = [get, list]networkpolicies = [create, delete, get, list]jobs = [create, delete, get, list, deletecollection]chaosEngines, chaosExperiments, chaosResults = [create, delete, get, list, patch, update]secrets = NApod eviction = NAnodes = NA*
 | 
| Pod network rate limit | Namespaced, Cluster | pod = [create, delete, get, list, patch, deletecollection, update]events = [create, get, list, patch, update]configMaps = NApods/log = [get, list, watch]pods/exec = NAdeployments, statefulsets = [get, list]replicasets, daemonsets = [get, list]networkpolicies = NAjobs = [create, delete, get, list, deletecollection]chaosEngines, chaosExperiments, chaosResults = [create, delete, get, list, patch, update]secrets = NApod eviction = NAnodes = NA*
 | 
| Time chaos | Namespaced, Cluster | pod = [create, delete, get, list, patch, deletecollection, update]events = [create, get, list, patch, update]configMaps = NApods/log = [get, list, watch]pods/exec = NAdeployments, statefulsets = [get, list]replicasets, daemonsets = [get, list]networkpolicies = NAjobs = [create, delete, get, list, deletecollection]chaosEngines, chaosExperiments, chaosResults = [create, delete, get, list, patch, update]secrets = NApod eviction = NAnodes = NA*
 | 
| Node-level faults | Mode (Scopes of chaos agent) | Permissions required | 
|---|
| Kubelet service kill | Namespaced, Cluster |  pod = [create, delete, get, list, patch, deletecollection, update]events = [create, get, list, patch, update]configMaps = NApods/log = [get, list, watch]pods/exec = [get, list, create]deployments, statefulsets = NAreplicasets, daemonsets = NAnetworkpolicies = NAjobs = [create, delete, get, list, deletecollection]chaosEngines, chaosExperiments, chaosResults = [create, delete, get, list, patch, update]secrets = NApod eviction = NAnodes = [get, list]
 | 
| Node CPU hog | Cluster | pod = [create, delete, get, list, patch, deletecollection, update]events = [create, get, list, patch, update]configMaps = NApods/log = [get, list, watch]pods/exec = [get, list, create]deployments, statefulsets = NAreplicasets, daemonsets = NAnetworkpolicies = NAjobs = [create, delete, get, list, deletecollection]chaosEngines, chaosExperiments, chaosResults = [create, delete, get, list, patch, update]secrets = NApod eviction = NAnodes = [get, list]
 | 
| Node drain | Cluster | pod = [create, delete, get, list, patch, deletecollection, update]events = [create, get, list, patch, update]configMaps = NApods/log = [get, list, watch]pods/exec = [get, list, create]deployments, statefulsets = NAreplicasets, daemonsets = NAnetworkpolicies = NAjobs = [create, delete, get, list, deletecollection]chaosEngines, chaosExperiments, chaosResults = [create, delete, get, list, patch, update]secrets = NApod eviction = [get, list, create]nodes = [get, list, patch, update]
 | 
| Node IO stress | Cluster | pod = [create, delete, get, list, patch, deletecollection, update]events = [create, get, list, patch, update]configMaps = NApods/log = [get, list, watch]pods/exec = [get, list, create]deployments, statefulsets = NAreplicasets, daemonsets = NAnetworkpolicies = NAjobs = [create, delete, get, list, deletecollection]chaosEngines, chaosExperiments, chaosResults = [create, delete, get, list, patch, update]secrets = NApod eviction = NAnodes = [get, list]
 | 
| Node memory hog | Cluster | pod = [create, delete, get, list, patch, deletecollection, update]events = [create, get, list, patch, update]configMaps = NApods/log = [get, list, watch]pods/exec = [get, list, create]deployments, statefulsets = NAreplicasets, daemonsets = NAnetworkpolicies = NAjobs = [create, delete, get, list, deletecollection]chaosEngines, chaosExperiments, chaosResults = [create, delete, get, list, patch, update]secrets = NApod eviction = NAnodes = [get, list]
 | 
| Node network latency | Cluster | pod = [create, delete, get, list, patch, deletecollection, update]events = [create, get, list, patch, update]configMaps = NApods/log = [get, list, watch]pods/exec = [get, list, create]deployments, statefulsets = NAreplicasets, daemonsets = NAnetworkpolicies = NAjobs = create, delete, get, list, deletecollectionchaosEngines, chaosExperiments, chaosResults = create, delete, get, list, patch, updatesecrets = NApod eviction = NAnodes = [get, list]
 | 
| Node network loss | Cluster | pod = [create, delete, get, list, patch, deletecollection, update]events = [create, get, list, patch, update]configMaps = NApods/log = [get, list, watch]pods/exec = [get, list, create]deployments, statefulsets = NAreplicasets, daemonsets = NAnetworkpolicies = NAjobs = [create, delete, get, list, deletecollection]chaosEngines, chaosExperiments, chaosResults = [create, delete, get, list, patch, update]secrets = NApod eviction = NAnodes = [get, list]
 | 
| Node restart | Cluster | pod = [create, delete, get, list, patch, deletecollection, update]events = [create, get, list, patch, update] configMaps = NApods/log = [get, list, watch]pods/exec = [get, list, create]deployments, statefulsets = NAreplicasets, daemonsets = NAnetworkpolicies = NAjobs = [create, delete, get, list, deletecollection]chaosEngines, chaosExperiments, chaosResults = [create, delete, get, list, patch, update]secrets = [get, list]pod eviction = NAnodes = [get, list]
 | 
| Node taint | Cluster | pod = [create, delete, get, list, patch, deletecollection, update]events = [create, get, list, patch, update]configMaps = NApods/log = [get, list, watch]pods/exec = [get, list, create]deployments, statefulsets = NAreplicasets, daemonsets = NAnetworkpolicies = NAjobs = [create, delete, get, list, deletecollection]chaosEngines, chaosExperiments, chaosResults = [create, delete, get, list, patch, update]secrets = NApod eviction = [get, list, create]nodes = [get, list, patch, update]
 | 
| Kubelet density | Cluster | pod = [create, delete, get, list, patch, deletecollection, update]events = [create, get, list, patch, update]configMaps = [get, list]pods/log = [get, list, watch]pods/exec = NAdeployments, statefulsets = NAreplicasets, daemonsets = NAnetworkpolicies = NAjobs = [create, delete, get, list, deletecollection]chaosEngines, chaosExperiments, chaosResults = [create, delete, get, list, patch, update] secrets = [get, list]pod eviction = NAnodes = [get, list]
 | 
| Spring boot | Mode (Scopes of chaos agent) | Permissions required | 
|---|
| Spring boot app kill | Namespaced, Cluster | pod = [create, delete, get, list, patch, deletecollection, update]  events = [create, get, list, patch, update]  configMaps = NA  pods/log = [get, list, watch]  pods/exec = NA  deployments, statefulsets = [get, list]  replicasets, daemonsets = [get, list]  networkpolicies = NA  jobs = [create, delete, get, list, deletecollection]  chaosEngines, chaosExperiments, chaosResults = [create, delete, get, list, patch, update]  secrets = NA  pod eviction = NA  nodes = NA* 
 | 
| Spring boot CPU stress | Namespaced, Cluster | pod = [create, delete, get, list, patch, deletecollection, update]  events = [create, get, list, patch, update]  configMaps = NA  pods/log = [get, list, watch]  pods/exec = NA  deployments, statefulsets = [get, list]  replicasets, daemonsets = [get, list]  networkpolicies = NA  jobs = [create, delete, get, list, deletecollection]  chaosEngines, chaosExperiments, chaosResults = [create, delete, get, list, patch, update]  secrets = NA  pod eviction = NA  nodes = NA* 
 | 
| Spring boot memory stress | Namespaced, Cluster | pod = [create, delete, get, list, patch, deletecollection, update]  events = [create, get, list, patch, update]  configMaps = NA  pods/log = [get, list, watch]  pods/exec = NA  deployments, statefulsets = [get, list]  replicasets, daemonsets = [get, list]  networkpolicies = NA  jobs = [create, delete, get, list, deletecollection]  chaosEngines, chaosExperiments, chaosResults = [create, delete, get, list, patch, update]  secrets = NA  pod eviction = NA  nodes = NA* 
 | 
| Spring boot latency | Namespaced, Cluster | pod = [create, delete, get, list, patch, deletecollection, update]  events = [create, get, list, patch, update]  configMaps = NA  pods/log = [get, list, watch]  pods/exec = NA  deployments, statefulsets = [get, list]  replicasets, daemonsets = [get, list]  networkpolicies = NA  jobs = [create, delete, get, list, deletecollection]  chaosEngines, chaosExperiments, chaosResults = [create, delete, get, list, patch, update]  secrets = NA  pod eviction = NA  nodes = NA* 
 | 
| Spring boot exception | Namespaced, Cluster | pod = [create, delete, get, list, patch, deletecollection, update]  events = [create, get, list, patch, update]  configMaps = NA  pods/log = [get, list, watch]  pods/exec = NA  deployments, statefulsets = [get, list]  replicasets, daemonsets = [get, list]  networkpolicies = NA  jobs = [create, delete, get, list, deletecollection]  chaosEngines, chaosExperiments, chaosResults = [create, delete, get, list, patch, update]  secrets = NA  pod eviction = NA  nodes = NA* 
 | 
| Spring boot fault | Namespaced, Cluster | pod = [create, delete, get, list, patch, deletecollection, update]  events = [create, get, list, patch, update]  configMaps = NA  pods/log = [get, list, watch]  pods/exec = NA  deployments, statefulsets = [get, list]  replicasets, daemonsets = [get, list]  networkpolicies = NA  jobs = [create, delete, get, list, deletecollection]  chaosEngines, chaosExperiments, chaosResults = [create, delete, get, list, patch, update]  secrets = NA  pod eviction = NA  nodes = NA* 
 |