SaltStack Interview Questions

Core Concepts & Architecture

1. What is SaltStack?

• Answer: SaltStack (often called Salt) is a powerful, Python-based open-source configuration management and remote execution tool. It allows system administrators to automate the management and configuration of servers (or ‘minions’) at scale from a central point (the ‘master’). It uses a Master/Minion architecture and communicates via a fast message bus (typically ZeroMQ).

2. Describe the basic SaltStack Architecture.

• Answer: The most common architecture involves:
• Salt Master: The central control server that sends commands and configurations to minions.
• Salt Minions: Servers managed by the Salt Master. They run the salt-minion service, listen for commands from the master, execute them, and return the results.
• Message Bus (ZeroMQ): The high-speed communication layer facilitating interaction between the master and minions. Minions connect to the master’s publisher (port 4505) and request server (port 4506).

3. What is the role of the Salt Master?

• Answer: The Salt Master acts as the central command and control hub. It authenticates minions, sends commands for remote execution, distributes configuration states and pillar data, and collects results from minions.

4. What is the role of the Salt Minion?

• Answer: The Salt Minion is an agent running on the managed servers. It authenticates with the Salt Master, listens for jobs published by the master, executes those jobs (like running commands or applying state configurations), and returns the job results or status back to the master.

5. How do Salt Masters and Minions communicate?

• Answer: They primarily communicate over ZeroMQ (ØMQ), a high-performance asynchronous messaging library. The Master listens on two TCP ports:
• Publisher Port (Default 4505): Used by the Master to publish jobs to minions.
• Request Server Port (Default 4506): Used by minions to send results back to the master and for file server/pillar requests.

Communication is encrypted using public/private key pairs (AES encryption).

6. Explain the Minion Key Authentication process.

• Answer: When a minion first starts, it generates cryptographic keys and sends its public key to the master. The master stores this key in a pending state (/etc/salt/pki/master/minions_pre/). An administrator must explicitly accept the minion’s key using salt-key -a <minion_id> (or similar commands) on the master. Once accepted, the key is moved to the accepted directory (/etc/salt/pki/master/minions/), and secure communication can begin.

7. What is Salt SSH? How does it differ from the standard Master/Minion setup?

• Answer: Salt SSH allows executing Salt commands and states over SSH without requiring a Salt Minion agent on the target machine. It’s useful for managing systems where installing an agent is not desirable or possible. Unlike the persistent ZeroMQ connection in the Master/Minion model, Salt SSH establishes an SSH connection for each command execution, which can be slower but is agentless.

8. Can you have a Masterless Minion setup? How?

• Answer: Yes. A minion can be run in ‘masterless’ mode. In this setup, the minion does not connect to a Salt Master. Instead, it fetches its configurations (states, pillars, etc.) from local file paths or git repositories and applies them locally using the salt-call --local command. This is useful for standalone machines or bootstrapping environments.

9. What is a Salt Syndic? When would you use it?

• Answer: A Salt Syndic is a special type of minion that also acts as a master to minions below it in a hierarchy. It relays commands from a top-level master (“Master of Masters”) down to the minions connected to it. You would use a Syndic architecture to manage very large infrastructures, segment networks, or distribute the load across multiple masters while maintaining centralized control.

10. What are Salt Runners?

• Answer: Salt Runners are modules executed directly on the Salt Master, primarily for orchestrating complex tasks, managing the Salt environment itself (like key management via manage), interacting with external APIs, or controlling cloud resources. They are executed using the salt-run command (e.g., salt-run manage.status).

Execution Modules & Commands (Questions 11-18)

11. What are Salt Execution Modules?

• Answer: Execution modules are the core components that perform the actual work on minions (or the master via salt-run or salt-call --local). They contain functions that can be called remotely. Examples include pkg (for package management), service (for service management), cmd (for running shell commands), file (for file operations), etc.

12. What is the salt command used for? Give an example.

• • Answer: The salt command is the primary tool used on the Salt Master to execute commands on targeted minions.
  • Example: salt '*' test.ping – This pings all minions whose keys have been accepted by the master (* targets all) using the test.ping execution module function.

13. What is the salt-key command used for? Give examples.

Answer: salt-key is used on the Salt Master to manage the public keys of minions, which is essential for authentication.

• Examples:
  • • salt-key -L: List all keys (accepted, pending, rejected).
    • salt-key -a <minion_id>: Accept a specific pending minion key.
    • salt-key -d <minion_id>: Delete a specific minion key.
    • salt-key -A: Accept all pending keys (use with caution).

14. What is the salt-call command used for?

• Answer: salt-call is used directly on a Salt Minion to execute execution modules locally. It’s useful for testing modules, running commands in masterless mode (--local), or troubleshooting a specific minion without involving the master.
• Example: salt-call pkg.install cowsay (runs on the minion itself).
• Example (Masterless): salt-call --local state.apply my_state

15. What is the salt-run command used for?

• • Answer: salt-run is used on the Salt Master to execute Salt Runner modules. Runners perform tasks on the master itself, often for orchestration, master-side management, or interacting with external systems.
  • Example: salt-run jobs.list_jobs

16. How can you check the status of connected minions?

• • Answer: Use the manage.status runner: salt-run manage.status. This lists minions currently connected (up) and those that are known but not currently connected (down). You can also use salt '*' test.ping.

17. How do you run a raw shell command on minions using Salt?

• • Answer: Use the cmd.run execution module function.
  • Example: salt 'web*' cmd.run 'ls -l /var/www'

18. What is the difference between cmd.run and cmd.script?

Answer:

• cmd.run: Executes a single shell command passed as a string.
• cmd.script: Downloads a script from the Salt Master’s file server (or another source like http/ftp) to the minion and then executes it. Useful for running more complex sequences of commands.

State Management

19. What are Salt States (SLS files)?

• Answer: Salt States define the desired configuration or ‘state’ of a system. They are typically written in YAML (with Jinja templating) using SLS (SaLt State) format. States declare what the system should look like (e.g., a package installed, a service running, a file present with specific content), and Salt figures out how to achieve that state idempotently.

20. What does ‘Idempotency’ mean in the context of Salt States?

• • Answer: Idempotency means that applying a state multiple times will result in the same final system state without causing unintended side effects after the initial application. If a package is already installed, a state ensuring its installation will simply report success without trying to reinstall it. Salt’s state modules are designed to be idempotent.

21. What is the top.sls file?

• • Answer: The top.sls file, located in the root of the state tree (/srv/salt/ by default), maps which states should be applied to which minions based on targeting criteria (minion ID, grains, etc.). It defines the high-level assignment of configurations across the infrastructure.

22. Explain the structure of a basic SLS file.

Answer: An SLS file typically consists of:

• ID Declaration: A unique ID for the state block (often related to the component being managed, e.g., apache-package).
• State Module: The Salt state module to use (e.g., pkg, service, file).
• Function: The specific function within the state module to call (e.g., installed, running, managed).
• Arguments: Parameters for the function (e.g., name: httpd, source: salt://files/httpd.conf).

Example:

install_apache:    # ID Declaration
  pkg:             # State Module
    - installed    # Function
    - name: httpd # Argument

23. How do you apply states to minions?

Answer: Use the state.apply or state.sls execution modules from the master.

• salt '*' state.apply: Applies the states defined for each minion in the top.sls file.
• salt 'db*' state.sls mysql: Applies the mysql.sls state (or mysql/init.sls) specifically to minions matching the db* target.

24. What are requisites in Salt States? Give examples.

Answer: Requisites define dependencies between state declarations, ensuring they are executed in the correct order. Common requisites include:

• require: The most common; ensures the required state is applied successfully before the current state.
• watch: Similar to require, but also triggers a re-run (refresh) of the watching state if the watched state changes. Useful for restarting a service when its configuration file changes.
• prereq: Checks if the prerequisite state would change; if so, the current state runs. If not, the current state is skipped.
• onchanges: Only runs the state if the dependency state(s) change.

Example (watch):

/etc/httpd/conf/httpd.conf:
  file.managed:
    - source: salt://apache/httpd.conf

apache_service:
  service.running:
    - name: httpd
    - enable: True
    - watch:
      - file: /etc/httpd/conf/httpd.conf # Restarts httpd if httpd.conf changes

25. What is Highstate?

Answer: Highstate refers to the complete collection of states defined for a minion in the top.sls file. Running salt <target> state.apply (or state.highstate) instructs the targeted minions to retrieve their assigned SLS files from the master (based on top.sls) and apply all those states to bring themselves into compliance.

26. How can you test an SLS file before applying it?

• Answer: Use the test=True argument. This performs a “dry run” of the state application, showing what changes would be made without actually making them.
• Example: salt 'minion1' state.apply my_state test=True

27. What is the Salt File Server?

• Answer: The Salt File Server is a component on the Salt Master that serves files (like SLS files, templates, configuration files) to minions. Minions access these files using the salt:// URL prefix. It primarily uses the directories specified in the file_roots setting in the master configuration (defaulting to /srv/salt/).

Pillars

28. What are Salt Pillars?

• Answer: Pillars are tree-like structures of data defined on the Salt Master and securely transmitted only to specific, targeted minions. They are primarily used for sensitive data (like passwords, API keys) and minion-specific configuration parameters (like user names, database settings) that shouldn’t be globally accessible like Grains.

29. How do Pillars differ from Grains?

Answer:

• Grains: Data generated on the minion about the minion itself (OS, kernel, IP address, CPU). Available to the master for targeting. Relatively static.
• Pillars: Data defined on the master and assigned to specific minions. Used for sensitive or targeted configuration data. Securely transmitted. Can be used in states and templates.

30. Where is Pillar data typically stored?

• Answer: Similar to states, pillar data is usually defined in SLS files (YAML+Jinja) within the pillar_roots directory on the Salt Master (defaulting to /srv/pillar/).

31. What is the Pillar top.sls file?

• Answer: Analogous to the state top.sls, the pillar top.sls file (/srv/pillar/top.sls by default) maps which pillar SLS files should be assigned to which minions based on targeting criteria.

32. How do minions get their Pillar data?

• Answer: Minions request their compiled pillar data from the master when needed (e.g., during a state.apply or when pillar.items is called). The master compiles the relevant pillar SLS files based on the top.sls mapping for that specific minion and sends only the resulting data structure securely to that minion.

33. How can you view the Pillar data assigned to a specific minion?

• Answer: From the master, use the pillar.items (or pillar.get) execution module function:
• salt 'minion1' pillar.items
• You can also use salt-call pillar.items directly on the minion.

Grains

34. What are Salt Grains?

• Answer: Grains are static pieces of information collected about a minion, such as its operating system, domain name, IP address, kernel version, CPU architecture, memory, etc. This data is collected when the minion service starts and is primarily used by the master for targeting minions.

35. How are Grains typically generated?

• Answer: Most core grains are automatically detected by the Salt Minion agent when it starts. Custom grains can also be defined.

36. How can you view the Grains for a specific minion?

• Answer: Use the grains.items (or grains.get) execution module function from the master:
• salt 'minion1' grains.items
• Or use salt-call grains.items directly on the minion.

37. How do you define custom Grains?

Answer: Custom grains can be defined in several ways:

• In Minion Configuration: Adding a grains: section to /etc/salt/minion or /etc/salt/minion.d/*.conf.
• Using /etc/salt/grains file: A simple YAML file on the minion.
• Using Custom Grain Modules: Writing Python modules in /srv/salt/_grains/ (or other configured grains_dirs) on the master, which are then synced to minions and executed locally to generate grain data.

38. Give an example of using Grains for targeting.

Answer: To apply a state only to minions running CentOS 7:

• salt -G 'os:CentOS' -G 'osmajorrelease:7' state.apply my_centos_state
• In top.sls:

base:
  'os:CentOS and osmajorrelease:7': # Grain Targeting
    - match: grain
    - my_centos_state

Targeting

39. Explain different ways to target minions in Salt.

Answer: Salt offers flexible targeting methods:

• Globbing: Using wildcards on minion IDs (e.g., web*, db?).
• Grains (-G): Targeting based on minion grain data (e.g., salt -G 'os:Ubuntu' ...).
• Pillar (-I): Targeting based on pillar data assigned to minions (e.g., salt -I 'role:webserver' ...). Note: Can be slower as it requires evaluating pillar data.
• List (-L): Explicitly listing minion IDs (e.g., salt -L 'minion1,minion2' ...).
• Compound (-C): Combining multiple target types using boolean operators (and, or, not) (e.g., salt -C 'G@os:Debian and web*' ...).
• Subnet/IP (-S): Targeting based on minion IP address or subnet.
• Nodegroups (-N): Using predefined groups of minions defined in the master configuration file.

40. What is Compound Targeting? Give an example.

• Answer: Compound targeting allows combining different targeting methods using boolean logic (and, or, not) and specifying the matcher type (glob, grain, pillar, list, etc.) using @.
• Example: Target all minions whose ID starts with web OR minions running Debian: salt -C 'web* or G@os:Debian' test.ping

41. What are Nodegroups? How are they defined?

Answer: Nodegroups are aliases defined in the Salt Master configuration file (/etc/salt/master or master.d/*.conf) that represent groups of minions specified using compound target expressions. This simplifies complex or frequently used targeting expressions.

Example Definition in Master Config:

nodegroups:
  webservers: 'G@role:web and L@web01,web02,web03'
  debian_dev: 'G@os:Debian and G@environment:dev'

Usage: salt -N webservers state.apply apache

42. How would you target a single, specific minion?

• Answer: Simply use its minion ID.
• Example: salt 'my-specific-minion-123' cmd.run 'hostname'

Templating & Renderers

43. What templating engine does Salt primarily use in SLS files?

• Answer: Jinja2 (often just called Jinja) is the default and most widely used templating engine for Salt States and Pillar files. It allows embedding logic (loops, conditionals, variables) directly within YAML files.

44. Give an example of using a Pillar value within a Jinja template in an SLS file.

Answer: If you have a pillar value admin_user: 'john', you can use it in an SLS file like this:

# /srv/salt/users/init.sls
{% set user = pillar.get('admin_user', 'default_admin') %} # Get pillar value

{{ user }}_account:
  user.present:
    - name: {{ user }}
    - fullname: Administrator User
    - shell: /bin/bash

45. What are Salt Renderers? Can you use something other than YAML+Jinja?

• Answer: Renderers are responsible for transforming the templated SLS file (or Pillar file) into the high-level data structure that Salt’s compiler understands. The default renderer pipeline is Jinja followed by YAML (jinja|yaml). Yes, Salt supports other renderers like py (pure Python), json, mako, wempy, allowing states to be defined in different formats if needed. You specify the renderer using a shebang-like notation at the top of the SLS file (e.g., #!py).

Advanced Concepts (Reactors, Runners, Orchestration)

46. What is the Salt Reactor system?

Answer: The Reactor system allows Salt to trigger actions (like running states or runners) automatically in response to specific events occurring on the Salt Event Bus. Events can be anything from minion startup (salt/minion/*/start) to custom events fired by applications or states. Reactors are configured on the Salt Master.

47. What is the Salt Event Bus?

• Answer: The Salt Event Bus is the core communication system used internally by Salt (built on ZeroMQ). Nearly every action within Salt (minion connects, key accepted, job starts, job returns, custom events) fires an event onto this bus. Components like the Reactor system listen to this bus to trigger actions.

48. What is Salt Orchestration? How does it differ from state.apply or state.highstate?

• Answer: Salt Orchestration allows defining and executing complex, multi-stage workflows across multiple minions in a controlled sequence, directly from the master using a Salt Runner (state.orchestrate or newer orch.* runners). While state.apply/highstate apply states concurrently to targeted minions based on their individual top.sls configurations, Orchestration controls the order of execution across different minions and different states. It’s useful for tasks like deploying multi-tier applications (e.g., configure DB server, then web server, then load balancer). Orchestration states are defined in SLS files similar to regular states but are executed by a runner on the master.

49. What are Beacons in SaltStack?

• Answer: Beacons are a monitoring system running on minions. They watch for specific system activities or conditions (e.g., file changes, service status, specific log messages, system load) and fire events onto the Salt Event Bus when those conditions are met. These events can then be used by the Reactor system on the master to trigger automated responses.

Security & Best Practices

50. Describe some security best practices when using SaltStack.

Answer: Key security practices include:

• Secure the Master: Restrict access to the Salt Master server itself. Harden its OS.
• Firewalling: Limit access to the Master’s ports (4505, 4506) only to known minion subnets or IPs.
• Key Management: Carefully manage minion keys. Use salt-key diligently, remove old/untrusted keys promptly. Avoid using salt-key -A in production without strict verification.
• Pillar for Secrets: Store all sensitive data (passwords, keys, tokens) in Pillar, not in States or Grains. Consider using Pillar encryption or integrating with external secret management tools (like HashiCorp Vault) via Salt modules/runners.
• Least Privilege: Use Salt’s Client ACL system (client_acl or publisher_acl in the master config) to restrict which users or external systems can run specific commands or target specific minions.
• Secure File Roots: Ensure file_roots and pillar_roots have appropriate file permissions on the master.
• Regular Updates: Keep Salt Master and Minions updated to patch security vulnerabilities.
• Transport: Ensure transport is configured for encryption (default AES).
• Audit Logs: Monitor Salt Master and Minion logs for suspicious activity.