Terminal Server uses etcd for data replication between the nodes. By default, etcd API that is called by Terminal Server is not secured. The etcd API, by default listening on port 2377, can be used to obtain all the data in the distributed key-value store, including usernames and print job metadata.

To mitigate this information disclosure risk, we recommend one of the following approaches, or both:

  1. Block inbound connection to the etcd API (client) port
  2. Enable authentication of the etcd API

Block inbound connection to the etcd API

The etcd client port can be configured in the management interface. The property is called etcdClientPort and defaults to 2377.

Since Terminal Server is the only service that uses this port, and both Terminal Server and etcd run on the same machine, it is not necessary to open this port to external connections for the product to function.

To enhance security, use your firewall configuration to block inbound connections to this port.

Make sure etcd is not being accessed remotely for maintenance purposes before blocking the port.

Enable authentication of the etcd API

In order to secure the etcd cluster, its authentication needs to be enabled and credentials for access need to be configured. Since all instances of Terminal Server in the cluster are calling the etcd API to get/set its distributed store, the authentication credentials need to be configured for all of them. This can be done centrally from the management interface.

It is recommended to first configure the etcd authentication for Terminal Server and then enable it in the etcd cluster.

Once the authentication is enabled, the cluster remains authenticated with one exception which is the reconfiguration/compensation process. This process, which is a part of Terminal Server, can be triggered only an existing cluster is modified by adding/removing nodes (servers). Be sure to check the authentication is enabled after the cluster is modified, and if not, follow the steps below to enable it again. For more information see How and When to Restart a Standalone SPOC and SPOC Group.

Prerequisites

  • Dispatcher Paragon Build 100 (with etcd v2.3) or higher
  • etcd cluster is healthy

In case a witness server is used as part of the etcd cluster, it needs to be upgraded to v2.3 as well, otherwise the authentication cannot be enabled.

Configure etcd authentication for Terminal Server

  1. In the management interface, go to System >  Configuration
  2. Select Expert settings
  3. Set Network >enableEtcdApiAuth property to Enabled
  4. Set Network > etcdApiUser property to root
    1. if required, you can later manually create different etcd user. Then set this property to chosen username. For more information see Additional etcd users
  5. Choose a secure-enough password and set Network > etcdApiPassword property to it
  6. Click Save changes
  7. Restart the Terminal Server services on all servers in the cluster.

Enable authentication in the etcd cluster

The steps below enable authentication for etcd cluster. From that point, the cluster becomes accessible only with the etcd credentials you set. Make sure these credentials are secure-enough but also not forgotten. They are used by Terminal server to access etcd, but will also be required for any maintenance works with the etcd cluster. If the credentials are forgotten, the cluster would probably need to be reset. See Forgotten etcd credentials for more details. 

To enable authenticated etcd cluster using official etcdctl utility, follow the steps below.

In the example commands below, replace the --endpoint URI if necessary to point to the any actual cluster node (Spooler Controller server).

  1. Locate etctctl.exe utility in the directory <install dir>\SPOC\terminalserver\etcd
  2. Check that etcd cluster is healthy:
    etcdctl.exe --endpoint http://127.0.0.1:2377 cluster-health
    The output should be "cluster is healthy". If the cluster is not healthy, do not continue until the cluster is repaired. See Reconfiguring or Recovering an etcd Cluster in Terminal Server
  3. Create a root user. When asked for password, input the password you chose in the previous section Configure authentication for Terminal Server:
    etcdctl.exe --endpoint http://127.0.0.1:2377 user add root
    The output should be "User root created".
    If the output "context deadline exceeded" please retry that again. There is short period for fill the password.
  4. Enable authentication:
    etcdctl.exe --endpoint http://127.0.0.1:2377 auth enable
    The output should be "Authentication Enabled".
  5. Remove guest role. Note that you need to input the credentials of the root user that was created previously. Replace the password in the command below with actual password:
    etcdctl.exe --endpoint http://127.0.0.1:2377 -u root:password role remove guest
    The output should be "Role guest removed".
  6. Check that etcd cluster is healthy:
    etcdctl.exe --endpoint http://127.0.0.1:2377 cluster-health
    The output should be "cluster is healthy".

Etcd API authentication is now enabled.

Alternative: Using etcd API (PowerShell)

In case the enablement using etcdctl utility fails, you can use a prepared script which uses REST API calls to enable the authentication. Copy the PowerShell script below and execute it using the parameters:

  • hostname - IP address or hostname of one of the nodes in the etcd cluster
  • port - port of one of the nodes in the etcd cluster for API communication (by default 2377)
  • password - password to be set for the root user (that you chose in the previous section Configure authentication for Terminal Server)
  • unattended (optional) - if the switch is provided, script does not ask for confirmation before executing

There is no need to run the script for all the nodes (servers) of the cluster, just run it once for any node that is part of the cluster. The script can be re-run if necessary, it will always check if authentication is properly enabled.

Example command:

.\Enable-EtcdAuthentication.ps1 -hostname "127.0.0.1" -port 2377 -password "password"

The script code is below. Copy it and save to a file Enable-EtcdAuthentication.ps1

Enable-EtcdAuthentication.ps1
param (
[Parameter(Mandatory=$true)]
[string]$hostname,
[Parameter(Mandatory=$true)]
[int]$port,
[Parameter(Mandatory=$true)]
[string]$password,
[switch]$unattended, # Optional flag for unattended execution
[switch]$disable # Optional flag for disabling authentication instead
)
 
# Global base URL
$baseUrl = "http://${hostname}:${port}"
$versionUrl = "${baseUrl}/version"
$healthUrl = "${baseUrl}/health"
$authBaseUrl = "${baseUrl}/v2/auth"
$rolesUrl = "${authBaseUrl}/roles"
$usersUrl = "${authBaseUrl}/users"
 
$rootUserName = "root"
$guestRoleName = "guest"
 
$rootUserCreated = $false
 
# Function to prompt for confirmation to continue
function Get-ContinueOrExit {
if (-not $Unattended) {
$confirmation = Read-Host "Do you want to proceed? Type 'yes' to continue or 'no' to exit"
if ($confirmation -eq 'yes') {
Write-Host "Proceeding..." -ForegroundColor Green
} elseif ($confirmation -eq 'no') {
Write-Host "Exiting..." -ForegroundColor Yellow
exit 0
} else {
Write-Host "Invalid input. Please type 'yes' or 'no'." -ForegroundColor Red
Get-ContinueOrExit
}
} else {
Write-Host "Unattended mode enabled, skipping confirmation..."
}
}
 
# Function to make an authenticated API call and handle errors
function Invoke-APIRequest {
param (
[string]$method,
[string]$url,
[hashtable]$body = $null
)
 
$authHeader = "Basic " + [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes("${rootUserName}:${password}"))
$headers = @{ "Authorization" = $authHeader }
if ($body -ne $null) {
$bodyJson = $body | ConvertTo-Json -Compress -Depth 3
return Invoke-RestMethod -Method $method -Uri $url -Headers $headers -Body $bodyJson -ContentType "application/json" -ErrorAction Stop
} else {
return Invoke-RestMethod -Method $method -Uri $url -Headers $headers -ErrorAction Stop
}
}
 
# Function to check if the version meets the minimum requirement
function Check-ETCDVersion {
try {
Write-Host "Checking etcd cluster version..."
$versionResponse = Invoke-APIRequest -method "GET" -url $versionUrl
if ([version]$versionResponse.etcdcluster -lt [version]"2.3.0") {
Write-Host "Error: Minimum etcd cluster version is 2.3.0. Please upgrade and try again. Version 2.3.0 is part of SafeQ 6 Build 100 or higher. If you already upgraded, check if all nodes are up to date and not misbehaving." -ForegroundColor Red
exit 1
}
Write-Host "Version check passed: etcd cluster version is $($versionResponse.etcdcluster)"
} catch {
Write-Host "Error during version check, request failed: $_" -ForegroundColor Red
Write-Host "Double check the hostname '${hostname}' and port '${port}' input parameters. For example, input '127.0.0.1' and '2377' for access from server locally." -ForegroundColor Red
exit 1
}
}
 
# Function to check cluster health
function Check-ClusterHealth {
try {
Write-Host "Checking etcd cluster health..."
$healthResponse = Invoke-APIRequest -method "GET" -url $healthUrl
if ($healthResponse.health -ne "true") {
Write-Host "Error: Cluster is not healthy, cannot continue." -ForegroundColor Red
exit 1
}
Write-Host "Cluster health check passed, cluster is healthy."
} catch {
Write-Host "Error during health check, request failed: $_" -ForegroundColor Red
Write-Host "Double check the hostname '${hostname}' and port '${port}' input parameters. For example, input '127.0.0.1' and '2377' for access from server locally." -ForegroundColor Red
exit 1
}
}
 
# Function to create user if not exist
function Ensure-UserExists {
param (
[string]$userName
)
$url = "${usersUrl}/${userName}"
try {
Write-Host "Creating user '$userName'..."
Invoke-APIRequest -method "GET" -url $url | Out-Null
Write-Host "User '${userName}' already exists, no need to create it."
return $false
} catch {
if ($_.Exception.Response.StatusCode -ne 404) {
Write-Host "Error during user creation, request failed: $_" -ForegroundColor Red
exit 1
}
}
try {
Write-Host "User '${userName}' does not exist. Creating user."
$createUserBody = @{
"user" = $userName;
"password" = $password
}
Invoke-APIRequest -method "PUT" -url $url -body $createUserBody | Out-Null
Write-Host "User '${userName}' created successfully."
return $true
} catch {
Write-Host "Error during user creation, request failed: $_" -ForegroundColor Red
exit 1
}
}
 
# Function to check if authentication is enabled
function Check-AuthenticationEnabled {
Write-Host "Checking if authentication is enabled..."
$authCheckResponse = Invoke-APIRequest -method "GET" -url "${authBaseUrl}/enable"
return $authCheckResponse.enabled -eq $true
}
 
# Function to enable authentication if not enabled
function Ensure-AuthEnabled {
try {
if (Check-AuthenticationEnabled) {
Write-Host "Authentication is already enabled."
} else {
Write-Host "Authentication not enabled, enabling..."
Invoke-APIRequest -method "PUT" -url "${authBaseUrl}/enable" | Out-Null
Start-Sleep -Seconds 3
if (Check-AuthenticationEnabled) {
Write-Host "Authentication enabled successfully."
} else {
Write-Host "Authentication was not enabled. See errors above, fix them and try running the script again." -ForegroundColor Red
exit 1
}
}
} catch {
Write-Host "Error during authentication enablement, request failed: $_" -ForegroundColor Red
exit 1
}
}
 
# Function to disable authentication if enabled
function Ensure-AuthDisabled {
try {
if (-not (Check-AuthenticationEnabled)) {
Write-Host "Authentication is already disabled."
} else {
Write-Host "Authentication is enabled, disabling..."
Invoke-APIRequest -method "DELETE" -url "${authBaseUrl}/enable" | Out-Null
Start-Sleep -Seconds 3
if (-not (Check-AuthenticationEnabled)) {
Write-Host "Authentication disabled successfully."
} else {
Write-Host "Failed to disable authentication. See errors above, fix them and try running the script again." -ForegroundColor Red
exit 1
}
}
} catch {
Write-Host "Error during authentication disablement, request failed: $_" -ForegroundColor Red
exit 1
}
}
 
# Function to revoke role permissions if necessary
function Ensure-RolePermissionsRevoked {
param (
[string]$roleName
)
try {
Write-Host "Checking '${roleName}' role permissions..."
$url = "${rolesUrl}/${roleName}"
$roleResponse = Invoke-APIRequest -method "GET" -url $url
$permissionsRevoked = ($roleResponse.permissions.kv.read.Count -eq 0) -and ($roleResponse.permissions.kv.write.Count -eq 0)
 
if ($permissionsRevoked) {
Write-Host "Role '${roleName}' permissions are already revoked."
} else {
Write-Host "Revoking role '${roleName}' permissions."
$revokeRoleBody = @{
"role" = $roleName;
"revoke" = @{
"kv" = @{
"read" = @("/*");
"write" = @("/*")
}
}
}
Invoke-APIRequest -method "PUT" -url $url -body $revokeRoleBody | Out-Null
Write-Host "Role '${roleName}' permissions revoked successfully."
}
} catch {
Write-Host "Error during permissions revocation, request failed: $_" -ForegroundColor Red
exit 1
}
}
 
function Output-Separator {
Write-Host "------------------------------"
}
 
# Main logic
try {
Write-Host "THIS SCRIPT ENABLES OR DISABLES AUTHENTICADED ETCD CLUSTER."
Write-Host "As part of enablement, root etcd user will be created with username '${rootUserName}' and password '${password}'. No other user will be able to access the etcd API since that point. This credentials needs to be configured in SafeQ 6 Management UI in order for Terminal Server to be able to use etcd."
Write-Host
Write-Host "PARAMETERS:"
Write-Host " * '-hostname <string>' - IP address or hostname of one of the nodes in the etcd cluster"
Write-Host " * '-port <int>' - port of one of the nodes in the etcd cluster for API communication (by default 2377)"
Write-Host " * '-password <string>' - password to be set for the root user"
Write-Host " * '-unattended' (optional) - if the switch is provided, script does not ask for confirmation before executing"
Write-Host " * '-disable' (optional) - if the switch is provided, the etcd cluster authentication will be disabled instead of enabled"
Write-Host
Write-Host "PREREQUISITES:"
Write-Host " * etcd cluster has version 2.3.0 or higher (SafeQ 6 Build 100 or higher)"
Write-Host " * etcd cluster is healthy"
Write-Host " * etcd authentication is configured properly in SafeQ 6 Management UI"
Write-Host
Write-Host "Etcd cluster to be configured is ${hostname}:${port}"
Write-Host
Get-ContinueOrExit
Output-Separator
 
Check-ETCDVersion
Output-Separator
Check-ClusterHealth
Output-Separator
 
if ($disable) {
Write-Host "The 'disable' switch was set, the authentication will be disabled"
Ensure-AuthDisabled
Output-Separator
 
Write-Host "All steps completed successfully, etcd authentication is disabled." -ForegroundColor Green
}
else {
$rootUserCreated = Ensure-UserExists -userName $rootUserName
Output-Separator
Ensure-AuthEnabled
Output-Separator
Ensure-RolePermissionsRevoked -roleName $guestRoleName
Output-Separator
Check-ClusterHealth
Output-Separator
Write-Host "All steps completed successfully, etcd authentication is enabled." -ForegroundColor Green
if ($rootUserCreated) {
Write-Host "WARNING: Please note down the username '${rootUserName}' and password '${password}'. If lost, the cluster would need to be recteated!" -ForegroundColor Yellow
}
Write-Host "See the logs of Terminal Server services to check all is configured correctly. You should see 'Etcd API authentication will be used' and no etcd errors."
}
} catch {
Write-Host "An unexpected error occurred: $_" -ForegroundColor Red
exit 1
}

(Optional) Additional etcd users

Multiple users can be created and even configured with different permissions (roles). This is not necessary for the product to work and in most cases default user and role will be sufficient, but it might be used for hardening the security even further.

To create a user, use etcdctl utility located in the <install dir>\SPOC\terminalserver\etcd. This example will create a user admin and assign him the role root while providing credentials of existing user root:password to authenticate:

etcdctl.exe --endpoint http://127.0.0.1:2377 -u root:password user add admin
etcdctl.exe --endpoint http://127.0.0.1:2377 -u root:password user grant admin -roles root
# due to bug in etcdctl you need to set the password after creation:
etcdctl.exe --endpoint http://127.0.0.1:2377 -u root:password user passwd admin

It is also possible to configure separate roles for access. For example, for even higher security, the root role can be dedicated only to root user, and additional role can be created for reading and writing to the cluster and assigned for users configured in Terminal Server and for support access. For more information, see official documentation.

Disable authentication of the etcd API

Prerequisites

  • Dispatcher Paragon Build 100 (with etcd v2.3) or higher
  • etcd cluster is healthy
  • etcd cluster has authentication enabled

Disable authentication in the etcd cluster

To disable authenticated etcd cluster using official etcdctl utility, follow these steps:

  1. Locate etctctl.exe utility in the directory <install dir>\SPOC\terminalserver\etcd
  2. Check that etcd cluster is healthy:
    etcdctl.exe --endpoint http://127.0.0.1:2377 cluster-health
    If the cluster is not healthy, do not continue until the cluster is repaired. See Reconfiguring or Recovering an etcd Cluster in Terminal Server
  3. Disable authentication:
    etcdctl.exe --endpoint http://127.0.0.1:2377 -u root:password auth disable
  4. Check that etcd cluster is healthy:
    etcdctl.exe --endpoint http://127.0.0.1:2377 cluster-health

This disables the authentication, however the users and roles that were created remain.

If you would like to re-enable the authentication, follow the steps in Enable authentication in the etcd cluster section except for the root user creation, which already exists.

Alternative: Using etcd API (PowerShell)

In case the enablement using etcdctl utility fails, you can use a prepared script which uses REST API calls to disable authentication. Copy the PowerShell script above and execute it with an additional disable switch.

Example command:

.\Enable-EtcdAuthentication.ps1 -hostname "127.0.0.1" -port 2377 -password "password" -disable

Disable authentication for Terminal Server

  1. In the management interface, go to System >  Configuration
  2. Make sure to select Expert settings
  3. Set Network >enableEtcdApiAuth property to Disabled
  4. Click Save changes

Restart the Terminal Server services on all servers in the cluster.

Forgotten etcd credentials

Unfortunately, if you forgot the credentials, etcd provides no way to recover them. Also, the cluster authentication cannot be disabled without the credentials with root role permissions.

If you do not have the credentials and you need to access the cluster, you can try contacting the support, because there is a chance for successful recovery in some scenarios, depending on at what point the credentials got lost. However, most probably the cluster would need to be reset. In order to do that, follow the same steps as for the case when etcd cluster is unhealthy that are described in the Reconfiguring or Recovering an etcd Cluster in Terminal Server guide.

Troubleshooting

In case of troubleshooting issues with etcd you can enable additional logging for Terminal Sever, which includes dump of standard output of the etcd process.

To enable it, add the configuration property TraceEtcdCommunication into the <appSettings> section in the <install_dir>\SPOC\terminalserver\TerminalServer.exe.config file. The value should be set to true. Afterwards, restart the Terminal Server service, and, if needed, repeat for other Terminal Servers in the cluster.

Example

<add key="TraceEtcdCommunication" value="true" />