Commands

The Four Letter Words

ZooKeeper responds to a small set of commands. Each command is composed of four letters. You issue the commands to ZooKeeper via telnet or nc, at the client port.

Three of the more interesting commands: "stat" gives some general information about the server and connected clients, while "srvr" and "cons" give extended details on server and connections respectively.

New in 3.5.3: Four Letter Words need to be explicitly white listed before using. Please refer to 4lw.commands.whitelist described in cluster configuration section for details. Moving forward, Four Letter Words will be deprecated, please use AdminServer instead.

• conf : New in 3.3.0: Print details about serving configuration.

• cons : New in 3.3.0: List full connection/session details for all clients connected to this server. Includes information on numbers of packets received/sent, session id, operation latencies, last operation performed, etc...

• crst : New in 3.3.0: Reset connection/session statistics for all connections.

• dump : Lists the outstanding sessions and ephemeral nodes.

• envi : Print details about serving environment

• ruok : Tests if the server is running in a non-error state. When the whitelist enables ruok, the server will respond with imok if it is running, otherwise it will not respond at all. When ruok is disabled, the server responds with: "ruok is not executed because it is not in the whitelist." A response of "imok" does not necessarily indicate that the server has joined the quorum, just that the server process is active and bound to the specified client port. Use "stat" for details on state wrt quorum and client connection information.

• srst : Reset server statistics.

• srvr : New in 3.3.0: Lists full details for the server.

• stat : Lists brief details for the server and connected clients.

• wchs : New in 3.3.0: Lists brief information on watches for the server.

• wchc : New in 3.3.0: Lists detailed information on watches for the server, by session. This outputs a list of sessions(connections) with associated watches (paths). Note, depending on the number of watches this operation may be expensive (ie impact server performance), use it carefully.

• dirs : New in 3.5.1: Shows the total size of snapshot and log files in bytes

• wchp : New in 3.3.0: Lists detailed information on watches for the server, by path. This outputs a list of paths (znodes) with associated sessions. Note, depending on the number of watches this operation may be expensive (ie impact server performance), use it carefully.

• mntr : New in 3.4.0: Outputs a list of variables that could be used for monitoring the health of the cluster.

  $ echo mntr | nc localhost 2185
              zk_version  3.4.0
              zk_avg_latency  0.7561              - be account to four decimal places
              zk_max_latency  0
              zk_min_latency  0
              zk_packets_received 70
              zk_packets_sent 69
              zk_outstanding_requests 0
              zk_server_state leader
              zk_znode_count   4
              zk_watch_count  0
              zk_ephemerals_count 0
              zk_approximate_data_size    27
              zk_learners    4                    - only exposed by the Leader
              zk_synced_followers 4               - only exposed by the Leader
              zk_pending_syncs    0               - only exposed by the Leader
              zk_open_file_descriptor_count 23    - only available on Unix platforms
              zk_max_file_descriptor_count 1024   - only available on Unix platforms

  The output is compatible with java properties format and the content may change over time (new keys added). Your scripts should expect changes. ATTENTION: Some of the keys are platform specific and some of the keys are only exported by the Leader. The output contains multiple lines with the following format:

  key \t value

• isro : New in 3.4.0: Tests if server is running in read-only mode. The server will respond with "ro" if in read-only mode or "rw" if not in read-only mode.

• hash : New in 3.6.0: Return the latest history of the tree digest associated with zxid.

• gtmk : Gets the current trace mask as a 64-bit signed long value in decimal format. See stmk for an explanation of the possible values.

• stmk : Sets the current trace mask. The trace mask is 64 bits, where each bit enables or disables a specific category of trace logging on the server. Logback must be configured to enable TRACE level first in order to see trace logging messages. The bits of the trace mask correspond to the following trace logging categories.

  Trace Mask Bit Values
  0b0000000000	Unused, reserved for future use.
  0b0000000010	Logs client requests, excluding ping requests.
  0b0000000100	Unused, reserved for future use.
  0b0000001000	Logs client ping requests.
  0b0000010000	Logs packets received from the quorum peer that is the current leader, excluding ping requests.
  0b0000100000	Logs addition, removal and validation of client sessions.
  0b0001000000	Logs delivery of watch events to client sessions.
  0b0010000000	Logs ping packets received from the quorum peer that is the current leader.
  0b0100000000	Unused, reserved for future use.
  0b1000000000	Unused, reserved for future use.

  All remaining bits in the 64-bit value are unused and reserved for future use. Multiple trace logging categories are specified by calculating the bitwise OR of the documented values. The default trace mask is 0b0100110010. Thus, by default, trace logging includes client requests, packets received from the leader and sessions. To set a different trace mask, send a request containing the stmk four-letter word followed by the trace mask represented as a 64-bit signed long value. This example uses the Perl pack function to construct a trace mask that enables all trace logging categories described above and convert it to a 64-bit signed long value with big-endian byte order. The result is appended to stmk and sent to the server using netcat. The server responds with the new trace mask in decimal format.

  $ perl -e "print 'stmk', pack('q>', 0b0011111010)" | nc localhost 2181
  250

Here's an example of the ruok command:

$ echo ruok | nc 127.0.0.1 5111
    imok

The AdminServer

New in 3.5.0: The AdminServer is an embedded Jetty server that provides an HTTP interface to the four-letter word commands. By default, the server is started on port 8080, and commands are issued by going to the URL "/commands/[command name]", e.g., http://localhost:8080/commands/stat. The command response is returned as JSON. Unlike the original protocol, commands are not restricted to four-letter names, and commands can have multiple names; for instance, "stmk" can also be referred to as "set_trace_mask". To view a list of all available commands, point a browser to the URL /commands (e.g., http://localhost:8080/commands). See the AdminServer configuration options for how to change the port and URLs.

The AdminServer is enabled by default, but can be disabled by either:

• Setting the zookeeper.admin.enableServer system property to false.
• Removing Jetty from the classpath. (This option is useful if you would like to override ZooKeeper's jetty dependency.)

Note that the TCP four-letter word interface is still available if the AdminServer is disabled.

Configuring AdminServer for SSL/TLS

• Generating the keystore.jks and truststore.jks which can be found in the Quorum TLS.

• Add the following configuration settings to the zoo.cfg config file:

  admin.portUnification=true
  ssl.quorum.keyStore.location=/path/to/keystore.jks
  ssl.quorum.keyStore.password=password
  ssl.quorum.trustStore.location=/path/to/truststore.jks
  ssl.quorum.trustStore.password=password

• Verify that the following entries in the logs can be seen:

  2019-08-03 15:44:55,213 [myid:] - INFO  [main:JettyAdminServer@123] - Successfully loaded private key from /data/software/cert/keystore.jks
  2019-08-03 15:44:55,213 [myid:] - INFO  [main:JettyAdminServer@124] - Successfully loaded certificate authority from /data/software/cert/truststore.jks
  2019-08-03 15:44:55,403 [myid:] - INFO  [main:JettyAdminServer@170] - Started AdminServer on address 0.0.0.0, port 8080 and command URL /commands

Available commands include:

• connection_stat_reset/crst: Reset all client connection statistics. No new fields returned.

• configuration/conf/config : Print basic details about serving configuration, e.g. client port, absolute path to data directory.

• connections/cons : Information on client connections to server. Note, depending on the number of client connections this operation may be expensive (i.e. impact server performance). Returns "connections", a list of connection info objects.

• hash: Txn digests in the historical digest list. One is recorded every 128 transactions. Returns "digests", a list to transaction digest objects.

• dirs : Information on logfile directory and snapshot directory size in bytes. Returns "datadir_size" and "logdir_size".

• dump : Information on session expirations and ephemerals. Note, depending on the number of global sessions and ephemerals this operation may be expensive (i.e. impact server performance). Returns "expiry_time_to_session_ids" and "session_id_to_ephemeral_paths" as maps.

• environment/env/envi : All defined environment variables. Returns each as its own field.

• get_trace_mask/gtmk : The current trace mask. Read-only version of set_trace_mask. See the description of the four letter command stmk for more details. Returns "tracemask".

• initial_configuration/icfg : Print the text of the configuration file used to start the peer. Returns "initial_configuration".

• is_read_only/isro : A true/false if this server is in read-only mode. Returns "read_only".

• last_snapshot/lsnp : Information of the last snapshot that zookeeper server has finished saving to disk. If called during the initial time period between the server starting up and the server finishing saving its first snapshot, the command returns the information of the snapshot read when starting up the server. Returns "zxid" and "timestamp", the latter using a time unit of seconds.

• leader/lead : If the ensemble is configured in quorum mode then emits the current leader status of the peer and the current leader location. Returns "is_leader", "leader_id", and "leader_ip".

• monitor/mntr : Emits a wide variety of useful info for monitoring. Includes performance stats, information about internal queues, and summaries of the data tree (among other things). Returns each as its own field.

• observer_connection_stat_reset/orst : Reset all observer connection statistics. Companion command to observers. No new fields returned.

• restore/rest : Restore database from snapshot input stream on the current server. Returns the following data in response payload: "last_zxid": String Note: this API is rate-limited (once every 5 mins by default) to protect the server from being over-loaded.

• ruok : No-op command, check if the server is running. A response does not necessarily indicate that the server has joined the quorum, just that the admin server is active and bound to the specified port. No new fields returned.

• set_trace_mask/stmk : Sets the trace mask (as such, it requires a parameter). Write version of get_trace_mask. See the description of the four letter command stmk for more details. Returns "tracemask".

• server_stats/srvr : Server information. Returns multiple fields giving a brief overview of server state.

• snapshot/snap : Takes a snapshot of the current server in the datadir and stream out data. Optional query parameter: "streaming": Boolean (defaults to true if the parameter is not present) Returns the following via Http headers: "last_zxid": String "snapshot_size": String Note: this API is rate-limited (once every 5 mins by default) to protect the server from being over-loaded.

• stats/stat : Same as server_stats but also returns the "connections" field (see connections for details). Note, depending on the number of client connections this operation may be expensive (i.e. impact server performance).

• stat_reset/srst : Resets server statistics. This is a subset of the information returned by server_stats and stats. No new fields returned.

• observers/obsr : Information on observer connections to server. Always available on a Leader, available on a Follower if its acting as a learner master. Returns "synced_observers" (int) and "observers" (list of per-observer properties).

• system_properties/sysp : All defined system properties. Returns each as its own field.

• voting_view : Provides the current voting members in the ensemble. Returns "current_config" as a map.

• watches/wchc : Watch information aggregated by session. Note, depending on the number of watches this operation may be expensive (i.e. impact server performance). Returns "session_id_to_watched_paths" as a map.

• watches_by_path/wchp : Watch information aggregated by path. Note, depending on the number of watches this operation may be expensive (i.e. impact server performance). Returns "path_to_session_ids" as a map.

• watch_summary/wchs : Summarized watch information. Returns "num_total_watches", "num_paths", and "num_connections".

• zabstate : The current phase of Zab protocol that peer is running and whether it is a voting member. Peers can be in one of these phases: ELECTION, DISCOVERY, SYNCHRONIZATION, BROADCAST. Returns fields "voting" and "zabstate".