The problem is when hosts go down, we cannot view the historic data of the down's host.
If you restart gmetad service, we will lost list of down's hosts although rrd files still persist in /var/lib/ganglia/rrd . It's hard to dump rrd files to a pretty graph like ganglia web frontend
The way to solve this problem has 2 steps.
First, If hosts go down and list of host still persist in list of cluster but we cannot access the historic data of down's hosts. We can access the data by comment out the block "if ($hosts_down)" in ganglia-webfrontend/hosts_view.php
Second, If you lost the list of hosts. It's hard to recovery the list of hosts but before it losts, we can prepare XML from gmond in normal situation by
$ telnet localhost 8649 > MetaHBaseAll
Keep the 'MetaHBaseAll' file to use in host down situation.
When the hosts go down, we can use this python script to delude the gmetad for all live of gmond.
the python script works as gmond and will always send the metadata to gmetad although all gmonds are dead. By this way, you can access historic data using ganglia webfront end although all hosts down
import socket
ตอบลบfrom time import gmtime, strftime
#Metadata file from $telnet localhost 8649 > MetaHBaseAll
file_metadata = 'MetaHBaseAll'
f = open(file_metadata,'r')
server_sock = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
server_sock.bind(('127.0.0.1',8649))
server_sock.listen(5)
while 1:
(client_sock, address) = server_sock.accept()
time = strftime("%Y-%m-%d %H:%M:%S", gmtime())
print time + " >> Accepted from ", address
client_sock.send(f.read())
client_sock.close()