ALL OUTAGE UPDATES WILL BE POSTED HERE FIRST
Published on: MAY 13, 2017 @ 14:00
Due to the worldwide ransomeware attack, we are updating all our Windows systems to prevent services being encrypted. All windows servers will be rebooted and your services will be restarted after the update.
Outage ETA ~10mins
Customers can patch their personal computers by going clicking HERE or by running a windows update.
UPDATE: All servers have been updated.
Published on: APR 22, 2017 @ 17:42
Server’s hacked with Crypto / Ransomeware
Server 12 has been hacked @ 3:24PM Today.
Server 13 has been hacked @ 3:30PM Today.
All other servers appear to be unaffected. As a result we have shut all windows servers down, until investigations are complete. Linux servers will continue to run.
It is advised that all customers have a backup ready to be restored.
Update: APR 22, 2017 @ 11:40PM
We are still trying to narrow down how the “Nemesis ransomware” got on two of our systems. This seems like a new variant, when we tried to decrypt a file.
Our plan is to move services onto un-affected servers. Although we see a bunch of game data encrypted, we have two options.
1. Restore your latest backup.
2. Some game servers are still running, the game process of these servers have periodic saves which are clean.
We will look at restoring these files.
Time frame is unknown, but should be up and running at-least some time Sunday (Tomorrow). Sorry for any inconvenience, unfortunately this world released some turds into our society.
UPDATE APR 23, 2017 @ 12:28AM
List of Service ID’s affected. The following services will need to be restored.
Services moved to server 14 are:
834,651,855,869,870,872
Services moved to server 238 are:
859,624,833,743
UPDATE APR 23, 2017 @ 12:00PM
Server 181 is online.
UPDATE APR 23, 2017 @ 12:42PM
Server 14 is online.
UPDATE APR 23, 2017 @ 1:36PM
Server 236 is online.
UPDATE APR 23, 2017 @ 2:15PM
Server 182 is online.
UPDATE APR 23, 2017 @ 3:33PM
Server 238 is online.
UPDATE APR 23, 2017 @ 4:52PM
All Services have been moved onto other servers, we have restored what we could. If you have backup you wish to use please contact us.
Published on: JAN 27, 2017 @ 00:14
Server 181 maintenance The downtime of this was 5 minutes and is Complete. All services are running.
Published on: FEB 5, 2017 @ 14:00
Web Server maintenance @2:30PM AEST. The downtime expected 5 minutes.
We had a site outage today @~12:30PM which lasted ~20 minutes
This was due to a flood of customers at this time. This put some heavy load on our website.
We will be upgrading our sites memory to handle much more load.
UPDATE: This is complete, the downtime was 3min’s
Published on: JAN 27, 2017 @ 00:14
Server 181 maintenance The downtime of this was 5 minutes and is Complete. All services are running.
Published on: Nov 9, 2016 @ 11:54
Web-sites offline whilst we patch our server against Dirty COW (CVE-2016-5195)
The downtime of this was < < 1 minute and is Complete. All services are running.
Published on: Jun 6, 2016 @ 14:23
We have come across a major disk issue on server 221.121.159.181. To protect in data loss we have copied services to a new server.
The downtime of this was < 5 minutes and is Complete. All services are running.
The disk has been replaced and rebuilt.
Published on: Dec 5, 2015 @ 12:32
Work being done on Server 180 & 182 expect outage, back online soon.
**** UPDATE **** 13:22
All Services are up and running. If you have any issues please contact support.
Published on: Dec 4, 2015 @ 22:53
Hi All,
Yesterday a update came out for the game ARK Survivor Evolved. We host a large number of game servers for ARK. We have spread the game servers across 8 Windows boxes. Each box contains 6 game servers.
This ARK update created a file on the server that Windows cannot read and will get stuck reading it. When this happens, the server box freezes and locks up, to the point where it is inaccessible every time that file is looked at by the operating system.
This causes heavy CPU usage, to the point that we have to hard shutdown the server to release the lock on the file. On these boxes we also host a number of other game servers, thankfully ARK is not installed on these boxes.
So our option is to submit a bug report in the Steam forums and wait for a reply. We do not want you out of service and we are trying to move the services around to keep you up and running. However the files are around 30G in size and this 1. Costs a lot of money on bandwidth, moving from server to server. 2. Takes a lot of time to copy across. 3. I will be up all night trying to get everyone up and running again and keeping you all happy.
We do not know if this is a global issue with a large number of hosts and 1 by 1 will start dropping.
Just keep in mind we are doing every single thing we can and working overtime to resolve this. I am sincerely apologetic to all customers.
You can follow the progress here
Nathum
