Smashing the Mega-d/Ozdok botnet in 24 hours

In my previous article, I talked about the Ozdok command and control architecture and its fallback mechanisms in great detail. That article was an attempt to highlight different approaches to take down this botnet theoretically. But when it comes to the actual shutdown, it's far more complex than just finding out the command and control server coordinates and fallback mechanisms. An actual shut down attempt requires someone to take the initiative and start a combined effort involving third parties like ISPs, registries, registrars, etc. 

Instead of playing a passive role, this time FireEye decided to come forward and start working with these groups to make this happen.  The good news is that at the time of writing this article, all the major Ozdok command and control servers (as mentioned in my last post) have been taken down.  As it turns out, no matter how many fallback mechanisms are in place, if they aren't all implemented properly, the botnet is vulnerable.

FireEye's formal effort to shutdown this botnet stared last night. The research team here worked in multiple directions simultaneously. The purpose was to work against all the fallback mechanisms so fast that bot herders wouldn't get a chance to counter react. 

The first step was to prepare all the evidence against the rogue domains and hosts in the form of pcaps and actual Ozdok malware samples. Once the evidence package was ready, these were the steps taken by our research team:

1. Abuse notifications to all the ISPs involved.

So far except for 4 hosts all others were promptly taken down as a result of these abuse notifications (Thanks to ISPs involved). CnCs which are still up and running are as follows: 

98.126.17.114
64.202.189.170
98.126.44.146
62.90.134.24

We hope that the relevant authorities will be investigating these IPs and we will get a positive reply from their side soon.

2. Working with registrars to take down all the registered CnC domains.

Here is the list of Ozdok's active CnC domains. Registrars were requested to take down these domains to cut the main command and control chain.

yokserezantia.net
billibonskanzas.com
yopilazankaza.net
foodcaters.info
gondolfrazrv3.com
pilimerkazana.biz
poelzmdayl.com
grezasadaf.info
zavaretalies.com
galileoboots.info
kuport.com
jamfzuyqyra.com
beztakrezt.info
mazerattikrak.info
pubdomainstr.com
cristymisty.info

So far we got confirmation that these domains listed below are taken down. We are very thankful to the authorities involved.

foodcaters.info
pilimerkazana.biz
zavaretalies.com
grezasadaf.info
beztakrezt.info
jamfzuyqyra.com

We'll keep this list updated once we confirm for other domains too..

3. Registration of all unused CnC domains.

Many domains in the Ozdok permanent CnC list were not registered due to some unknown reasons. FireEye registered all such domains to prevent the bot herders using them to regain control.

These are the CnC domains registered by FireEye yesterday:

ADMZJYDA.BIZ
AJZPLRAKZUI.ORG
ALFAHARPUN.ORG
BLAGOINC.INFO
DFCZNU9Q.BIZ
GREATPUNNETT.COM
HAKASIMQ.INFO
HARMZOAKE.INFO
HOTOPIKALAR.INFO
IZTEP14MRKDE.INFO
JOPITERAZANIA.NET
MAMAFOBIKE.ORG
MICRALOKP.BIZ
MILFIFEZABOQ.ORG
MIRAKLEGROUP.INFO
MIREXINT.BIZ
MKZYAJIUJOIQ.INFO
NAYZIELZP.BIZ
RAFFAELLOPAOLINO.NET
SKILOPER.NET
TYPIREW.ORG
UPOYANSA.COM
WIKIROCKSA.INFO
YANKDREAM.INFO
YOURWAYBASKETS.COM
ZMCBY6VG.BIZ

All of these domains are pointing to our sinkhole server now. What this means is that all the Ozdok zombies instead of connecting to their real CnCs are coming to this sinkhole server. Data collected from the sinkhole server logs will be used to identify the victim machines and help them recover their machines back to a normal state. So far we have seen 264,784 unique IPs connecting to our sinkhole server in a 24 hour time frame. This could be a rough estimate of the current size of Mega-d botnet.

4. Registration of all unused CnC domains.

As I explained in my last article, Ozdok is also capable of generating random CnC domains based on the current date and time. As these domains could also be used by the bot herder to regain control in case all the other domains become unavailable. FireEye has registered these auto-generated domains for the next 3 days..

These domains are

4th Nov 2009 = dfcznu9q.biz

5th Nov 2009  = q0hgbn4t4g5a.info

6th Nov 2009  = lpygopoytqd6mrak.org

 

It looks like everything went right according to plan. This combined effort has been quite successful in retaining this beast for the next couple of days. I just talked to Phil Hay from Marshal TRACE in order to find latest SPAM trends against Ozdok. In his words:

"The last spam message we saw from Ozdok today was some 7 hours ago, looks like you had an impact". 

We are very relieved to see the amount of cooperation offered by most of the ISPs and registrars against our abuse notifications. It clearly shows that it's difficult but not impossible to take down some of the nastiest botnets of the world.

Note: We are currently unsure how long we can keep up with these future domains. We also looking closely how the bot herders will react to this situation. We'll keep you all informed.

Atif Mushtaq @ FireEye Malware Intelligence Lab

Detailed Question/Comments : research SHIFT-2 fireeye DOT COM

Comments

[Robert]

This is excellent work. Spam received by our Barracuda firewall has reduced from over 80K per day (95% of all email received) to under 45K (91% of email). If only some kind of organised international alliance could be formed to systematically work on this issue of botnets. Decimating botnets consistently could eventually make illegal control of PCs and phishing spam campaigns unprofitable. This could bring PCs to the same low level of security risk as Apple Macs. Of course, other measures will also be required such as efficiently apprehending malware authors from all jurisdictions (whether based in Nigeria, Ukraine or US).

I'm concerned that the larger firms (Microsoft, Symantec) may have a conflict of interest and might be less aggressive than they could be towards botnets so that they can sell security solutions (and "Genuine Windows") to their customers. I sincerely hope this isn't the case.

[The Big Bad Wolf]

Wow, this is one hell of a PR stunt.
Iwe never herd of u guys until now so it's one hell of a way to put the company on the map.

Btw i usaly get around 20 spam mail a day and the last cupple of days iwe only gotten around 2-6 spam mails :)

[Sajid]

Interesting... Great effort, and nice job.

[Atif Mushtaq]

Q2: How was my experience while interacting with ISPs/Hosting providers.

Overall it was a great experience.

1. There were some ISPs who never replied to our abuse notifications but pulled the plug silently.

2. There were some ISPs who promptly replied to our abuse notifications and asked for more evidence. After seeing the evidence they pulled the plug and replied with a Thanks.

3. There were some ISPs who did not respond to our notifications at all (mostly non US ). Result is that those servers are still up. Luckily it looks that bot herders simply abandoned those servers, as I am no more seeing any response coming out of these servers.

One thing which was common in all the replies, was something like this:

"We have notified our client about this problem and it looks that these servers were compromised and are now being re-imaged by the our clients".

It looks that ISPs are not convinced that these server might actually be owned by the bot herders. So It doesn't matter what I personally think of this 'owned vs compromised' theory, it is what we have to believe in too..

[Atif Mushtaq]

There are many questions which are being asked by our readers, I would like this opportunity to answer most of these frequently asked questions one by one.

Q1: Killing the zombies machines or killing the malware itself by sending a specially crafted packet.

There are two main points to be noted here..

1.Is there any self destruction mechanism hidden in the code to force Ozdok for killing itself? So far we are unable to find any such mechanism. Our investigation for the Ozdok as a malware is still going on. I'll let you guys know in case we come across any such mechanism.

2. Even if there is any such mechanism, it will be completely illegal to do so, US and international laws do not permit any such activity even if the intention behind is good. So there is no chance that FireEye will involve itself in any such activity now and in future. It is sad but it is how it goes....

[paul b]

It's looking as if bounceback is starting to happen - is this because you can't afford to keep buying domains in front of yourself, or are the herders routing around the damage?

At our location on Nov 7-8 we saw about 50% less spam than normal for a weekend, which is pretty remarkable. The trend is reversing, with the 9th running around 60% of normal and the 10th ~80% of normal spam volume (all "days" are PST, not GMT, sorry.)

Great work on showing what can be done with coordination and an understanding of command and control channels, though!

[Patrick M]

You guys are making the Internet a better place for everyone, I wish more companies would do this sort of thing. As an email server and network administrator, I thank you from the bottom of my heart.

[J]

Inspiring work! Its terrifying how big these things get.

[Ross Thomas]

Fantastic! Another one bites the dust. Good work, guys :)

[anon]

Have you considered trying to establish your own c&c facility to instruct infected machines to clean themselves?

[Meh]

So... you worked for free and now have nothing more to talk about? Oh! I know, you can tell us how your email campaign with isps works out for you.

It's generally a good idea not to do things that put yourself out of business. Well maybe you can have a good cry over a beer with these spammer monkeys.

[Eric]

Fascinating, I wish Google or Microsoft or the U.S. Government could fund more such efforts.
Eric

[GarWarner]

FireEye does it again! Tremendous job! Its great to see security companies that fight computer crime instead of just profiting from it. You may be small, but you are a significant leader in this area! Well done!

[joe blow]

i would just like to say, that i applaud your actions in getting this botnet shut down. as soon as this one goes down, please keep shutting them all down.

[James McQuaid]

Excellent work Atif!

[tw]

Spamhaus has SBL record indicating that is yopilazankaza.net is pointing to 195.161.113.218, Ref: SBL80926
Great to hear that 174.139.16.50 is offline.

[Atif Mushtaq]

tw,

Are you trying to say that 'yopilazankaza.net' has started pointing to some IP other than 174.139.16.50. I am not seeing this change at all, the domain is still pointing to 174.139.16.50 which was taken down by the ISP involved recently. In other words I am not seeing any re-gain attempt made by bot herders so far. May be they are waiting for the right time...

[tw]

Very nice research--have passed it on to several powers that be.
At least one of the domains in the list (yopilazankaza.net) already appears on a new IP address with nameservers associated with Conficker and Braviax. Hope Ozdok efforts to recover simply exposes more of the participants and malware infrastructure.

[AlphaCentauri]

Nice work! This is extremely significant, because the biggest obstacle to fighting these botnets is the attitude that "There's no point trying, it will never work, there are too many of them, none of the other ISP's would cooperate, there are too many computers running unpatched pirated Windows OS's," yadda, yadda.

Now everyone can see that with knowledgeable planning and a coordinated evening's work, you can take 264,784 bots off line, with only four C&C IP's left to take down. It makes it look a lot more feasible to get a whole botnet at once, and puts a lot more pressure on the ISP's that fail to cooperate. With further coordination between the multiple entities working on the problem, it should be possible to take multiple botnets down simultaneously, greatly reducing the concern about retaliatory attacks against cooperating hosts.

[Nart]

Great work! I'm wondering if you guys might post a sort of "lessons learned" on how to do effective notification. What is the best way to contact ISP's, registrars, and CERTs? What kind of "evidence" do they require to take action? How do they prefer the evidence to be presented? Etc... This would be especially for individual researchers. Thanks. -nart

[diocyde]

On a side note, it is truly embarrassing and goddamn shame that it takes a bright SNappy company young upstart such as yourselves to actually make a HUGE impact on the entire security of the Internet. Larger companies like Symantec, Microsoft, Trend, and McAfee should be embarrassed that they can grasp these simple concepts and take down some of the larger botnets the same way. on a broader scale and with much more speed and tactical surprise. SHAME on them. They clearly have the resources to do it. If you get rid of the background noise, Top 15 largest botnets, you can focus on the smaller more damaging botnets, that at times can raid and pillage a company before they even know what hit them. Think CoreFlood. and the over 500 and growing different Zeus networks, Clampi, and ilomo.

Did you collaborate with any other vendors or was this a one man show? Maybe you can partner with someone to Start taking out the top 15 - 1 by 1 until their finances dry up. Or get some seed money and Payoff and empower 1 or 2 of the largest with bribes, implants into their organization or coersion, to get them to roll or ratt out the leadership on say the bottom 8 botnets. Typically if they are going to be Russian Eastern Europa based, all the hackers primarily know each other in the under ground over there.

My suggestion is follow up with the Trend Research report on the Asprox guys. Trend went literally to their doorstep and then stopped. They operate with impunity and I would suggest they are NEXT on the SMASHING BLOCK>

Go for it. LEAD or DIE. or just continue to operate and make money but dont do a dam thing about the problem like most of the other major security companies do.

[Sim]

great job guys!
I wish a lot of people will follow your exemple and take down the botnets they are working on!

[diocyde]

This is awesome, however I am concerned about the lack of response from some registrars and ISPs on the abuse and takedown notices. As you know any gaps could allow for an Update to slip in.

It should be possible for you to contact the ISPs for a file extraction pull of the Command and Control backend software (usually PHP) so that you can analyze that and compare it to other C&C consoles) As well the C&C's stupidly use the same sites or near IPs for their drop sites. You may well be able to capture miscreant intel on the CUSTOMERs of the spam services. Publicising this would go a long way and help demonstrate the volumne of their operations and their cashflows.

Additionally IP logs, may be useful coming inbound to the C&C's if they stupidly are not coming in via chained proxies or some other anon service or botnet authenticated proxy. Please Do your best and get a copy of the C&C and publish an analysis, this type of intel is sorely lacking.

In a perfect cyber warfare style (no freaking lawyer world) if the other ISPs are complicit or obstinate or on the take and try to undermine your efforts, a Sustained DDOS on the few remaining C&Cs would be needed to ensure they stay down (or you can brick the systems if they were dedicated hosts and not virtual hosts, or you can pop the box and block any incoming connections to the interface)

You could additionally call the local police in that jurisdiction to get them to respond or better yet the FBI could could most likely ensure take down (if they had the will, this is debatable and doubtful)

Do you have any intel on the origins of the operators? Is it dedicated or rented out to multiple parties? What geographic location is the owners or are they globally distributed?

Hopefully this will go towards more damaging, industry coordinated attacks on Malware infrastructure which as you know if done right, can be enormously successful.

You know your doing right when you get DDOS some time in the near future. Hurt the cash flow and make an impact.

A great article would be for other providers on how to assemble a evidence package. Standardizing this for the industry would go a long way.

Diocyde

http://diocyde.wordpress.com/

[Steven Burn]

Nice one :o)

[spamislame]

I am uncertain if this is related but a forum I maintain with several colleagues was under a very severe, sustained attack until sometime last night. This may have coincided with your efforts.

Either way this is excellent news, and as usual extremely good research.

SiL / IKS / concerned citizen