April 3, 2020
  • 5:53 pm Joe Biden to address impeachment and Trump-Ukraine whistleblower call , live stream
  • 5:53 pm President Trump’s 2020 State of the Union address and the Democratic response (FULL LIVE STREAM)
  • 5:53 pm Video – Need help? Call Blue Card Services!
  • 5:53 pm Temple University Student’s Viral Tik Tok Video Calling North Philadelphia ‘The Ghetto’ Causes Outra
  • 4:52 pm @TorontoPolice News Conference Re: Homicide #54/2016 Jarryl Hagley, 17 | Fri Oct 21st, 1pm
SANS DFIR WebCast  – Virtualization Incident Response & Forensics:


– [AJ] Hello, and welcome
to the SANS APAC webcast IR and computer forensics
in the virtual realm. I am AJ Boyle, and I will be
moderating the course today. The instructor is Paul Henry. Before I turn the
time over to him, I would like to cover
features of the interface for those who have not attended
a SANS APAC webcast before. The screen is split up
into three main regions, the participants list, the
direct message chat window, and the whiteboard, which
has already been loaded. The participants list
has several columns: the first with the
little hand indicates someone has raised their
hand to ask a question, to raise your hand,
click the hand button right below the
participants list. The next column is
a yes/no indicator, you can click the
toolbar button with the green check or
the red x to indicate a yes or no answer to a question from a moderator
or the instructor. The next column
is the microphone, if this icon has a
yellow background, someone is talking. The microphone will be
blocked, so only one person can speak at a time. If you have a turn to
speak, you can click the microphone button in
the audio section. Do note that Illuminate
software will buffer speech, so if you experience
network delays playback will speed up, causing
a chipmunk effect. The next column is
the chat indicator, anyone who is typing in
the messaging entry box will have a yellow
background here. I will cover how messaging
works in a moment. The next column
with the pin icon is a Whiteboard editing
indicator; I have disabled Whiteboard editing
for participants. The last icon before names
is application sharing, which has also been
disabled for participants. Finally, the list of names
you see are your classmates, moderators, and the instructor, who is also a moderator. Let me explain how the
messaging system works: to enter a message,
point your cursor at the entry box below the main
window before sending, use the drop-down
to select who the message will go to. Now, the choices are
all participants, only moderators, and only
selected participants, you can select names
of other students, the instructor, or
only moderators to send a message only that person, but do note that moderators
will see all messages, both public and private. As mentioned, the
whiteboard already has the slides loaded, the
instructor will advance the slides as he moves
through the course material, and it will automatically
advance for all students. If you experience
technical difficulties during the class,
please send an email to [email protected]
or notify your moderator. Thank you for your attention, now let me turn the mic
over to your instructor, Paul Henry. – [Paul] Hello everyone, my
name is Paul Henry with SANS, today’s topic is going to be
“IR and Computer Forensics “in the Virtual Realm”. You know, according to
the numbers from Gartner, 80 percent of those
servers that could be virtualized have
been virtualized, yet 60 percent are
actually not as secure as their physical counterparts, hence, we can count on
having incidence response and computer forensic issues
within the virtual realm. Critically important
that people very quickly update their policies
and procedures to address providing for IR
and forensics within the virtual realm. Now Again, just a little
background on myself, I’ve been with SANS
for a number of years, many of you may be
familiar with me, I did quite a bit of work
back in my cyber guard days in Southeast Asia. For about the last
four years I’ve run a forensics practice
here in South Florida called Vnet Security,
we primarily focus on providing incident response
and computer forensics in the virtual realm. I’m the board VP of FACCI, the Board USA seat for ISFCC, board member of Ashlar, member of the Miami
Electronic Crimes Task Force, a member of
HTCIA and HTCC as well. I hold a large number
of certifications within the industry,
so I’ve sat in the seat many of you are
sitting in today, I sincerely believe in
the value of third-party validation for credentials,
et cetera, et cetera, as you can see from my sig. And I’ve written quite
a few book chapters over my career, including
many of those that deal specifically with both
security and forensics. Now, at SANS, what do I teach? Well, I teach 401
Security Essentials, I teach all the
forensics tracks at SANS, as well as some of the fun
stuff, such as SEC 553, and of course, the
newer, in fact, we just released the
new 579 virtualization security around the
first of the year, and of course, I’m the
author on the VoIP security class for SANS. Now SANS is the most
overall complete curriculum out there today with respect to computer forensics,
and we typically bring people in with FOR 408, Computer
Forensic Investigations, Windows in-depth,
they would advance to FOR 508, Advanced
Computer Forensic Analysis and Incidence Response,
and then you have a decision to make, are
you going to move onward with Network Forensics, FOR 558, or perhaps mobile
device forensics, FOR 563, you can then advance to FOR 610, Malware Analysis
Tools and Techniques, and of course, FOR 526,
Advanced File System Recovery and Memory Forensics. Now on this slide, it
does indicate the website for the Forensics
Tracks app SANS, where you can find very
valuable information, there’s also a tremendous
amount of insight available from industry
experts on the SANS forensics blog, I would
highly encourage you to take a look. Now, if you have in
fact taken a SANS class and you do have a portal
account, there’s a number of really exceptional
free tools available from SANS, such as
the SIFT Workstation. You know, I have access
to pretty much every commercial tool out there, and regularly, I find
myself migrating back to the SIFT Workstation
from SANS when I have to get something
done very quickly, simple as that. We also have some really
cool digital forensics challenges, if you want
to test out your skills, also available on
the SANS website, and of course, most of
the instructors at SANS do use Twitter, so
again, you can follow most of us on Twitter,
my Twitter handle would be phenryCISSP. Now, moving forward,
we’re gonna jump into a little bit
of the basics here. Looking at virtualization
technology today, most of us are
primarily familiar with server virtualization,
that’s taking that physical server and
basically abstracting it into a virtual environment. We of course have
application virtualization, application virtualization
is kind of like sand boxing, a number
of vendors have taken this approach, in fact,
in the new version of the operating system
for Apple that’s coming up here in the July timeframe, they’ve now taken a
turn and they’re going to be using sand boxing and/or
application virtualization within the Apple
platform itself. Now on the desktop
virtualization, what we’re talking about there
primarily are products like Workstation, and of
course, Fusion for the Mac. On storage virtualization,
we’ve really seen this roll out
through VMware here in the last couple
of product releases. You now have the ability
to fully abstract storage within VMware,
hence you’re able to move that entire
storage array using a storage V motion
off of the hardware, repair that hardware, if you
have to apply firmware patches, et cetera, et cetera, and then simply storage
Vmotion it back. Now the primarily players
today in server virtualization would of course be VMware,
they’re the 800-pound gorilla, Citrix, and of course Microsoft. Microsoft is putting a major
push into virtualization currently, it’s clearly
reflected in their pricing. If you price out at
configuration with literally the same capabilities,
now, mind you, you’re gonna have to turn it
down a notch for Microsoft, they’re pretty much
following VMware today, you’ll find that the pricing
is significantly less for the Microsoft platform. Now just to get some
terminology squared away here, with respect to
server virtualization, when we’re referring to a host, we’re talking about the
server virtualization platform that you would
host virtual machines upon. When we’re talking
about a guest, we’re talking about
virtual machines running on top of a host platform. Now, speaking of
hypervisors, this would be your virtual machine
monitor, or VMM, that’s the software that
enables virtualization on the host. You have your Type 1 hypervisor, that would be your
bare metal hypervisor, it really is somewhat of
a self-contained platform. You know, VMware today
primarily operates in a bare metal configuration,
that’s where you get your best
performance with VMware. A Type 2 hypervisor
runs on top of a traditional operating system. A good example of
a Type 2 hypervisor would be of course, VMware
Workstation or VMware Fusion. Again, you have the
overhead of dealing with the underlying operating
system, hence you’ll always get better performance
of a Type 1 hypervisor, the bare metal implementation. Now in virtualization risks,
it all drills back down to the old target we’ve
all seen in the past. In the x86 platform
privilege levels are as follows: ring 0 is your
most privileged level, and it’s where you’re
essentially controlling
the hardware. Ring 3 is where your
applications themselves typically function. Now, a hypervisor’s job
is to present securely that virtual ring 0 to each of
your virtual machine guests. Now the problem that could
potentially arise of course, would be that a compromise
of the hypervisor could in fact lead to
control in ring 0. Just yesterday US Cert
published a bulletin that a guest to host
exploit has been discovered within virtualization
that could potentially allow a guest to
compromise a hypervisor. Now, I read through the
bulletin myself last night, it does not impact VMware. That being said, it looks
like the primary risk today would be associated
with the Citrix product as well as the
Microsoft product, you might want to
Google that, again, that would be a cert
bulletin on virtualization whereby a guest could be used in the compromise of a host. Now within VMware,
looking at the most common infrastructure
out there today, we’re gonna be
looking at version 4, it’s very similar to
version 5, mind you, but we’re seeing the largest
deployments out there are still four, as people
migrate to version 5. You of course have your ESX
and your ESXi server itself, you’ll see that as the
all-encompassing block surrounding everything. You of course have
your virtual switches, you have your vCenter
management console, you have your vSphere
clients that allow you to connect either to
vCenter or directly to the or directly to
the ESX/ESXi host, you would then have
your virtual machines running on top of
ESX and/or ESXi, and of course your
storage, so that just gives you a very quick overview
of the infrastructure associated with
the VMware product. Now we’re gonna look at a
couple of different options available to users
today, and how they’re gonna handle their DMZ. The picture shown on
your screen right now is a partially collapsed
DMZ, I’m seeing a lot of deployments
utilizing this methodology of configuring their DMZ
so they can take advantage of still using
their still existing physical security products. You’ll note that many DMZ
systems are being virtualized today, the network
connections are still physically distinct,
hence you’re able to connect those
network connections to your existing firewall
products, IDS, IDP, et cetera. This provides the most
flexibility with existing network security tools. Again, looking carefully
at the picture, you’ll notice that we’ve
actually separated our servers into individual containers
utilizing ESX server, we’re bringing that
output out through a physical net, and
you’re able to isolate your trust zones
using your existing physical security products. Now looking at a
partially-collapsed DMZ. In a partially-collapsed
DMZ we’re running multiple servers on
a single copy of ESX. In the most current
VSS from PCI, they indicated that the
level of trust available on VS separation within
a copy of ESX server is in fact equivalent to
a physical separation; hence, your able to run
distinctly separate servers on top of a single
copy of ESX server. Now you’ll note in this
drawing, we’re still bringing out the IO to
physical interfaces, so even though we’re
running multiple servers on top of a single
copy of ESX or ESXi, we’re still able to
take full advantage of the existing, the physical
infrastructure that we may have for our
existing security products. Now we have a
fully-collapsed DMZ. In a fully collapsed
DMZ you’re really gonna get your money’s worth
out of the investment that you’ve made
in virtualization. Now, you’ll not that
we have a single copy of ESX server, we’re
running multiply individual servers, taking advantage
of the separation provided by ESX, but
we’re using virtual firewalls, we’re no
longer using our existing physical firewalls. Now again, the problem
that we have today with respect to fully
collapsed DMZs is we have very little
history behind today’s virtual firewalls. Now, that’s not to
say that they’re not equivalent to yesterday’s
physical products, but the bottom line
is, we really don’t have the history with them. You know, you’ll note
that it really seems that firewalls
vendors as an example, thought this whole
virtualization thing was only gonna be a fad, and
very few of them wrote the necessary software to
interact with the Vsafe EI to able to apply policy
to intra-VM traffic, it’s kind of a problem. I mean again, we had
Trend Micro, we had Altor, et cetera that had been in the
game for quite some time now, but if you go back
just one year ago, you could count the
number of firewalls that could be deployed
within a virtual environment that could actually
see and act upon the intra-VM traffic, you
could count them on one hand. Now it’s changing rapidly,
I know that Checkpoint as an example, the market
leader in physical firewalls has actually now released
a product, it was about six months ago, that can
actually be quite effective within the virtual realm. So it remains to be
seen just how well virtualization firewall
vendors are going to perform and how
good of a job they’re gonna be able to do, again. Six months of a product
being out there is not quite enough time
to make a determination as to how much security
it’s gonna be able to afford, so for myself,
I’m not recommending a fully collapsed DMZ yet,
I want to see a little bit more operational time,
see a little bit more track record and history
behind these products before I’m willing to
commit them for my clients. Now in looking at your
overall ESX architecture, it’s kind of interesting. You had of course your
virtual machine monitor, and you had a service console. Now, with an ESX that
service console was based upon Red Hat, simply put, you had a full Red Hat
console available within the ESX platform. Now ESX is being replaced,
it’s being replaced by ESXi, it’s a
major shift of ESX, there is reportedly no
service console at all. Well actually, it’s still there, it’s just somewhat
unsupported, it has less than a 90 megabit footprint,
it can be embedded within the hardware itself, I
see a lot of Dells today being shipped with
VMware ESXi embedded, or of course, it can be
installed using media. It of course does support
most major vSphere features, and ESXi is free from
VMware, where they’re making ESX a little bit
difficult to get nowadays. VMware is clearly moving
to ESXi, that’s where the future is, so again,
if you’re running ESX, migration to ESXi
is in your future. Now shared storage
in VMware, all of the cool things that you
can do with VMware really requires shared storage, Vmotion, HA, DRS
simply cannot be done from local storage,
local storage is typically only used for
storing things such as templates, ISOs, et cetera, highly likely the
VM you wanna get to from an IR or forensics
perspective is stored on shared storage. I find it somewhat
laughable, I’ve seen a number of people
come in and perform investigations within
the virtual realm, whereby they would go
the system administrator and they would ask the
system admins specifically which one of those
one use servers was running the virtual machine. System admin would
point to that Dell 1950 on the rack, the law
enforcement typically would unplug the machine,
image the hard drive, and go back to their
lab to analyze it, only to find in fact,
the virtual machine does not exist on that
hardware platform. Again, most virtual machines
today do not operate on the local storage of
the hardware platform that is running the
virtual machine, it does exist on shared storage. Being able to actually
locate where that virtual machine
VMDK folder exists and being able to carve
it out is key to doing an effective job in incidence
response and forensics. There a couple of more
gotchas associated with that, and we’ll talk about that
as we progress forward. You have a number of
different types of shared storage available today. We have network-attached
storage, NAS, typically using NFS,
seeing this really beginning to take hold today. You of course could
have your SAN running on top of Fibre
Channel, it in fact is the best-performing
solution out there. If you’re virtualizing
something such as a web server that’s
used in eCommerce, where you’re gonna
have a large number of simultaneous connections,
Fibre channel will in all likelihood
be your first choice. I’m seeing a lot of
SANS use using iSCSI, still within small business. You can actually go
out and download a free ISCSI server from
the VMware marketplace, install it on top of
some older existing hardware that you may
have, and do a good job of handling shared
storage, but again, you’re not going to
be able to handle the higher IO that
might be associated with an eCommerce site in
comparison to something such as a Fibre channel SAN. Now the vast majority
of people that I’ve worked with
here in South Florida are running their
SAN over iSCSI, that being said, on
newer deployments today I’m seeing a lot of
pickup in the NAS running NFS, I’m running
here in my lab myself, a year ago we had
issues because very few NAS devices were in fact
on the HCL from VMware, but that’s changing
rapidly, I just recently switched my own lab
to NFS for my NAS, I’m running a ready NAS here,
and is working quite well, I have no downtime
with it whatsoever, it is much simpler
to deploy that iSCSI. Now we have to talk about
VMware’s VMFS file system. This is the virtual
machine file system, it’s a journal file
system that was created by VMware, it can
handle multiple disks, multiple LUNs, it is
able to work with up to 256 VMFS volumes
per ESX host, that would be Version 3, and again, multiple ESX
hosts can concurrently access the very same
VMFS volume and files. Now typically you
would manage VMFS using the VMKFS tools
command on the command line. All right, now one of
the issues you’re gonna have is occasionally you’re
gonna have to work with that VMFS file system
outside of VMware, as an example there,
if you have a client use a third-party to
come in and create a DV image of that
VMFS file system and present you with either a DV or a EL1 file, you then, as
the forensic investigator, would have to mount
that to be able to carve the VMDK to
get to the files associated with that
virtual machine. Now there are some great
third-party tools out there, VMFS from open source
virtual file machine system driver is a great tool,
I’ve provided the URL for it here, it’s up on
the Google code site, it can work on a command line, or it use it with WebDAV. There are also a couple
of new Java applications out there that can
also work with VMFS. Now it’s critical that
you understand that there is not a single
commercial tool out there today, end
case FTK et cetera as an example, that can
actually understand VMFS. They can all work with the VMDK, but none of them have
the ability to carve out a VMDK from VMFS, that really
is a limiting function here, it’s been a problem
for many, many people. Again, it would seem
that virtualization with respect to forensic
vendors looks like they really dropped
the ball again, it would seem that
they kind of thought this whole virtualization
thing was nothing more than a fad, and
again, we see no support from either guidance
or from FTK for that underlying VMFS file system. Now again, on the
vendor support, it simply does not
exist, it’s really tough to blame the vendors, though, as VMware never released
any standard that they could write code for. Now again, while
there is no support for VMFS, the VMware
or VMDK folder itself, which is the folder
that would contain the entire abstraction of
the virtual machine is a published standard,
and most forensic vendors can in fact now
and analyze a VMDK, but you first have
to carve it out of the VMFS file system; that
can be a little tricky. Now, we show a number
of different ways to accomplish this
within our course itself, we work with primarily,
the Java capabilities that was brought out,
as well as a couple of binaries that are
available today that do allow you to actually
mount a VMFS file system and then carve out
the respective VMDK. Now Encase and of
course and VMware, Encase can in fact
analyze a VMDK data file, this is the folder that
contains the abstraction, but again, you have to
keep in mind that they cannot carve it from the VMDK. Now what’s really cool
about VMware is that all of the files
associated with a specific virtual machine are contained
within that VMDK folder. What’s even cooler yet
is they’ve abstracted the physical hard
drive, so effectively, when you make a
forensically-sound copy of the VMDK, you’re actually
getting that virtual abstraction of the hard drive,
which would include both allocated and unallocated space. So you’re able to
actually then carve out deleted files that
would typically be associated unallocated
space from within that VMDK, very cool
capability, I’ve done it regularly myself. Again, I would simply
mt VMDK, either using Encase or FTK, and I’m
able to recover all of the deleted files from
the abstraction of the hard drive, because
it does include within the abstraction all of
the unallocated space. Now there are a number
of other products out there, some
pretty cool ones, it you’re working on VMFS
itself off of a VMware platform, you can download a free
copy from Sanbarrow of the product called MOA. MOA will allow you
as well to work with an actual VMFS outside
of a VMware environment so again, another great
tool for working with VMFS outside of VMware. Now on recovering
deleted files within a VMFS file system; this
gets a little bit tricky. If we look at ESX in
version 3.5 update 3, we actually had a really
good undelete capability. They created a list
of all of the blocks where the files were
stored, you would simply log into the service
console and enter the VMFS undelete
command, and you could recover files that
had been deleted from version 3.5 of the
VMFS file system. However, since version
4 from VMware, there is no longer any simple
way to undelete a file. VMFS no longer keeps a
backup block list associated with any specific file. Hence, no block list equals
no recover capability, this absolutely is a
sore point within VMware. Again, it’s hard to
understand why this capability would have been
removed, but we all have to clearly understand
that VMware is a high-performance
virtualization platform, it is not necessarily an
IR or forensics platform. Now from a forensic
considerations perspective, looking at VMFS,
if you’re trying to get at a single
virtual machine, it can be a bit of a problem. You may have in
fact, hundreds if not thousands of virtual
machines that could potentially be impacted
by simply taking the shared storage VMFS
file system offline. We also have to
consider privacy; again, you could have
hundreds of thousands of virtual machines running
on that shared storage, if you copy out a copy
of the VMFS file system, are you going to have possession
of intellectual property belonging to other clients
that are not really within the scope of
the search warrant? Probability of recovery
of any deleted materials is low, so why not focus
on the specific VMDK for the virtual machine
that really is the focus of the investigation
that you’re doing? Now in the VMware VMDK
the simplest of terms, it’s your container for
all the abstracted files associated with a
specific virtual machine. Now again, the word
“abstracted” is really key here, as the VMDK actually
in some respects, emulates a physical hard disk. So again, you’re gonna
get both allocated and non allocated space
for the abstracted hard disk in your
copy of the VMDK. Now VMDK file types,
there’s a number of different file types available
when you create your VMDK. You have a zero thick,
eager zeroed thick, thick, as well as thin. Now a thin would be your
default on NFS volumes in your VMware workstation
product itself, some of these disks formats
will actually overwrite the disk itself with
zeroes as they construct your virtualized disk. Others in fact will
overwrite with a zero only as they’re actually
creating a specific place for the file within that file system structure, so again, a careful thing to
consider there is that in some cases, as you’re
creating an abstraction of the hard disk within
VMware, it’s actually overwriting all unallocated
space as you create the disk, you’re gonna gain
a little bit there with respect to performance,
but it’s going to take longer to create the disk. The alternate of course,
is to only overwrite with zeros as you’re
actually creating disk that you’re going to be using. Now in that light you’re going
to pay a performance hit, but you’ll be able to
very quickly create that disk, because
you’re not allocating the entire disk instantly,
you’re only allocating space as you need it. I primarily see that
methodology being used in production today, people
tend to over commit disk space so they will run in a
manner whereby they’re only creating the actual disk
space as they need it. Now you’ll also have a
couple of different modes of operation
available with VMDKs. You have persistent
mode, which is in fact is the default mode,
whereby the VMDK behaves just like a standard
physical disk, as you make changes,
they are instantly written to the disk. We have an alternate
mode of operation called nonpersistent mode,
once set as nonpersistent, changes are written
essentially to RAM, they’re never written
to the hard disk. You’ll find that operations
such as a kiosk, et cetera, would be great application
of nonpersistent mode, users could walk
up to the console, take care of whatever
business they have to take care of,
and essentially, nothing they have
done is written tot eh hard dis. Hey, if you’ve got kids,
nonpersistent mode’s another great mode of
operation you might wanna consider for them. I have five children myself,
they like playing games such as WOW et
cetera, and again, they’re constantly
downloading those cheat codes, and the malware that
comes with them, so they’re operating in
non-persistent mode in my house, when they download
that stuff and they include malware,
it’s never written to the hard disk,
so essentially, each time they log into
their VMDK in VMware, they’re logging into
a clean machine and I’m not wasting my
weekends cleaning up PCs infected with malware. So again, you might
find that within your own organization, the
use of nonpersistent mode may in fact be a benefit,
and could in fact be usable within your environment. Again, I primarily
see it used in terms of a kiosk-like
operation, you may want to consider that let’s say,
as a mode of operation you would use in
your office lobby, where you’re allowing
vendors, et cetera, that are visiting your
company to check email, et cetera, again, it
does not save any changes that are made to the
system, nothing is written to the hard drive. Now let’s go ahead and
create a virtual machine called SANS and see
what kind of files does in fact generate. We of course are going
to have a SANS.vmx, that’s our actual VM
configuration file, you can actually take
a look at VMX file using a text editor, it
really is kind of cool, you’re able to see
the full configuration simply looking at
it with a text file. You have your SANS.vmdk,
this essentially is the data associated
with the abstraction of the hard drive itself. Now again, you can
view the VMDK with the text editor,
and it will show you exactly what you
would think you see in a configuration file
for a hard disk, it will talk about clusters,
sectors, and everything that we’re primarily
familiar with in describing a hard disk. Now, we would also
generate a SANS-flat.vmdk, that’s the actual binary
file that represents the abstraction
of the hard disk, so no, you would not
be able to view that with a text editor. It also creates a
SNAS.nvram file, which would be the virtual
machine’s BIOS file, you can view that
with a text editor, and of course, again,
what you would expect to see in a BIOS file,
such as boot from CD, et cetera, et
cetera, you will find within this BIOS file. We have a number of
different SANS startup log files that are in
fact created as well, that would be associated
with that individual virtual machine. We get a SANS.vswp,
which would represent the virtual machine’s swap file, we would get a
SANS.vmsn, or vmsd, depending on whether or not we are able to run
under a workstation or run under the bare
metal hypervisor. Now I’ve got a question
here, I’m gonna go ahead and answer
that real quickly, been paying attention
to the slide deck, not looking at chat, I
assume you’re gonna hold questions till after,
but we’re gonna go ahead and jump on these now. So the question is, is
nonpersistent hardened? No, absolutely not,
it is not hardened, it does give the benefit
in fact, of not writing changes to a hard
disk, but it really is not hardening. That’s one of the
issues I see in virtualization today
is so many people are mistakenly assuming
that simply because they’re running on
top of a hyper visor, they don’t need to
harden that guest, you absolutely are skipping
a critically important step if you’re not
hardening the guest. So again, would running
in nonpersistent mode replace going through
and either using DSS dig or CIS hardening
guide on the guest OS, absolutely not, you
still have to harden that operating system. I’ll go ahead and jump
back to our slides here. So again, you would
get the SANS.vmsn or the vmsd, depends on
whether or not you’re running on top of a
bare metal hypervisor or on top of work station,
it’s actually kind of cool, it’s your virtual machine
snapshot metadata, we of course have
our SANS delta files, this would be a real-time
snapshot write file, you know, when you
create a snapshot, you’re basically telling
the system, hey no longer write this to the original
abstracted hard drive, create a new hard drive. The original abstracted
hard drive remains static, and you’re writing
only the new we’ll say delta.vmdk representation
of the hard drive. Now we have another
file here that is absolutely worth
mentioning, that would be the SANS-***.vmss; this is
actually a snapshot of memory that’s created when you
suspend the virtual machine. I really like imaging
suspended virtual machines, I get a static disk that
I can in fact image, hence my hashers
are gonna match, and on top of that, I
actually get a complete image of RAM at the moment
I suspended that disk, I’m really big today on
actually performing a full forensic analysis of RAM. We’re seeing time
and time again today where the bad guys are
inserting malware into a running process such as a DLL. Now if you follow the
traditional IR mantra, when you have a compromised
machine you pull the plug and you image the hard
drive, you may have in fact wiped out all of
your evidence, again, even using metasploit,
I can insert my malware into a running process. If you’re not capturing
an image of RAM, you’re not capturing
your evidence. So I think again, it’s
really really cool that by suspending the
virtual machine I can also get the copy of RAM,
and I gotta tell you, I can do the same thing
with the snapshot, but it’s an option, you
have to turn on that option as you’re creating
that snapshot, but it absolutely the
way to go, you gotta get that copy of RAM, you gotta
look at what processes were running and in fact,
you may find that you had malware existing within
a running process in RAM, and without that image of ram,
you would completely miss it. So let’s image the VMDK, well, most of the old-school
forensics guys out there are used to
using things like DD, everybody should remember
DD here on the event. Again, with DD you’re
making a bit by bit copy of an entire hard disk, and we’ve all got used to
its use over the years, DD was the only way
to really perform a solid, sound forensic
copy job of a hard disk, you had to make that
bit by bit copy. Again, today we’re
talking about everything being abstracted within
a VMDK container, so all of the allocated
and unallocated space is available there. So again, is it really
necessary to use DD? In fact, I say it is not, again, I can run an MD5 against
the original VMDK, and then simply copy
it out using any number of different tools, such
as FastSCT from Veeam, an SCT on the command line,
and even within VMware, I can write out that
VMDK using the GUI to a a new destination and
then I simply run an MD5 or a SHA-1 for that
matter, against the copy. Bottom line is, if the MD5
or the SHA-1 of the original matches the MD5 of the copy, we have a forensically
sound copy. Now, I do give examples
in the 579 class, we do let the old-timers
like myself run DD to make their copy, but again,
it really is unnecessary. We have an abstraction
of a hard disk within the VMDK, so
again, simply making a forensically sound copy
of that VMDK folder itself is a forensically sound method, again, it’s a method
I recommend myself. You can use DD if like I
said, you’re an old-timer and you’re stuck on it, we
do give examples of it, but again, it’s become a
bit too time-consuming. Again, if you’re using the tool
such as FASTSCP from Veeam, which we give a copy
of within our course, it essentially is well
over 10 times faster than running DD
over SSH et cetera, so absolutely a
time-saver, it can make the difference in
hours when it comes to imaging an individual
VMDK versus DD. Now, let’s look at some
VMDK state considerations from a forensics perspective. You know, a VMDK can
be copied in any one of the following states. Well, it could be simply
be running, turned on, but that’s really useless
from a forensic perspective, because you’re never
going to be able to get a hash that matches. Again, if the machine is on, it’s gonna be constantly
changing that hard disk, so you’re never gonna
be able to verify that in fact, the copy that you
made is forensically sound, because when you create
the original hash and then attempt to copy it out, it’s constantly changing. When you make your final
hash and compare the two, it’s simply not going
to match, excuse me. We could simply turn it off, that’s been what’s
been traditional in the forensic imaging in
the past, however, if you’re turning it
off, you are in fact impacting production. Many organizations
simply don’t want you taking their system
offline, so we have to find an alternative
to turning it off. Well, we could suspend
the virtual machine, now again, as I noted
earlier, you also get a copy of what was memory,
and that was really a bonus for you, and of
course, you could snapshot the virtual machine,
create the image of the now unchanging disk and
let the machine run along with the new file
that it created. So again, referring
to the snapshot, it really is nice as
the VMDK is no longer actually changing, it’s
writing data to that new hard drive that it did
create, and you’re also gonna get that image
of RAM when you created the snapshot
of that moment in time. So for myself, a vast
majority of IR and forensics I do, it
would be against a snap shotted virtual
machine, and truly is the best way in my opinion, again, you’re able to get that
forensically sound copy, the hard drive itself
is no longer changing, so you’re able to
verify the hash, plus you’re gonna get
a bonus of getting that RAM image at the time
the snapshot was created, so it really is the
best of all worlds, truly is today, so
again, in my practice, and the vast majority of
time we’re performing IR in the virtual realm,
we’re performing that incidence response
against a snap shotted VM. Now some consideration
regarding snapshots, again, snapshots are a very
useful tool for creating an image without
having to shut down, but there are some
drawbacks to snapshots. The problem with
snapshots, simply put, is people tend to
get snapshot happy; it’s not uncommon to
find instances where a given virtual machine may
have dozens of snapshots, and that makes life very
difficult in performing a full analysis,
because effectively you have to revert each
individual snapshot, create an image, and
then perform an analysis and then move on to
the next snapshot. If you simply reverted
all snapshots all at once, you could in fact be
overriding evidence within that virtual machine. Now when you create a snapshot, you’re basically
telling VMware, okay, I’m going to create a
new disk for this VM, so from now on, you only
write to this new disk, and no longer write
to the original disk, that’s the beauty
of it, and again, after you snapshot,
you create your actual image of the original
disk, you’re not creating it of the new disk,
you’re letting that new disk go ahead and
accept the changes. So again, you’re static
on the original disk, so it’s going to be
forensically sound. Now, this is great
if you’re only going to create a single
snapshot, but gets very complex again with
multiple snapshots. There is not tool
available today that can fully analyze across
multiple snapshots, so again, the workaround
is to restore and analyze each one sequentially,
that is a tremendous amount of work, I tell my
clients, take my proposal for this incident response
and simply multiply the overall cost times
the number of snapshots, it truly is a full multiple. Again, you’re gonna
have to restore each individual snapshot and
perform a full analysis, that gets very very expensive. Now, snapshots are
in fact an asset when they’re properly managed,
the graphic shown here would depict a good
use of a snapshot. You’ll note we’re
running Windows XP, we had our snapshot
that was created, we have our Day One
snapshot as well, very easy to perform an analysis on this given implementation. Now let’s contrast
that with something a little bit different. This is a snapshot
happy configuration, this is simply way
too many snapshots, I mean look at this graphic,
where would you even begin in performing a thorough
forensic analysis in an environment where you
have not only a high count of snapshots, but so
many branched off? Now again, if you simply
came in and restored everything back
to a single image, you would absolutely be
potentially overriding evidence. The only way to approach
something this bad would be to simply revert
snapshots individually, create new images, and
perform a full analysis on that snap
shotted disk itself. Again, it could become
very very costly, we really have to reign
in the number of snapshots that people are creating. Again, I refer to the term
as being “snapshot-happy”, you simply keep creating
snapshots at will. There’s really, in my
opinion, very little need to have more than one,
or perhaps two snapshots within any environment. Again, in the graphic
that’s show here, that would be an
absolute nightmare, that would be the type
of IR or forensics job I’d happily let my
competition have. Now, more on snapshot
considerations, again, you have to
consider that when you consolidate your snapshots, you’re effectively
sequentially applying each snapshot and the respective
changes contained within that snapshot
to the original VMDK. You have to do it in
the right sequence, and again, you have
to actually image the VMDK as you are in fact
restoring individual snapshots or you do pose yourself
some risks in potentially overriding evidence by
actually restoring more than one snapshot
without creating that new image of the drive itself. Now, valuable evidence
could of course in fact, potentially
be found literally be found in between
snapshot events, therefore of course
it’s important to analyze the VDMK after
each individual snapshot is restored, and again,
that’s where the real labor comes into play here. Again, if you didn’t have
a clue as to how many snapshots existed before
you submitted your proposal, you could find yourself
absolutely underwater with respect to that
IR or forensics job you had put that bid in. Be very, very careful
that you understand how many snapshots you’re
gonna have to deal with. As I said, you’re gonna
have to restore each one individually, create a
new image of the VMDK and perform a full
analysis with respect to each individual snapshot,
so be very very careful there. Now more on snapshot
considerations, in my opinion, a thorough
forensic analysis of course would require
that the original VMDK and each individual
restored snapshot should be analyzed,
that’s a major time sink. Well, in products such as
Shadow Analyser for Windows, it does have this
really cool capability of being able to analyze
across shadow files in a Windows environment. This is the kind of
tool we absolutely need within a virtual realm,
but simply does not exist today. I’ve reached out to the
authors of Shadow Analyser, and I’ve expressed my
opinion that we really need this capability being
brought into snapshots within the virtual realm. Hopefully they’ll get
started on a product that can meet that need
in the very near future. Now again, all forensic
product vendors need to take note that we
need something like Shadow Analyser to
reduce the burden of performing the forensic
analysis an environment where we have
multiple snapshots. Now we’re gonna jump
into a bit of a mini lab, you’re not gonna be able
to get any real hands-on with this, but I’ll walk
you through it myself. So here’s our situation: the client needs a
forensically sound copy of a potentially
compromised virtual machine. The client can provide
access to their vCenter admin console, and can
provide local console access, bu the client did note
that the ESXi console remotely is currently
not enabled, so we’re not gonna go in through SSH
directly, we’ll have to turn a few things on
to be able to perform our IR in this particular case. So VM itself is a
database server running on top of Linux, files
are stored locally on hardware, and there
are currently no snapshots of the virtual machine itself. You know, the client
again, has no issue with a reasonable
amount of downtime, after normal working
hours, so again, it might be in our
best interest to work either from a snapshot
or actually suspending the VM for a moment
to get the image to prevent having to work
after normal working hours. So our proposed
process, we’re going to first enable the
remote console in ESXi, we’re then going to
simply quickly suspend the VM using the vCenter
Client, we’re gonna navigate to the storage medium
using the remote console, we’re going to hash
the respective VMDK and the memory
snapshot, the VMSS. We’re gonna copy the
VDMK and the VMSS to a removable
temporary NAS device, using DD just to
keep it within the scope for the old-timers
like me out there, we’re gonna hash the
copies, and we’re then gonna verify the hashes actually
match the original. If they in fact do
match, we do have a forensically sound
copy, and we will then resume the
virtual machine and get that client right
back into production. So here we simply go in
through the ESXi console locally on the machine,
and we’re going to find the support
mode has in fact been disabled, so we’re going
to have to enable that. We go back into the GUI, one problem we have
today within the ESXi is you can get to the
GUI, but no root password is typically required,
it comes by default with no root password,
so the first thing we have to do is
set up a password so that we can
come into the SSH. Once that’s taken care of, we simply go in and we enable
local technical support mode. It’s done here as
shown on the GUI, you would simply select
Enable Remote Tech Support over SSH. Again, we’re able to then get to our underlying console,
again, with ESXi, they’re running a
busy box console with ESX, where we’re
used to working within a Red Hat enabled console. Now the key there
is both of them do provide the tools
you need to perform ESX, I should say,
to perform IR, so again, with ESX
and ESXi you will find some ends such as MD5 some NDD, the basic tools that
you need to perform an incident response. Now here we’re
simply connecting up a NAS box, what I’ve
done here is again, using the ESX CFG-NAS command, I’ve established a connection
to a remote NAS box and I was able to
hang on the wire at IP address 9.16.11.134. Now as I’ve connected a NAS up, I have a great destination
to store my image in of that virtual machine. So here we’re back in
virtual center itself, and I’m able to simply
right-click on the individual virtual machine
and suspend that VM. With the virtual
machine suspended, I go back to my console
and I simply seek out the location of my files. So again, I first pinged
my storage device to make sure I had connectivity,
I then navigated down through the VMFS
file system to Volumes, within Volumes it listed
out the virtual machines that are actually
available to me, and I was able to go
then down and change directories to the individual
VMDK folder that is associated with the virtual
machine that we want to image. Now once I know where
my virtual machine is, I change to that directory,
you’ll note it’s a rather long string
leading to the directory, but within the directory
here we have our files. We have our Linux
Red Hat 6.2 VMDK, our VMSD, our VMX file,
et cetera, et cetera, all the pertinent
files with respect to that virtual machine
located within the VMDK folder, which
again, is a container for the abstracted
virtual machine. So now we have some
command line kung-fu, we have to actually
DD out a copy of the respective files that are
pertinent to our investigation. In this slide we’re
showing the DD command, the IF of course equals
would be the input file, so we’re listing
the full path to the flat.vmdk file, that
of course will be the actual binary file
that represents the abstraction of the hard disk, our output file is noted
here in the command line, is essentially pointing to
the NAS that we connected, so again, keep this,
this is kind of a reference for yourself
in using DV to copy out an individual file. In this slide we’re
simply showing the same thing, but this time
we’re actually copying out the .VMSS file, this
would be the actual image of RAM that was
created at the exact moment that we suspended
that virtual machine. Again, a great
reference for you here on the command line,
and again, DD is in fact included both in the
ESX and the ESXi. So again, makes kind
of a trivial way to copy out a file, you
know you’re getting that bit by bit copy
but again, there are easier and faster ways to do it. Again, I included DD in here, just for the old
timers like myself, so that you would
have something shown that is something
somewhat familiar to how you’ve done
it in the past. Now here we’re actually
then creating out a copy of the
files on our NAS to an alternate store
so we can then work with the files. So again, you can simply use the copy command to move
the files around just as well as you
could a DD command. Now here we simply
show the actual files that I copied out to my
network storage device, I simply connected to it
in a Windows environment, so you can see that in
fact we got our VMDK and we got our
suspended RAM image, and I also created MD5 hashes
of the files themselves. Now, here we’re simply
comparing the MD5s from before the copy
and after the copy, you’ll note that the
MD5s actually match, so in any court, in
any court in the land, I should say, as long as
the MD5 of the original file matches the
MD5 of the copy, you do have a
forensically sound copy. Now, we go through about
four different methodologies in doing this in the 579 class, we of course use DD, we also use FAST SCP and we
use SCP over SSH, we go through the
process of working with a suspended virtual
machine, and snapshot of virtual machine, as well as
an offline virtual machine, just to give users the
experience of doing it in the common ways
that they’re going to have to out there in the wild. Again, we try to
cover each one of them within the class to
give you a little bit more hands-on
experience, as again, we want you to leave the
class with the ability to handle this in
the real world. So in summary here,
virtualization changes many things, and it
of course does change how we actually respond
for both IR and forensics. You know, imaging the
entire VMFS is not always an option, and
is simply not necessary to analyze a specific
virtual machine. I find it somewhat laughable
that here in the USA we’re constantly
reading stories about law enforcement that
was called in because an individual virtual
machine may have been serving up malware, and
they actually take down the entire shared storage
array, impacting hundreds, if not thousands of
virtual machines for days on end. We had one care recently
where law enforcement actually came in and
physically seized the shared storage
server so they could take it back to
their lab environment to image it. They were literally down
for well over a week, there is absolutely
no need to do that, you can easily carve out
the VMDK associated with and individual virtual machine, and you can have no
impact on the production, not only of that
virtual machine, but all the other virtual machines
that might be running on the shared storage that
is associated with it. So continuing on here,
snapshots can of course both be a friend or a foe. Again, if you walk into
a environment where they manage snapshots
correctly, you’re typically never gonna have
more than one or two. It’s where snapshots
get out of control, as I said, the user
gets snapshot happy, that it can create
some serious issues. Again, from a time
perspective, take the time that you would
allocate it for both the imaging and the analysis
and simply multiply times the number of snapshots that
are made on that machine, it truly is a labor
intensive process to do it right. Again, you have to literally
restore individual snapshots, create new images, and
then fully analyze that image or you risk in fact, overriding the
potential evidence. Now, the VMDK is a
complete abstraction of a server and can be
imaged in various states. For myself, I like
suspending the VM for imaging, you
get RAM as well, and of course, we
can’t forget that you that you can also
snapshot that VM, you’re gonna get
the static drive as well as the RAM itself. Now, my recommendation
for you is you really need to get
some of that priceless hands-on experience with
VMware on a lab machine before ever trying this in
a custodian environment. You can run workstation
on pretty much any Windows machine today,
and on top of Workstation, you can load up a
copy of ESXi and then put multiple
virtual machines on top of ESXi and you can do
that all on your laptop, again, you don’t want
the first time you’re having to deal with a
virtual machine IR response to be in a running environment, you really want to
get that hands-on well beforehand, so again, my recommendation is
using a Windows PC, run workstation,
load a copy of ESXi, and multiple virtual
machines and work at copying out individual VMDKs. For you MAC users
out there, you can do the very same thing with Fusion. Again, in our classes
we typically have the students running
a Windows environment, but we always have a
student bring in a Mac, and as long as you’ve
got at least eight gig of RAM, that’s
typically not an issue. Be aware that Fusion
does not have the network configuration
capability of workstation, so it’s a little bit
more work for the student to handle that. I see a question here,
“How are swap spaces “on VMs managed?” Well, that’s a
very good question. Bottom line is that within ESXi, you have a swap file
associated with it. Within the virtual
machine you have a separate swap file
that’s associated with the individual
virtual machine. Now the actual swap
space associated with an individual
virtual machine is a component of the VMDK. Nothing associated
with an individual running virtual machine
is stored outside of the VMDK, so again,
the evidence that you seek is generally
always found within the individual
VMDK, most all data associated with that
virtual machine is written within the
VMDK folder itself, and we typically do
not have issues with trying to grab the
underlying copy of ESXi or ESX swap file off of
the machine itself, our investigation is focusing on an individual virtual machine. Now mind you, if your
investigation is broader in scope, you’re not
focusing on an individual virtual machine, it’s
a major change in what we’re talking about here. Now you’re gonna have to
get copies of pretty much everything, you’re gonna
need to get a copy of the Oracle or MySQL database
associated of course, with vCenter, you’re
gonna have to get copy of the respective swap
space for the vertical machine that’s running
within ESX or ESXi, and you’re gonna have to
grab copies of any templates that may associated
with the running virtual machine that might be
just sitting out there on shared storage, that’s a
totally different scenario. Again, I thought I’d
keep it kind of simple for you guys in this, and
here in the analysis that we just walked through,
we were targeting an individual virtual
machine that may in fact have been compromised, so
we were limiting the scope to an individual
virtual machine. It truly does expand the
scope, when you’re getting outside of an individual
virtual machine. All right, and I’ll
get back on topic here, so that pretty much wraps it
up for today’s presentation, wanted to call a
couple of events into the session here real quick, let you know what’s going on. We of course do have
SOS Singapore 2012, that’s the 15th through
the 20th of October, a number of great courses,
we have SANS Security 401, a great course written
by Dr. Eric Cole, I’ve taught that a
number of times myself, we have Security 503,
Intrusion Detection In-Depth, Security 560, Network Pen
Testing and Ethical Hacking, and we have yours
truly, I’ll be teaching Security 579, six days
of intensive hands-on with respect to virtualization
and private Cloud security, offense and defense. And of course, we
have Forensics 508, the new version, it’s Advanced
Computer Forensic Analysis and Incident Response,
truly is a very cool course; its’ been recently
upgraded to include a full APT, I teach that
myself, love working through the lab on that. We also have SANS
Bangalore 2012, 29th of October through
the 3rd of November, there you’ll find
again, Security 401, SANS Security Essentials
bootcamp-style, Security 542, Web App Pen
Testing and Ethical Hacking, and of course, Security 560,
Network Penetration Testing and Ethical Hacking. So that wraps those up for
getting Suresh’s information out there, and the
two events upcoming in Southeast Asia. That does wrap it
up for me today, I trust you found this
presentation both interesting and informative, I
thank everyone for coming along, if you
have questions I will hang out for a little bit,
again, happy to answer them. If you prefer, you
can reach out to me, at [email protected], I do actually answer
emails, so feel free to reach out, be
happy to chat with you about any specific IR
or forensic questions you may have within
the virtual realm. Again, I thank
everyone for attending, and we really are
good to go here, so again, I’ll hang
out for a minute now. Oh, did have another question, “Does 508 have
prerequisite as 579?” Actually, I recommend
people take 408 to get their essentials covered
before taking 508, 579 could really be
considered to be a bolt-on to either 408 or 508. In 408 you’re primarily
focusing on Windows; in 508 you’re primarily
focusing on Linux, and they do cover the
essentials in both. 579 we’re really
focusing only on the virtualization
environment, so again, it’s not really a
prerequisite, but it is somewhat recommended
that you have the essentials covered first. Any other questions
out there, folks? Well actually, with
respect to ESXi as a mandated standard,
it’s not really a mandated standard, however, ESX is going to be end of life, ESX is going away,
VMware is adamantly pushing people to ESXi. If you’re still running ESX, the writing is already
on the wall that it will be an unsupported
product from VMware. Again, they took away
that free copy of ESX, it’s no longer free, eventually you will
find that perhaps ESX will no longer work
within vCenter long term. Again, you really do
need to migrate to ESXi in any environment. I mean, again, they’re
end of lifing ESX. Again, I’ll hang out
here for a few minutes, if anybody has questions. Yeah, I hear that
on the “get well”. Bit of a cold here,
that’s what I get for traveling to Denver
at 97 degrees Fahrenheit, and then traveling to Canada,
30 degrees in the rain. That’s dedication
to the cause, folks. Sure, go ahead and
ask your question. Shiv, it looks like
you might be having problems with the GUI there, and formulating your question. Feel free to email me,
at [email protected], and I’ll be happy to answer
your question regarding security levels, again, Shiv, I did not see your full question come through, so I
really can’t answer it. So again, reach
out to me on email at [email protected]
and we’ll be good. Okay Suresh, time for me
to sign off here, friend. All right, thank
you very much folks, I’m gonna go ahead
and sign off for now, I enjoyed it, hope
you did as well.

Robin Kshlerin

RELATED ARTICLES
LEAVE A COMMENT