Securing Traffic between 2 or more servers (Domain Isolation) using IPSec without domain group policy

I was researching for a colleague of mine whether there is any documentation out there to secure 2 or more computers using IPSec(IP Security). Surprisingly, there isn’t any easy one. Those that are out there requires you to configure group policy at the domain level, and domain controller(from what I found). What if you are neither a Domain Admin or GP Admin?? Some forum even suggested that IPSec without group policy isn’t possible. This is true, but you can always configure the policy locally on each computer, without having to go through domain policy. If you are implementing IPSec on a large scale, of course, domain group policy would be the way to do it.

If you are not familiar with IPSec, this article from technet is probably the best one I can find.

So, just like the title of my blog, I’ll try to post things I can’t find Googling. Not only will I spell out the solution, I will explain what each of the step does so that you are not just clicking through dialog boxes. Bold fonts in a sentence indicate action you need to perform. Italic fonts indicate label.

Goal:

IPSec without Group Policy
IPSec without Domain Group Policy

You have a central server(web server, file server, database, etc) in your company, and you have a small number of workstation accessing the server(as pictured on the right). Let’s say you have 3 workstations, and you’d like to accomplish the following:

  • The traffic between the server and 3 stations needs to be secured.
  • ONLY those 3 stations  are able to access the server, all other stations are not permitted
  • In addition to limiting access only from specific stations, you’d also like to limit access to ONLY specific users
  • If domain user 1 uses computer W to access the protected server, the traffic will be denied because computer W is not on the ‘allowed computers list’
  • If domain user 4 uses computer X, Y or Z to access the protected server, the traffic will be denied because domain user 4 is not on the ‘allowed users list’
  • Domain user 2 can use computer X, Y, or Z to access the protected server.
  • Computer X, Y and Z can still be used to communicate with other servers as usual(can still be used for browsing and normal business tasks). This is optional. You will find this on step IV.17 below

If you are planning to deploy this solutions to hundreds of computers, of course Group Policy will make your life easier.

Short Solution and its shortcoming:

If it’s a web server, slap an SSL certificate on it and restrict the web port to allow only certain IP. That will secure traffic between all the workstations and the server. Shortcomings:

  • Need to buy SSL
  • Need static IPs on all workstations. Complicate to keep track when you need to buy new hardware
  • Can’t really restrict based on users

Long, complicated Solution, and its benefit

It’s long and complicated, but I’ll try to spell it out here with screen shot and explanation. Basically, you need to implement IPSec in Windows Firewall. Benefits are:

  • Don’t need to buy SSL certificate. IPSec is encrypted at the lower level of the OSI layer model.
  • Don’t need to maintain static IP of workstations
  • Can restrict based on users
  • Seamless to users. As long as users log in to their computers using their domain username and password, they will not be prompted for username and password. They won’t even know that the traffic is secured(it’s your job to tell them that it is secured with IPSec).

Requirements/assumptions

There are a lot of variables in this setup. I definitely can not list and explain every possible OS scenario. So, I’m going to pick the simplest one. This is what you need:

  • Computers/workstations and a server obviously(duuuh!)
  • You are in a domain environment that you or your users can log in to.You don’t need to be Group Policy admin or Domain admin.
  • Workstations running Windows 7
  • Server running Windows 2008 R2 with static IP
  • Both workstations and server are joined to the domain
  • Users log in to their stations using domain account. This is required only if you are restricting access to BOTH computers and users
  • Windows Firewall are enabled on workstations and server

Enough explanation. Let’s get down to it.

I.  Configure your Server Security Policy

1. launch Local Security Policy MMC by typing secpol.msc from start menu. Make sure your server is joined to your company’s domain. A security policy contains packet filter(s) that specify one or more network traffic going in or out of the server.

2. Right Click on IP Security Policies on Local Computer, and click Create IP Security Policy, and click next on the initial wizard screen

3. Name and describe your Policy. I usually name mine based on what project I’m securing. Let’s say the super secret I’m protecting is called: Salary History Data Policy. Description: To secure salary data only to person 1, person 2 and person 3, using computer 1, computer 2 and computer 3. Next. Please Note that I’m not actually storing anyone’s salary. This is merely an example.

4. On Request for secure communication window, I will leave the checkbox(Activate the default response rule) unchecked. Next

5. Leave Edit properties checked. Finish

6. You’ll be prompted with a property window of a policy you just created. Click Add to add a new security Rule and click Next on the welcome screen

New IPSec Rule
New IPSec Rule

7. We won’t be using a Tunnel Endpoint because the sever and workstations are in the same network. Choose This rule does not specify a tunnel. Next

8. On Network Type Window, you can choose to apply this policy to several network type. In this scenario, All network connections and Local area network(LAN) will yield the same result. I’ll go ahead and pick All Network Connections. Next

9. You are now on IP Filter List window.  This specify the source and destination IPs you’d like to be included in your protected environment(List of IPs that this rule will be enforced on). Click Add to add a new filter list. Give it a name. I’ll name mine Super Secret Employees Salary IP Filter List. Click Add again

Adding IP Filter List
Adding IP Filter List

10. Getting tired of all those dialog box?? Don’t worry, we still have a long way to go(well, in that case, DO worry then).

11. Click Next. You’ll be prompted for the description. You can skip this. Be kind to your brain, and provide some description. I’m sure you are not going to remember any of this a year from now :). Leave the Mirrored option Checked.

12. Leave the Source Address to Any IP Address and click Next. You might ask “Are you crazy?? I thought we are restricting access to only specific sets of workstations to be able to access this server??” We’ll cover that part later using Kerberos authentication.  I promise. You won’t have to worry about workstations IPs.

13. Change the destination address to My IP Address. Click Next.

14. On protocol type, select the protocol you’d like this rule to be enforced on. If you are protecting a web, file or database server, they are mostly using TCP protocol. Click Next.

15. In this example, I will be protecting a web server. So, I will choose to enforce this rule for TCP traffic coming from ANY port, going to port 80 on this server. Click Next, and Finish

IPSec Ports
IPSec Ports

16. You are back to IP Filter list window on step 9 above. Click OK.

17. Select the filter list you just created and click Next

IPSec Filter List
IPSec Filter List

18. Now that you have specified a filter(IP, protocol and port), what are you going to do with it when a traffic that match the filter criteria arrive on your server(permit, deny, or negotiate with it)? This is where Filter Action comes along. You can also think of this as “In order for other computers to access this server, their IP traffic has to be….(requirements you will specify later)”. On Filter Action window, click on Add. Make sure the Use Add Wizard is selected. Click Next on the Filter Action Wizard welcome screen.

19. I’ll name my action Super Secret Employee Salary Filter Action. I’m getting tired of typing description. I’m leaving it blank. Click Next.

20. On Filter Action General Options window, pick Negotiate security. Next.

21. Click Do not allow unsecured communication. Next

22. Click Integrity, and encryption to support encryption and authentication. Next, Finish

23. Select the filter action you just created and click Next

Filter Action
Filter Action

24. Pick Active Directory default(Kerveros V5 protocol). In order for this to work, your server needs to be joined to the domain. You can use certificate or shared key(not recommended) too, but I will cover only using AD for authentication. You don’t need to be domain admin. Next, Finish

25. Click Ok to close the dialog box. At this point you have created a security policy, BUT you haven’t enabled it yet.

26. To enable/use this policy, you need to Assign it. Right Click on the policy and click Assign

Assign IPSec Policy
Assign IPSec Policy

At this point you have created and enabled this policy. It’s time to do some testing!!

I’m enforcing this policy on a web traffic(TCP port 80). Assuming that my server is called protectedserver, with an IP of 192.168.1.200 and my company domain is called mycompany.com, with this policy applied, I should NOT be able to access my server using http://protectedserver.mycompany.com or http://192.168.1.200 from other location. However, I can still launch an internal browser on protectedserver and navigate to http://protectedserver.mycompany.com or http://192.168.1.200. This is okay. Traffic within protectedserver is not included in the filter.  If you Un-assign the policy by right clicking and click on Un-assign, now the http traffic should work again.

This means you have successfully restrict connection to the server. You need to test it accordingly of course. If you are restricting file share, then test if file shares work correctly after you assign the policy.

You are half way there. IPSec works together with Windows Firewall. So, the next step is to configure Windows Firewall.

II. Configure your Server Advance Firewall Rules

You need to create a connection security rule in your firewall. “What does it do? clean my car? It’s been too many steps”, one might ask(or complain). This is where you can request IPSec authentication based on criterias such as: users, computers or both. This governs when to request authentication and how it’s done(it’ll be clearer below). It doesn’t restrict users. You would need to configure that separately, which I will also explain.

1. Launch Windows Firewall with Advanced Security

2. Right click on Connection Security Rules and click New Rule

Firewall Security Rule
Firewall Security Rule

3. Select Custom, Next

4. Add the IP address of the server you are protecting to Enpoint 2. In my case, I’m adding 192.168.1.200 to Endpoint 2. Next

5. Pick Require authentication for inbound and outbound connections. This is the WHEN part I explained above. The verb Request means it’s not required. It will be requested, but not required. Click Next.

6. This is the HOW part I explained above. You can choose to authenticate based on Computers(any user can access my web server as long as they use computers I specified in my policy), Users(only a specific users can access my server regardless of which computers they use. It’s not displayed on this screen. I will explain more below), Computer and User(only specific sets of computers used by users I specify are allowed to access my server). I’ll go ahead and Pick Computer and user (kerberos v5). Any users you specify can use any computer you specify to access your servers. Click Next

7. Specify a protocol type and port. In my case, I’m using TCP protocol because I’m using http over port 80. Select All ports for Endpoint 1, and Port 80 for Endpoint 2. Click Next

8. You’ll have the option to apply it only to a specific network Profile. Obviously we need to enable this rule on at least one the Domain profile. You can apply them to all profiles(domain, private and public). Click Next

9. Name it again. I name mine Super Secret Employees Salary Firewall Connection Rule. Click Finish

You are almost done. Remember how I told you that you can choose to only allow certain users only, regardless of what computer they use?? You can do this by changing the properties of this Connection Security Rule. Now, your screen should look like the screenshot below. Make sure the Enabled column says Yes. :

Connection Security Rule
Connection Security Rule

The red-dotted column indicates that you are authenticating your users using BOTH their username and Computer. If you want to change this, right click on the connection rule you just created, click Properties, and click on the Authentication tab. You can change the authentication method to authenticate by User only. The User option is not available on the Wizard screen in step II.6 above. For the rest of this article, I’ll be using Computer and User authentication method.

At this point you did all the HOW and WHEN parts of authentication. Now you need to configure WHO or WHAT are you allowing to access this server. You will be using Windows Firewall with Advanced Security. It’s so much easier now that IPSec is integrated with Firewall.

III. Restricting server access using Windows Firewall

1. Launch Windows Firewall with Advanced Security on the server you are protecting

2. Click Inbound Rules

3. If the port you are securing is not listed under Local Port column, do the following. If it is, skip this step, and continue to step 4.

3.1 Click New Rule

New Firewall Rule
New Firewall Rule

3.2 Select Port, Next

3.3 Pick a Protocol(i.e TCP), and specify the incoming port number you are trying to secure. In this article I’m securing port 80. Next

3.4 Click Allow the connection if it is secure, and click customize.

3.5 If all you need is to make sure that network packets are authenticated and not modified between 2 points(NO encryption), pick Allow the connection if it is authenticated and integrity-protected.  If you need to add encryption on top of authentication and integrity protection between the 2 points, pick Require the connections to be encrypted. Click Ok and Next.

3.6 If you are using Computer and User or just User authentication method, check Only allow connections from these users, and click Add. Add DOMAIN users you want to allow to access this server. You can NOT use local users. It HAS TO BE domain users. Specify any exception if necessary. If you are authenticating using Computer only, you can skip this part, and click Next

3.7 If you are using Computer and User or just Computer authentication method, check Only allow connections from these computers, and click Add. Add DOMAIN computers you want to allow to access this server. The computers need to be previously joined to the domain prior to doing this. You do not need a DNS entry for these computers. if you joined all the computers to the domain, you should be able to just type the computer names here, separated by semi colon. Specify any exception if necessary. If you are authenticating using User only, you can skip this part, and click Next

3.8 Apply this rule to at least the Domain profile. I usually apply this to all Profiles. Click Next

3.9 Name your Inbound Rule. I named mine Super Secret Employee Salary Incoming Traffic. Click Finish. Make sure the incoming rule you created appears on the Inbound Rules List, and make sure it is Enabled. Do NOT continue to step 4. You are done configuring your server

4. Locate the entries where the local port matches the port you are trying to secure. In my case my existing entry is called World Wide Web Services (HTTP Traffic-In)

4.1 Right Click the entry and click properties.

4.2 Under the General tab, select Allow the connection if it is secure, and click Customize

4.3 If all you need is to make sure that network packets are authenticated and not modified between 2 points(NO encryption), pick Allow the connection if it is authenticated and integrity-protected. If you need to add encryption on top of authentication and integrity protection between the 2 points, pick Require the connections to be encrypted. Click Ok.

4.4 If you are using Computer and User authentication method, go to both the Computers and Users tab and add computers and users you’d like to allow to access this server. Users need to be domain users. It can NOT be local users. Computers need to be joined to the domain prior to being added to the computers tab. You do NOT need to add a DNS entry manually for these computer.  You can add exception accordingly to both tab. If you are authenticating users only, you can leave the Authorized Computer within the Computers tab blank. The same thing apply to Authorized Users within the Users tab. Click Ok after you are done configuring.

—————————————————–

Your Inbound rule should now be enabled, and look like this(note the lock-indicating it is secured):

Configured Inbound Firewall Rule
Configured Inbound Firewall Rule

You are now done configuring the hard part, your server. Now, you need to configure your client(workstations/computers that needs to access your server) so that it can securely talks with your server using IPSec.

IV. Configuring workstation/desktop computers

Go ahead and try accessing the protected server now. In my case, I’m going to launch my web browser and browse to http://http://192.168.1.200. You should NOT be able to access your server. Because you have protected it and it’s not recognizing your computer as one of the authorized users/computers. If you are still able to access the server you are missing one or more step. Make sure you did everything above correctly.

You now need to re-do step I.1- I.26 on all workstations you’d like to allow to access the protected server……. WAIT!! don’t shoot yourself just yet. This can be easily done. You need to do this on ALL workstations you’d like to allow access to your protected server.

1. Login to your protected server and launch Local Security Policy MMC by typing secpol.msc from start menu

2. Right Click IP Security Policies on Local Computer, All tasks, Export Policies, band save it to a location of your choice. Copy this exported file to a workstation that needs to access your secure server.

Exporting IPSec Policy

3. On the workstation, launch Local Security Policy MMC by typing secpol.msc from start menu

4. Import the policy you just exported by repeating step 2 above, but choose Import instead.

5. Enable this policy by right clicking it, and click Assign

6. Configure Connection Security Rules in Windows Firewall with Advanced Security so that traffic going to your protected server will use IPSec. Repeat step II.1 to II.9 above on all the workstations you’d like to allow to access the protected server.

7. If you are only setting up for authentication and integrity protection(step III.3.5 or step III4.3 above) WITHOUT encryption, you are done. You can skip step 8 and on. Otherwise, proceed to step 8.

8. Now you need to setup network traffic that originates from this workstation to be encrypted if they have destination IP of the protected server in their network packet. Launch Windows Firewall with Advanced Security to do this.

9. Click Outbound Rules and select New Rule…

10. Select Port. Next

11. Select Protocol and Port you are using to connect to your destination server. I’m using TCP and port 80. Next

12. Select Allow the connection if it is secure, click Customize.

13. Select Require the connections to be encrypted. Ok. Next

14. Click Next on Computers Window

15. Apply this rule to at least the Domain Profile. I apply mine to ALL Profile. Next

16. Name and Describe this outbound rule

17. This rule should now appear on your Outbound Rules list. Right click on the rule, click Properties, click Scope tab. Under Remote IP Address, add the IP of the protected server. This is important. If you skip this part workstation will only be able to communicate with the protected server but NOTHING ELSE.

18. That’s it. Now, Log off from your workstation and log back in. If things aren’t working, try restarting both the server and workstations. You shouldn’t need to.

V. Debugging/testing the secured connection

If things goes well, you should be able to connect to the protected server using a combination of user and computers you specified on step III above. If you are encrypting traffic, your quick mode within Windows Firewall with Advanced Security should look like:

IPSec-Quick-Mode-Encrypted
IPSec-Quick-Mode-Encrypted

If you are just making sure that the connection is authenticated and integrity-protected, but NOT encrypted (step III.3.5 or step III4.3 above), your Quick Mode should look like

IPSec-Quick-Mode-UNEncrypted
IPSec-Quick-Mode-UNEncrypted

If you can’t get an authorized connection to work. Look at your main mode. If there is an entry in your main mode, that means, IPSec connection is established, but the connection is refused due to security rule you specified(may be you require both computer and user authentication, but the client connection is only using computer authentication, or the user of the computer is not authorized). Make sure you do step I-IV right. This is a very long process. It can be shortened using command line, but that’s for another post.

Make sure you check the main-mode first. Main-mode always comes first before the quick mode.

After you successfully implement this, you might still be paranoid, and are still asking “How do I know for sure that the connection is really encrypted?? I’m not going to just trust what is reported. I want PROOF or I’LL SUE”(If this is really the case, you need to ask for “chill-pill” prescription). In this scenario(protecting http traffic), it’s simple. Just enable basic authentication, and run a network monitoring tools to monitor a network packet going in to your server from an authenticated station(or the other way around). Without encryption enabled, you will be able to see user’s user name and password in clear text in your network monitoring tools during the authentication process. You normally protect this with SSL. However, if you have IPSec encryption enabled, you can see that all the http traffic traveling in TCP protocol over port 80 is encapsulated(encrypted) in ESP protocol. Hence, no need for SSL. Here is a screen shot from Microsoft network Monitor:

Encrypted-Network-Packet-with-IPSec
Encrypted-Network-Packet-with-IPSec

That’s it!! As usual, if you see any mistake on this post, feel free to let me know.

4 thoughts on “Securing Traffic between 2 or more servers (Domain Isolation) using IPSec without domain group policy

  1. This is very complicated. You don’t need to make IPSEC rules and then make connection security rules. On Vista and above you can just use connection security rules. Or if you have mixed environment (XP, Win 7, 2003, 2012, linux etc.) you can use (well you don’t have another option in mixed env, because XP doesn’t have such thing like a con sec rules) just IPSEC rules and it will be sufficient to encrypt traffic.

  2. I’ve been through this a couple of times now and am having trouble getting it to work properly. I’m trying to use the user and computer method for a file share on port 445. I can get everything to communicate just fine, but when I try to test by removing a computer from the AD group that’s given access previously I can still connect to the share. I just tried restarting the server and workstation with no difference in the outcome. Any suggestions would be greatly appreciated.

Leave a Reply

Your email address will not be published. Required fields are marked *

*