Aug 282014
 

Hope this saves someone some time:  RDP was not listening on TCP 3389 on a new server build.  Followed the standard troubleshooting (verify registry, re-create RDP listener, etc). When showing hidden devices in device manager, tdtcp (tdtcp.sys) was not enabled.  NIC is HP373i (Broadcom), but the tdtcp driver would not load until after I installed the HP Network Configuration Utility (NCU)  + reboot.  Tip your waitress.

Aug 262014
 

Merely as a thought experiment (I like SCOM and Nagios both), if one were to implement Nagios [Icinga, et al.] to cover the same functionality as SCOM, what would be the quickest and surest way?

SCOM and Nagios both do nothing — they’re just frameworks for collecting data and figuring out if it meets criteria to alert or report on.  SCOM without Management Packs is like Nagios without plugins: useless.  The most important part of monitoring is knowing what you want to monitor, but the documentation for hardware and software aren’t generally very specific.  “Monitor the event logs for stuff with Red by them” isn’t useful.  SNMP MIBs without any corresponding declaration of ‘normal’ is a start, but it’s not good.  It’s time-consuming to start from scratch (even from “Best Practices” whitepapers) and try to figure out what to monitor, what’s normal, and what the severity is when things aren’t normal.

The best documentation for monitoring Microsoft products (or anything with a SCOM management pack), is the management pack.  Load up a SCOM console (eval copy works fine), download the free MPViewer utility, download the management pack(s) you need, install them, then open MPViewer and view the management pack contents.  Discovery is much more difficult with Nagios (an exercise for the reader), but if you’re looking for monitors and rules for 90% of use cases (eventlog id’s, service status, perfmon counters, network status), it’s pretty trivial to convert to NSClient++/NSCP checks.   Someone savvy enough could probably automate a great deal of it, though I bet Microsoft’s lawyers might object….

 

 

Heartbleed

 Computers, Main  Comments Off
Apr 132014
 

My WordPress installation sent me an email this morning, saying I was the only person with a blog who hadn’t written about heartbleed, and I only have one thing to add, really.

Heartbleed describes a bug in OpenSSL, most succinctly explained by the xkcd: Heartbleed Explanation comic.  It’s a routine programming bug, but the incomprehensible aspect is that the OpenSSL authors actively worked around system malloc’s and rolled their own: to make you vulnerable that much faster (“exploit mitigation countermeasures”).  Other bugs that have been logged for years are presumably being reviewed.  Nothing new to add there.

But I say that critical infrastructure exploits should all have better names, though ‘heartbleed’ isn’t bad — a heartbeat function that bleeds information.  But instead of rolling your own, or going by generic CVE entries, we should pre-allocate alphabetical names, like they do for hurricanes.  Except I’d name them after waitresses, not just generic female names.  So, instead of ‘heartbleed’ we’d start with ‘Amanda.’

  • Amanda (one of my favorite waitresses)
  • Betty (I don’t know a waitress named Betty, but I imagine she’d be really good).
  • Chelsea (who isn’t a waitress any more; she’s a parole officer).
  • Dotty (Like Betty, I don’t know a waitress named Dotty.  I bet Dotty’s a good waitress, but not as good as Betty).
  • and so on.

Everyone could have action plans and themed PowerPoint templates ready.  Next critical infrastructure bug?  Pull out the “Torie” slides, and Bob’s your uncle.  That’s all I have to say.  Thanks.

Mar 072014
 

SCCM 2007 report to show patch status details per Update List and Collection.  You can get here by drilling down 4 reports deep, individually, for hundreds or thousands of your servers, or you can just run this, throw it into Excel, and filter and munge at will.  Much faster.


-- Shows all patch status details, given an Update List and a Collection. Export it to Excel and monkey with it there.
-- Based off of the stock "Compliance 1 - Overall Compliance" report.

DECLARE @AuthListLocalID AS INT
SELECT @AuthListLocalID=CI_ID
 FROM v_AuthListInfo
 WHERE CI_UniqueID=@AuthListID

SELECT
 fcm.Name,
 ps.UpdateID,
 ps.ID,
 ps.Title,
 ps.QNumbers,
 ps.LastStatusMessageIDName,
 ps.LastStateName,
 ps.AgentInstallDate,
 v_UpdateInfo.DatePosted As UpdateDateReleased,
 v_UpdateInfo.DateRevised AS UpdateDateRevised,
 v_UpdateInfo.InfoURL AS UpdateInfoURL,
 v_UpdateInfo.Description AS UpdateDescription

FROM v_UpdateInfo

 INNER JOIN v_GS_PatchStatusEx AS ps ON v_UpdateInfo.CI_UniqueID = ps.UniqueUpdateID
 INNER JOIN v_FullCollectionMembership AS fcm ON ps.ResourceID = fcm.ResourceID
 INNER JOIN v_CIRelation cir ON cir.ToCIID= v_UpdateInfo.CI_ID
 INNER JOIN (v_CICategories_All
 INNER JOIN v_CategoryInfo
 ON v_CICategories_All.CategoryInstance_UniqueID = v_CategoryInfo.CategoryInstance_UniqueID
 AND v_CategoryInfo.CategoryTypeName = 'Company')
 ON v_CICategories_All.CI_ID = v_UpdateInfo.CI_ID

WHERE fcm.CollectionID = @CollID
 AND ps.AgentInstallDate IS NULL --this shows errors only. Comment it out for reports on installed updates.
 AND cir.FromCIID = @AuthListLocalID
 AND cir.RelationType = 1

ORDER BY fcm.Name

-----------
-- Create two prompts, for Update List, and Collection
--
--
-- AuthListID
-- Update List ID (Required)
begin
if (@__filterwildcard = '')
 select distinct CI_UniqueID as AuthListID, Title as Title from v_AuthListInfo order by Title
else
 select distinct CI_UniqueID as AuthListID, Title as Title from v_AuthListInfo
 where ((CI_UniqueID like @__filterwildcard) or
 (Title like @__filterwildcard))
 order by Title
end
-------
-- CollID
-- Collection ID (Required)

begin
 if (@__filterwildcard = '')
 select CollectionID as CollectionID, Name as CollectionName from v_Collection order by Name
 else
 select CollectionID as CollectionID, Name as CollectionName from v_Collection
 WHERE CollectionID like @__filterwildcard or Name like @__filterwildcard
 order by Name
end
 ---

Mar 172013
 

UPDATE: Disk Cleanup Wizard addon lets users delete outdated Windows updates on Windows 7 SP1 or Windows Server 2008 R2 SP1 (KB2852386)

 

I picked up an HP t5740e thin client off eBay, as I had deployed some at a prior job.  Window Embedded Standard 7 (32-bit), with 2GB RAM and 4GB flash.  Set it up the way I want it, re-enable the write filter, and Bob’s your uncle.   But the default HP build includes components that take up a lot of space, and I have no need for them — namely, the text-to-speech components, the natural language components, and the SAT performance tests (sample movies).

While logged in as Administrator, with the write filter disabled:

dism /online /Get-Packages

You’ll get a list of all packages installed in the running image. Find the ones you want to delete. Then delete them. Reboot.

dism /online /Get-Packages

dism /online /Get-PackageInfo /packagename:WinEmb-Natural-Language~31bf3856ad364e35~x86~~6.1.7601.17514

dism /online /Remove-Package /PackageName:WinEmb-Accessibility~31bf3856ad364e35~x86~~6.1.7601.17514
dism /online /Remove-Package /PackageName:WinEmb-Natural-Language~31bf3856ad364e35~x86~~6.1.7601.17514
dism /online /Remove-Package /PackageName:WinEmb-Speech-LP-ENU~31bf3856ad364e35~x86~~6.1.7600.16385
dism /online /Remove-Package /PackageName:WinEmb-Speech~31bf3856ad364e35~x86~~6.1.7601.17514
dism /online /Remove-Package /PackageName:WinEmb-Diagnostics-Performance~31bf3856ad364e35~x86~~6.1.7601.17514
Mar 112012
 

DISCLAIMER – I DON’T DO C. And my Perl isn’t great, either.

I routinely browse the openbsd-cvs mailing list, and I saw this easy openbsd-cvs bug fix (“Fix a stupid bug in tcpdump print-bgp.c“) the other night when doing some really late-night, partial-involvement sysadmin work.  So I decided to pass the time (“stay awake”) by doing a regex exercise to find similar patterns in the OpenBSD source tree.

Continue reading »

Aug 222011
 

On UNIX, it’s simple to delete matching files in a directory, over a certain age (e.g., all .txt files over 2 days old):

find /some/directory -type f -ctime +2 -name \*.txt -exec rm -f '{}' \;

On Windows, it’s ridiculously complicated.  Here’s a PowerShell script, modified from somewhere….:

$a = Get-ChildItem 'C:\Temp\subdir\*' -include *.txt
if ($a.count -gt 1) {
    foreach($x in $a) {
        $y = ((Get-Date) - $x.CreationTime).Days
        if ($y -gt 7 -and $x.PsISContainer -ne $True) {
            #$x.Delete()  #uncomment here to delete
            Write-Host $x
        }
    }
}

 

May 082011
 

File:  squid.zip

Miscellany.  Overview diagram in Visio and PNG.  adzapper postmatch.  squid examples to redirect based on source IP.  squid tee.  Wiki squid/adzapper template.  Wiki template for Java client config file (you’d be surprised how many people don’t know about it…).

Mar 112011
 

Here’s the quickest way I’ve found to produce good diagrams for certain types of technical diagrams.  You can export them to SVG or EPS and use them in most anything.

Protocol Diagrams (packet diagrams): use the LaTeX bytefield package.  It’s easy.

Sequence Diagrams: use Mscgen.  Easier than Graphviz on this.

Graphs (general): use Graphviz. And the Perl module generally makes life easier.

State Diagrams: use Graphviz, or Tikz.  Tikz is intimidating, but state diagrams are simple enough.

General drawing: try Inkscape, and don’t forget OpenClipArt.org.

 

 

My Password

 Computers, Main  Comments Off
Feb 252011
 

It’s terrible practice, I know, but my password is the name of my dog.  My dog’s name is H^7a(;tQ.

 

UPDATE:  I’m being notified by various systems that my dog’s name doesn’t meet the minimum name length, so I have renamed my dog to Y05!z[2@,*HUps%3.  If it weren’t for the number and special character requirements, I had planned to just name him the Diceware passphrase ‘doublespadeambiguouslactationconsultant’.

Feb 062011
 

Here’s how to use Perl and libpcap to process a capture file offline.  This particular example parses RDP cookies….

# ./watch_rdp_cookies.pl session.cap
192.168.0.93:46890:192.168.0.88:3389:mstshash=corp\jaso
192.168.0.93:46890:192.168.0.88:3389:0
192.168.0.93:49469:192.168.0.89:3389:msts=1493215424.15629.0000(192.168.0.89:3389)

Continue reading »

RDP Cookies

 Computers, Main  Comments Off
Feb 062011
 

Overview

Windows terminal server (remote desktop session host) environments needing load balancing and high availability may use 3rd-party load balancers to distribute initial connections and persist further network connections. Load balancers utilize RDP cookies — short text strings sent by RDP clients in initial connection requests.

Native Windows functionality may be sufficient for some needs, say by creating a terminal server farm and using round-robin DNS to distribute initial connections (“Poor Man’s Load Balancing”).  For example, given three terminal servers (TERM{1,2,3}) with IP addresses 10.10.10.{1,2,3}, the administrator would publish a DNS A record (e.g., “TERMFARM”) with the multiple IP addresses of each terminal server.  The client requests the address for “TERMFARM” is [somewhat] randomly given one of the IP addresses and connects to that server.  Of course, when the client disconnects and tries to connect again, a different IP address may be returned, and the client would log in at that server, causing two active sessions to be open.  This is very unproductive for users, and it’s a waste of resources.  Microsoft Session Broker (Connection Broker) can be used to keep track of open sessions, and subsequent reconnects redirect the user back to the server hosting their existing session (“revectoring”).  This generally works fine. There are a few downfalls: 1) the initial connection may fail if the randomly-returned server is down, and the client will wait 30 seconds before trying the next, 2) Windows Server 2003 must be the more expensive Enterprise Edition to use Session Broker, 4) Windows 2003′s Session Broker doesn’t do any actual load balancing, and 5) Windows Server 2008′s Connection Broker is better (load balances solely on session count), but it incorporates no other intelligence.  Additionally, RDP clients have to have direct IP connectivity with the terminal servers, which may not be desirable in some cases.

As an alternative, consider putting a couple of UNIX servers out in front, running HAProxy on a shared IP address, by using keepalived or similar.  RDP clients would connect to a DNS name pointing to an IP address.  keepalived manages the failover IP address so that if the primary load balancer fails, the second one picks up the traffic.  The HAProxy load balancing software understands RDP cookies and can load balance connection requests.  Savvy admins and value-added companies can add more intelligence to the load balancing.  You could weight some servers higher than others, or include memory and CPU load, rather than using a sessions-per-host metric.  Also, using a 3rd-party load balancer can hide the servers, simplifying network and firewall administration effort.

Connection Request Protocol Details

Remote Desktop Protocol (RDP) clients send a small text string – a cookie or routing token– when starting a new connection. RDP is implemented through the ITU X.224 protocol, where commands are encapsulated in Protocol Data Units (PDU). RDP connection requests use X.224 Connection Request PDU.

ITU “T.123 : Network-specific data protocol stacks for multimedia conferencing”: http://www.itu.int/rec/T-REC-T.123/en

ITU “X.224 : Information technology – Open Systems Interconnection – Protocol for providing the connection-mode transport service”: http://www.itu.int/rec/T-REC-X.224/en

Microsoft “2.2.1.1 Client X.224 Connection Request PDU” : http://msdn.microsoft.com/en-us/library/cc240470

A X.224 Connection Request PDU has the following layout:

Cookies

Cookies are simple text strings and have nothing to do with authentication or any type of negotiation — they are just a string of text set by the RDP client. There are two types of cookies, and they’re mutually exclusive – a connection request can only contain one type of cookie. They are:

  • User cookie (”mstshash”): ANSI text string that almost always contains a username.
  • IP cookie (”msts”): numeric string that encodes the IP address and port number to route the connection to.

User Cookies (mstshash:)

A user cookie is intended to map a user to a specific terminal server, even if the user disconnects and reconnects from a different client and IP address.

The format of the user cookie is:

Cookie:[space]mstshash=[ANSI string][0x0d0a]
10:57:26.321918 192.168.0.201.33520 > 192.168.0.93.3389:
P 2000684428:2000684480(52) ack 3864383132 win 5840
<nop,nop,timestamp 2842032495848714> (DF)
0000: 4500 0068 53a2 4000 4006 6477 c0a8 00c9 E..hS¢@.@.dw(.?
0010: c0a8 005d 82f0 0d3d 7740 058c e655 ce9c (.].?.=w@..?U?.
0020: 8018 16d0 3908 0000 0101 080a 0004 562b ...?9.........V+
0030: 94c3 a10a 0300 0034 2fe0 0000 0000 0043 .á....4/?.....C
0040: 6f6f 6b69 653a 206d 7374 7368 6173 683d ookie: mstshash=
0050: 6a61 736f 6e74 6573 7440 636f 7270 0d0a jasontest@corp..
0060: 0100 0800 0000 0000 ........

The cookie is then “mstshash=jasontest@corp”.

A load balancer will keep a table listing the cookie and target server IP address, using the cookie to balance the initial connection if no state already exists, or routing the connection to the appropriate server if an existing session does exist.

For example, USER1 has no existing session on any terminal server. USER1’s RDP client sends an RDP Connection Request to the load balancer (with cookie “mstshash=USER1”), and the load balancer sees that “mstshash=USER1” has no existing state (no active sessions). The load balancer, depending on configuration, applies an algorithm to determine where the request should be sent. Most load balancers default to round-robin, but different implementations may have more intelligent methods, like using terminal server agents to report back load (memory, CPU, etc.,) that the load balancer can use when routing initial connections. USER1’s connection request is sent to a terminal server.

Cookie Size and Interoperability

Different RDP client implementations impose different size limits on the user cookie:

  • The Windows RDP client (mstsc.exe) truncates the user cookie to 9 characters.
  • Newer versions of the Windows RDP client can use the LoadBalanceInfo property and set the cookie up to 110 characters.  More
  • rdesktop limits the cookie to 127 characters.
  • ProperJavaRDP and Elusiva’s “Open Source Java RDP” fork both truncate the user cookie to 9 characters (iso.java: “if(uname.length() > 9) uname = uname.substring(0,9);”)

This truncation causes certain issues.

The most well-known issue is a collision that occurs when cookies are truncated to 9 characters.  For instance, the default Microsoft client cookie for  DOMAIN\USER1 is “DOMAIN\US” (9 characters).  The default Microsoft client cookie for  DOMAIN\USER2 is also “DOMAIN\US” (9 characters).   The cookies are the same for both connection requests, and a load balancer would route them to the same server.  The consensus solution for this is to use the USER@DOMAIN login format.  So, USER1@DOMAIN would have cookie “USER1@DOM”, while USER2@DOMAIN would be “USER2@DOM”.

But note that the user can’t switch formats.  “CORP\Bob” connects with cookie “CORP\Bob” and disconnects.  He connects again, but uses “Bob@CORP” with cookie “Bob@CORP”.  These cookies are different, so a load balancer will treat the connections differently — he may end up with different sessions on different servers.

Another snafu occurs when switching between clients that truncate the cookie and those that don’t.  CORP\ROBERT logs in from a Windows XP RDC client and opens a session on TERM-01.  He heads out to the warehouse and logs in from a UNIX thin client using rdesktop.  But he ends up with a new session on TERM-02.  His first connection was with cookie “CORP\ROBE”, while his second connection used cookie “CORP\ROBERT”.

IP Cookies

IP-based routing tokens “msts=” are handed back by Windows Session Broker / Connection Broker when “Use IP Address Redirection” is disabled.

Cookie:[space]msts=[ip address].[port].[reserved][0x0d0a]
09:39:40.133269 192.168.0.93.34764 > 192.168.0.87.3389:
P 3072855348:3072855403(55) ack 1295660978 win 46
<nop,nop,timestamp 2836782526209545> (DF)
0000: 4500 006b d090 4000 4006 e7f7 c0a8 005d E..k?.@.@.??(.]
0010: c0a8 0057 87cc 0d3d b728 0d34 4d3a 37b2 (.W.?.=·(.4M:7²
0020: 8018 002e 8262 0000 0101 080a a915 ddbe .....b......©.?
0030: 0003 3289 0300 0037 32e0 0000 0000 0043 ..2....72?.....C
0040: 6f6f 6b69 653a 206d 7374 733d 3134 3933 ookie: msts=1493
0050: 3231 3534 3234 2e31 3536 3239 2e30 3030 215424.15629.000
0060: 300d 0a01 0008 0000 0000 00             0..........

Decode the IP address as follows:

  1. Convert decimal to hex:  1493215424 = 59 00 A8 C0.
  2. Reverse the bytes:  C0 A8 00 59.
  3. Convert back to decimal:  192 168 0 89 (192.168.0.89).

Decode the TCP port likewise:

  1. Convert decimal to hex:  15629 = 3D 0D.
  2. Reverse the bytes:  0D 3D.
  3. Convert back to decimal:  D3D = 3389.

The reserved section should always be “0000“.

Monitoring

I wrote a short Perl script to parse offline tcpdump packet dumps for RDP cookies:

http://www.snakelegs.org/2011/02/06/perl-and-libpcap-parsing-capture-files-offline-rdp-cookies/

tcpdump

tcpdump -s 65535 -X ’dst port 3389 and tcp[37] == 0xe0’

Wireshark

Wireshark’s RDP documentation

Filter:

x224.rdp_rt

 

Network Monitor

Description == “X224:Connection Request”

Make sure the Windows parser profile is loaded.

Connection Request Protocol Details
Remote Desktop Protocol (RDP) clients send a small text string – a cookie or routing
token– when starting a new connection. RDP is implemented through the ITU
X.224 protocol, where commands are encapsulated in Protocol Data Units (PDU).
RDP connection requests use X.224 Connection Request PDU.

http://msdn.microsoft.com/en-us/library/cc240470

http://207.46.16.248/en-us/library/dd644639

A X.224 Connection Request PDU has the following layout: