ThinVNC - Digging into the code, Part I

Introduction

I found interesting the idea of a pure HTML5 Remote Desktop client for cross-browser cross-platform access to a PC. Despite that there are some AJAX VNC/RFB implementations, I thought in a simpler way to build a remote desktop solution using web standards and simple image processing.

This solution is built using AJAX, HTML5, JSON and simple JPEG/PNG images. The required server-side code is written in Delphi 2010.

The code - Part I

In this first part we'll see the code to do the screen capture. The standard aproach is to capture the whole desktop. However, in this case, we'll capture every window individually, applying clipping regions and saving the individual bitmap for later comparison and difference extracting.

Firstly we need to enumerate all visible top windows:

TWin = class(TObject)
private
  Wnd : Hwnd;
  Rect : TRect;
  Pid : Cardinal;
public
  constructor Create(AWnd:HWND;ARect:TRect;APid:Cardinal);
end;

function EnumWindowsProc(Wnd: HWnd; const obj:TList<TWin>): Bool; export; stdcall;
var ProcessId : Cardinal;
  R,R1 : TRect;
  Win : TWin;
begin
  Result:=True;
  GetWindowThreadProcessId(Wnd,ProcessId);
  if IsWindowVisible(Wnd) and not IsIconic(wnd)then begin
    GetWindowRect(Wnd,R);
    IntersectRect(R1,R,Screen.DesktopRect);
    if not IsRectEmpty(R1) then begin
      win := TWin.Create(Wnd,R,ProcessId);
      obj.Add(win);
    end;
  end;
end;

procedure GetProcessWindowList(WinList:TList<TWin>);
begin
  WinList.Clear;
  EnumWindows(@EnumWindowsProc, Longint(WinList));
end;

We want to keep a list of windows, with their basic attributes and their
bitmaps, so we can compare with the new ones and send the differences to the
client. Here we merge the window list into a list of TWindowMirror:

TWindowMirror = class
private
  FIndex : Integer;
  FRgn : HRGN;
  FHandle : THandle;
  FBoundsRect : TRect;
  FProcessId : Integer;
  FImage : TBitmap;
  FDiffStreamList : TList<TImagePart>;
  ...
  ...
end;

procedure TMirrorManager.RefreshMirrorList(out OneMoved:Boolean);
  procedure GetProcessWindowList(WinList:TList<TWin>);
  begin
    WinList.Clear;
    EnumWindows(@EnumWindowsProc, Longint(WinList));
  end;

var
  wl : TList<TWin>;
  n : Integer;
  wm : TWindowMirror;
begin
  OneMoved:=False;

  wl := TList<TWin>.Create;
  try
    // Enumerates top windows
    GetProcessWindowList(wl);
    try
      for n := wl.Count - 1 downto 0 do begin
        // Looks for a cached window
        wm:=GetWindowMirror(FMirrorList,wl[n].Wnd);
       if assigned(wm) then begin
        if IsIconic(wl[n].Wnd) then
           wm.SetBoundsRect(Rect(0,0,0,0))
       else wm.SetBoundsRect(wl[n].Rect);

          // Returns true when at least one window moved
       OneMoved:=OneMoved or (DateTimeToTimeStamp(Now-wm.FMoved).time<MOVE_TIME);
        end else begin
         // Do not create a TWindowMirror for invisible windows
         if IsIconic(wl[n].Wnd) then Continue;

       wm:=TWindowMirror.Create(Self,wl[n].Wnd,wl[n].Rect, wl[n].pid);
       FMirrorList.Add(wm);
        end;
        // Saves the zIndex
        wm.FIndex:=wl.Count-n;
        // Generates clipping regions
        wm.GenRegions(wl,n);
      end;
    finally
      ClearList(wl);
    end;
    // Sorts the mirror list by zIndex
    FMirrorList.Sort;
  finally
    wl.free;
  end;
end;



Finally we do the capture:






function TWindowMirror.Capture(ANewImage:TBitmap): Boolean;
  function BitBlt(DestDC: HDC; X, Y, Width, Height: Integer; SrcDC: HDC;
                   XSrc, YSrc: Integer; Rop: DWORD): BOOL;
  begin
    // Capture only visible regions
    SelectClipRgn(DestDC,FRgn);
    result:=Windows.BitBlt(DestDC, X, Y, Width, Height, SrcDC,
                           XSrc, YSrc, Rop);
    SelectClipRgn(DestDC,0);
  end;

var
  DC : HDC;
  RasterOp,ExStyle: DWORD;
begin
  RasterOp := SRCCOPY;
  ExStyle:=GetWindowLong(FHandle, GWL_EXSTYLE);
  if (ExStyle and WS_EX_LAYERED) = WS_EX_LAYERED then
  RasterOp := SRCCOPY or CAPTUREBLT;

  DC := GetDCEx(FHandle,0,DCX_WINDOW or DCX_NORESETATTRS or DCX_CACHE);
  try
    Result:=BitBlt(ANewImage.Canvas.Handle,0,0,
    Width(FBoundsRect),Height(FBoundsRect),DC,0,0, RasterOp)
  finally
    ReleaseDC(FHandle,DC);
  end;
end;





Now that we have captured all visible regions we need to get the bitmap

differences against the previous capture. We do this by looping through the

windows, then their visible regions and finally calculating the regions where we

find bitmap differences:






function TWindowMirror.CaptureDifferences(reset:boolean=false): Boolean;
  ....
begin
 ...
 result:=Capture(TmpImage);
 if result then begin
   ...
   ra:=ExtractClippingRegions(Rect(0,0,TmpImage.Width,TmpImage.Height));
   for n := 0 to Length(ra) - 1 do begin
     ra2:=GetDiffRects(FImage,TmpImage,ra[n]);
     for m := 0 to Length(ra2) - 1 do begin
       Jpg := TJpegImage.Create;
       ...
       CopyBmpToJpg(Jpg,TmpImage,ra2[m]);
       FDiffStreamList.Add(TImagePart.Create(rbmp,'jpeg'));
       Jpg.SaveToStream(FDiffStreamList[FDiffStreamList.Count-1].FStream);
       ... 
       Bitblt(FImage.Canvas.Handle,
       ra2[m].Left,ra2[m].Top,Width(ra2[m]),Height(ra2[m]),
       TmpImage.Canvas.handle, rbmp.Left,ra2[m].Top,SRCCOPY);
     end;
   end;
   ...
end;





On the next post we'll focus on the protocol with the client and the client code.



HTML5 remote desktop open source

You can download the full source code from Source Forge.





UPDATE:
There's a commercial version with PRO features available here.