Week 23 – sorry I haven’t posted in 12 weeks

My last week-by-week post was 12 weeks ago! Here is a summary of what we’ve accomplished

The lwio manager – The lwio kernel is fully asynchronous. IRP pending works like a charm.

The Posix Virtual File System -  the pvfs system has a complete lock manager implemented. As as result, we have full byte-range locking semantics. We actually pass the smb torture tests for byte range locks. For oplocks and BRLs, the  PVFS driver cleanly handle asynchronous semantics.

The lwmsg system  – our internal ipc mechanism support asynchronous semantics as well. Thus there are no blocking threads for calls on the server. Even when the client process makes synchronous calls, we translate them to async calls on the lwio kernel.

The smb server has been fully refactored so that the transports, and the smb1 and smb2 protocols are independently developed. See the last post  – we have almost completely finished the smb2 wire protocol engine.  Remember that our protocol engines are independent of the underlying file system, so when I say that the wire protocol engine is complete that means you have a complete smb2 file server.  We just wire the protocol engine to the underlying file system which was already in place for smb1.

lsass has gone through a significant upgrade. We have closed several large OEM deals so for each of them we were doing bits of fit and finish work.  The major improvements are the fully operational samdb backend as the local authentication provider, significant reduction in the number of shared object libraries and a much more simpler source code layout.

Coming soon… We will shortly be releasing our registry subsystem – all lsassd, eventlogd, lwiod infrastructure will store all its configuration information in the registry. At install time, we will provide utilities that export the text base configuration information for a subsystem into the registry. We will be providing, registry import-export tools, a registry shell and a graphical LAC plugin to graphically view, edit and manage the registry.  This is really important because it lets us manage upgrades smoothly, allow us to introduce new configuration parameters without having to write tons of parser and lexical analysis code.

Releases – We’re going to be releasing 5.2 very very soon .. so stay tuned.

Thanks for reading.

A month after Samba XP; onward to SNIA

I’ve gotten quite a few queries as to why I haven’t posted for the past six weeks. Sorry! Been awfully busy.
Well its been a month since Samba XP. But we continue to make huge strides in new features and functionality.

We gave a formal preview of the Likewise LWIO infrastructure at Samba XP. The numbers were: 4 demos, 5 developers, 60 slides and 60 minutes. We probably broke all of the standard presentation rules. But we made quite an impact.

We demonstrated the full lwio kernel architecture

We demonstrated the SMB redirector integrated with a FUSE client

We demonstrated the entire SMB server. Jerry Carter did a fantastic demo.

I’ll add more notes to this as I go.

lwio watch: Weeks 9, 10, 11 ending April 3rd , 2009

It has been a hectic three weeks and I haven’t been able to get to write a whole lot. I have been on the road quite a bit meeting with customers and I’d forgotten how busy things get on the road.

Anyway we’ve made huge strides ..

First, we’re dogfooding the SMB server on all our internal servers. We now have full security descriptor support in the Posix file system driver. Our first attempt was to write clean mapping functions from NT Security descriptors to Posix ACLs and vice versa, but we’ve settled to store native NT security descriptors as Posix file EAs. This allows us to quickly get the Windows MMC pieces to allow us to edit security descriptors.

Second, we now have a uniform access token model across the entire lwio “kernel”. What this means is that similiar to Windows ConvertAuthDataToToken function, we can pass in authorization data from an underlying security mechanism (in the case of the SMB server at session setup time) and create a generic lwio token that now holds the authentication state for the connection. This is now one of the parameters passed to IoCreateFile to the posix driver when creating/opening a handle to a(n) new/existing file managed by the posix driver. This access token model is a generic model common across all Likewise protected subsystems.

Brian Koropoff has finished asynchronous semantics in the lwmsg layer. What this means is that from “user space” to “kernel space”, we can support system calls with overlapped I/O semantics. Thus an NtCreateFile or a NtConnectNamedPipe call can immediately return back to the calling process, and when the “kernel” has data to indicate we can signal an event in the “user-mode” calling process. This was necessary because without this asynchronous behavior, each named pipe RPC server would be plumbing a ConnectNamedPipe call which would be consuming one blocking thread. This way, all named pipe RPC servers will be able to support asynchronous semantics. Huge piece of work.

The biggest piece of work left and we’ve made great progress on is the refactoring of our local authentication database in preparation of our lsarpc, netlogon, samr RPC server. I’m going to write a separate post on that. This is our SAM for Linux strategy.

The posix virtual file system has support for alternate data streams. I should clarify: the smb server will pass requests to the posix virtual file system driver which can implement them in a variety of ways: we can create a hidden directory where the additional data streams are stored as separate files. Alternatively, if the underlying file system has native support for alternate data streams, the posix virtual file system driver can support that.

Two weeks from now is Samba XP: I’m looking forward to this trip to Gottingen. It has become a yearly event and a greate time to meet with people.

Thanks for reading!

http://www.HowSoftwareisBuilt.com

Short post: I was recently interviewed by http://www.HowSoftwareisBuilt.com.

lwio watch: Week 8 ending March 13, 2009

This one is another short post. Here are our highlights

1) Share management is fully operational: This means you can launch MMC from your Windows client; load up the share management snapin and point to Linux box running the lwio SMB file server and srvsvc daemon. The result is that you can browse shares from a Windows client; you can create a new share on your Linux server as well and do all of the file share operations. We have yet to complete the “open files” and “open sessions” functionality. But we’re so pleased with seamless management that we’ve punted on this for now.
2) Our Likewise FuSE driver is fully operational: This means that a Linux client joined to Active Directory can easily mount SMB shares on a Windows/Samba/Likewise SMB file server. Here is the cool part: you login on your Linux machine using your Active Directory credentials and single sign-on mount your SMB file share. If your machine is multi-user and simultaneously your buddy logs on to the same machine and mounts his SMB file share using our FuSE driver; he connects seamlessly with his credentials to the SMB server. This is a really elegant piece of work. Brian tells me that OpenOffice running on a Linux client can save files to a Likewise/Samba/Windows SMB File Server through our SMB redirector. As you can notice, I am getting a kick out of saying Likewise/Samba/Windows SMB File Server. The cool thing here is not the file server, it is our FuSe driver that calls our SMB redirector (file client) within the LWIO Framework.

3) A ton of other coolness; I thought I’d just copy Jerry’s mail to me

“What works:

* User profiles load from the network
* Home directories mount properly at login time.
* My Documents redirection to %HOMESHARE%\%HOMEPATH%\Documents
works fine.

Also ran some tests with iTunes and was able to successfully import and existing library XML file and 4k mp3s from a network share to my current iTunes library. So now all my music is being served (and playing 24×7) from our server. Also editing mp3 song properties (artist, genre, etc…) works properly.

What is not working:

* Take ownership of files fails
* Roaming profiles are not saving

I believe that both of these are due to the lack of properly support partial security descriptor set operations (based on the SECURITY_INFORMATION flags) which I will implement today.”

I’ll post some screenshots once I get some cycles…

Thanks for reading!

lwio watch: Week 7 ending March 6th, 2009

This is a really short post, but I hope people get a kick out of it. Here is the current set of applications that the SMB server supports. This means an Windows XP client can run the applications below, read files from and write files to a file share on Linux machine running the Likewise SMB Server!

1) Word
2) Excel
3) Powerpoint
4) Explorer
5) IE
6) cmd shell
7) Tortoise SVN for checkouts
Visio
9) Project
10) Wireshark
11) Notepad
12) Outlook
13) Firefox
14) Safari
15) iTunes
16) Adobe Reader
17) Publisher
18) OpenOffice.org
19) QuickTime
20) Software Extraction and Installation/.MSI files all work
21) Image Previews and Thumbnails

Thanks for reading!

lwio – the SMB redirector: Building the integrated Linux SMB client experience

One of the more painful things about using Linux as a client in a Windows corporate intranet is the clunkiness of integration. The most basic requirement is that you be able to bring up the Nautilus or your favorite graphical file browser and browse through files that live on a Windows (Samba) file server. I’ll reiterate: this is one of the most basic requirements.

Now most affficianados will be up in arms when I say this. Of course, Linux does this and Ubuntu does this out of the box. But here is the rub: Ubuntu’s Nautilus file browser will does this fine. But now I want to go to my bash command shell and I want it to also be able to browse to my Windows file shares. This does not happen out of the box.

Now even if you get all of the pieces to work, the credentials are messed up. The single-sign-on experience needs to immediately transfer itself to all client applications. Even though you can sign in with your AD credentials, if I have to re-enter my credentials, it adds to the clunkiness of the experience.

Introducing the SMB redirector in lwio.

Our SMB redirector driver provides all of the semantics necessary for a file client API. The obvious next step was to pick up FUSE and begin work on a FUSE driver that would call our lwio APIs – NtCreateFile, NtReadFile, NtWriteFile, NtClose. Once you provide a FUSE driver to our SMB client, automatically, a client can mount smb shares without reentering his/her credentials. Once the client mounts a share, he can browse the directory space, copy files, move files, create directories, delete files and perform every UNIX file system operation against the SMB volume. Even better, Nautilus and the Gnome shell work without any additional work.

We’ve started construction of this FUSE driver and its is amazing how fast we’re putting together the pieces. Our Posix Virtual File System (which is the most complete lwio driver) is proxying for the SMB Redirector File System (we’ve implemented enough for the named pipe interface and its missing some of the QueryInfo and SetInfo semantics)

Now here is the even more interesting part: FUSE driver run as a daemon themselves, so a client (like the shell) has to communicate to the FUSE daemon which in turn communicates to the lwio daemon. That’s a transition of 2 processes which means at least 4 context switches to service a call. But lwio can implement a FUSE driver as another lwio driver. Just like the SMB server driver calls the IO Manager to route requests to the Posix Virtual File System, the FUSE lwio driver calls the IO Manager to route requests to the SMB Redirector File System. This implies we reduce the number of context switches between processes to half what we would have if we ran FUSE independently as a separate daemon. This was Brian Koropoff’s epiphany over the weekend. We’re still going to do this first as a separate daemon, iron out the end-to-end scenario, but then placing it into the lwio kernel is a natural optimization.

Well, I thought we’d be doing three key scenarios by April 15th. It looks like we’re going to go for a fourth scenario.

Thanks for reading!

lwio watch: Week 7 ending February 28, 2009

In the course of building a new software system, there are these inflection points when everything comes together. Last week was one such inflection point. Here are the highlights
- Windows explorer, dir, Acrobat reader, thumbnails views on explorer all work
- copying files, deleting files, drag and drop, xcopies are all smooth and seamless
- Word now works – we can click on a Word document on a share and we can open the document and edit it
- Multiple connections and multiple large file downloads work smoothly
- We’ve set up an internal file server that hosts over half a terrabyte of data (with full iso OS images and VMs) and are using this as our internal dog food server.
- The MMC share management snapin and wizard works smoothly; we can point MMC at our Linux server create, delete and manage shares on the server.
- The installation is very simple. There is next to nothing to configure and setup. Install the bits and you have a file server available.

The month of March promises to be very interesting for us. I’ll write another post on what we plan on accomplishing by end of March.

Thanks for reading

lwio watch: Week 6 continued (ends February 20th, 2009 today)

It has been a super productive week.

First the highlights. The lwio SMB File Server has hugely advanced. Most operations from the Windows XP command line work. Copying files, xcopying, making directories, net use, net use /del, single sign-on are all fully operational. In addition, the Windows explorer is almost completely functional. We’ve yet to support server-side file change notifications (i.e changes to the remote directory are not immediately visible in the Windows Explorer, the simple workaround is to hit the refresh button). You can view thumbnails of pictures and photographs, you can bring up notepad on a file directly. In all, we have a pretty useful file server.

Today we plan on starting internal dog-fooding. We will be hosting our own server and placing large size OS iso images which we use, so that everyone in the engineering team can upload/download files to the server. We’re putting all our VMs as well on the server and will subject the server to a significant amount of stress.

The day is not yet done. It is 8:38 hrs PST out here in Bellevue WA. We plan to have the file share management work done by end of day today. This will allow us to create file shares on the Likewise SMB File Server via MMC’s Add Share Wizard. Also, we plan on add a File Management plugin on the Likewise Administrators Console that will allow Linux Administrators to graphically manage their own Linux Likewise File Servers from Linux desktops.

Its been a good week!

Windows Explorer works!

Yesterday, we got Windows Explorer to work against the Likewise SMB File Server

Windows Explorer copying a file from the Likewise SMB File Server