Archive for the ‘File Operations’ Category.

Excel as a Really Bad Database

Do you know what's worse than having a bunch of data in an Excel workbook that really should be in a relational database? Having a bunch of data in multiple Excel workbooks that really should be in a relational database. It's the Excel-workbook-as-a-record model and it's how we currently store our quotes.

Each quote is a separate Excel file and all the files are stored in a folder. I need some information out of these files, so I have to create my own table. I set a reference to the Microsoft Scripting Runtime dll so that I can use the FileSystemObject.

tools references excel scripting runtime

Now I can loop through all the files in the folder.

Sub GetQuotes()
   
    Dim fso As Scripting.FileSystemObject
    Dim dtOldest As Date
    Dim oFile As Scripting.File
    Dim sPath As String
    Dim wb As Workbook
   
    Set fso = New Scripting.FileSystemObject
    dtOldest = Date - 30
    sPath = "S:\AIMUSA\Sales\Quotes\AIM Quotes\"
   
    For Each oFile In fso.GetFolder(sPath).Files
        If UCase(Right$(oFile.Name, 3)) = "XLS" Then
            If oFile.DateCreated> dtOldest Then
                WriteQuotes oFile
            End If
        End If
    Next oFile
 
End Sub

I only want Excel files, and get them by looking at the last three characters of the file name. I also only want those files that were created in the last month, or so. This was the primary reason why I didn't use Application.FileSearch. FileSearch has a LastModified property, but it doesn't seem very flexible. I could use msoLastModifiedLastMonth, and that would have worked well today, but it would not have worked so well on, say, August 15th. I'm also not really that interested in the date it was last modified and I didn't see any facility for creation date in FileSearch.

The down side to using the FileSystemObject is that I end up looping through 2,500 files. I'm not sure how to limit the number it loops through.

Here's the WriteQuotes sub, although not really that interesting:

Sub WriteQuotes(oFile As Scripting.File)
   
    Dim wb As Workbook
    Dim rStart As Range
    Dim rQuoteParts As Range
    Dim rCell As Range
    Dim lRow As Long
   
    Set wb = Workbooks.Open(oFile.Path)
    Set rStart = wshQuotes.Range("A65536").End(xlUp).Offset(1, 0)
    Set rQuoteParts = wb.Sheets(1).Range("A21", wb.Sheets(1).Range("A45").End(xlUp))
    lRow = 0
   
    For Each rCell In rQuoteParts.Cells
        rStart.Offset(lRow, 0).Value = wb.Sheets(1).Range("C9").Value
        rStart.Offset(lRow, 1).Value = oFile.DateCreated
        rStart.Offset(lRow, 2).Value = rCell.Offset(0, 1).Value
        rStart.Offset(lRow, 3).Value = rCell.Offset(0, 7).Value
        lRow = lRow + 1
    Next rCell
   
    wb.Close False
   
End Sub

Note that the Path property of the Scripting.File object includes the filename, which is different than the Path property of the Excel.Workbook object.

The New Excel 2007 File Format

Most of you will know that Excel 2007 (well, Office 2007) comes with a brand new file format, based on what MSFT calls Open XML.

This suddenly enables us to write code that can easily generate/change Office 2007 files without the need for an Office installation. For instance on a server.

Whilst there is proper documentation on this file format, the document with detailed descriptions of the Open XML format ("part 4" in the aforementioned link) counts an astonishing 4721 pages !!

This is why I decided to write up a couple of basic pages on how to do stuff with these Open XML files.

My first one is about reading and editing cells:

Working With Worksheet Data In An Excel 2007 File

Enjoy!

Regards,

Jan Karel Pieterse
www.jkp-ads.com

Unprotect all Worksheets in all Workbooks

Here is one for the Code Library.

Somehow I end up misplacing this bit of code. So every time I need to do it, I end up re-writing it.
Perhaps I'll save someone the same frustration along the way.

This code snippet will loop through each file in your folder (and subfolders).
For each workbook opened, it will unprotect each worksheet using the supplied password.

Const cStartFolder = "D:\MySecretSpreadsheets" 'no slash at end
Const cFileFilter = "*.xls"
Const cPassword = "trustno1"
 
Sub UnprotectAllWorksheets()
    Dim i As Long, j As Long, arr() As String, wkb As Workbook, wks As Worksheet
 
    ExtractFolder cStartFolder, arr()
 
    On Error Resume Next
    j = -1: j = UBound(arr)
    On Error GoTo 0
 
    For i = 0 To j
        Set wkb = Workbooks.Open(arr(i), False)
        For Each wks In wkb.Worksheets
            wks.Unprotect cPassword
        Next
        wkb.Save
        wkb.Close
    Next
End Sub
 
Sub ExtractFolder(Folder As String, arr() As String)
    Dim i As Long, objFS As Object, objFolder As Object, obj As Object
 
    Set objFS = CreateObject("Scripting.FileSystemObject")
    Set objFolder = objFS.GetFolder(Folder)
 
    For Each obj In objFolder.SubFolders
        ExtractFolder obj.Path, arr()
    Next
 
    For Each obj In objFolder.Files
        If obj.Name Like cFileFilter Then
            On Error Resume Next
            i = 0: i = UBound(arr) + 1
            On Error GoTo 0
            ReDim Preserve arr(i)
            arr(i) = objFolder.Path & Application.PathSeparator & obj.Name
        End If
    Next
End Sub

Creating Folders with MkDir

Scott wants to create folders based on the information in certain cells. I suggest the MkDir function.

Check out the line below "Make sure base folder exits". Is that the best way to do that. For some reason I thought there was a problem with that method, but I can't think of what it was.

Sub StartHere()
   
    Dim rCell As Range, rRng As Range
   
    Set rRng = Sheet1.Range("A1:A2")
   
    For Each rCell In rRng.Cells
        CreateFolders rCell.Value, "C:\Test"
    Next rCell
       
End Sub
 
Sub CreateFolders(sSubFolder As String, ByVal sBaseFolder As String)
   
    Dim sTemp As String
   
    'Make sure the base folder is ready to have a sub folder
    'tacked on to the end
    If Right(sBaseFolder, 1) <> "\" Then
        sBaseFolder = sBaseFolder & "\"
    End If
   
    'Make sure base folder exists
    If Len(Dir(sBaseFolder, vbDirectory))> 0 Then
        'Replace illegal characters with an underscore
        sTemp = CleanFolderName(sSubFolder)
        'See if already exists: Thanks Dave W.
        If Len(Dir(sBaseFolder & sTemp)) = 0 Then
            'Use MkDir to create the folder
            MkDir sBaseFolder & sTemp
        End If
    End If
   
End Sub
 
Function CleanFolderName(ByVal sFolderName As String) As String
   
    Dim i As Long
    Dim sTemp As String
   
    For i = 1 To Len(sFolderName)
         Select Case Mid$(sFolderName, i, 1)
            Case "/", "\", ":", "*", "?", "<", ">", "|"
                sTemp = sTemp & "_"
            Case Else
                sTemp = sTemp & Mid$(sFolderName, i, 1)
        End Select
    Next i
   
    CleanFolderName = sTemp
   
End Function

img: excel range and windows folder showing new sub folders

Log Worksheet Changes

I have a requirement to log all the changes made to a particular worksheet. I cringe whenever I'm faced with that particular task. I could use a Worksheet_Change event coupled with a Worksheet_SelectionChange event to log the old cell value and the new cell value. I really hate that option for a couple of reasons. First, I would have code running all the time which would be a drag on the system. Also, it logs way too much stuff. For instance, if a user changes a cell value from 1 to 2, then changes it back from 2 to 1, you have two log entries that don't add much to the log. It makes the log file messy and lessens its value in total. Another problem occurs when the user changes a whole slew of cells at once. Make a Worksheet_Change log procedure then delete the used range of the worksheet. With just a 10 column x 10 row worksheet, you've just made a 100 log entries with a noticeable lag in performance.

Speaking of changing a bunch of cells, I would have to save the old value in a module level variable using the SelectionChange event so that I could log it when the user fires the Change event. That wouldn't be such a big deal if I could use the Target variable from each of those event procedures, but when the user selects a range of cells, I have to save the "old value" from every cell in the selection. What a pain.

Normally, I would recommend that people don't log their changes. But I have situation that's hard to defend. There's a one worksheet workbook with a fairly limited UsedRange. It's an important table that a lot of other sheets will use and that many users will need to update as necessary. Everything happens (or should happen) in this limited UsedRange and there will never be any other sheets in the workbook. It's a prime candidate for change logging because many users can change the data and that data affects every other user. If things get screwy, there needs to be some accountability.

Nevertheless, I wasn't going to use a Worksheet_Change event. There's just too much to track and too many problems that can arise. So I settled on a different tack, which I (finally) describe here.

The basic structure is that I save a copy of the file when it's opened and I compare the current file with this saved copy to see what's changed. This has the benefit of keeping the log file fairly clean because the whole change/change back scenario isn't logged. Only the final changes against the original are logged. It also has the advantage that the code is running less. With a Change event, the code runs whenever you do absolutely anything. With this, it only runs on Open, on Save, and on Close (as described in a bit).

I'm not saying there aren't challenges with this because there are. At first, I decided that I would save a copy on Open and compare on Close. I'm sure the brighter of you have already figured out the problem with that. You can close a file without saving it. So now I have to determine if the user cancels the close, closes without saving, or closes and saves. That's all well and good, but I only have a Before_Close event and these user decisions happen after that fires. Ultimately, I had to settle on logging at every save. If my goal was to only log real changes, this takes me farther away from that goal. Now a user can change a cell, save, change it back, and save it again. It's still less than the alternative. Here's the basic structure:

When the file is opened, I use the SaveCopyAs method to save it in its current state, heretofore referred to as OldCopy
When the file is saved, I compare the two workbooks and log the changes. Then I close the OldCopy, delete it with a Kill statement, and resave the NewCopy so OldCopy is now current.
When the file is closed, I kill the OldCopy. Not so fast my friend, as Lee Corso might say. I had to control the hell out of the BeforeClose event for this to work. The long and the short of it is that I had to eliminate the Cancel option and program my own "Save and Close", "Don't Save and Close" options. Either way, that file was closing. Not ideal, but that's life.

Figuring out which cells changed posed another problem. Do I loop through the newer file and compare it to the older file or vice versa. Or do I compare all 65,000 rows x 256 columns to make sure I don't miss something. I don't think so. I ended up looping through the newer and comparing to the older, but I didn't necessarily use the UsedRange of the newer. I counted the cells in the UsedRange from both sheets and used the Address property of the larger of the two to define the corpus of cells to change. This, however is not foolproof.

The main issue with differences in the UsedRange was deleting rows and columns. If the user deleted half the rows in the newer file, then the newer file's UsedRange might be quite a bit smaller that the UsedRange of OldCopy. If I just loop through the UsedRange of NewCopy, then I'm not actually logging all the changes because I'm missing all the zeros in NewCopy that had values in OldCopy. Counting the cells gets close, but it's not perfect. If I delete a bunch of columns in NewCopy but add a bunch of rows, the UsedRange may have a higher count, but still not catch every change.

Now that I type this, I can see that I should count the rows and columns separately and use the greater of each independent count to determine which range I should compare. Okay, I'll change the code, but I don't guarantee that I won't screw it up.

I haven't commented this code very well (count of comments = 0) so you'll have to rely on the above description. I think the variables speak for themselves. Anything that starts with a 'g' is a global variable. Anything in all caps is a constant, proper case is a variable. The error handling scheme is straight out of Professional Excel Development. The variable gsIdentStandardsOpen holds the name of the person who opened. This global variable is set when the workbook is opened and a password is supplied. I'll describe the password scheme in a future post. I don't know if any of that helps, but there it is.

Public Function LogStandardsChanges() As Boolean
 
    Dim bReturn As Boolean
    Dim lFnum As Long
    Dim sOutput As String
    Dim wbOldStan As Workbook
    Dim wbNewStan As Workbook
    Dim rCell As Range
    Dim bChanges As Boolean
    Dim lMaxRow As Long
    Dim lMaxCol As Long
    Dim rRng As Range
   
    Const sSOURCE As String = "LogStandardsChanges()"
    On Error GoTo ErrorHandler
    Application.EnableCancelKey = xlErrorHandler
    Application.ScreenUpdating = False
    bReturn = True
   
    Set wbNewStan = Workbooks(gsSTAN)
    Set wbOldStan = Workbooks.Open(gsTemplateFldr & gsOLDSTAN)
   
    lMaxRow = Application.WorksheetFunction.Max(wbNewStan.Sheets(1).UsedRange.Rows.Count, _
        wbOldStan.Sheets(1).UsedRange.Rows.Count)
    lMaxCol = Application.WorksheetFunction.Max(wbNewStan.Sheets(1).UsedRange.Columns.Count, _
        wbOldStan.Sheets(1).UsedRange.Columns.Count)
       
    With wbNewStan.Sheets(1)
        Set rRng = .Range(.Cells(1, 1), .Cells(lMaxRow, lMaxCol))
    End With
   
    bChanges = False
   
    sOutput = String(60, "-") & vbNewLine
    sOutput = sOutput & "Saved: " & Format(Now, "yyyy-mm-dd hh:mm:ss") & _
        vbTab & "By: " & gsIdentStandardsOpen & vbNewLine
       
    For Each rCell In rRng.Cells
        If rCell.Value <> wbOldStan.Sheets(1).Range(rCell.Address).Value Then
            sOutput = sOutput & rCell.Address & vbTab & _
                "Old Value: " & wbOldStan.Sheets(1).Range(rCell.Address).Value & vbTab & _
                "New Value: " & rCell.Value & vbNewLine
            bChanges = True
        End If
    Next rCell
   
    sOutput = sOutput & String(60, "-")
   
    If bChanges Then
        lFnum = FreeFile
       
        Open gsTemplateFldr & gsSTANLOG For Append As lFnum
       
        Print #lFnum, sOutput
       
        Close lFnum
    End If
   
ErrorExit:
    On Error Resume Next
    LogStandardsChanges = bReturn
    wbOldStan.Close False
    Kill gsTemplateFldr & gsOLDSTAN
    wbNewStan.SaveCopyAs gsTemplateFldr & gsOLDSTAN
    Application.ScreenUpdating = True
    Exit Function
 
ErrorHandler:
    bReturn = False
    If bCentralErrorHandler(msMODULE, sSOURCE) Then
        Stop
        Resume
    Else
        Resume ErrorExit
    End If

End Function

So there it is. I'm tired of typing, but if there's something I didn't explain be sure to let me know. You comments on this strategy, as always, are welcome and expected.

FTP Via VBA

One of my new year's resolutions was to write a procedure to ftp picture files to this blog to simplify the procedure. It took two days, but I finally got it done. I don't know if it's well done, but it works. I got a little help along the way, specifically:

Forestasia has code to determine the image size. Another resolution is to put height and width arguments in my img tags. I tried about 17 other things before finding this page, such as looking at the extended file properties like Walkenbach did with MP3s. I'm glad I found it, though, because I learned something about the GIF file format. Nice use of Get.

Speaking of that, I read the GIF89a file specification to see what else I could learn. I had a little trouble applying the spec to the bits (which I printed to a worksheet for inspection). For one, it was clear from Forestasia's code that the dimensions were stored in two bits; one with size mod 256 and the other with the number of full 256's, but I couldn't find that in the spec. I was going to make an all white GIF and an all black GIF and compare the bits, but I lost interest.

For the FTP stuff, I went to bygsoftware. I read about ftp subcommands in Windows help, but all my attempts to change bygsoftware's code failed. It ended up looking pretty much like they have it.

Finally, since I can never remember how to stuff text into the clipboard, I went back (as always) to Chip's Clipboard page. Forms 2.0. Oh yeah, now I remember.

The code is a bit long, but there's some good stuff in there.

Enum gdGifDims
    gdHeight = 0
    gdWidth = 1
End Enum
 
Sub UploadPicture()
   
    Dim vFname As Variant
    Dim i As Long
    Dim sTags As String
    Dim lHeight As Long
    Dim lWidth As Long
   
    Const sIMG As String = "<img src="""
    Const sPATH As String = "http://www.dicks-blog.com/blogpix/"
    Const sIMGEND As String = " alt="""" />"
   
    'get one or more gif files
    vFname = Application.GetOpenFilename("*.gif, *.gif", , , , True)
   
    'Make the img tags
    For i = LBound(vFname) To UBound(vFname)
        lHeight = GetGifDim(vFname(i), gdHeight)
        lWidth = GetGifDim(vFname(i), gdWidth)
        sTags = sTags & sIMG & sPATH & Dir(vFname(i)) & Chr$(34) & _
            " height=" & lHeight & _
            " width=" & lWidth & _
            sIMGEND & vbCrLf
    Next i
   
    SendViaFtp vFname
   
    'put string in clipboard
    PutInClip sTags
   
End Sub
 
Function GetGifDim(ByVal sFname As String, ByVal eDim As gdGifDims) As Long
   
    Dim btBuffer(10) As Byte 'to get the first 10 bits
    Dim lFnum As Long
   
    lFnum = FreeFile
   
    'open the file and read in the bits
    Open sFname For Binary As lFnum
    Get lFnum, 1, btBuffer
    Close lFnum
   
    If eDim = gdHeight Then
        GetGifDim = btBuffer(8) + (btBuffer(9) * 256)
    Else
        GetGifDim = btBuffer(6) + (btBuffer(7) * 256)
    End If
   
End Function
 
Sub SendViaFtp(vFname As Variant)
   
    'code modified from http://www.bygsoftware.com/Excel/VBA/ftp.htm
   
    Dim i As Long
    Dim lFnumFtp As Long, lFnumBatch As Long
    Dim sFname As String
   
    Const sPATH As String = "C:\"
    Const sSITE As String = "123.45.678.901"
    Const sUSER As String = "MyUserName"
    Const sPASS As String = "MyPassword"
    Const sDIR As String = "www\blogpix\"
           
    sFname = sPATH & Format(Now, "yyyymmddhhmm")
    lFnumFtp = FreeFile
   
    'Create text file with ftp commands
    Open sFname & ".txt" For Output As lFnumFtp
    Print #lFnumFtp, "open " & sSITE 'open the site
    Print #lFnumFtp, sUSER
    Print #lFnumFtp, sPASS
    Print #lFnumFtp, "binary" 'set file transfer mode
    Print #lFnumFtp, "cd " & sDIR
    For i = LBound(vFname) To UBound(vFname)
        Print #lFnumFtp, "send " & Dir(vFname(i)) 'send files
    Next i
    Print #lFnumFtp, "bye" 'close ftp session
   
    Close lFnumFtp  'close text file
   
    lFnumBatch = FreeFile
   
    'open a batch file
    Open sFname & ".bat" For Output As lFnumBatch
    Print #lFnumBatch, "ftp -s:" & sFname & ".txt"
    Print #lFnumBatch, "Echo ""Complete""> " & sFname & ".out"
    Close lFnumBatch
   
    'run the batch file
    Shell sFname & ".bat"
   
    'what until the ftp session is closed
    Do While Dir(sFname & ".out") = ""
        DoEvents
    Loop
   
    Application.Wait Now + TimeValue("0:00:03")
   
    'clean up files used
    On Error Resume Next
        Kill sFname & ".txt"
        Kill sFname & ".bat"
        Kill sFname & ".out"
    On Error GoTo 0
   
End Sub
 
Sub PutInClip(sTags As String)
   
    Dim doObject As DataObject
   
    Set doObject = New DataObject
   
    doObject.SetText sTags
    doObject.PutInClipboard
   
End Sub

What kind of dink would I be if I didn't actually have a picture on this post? Here's the first part of the 12,365 bits of GIF file on my computer. The bits actually start at 1 (if you're using the Get Statement), but the array they were stored in was zero based.

Excel range listing first few bits of GIF format