Jim Pickering's Radio Weblog

Fri, 11 Mar 2005 22:58:09 GMT

Remember that there are typically two 8-bit code pages active, the so-called "ANSI" code page and the so-called "OEM" code page. GUI programs usually use the ANSI code page for 8-bit files (though utf-8 is becoming more popular lately), whereas console programs usually use the OEM code page.

This means, for example, when you open an 8-bit text file in Notepad, it assumes the ANSI code page. But if you use the TYPE command from the command prompt, it will use the OEM code page.

This has interesting consequences if you switch between the GUI and the command line frequently.

The two code pages typically agree on the first 128 characters, but they nearly always disagree on the characters from 128 to 255 (so-called "extended characters"). For example, on a US-English machine, character 0x80 in the OEM code page is Ç, whereas in the ANSI code page it is €.

Consider a directory which contains a file named Ç. If you type "dir" at a command prompt, you see a happy Ç on the screen. On the other hand, if you do "dir >files.txt" and open files.txt in a GUI editor like Notepad, you will find that the Ç has changed to a €, because the 0x80 in the file is being interpreted in the ANSI character set instead of the OEM character set.

Stranger yet, if you mark/select the file name from the console window and paste it into Notepad, you get a Ç. That's because the console window's mark/select code saves text on the clipboard as Unicode; the character saved into the clipboard is not 0x80 but rather U+00C7, the Unicode code point for "Latin Capital Letter C With Cedilla". When this is pasted into Notepad, it gets converted from Unicode to the ANSI code page, which on a US-English system encodes the Ç character as 0xC7.

But wait, there's more. The command processor has an option (/U) to generate all piped and redirected output in Unicode rather than the OEM code page.

(Note that the built-in documentation for the command processor says that the /A switch produces ANSI output; this is incorrect. /A produces OEM output. This is one of those bugs that you recognize instantly if you are familiar with what is going on. It's so obviously OEM that when I see the documentation say "ANSI", my mind just reads it as "OEM". In the same way native English speakers often fail to notice misspellings or doubled words.)

If you run the command

 cmd /U /C dir ^>files.txt

then the output will be in Unicode and therefore will record the Ç character as U+00C7, which Notepad will then be able to read back.

This has serious consequences for batch files.

Batch files are 8-bit files and are interpreted according to the OEM character set. This means that if you write a batch file with Notepad or some other program that uses the ANSI character set for 8-bit files, and your batch file contains extended characters, the results you get will not match the what you see in your editor.

Why the discrepancy between GUI programs and console programs over how 8-bit characters should be interpreted?

The reason is, of course, historical.

Back in the days of MS-DOS, the code page was what today is called the OEM code page. For US-English systems, this is the code page with the box-drawing characters and the fragments of the integral signs. It contained accented letters, but not a very big set of them, just enough to cover the German, French, Spanish, and Italian languages. And Swedish. (Why Swedish yet not Danish and Norwegian I don't know.)

When Windows came along, it decided that those box-drawing characters were wasting valuable space that could be used for adding still more accented characters, so out went the box-drawing characters and in went characters for Danish, Norwegian, Icelandic, and Canadian French. (Yes, Canadian French uses characters that European French does not.)

Thus began the schism between console programs (MS-DOS) and GUI programs (Windows) over how 8-bit character data should be interpreted.

[The Old New Thing]

Fri, 10 Dec 2004 20:28:18 GMT

For an interesting discussion as to why you should not browse the web or read email while using an Administrator account, please review the info in this MSDN article:

http://msdn.microsoft.com/security/securecode/columns/default.aspx?pull=/library/en-us/dncode/html/secure11152004.asp

Sat, 20 Mar 2004 00:13:39 GMT

Some of the following may be of interest:

OLEXP: Unintended Text May Be Sent with All New Messages:
http://support.microsoft.com/default.aspx?scid=kb;en-us;Q255827

OLEXP: How to Troubleshoot E-mail Messages That Are Stuck in Your Outbox:
http://support.microsoft.com/default.aspx?scid=kb;en-us;Q262052

OLEXP: Error Finding POP3 or SMTP Servers When You Send or Receive E-mail Messages:
http://support.microsoft.com/default.aspx?scid=kb;en-us;Q274456

OLEXP: Error Message: MSIMN Caused an Invalid Page Fault in Module Kernel32.dll:
http://support.microsoft.com/default.aspx?scid=kb;en-us;Q276727

OLEXP: Address Book Group Does Not Display Members:
http://support.microsoft.com/default.aspx?scid=kb;en-us;Q278484

OLEXP: Question Marks Are Displayed in the "To" and "Subject" Boxes:
http://support.microsoft.com/default.aspx?scid=kb;en-us;Q245299

Outlook Express does not start after you upgrade your operating system to Windows XP Home Edition or Windows XP Professional Edition:
http://support.microsoft.com/default.aspx?scid=kb;en-us;Q873021

OLEXP: Closing "Find Message" Window Closes Reply Message in Outlook Express:
http://support.microsoft.com/default.aspx?scid=kb;en-us;Q243264

Outlook Express Support Center

Microsoft Support Center for Outlook Express

Recent Arrivals
OLEXP: Using Virus Protection Features in Outlook Express 6 (291387)
OLEXP: "Most Recently Used" Feature Does Not Retain Encoding Options on the Encoding Menu (296504)
OLEXP: Can Send and Receive Messages in Only 32 POP3 Accounts in Outlook Express 6 (296513)

275543 - How to trouble shoot the invalid page fault error message.

Updated - Blank Subject line in email update.

293907 - How to uninstall Internet Explorer 6.

191946 - Err Msg: Address Book Failed to Load

259015 - Unable to move newsgroup messages.

265230 - Blank attachments when messages starts with begin.

257234 - How to uninstall Internet Explorer 5.5

268986 - Using Outlook Express over Internet Connection Sharing

259927 - OE stops responding and pointer appears as hourglass.

263837 - How to manually uninstall and reinstall OE in Windows 2000.

Cannot do.....?

221486 - Cannot send mail with attachment.(Happy99.exe damage)

176571 - Cannot specify mail program in Outlook Express.

178895 - Cannot change Text Format in Outlook Express Express

224176 - Cannot Spell Check Messages in Outlook Express 5.

235236 - Cannot type text in Outlook Express or Internet Explorer.

229130 - Cannot use Add/Remove Programs to Install Outlook Express 5

HotMail

222984 - Known Issues with Hotmail

241841 - Cannot Create an HTTP or Hotmail Account in Outlook Express 5

246340 - Description of Hotmail Advertising in Outlook Express

247106 - Unable to Retrieve HTTP E-Mail Using Cable Modem

247335 - Outlook Express Cannot Connect to Hotmail through Proxy Server

252518 - "Unknown Error" in Outlook Express When Accessing a Hotmail Account

252840 - Problems Downloading E-mail from Hotmail Account

253474 - HotMail Messages Disappear in Outlook Express Folders

HTML and Stationery

165531 - How to View the HTML Source for Outlook Express Messages

171245 - How to Use Stationery in Outlook Express

180819 - How to Edit Stationery in Outlook Express Using an HTML Editor

174501 - How to Change Reply Font and Line Color in Outlook Express

170968 - How to Insert an HTML Signature in Outlook Express Messages

Address Book (WAB)

175017 - How to Import and Export Address Books in Outlook Express

175585 - Address Book Failed to load. (MSN members)

177123 - How to Import Outlook Contacts into Outlook Express

182833 - How to Copy an Outlook Express WAB to Another Computer

189396 - OL98: Windows Address Book is missing.

191946 - Address Book failed to load.

192321 - Unable to open Address Book, a component is missing

211040 - "Address Book Export Process Has Completed" Message Is Incorrect

210948 - () Add Name to Address Book Prompt When Name Is Added to Group

217125 - Command-Line Switches for Windows Address Book (Wab.exe)

222068 - Shared Windows Address Book Does Not Display Contacts

224802 - Error Message: No Matches Found for <User Name>

236342 - No .pab File Created When Exporting Address Book from Outlook Express

238892 - Unable to Import Netscape 4.6x Address Book in Outlook Express

239135 - The Message could not be sent. Some recipients not valid

241875 - Cannot Export Address Book Subfolders in Outlook Express

244459 - Address Book Entries Missing After Uninstalling Outlook Express 5

Error Messages

247706 - Error Message: MSIMN Caused an Invalid Page Fault in Module Wab32.dll

202498 - Error Messages After You Uninstall Outlook Express

211005 - Err Msg: No Other E-mail Accounts Could Be Found

227162 - Err Msg: Not All of the Messages Needed to Assemble This...

225096 - Error Message Using Some Rules in Outlook Express

224141 - Error Message: MSIMN Caused an Invalid Page Fault in Module...

224188 - Err Msg: MSIMN Caused an Invalid Page Fault in PIMAB32.DLL

224421 - Error Message: Unable to Connect to Server. Host Not Found

224513 - Error Message: Uninstalling Outlook Express 5 May Prevent Outlook from Functioning

224548 - Error Message: An Error Occurred While Establishing a Dial-Up...

225867 - Err Msg: MSIMN Caused an Invalid Page Fault in Module DIRECTDB.DLL importing ods message files from beta 2.

233264 - Err Msg: MSIMN Causes an Invalid Page Fault in Module DIRECTDB.DLL if folders.dbx is missing or damaged

238585 - Err Msg: "MSIMN Caused an Invalid Page Fault in Module Unknown"

238656 - Err Msg: "This Program Has Performed an Illegal Operation and Will Be Shut Down"

239135 - Error Message: The Message Could Not Be Sent. Some of the Recipients for This Message Are Not Valid.

233079 - Error Message When Choosing to Work Offline

241256 - Err Msg: "An Unknown Error Has Occurred" in Outlook Express

241552 - Err Msg: Unable to Logon to Server Using Secure Password Authentication Components

253351 - Error Message: An Error Occurred While the Spelling Was Being Checked

253795 - Error 0x800C013B When You Attempt to Send a Message in Outlook Express 5

187965 - Err Msg: Mprexe Caused an Invalid Page Fault in Module Kernel...

202481 - Error Message: Digitally Signed - Signature Invalid

How To or How do I.....

166514 - How to Delete News Messages and Headers in Outlook Express

171240 - How to Work Offline with Outlook Express

171163 - How to Configure Outlook Express for Internet Mail

171164 - How to Configure Outlook Express for Internet News

171190 - How to Subscribe to Newsgroups in Outlook Express

168726 - How to Digitally Sign and Encrypt Messages in Outlook Express

175037 - How to Change the Default Location of Mail and News Folders

175134 - How to Restore Newsgroup Messages After Local File Clean Up

176267 - How to Import MMF File into Outlook Express

176660 - How to Configure Outlook Express for Unattended Dial-Up

175720 - How to Enable Directory Services and Change the Search Order

174992 - How to Add Socket Support for Secure Directory Service Servers

178077 - How to set Outlook Express as Your Default Simple MAPI Client

181084 - How to Back Up Your Inbox Assistant Rules in Outlook Express

177402 - How to Import an Internet Mail 3.x for Macintosh Address Book

179414 - How to Store Messages on Mail Server with Outlook Express

177897 - How to Change the Default Font in Outlook Express

179590 - How to Identify the Version of Outlook Express in Use

181985 - How to Create a Shortcut for an Outlook Express E-mail Message

183853 - How to Move Messages to Exchange Server in Outlook Express

188093 - How to Back Up the Account List in Outlook Express

192341 - How to Send Outlook Express Mail from a Command Line

205480 - How to Combine Multiple-Part Messages in Outlook Express

209169 - How to Create and Use Identities in Outlook Express 5

224411 - How to Manually Convert Messages from an Outlook Express 5 Beta 2 Message Store

224416 - How to Change the Order in Which Outlook Express Mail Accounts Are Synchronized

230208 - How to Back Up E-mail Messages in Outlook Express 5

234489 - How to Disable the Picture Slide Show Feature in Outlook Express

242537 - How to Use Outlook 2000 Contacts in Outlook Express 5

252774 - How to Use Outlook 97 Contacts in Outlook Express 5

255186 - How to Configure Outlook Express for CompuServe 2000 IMAP E-Mail

256219 - How to Manually Uninstall and Reinstall Outlook Express 5

Miscellaneous Items

177054 - Internet Shortcuts in Outlook Express Do Not Start Web Browser

168019 - Update Available For Outlook Express Security Issue

176581 - Nothing Happens When You Click a Link in Outlook Express

177233 - Command Line Options for Outlook Express

182977 - Unable to Print a New Message Before You Send It

184340 - Deleted Dial-Up Networking Connection Appears

190056 - Right-Click May Not Display Context Menu

190277 - Outlook Express Opens Messages in a Small Window

190285 - E-mail Address Missing in Message Exported to Outlook 97

191219 - Cannot Find Program.exe...

192422 - IPF in Module Msconv97.dll or <Unknown> Pasting from TechNet CD-ROM

191573 - Message Saved to Drafts Folder Automatically Wraps Text

196473 - New IMAP Folders Do Not Appear in Outlook Express

196518 - Outlook Express Does Not Prompt You for an Identity

216884 - Missing Dialog Box When Message Has a Blank Subject Line

217243 - Message Rules in Outlook Express May Not Work as Expected

217246 - Default Signature Used for News Messages

219270 - Outlook Express Hangs After Sorting by From or Subject Column

219279 - Messages Are Not Marked as Read after Previewing

221095 - Printed Message Contains Different Date and Time

221157 - Forgotten Identity Passwords are nor recoverable

220512 - Spell Check Unavailable in Outlook Express 4.0 After Installing Office 2000

221203 - Unable to Start Outlook Express If Installed Over PGP

221161 - Cannot Add a Signature File in Outlook Express

221192 - Outlook Express Reopens the Folder List After You Close It

222624 - Unable to Remove InfoPane in Outlook Express 5

224010 - Outlook Express 5 Icons Remain After Uninstalling Outlook Express 5

225193 - Unable to Synchronize Palm Pilot to Outlook Express 5

224176 - Cannot Spell Check Messages in Outlook Express 5

226223 - Unable to Print in Outlook Express 4.x

224463 - "Identity Switch Canceled" When Starting Outlook Express

225540 - Unable to Print When the Preview Pane Is Not Enabled

226313 - MSIMN Is Still Running After You uit Outlook Express

228713 - Cannot Save E-Mail Account Password in Outlook Express

227928 - IMAP and HTTP Messages Do Not Appear in Inbox

227958 - Cannot Drag and Drop Folders on Internet Message Access Protocol Server

226985 - Closing Windows Address Book While Composing a Message Closes Message

226987 - Unable to Retrieve E-Mail When Line Contains Only 2 Periods

229134 - Newsgroups Views May Not Work

229606 - Remain Connected to Internet After Send or Receive

230224 - Blank Reply Message Sent if "Include Message in Reply" Not Selected in Outlook Express

230320 - Outlook Express Opens Connection Dialog Box at Startup

229910 - Problems Sending Messages to a Shared E-mail Address

230648 - Image Is Not Displayed in Your HTML Signature File

232001 - The Font Size Changes When You Press ENTER in a New Message

230768 - Mailto Link Does Not Work with % Character and 2 Letters

231682 - Insert Business Card Option Unavailable After Creating a New Business Card

232381 - Outlook Express Mail Folders Appear as Icons on the Desktop

232480 - Outlook Express 5 Remains After Uninstalling Internet Explorer 5

231796 - Cannot Import Outlook Express 4 Messages in Outlook Express 5

232497 - Time Preview in Outlook Express Is Affected by Daylight Saving Time

232738 - Outlook Express May Not Disconnect After Sending and Receiving

232904 - Import of Text File Ignores Last Field

232950 - Spelling and Grammar Options Not Available in Outlook Express 5

233180 - Send to Fax Recipient Opens a New E-mail Message

234061 - Unable to Import Messages from Outlook Express 4.x to Outlook Express 5

235300 - Microsoft Outlook Express 5 Starts Very Slowly

235378 - Text String Automatically Converted to Live Link

235695 - Stationery May Not Be Displayed in E-Mail Messages

240730 - Default Font Size Is Not Applied in a New HTML E-mail Message

240915 - Description of Read Receipts in Outlook Express

241008 - Description of Certificate Revocation Behavior in Outlook Express

241841 - Cannot Create an HTTP or Hotmail Account in Outlook Express 5

242234 - Outlook Express E-mail Signature Does Not Change with Different Accounts

245200 - News Post with Invalid Newsgroup Name Is Not Sent

245199 - Text in the Subject Line Followed by a Colon Is Removed

245419 - Cannot Start Outlook Express 5

245620 - Outlook Express 5 Downloads the Same E-Mail Message Multiple Times

245551 - Unable to Change Font When Using Plain Text in Outlook Express

244441 - Messages Received in Outlook Express Have Different Characters in the Text

244672 - Unable to View New Newsgroup Posting in Outlook Express

246186 - Image May Not Appear in Message Received with Outlook Express

243266 - Uninstalling Internet Explorer 5.01 Removes Outlook Express

248005 - Problems Using Outlook Express and Prodigy Version 5 Software

243716 - "Replies to My Messages" View Is Changed After You View the Inbox

246396 - Outlook Express Startup Informational Pane is blank

248451 - Connection Dialog Box Displayed Even Though You Are Already Connected to the Internet

252603 - Unable to Import E-mail Messages into Outlook Express 5

225254 - MSIMN Causes Invalid Page Fault When You uit Outlook Express

226964 - Cannot Use SPA in Outlook Express to Connect to an SMTP Server

235236 - Cannot Type Text in Outlook Express or Internet Explorer

255827 - Unintended Text May Be Sent with All New Messages

Jim Pickering's Radio Weblog

Recent Arrivals OLEXP: Using Virus Protection Features in Outlook Express 6 (291387) OLEXP: "Most Recently Used" Feature Does Not Retain Encoding Options on the Encoding Menu (296504) OLEXP: Can Send and Receive Messages in Only 32 POP3 Accounts in Outlook Express 6 (296513)

Recent Arrivals

275543 - How to trouble shoot the invalid page fault error message.

Cannot do.....?

HotMail

HTML and Stationery

Address Book (WAB)

Error Messages

How To or How do I.....

Miscellaneous Items

Recent Arrivals
OLEXP: Using Virus Protection Features in Outlook Express 6 (291387)
OLEXP: "Most Recently Used" Feature Does Not Retain Encoding Options on the Encoding Menu (296504)
OLEXP: Can Send and Receive Messages in Only 32 POP3 Accounts in Outlook Express 6 (296513)