How `[System.Console]::OutputEncoding/InputEncoding` with Python?

Posted on

Problem :

Under Powershell v5, Windows 8.1, Python 3. Why these fails and how to fix?

[system.console]::InputEncoding = [System.Text.Encoding]::UTF8; 
[system.console]::OutputEncoding = [System.Text.Encoding]::UTF8; 
"import sys
" | 
sc -Encoding utf8; 
[char]0x0422+[char]0x0415+[char]0x0421+[char]0x0422+"`n" | py -3


Active code page: 65001

Solution :

You are piping data into Python; at that point Python’s stdin is no longer attached to a TTY (your console) and won’t guess at what the encoding might be. Instead, the default system locale is used; on your system that’s cp1251 (the Windows Latin-1-based codepage).

Set the PYTHONIOENCODING environment variable to override:

If this is set before running the interpreter, it overrides the encoding used for stdin/stdout/stderr, in the syntax encodingname:errorhandler. Both the encodingname and the :errorhandler parts are optional and have the same meaning as in str.encode().

PowerShell doesn’t appear to support per-command-line environment variables the way UNIX shells do; the easiest is to just set the variable first:


or even

Set-Item Env:PYTHONIOENCODING "cp65001"

as the Windows UTF-8 codepage is apparently not quite UTF-8 really, depending on the Windows version and on wether or not pipe redirection is used.

Why not embed CPython in powershell?! CPython is so easy to embed, and powershell is very good REPL to play with .NET and COM objects. Here is a simple introduction to using pythonnet from PowerShell. Note how encoding is automatically propagated from powershell to python.

Windows PowerShell
Copyright (C) 2015 Microsoft Corporation. All rights reserved.

PS C:Usersdenfromufa> [system.console]::InputEncoding = [System.Text.Encoding]::UTF8;
PS C:Usersdenfromufa> [system.console]::OutputEncoding = [System.Text.Encoding]::UTF8;
PS C:Usersdenfromufa> [Reflection.Assembly]::LoadFile("C:PythonMiniconda3_64bLibsite-packagesPython.Runtime.dll")

GAC    Version        Location
---    -------        --------
False  v4.0.30319     C:PythonMiniconda3_64bLibsite-packagesPython.Runtime.dll

PS C:Usersdenfromufa> $gil = [Python.Runtime.Py]::GIL()
PS C:Usersdenfromufa> $sys=[Python.Runtime.Py]::Import("sys")
PS C:Usersdenfromufa> $sys.stdin.encoding.ToString()
PS C:Usersdenfromufa> $sys.stdout.encoding.ToString()
PS C:Usersdenfromufa> $gil.Dispose()
PS C:Usersdenfromufa> [Python.Runtime.PythonEngine]::Shutdown()
PS C:Usersdenfromufa>


Here is snek package that was released by one of powershell developers for embedding Python in powershell:

Leave a Reply

Your email address will not be published. Required fields are marked *