Home » Php » php – glob() can't find file names with multibyte characters on Windows?

php – glob() can't find file names with multibyte characters on Windows?

Posted by: admin April 23, 2020 Leave a comment

Questions:

I’m writing a file manager and need to scan directories and deal with renaming files that may have multibyte characters. I’m working on it locally on Windows/Apache PHP 5.3.8, with the following file names in a directory:

  • filename.jpg
  • имяфайла.jpg
  • file件name.jpg
  • פילענאַמע.jpg
  • 文件名.jpg

Testing on a live UNIX server woked fine. Testing locally on Windows using glob('./path/*') returns only the first one, filename.jpg.

Using scandir(), the correct number of files is returned at least, but I get names like ?????????.jpg (note: those are regular question marks, not the � character.

I’ll end up needing to write a “search” feature to search recursively through the entire tree for filenames matching a pattern or with a certain file extension, and I assumed glob() would be the right tool for that, rather than scan all the files and do the pattern matching and array building in the application code. I’m open to alternate suggestions if need be.

Assuming this was a common problem, I immediately searched Google and Stack Overflow and found nothing even related. Is this a Windows issue? PHP shortcoming? What’s the solution: is there anything I can do?

Addendum: Not sure how related this is, but file_exists() is also returning FALSE for these files, passing in the full absolute path (using Notepad++, the php file itself is UTF-8 encoding no BOM). I’m certain the path is correct, as neighboring files without multibyte characters return TRUE.

EDIT: glob() can find a file named filename-äöü.jpg. Previously in my .htaccess file, I had AddDefaultCharset utf-8, which I didn’t consider before. filename-äöü.jpg was printing as filename-���.jpg. The only effect removing that htaccess line seemed to have was now that file name prints normally.

I’ve deleted the .htaccess file completely, and this is my actual test script in it’s entirety (I changed a couple of file names from the original post):

print_r(scandir('./uploads/')); 
print_r(glob('./uploads/*'));

Output locally on Windows:

Array
(
    [0] => .
    [1] => ..
    [2] => ??? ?????.jpg
    [3] => ???.jpg
    [4] => ?????????.jpg
    [5] => filename-äöü.jpg
    [6] => filename.jpg
    [7] => test?test.jpg
)
Array
(
    [0] => ./uploads/filename-äöü.jpg
    [1] => ./uploads/filename.jpg
)

Output on remote UNIX server:

Array
(
    [0] => .
    [1] => ..
    [2] => filename-äöü.jpg
    [3] => filename.jpg
    [4] => test이test.jpg
    [5] => имя файла.jpg
    [6] => פילענאַמע.jpg
    [7] => 文件名.jpg
)
Array
(
    [0] => ./uploads/filename-äöü.jpg
    [1] => ./uploads/filename.jpg
    [2] => ./uploads/test이test.jpg
    [3] => ./uploads/имя файла.jpg
    [4] => ./uploads/פילענאַמע.jpg
    [5] => ./uploads/文件名.jpg
)

Since this is a different server, regardless of platform – configuration could be different so I’m not sure what to think, and I can’t fully pin it on Windows yet (could be my PHP installation, ini settings, or Apache config). Any ideas?

How to&Answers:

It looks like the glob() function depends on how your copy of PHP was built and whether it was compiled with a unicode-aware WIN32 API (I don’t believe the standard builid is.

Cf. http://www.rooftopsolutions.nl/blog/filesystem-encoding-and-php

Excerpt from comments on the article:

Philippe Verdy 2010-09-26 8:53 am

The output from your PHP installation on Windows is easy to explain :
you installed the wrong version of PHP, and used a version not
compiled to use the Unicode version of the Win32 API. For this reason,
the filesystem calls used by PHP will use the legacy “ANSI” API and so
the C/C++ libraries linked with this version of PHP will first try to
convert yout UTF-8-encoded PHP string into the local “ANSI” codepage
selected in the running environment (see the CHCP command before
starting PHP from a command line window)

Your version of Windows is MOST PROBABLY NOT responsible of this weird
thing. Actually, this is YOUR version of PHP which is not compiled
correctly, and that uses the legacy ANSI version of the Win32 API (for
compatibility with the legacy 16-bit versions of Windows 95/98 whose
filesystem support in the kernel actually had no direct support for
Unicode, but used an internal conversion layer to convert Unicode to
the local ANSI codepage before using the actual ANSI version of the
API).

Recompile PHP using the compiler option to use the UNICODE version of
the Win32 API (which should be the default today, and anyway always
the default for PHP installed on a server that will NEVER be Windows
95 or Windows 98…)

Then Windows will be able to store UTF-16 encoded filenames (including
on FAT32 volumes, even if, on these volumes, it will also generate an
aliased short name in 8.3 format using the filesystem’s default
codepage, something that can be avoided in NTFS volumes).

All what you describe are problems of PHP (incorrect porting to
Windows, or incorrect system version identification at runtime) :
reread the README files coming with PHP sources explaining the
compilation flags. I really think that the makefile on Windows should
be able to configure and autodetect if it really needs to use ONLY the
ANSI version of the API. If you are compiling it for a server, make
sure that the Configure script will effectively detect the full
support of the UNICODE version of the Win32 aPI and will use it when
compiling PHP and when selecting the runtime libraries to link.

I use PHP on Windows, correctly compiled, and I absolutely DON’T know
the problems you cite in your article.

Let’s forget now forever these non-UNICODE versions of the Win32
API (which are using inconsistantly the local ANSI codepage for the
Windows graphical UI, and the OEM codepage for the filesystem APIs,
the DOS/BIOS-compatible APIs, the Console APIs) : these non-Unicode
versions of the APIs are even MUCH slower and more costly than the
Unicode versions of the APIs, because they are actually translating
the codepage to Unicode before using the core Unicode APIs (the
situation on Windows NT-based kernels is exactly the reverse from the
situation on versions of Windows based on a virtual DOS extender, such
as Windows 95/98/ME).

When you don’t use the native version of the API, your API call will
pass through a thunking layer that will transcode the strings between
Unicode and one of the legacy ANSI or CHCP-selected OEM codepages, or
the OEM codepage hinted on the filesystem: this requires additional
temporary memory allocation within the non-native version of the Win32
API. This takes additional time to convert things before doing the
actual work by calling the native API.

In summary: the PHP binary you install on Windows MUST be different
depending on if you compiled it for Windows 95/98/SE (or the old
Win16s emulation layer for Windows 3.x, which had a very mimimum
support of UTF-8, only to support the Unicode subsets of Unicode used
by the ANSI and OEM codapges selected when starting Windows from a DOS
extender) or if it was compiled for any other version of Windows based
on the NT kernel.

The best proof that this is a problem of PHP and not Windows, is that
your weird results will NOT occur in other languages like C#,
Javascript, VB, Perl, Ruby… PHP has a very bad history in tracking
versions (and too many historical source code quirks and wrong
assumptions that should be disabled today, and an inconsistant library
that has inherited all those quirks initially made in old versions of
PHP for old versions of Windows that are even no longer officially
supported, by Microsoft or even by PHP itself !).

In other words : RTM ! Or download and install a binary version of
PHP for Windows precompield with the correct settings : I really think
that PHP should distribute Windows binaries already compiled by
default for the Unicode version of the Win32 API, and using the
Unicode version of the C/C++ libraries : internally the PHP code will
convert its UTF-8 strings to UTF-16 before calling the Win32 API, and
back from UTF-16 to UTF-8 when retrieving Win32 results, instead of
converting PHP’s internal UTF-8 strings back/to the local OEM codepage
(for the filesystem calls) or the local ANSI codepage (for all other
Win32 APIs, including the registry or process).

Answer:

PHP on windows does not use the Unicode API yet. So you have to use the runtime encoding (whatever it is) to be able to deal with non ascii charset.

Answer:

Starting with PHP 7.1 long and UTF-8 paths on Windows are supported directly in the core.