UTF16 for Microsoft Windows

If you found a bug in our library or on our website, please report it in this section. In this forum you can also make concrete suggestions or feature requests.

Moderators: CEGUI MVP, CEGUI Team

itten
Just popping in
Just popping in
Posts: 1
Joined: Thu Feb 18, 2010 04:13

UTF16 for Microsoft Windows

Postby itten » Thu Feb 18, 2010 04:25

Hi
i had a bug on Microsoft Windows when opening files with non Latin symbols. There is std::ifstream in CEGUIDefaultResourceProvider.cpp that opens all files and it wants ANSI path on MS Windows, not utf8. So, as you may suggest it fails to open any file.
I changed file to this and now it works fine

Code: Select all



/***********************************************************************
filename:    CEGUIDefaultResourceProvider.cpp
created:   8/7/2004
author:      James '_mental_' O'Sullivan

purpose:   Implements the Resource Manager common functionality
*************************************************************************/
/***************************************************************************
*   Copyright (C) 2004 - 2006 Paul D Turner & The CEGUI Development Team
*
*   Permission is hereby granted, free of charge, to any person obtaining
*   a copy of this software and associated documentation files (the
*   "Software"), to deal in the Software without restriction, including
*   without limitation the rights to use, copy, modify, merge, publish,
*   distribute, sublicense, and/or sell copies of the Software, and to
*   permit persons to whom the Software is furnished to do so, subject to
*   the following conditions:
*
*   The above copyright notice and this permission notice shall be
*   included in all copies or substantial portions of the Software.
*
*   THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
*   EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
*   MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
*   IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR
*   OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
*   ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
*   OTHER DEALINGS IN THE SOFTWARE.
***************************************************************************/
#include "CEGUIDefaultResourceProvider.h"
#include "CEGUIExceptions.h"

#include <fstream>
#include <iostream>

#if defined(__WIN32__) || defined(_WIN32)
#   include <io.h>
#   include <windows.h>
#   include <string>

std::wstring Utf8ToUtf16( const std::string& utf8text )
{
   int textLen = MultiByteToWideChar(CP_UTF8, 0, utf8text.c_str(), utf8text.size()+1, NULL, 0);
   if(textLen == 0)
   {
      throw CEGUI::InvalidRequestException("Utf8ToUtf16 - MultiByteToWideChar failed");
   }

   std::wstring wideStr(textLen, 0);
   MultiByteToWideChar(CP_UTF8, 0, utf8text.c_str(), utf8text.size()+1, &wideStr[0], wideStr.size());
   return wideStr;
}

#else
#   include <sys/types.h>
#   include <sys/stat.h>
#   include <dirent.h>
#   include <fnmatch.h>
#endif

// Start of CEGUI namespace section
namespace CEGUI
{

   void DefaultResourceProvider::loadRawDataContainer(const String& filename, RawDataContainer& output, const String& resourceGroup)
   {
      if (filename.empty())
      {
         throw InvalidRequestException(
            "DefaultResourceProvider::load - Filename supplied for data loading must be valid");
      }

      String final_filename(getFinalFilename(filename, resourceGroup));

#if defined(__WIN32__) || defined(_WIN32)
      std::wstring Filename = Utf8ToUtf16(final_filename.c_str());
      FILE* file = _wfopen(Filename.c_str(), L"rb");
#else
      FILE* file = fopen(final_filename.c_str(), "rb");
#endif

      if(file == 0)
      {
         throw InvalidRequestException(
            "DefaultResourceProvider::load - " + filename + " does not exist");
      }
      fseek(file, 0, SEEK_END);
      long size = ftell(file);
      fseek(file, 0, SEEK_SET);

      unsigned char* buffer = new unsigned char [size];

      size_t SizeRead = fread(buffer, sizeof(char), size, file);
      fclose(file);
      
      if(SizeRead != size)
      {
         delete [] buffer;
         throw GenericException(
            "DefaultResourceProvider::loadRawDataContainer - Problem reading " + filename);
      }

      output.setData(buffer);
      output.setSize(size);
   }

   void DefaultResourceProvider::unloadRawDataContainer(RawDataContainer& data)
   {
      uint8* ptr = data.getDataPtr();
      delete [] ptr;
      data.setData(0);
      data.setSize(0);
   }

   void DefaultResourceProvider::setResourceGroupDirectory(const String& resourceGroup, const String& directory)
   {
      if (directory.length() == 0)
         return;

#if defined(_WIN32) || defined(__WIN32__)
      // while we rarely use the unportable '\', the user may have
      const String separators("\\/");
#else
      const String separators("/");
#endif

      if (String::npos == separators.find(directory[directory.length() - 1]))
         d_resourceGroups[resourceGroup] = directory + '/';
      else
         d_resourceGroups[resourceGroup] = directory;
   }

   const String& DefaultResourceProvider::getResourceGroupDirectory(const String& resourceGroup)
   {
      return d_resourceGroups[resourceGroup];
   }

   void DefaultResourceProvider::clearResourceGroupDirectory(const String& resourceGroup)
   {
      ResourceGroupMap::iterator iter = d_resourceGroups.find(resourceGroup);

      if (iter != d_resourceGroups.end())
         d_resourceGroups.erase(iter);
   }

   String DefaultResourceProvider::getFinalFilename(const String& filename, const String& resourceGroup) const
   {
      String final_filename;

      // look up resource group directory
      ResourceGroupMap::const_iterator iter =
         d_resourceGroups.find(resourceGroup.empty() ? d_defaultResourceGroup : resourceGroup);

      // if there was an entry for this group, use it's directory as the
      // first part of the filename
      if (iter != d_resourceGroups.end())
         final_filename = (*iter).second;

      // append the filename part that we were passed
      final_filename += filename;

      // return result
      return final_filename;
   }

   size_t DefaultResourceProvider::getResourceGroupFileNames(
      std::vector<String>& out_vec,
      const String& file_pattern,
      const String& resource_group)
   {
      // look-up resource group name
      ResourceGroupMap::const_iterator iter =
         d_resourceGroups.find(resource_group.empty() ? d_defaultResourceGroup :
         resource_group);
      // get directory that's set for the resource group
      const String dir_name(iter != d_resourceGroups.end() ? (*iter).second : "./");

      size_t entries = 0;

      // Win32 code.
#if defined(__WIN32__) || defined(_WIN32)
      intptr_t f;
      struct _finddata_t fd;
      if ((f = _findfirst((dir_name + file_pattern).c_str(), &fd)) != -1)
      {
         do
         {
            if ((fd.attrib & _A_SUBDIR))
               continue;

            out_vec.push_back(fd.name);
            ++entries;
         } while (_findnext(f, &fd) == 0);

         _findclose(f);
      }
      // Everybody else
#else
      DIR* dirp;
      if ((dirp = opendir(dir_name.c_str())))
      {
         struct dirent* dp;
         while ((dp = readdir(dirp)))
         {
            const String filename(dir_name + dp->d_name);
            struct stat s;

            if ((stat(filename.c_str(), &s) == 0) &&
               S_ISREG(s.st_mode) &&
               (fnmatch(file_pattern.c_str(), dp->d_name, 0) == 0))
            {
               out_vec.push_back(dp->d_name);
               ++entries;
            }
         }

         closedir(dirp);
      }
#endif
      return entries;
   }

} // End of  CEGUI namespace section



Actually i changed loadRawDataContainer function and include some headers.
Hope this code will go to trunk.

User avatar
CrazyEddie
CEGUI Project Lead
Posts: 6760
Joined: Wed Jan 12, 2005 12:06
Location: England
Contact:

Re: UTF16 for Microsoft Windows

Postby CrazyEddie » Thu Feb 18, 2010 09:38

Hi,

Thanks for this, though can I ask that you resubmit this as a patch in unified diff format to our mantis tracker; we do not accept modifications in any other form because it's hard to see - at a glance - exactly what's changed, and by submitting to the tracker we ensure that the submission is not lost and forgotten about on the forum.

For more information as regards to contributing code to CEGUI, please see Contributing to the development of CEGUI in the docs.

Thanks,

CE.

User avatar
CrazyEddie
CEGUI Project Lead
Posts: 6760
Joined: Wed Jan 12, 2005 12:06
Location: England
Contact:

Re: UTF16 for Microsoft Windows

Postby CrazyEddie » Sat Feb 20, 2010 15:36

Hi,

This change has now been comitted in the stable branch: branches/v0-7. It will hit trunk with the next merge (tomorrow or Monday).

A small request for future patches: please try to minimise the amount of lines that get picked up in the patch by disregarding whitespace changes ;) Having said this, I do not know how to do that except via the command line diff (which would be, for example: 'diff -dwu <file1> <file2>' or via 'svn diff' via the -x option to pass options to the external diff command), I looked in TortoiseSVN but did not see an option to achieve the same results :(

CE.


Return to “Bug Reports, Suggestions, Feature Requests”

Who is online

Users browsing this forum: No registered users and 15 guests