如何对将标点符号分类为空格的（单词）进行分词

|| 基于这个问题，很快就结束了：尝试创建一个程序来读取用户输入，然后将数组拆分为单独的单词，我的指针是否全部有效？我认为与其结束讨论，不如说可以做一些额外的工作来帮助OP澄清问题。问题：我想标记用户输入并将标记存储到单词数组中。我想使用标点符号（。，-）作为分隔符，因此将其从令牌流中删除。在C语言中，我将使用“ 0”将数组拆分为令牌，然后手动构建数组。像这样：主要功能：

char **findwords(char *str);

int main()
{
    int     test;
    char    words[100]; //an array of chars to hold the string given by the user
    char    **word;  //pointer to a list of words
    int     index = 0; //index of the current word we are printing
    char    c;

    cout << \"die monster !\";
    //a loop to place the charecters that the user put in into the array  

    do
    {
        c = getchar();
        words[index] = c;
    }
    while (words[index] != \'\\n\');

    word = findwords(words);

    while (word[index] != 0) //loop through the list of words until the end of the list
    {
        printf(\"%s\\n\", word[index]); // while the words are going through the list print them out
        index ++; //move on to the next word
    }

    //free it from the list since it was dynamically allocated
    free(word);
    cin >> test;

    return 0;
}

行标记器：

char **findwords(char *str)
{
    int     size = 20; //original size of the list 
    char    *newword; //pointer to the new word from strok
    int     index = 0; //our current location in words
    char    **words = (char **)malloc(sizeof(char *) * (size +1)); //this is the actual list of words

    /* Get the initial word, and pass in the original string we want strtok() *
     *   to work on. Here, we are seperating words based on spaces, commas,   *
     *   periods, and dashes. IE, if they are found, a new word is created.   */

    newword = strtok(str, \" ,.-\");

    while (newword != 0) //create a loop that goes through the string until it gets to the end
    {
        if (index == size)
        {
            //if the string is larger than the array increase the maximum size of the array
            size += 10;
            //resize the array
            char **words = (char **)malloc(sizeof(char *) * (size +1));
        }
        //asign words to its proper value
        words[index] = newword;
        //get the next word in the string
        newword = strtok(0, \" ,.-\");
        //increment the index to get to the next word
        ++index;
    }
    words[index] = 0;

    return words;
}

对以上代码的任何评论将不胜感激。但是，此外，在C ++中实现此目标的最佳技术是什么？

已邀请:

2 个回复

凡夕

看一下Boost令牌生成器，它在C ++上下文中要比strtok()好得多。

窃誓额

许多问题已经涉及如何在C ++中标记流。示例：如何使用C ++读取文件并获取单词但是更难找到的是如何获得与strtok（）相同的功能：基本上，strtok（）允许您将字符串拆分为一堆用户定义的字符，而C ++流仅允许您使用white space作为分隔符。幸运的是，white space的定义是由语言环境定义的，因此我们可以修改语言环境以将其他字符视为空格，然后使我们能够以更自然的方式标记流。

#include <locale>
#include <string>
#include <sstream>
#include <iostream>

// This is my facet that will treat the ,.- as space characters and thus ignore them.
class WordSplitterFacet: public std::ctype<char>
{
    public:
        typedef std::ctype<char>    base;
        typedef base::char_type     char_type;

        WordSplitterFacet(std::locale const& l)
            : base(table)
        {
            std::ctype<char> const&  defaultCType  = std::use_facet<std::ctype<char> >(l);

            // Copy the default value from the provided locale
            static  char data[256];
            for(int loop = 0;loop < 256;++loop) { data[loop] = loop;}
            defaultCType.is(data, data+256, table);

            // Modifications to default to include extra space types.
            table[\',\']  |= base::space;
            table[\'.\']  |= base::space;
            table[\'-\']  |= base::space;
        }
    private:
        base::mask  table[256];
};

然后，我们可以在本地像这样使用此方面：

    std::ctype<char>*   wordSplitter(new WordSplitterFacet(std::locale()));

    <stream>.imbue(std::locale(std::locale(), wordSplitter));

问题的下一部分是如何将这些单词存储在数组中。好吧，在C ++中您不会。您可以将此功能委托给std :: vector / std :: string。通过阅读您的代码，您将看到您的代码在代码的同一部分中做了两项主要工作。它正在管理内存。它正在标记数据。有一个基本原则“ 8”，您的代码应仅尝试执行以下两项操作之一。它应该执行资源管理（在这种情况下为内存管理），或者应该执行业务逻辑（数据标记）。通过将它们分成不同的代码部分，可以使代码更易于使用和编写。幸运的是，在此示例中，所有资源管理已由std :: vector / std :: string完成，因此使我们能够专注于业务逻辑。如许多次所示，标记流的简单方法是使用运算符>>和字符串。这会将信息流分解为文字。然后，您可以使用迭代器自动遍历整个流，从而对流进行标记化。

std::vector<std::string>  data;
for(std::istream_iterator<std::string> loop(<stream>); loop != std::istream_iterator<std::string>(); ++loop)
{
    // In here loop is an iterator that has tokenized the stream using the
    // operator >> (which for std::string reads one space separated word.

    data.push_back(*loop);
}

如果我们将其与一些标准算法结合起来以简化代码。

std::copy(std::istream_iterator<std::string>(<stream>), std::istream_iterator<std::string>(), std::back_inserter(data));

现在将以上所有内容组合到一个应用程序中

int main()
{
    // Create the facet.
    std::ctype<char>*   wordSplitter(new WordSplitterFacet(std::locale()));

    // Here I am using a string stream.
    // But any stream can be used. Note you must imbue a stream before it is used.
    // Otherwise the imbue() will silently fail.
    std::stringstream   teststr;
    teststr.imbue(std::locale(std::locale(), wordSplitter));

    // Now that it is imbued we can use it.
    // If this was a file stream then you could open it here.
    teststr << \"This, stri,plop\";

    cout << \"die monster !\";
    std::vector<std::string>    data;
    std::copy(std::istream_iterator<std::string>(teststr), std::istream_iterator<std::string>(), std::back_inserter(data));

    // Copy the array to cout one word per line
    std::copy(data.begin(), data.end(), std::ostream_iterator<std::string>(std::cout, \"\\n\"));
}

要回复问题请先登录或注册

如何对将标点符号分类为空格的（单词）进行分词

2 个回复

发起人

c++

locale

tokenize

问题状态

如何对将标点符号分类为空格的（单词）进行分词

与内容相关的链接

2 个回复

发起人

c++

locale

tokenize

问题状态