boost::tokenizer and BOOST_FOREACH

I looked at the documentation for what all could foreach macro support? The list didn't have boost tokenizer in it, which was expected but it said something that meant, it should work: "The support for STL containers is very general; anything that looks like an STL container counts. If it has nested iterator and const_iterator types and begin() and end() member functions, BOOST_FOREACH will automatically know how to iterate over it."

So, I tried it out and yes, it worked. The following compiled and worked fine with VC++ 2005. The code should be easy to understand that basically, reads a file line by line and tokenizes the lines assuming a space or punctuation as the seperator.

[code]
    //standard headers
    #include <iostream>
    #include <fstream>
    #include <string>

    //boost headers
    #include <boost/tokenizer.hpp>
    #include <boost/foreach.hpp>

    //make BOOST_FOREACH prettier! ;-) :-)
    #define foreach BOOST_FOREACH

    void tokenizeLines(std::istream& inputStream)
    {
        if(inputStream)
        {
            std::string line;
            while(std::getline(inputStream, line, '\n'))
            {
                boost::tokenizer<> tokens(line);
                foreach(const std::string & str, tokens)
                {
                    std::cout << str << "\n";
                }
            }
        }
        else
        {
            std::cerr << "Error: Invalid stream object\n";
        }
    }

    int main()
    {
        std::ifstream inputStream("C:\\testfiles\\myfile.txt");
        tokenizeLines(inputStream);
    }