java - 表达式分词器

Question

我正在尝试创建一个读取表达式的分词器。当它获得 token 时，我打印类型和词素，它可以是运算符的数字。它目前单独打印每个字符，但我需要它一起打印两位数和十进制数字。这是我的代码。

    public class Tokenizer {
String type;
String lexeme;
/**
 * @param args the command line arguments
 */
void getToken(String expression, int i) {

        if (expression.charAt(i) == '+')
        {
            type = "operator";
            lexeme = "+";       
        }
        else if (expression.charAt(i) == '-')
        {
            type = "operator";
            lexeme = "-";
        }
        else if (expression.charAt(i) == '*')
        {
            type = "operator";
            lexeme = "*";
        }
        else if (expression.charAt(i) == '/')
        {
            type = "operator";
            lexeme = "/";
        }
        else if (expression.charAt(i) == ' ')
        {
            ; //empty statement. Does nothing if the character is white space
        }
        else 
        {
            type = "number";
            lexeme = Character.toString(expression.charAt(i));
        }
}
public static void main(String[] args) {
    // TODO code application logic here
    String expression = "3+66*2.5";
    Tokenizer tokenizer = new Tokenizer();
    for (int i = 0; i < expression.length(); i++)
    {
        tokenizer.getToken(expression, i);
        System.out.print("Type: " + tokenizer.type + "\t" + "Lexeme: " + tokenizer.lexeme + "\n");
    }
  }
}

示例输出 - 实际

Type: number    Lexeme: 3
Type: operator  Lexeme: +
Type: number    Lexeme: 6
Type: number    Lexeme: 6
Type: operator  Lexeme: *
Type: number    Lexeme: 2
Type: number    Lexeme: .
Type: number    Lexeme: 5

示例输出 - 预期

Type: number    Lexeme: 3
Type: operator  Lexeme: +
Type: number    Lexeme: 66
Type: operator  Lexeme: *
Type: number    Lexeme: 2.5

Answer 1

此语句会覆盖每次执行时的词法

 lexeme = Character.toString(expression.charAt(i));

应该是

lexeme += Character.toString(expression.charAt(i));

即识别出单个数字后，将其添加到词位中。

除此之外，您的 getToken 方法不正确，那么通过将 i 传递给它，您会假设词素只有一个字符长，实际上它可以是 1或更多，例如 66 或 2.5。所以你需要重新设计你的 Tokenizer 类。您应该使用表达式初始化 Tokenizer 对象，然后让它有一个无参数方法 getNextToken()，该方法根据需要进行多次迭代来识别 token ，然后返回它。它还应该有一个方法来检查是否还有任何输入，例如 hasMoreTokens()。

这是您可以构建的东西

public class Tokenizer {
    int pos;
    char[] expression;

    Tokenizer(String expression) {
        this.expression = expression.toCharArray();
        this.pos = 0;
    }

    enum Type { OPERATOR, NUMBER, UNKNOWN }

    class Lexeme {
        String type, token;
        Lexeme(String type, String token) {
            this.type = type;
            this.token = token;
        }
    }

    Lexeme getNextToken() {
        StringBuilder token = new StringBuilder();
        boolean endOfToken = false;
        Type type = Type.UNKNOWN;
        while (!endOfToken && hasMoreTokens()) {
            while(expression[pos] == ' ' && hasMoreTokens())
                pos++;
            switch (expression[pos]) {
                case '+':
                case '-':
                case '*':
                case '/':
                    if(type != Type.NUMBER) {
                        type = Type.OPERATOR;
                        token.append(expression[pos]);
                        pos++;
                    }
                    endOfToken = true;
                    break;
                case ' ':
                    endOfToken = true;
                    pos++;
                    break;
                default:
                    if(Character.isDigit(expression[pos]) || expression[pos] == '.') {
                        token.append(expression[pos]);
                        type = Type.NUMBER;
                    } else {
                        System.out.println("Systax error at position: " + pos);
                    }
                    pos++;
                    break;
            }
        }
        return new Lexeme(type.name().toLowerCase(), token.toString());
    }

    boolean hasMoreTokens() {
        return pos < expression.length;
    }

    public static void main(String[] args) {
        String expression = "3+66*2.5";
        Tokenizer tokenizer = new Tokenizer(expression);
        while (tokenizer.hasMoreTokens()) {
            Lexeme nextToken = tokenizer.getNextToken();
            System.out.print("Type: " + nextToken.type + "\tLexeme: " + nextToken.token + "\n");
        }
    }
}

输出

Type: number    Lexeme: 3
Type: operator  Lexeme: +
Type: number    Lexeme: 66
Type: operator  Lexeme: *
Type: number    Lexeme: 2.5